methods moment hypothesis vol. 32 no.1 fall/winter 2020 introducing methods moment nina exner, phd, mls assistant professor / research data librarian virginia commonwealth university richmond, va a new column is being introduced to hypothesis, called methods moment. as the name suggests, the column will focus on brief discussions of methodological topics for librarians who are doing, or wish to do, their own research. librarians typically do not delve deeply into research class in library school or hear research concepts sprinkled throughout their library school courses. nevertheless, librarians often want or need to do research! regrettably, much of the methods literature is not geared towards practitioner-researchers who want to start or expand their research career. to support librarians new to research or expanding their interests, methods moment wants to introduce various methods for librarians to explore and apply. methods moment aims to give short, practical introductions to various methods of use to medical librarians. our goal is to help librarians to familiarize themselves with various methods, and to think about which methods will or won’t work for answering their own research questions. each column should serve the goal of increasing the breadth of methodological concepts that readers are familiar with. submissions to the column are subject to blind peer review. if prospective authors would like advice on developing an idea into a column, column editor nina exner, nexner@vcu.edu, will give guidance and feedback to authors about style and scope-fit. the hypothesis journal editor(s) have final discretion over the acceptability of any column submitted. column format methods moment columns should be 1,250-2,000 words per column. they should be written in a casual, practical style. a given column will not aim to be comprehensive of a topic; instead, it should give enough detail to familiarize the reader with the topic, help them imagine the topic’s practical application in health librarian research, and lead interested readers to explore further. the structured abstract serves as a place for librarians to determine whether a given method is relevant to them at all. it covers: • objective: what is this method intended to do or discover? • approach: how does the method work? what describes the unique features or technique of this method? • data type(s) used: what data goes into this method? is it interviews, health records, national surveys, big data, etc.? mailto:nexner@vcu.edu methods moment hypothesis vol. 32 no.1 fall/winter 2020 • strengths: what is uniquely good about this method? • limitations: what are common limits or issues to be aware of when using this method? each methods moment column is intended to provide (1) an overview of the topic; (2) a practical example or illustrative mini-case; and (3) resources for further exploration. • overview: the overview should include a short definition or description of the method or topic. it will then explain the method or topic, its purpose or role in producing research findings, and why it might be of interest to health science librarians in their research. the overview should stand on its own for readers who are curious but do not want to explore further. • example: the example section should illustrate how the method or topic plays out during research in practice. the example might be a narrative description of the topic in action, a personal experience of application, or a mock case study describing an imaginary scenario of when and how the topic would be applied. • resources: the resources section should point readers to places for further exploration and self-guided learning. this should be a curated list of materials for interested learners. wherever possible, if learning materials exist, they should be pointed to in the resources rather than trying to include instructional content in the overview. most of the resources should be freely accessible, rather than subscription resources with limited availability. however, alternative formats may be considered! good alternative formats could include compare-and-contrast between two related methods, brief reports of methods focusing on ideas for self-guided exploration, or summaries of discussion events among librarians who use a certain method. creative approaches to addressing the needs of practitioners new to research are encouraged. invited commentary hypothesis vol. 32 no.1 fall/winter 2020 message of solidarity to research community members alexander j. carroll, msls, ahip librarian for stem research jean and alexander heard libraries vanderbilt university nashville, tennessee dear research caucus community members, as the chair of the research caucus, i want to express my solidarity and love for our black, indigenous, and people of color (bipoc) library community members. i can only imagine the pain, trauma, and exhaustion that you have been experiencing as you have seen the images of anti-black violence perpetrated across the united states over the last week. the horrifying and heartbreaking images that we have seen across the country reflect the systemic racial inequalities that have been a pervasive and persistent aspect of the united states since its founding, and indeed preceding even that. as medical librarians, it’s critical that we recognize that libraries are not immune from the systemic inequities that permeate our society; rather, libraries often reflect and reinforce those inequitable values through our systems, spaces, collections, budgets, and through the makeup of our staff. for white members of our research caucus community who have found the images of police brutality shocking, i would urge you to spend some time listening to our bipoc colleagues. while seeing this violence up close may be news to many of us, it is not news to our bipoc colleagues. for members of our community who seek to criticize the ongoing protests against police violence rather than criticizing the over-aggressive policing that has led to protests, i would encourage you to read dr. martin luther king jr.’s “letter from a birmingham jail.” written just over 57 years ago, dr. king’s words remain as convicting as ever: “first, i must confess that over the past few years i have been gravely disappointed with the white moderate. i have almost reached the regrettable conclusion that the negro's great stumbling block in his stride toward freedom is not the white citizen's counciler or the ku klux klanner, but the white moderate, who is more devoted to "order" than to justice; who prefers a negative peace which is the absence of tension to a positive peace which is the presence of justice; who constantly says: "i agree with you in the goal you seek, but i cannot agree with your methods of direct action"; who paternalistically believes he can https://www.africa.upenn.edu/articles_gen/letter_birmingham.html https://www.africa.upenn.edu/articles_gen/letter_birmingham.html invited commentary hypothesis vol. 32 no.1 fall/winter 2020 set the timetable for another man's freedom; who lives by a mythical concept of time and who constantly advises the negro to wait for a "more convenient season." shallow understanding from people of good will is more frustrating than absolute misunderstanding from people of ill will. lukewarm acceptance is much more bewildering than outright rejection. i had hoped that the white moderate would understand that law and order exist for the purpose of establishing justice and that when they fail in this purpose they become the dangerously structured dams that block the flow of social progress. i had hoped that the white moderate would understand that the present tension in the south is a necessary phase of the transition from an obnoxious negative peace, in which the negro passively accepted his unjust plight, to a substantive and positive peace, in which all men will respect the dignity and worth of human personality. actually, we who engage in nonviolent direct action are not the creators of tension. we merely bring to the surface the hidden tension that is already alive. we bring it out in the open, where it can be seen and dealt with. like a boil that can never be cured so long as it is covered up but must be opened with all its ugliness to the natural medicines of air and light, injustice must be exposed, with all the tension its exposure creates, to the light of human conscience and the air of national opinion before it can be cured.” for those of us seeking to learn more about how we can get involved in anti-racism work, i would encourage you to explore some of the resources highlighted in the following lists: • “disrupting whiteness in libraries and librarianship: a reading list,” a curated list of lis scholarship via the university of wisconsin-madison libraries: https://www.library.wisc.edu/gwslibrarian/bibliographies/disrupting-whiteness-inlibraries/ • “anti-racism resources for all ages,” a curated reading list developed by nicole cooke, the augusta baker chair in childhood literacy at the university of south carolina school of library and information science: https://padlet.com/nicolethelibrarian/nbasekqoazt336co • “anti-racist reading list from ibram x. kendi” (via the chicago public library): https://chipublib.bibliocommons.com/list/share/204842963/1357692923 • we here’s “anti-racism resources for white people”: https://docs.google.com/document/d/1brlf2_zhne86sggha6-vlboqgiritwctugsfkie5fs/preview?pru=aaabcot0otc*qoqtmq_bibdnbdzioyfd-g https://nam04.safelinks.protection.outlook.com/?url=https%3a%2f%2fwww.library.wisc.edu%2fgwslibrarian%2fbibliographies%2fdisrupting-whiteness-in-libraries%2f&data=02%7c01%7calexander.j.carroll%40vanderbilt.edu%7c20e4aad8890b484b194808d8070a4ca5%7cba5a7f39e3be4ab3b45067fa80faecad%7c0%7c0%7c637267088102501188&sdata=wuig9dbfoliltpxmnleejzrgoklvigojxwqmm5ahjpg%3d&reserved=0 https://nam04.safelinks.protection.outlook.com/?url=https%3a%2f%2fwww.library.wisc.edu%2fgwslibrarian%2fbibliographies%2fdisrupting-whiteness-in-libraries%2f&data=02%7c01%7calexander.j.carroll%40vanderbilt.edu%7c20e4aad8890b484b194808d8070a4ca5%7cba5a7f39e3be4ab3b45067fa80faecad%7c0%7c0%7c637267088102501188&sdata=wuig9dbfoliltpxmnleejzrgoklvigojxwqmm5ahjpg%3d&reserved=0 https://nam04.safelinks.protection.outlook.com/?url=https%3a%2f%2fpadlet.com%2fnicolethelibrarian%2fnbasekqoazt336co&data=02%7c01%7calexander.j.carroll%40vanderbilt.edu%7c20e4aad8890b484b194808d8070a4ca5%7cba5a7f39e3be4ab3b45067fa80faecad%7c0%7c0%7c637267088102501188&sdata=u%2fvbaqf7babej5%2fbbqzfvdbih%2bmyw52ra1fpjixuyek%3d&reserved=0 https://nam04.safelinks.protection.outlook.com/?url=https%3a%2f%2fchipublib.bibliocommons.com%2flist%2fshare%2f204842963%2f1357692923&data=02%7c01%7calexander.j.carroll%40vanderbilt.edu%7c20e4aad8890b484b194808d8070a4ca5%7cba5a7f39e3be4ab3b45067fa80faecad%7c0%7c0%7c637267088102511185&sdata=itwq1vpnol5dlydeuzcptf9408v61yp4ouytoykzemu%3d&reserved=0 https://docs.google.com/document/d/1brlf2_zhne86sggha6-vlbo-qgiritwctugsfkie5fs/preview?pru=aaabcot0otc*qoqtmq_bibdnbdzioyfd-g https://docs.google.com/document/d/1brlf2_zhne86sggha6-vlbo-qgiritwctugsfkie5fs/preview?pru=aaabcot0otc*qoqtmq_bibdnbdzioyfd-g invited commentary hypothesis vol. 32 no.1 fall/winter 2020 however, listening and educating ourselves is not enough. speaking out about systemic racism only during widespread anti-black violence like we’ve seen over the last week is not enough, either. the research caucus’ role is to encourage mla members to use research to answer the questions that matter. as members of the research caucus, i encourage each of you to consider asking questions about systemic inequities within medical libraries. how do our systems and services reflect and reinforce systemic inequality, and what can we do to reform those systems to create more equitable libraries, institutions, and communities? these questions are challenging; asking them will require courage and will lead to discomfort. but if we want to live up to the spirit of “i am mla,” which mla past president beverly murphy describes as “the collective understanding…that it is up to us, as members and as volunteers, to do what needs to be done for our association,” we must rise to meet that challenge. sincerely, alexander j. carroll, msls, ahip mla research caucus chair 2020-2021 awards hypothesis vol. 31 no.1 fall/winter 2019 hypothesis research training quality award this year marks the inaugural offering of the hypothesis research training quality award. this award is intended to recognize quality (online) trainings that cover the conduct or support of research and were offered to health sciences librarians between august 2016-august 2018. the scope/criteria for this award include: • training sessions created by an individual or team of practitioners. • training topic concerning the conduct or support of research. • training must have been offered to health sciences librarians in the two years prior to the nomination deadline. • training session must be currently available online. qualifying trainings were evaluated based on a rubric that included: audience, learning objectives, relevance to profession, accessibility & design, command of subject matter, and topical focus. many thanks to those who participated in the peer review process of these trainings. awards were available for three trainings and the following were awarded following review: 1st place – 8 tips for using metrics in research evaluation • karen gutzman impact & evaluation librarian, galter health sciences library, northwestern university, chicago il • patricia l. smith senior engagement manager, altmetric, chicago il • kristi l. holmes director, galter health sciences library, northwestern university, chicago il this webinar offers librarians a quick, practical way to stay on top of the up-andcoming area of research evaluation and the role librarians can play in it. attendees will learn how to easily transform their skills in expert searching and metadata into the valuable skills needed in the research evaluation process. presenters address various types of publication data, review categories of bibliometric indicators, highlight the uses of alternative metrics, cover data sources and tools, and touch on best practices for evaluation. available at: http://www.medlib-ed.org/products/1535/8-tips-for-using-metrics-inresearch-evaluation-recording http://www.medlib-ed.org/products/1535/8-tips-for-using-metrics-in-research-evaluation-recording http://www.medlib-ed.org/products/1535/8-tips-for-using-metrics-in-research-evaluation-recording awards hypothesis vol. 31 no.1 fall/winter 2019 2nd place – community health assessment tools • dana abbey associate professor, national network of libraries of medicine, midcontinental region, strauss health sciences library, anschutz medical campus, aurora co there are numerous factors that have the potential to influence the health of your community members including quality of life, health behaviors, utilization of and access to health care, social and economic factors, and the physical environment. in this webinar participants will: 1) learn how to utilize tools for researching these factors at the local level; 2) identify authoritative health information resources for program planning; and 3) identify potential community partners. available at: https://nnlm.gov/class/community-health-assessment-tools/6759 3rd place – creating a campus-wide research data services committee (parts 1 & 2) • amy koshoffer science informationist, uc libraries, university of cincinnati, cincinnati oh • renaine julian associate director for stem libraries, florida state university libraries, florida state university, tallahassee fl • cinthya ippoliti university librarian and director, auraria library, university of colorado denver, denver co • betty rozum research data librarian, merrill-cazier library, utah state university, logan, ut • christine kollen data curation librarian, office of digital innovation & stewardship, main library, the university of arizona, tucson az • david minor program director for data curation, uc san diego library, university of california at san diego, la jolla ca this two-part webinar covers how academic libraries are taking the lead in developing cross-campus collaborations in establishing research data committees to spearhead institutional efforts related to data stewardship and digital projects. this webinar session leads participants through the various steps needed in order to initiate a similar effort within their institutional context. available at: https://nnlm.gov/class/creating-campus-wide-research-data-servicescommittee-part-1-building-bridges-and-planting we are excited to present these as excellent continuing education opportunities for health sciences librarians! https://nnlm.gov/class/community-health-assessment-tools/6759 https://nnlm.gov/class/creating-campus-wide-research-data-services-committee-part-1-building-bridges-and-planting https://nnlm.gov/class/creating-campus-wide-research-data-services-committee-part-1-building-bridges-and-planting reflection hypothesis vol. 31 no. 1 fall/winter 2019 a year in review: mla-rti fellow reflection margaret hoogland, mls, ahip assistant professor, clinical medical librarian mulford health sciences library university of toledo toledo, oh the medical library association-research training institute (mla-rti) is an opportunity for librarians to get hands-on experience and mentoring from faculty experts on how to design, execute, present, and publish the results of a one-year research project. the mla-rti program director, susan lessick, worked with five faculty members (sally gore, lori kloda, mark maceachern, jodi phelps, and emily vardell) to design pre-institute online training modules, a five day in-person training, and a mentoring structure for the projects that fellows design during the fellowship year. in early december, mla-rti applicants submitted a brief biography describing their research experience, a study proposal, proposed research project timeline, letter of intent, and a letter of support from the fellows’ library. participants in the mla-rti can request financial assistance and most of the 2018 mla-rti fellows received financial assistance. in march 2018, i received an email inviting me to be a member of the inaugural cohort of the medical library association-research training institute (mlarti). after completing the pre-institute training, i arrived for the in-person training, eager to learn and ready to adjust my proposed mla-rti project. at mla-rti in-person training in general, i found accepting and providing feedback during the in-person training to be challenging. it meant keeping my mind open to comments, feedback, and criticism. i also found, when providing feedback or criticism to others, it was best to be constructive (e.g., “i find this confusing”, “have you considered?”) and candid (i.e., “i really like this”). if authors located and stated a quote that i found interesting and pertinent, i might respond, “great quote!” i arrived expecting to be challenged and to get guidance on how to improve my project; however, it quickly became clear that i was a fellow and this was my project. i had no list of instructions or bullet points on how to proceed and, if i had questions or concerns, it was my job to approach other fellows and faculty members to get answers. upon reaching this conclusion, i used my 30-minute mentor session and discussions with fellows and experienced faculty members, to expand my knowledge and improve my project. following this in-person training, the responsibility of solving issues and completing the project in one year rested completely on me. reflection hypothesis vol. 31 no. 1 fall/winter 2019 during mla-rti fellowship year each mla-rti faculty member had mentor groups and four mentees. mentor groups and mentees were pre-assigned based on project if possible. for example, if a fellow was doing a qualitative study, they might be paired with emily vardell. if working on a survey project, they might be paired with jodi philbrick. my mla-rti mentor, mark maceachern, established a monthly check-in opportunity that i found to be essential. we primarily communicated via email with me describing, in 3-8 sentences, what i had done and what i hoped to do in the upcoming month. quarterly reports were also used to keep a fellow’s project moving forward and to keep the mla-rti program director, susan lessick, and the assigned mentor apprised of how the project was progressing. in each report, fellows describe issues with the project, collaborations, and outcomes from participating in the mla-rti. one problematic issue i faced was including interviews in my mla-rti project. i had no prior experience with conducting interviews for a study and when it came to analyzing the transcripts, i spent about three months deciding how to organize the information from the interview and trying to determine what form of analysis to use. i contacted my assigned mentor for ideas, but i found a conversation with the educational, research, and measurement program director at the university of toledo to be most helpful. after consulting a handbook on qualitative research, i had a plan and could move forward with analyzing my data. figuring out how to analyze the data, and recognizing that i should have structured interview questions differently, proved to be the most challenging aspect of my mla-rti project. as part of the mla-rti, all fellows participate in the mla-rti session at the mla annual business meeting and the conference poster presentation. i confirmed my mla annual business meeting and conference attendance early in january and received additional instructions in subsequent months on how to proceed. i also gave a lightning round presentation on my mla-rti project. several of the mla-rti fellows also presented posters. quite a few fellows could not attend due to conflicting meetings and delayed providing a final report until the mla-rti session at mla in 2020. unanticipated (but positive!) developments due to mla-rti participation in january 2017, i designed a session with first year medical students that incorporated use of medical information tools, such as dynamed, uptodate, epocrates, and visualdx. i was able to teach the session in both fall 2017 and fall 2018 and my project investigated how the students discover and use these medical information tools. my study highlighted positive changes resulting from instituting the new curriculum in fall 2017. as a result, students are now approaching me with ideas for how the library can improve the student experience. additionally, after showing my mla-rti project survey results to my faculty and administration, they urged me to write-up and publish the results of my mla-rti project. reflection hypothesis vol. 31 no. 1 fall/winter 2019 as a result of my mla-rti project, i now have a rapport with key members of the preclinical curriculum. in fall 2019, these faculty members agreed to let me meet with smaller groups of students for 15-and-30-minute sessions, which they can schedule in the fall. the timing of these sessions is more applicable to when students could use the knowledge of how to search the literature and evaluate the study design, results, and publication types of articles. i am becoming the go-to librarian for students, faculty, and staff in the college of medicine and life sciences and i am less hesitant to discuss potential grant and research ideas with faculty members. i would not be comfortable doing either of these, if i had not participated in the medical library associationresearch training institute experience. i strongly encourage everyone to give research a try and apply for a future mla-rti! acknowledgements i thank the mla research training institute for its training, support, and encouragement to carry out this research. this project was made possible in part by the institute of museum and library services (re-95-17-0025-17). research section spotlight hypothesis, vol. 30, no. 1, fall/winter 2018 54 diversity and inclusion: potential trends for data in research charlene finley, mls phd student school of information and library science university of north carolina at chapel hill finleyc@email.unc.edu introduction from the pre-meeting emails by mla leadership soliciting feedback on the proposed community models to the closing plenary speakers, the theme of the annual meeting, “adapting, transforming, and leading,” was thoroughly integrated with all aspects of the 118th medical library association annual meeting in atlanta, ga. amid the standard conference program, and the historical naming of beverly murphy as the first africanamerican mla president, the meeting fostered an environment of engagement and inclusiveness. the mla’s diversity and inclusion task force and the communities strategic goal task force promoted safe spaces for all mla attendees to ask questions and voice their opinions on issues relevant to governance of mla and their perceptions of diversity within the health and medical information profession. for example, attendees were afforded numerous venues and opportunities to collectively or individually speak with representatives from mla’s communities strategic goal task force with questions, comments, or concerns to the proposed restructuring of the mla section and special interest groups (sigs). the dialogues extended beyond formal spaces to informal spaces providing mla attendees the opportunity for meaningful exchanges with health and medical information professionals from diverse perspectives and experiences. however, my main takeaway from the annual meeting was the overview by dr. dara richardson-heron of the all of us research program (aou). the presentation directly aligned with the annual meeting’s theme of “adapting, transforming, and leading,” and connected to my future research interest. transformative research in practice dr. richardson-heron is the chief engagement officer of the national institute of health all of us research program (aou) and a health advocate. the main objective of aou is to advance the precision medicine initiative through the collection of data from more than a million participants over the span of a decade. the precision medicine initiative is a transformative approach for treatment and disease prevention that focuses on the uniqueness of individuals and accounts for distinct factors, such as lifestyle, genetics, and habits to deliver “the right treatment for the right person at the right time” [1]. http://www.mlanet.org/blog/dara-richardson-heron http://www.mlanet.org/blog/dara-richardson-heron http://www.mlanet.org/blog/dara-richardson-heron https://www.joinallofus.org/en https://www.joinallofus.org/en https://www.joinallofus.org/en https://ghr.nlm.nih.gov/primer/precisionmedicine/initiative https://ghr.nlm.nih.gov/primer/precisionmedicine/initiative research section spotlight hypothesis, vol. 30, no. 1, fall/winter 2018 55 committed to implementing innovative policies and practices to increase the success of the program, aou has established a set of core values as a framework to achieve widespread participation from diverse people, geography, health status, and data types. dr. richardsonheron states these core values are very important to her. in response to unethical historic research practices (for example, henrietta lacks and the tuskegee syphilis study) that caused harm or excluded vulnerable populations, aou has initiated policies and practices to re-establish participant trust and increase participation with, and transparency of, the research process and data collection through education, awareness, and immediate notifications of data breaches. aou’s implementation of a transparent approach to research has the potential to benefit participants in meaningful ways. some of the key benefits for participants include knowledge of health conditions and personal risk factors that may affect long-range health. in addition, participants will have access to their data to share with health care teams. similarly, researchers will also benefit from the data collected in the ten-year longitudinal study at no cost in what dr. richardson-heron describes as a democratized research model. the democratized model will afford researchers (1) a rich resource of data, (2) a longitudinal dataset, (3) a diverse cohort of research participants, (4) de-identified and anonymized raw data and data, (5) an existing secure data infrastructure, and (6) a collaborative working environment. finally, expressing her appreciation for libraries and librarians, dr. richardson-heron shared the plans of a three-year partnership with the national network of libraries of medicine to design, develop, and implement activities for public libraries to make available to their local communities. the libraries role would be to educate and increase awareness of the all of us research initiative, in addition to providing technology resources for individuals without access to the internet. research and advocacy the all of us research program provides a framework for research considerations for all disciplines. in my future research, i will focus on the health information and decisionmaking of veteran women. as a veteran woman and researcher, it is important to me to advocate in the interest of veteran women. my research objective is to examine the history of women veterans in the united states and propose a possible conceptual framework for describing and understanding how the information needs, access, and use of health information reflects and contributes to the marginalization of women in the veterans health administration (vha). despite the historically strong opposition of women in the military and in combat roles, women have demonstrated valor in combat since the revolutionary war [2], preceding policy and withstanding resistant societal and cultural norms with rigid standards and perceptions of women. today, there are approximately 1.84 million living female veterans research section spotlight hypothesis, vol. 30, no. 1, fall/winter 2018 56 of more than 23 million surviving veterans in the united states [3], and by the year 2020 the number of woman veterans is projected to reach 2 million [4]. the priorities for women at the va have ranked relatively low level in the past due to their small numbers, and military law and policy that limited their military involvement [5]. thus, the medical needs and long-range effects of combat on the health of women have historically been poorly understood in medicine and research [5][6]. women are a numerically small population within a largely male-oriented vha healthcare system [6]. historically, due to insufficient data collection on female veterans, there remains a gap in understanding how veteran women utilize va healthcare services and benefits [7]. another challenge for researchers has involved the representation of health issues related to vulnerable and marginalized groups, such as african-american and other minorities, the homeless, rural communities, and lesbian, gay, bisexual, transgender, and queer (lgbtq) veterans who have not been traditionally represented in research. addressing intersectionality in research can potentially improve health outcomes and inform stakeholders in the development of health information resources and technology to address the specific needs of veteran women. whatley and worcester propose that when women have access to health information, not only do they have a better understanding of their body; they also are more likely to make better health-related decisions [8]. a copious amount of research focused on various aspects of women in general, and health information behavior, has provided a potential foundation for comparing and identifying key behaviors that may be salient to understanding the health information behavior and decision-making of veteran women. as the vha implements quality initiative to improve care for female veterans, understanding their health information needs will help to tailor health care delivery to veteran women and address health disparities. conclusion the success of the all of us research program will have a widespread impact on health, research, and the value of libraries. the availability of a rich source of data will allow comparison between veterans and non-veterans, in particular female veterans, to identify gaps in research to improve the immediate and long-range health of the veteran population. gaps in research will be more identifiable, eliminating duplicate research efforts. furthermore, partnerships with initiatives like all of us will continue to enhance the value of librarians and health information professionals, and help connect people from diverse backgrounds to health-related resources and information. research section spotlight hypothesis, vol. 30, no. 1, fall/winter 2018 57 works cited 1. richardson-heron d. mla 2018 plenary session 4. cited in allen m, mla '18 plenary 4 recap: dr. dara richardson-heron. retrieved 11/9/2018 from https://www.mlanet.org/p/cm/ld/fid=1122&&blogaid=2073. 2. mcgraw k, koehlmoos tp, ritchie ec. women in combat: framing the issues of health and health research for america’s servicewomen. mil med, 2016;181, 7–11. doi: https://doi.org/10.7205/milmed-d-15-00223. 3. resnick, e.m., mallampalli, m., carter, c.l. (2012). current challenges in female veteran’s health. j women’s health, 21(9) 895-900. doi: https://doi.org/ 10.1089/jwh.2012.364 4. american college of obstetricians and gynecologists. committee opinion no. 547: health care for women in the military and women veterans. obstet gynecol. 120(6), 15381542. 5. tepe v, yarnell a, nindl bc, van arsdale s, deuster pa. women in combat: summary of findings and a way ahead. mil med, 2016;181, 109–118. doi: https://doi.org/10.7205/milmed-d-15-00409. 6. yano em, hayes p, wright s, schnurr pp, lipson l, bean-mayberry b, washington dl. (2010). integration of women veterans into va quality improvement research efforts: what researchers need to know. j gen int med, 2010; 25(1):56–61. doi: https://doi.org/10.1007/s11606-009-1116-4. 7. mcclerking ca, wood f. health policy initiatives for african american women veterans. policy politics nurs practice, 2016;17(3):118-124. 8. whatley mh, worcester n. women’s health: readings on social, economic, and political issues. dubuque, iowa : kendall/hunt pub. co., 1988. -- https://www.mlanet.org/p/cm/ld/fid=1122&&blogaid=2073 https://www.mlanet.org/p/cm/ld/fid=1122&&blogaid=2073 https://doi.org/10.7205/milmed-d-15-00223 https://doi.org/10.7205/milmed-d-15-00223 https://doi.org/10.7205/milmed-d-15-00223 https://doi.org/10.7205/milmed-d-15-00409 https://doi.org/10.7205/milmed-d-15-00409 https://doi.org/10.7205/milmed-d-15-00409 https://doi.org/10.1007/s11606-009-1116-4 https://doi.org/10.1007/s11606-009-1116-4 https://doi.org/10.1007/s11606-009-1116-4 research section spotlight hypothesis, vol. 29, no. 2, fall/winter 2017 39 mla 2017 research section research awards congratulations to the winning research papers and posters from mla ’17! the mla research section is pleased to announce the winners for best research papers and posters presented at the mla 2017 annual meeting in seattle, wa. thank you to the 57 judges who volunteered their expertise to help select these deserving awardees. to learn more about the awards and selection process, visit the research section website at http://www.mlanet.org/p/cm/ld/fid=938. contributed papers 1st place authors: christy jarvis, ahip, head of information resources and digital initiatives, university of utah, salt lake city, ut; melissa rethlefsen, ahip, deputy director / associate librarian, spencer s. eccles health sciences library, salt lake city, ut title: daring to dive deep into the citation data: going head to head with scihub abstract: objectives: to analyze the extent to which the library is fulfilling the information needs of its patrons in light of recent data showing heavy usage of scihub in [our metro city] and specifically to gain insight into the percentage of referenced resources and content areas that were not made available to researchers through library funded subscriptions. methods: using scopus, we identified papers published by the institution’s health sciences faculty in each of the previous 5 years. papers co-authored by researchers at other institutions were excluded in order to focus the analysis solely on information resources available to scholars at the target institution. the citations from the resulting set of publications were extracted to spreadsheets, where they were subjected to a data normalization process. incomplete or obviously erroneous references were removed. the remaining set of citations was sorted alphabetically by journal title, then secondarily by citation year. using a combination of data sources, including publisher entitlement reports, catalog records, and previously downloaded holdings reports, we compared each citation to the library’s collection at year of http://www.mlanet.org/p/cm/ld/fid=938 research section spotlight hypothesis, vol. 29, no. 2, fall/winter 2017 40 citation to determine if access to the cited resource was provided and paid for by the library. results: published literature authored by university of utah health sciences faculty between 2012 and 2016 yielded 119,794 citations for analysis. the libraries had print or electronic access to 99,298 (82.89%) of these cited resources. another 9,220 (7.7%) were accessible from open access platforms, leaving only 11,084 (9.25%) citations that needed to be obtained from other sources, such as interlibrary loan, pay-per-view, or illegitimate sites such as sci-hub. of the cited, but not provided, literature, 26% comes from backfile content whereas 74% comes from more recently published work. identified collection gaps include the disciplines of neurology, cardiology, and oncology. conclusion: a comprehensive evaluation of health sciences author citations over a 5-year period demonstrated that the vast majority were available from library-funded resources, thereby suggesting that the library’s collection has been adequately meeting the needs of researchers and scholars. this study provides valuable insight into faculty information-seeking behaviors and has implications for future collection development, service offerings, and funding priorities within the library. further study is needed to explore correlations between library-funded access to cited resources and other university metrics such as faulty recruitment and grant funding, as well as to investigate purported sci-hub activity in light of our findings. 2nd place authors: joanne marshall, ahip, fmla, research professor, school of information and library science chapel hill, nc; amber wells, doctoral graduate, dept of sociology, chapel hill, nc; kathel dunn, associate fellowship program director, national library of medicine, bethesda, md; joyce backus, associate director for library operations, national library of medicine, bethesda, md title: the role of medline in patient care: results of a secondary analysis of the value of libraries study abstract: objectives: what role does the medline database play in relation to other information resources that are available to health care providers? what is the role of medline in positively impacting patient care? since health care providers use multiple information resources in research section spotlight hypothesis, vol. 29, no. 2, fall/winter 2017 41 providing patient care, what is the specific role of medline? methods: a previous survey on the use of health information resources for patient care obtained 16,122 responses from 56 hospitals in the u.s. and canada. the study asked respondents to indicate resources used in answering specific clinical questions. on average, respondents reported using 3.5 resources to answer their question. this analysis used advanced descriptive statistics and regression analysis to examine the specific information resources used and how they were used in combination with one another. the use of more resources was associated with more changes made to patient care and increased avoidance of adverse events. medline was more likely to be among the resources consulted than any other information resource except journals. the analysis reported in this paper provides new insights into how medline is used in conjunction with other resources to answer clinical questions. results: our additional analysis of the value study data found that medline and online journals were the two most frequently used information resources. respondents reported using an average of 3.5 resources when seeking additional information related to a specific patient care decision making situation. using more information resources was associated with improved clinical decision making and a higher probability of making changes to patient care and avoiding adverse events. medline was most likely to be among the combinations of information resources used by physicians, residents and nurses. conclusion: medline continues to be a key information resource for health care providers as they seek answers to patient care questions. the medline database is also used in the preparation of many other specialized health information resources and point of care information tools. since health professionals use multiple information resources, libraries and librarians continue to have an important role in providing access to and supporting the use of a wide range of information tools. honorable mention authors: tanja bekhuis, ahip, principal scientist, tcb research & indexing llc, pittsburgh, pa title: training in support of data-driven research: a qualitative study of library workshops in top national institutes of health-funded and national science foundation-funded universities research section spotlight hypothesis, vol. 29, no. 2, fall/winter 2017 42 abstract: objectives: in an age of data-driven research and competitive funding, libraries help their patrons acquire new skills. to do so, some offer workshops about knowledge discovery, analysis, and management of various kinds of data. the primary objective of this qualitative study was to explore the nature of workshops offered by libraries to support data-driven research in top nihand nsf-funded universities. additionally, we developed a catalog of workshops, and indexed the resources and thematic content. methods: to identify top-funded universities, we used nih research portfolio online reporting tools and the nsf budget internet information system. from corresponding websites, we extracted information on 99 workshops offered by health sciences libraries (n=5) and main libraries (n=5) in schools funded by nih and nsf, respectively. workshop title, duration, and description were catalogued by library and source of federal funding for the university. we used nvivo 11 pro (qsr international) for qualitative data analysis and textract® (texyz) to semi-automate indexing the content of the catalog. themes were first identified in textual patterns and then were refined by an analyst. thematic overlap was described across funding source. additionally, we identified themes unique to each subset. results: main libraries in nsf-funded schools offered 36% more workshops than health sciences libraries in nih-funded schools (57 vs 42). overall workshop duration ranged from 1 to 16 hours in a bimodal distribution (1st mode = 1 hour; 2nd mode = 3 hours). the distribution of duration for nsf schools differed from the nih distribution. we identified 15 main themes overall: statistical programming and data visualization occurred most often, and finding funds for research and open science least often. thematic distributions varied with funding source. for example, bioinformatics occurred most often in the nih-funded subset and statistical programming in the nsf subset. for each subset, 20 most informative indexing terms were identified after sorting and discretizing into 7 quantiles. top indexing terms included: data visualization, pathway analysis, and data management (nih schools); data analysis, data management plan (dmp), and python (nsf schools). conclusion: a catalog of workshops organized by university funding source and library, along with 2 indexes (resource and subject), will be publicly available. the analytical results, as well as index content, yield insights regarding workshop coverage. implications for strategic planning and development of library workshops in support of data-driven research will be discussed. research section spotlight hypothesis, vol. 29, no. 2, fall/winter 2017 43 contributed posters 1st place authors: angela spencer, manager, c. alan mcafee md medical library, chesterfield, mo; elizabeth laera, medical librarian, brookwood baptist health, birmingham, al; halyna liszczynskyj, director, library services, st. elizabeth medical center, utica, ny; louise mclaughlin, information specialist, woman's health sciences library, baton rouge, la; kathy zeblisky, medical library manager, phoenix children's hospital, phoenix, az title: solo librarians: demographics, duties, needs, and challenges abstract: objective: to obtain data on how many librarians classify themselves as solo librarians within a medical/hospital setting. solo librarians constantly face challenges to maintain and expand services vital to their users. by quantifying their number and needs, a stronger voice can be developed. methods: a ten question survey using surveymonkey was sent to various medical library related listservs of interest to solo librarians. results: 383 surveys were returned, the majority from hospital and academic librarians. other settings include clinics, organizations, research institutions and veteran’s institutions. duties showed the variety of hats a solo can wear. duties included: reference, interlibrary loan, teaching, committee work, website development, marketing, creating policies/procedures, writing grants, archives, informatics and other work. the “best challenges” question was the most insightful into what the needs are for solos. major challenges included: funding/budget, awareness/visibility, time management, value/roi/proving your worth, staffing, space, promotion/marking/outreach, professional development, technology and organizational mergers. discussion: the full survey results quantify the size of the solo librarian population, and the contributions and challenges they face working in solo settings. this data can contribute useful information to discussions on best ways to support, educate, inform and advocate for this research section spotlight hypothesis, vol. 29, no. 2, fall/winter 2017 44 population. conclusion/next steps: solo librarians are faced with similar financial, marketing and operational challenges regardless of setting. we hope to encourage peers to share their challenges and concerns and work with nn/lm and mla to educate them about solo librarians’ needs and concerns so that we can sustain our future. 2nd place authors: nicole theis-mahon, liaison to the school of dentisty & hsl collections coordinator, university of minnesota, minneapolis, mn; shanda hunt, public health library liaison & data curation specialist, health sciences libraries, minneapolis, mn title: my doctor said what!? identifying and assessing online health information resources abstract: objectives: health information consumers look to the internet to find answers to questions about their health or that of a loved one. we conducted a study to identify where individuals find online health information, how they use it, and what they think is missing. results from this study are being used to make recommendations of how to improve services to this population. methods: the university of minnesota health sciences libraries conducted a cross-sectional study of adults in august 2016. the survey instrument was adapted from the ehealth literacy scale (eheals) and the patient activation measure (pam-13), administered electronically on tablets at the minnesota state fair, and took approximately six minutes to complete. convenience sampling yielded a total of 281 participants. analysis of descriptive statistics and statistics to explore relationships between variables were conducted using r, and a qualitative analysis of one survey item was conducted using nvivo. results/conclusion: preliminary results show that a majority of participants use a search engine, such as google, webmd, or the mayo clinic website, to locate online health information. while most respondents were confident in their ability to evaluate the health resources they find online, only half identified indicators of quality health information. this result was confounded by the high number of participants who were health providers. participants identified personalization of and interactivity with health websites as highly desirable. research section spotlight hypothesis, vol. 29, no. 2, fall/winter 2017 45 honorable mention authors: hannah norton, interim fackler director, associate university librarian, university of florida, gainesville, fl; mary edwards, reference & liaison librarian, health science center library, gainesville, fl; ariel pomputius, health sciences liaison librarian, health science center library, gainesville, fl; michele tennant , interim fackler director, health science center libraries, gainesville, fl title: tracking tech trends: studying patron technology use through annual surveying abstract: objectives: at an academic health sciences library serving students, faculty, and staff across a wide variety of disciplines, studying library patrons’ technology use, particularly in areas of mobile technology, provides necessary information on intersection points for library services. administering a similar survey annually for five years generates a holistic view of patrons’ technology needs and preferences over time. methods: beginning in 2012, the university of florida (uf) health science center library (hscl) began administering a 16-question survey designed by the university of southern california norris medical library to address technology use of health professional students and faculty and their interest in related library services. for three years we participated in a multiinstitution implementation of this survey; when the collaboration ended, we continued to administer the survey at uf. while some questions have been modified over time for clarity or changes in available technology, many are consistent across the five years of survey implementation, allowing analysis of trends over time in use of specific technologies and service needs at our institution. results: smartphone ownership among survey respondents is nearly universal (ranging from 87.6% to 95.7% over the past 5 years), and a majority of respondents also own a tablet (from 51.1% to 70.2%). while respondents were likely to check library hours, use medical apps, and use library electronic resources from their smartphone or tablet, they reported being unlikely to friend or follow the library on facebook or twitter or send a call number from the catalog. one simple change implemented in response to survey results was to add the library’s hours to the “quick links” portion of the library’s website; while the hours are featured on other parts of the site, the quick links are the most prominent portion of the site’s mobile version. likewise, when survey data indicated that respondents were highly interested in research section spotlight hypothesis, vol. 29, no. 2, fall/winter 2017 46 training on mobile device apps, the hscl developed a stand-alone workshop entitled “mobile resources for health.” trends that have not yet been explored further include respondents’ preference for print books for both academic (53.1% to 57.3%) and leisure (53.2% to 55.2%) reading, as compared to ebooks. conclusions: annual review of survey results has led to incremental changes in services offered. reviewing the aggregate data allows for more strategic consideration of future directions, with implications towards marketing the library’s resources, training development, and service development. -- awards hypothesis vol. 31 no.1 fall/winter 2019 congratulations to the winning research papers and posters from the medical library association (mla) 2019 annual meeting! the mla research caucus is pleased to announce the winners for best research papers and posters presented at the mla 2019 annual meeting in chicago, il. thank you to the 25 judges who volunteered their expertise to help select these deserving awardees. to learn more about the awards and selection process, visit the research caucus website at http://www.mlanet.org/p/cm/ld/fid=938. contributed papers 1st place investigating emerging roles for medical librarians at college and university libraries • alexander carrol, lead librarian for research engagement, nc state university, raleigh, north carolina • jason reed, health sciences information specialist, purdue university, west lafayette, in objectives: this study seeks to identify emerging roles for health sciences librarians by examining position descriptions at u.s. college and university libraries. we hypothesize that because of increasing interdisciplinary research within colleges and universities, there may be emerging roles for medical librarians at academic institutions without dedicated academic health sciences libraries as signified by association of academic health sciences libraries (aahsl) membership. methods: to gather these position descriptions, we are using two methods. we are collecting position advertisements posted to several job boards and health sciences librarian mailing lists between september 1 2018 and march 1 2019. positions that include at least one of the following terms in the position title, qualifications, or responsibilities are flagged for review: medic*, health, life, nursing, veterinary, bio*, pharma*. we are also advertising this study on several health sciences librarian mailing lists, soliciting individuals who meet these criteria to deposit their position descriptions into a repository via a qualtrics survey. we are using grounded theory (an inductive, qualitative research methodology) to identify themes within the collected position descriptions that meet the inclusion criteria. results: we found a total of 104 job postings that met our inclusion criteria. of those 104, 60 were advertised at aahsl member institutions, and 44 from non-aahsl institutions. 81 of these advertisements were for front-line librarian positions, with the remaining 23 for management level positions. job postings were listed in 32 distinct states, with at least one posting in all 8 national network of libraries of medicine (nnlm) regions. we found the highest number of postings in region 2 (southeastern / atlantic region), and the fewest in region 6 (pacific northwest http://www.mlanet.org/p/cm/ld/fid=938 awards hypothesis vol. 31 no.1 fall/winter 2019 region). qualitative and additional quantitative analyses of the content of the collected position descriptions are ongoing at this time. conclusions: our preliminary quantitative data suggest the existence of a relatively high number of health sciences librarian job postings at institutions outside of aahsl. librarians entering into these advertised positions, specifically into front-line positions, will likely seek out professional development opportunities that will align with the mla competencies for lifelong learning and professional success. this suggests an opportunity for mla to expand its membership by designing continuing education opportunities for health sciences librarians employed at academic institutions without dedicated academic health sciences libraries. 2nd place burnout among health information professionals: elevating the issue to inspire change • angela spencer, medical library manager, st. luke’s hospital, chesterfield, missouri • elizabeth laera, medical librarian, brookwood baptist health, birmingham, alabama objectives: to obtain data on how many health information professionals meet the copenhagen burnout inventory criteria and how many feel they are burned out or close to burning out despite survey results. this study also seeks to determine which category of health information professionals are facing the highest risk of burnout and what is being done to combat or prevent burnout. methods: potential study participants were contacted via various medical and special library listservs. participation was limited to health science information professionals, including paraprofessionals. using surveymonkey, study participants first completed a modified version of the copenhagen burnout inventory (cbi) and then were asked about their general experiences with burnout, what, if anything, they are doing to relieve stress or prevent burnout, and demographic questions to quantify their results with others in similar settings, age ranges, and experience levels. results: 497 responses were recorded. of these, 41 respondents did not complete enough of the survey to be counted, bringing the total number of usable surveys to 456. 179 participants reported an overall level of burnout. using average cbi scores, we determined that 117 of those reporting burnout had scores suggested at least a moderate level of burnout. 41.88% of these scored at the significant level of burnout or higher. 5 participants or 1.14% reported not feeling burned out scored at the moderate or significant level in the cbi. of those scoring at least moderate in the cbi (n=143), 52.45% were from an academic environment and 62.22% had 15 or less years of experience. the participants with the highest cbi averages (n=26) had very little in common among their reasons for burnout or demographics. awards hypothesis vol. 31 no.1 fall/winter 2019 conclusions: are health information professionals burned out? research shows that burnout has been a theme in librarianship literature since the early 1980s. public, academic, school, and special library journals have published articles on recognizing signs of burnout, preventing burnout in the workplace, and ways to combat burnout in librarians and information professionals. however, there is limited research to determine the number of librarians and information professionals experiencing burnout. this study helps to determine the level of burnout health information professionals are experiencing, and in the future, may help librarians and other information professionals find ways to prevent burnout. honorable mention elevating nursing librarianship: assessing trends in distance librarian services for advanced practice nursing programs • gregg stevens, health sciences librarian, stony brook university, stony brook, new york • elizabeth hinton, instruction and research librarian, university of mississippi medical center, jackson, mississippi • roy brown, research and education librarian, virginia commonwealth university, north chesterfield, va objectives: with the increasing popularity of distance education among universities and busy students, many advanced practice nursing (apn) programs have shifted to become either online or hybrid programs. to meet the research and instruction needs of these students, some nursing librarians are using technology for virtual research and instruction. methods: this study was designed to assess the extent to which nursing librarians in north america are providing virtual research and instruction services for apn students. an irb-approved survey with ten objective questions was developed to determine how librarians are providing services for apn students at their universities. it was announced in october 2017 through several health sciences librarian listservs. the survey ran for four weeks. data were analyzed using qualtrics and excel. results: eighty complete responses were received. the majority of respondents (66%) indicated that their universities' apn programs were conducted in a hybrid format and the same percentage also indicated that they generally provide library instruction in person. most librarians indicated that they have provided research assistance through some virtual method (phone or email) and some have also used online chat (11%) and video chat (9%). a strong majority of librarians (95%) indicated that they feel comfortable using technology to provide research assistance and instruction. conclusions: many opportunities exist for nursing librarians to use technology to provide virtual research assistance and library instruction. greater promotion of awards hypothesis vol. 31 no.1 fall/winter 2019 these alternate methods can supplement traditional in-person services, providing greater flexibility for the busy schedules of graduate nursing students. some outreach may be necessary to highlight the advantages of virtual services. contributed posters 1st place knowing and doing: quantitative and qualitative exploration of wellness behaviors among health sciences librarians • susan keller, librarian, children's national medical center, college park, maryland • layla heimlich, medical librarian, medstar washington hospital center, bethesda, maryland • fred king, medical librarian, medstar washington hospital center, silver spring, maryland • jory barone, medical librarian, medstar washington hospital center, washington, district of columbia objectives: the connection between abundant and easy access to information among health science librarians and the practice of wellness behaviors is unknown. this project looks at the behaviors of both health science and non-health science librarians in order to answer the question: "is having access to high-quality health information associated with a high level of wellness behaviors among health science librarians?" methods: after consultation with a biostatistician and approval by the institutional review board, we designed a survey consisting of seven demographic items: two items on workplace support for wellness behavior (e.g., vaccines, wellness programs), three items to assess access to health information and awareness of healthy behavior recommendations, seven specific healthy behavior practice items, and three open ended questions. we contacted 88 library organizations and 49 agreed to help us distribute our survey. these organizations represented public, school, academic, and health science libraries. we also posted the survey on the medlib-l (medical library association listserv), special library association listserv, as well as seven sla and mla special interest groups. we opened the survey on march 7, 2018 and closed it on april 25, 2018. survey analysis tools included redcap, sas 9.4, and dedoose. results: 1913 librarians responded to the survey 663 health science librarians (hsl) and 1250 non-health science librarians (non-hsl). we found no significant demographic differences between the two groups. hsl are more likely than nonhsl to eat 5 or more fruits/vegetables per day (p value 0.031). hsl were more likely than non-hsl to exercise moderately (p value <0.001). hsl were less likely than non-hsl to engage in prayer or other spiritual practices (p value 0.009). using awards hypothesis vol. 31 no.1 fall/winter 2019 dedoose, we identified themes concerning the barriers, facilitators, and connections between the profession of librarianship and wellness behaviors. conclusions: overall, health science librarians have more access to quality health information and have healthier eating and exercise behaviors, but spend less time in prayer and other spiritual activity than non-health science librarians. we expect to gain more insight into the barriers and facilitators, as well as the connections between the profession of librarianship and the practice of wellness behaviors, after analyzing the qualitative results from the survey. 2nd place the impact of institutional repositories: a systematic review • michelle demetres, scholarly communications librarian, weill cornell medical college, new york, new york • diana delgado, associate director, information, education and clinical services, weill cornell medicine, new york, new york • drew wright, research librarian, weill cornell medical library, sunnyside, new york objectives: institutional repositories are an ideal outlet to present and publicize an academic institution's output. however, there are many challenges associated with its startup and upkeep. the objective of this study is to concretely define the various impacts that an institutional repository can provide for an academic institution, thusly justifying its implementation and/or maintenance. methods: a comprehensive literature search was performed in march 2018 in the following databases from inception: ovid medline, ovid embase, the cochrane library (wiley), eric (proquest), web of science (core collection), scopus (elsevier), and library, information science & technology abstracts (ebsco). a total of 6,331 citations were screened against predefined inclusion/exclusion criteria. results: the 13 included studies were divided into three areas of impact: citation count; exposure/presence; and administrative. those focusing on citation count (5) and exposure/presence (7) all saw a positive impact for the institution and/or researchers. the one study focusing on administrative benefit saw a unique tie-in with automated orcid profile population. conclusions: based on the available literature, institutional repositories do appear to have a positive impact in the following areas: citation count, exposure/presence, and administrative burden. in order to make more concrete conclusions, additional, higher quality studies are needed. honorable mention e-books and print books: what do medical students and residents prefer and why • erin watson, health sciences librarian, university of saskatchewan, saskatoon, saskatchewan, canada awards hypothesis vol. 31 no.1 fall/winter 2019 objectives: the objective of this research is to determine in which format medical students and residents prefer to use monographs to support their learning and clinical work. informal feedback from library users at the researcher's institution, and results reported in the literature, have both varied, making acquisition decisions difficult. methods: the researcher surveyed medical students and residents at her institution to determine their preferred reading format, and the reasons and circumstances surrounding it. the survey also asked what respondents do if their preferred format is not available. medical education is delivered through a distributed program at the researcher's institution, therefore, respondents were asked to indicate where they are located; this will help determine if students studying through the distributed program prefer to use electronic materials, or whether they borrow/acquire print materials. results: of 844 possible respondents, 232 answered the survey, a response rate of 27.5%. not all respondents answered all questions. when reading only a few pages, the vast majority (72%, n=166) preferred to use e-books. when reading a chapter of 10 pages, the largest number (48%, n=105) preferred e-books. when reading an entire book, the majority (72%, n=166) preferred print books. distributed students and residents were not statistically more likely to prefer e-books to print books. the top reasons for choosing to use e-books were immediate access to content (n=205; 89%), searchability (n=175; 76%), and accessibility on the go (n=166; 70%). top reasons for choosing print books were reduced strain on the eyes (n=177; 77%), easier absorption of information (n=128; 56%), and being able to hold the book in one's hands (n=128; 56%). when respondents preferred to use e-books, but only the print format was available from the library, a majority (n=140; 61%) would use the print book, though a sizeable number (n=109; 47%) felt they would get the ebook from elsewhere. when print was preferred but only e-books were available from the library, the majority (n=173, 76%) would use the e-book, though 34% (n=78) might buy their own copy of the print. conclusions: while medical students and residents enjoy using e-books for their convenience and accessibility, they prefer print books for sustained reading. libraries should continue to make print books available to their patrons, even to those in distributed locations. research section spotlight hypothesis, vol. 29, no. 2, fall/winter 2017 32 “librarian’s role in reproducibility of research” 2017 symposium: research section stipend winner reflection kimberly r. powell, mlis life sciences informationist woodruff health sciences center library emory university atanta, ga krpowel@emory.edu introduction at the 117th annual meeting and exhibition of the medical library association (mla) in 2017, eight sections of the association jointly sponsored a symposium entitled librarian’s role in reproducibility of research. the four hour symposium was held saturday, may 27th as part of the mla pre-conference activities. shona kirtley, knowledge and information manager for the equator network at the centre for statistics in medicine at the university of oxford, served as the keynote speaker. invited panelists discussed their roles in initiatives aimed to reduce the research waste caused by irreproducible reporting of scientific efforts within the published literature. the four-hour symposium concluded with a hands-on brainstorming activity that asked each of the attendees to propose and reflect on increasing the reproducibility of science. the following is a summary of the information shared and a reflection on the brainstorm suggestions made at the librarian's role in reproducibility of research symposium. a libguide for the event, including agenda and speaker slides are located at http://mlasymposium.libguides.com/c.php?g=584462&p=4036194. defining the reproducibility crisis in march 2012, a commentary on the reproducibility of preclinical cancer studies was published in nature. this report by the company amgen highlighted the disappointing success rate of translating basic science findings into clinical therapeutics, especially with regards to cancer studies. amgen researchers conducted a review of 53 published studies finding only six (11%) of the results could be replicated [1]. this endorsed an earlier report from the pharmaceutical http://mlasymposium.libguides.com/c.php?g=584462&p=4036194 research section spotlight hypothesis, vol. 29, no. 2, fall/winter 2017 33 company bayer that from a sample set of 67 published drug reports, only 20-25% were reproducible [2]. in response to these concerning observations, the lancet launched a series of papers that discuss increasing the value and reducing research waste in the published literature, ultimately becoming the lancet’s reduce research waste and reward diligence (reward) campaign in 2015. in 2016, nature published survey results from 1,576 researchers, 90% of whom affirmed a reproducibility crisis within the published literature [4]. in 2015, the u.s. national science foundation (nsf) released recommendations authored by the subcommittee on replicability in science. among other findings, this report noted that many terms used to discuss the ability to replicate or generalize a study were applied inconsistently. the report thereby offered the following definition: “reproducibility refers to the ability of a researcher to duplicate the results of a prior study using the same materials and procedures as were used by the original investigator (…) reproducibility is a minimum necessary condition for a finding to be believable and informative.” [3] this differs slightly from replicability, defined by the report as “the ability of a researcher to duplicate the results of a prior study if the same procedures are followed but new data are collected” [3]. in contrast, the nature survey of researchers asked if respondents were able to reproduce results in a “similar experimental system” which “may include slight variations in methods or materials” [4]. moving forward, as funders, publishers, researchers, librarians, and other stakeholders work to formulate strategies to address the concern, it will be important to work from a standard definition of the problem. many approaches would conceivably impact both reproducibility and replicability; however, when discussing the scale of the crisis, and proposing focused solutions, it will be important to note the difference. policies: what are funders and publishers doing to support reproducible research? proposed solutions for increasing the reproducibility (and replicability) of the published literature can typically be broken into two main approaches: those that target data reporting and those that target process reports, or methodologies. selective data sharing was cited as one of the top factors contributing to the crises by respondents of the nature survey. selective reporting may occur when authors publish a clean story, leaving out factors such as replicates that did not meet expectations, outliers, or statistical tests that did not show desired research section spotlight hypothesis, vol. 29, no. 2, fall/winter 2017 34 results. data sharing initiatives help address these selective pressures by requiring all underlying data of published summaries and visualizations be made available. this creates options for timely, independent verification and may limit exaggerated reporting. however to address the reproducibility crises rather than just re-use initiatives, more rigorous sharing of research design and data collection methods are also required. funders of research have put together recommendations and guidelines in both these areas as seen by the nsf data sharing policy (https://www.nsf.gov/bfa/dias/policy/dmp.jsp) and the nih rigor and reproducibility web portal (https://www.nih.gov/research-training/rigorreproducibility). additionally, organizations such as the center for open science (cos) (https://cos.io/) in the u.s. and the enhancing the quality and transparency of health research (equator) network (http://www.equator-network.org) in the u.k. also provide resources to address the process of sharing scientific findings. these organizations provide frameworks and guidelines for conducting and reporting reproducible scientific efforts. guidelines and toolkits for reporting common study types are linked on the equator network, while cos provides a framework for project management support, training, and updates on ongoing projects looking at field specific reproducibility issues. journal publishers have also responded to the call for increased reproducibility, though most have focused on data sharing strategies. science, nature, public library of science (plos), and proceedings of the national academy of sciences (pnas) have all established data sharing or data accessibility policies for their authors. authors in these journals are expected to share the data and statistics underlying their findings, preferably in a public domain repository. the sharing of such data allows for both reproducibility and confirmatory actives as well as the reuse of data for new discoveries. journal publishers focusing on the increased rigor of published methodologies have done so primarily by exploring new publication types. the journal of visualized experiments (jove) has been a leader in this area, developing a publication module of instructional videos aimed to better communicate experimental processes. as a panelist at the 2017 mla reproducibility symposium, jove co-founder and ceo, moshe pritsker, noted that the classic structure of the journal article has remained largely unchanged since the first publication in 1665 and called emphatically for detailed methods as a stand-alone publication model [5]. reinforcing his call, another panelist shared an example of the importance of robust materials reporting when a study found that variations in software version, workstation type (mac or pc), and mac https://www.nsf.gov/bfa/dias/policy/dmp.jsp https://www.nih.gov/research-training/rigor-reproducibility https://www.nih.gov/research-training/rigor-reproducibility https://cos.io/) http://www.equator-network.org/ research section spotlight hypothesis, vol. 29, no. 2, fall/winter 2017 35 operating system all had a significant impact on the analytic readings of 30 mri scans [6]. when variables such as computer type and operating system are shown to play a role in reproducible data analysis, there is a clear need for more robust reporting than just product names. registered reports offer another emerging publication type in support of reproducibility. these are detailed study designs which undergo the peer-review process separate from the final study analysis. registered reports describe key methodologies such as experimental model, data collection instruments, and statistical methods used in the proposed analysis. these are then reviewed for rigor and reproducibility before data are collected. currently, cos lists 52 journals which have adopted the publication of registered reports in some capacity. journals such as royal society open science and bmc biology will then provisionally accept results for publication, contingent upon adherence to registered study design, regardless of study outcomes. this slow shift in accepted publication types represents a positive cultural shift in the scientific community by placing emphasis on rigorous scientific processes rather than focusing solely on novel, positive results. library services: how can librarians support a culture of reproducible research? as kirtley emphasizes in her 2016 the lancet commentary, librarians are well posed to be part of the answer to managing the current reproducibility crisis [7]. academic and medical librarians are familiar partners in the research lifecycle, from developing robust and comprehensive search strategies, to selecting a journal and assisting with data sharing and management plans. librarians who work on systematic review projects are intimately familiar with the challenges of adapting complex methods to new database environments. by attuning specific knowledge and services to the language and needs of addressing reproducibility, librarians are equipped to serve as advocates and partners. even traditional roles such as collection development and access training can be easily adapted to addressing reproducibility as it relates to growing the awareness, availability, and utilization of new publication types. panelists at the 2017 mla reproducibility symposium shared unique ways librarians at their institutions have contributed to creating reproducible research. cynthia hudson-vitale, at the university of washington in st. louis, works with the institute for clinical and translational sciences to establish a framework for reproducible methods when working with electronic health records (ehrs). librarians collaborated with researchers to identify 103 variables needed to ensure the reproducibility of ehr analysis: beginning with stating a clear, focused hypothesis research section spotlight hypothesis, vol. 29, no. 2, fall/winter 2017 36 through tracking query language with version notes and access dates, identifying statistical tests and packages, and reporting such specifics with standard documentation. all 103 variables would be needed in any resulting publications or data codebooks or the analyses could not be reliably replicated, regardless if the raw de-identified data is available and accessible. bart ragon from the university of virginia and kristi holmes from northwestern university, echoed the vital importance for librarians to offer collaborative support for open science and data management. to further explore actionable roles and services, participants at the symposium completed a hands-on exercise in which each person was asked to propose a specific idea for librarian involvement and collaboration. ideas were then anonymously scored for how well the proposal resonated for an individual’s library and institution. collected from over the 30 responses, the highest scoring suggestions are listed below. while specifics of each suggestion were not discussed at the symposium, one possible interpretation of the proposal follows each participant idea. • host “reproduce-my-research” events such outreach or training events may take many different approaches. the idea seems to suggest giving researchers and students a formalized setting to reflect on and engage with each other specifically around how to improve the scientific reporting of their manuscripts. as this was the high scoring suggestion, it clearly resonated with many librarians who saw inspiration in this event title within their institutional outreach even without further details. • offer training to students and early career researchers many libraries already provide training opportunities to gain a better understanding and engage with resources, publishers, and services. refocusing or adopting specific language to target reproducibility concerns may be an easy adaptation for services already in place. • incorporate into the researcher workflow again, many libraries already have collaborations or specific services targeting various aspects of the research lifecycle. emphasizing these services as essential for addressing the reproducibility crisis may further campus collaborations and refine librarian roles. • establish data management best practices research section spotlight hypothesis, vol. 29, no. 2, fall/winter 2017 37 when working with data management strategies, many initial approaches supported by libraries may have been reactive to specific funder mandates. by proactively establishing best practices in line with robust and reproducible science, librarians are natural champions of more accessible research and open data. • educate librarians by seeking out learning and professional networking opportunities such as the 2017 reproducibility symposium, librarians are educating themselves about the problem, understanding researcher frustrations and emerging policies, and collaborating as a profession to explore targeted services. opportunities to brainstorm and share challenges and success stories are invaluable for the development of robust and innovative services. • collaborate with institutional offices collaboration is implied in many of the top-scoring suggestions. continued outreach to researcher groups and campus partners will be essential to the recognition of librarians as part of the solution. • provide high-quality, reproducible search results for those librarians working on systematic reviews or providing in depth reference support, this is an opportunity to lead by example. providing details of the search query along with specifics such as database coverage, search date and applied filters, raises subtle awareness of the necessity of these details. this may then provide the opportunity for discussion of similar essentials when reporting results. the majority of these high scoring responses reflect activities and services already underway, with some variation, at many academic libraries. by educating librarians and raising awareness among researchers in the specific areas of reproducible research, many library services can be easily adapted to address the reproducibility crises. hosting additional symposiums and workshops around the topic will encourage other librarians to share the specific services and outreach initiatives that have been successful. research section spotlight hypothesis, vol. 29, no. 2, fall/winter 2017 38 references 1. begley cg, ellis lm. drug development: raise standards for preclinical cancer research. nature. 2012;483(7391), 531-533. 2. prinz f, schlange t, asadullah k. believe it or not: how much can we rely on published data on potential drug targets? nat rev drug discov. 2011;10(9), 712-712. 3. bollen k, cacioppo jt, kaplan rm, krosnick ja, olds jl, dean h. reproducibility, replicability, and generalization in the social, behavioral, and economic sciences. 2015. retrieved from https://www.nsf.gov/sbe/sbe_spring_2015_ac_meeting_presentations/bollen_report_on_re plicability_subcommitteemay_2015.pdf 4. baker m. 1,500 scientists lift the lid on reproducibility. nature. 2016;533(7604), 452-454. doi:10.1038/533452a 5. pritsker m. the science reproducibility crisis: how librarians can solve it and why they should care? paper presented at the the librarian's role in reproducibity of research symposium, seattle, wa. 2017. 6. gronenschild ehbm, habets p, jacobs hil, mengelers r, rozendaal n, van os j, marcelis m. the effects of freesurfer version, workstation type, and macintosh operating system version on anatomical volume and cortical thickness measurements. plos one. 2012;7(6), e38234. doi:10.1371/journal.pone.0038234 7. kirtley s. increasing value and reducing waste in biomedical research: librarians are listening and are part of the answer. the lancet. 2016;387(10028), 1601. doi:http://dx.doi.org/10.1016/s0140-6736(16)30241-0 -- https://www.nsf.gov/sbe/sbe_spring_2015_ac_meeting_presentations/bollen_report_on_replicability_subcommitteemay_2015.pdf https://www.nsf.gov/sbe/sbe_spring_2015_ac_meeting_presentations/bollen_report_on_replicability_subcommitteemay_2015.pdf http://dx.doi.org/10.1016/s0140-6736(16)30241-0 projects & programs hypothesis vol. 31 no. 1 fall/winter 2019 research roadmap: disseminating your research (part 2) katherine g. akers, phd biomedical research/research data specialist shiffman medical library wayne state university detroit, mi kristine m. alpi, phd, mph, mls, ahip university librarian oregon health & science university library oregon health & science university portland, or jonathan d. eldredge, phd, mls, ahip evidence based and translational science collaboration coordinator health sciences library and informatics center university of new mexico new mexico, nm heather n. holmes, mls, ahip associate director of libraries musc libraries medical university of south carolina (musc) charleston, sc kimberly r. powell, mlis research impact informationist woodruff health sciences center library emory university atlanta, ga margaret a. hoogland, mls, ahip assistant professor, clinical medical librarian mulford health sciences library university of toledo toledo, oh editor’s note: this piece is the second of two parts from the “research roadmap” session delivered at the 2018 annual meeting of the medical library association. part 1 covered advice and guidance about doing research while part 2 discusses how to share the products of your research. part 1 can be found in hypothesis vol. 30, no. 1, fall/winter 2018. https://www.mlanet.org/p/cm/ld/fid=737&tid=501&sid=2916 https://www.mlanet.org/p/cm/ld/fid=737&tid=501&sid=2916 projects & programs hypothesis vol. 31 no. 1 fall/winter 2019 presenting at a conference as you near the completion of your research project, you will probably begin thinking about whether to present your project at an upcoming regional or national conference. should you present your research at a conference? the answer is “yes,” unreservedly “yes.” there is no point to conducting research if you do not disseminate your results to other researchers and practitioners, this is the distinction between data gathering for quality improvement for internal use and research intended for sharing. presenting at a conference is often a first step in disseminating your research and can have several benefits, including building your professional identity as a researcher, forming new connections with colleagues and potential collaborators, allowing your colleagues to build upon your experiences and findings, adding another line to your cv to boost your tenure, promotion, or future career endeavors, and getting preliminary feedback on your project before you write it up and submit to a journal. perhaps a more pertinent question is: should you present your research as a poster or as a talk? the answer to this question is “it depends”—on factors such as your preferred mode of communicating with others, the content of your presentation, your primary motivation for presenting, and the size of the conference. talks are often considered more prestigious than posters. if you seek public recognition as a researcher and an expert in your research area, you may opt to give a talk so that your name receives more emphasis on the conference program and your face becomes more familiar to your peers. however, weigh your personal preferences for social interaction into this decision. perhaps the thought of standing alone at the front of a room and speaking to an audience induces deep fear or anxiety. if you are this kind of person, remember that the fear of public speaking often diminishes gradually over time with repeated experience—it really does get easier the more you do it. on the other hand, perhaps you dread the thought of standing awkwardly at a poster for hours, enticing not a single viewer while your poster-presenting neighbors draw in crowds. if you are this kind of person, it may be more appealing to “get it over with” by standing at a podium and speaking for 15 minutes, provided that you can sufficiently quell your nervousness. essentially, if you can stomach the thought of giving a talk at a conference, then go for it! in some circumstances, presenting a poster may clearly be a better choice. for instance, a poster is better for showing information visualizations that require closer examination than a brief glimpse on a slide, such as network graphs, flow diagrams, timelines, or other complex charts or graphs. you may also be more inclined to present a poster if your goal is to solicit a lot of feedback on your research or to generate conversations with colleagues. an abundance of tips for creating visually appealing posters that clearly and effectively convey your intended message can be found online and in the published literature; essentially, aim for clean and concise. projects & programs hypothesis vol. 31 no. 1 fall/winter 2019 finally, the potential impact of a talk versus a poster can differ markedly depending on the size of the conference. if you are attending a large national or international conference with multiple streams, a talk will compete with other talks or conference events occurring at the same time, which can result in a disappointingly small audience. thus, a poster might ultimately receive more viewings, particularly if it can be left up for one or more days. by contrast, if you are attending a small local or regional conference with only one stream, you will have a more “captive audience” for your talk, whereas a poster session may be more likely to be skipped by attendees opting for down-time or catching up with colleagues. in other words, a poster may be more impactful at a larger conference, and a talk may be more impactful at a smaller conference. you can hardly make a wrong decision; presenting your research at a conference can be extremely meaningful—to both you and your audience—regardless of its presentation format. understanding the publishing process next comes the more daunting question. should you publish your research in a journal? the answer to this question is “yes, but . . .”. publishing your research in a journal is extremely satisfying, can mark the successful completion of your project, and is important if not critical for building your identity as a researcher. however, publishing a journal article takes a lot of time and effort, requires strategic decision-making, and can be an arduous if not maddening process. it is worth it. before starting to write a manuscript describing your research, identify a target journal so that you can write to their specifications. we provide guidance for tackling each of the major sections of an article in an imrad (introduction, methods, results, and discussion) format. however, it is often easiest to write a manuscript “from the inside out”—first writing the methods and results as you prepare your protocol and analyze your data, and then writing the introduction and discussion. writing the manuscript a. introduction the introduction section is very important for framing your study and its purpose. it is wise to compose the introduction as a logically constructed argument that convinces reviewers (and future readers) that your study is a valuable contribution to the literature. the lack of a previously published study investigating your research question is not sufficient motivation; rather, it is important to identify a clear gap in knowledge or practice that your study seeks to fill. writing an introduction section can be as easy as following a three-step formula: (1) dedicate the opening paragraph to describing a higher-level (e.g., national, disciplineor profession-wide) issue or trend that provides broad context for your study and attracts readers, (2) use one or more paragraphs to present specific complexities or problems that serve to narrow into your research projects & programs hypothesis vol. 31 no. 1 fall/winter 2019 question, and (3) conclude with a paragraph that describes the specific objective(s) of your study. if the journal you are submitting to employs the imrad format, a separate literature review section is likely unnecessary. instead, references to the existing literature should be woven into your argument (and not simply presented as a list, such as “author a found this. author b found that.”). a well-constructed introduction can be accomplished in as few as 1-3 double-spaced pages. b. methods the methods section explains your procedures and analyses with enough detail to allow others to replicate them. from the perspective of a journal editor, reviewers hardly ever complain that the methods section is too detailed; instead, unclear or incomplete description of the methodology is a much more common reviewer complaint. your methodology will be more easily understood by readers if the methods section is well-organized (e.g., in temporal order, by technique or approach) and employs subheadings. c. results the way in which you lay out the results section can vary dramatically depending on your study design and actual findings. the goal is to present your results in a manner that is intuitive to readers using text, tables, and visualizations (i.e., figures). sometimes your results section will “write itself”; other times you may need to experiment with different ways of presenting your results before you discover the most effective approach. do not include tables or figures “just because”; reserve the use of tables and figures for data that cannot be easily described in a couple of sentences. for example, a simple one-or two-columned table, a pie chart, or a bar graph showing a single difference between two groups is often unnecessary, as these data can most likely be easily reported in the text. at the same time, however, try not to make your tables and figures overly complicated. the purpose of tables and figures is to help readers discern patterns in your data; reader should not have to spend a great amount of time studying a table or figure to comprehend the key information it is intended to convey. avail yourself of resources and training on the effective visualization of data and other information. furthermore, there should be no redundancy between the text and the information presented in tables and figures; although the content of tables and figures should be generally described in the text, specific numerical values shown in tables and figures should not be duplicated in the text. nearly all research articles will report some type of descriptive statistics, such as frequencies (e.g., counts, percentages) and measures of central tendency (e.g., mean, median) and spread (e.g., standard deviation, range). however, to make certain claims, you may also need to perform inferential statistical analyses, such as chiprojects & programs hypothesis vol. 31 no. 1 fall/winter 2019 squared tests or correlation coefficients. for example, if you state that students in an active learning condition scored higher on a test than students in a passive learning condition, this must be supported by inferential statistics showing that the difference between groups is greater than would be expected by chance. when inferential statistics are needed, report not only p-values but also other test statistics such as the χ2 value, t value, or pearson correlation coefficient (r) and the corresponding degrees of freedom. when in doubt, consult with someone who is knowledgeable about statistical analysis (and consider including them as a co-author!). be sure to keep the results section free of methods and discussion. while it may be important to remind readers why or how a certain analysis was performed, methodological procedures should not be described again or for the first time in the results section. also, while data can be objectively characterized in the results section, their subjective interpretation should be restricted to the discussion section. d. discussion the discussion section can be more free-flowing than other sections, but it should still be a concise, logically constructed argument that achieves strategic goals. most importantly, do not simply restate your results in the discussion section; rather, go beyond a summary of your findings and discuss what those findings might mean, whether they support or contradict previous research findings or existing theories, and how they extend the knowledgebase or advance practice. address any limitations of your study and how they might impact your results or their interpretation. explore the implications of your findings for future research or practice, particularly considering the journal’s primary audience. engage the literature; the discussion section should be rich with references to previous work in the area. each paragraph in the discussion should make its own unique point, with smooth transitions between paragraphs. although subheadings can be used in the discussion section, good writing can eliminate their need. reviewers and readers look favorably upon a concluding paragraph that briefly synthesizes your findings and reiterates the study’s contributions to the literature and implications. navigating the peer review process at some point, after taking great care to produce a high-quality research manuscript, you must turn it over to a journal and hope that it survives the gauntlet of peer review. reviewer comments can never be fully anticipated, and the path that a manuscript can travel through the peer review process can be long and convoluted. as you are already likely familiar with the basics of peer review, here are “pro tips” for increasing your chances of manuscript acceptance. projects & programs hypothesis vol. 31 no. 1 fall/winter 2019 a. sending pre-submission inquiry if you are uncertain about whether your manuscript would be a good fit for a journal, you should send the journal editor an email to gauge his or her interest. b. submitting the manuscript and associated documents cover letters used to be standard when manuscripts were physically mailed to the editor. more recently, cover letters have become optional, although they are still required by some journals. should you submit a cover letter with your manuscript? if you have specific information that you want to convey to the editor, then “yes”. for instance, you may want to highlight why your manuscript is particularly valuable to the field or relevant to that journal’s readership. you may want to explain the history of the manuscript (e.g., if parts have been published elsewhere) and any potential or perceived conflicts of interest. also, you may want to suggest specific individuals who would be good reviewers of your manuscript or who you would prefer not to review your manuscript due to personal or professional reasons (although the journal editor is under no obligation to honor those suggestions). if the submission system requires you to recommend reviewers, one way is to find them is to look for authors who have published similar work but have no affiliation with your project. c. checking in on manuscript status online journal submission systems will often show the status of your manuscript as it moves through the editorial and peer review process. however, if your manuscript appears to get “stuck” at a certain point or you feel that the review is taking an unreasonably long time, you may judiciously and politely contact the editor to inquire about the status of your manuscript. typical reasons for delays include difficulty finding reviewers for the manuscript or tardy reviews. d. receiving reviewer comments and the editorial decision even before you receive an editorial decision on your manuscript, you should be mentally prepared to revise your manuscript to address the reviewers’ concerns. outright acceptance of a peer-reviewed manuscript without revisions is extremely rare, as reviewers will invariably spot a weakness or make recommendations for improving the manuscript. also know that sometimes editorial decisions are very difficult to reach, especially if reviewers provide very different evaluations of the manuscript. it is completely normal to feel disappointed, frustrated, or angry upon your first read of the reviewers’ comments and editorial decision. let that emotional response settle overnight or for a few days before deciding how to respond. often, what initially seems to be insurmountable reviewer criticisms can often be addressed without major alterations to the manuscript. projects & programs hypothesis vol. 31 no. 1 fall/winter 2019 e. addressing reviewer comments with few exceptions, every reviewer comment should be addressed by a change, however slight, to the manuscript. even in instances in which a reviewer overlooked or misunderstood some aspect of your study, take that as an indication that you might not have been completely clear or direct in your writing, and edit accordingly. the editor may ask you to submit a point-by-point reply to the reviewers’ comments along with the revised manuscript. in this reply letter, compose your responses with a respectful yet confident tone and indicate briefly what changes you made in the revised manuscript to address each comment. keep in mind that the editor may send this reply letter along with the revised manuscript back to the original reviewers, who may be asked to re-evaluate the revised manuscript. expect that the reviewers will want to see evidence that you seriously considered their concerns. if you are completely perplexed about how to adequately address a reviewer’s comment, such as when two reviewers make contradictory recommendations, you can contact the editor for clarification or advice. f. asking for reconsideration if an editor declines your manuscript, that decision is probably final. however, if you strongly feel that the reviewers’ evaluations of your manuscript were biased or inaccurate or that you would be able to perform major revisions (e.g., collecting new data, using different approach to analysis) that would resolve the critical issues, you could send the editor an email asking them to reconsider their decision or inquiring whether they would be amenable to receiving a significantly reworked manuscript as a new submission. [see also the q&a in part 1] conclusion writing this article about the session was our first step in sharing our experiences in research with mla members and broader audiences. it was very enlightening to encourage research questions from our members in this approach. while each of us has served as a research mentor, those questions tend towards either the project-specific or personal development. to have multiple researchers considering the same questions for a diverse audience was a unique experience. while many of us would provide similar feedback, there are areas in this paper where each author may have a different perspective, particularly on overcoming barriers and obstacles. depending on your job title, duties, concerns for life balance, the guidance to find time or change duties may not resonate. please be in touch with the research caucus if you have questions or suggestions for further research education programming opportunities. hypothesis: failure hypothesis vol. 31 no.1 fall/winter 2019 introducing hypothesis: failure heather n. holmes, mlis, ahip associate director of libraries, associate professor medical university of south carolina charleston, sc welcome to the inaugural hypothesis: failure column. it is our sincere hope that readers will benefit from the lessons learned by their colleagues, and will contribute their own stories to help others. our first article comes to us from elaine wells, library director of the suny college of optometry, about a renovation project. this article is especially exciting to me because my current library is preparing for a renovation to begin next year. renovation is one of the words that librarians love to hear and then immediately cringe at what it might actually mean. in my past experience, when a library undergoes a renovation it is because someone else needs space. this isn’t always a bad thing, and in fact i think in some cases it has worked out to the benefit of the library; however, i think it is fair to say that in a good number of cases the library (and more importantly, our patrons) gets the short end of the stick. that is not always the case though, as you’ll see in the contributed article. as far as the renovation at my library, i think we’ll be seeing both great benefits, and having some pains due to losing a lot of space. the loss of space, however, isn’t really being taken from the students but rather being redistributed throughout the building which houses other spaces besides the library. there will be some muchneeded improvements to our built-in-1968-building, the most anticipated being access to electricity from places other than the perimeter of the building. furniture upgrades, and soundproofing of walls are also things we’re really looking forward to. the effects of the loss of space remain to be seen, but one positive is that all of the faculty and staff (except for 2 people) will be on the same floor. we hope this leads to improved workflows but there is a fair bit of concern about staff who currently have substantial workspaces being moved into cubicles. perhaps i can contribute my own article to this column in a couple years when ours is completed! sharing information is at the base of what we as librarians do so we owe it to each other to let others learn from our experiences. we hope you’ll consider submitting your own articles for future issues. heather n. holmes awards hypothesis vol. 32 no.1 fall/winter 2020 congratulations to the biannual jmla best research paper and the winning research papers and posters from mla ‘20! the mla research caucus is pleased to announce the winners for best research papers and posters presented at the mla 2020 virtual meeting. thank you to all the judges who volunteered their expertise to help select these deserving awardees both in the pre-judging phase and at the virtual conference. to learn more about the awards and selection process, visit the research section website at http://www.mlanet.org/p/cm/ld/fid=938. jmla biannual best research paper 2017-2019 mendez im, pories ml, cordova l, malki a, wiggins mf, lee jgl. a pilot project to increase health literacy among youth from seasonal farmworker families in rural eastern north carolina: a qualitative exploration of implementation and impact. j med libr assoc. 2019;107(2):179–186. https://doi.org/10.5195/jmla.2019.560 mla ‘20 winning research papers first place is the open access citation advantage real? a systematic review • caitlin bakker, research services librarian, university of minnesota, minneapolis, minnesota • amy riegelman, social sciences librarian, university of minnesota, minneapolis, minnesota • allison langham-putrow, scholarly communications librarian, university of minnesota, minneapolis, minnesota objective: the potential for open access (oa) publication to increase citation rates of articles was first articulated in 2001. since then, support for and refutation of the oa citation advantage has been abundant. oa’s influence on citation remains unclear, particularly across disciplines, data sources, and methodological approaches. this systematic review aims to determine if the oa citation advantage is real. methods: we conducted a systematic search of the literature in accordance with mecir standards to identify all publications that compared citation rates of oa and non-oa publications. we executed this search across seventeen databases representing a broad range of disciplines. title and abstract screening, full-text screening, data extraction, and risk of bias assessment were completed by two independent reviewers and discrepancies were resolved through consensus or by a third party where necessary. we extracted data to describe both the exposure (oa) and control (non-oa) groups, including number of included studies, as well as cumulative citations and data source of citations. we also recorded how open access was defined, how samples were identified, and the citation window http://www.mlanet.org/p/cm/ld/fid=938 https://doi.org/10.5195/jmla.2019.560 awards hypothesis vol. 32 no.1 fall/winter 2020 considered. risk of bias assessment was completed to assess underlying methodological quality of the component studies. results: with duplicates removed, we screened 2,108 titles and abstracts. at this phase, 1866 items were removed, leading to 242 full-text articles being assessed and ultimately 115 items being included in qualitative synthesis. these articles represented a broad range of disciplines, data sources, and outcome measures. data extraction also uncovered notable issues with incomplete reporting. fifty-four of the included studies reported an open access citation advantage while 28 reported an advantage in subsets and 32 reported no citation advantage. one study reported inconclusive results. risk of bias assessment and quantitative synthesis are currently underway. second place flawed research in focus: retracted publications in pharmacy systematic reviews • caitlin bakker, research services librarian, university of minnesota, minneapolis, minnesota • sarah jane brown, liaison librarian to the college of pharmacy and medical school, university of minnesota, minneapolis, minnesota • nicole theis-mahon, liaison to the school of dentistry & collections coordinator, university of minnesota, minneapolis, minnesota objective: publications are retracted for a multitude of reasons; however, identification of retraction notices remains inconsistent and post-retraction use of materials continues. we investigated how retracted publications influence systematic reviews and other evidence-based literature in pharmacy. we analyzed retracted publications cited in systematic and other comprehensive reviews and examined the application of quality assessment or risk of bias tools. methods: retracted research articles and clinical studies in the fields of pharmacology, drug design, and toxicology were identified through the retractionwatch database. searches were performed in scopus and web of science to identify all articles and reviews citing each retracted item. these results were collected and deduplicated in endnote and uploaded to rayyan for screening. the included systematic and other comprehensive reviews were then assessed to determine whether the retracted publication was cited positively, negatively, or neutrally in support of the findings. we also examined which, if any, quality assessment or risk of bias tool was used in the systematic review and what the results of that evaluation were. we conducted an analysis of citation trends to show the impact of retracted publications in systematic reviews and the methodological quality of those reviews. awards hypothesis vol. 32 no.1 fall/winter 2020 honorable mention understanding the health information practices of lgbtq+ communities to improve medical librarian services • travis wagner, doctoral candidate, university of south carolina, columbia, south carolina • nick vera, phd student, university of south carolina, columbia, south carolina • vanessa kitzie, assistant professor, university of south carolina, columbia, south carolina objective: this multi-method, three-year qualitative study addresses the following research questions: (1) how does socio-cultural context shape the information creation, seeking, sharing, and use of health information among lgbtq+ communities? (2) how can these findings inform medical librarian services to lgbtq+ communities for health promotion? methods: data collection consists of 30 individual semi-structured interviews with lgbtq+ community leaders from state (completed), 6-8 focus groups with leaders' communities (in progress), and a community forum informed by the world café methodology between 30-40 leaders and librarians (in progress). individual and focus group interview participants also engaged in information worlds mapping, a visual arts-based elicitation method. data for analysis are verbatim transcripts, analytical memos of information worlds maps, community forum notes, and researcher field notes and reflexivity journals. data analysis follows qualitative open coding and constant comparison methods. line-by-line, first-cycle process coding identifies initial codes, which the researchers compare, combine, and refine via subsequent data collection and analysis. second-cycle axial and theoretical coding informs development of a conceptual model that describes key coding categories and the relationships between them. peer debriefing and participant memberchecking serve as validity checks. results: preliminary interview findings invert deficit models of lgbtq+ health and information practices. these models position communities as lacking resources and knowledge to improve their social conditions and envision experts as able to “correct” this deficit. participants challenged these presumptions by identifying social and structural factors, including experts, as hindrances to achieving positive health outcomes and tactically responding to these constraints. for example, several communities stated that being misgendered at the doctor’s office took a significant toll on their mental health. they responded to this lack of expert competency by developing lists of community-approved medical professionals based on collective information assessment. conclusions: inverting the deficit model to view experts rather than lgbtq+ communities as lacking has implications for social and structural change. from the position of medical librarianship, this change can occur via a shift from outreach, which focuses on information and resource provision, to engagement, which centers awards hypothesis vol. 32 no.1 fall/winter 2020 community expertise as the driver for information and resource development. three specific implications informed by this shift and emergent research findings are establishing partnerships with community health workers, facilitating cultural competency training for medical professionals, and offering harm reduction workshops. honorable mention seeing our open access (oa) options: a comparison of full text finders • elizabeth moreton, clinical librarian, university of north carolina, chapel hill, north carolina • jamie conklin, health sciences librarian, university of north carolina, chapel hill, north carolina • adam dodd, data analyst, health technology and informatics, university of north carolina, chapel hill, north carolina objective: oa full text finders offer the potential to save researchers time and money in article retrieval with just the click of a button by finding free copies of articles, but to date, studies comparing the user experience and retrieval capabilities of these tools are scarce. this study analyzes the features and effectiveness of oa full text finders in health disciplines. methods: the investigators tested several types of oa full text finders by attempting to retrieve the full text of a random set of articles in a variety of health disciplines. they performed a structured analysis of the software, looking at quantitative and qualitative dimensions such as data sources, number of steps to retrieve text, and overall ease of use. the investigators also tested the effectiveness of the tools by comparing the number of test articles retrieved by each tool. results: overall, google scholar button performed the best regardless of browser. lazy scholar and endnote also performed well. a combination of google scholar button in chrome, which found the most articles, combined with endnote, which missed the fewest articles, may be the best approach. it would also be easy to install several tools at once and check multiple sources almost instantly. conclusions: though there are many types of oa full text finders, it may save potential users time and money to know which tool is easiest to use and provides access to the most free resources. mla ‘20 winning research posters first place applying citation and usage analysis to evaluate the e-journal package collection in a medical university library • hua-yu hsu, librarian, taipei medical university library, taipei, taiwan awards hypothesis vol. 32 no.1 fall/winter 2020 • tzu-heng chiu, director, taipei medical university library, taipei, taiwan • chun-huei shen, head of technical services, taipei medical university library, taipei, taiwan objective: e-journal packages accounts for 60% of overall e-resources budget in taipei medical university library (tmul); however, some titles included in them are not needed by users. in 2019, the authors tried to analyze the title fill rate and cost per full-text download of all e-journal packages. the research results can serve as the reference for our e-journal collection development and decision making in the future. methods: methodologies of citation analysis and cost of usage were applied in this study. the authors believe that the cited journals of our faculty’s papers represent their real research demand, and it could reflect the utilization efficiency of our ejournal collection. therefore, we exported 2016-2018 publication of tmu faculty from the wos sci/ssci database, which results in 4,698 journal articles with 181,422 citations. we then utilized the list-checking method to compare bibliographic data of these 181,422 citations with our e-journal collection for title fill rate to find out how our collection could support the research need of tmu faculty. we also analyzed source journals of those 181,422 citations, including its frequency of publication, times cited, and publisher/ package which it belongs to. in addition, the costs per full-text download for each e-journal package were calculated based on their subscription fee and download statistics. results: the title fill rate of 181,422 citations is 74% (among which 134,749 are the tmul e-journal collection). the top three most cited e-journal packages in 20162018 are "sdol" (1,080 titles), followed by " wileyonlinelibrary " (507 titles), and “springer” (496 titles). the cited rate and cost per download of each e-journal package are shown in figure 1 of the poster. the highest utilization efficiency ejournal packages are "jama" and "cell press," which have lowest cost per full-text download (us$ 0.81, us$ 1.27) and highest cited rate (90.9%, 90.5%). in addition, the lowest utilization efficiency e-journal package is "wileyonlinelibrary" (us$ 3.6, 53.8%). second place comparing three models for librarian office hours in a school of pharmacy • emily gorman, research, education & outreach librarian, university of maryland school of pharmacy, baltimore, maryland objective: to determine which of three office hour models was most effective in increasing librarian visibility in and engagement with a school of pharmacy. setting/population: one librarian is the liaison to a school of pharmacy (sop) that includes approximately 90 full-time faculty, 300 staff, 800 affiliate and preceptor awards hypothesis vol. 32 no.1 fall/winter 2020 faculty, and 900 students. the library is across campus from the buildings where most sop personnel are located. methods: the librarian held weekly office hours in the main sop building. the “lobby” model involved two hours per week at a table in the lobby, the “hybrid” model increased the lobby time to four hours and added additional hours in an office, and the “office” model removed lobby hours and increased the office time to a full day (7.5 hours). the librarian tracked all interactions and classified them in the following categories: brief chat, brief hello, brief question, and in-depth question. descriptive statistics were calculated to compare the number of interactions by category and population. results: results indicate that the lobby model had the most interactions (71), followed by the office model (39), with the hybrid model having the least (29). the office (9) and hybrid (8) models had more in-depth questions than the lobby model (3), but the lobby model had a higher number of brief questions (25) than the office (13) or hybrid (9) models. the number of student interactions differed most drastically, with the lobby model having far more (28) than the hybrid (6) or office (3) models. conclusions: based on these results, holding office hours in the building lobby is most effective for increasing librarian visibility in the sop. however, people appear to be more comfortable asking in-depth questions in the office setting. there are advantages and disadvantages to each model depending on the goal of the office hours—visibility versus in-depth engagement and support. honorable mention focus on nursing point-of-care tools: developing criteria for an evaluation rubric • emily m. johnson-barlow, regional health sciences librarian, university of illinois at chicago, chicago, illinois • annie nickum, information services and liaison librarian, university of illinois at chicago, chicago, illinois • rebecca raszewski, information services and liaison librarian, university of illinois at chicago, chicago, illinois • ryan rafferty, regional health sciences librarian, university of illinois at chicago, chicago, illinois objective: registered nurses have unique practice needs and many resources are marketed to support their practice. point-of-care tools provide evidence-based information on patient care and procedures at the time of need. this study aims to review five point-of-care tools based on their coverage, content, and transparency to support selection of a point-of-care tool for the registered nurse. methods: investigators selected five point-of-care tools cited in the literature: awards hypothesis vol. 32 no.1 fall/winter 2020 clinicalkey for nursing, dynamed, lippincott’s advisor and procedures, nursing reference center plus, and uptodate. the investigators developed a rubric containing evaluation criteria based on these point-of-care tools’ content, coverage of nursing topics, transparency of the evidence, user perception, and customization of the tools for supporting nursing practice. thirty-five identified classified nursing terminologies, nanda (13), nic (11), and noc (11), were used to examine the breadth of coverage within each point-of-care tool. four investigators independently extracted criteria using the rubric and reported descriptive statistics of the results. results will inform the decision-making process of recommending a point-of-care tool for nurses at our academic medical center. results: lippincott had the highest coverage of diagnoses (nanda) while clinicalkey for nursing had strong content focused on intervention (nic) and outcomes (noc). nursing reference center plus provided the most well-rounded coverage of terminology. dynamed and uptodate were more transparent with indicating conflict of interest but included little content on core measures (jcaho) or cultural competencies compared to the nursing-specific point-of-care tools. both uptodate and dynamed had lower coverage of nursing terminology and care processes. user perception was evaluated; however, the criteria was deemed to be influenced by our librarian expertise. conclusions: none of the five tools successfully met all of the evaluated criteria. the rubric developed for this study highlights each tools’ strengths and weaknesses that can then be used to inform the decision-making process to select a point-of-care tool based on priorities and budget. of the tools reviewed for this study, the investigators recommend utilizing two or more to provide comprehensive, evidence-based, patient care coverage and meet the diverse information needs of nurses. jmla biannual best research paper 2017-2019 volume 29, no 1 spring/summer 2017 hypothesis the journal of the research section of the medical library association hypothesis, vol. 29, no. 1, spring/summer 2017 1 volume 29, no 1 spring/summer 2017 hypothesis the journal of the research section of the medical library association editorial springing into the new year: hypothesis 2.1 ……...... 3 erin d. foster original research mla research section’s research agenda committee systematic review project: a status report ……………………………………………………. 5 marie t. ascher & jonathan d. eldredge of interest introducing the column ……………….…………….18 carol l. perryman research section news mla 2017 debrief ……….…………………………....... 20 erin d. foster linear regression from wikimedia commons, the free media repository https://commons.wikimedia.org/wiki/ file:linear_regression.svg hypothesis (issn 1093-5665) is the official journal of the research section of the medical library association. hypothesis is indexed in the cumulative index to nursing and allied health literature™ (cinahl). hypothesis is available online at http://www.mlanet.org/p/cm/ld/fid=503. http://www.mlanet.org/p/cm/ld/fid=503 hypothesis, vol. 29, no. 1, spring/summer 2017 2 editors erin d. foster, msls data services librarian ruth lilly medical library indiana university school of medicine edfoster10@gmail.com carol l. perryman, phd associate professor school of library and information studies texas woman’s university cperryman@twu.edu editorial board abby adamczyk, mlis liaison librarian for medicine drexel university libraries ala99@drexel.edu brooke l. billman, mlis research section chair-elect program chair medical librarian college of american pathologists bbillma@cap.org krystal bullers, mls emerging technologies librarian, pharmacy liaison shimberg health sciences library university of south florida health kbullers@health.usf.edu susan fowler, mls systematic review services coordinator school of medicine washington university fowler@wustl.edu ayaba logan, mlis, mph research and education informationist musc libraries medical university of south carolina (musc) loganay@musc.edu ex officio marie ascher, ms, mph, ahip research section immediate past chair nominating committee chair section council rep. lillian hetrick huber endowed director, health sciences library new york medical college marie_ascher@nymc.edu michelle b. bass, phd, msi research section chair section council alt. rep. population research librarian lane medical library stanford school of medicine michellebbass@stanford.edu margaret hoogland, mls, ahip research section community manager assistant professor, clinical medical librarian, health science library mulford health sciences library university of toledo margaret.hoogland@utoledo.edu editorial erin d. foster hypothesis, vol. 29, no. 1, spring/summer 2017 3 ‘springing’ into the new year – hypothesis 2.1 erin d. foster, msls co-editor, hypothesis data services librarian ruth lilly medical library indiana university school of medicine indianapolis, in welcome to the spring/summer 2017 issue of hypothesis! as of january 2017, i stepped into the role of editor many thanks to dr. christine marton for her leadership in revitalizing hypothesis and her time as lead editor. in addition, thanks to the hypothesis editorial board, marie t. ascher, and co-editor, carol l. perryman, for their hard work in putting together the spring/summer 2017 hypothesis issue! i am grateful for the time, interest, and support contributed by all in realizing this issue of hypothesis. with that, i would like to take this opportunity to introduce dr. carol l. perryman as the new co-editor of hypothesis! carol served on hypothesis’s editorial board previously, but expressed interest in becoming a co-editor following christine marton’s departure. a former hospital librarian who began an academic career after many years of practice, carol’s research interests include evidence-based practice, critical evaluation, and more recently, the changing roles of medical librarians in practice. i look forward to carol’s insight, expertise, and guidance given her significant experience in the field of library science research. the centerpiece of this issue is an update on the status of the research section’s research agenda committee systematic review project. written by marie t. ascher and jonathan d. eldredge, this article provides an overview of the questions posed by the 15 systematic review teams, brief updates as to the various teams’ progress, and a discussion of the challenges faced so far in this project. this issue also introduces a new column that aims to highlight a variety of topics that may be of interest to medical librarians conducting research including: research ‘works in progress’ taking place in the medical librarian community, summary results of failed and/or unpublished research, as well as links to and information about relevant research studies published in other disciplines. finally, since we’ve recently wrapped up mla 2017, the editorial erin d. foster hypothesis, vol. 29, no. 1, spring/summer 2017 4 research section news portion of this issue provides a debrief on section sponsored mla programming and a listing of some research section member contributed papers and posters! lastly, i want to emphasize hypothesis’ role as a platform for medical librarian research. as our profession grows in diversity and our roles shift and adapt in the larger communities we are part of, dissemination of the research we do is vital to furthering our profession and emphasizing the unique lens that medical librarians provide as part of the research enterprise. we accept submissions to hypothesis on a rolling basis – please, do not hesitate to reach out if you are interested in contributing. while we may be revamping our submission process soon, the easiest way to submit your work is to email either of the editors at the email addresses listed at the beginning of this issue. feel free to get in touch with any questions/comments. you can keep track of hypothesis news and updates through the research section’s webpage on mlanet.org as well as via the research section of the medical library association facebook group. we hope to see your research soon. --http://www.mlanet.org/p/cm/ld/fid=503 http://www.mlanet.org/p/cm/ld/fid=503 https://www.facebook.com/groups/mlaresearch https://www.facebook.com/groups/mlaresearch research section spotlight hypothesis, vol. 30, no. 1, fall/winter 2018 61 mla 2018 research section research awards congratulations to the winning research papers and posters from mla ’18 and the winner of the best biennial jmla research paper! the mla research section is pleased to announce the winners for best research papers and posters presented at the mla 2018 annual meeting in atlanta, ga. thank you to the 31 judges who volunteered their expertise to help select these deserving awardees. to learn more about the awards and selection process, visit the research section website at http://www.mlanet.org/p/cm/ld/fid=938. contributed posters 1st place authors: sarah wright – clinical librarian, university of north carolina-chapel hill, chapel hill, north carolina rebecca mccall – clinical librarian, university of north carolina-chapel hill, chapel hill, north carolina title: clinical librarians and their essential services in academic and health care settings: a comparative survey objectives: while there are many indicators and models of individual successful clinical librarian services and programs in health sciences libraries in the literature, there is no recent, comprehensive summary of services provided by clinical librarians. this poster, based on a current survey of clinical librarians, provides summary data of clinical librarian service models in the us and internationally. methods: in february, 2018, we distributed an online survey to librarians who currently provide clinical services in academic and/or hospital settings. requests to complete the survey went to multiple library listservs frequented by clinical librarians. the survey collected statistics in the following areas: clinical team rounding; participation in departmental activities; instruction for students, residents, or professional staff; expert searching and research involvement; identification of clinical teams served; the ratio of clinical librarians to all health sciences librarians at institution, the average number of clinical departments served; and their clinical environments and geographic locations. the authors asked survey questions regarding clinical librarians’ tasks and their perception of service impact on their constituents. results: three hundred and thirty-eight participants responded to the 14 question survey. responses to the ranked questions regarding services were coded by the authors. overall, 180 respondents answered all questions. the remaining participants answered http://www.mlanet.org/p/cm/ld/fid=938 research section spotlight hypothesis, vol. 30, no. 1, fall/winter 2018 62 some of the survey questions. 27% survey responses were from individuals at academic libraries, and 63% of responses were from hospital libraries. 52% of respondents were from the united states, while 48% were from international respondents, including canada, uk, and europe. the majority of clinical librarians serve physicians, residents, allied health professionals, and nurses. from this survey, the authors can make the following statements: clinical librarians serve multiple specialties working with a variety of clientele. in ranking their services, clinical librarians feel that they provide these top impact services to their clientele: 89% save clinicians time, 87% advance evidence-based practice, and 76% increase teaching skills of clinicians. conclusions: our analysis of current trends in clinical librarian services provides an overview of service models and the areas of focus clinical librarians have in their current positions. these results can be used to: 1) give creative ideas for services to current clinical librarians, 2) help justify a clinical librarian service to management, and 3) assist a new clinical librarian in providing creative and meaningful services to health care professionals. 2nd place authors: tallie casucci – assistant librarian, j. willard marriott library, salt lake city, utah amy locke – associate professor of family and preventive medicine; co-director, resiliency center, salt lake city, utah title: leading the way to transform burnout among health sciences librarians objectives: measure burnout among health sciences librarians and determine if a wellness game intervention improved personal and workplace wellness. methods: a burnout and satisfaction survey was administered to health sciences faculty in summer 2016 and fall 2017. a single item assessed emotional exhaustion, validated to the maslach burnout inventory. the survey was a part of a health sciences initiative to address faculty burnout and job satisfaction. each department selected wellness champions to develop programs to meet identified departmental wellness priorities with specific metrics. the library wellness champion created a team-based game after participatory interviews. players collected points for activities related to appreciation, social, mental, and physical wellness. at the game conclusion, a paper-based survey was administered to library employees. the survey included multiple choice and free-text questions. the data were analyzed with descriptive statistics and the grounded theory. game participants celebrated with an awards lunch. results: twelve library faculty completed the wellbeing and burnout survey. they scored poorly on burnout indicators in both 2016 and 2017. emotional exhaustion increased from 43% to 73%. faculty felt a great deal of stress due to their job (58% in 2016 and research section spotlight hypothesis, vol. 30, no. 1, fall/winter 2018 63 91% in 2017 compared to 45% among other health sciences employees). factors predicting burnout include sense of control over workload, job satisfaction, stress because of job and finding meaning in work. highest areas of concerns were a chaotic work environment, work flexibility, sense of control and team collegiality.30 out of the 59 employees completed the post-game survey. 70% reported the game encouraged them to socialize with colleagues. after coding qualitative data, five categories emerge: socialize (19), motivation (6), fun (5), game play (3), and recognize habits (3). participants found the wellness game to be a useful strategy in encouraging a more social culture with fun activities. conclusions: similar to previous studies on bibliometric librarians and health professionals (mainly physicians and nurses), health sciences librarians experience burnout. although the game intervention did not improve burnout or job satisfaction, it did build collegiality and recognition amongst employees. a wellness game can encourage team building, but may not sufficiently address the root causes for health sciences librarian burnout. honorable mention authors: rachel hinrichs – health sciences librarian, university library, indianapolis, indiana title: evidence-based practice skill retention and use by dietetic interns: did library instruction have an impact? objective: to determine if dietetic interns retain the evidence based practice (ebp) knowledge and skills that they were taught in three library instruction sessions in the fall by the end of the 10-month internship, and whether there is a change in their ebp clinical behaviors. methods: this non-randomized before and after study will use a validated survey to measure ebp knowledge, and ebp clinical behaviors. dietetic interns (n=16) from a large, midwestern university will be given the survey after ebp library instruction in the fall, and at the end of their internship in the spring. library instruction sessions will cover pico questions, database searching, filtered and unfiltered resources, and critical appraisal. a paired t-test will be used to compare interns’ scores in the fall and spring. results: fourteen interns (n=14) completed both surveys. on the ebp knowledge assessment, interns scored an average of 18/24 (75%) in the fall after library instruction, and 13/24 (54%) in the spring, a difference that is considered statistically significant (t(13)=7.0, p < .0001). the decrease was primarily due to missing questions on statistics and advanced boolean searching. interns retained and even improved their scores on pico, mesh, and the evidence pyramid. a slightly statistically significant change in evidence-based practice behaviors was found between the fall and spring (t(26)=2.1, research section spotlight hypothesis, vol. 30, no. 1, fall/winter 2018 64 p=.046). in particular, interns reported that they searched pubmed (t(13)=2.8, p=.016), and critically appraised articles more frequently (p(13)=2.2, p=.045). conclusion: despite the three library sessions occurring early in the internship, these results suggest that interns retain information on pico, mesh, and the evidence pyramid, but not on statistics or complex boolean searching. it is possible that these skills were not used frequently in the internship, so they did not retain the information. interns did report, however, that they more frequently performed all evidence-based behaviors including searching pubmed, accessing systematic reviews, and critically appraising articles, while decreasing their use of textbooks. while the sample size is small and not necessarily generalizable to other populations, this study suggests that dietetic interns retain some information from library ebp instruction, and do make use of the ebp resources and skills demonstrated by the librarian during their internship. future studies could examine different health professional students, and test whether spreading library sessions over the course of the year would increase retention and evidence-based behaviors further. contributed papers 1st place authors: frances delwiche – library associate professor, university of vermont, burlington, vermont title: bibliometric analysis of scholarly journal literature on the zika virus (zikv) 19522016 objectives: the primary objective of this study was to characterize the scholarly journal literature on the zika virus (zikv) on the basis of publication date, source journal, subject of source journal, country of first author, and publication type. a secondary objective was to develop a reproducible method of bibliometric analysis for this topic that would enable future updates to this study. methods: a search on the word zika or zikv was conducted in pubmed, using both mesh headings and keywords. the search was limited to articles published between 01/01/1952 and 12/31/2016. results were imported into an endnote library using a modified import filter, then exported to an excel file using a customized output style. exclusions included duplicates, off-topic articles, errata or corrections, and articles from popular magazines. data were analyzed by publication date, in five-year increments. a list of journals in which the articles were published was generated, and frequency ranked. frequency ranked lists were also obtained for the subjects of the journals and countries of research section spotlight hypothesis, vol. 30, no. 1, fall/winter 2018 65 first authors. finally, a comparison of the number of opinion pieces versus research articles was conducted for six infectious diseases, including zikv. results: approximately 93% of articles in pubmed on zikv were published in 2016, with the remaining 7% published between 1952 and 2015. though nearly 600 journals contributed articles on the topic, one third of the articles were contributed by just 19 journals. conversely, over 81% of journals contributed three articles or fewer, and 55% contributed only one article. articles were contributed by authors working in over eighty countries scattered worldwide. the publication type comparison for six infectious diseases showed that zikv had both the highest percentage of opinion pieces and the lowest percentage of research articles. conclusions: this study provided a broad overview of the state of scholarly journal publishing on zikv as of the end of 2016. the analysis revealed a remarkably rapid response to the zikv epidemic of 2015-2016 by the medical, scientific, and publishing communities, drawing contributions from scholars worldwide, working across dozens of disciplines. by describing a reproducible methodology, the study provided a mechanism for conducting periodic updates that will track the evolution and maturation of the zikv literature well into the future. 2nd place authors: julia esparza, ahip – head, user education and outreach services, medical library, louisiana state university health sciences center, shreveport, louisiana grace dodd – student researcher, lsuhsc library, shreveport, louisiana derrick murcia – student worker, neurosurgery, frierson, louisiana jessica sims, student, multicultural affairs, shreveport, louisiana gunjan kahlon – chair of hospital medicine, department of internal medicine, philadelphia, pennsylvania title: community knowledge assessment of hpv vaccination in males objectives: to ascertain community member perspectives regarding transmission of human papillomavirus infection, associated diseases and to identify barriers which prevent these groups from ensuring that males 9-26 receive the three-shot vaccine series to prevent hpv infection. using this information develop and implement health professional and patient education. methods: after institutional board approval at an academic medical center, a community survey using a convenience sample of visitors and patients to outpatient clinics was done. to establish a margin of error ±6%, 270 paper surveys were needed. the survey was 15 questions broken into a demographics section, hpv vaccination status of their children or research section spotlight hypothesis, vol. 30, no. 1, fall/winter 2018 66 self (if a young adult), their knowledge of transmission of hpv and if they would vaccinate their children or self based on the information provided in the survey. participation was sought from visitors in the outpatient internal medicine and pediatric clinics. over 800 individuals were approached to complete the survey. surveys were collected during june through july, 2016. the medical library, internal medicine, and pediatric faculty with the hospital and outpatient pharmacy spearheaded the project. the medical library faculty recruited three research apprentices to collect data. results: the survey was completed by 385 (95% confidence interval with a ±5% margin of error) participants. over 80% of the participants were female (80%) with most being african american (65%) or caucasian (29%). results show that 50% of the survey participants never had a physician discuss hpv vaccination, 39% didn’t know the diseases associated with persistent hpv infection and only 39% knew that sexual contact is the mode of transmission. “other” comments demonstrate the information needs of the participant population with statements such as “heard the vaccine doesn’t work” and “don’t trust vaccines” still a part of the community knowledge. conclusions: to date this is the largest study united states face-to face (not telephone) study of community knowledge on male hpv vaccination, knowledge of the results of persistent infection, methods of transmission, and change in vaccination acceptance. the results provided crucial insights. healthcare providers were not always having conversations about vaccination of males against hpv and that there was a gap in community knowledge on the cancers associated with persistent infection with hpv, and how hpv was transmitted. with this data a grant was obtained to create a comprehensive educational plan for healthcare providers and the community. honorable mention authors: kaitlin costello – assistant professor, rutgers university, department of library and information science, new brunswick, new jersey norris brown – research assistant, department of library and information science, new brunswick, new jersey title: coverage of core competencies for health information professionals in american library association programs objectives: what health-related courses are available to students in master’s programs accredited by the american library association? which of the core competencies necessary for work in health informatics, as defined by both the medical library association and the american medical informatics association, are covered in these classes? what curriculum gaps should instructors in these programs focus on for future course development? research section spotlight hypothesis, vol. 30, no. 1, fall/winter 2018 67 methods: a list of 60 accredited programs was obtained from the american library association directory in january 2017. during a six-month period from february 2017 through july 2017, the research team visited program websites and institutional course catalogs, and searched for course listings covering health-related topics between fall 2014 and fall 2017. course descriptions and other related data (e.g., date of last offering, number of credits, whether the class was part of a larger concentration in health informatics or similar, last available syllabi) were collected when available. the medical library association and american medical informatics association competencies were discussed among the research team and six codes, each corresponding to one of the six objectives outlined by both frameworks, were created. deductive quantitative content analysis was then applied to the available course descriptions to determine the main competency addressed in each class. results: thirty-nine of the 60 schools offer at least one class focused on health. nine programs offer a concentration in health informatics, while 18 of them offer one health class. most (32) classes focus on health information services and health reference. nineteen are related to the development of the health information professions; 16 cover leadership, management, and organizational communication; 12 address technical skills like data structures and programming; 8 are on instructional design and the design of sociotechnical systems; and 5 focus on evidence-based medicine and fundamental theoretical frameworks. programs with concentrations in health offer more comprehensive coverage of the competencies. conclusions: students in these programs will often need to supplement the health-related course offerings with classes in programming, coding, instructional design, humancomputer interaction, and theoretical frameworks in information science and health in order to gain the core competencies required for a career in health information. instructors in these programs looking to develop classes in health information should focus on developing those that address competencies that are not currently addressed fully, including the design of instructional programs and of sociotechnical systems for health; theoretical frameworks in information science, evidence-based medicine, and health behavior; and on technical skills like structuring data and coding. best hospital paper/poster authors: marilyn teolis – clinical medical librarian, library service (142d), tampa, florida priscilla stephenson – chief, library service, library service, tampa, florida mary virginia taylor – retired librarian, library, shreveport, louisiana edward poletti – chief, learning resources, learning resources, little rock, arkansas research section spotlight hypothesis, vol. 30, no. 1, fall/winter 2018 68 title: leading the way: evaluating health information professionals' satisfaction with discovery systems objectives: libraries are using discovery services to provide users with a one-box search engine to consolidate library content using a centralized index. the literature discusses use in academic libraries, but there is little data on usage in hospital libraries. we evaluated the experiences and satisfaction of hospital library staff concerning the purchase, implementation, and maintenance of a discovery service. methods: this study is based on a previous survey from 2015. an online questionnaire was sent to various health library mail groups, requesting participating librarians’ level of satisfaction regarding discovery products. participants were queried regarding which discovery products they had evaluated and whether a discovery product was purchased. respondents rated specific features, their satisfaction, and whether the tool met their expectations as a single search box resource. we inquired about the factors which influenced their purchasing decisions. individuals who purchased a discovery service were asked if the time to implement and maintain the product was what they had anticipated. participants were given the opportunity to share lessons learned during the process. finally, we invited health information professionals to share their experiences and questioned respondents regarding their perceptions of user acceptance and usage. results: over 87% of the responding librarians were satisfied with discovery services. in the current study, 92% of respondents said they planned to purchase or had purchased a discovery product, in comparison with only 41% of health information professionals who reported purchases or purchase plans in the 2015 study. many participants said discovery services not only increased utilization of the libraries’ owned resources, but also enabled patrons to find underutilized resources they might not have located without a discovery service. academic librarians said discovery services were essential to their libraries, because they integrated resources with link resolvers, and they revealed doctoral dissertations and other information in their collections which led to increased usage. conclusions: we anticipated librarians’ satisfaction would increase as discovery services evolved, but we didn’t expect the results would show a such a large, positive shift in librarians’ attitudes regarding the purchase and utilization of discovery tools. future research efforts should focus on several barriers librarians mentioned, such as enabling discovery tools for mobile devices, improving the platform to work seamlessly with different vendors products; refining search filters to find the most specific, relevant, and high-quality resources; and simplifying maintenance and updating procedures. research section spotlight hypothesis, vol. 30, no. 1, fall/winter 2018 69 best biennial jmla research paper mi m, zhang y. culturally competent library services and related factors among health sciences librarians: an exploratory study. j med libr assoc. 2017 apr;105(2):132-138. doi: 10.5195/jmla.2017.203. pmid: 28377675; pmc5370603. authors: misa mi, phd, mlis, ahip – associate professor and medical librarian, oakland university william beaumont school of medicine library rochester, mi yingting zhang, mls, ahip – research services librarian, robert wood johnson library of the health sciences, rutgers, the state university of new jersey, new brunswick, nj objective: this study investigated the current state of health sciences libraries’ provision of culturally competent services to support health professions education and patient care and examined factors associated with cultural competency in relation to library services and professional development. methods: this was a cross-sectional study. data were collected with a survey questionnaire that was distributed via surveymonkey to several health sciences librarian email discussion lists. results: out of 176 respondents, 163 reported serving clients from diverse cultural backgrounds. various services were provided to develop or support initiatives in cultural competency in health professions education and patient care. a considerable number of respondents were unsure or reported no library services to support initiatives in cultural competency, although a majority of respondents perceived the importance of providing culturally competent library services (156, 89.1%) and cultural competency for health sciences librarians (162, 93.1%). those who self-identified as nonwhites perceived culturally competent services to be more important than whites (p=0.04). those who spoke another language in addition to english had higher self-rated cultural competency (p=0.01) than those who only spoke english. conclusions: these findings contribute to our knowledge of the types of library services provided to support cultural competency initiatives and of health sciences librarians’ perceived importance in providing culturally competent library services and cultural competency for health sciences librarians. the results suggest implications for health sciences libraries in fostering professional development in cultural competency and in providing culturally competent services to increase library use by people from a wide range of cultures and backgrounds. -- research hypothesis vol. 32 no.1 fall/winter 2020 factors affecting clinical referrals to the medical library liz kellermeyer, msls biomedical research librarian library & knowledge services national jewish health denver, co matthew j. strand, phd professor division head, biostatistics and bioinformatics national jewish health denver, co abstract objective: to determine why and how often clinical care teams refer patients to the medical library and to determine facilitators and barriers to referral. methods: a 2018 survey of clinical care teams at a research hospital measured awareness of library services available to patients, facilitators and barriers to referral, and likelihood of future referral. spearman correlations were used to determine the strength of relationships between familiarity with the services and how often respondents referred those services. referral rate distributions were compared between job type groups. results: overall, self-reported referral rates were low. there was a marginally significant relationship between referral rate and job type (p=0.01), with providers having lower referral rates. there was a positive correlation between familiarity with library services and service referral frequency (rs=0.78 for combined data) and between current referral rates and likelihood of future referral (rs=0.43 for combined data, p<0.0001 for both). among respondents who had never referred patients, the top reasons were lack of awareness of library services and uncertainty about how to make referrals. conclusions: the results suggest that lack of knowledge, rather than lack of interest and support, results in lower clinic referrals to the library. providers who are aware of the library services refer patients. similarly, providers who currently refer are more likely to continue making referrals in the future. the qualitative responses show agreement, linking the lack of referrals to marketing and procedural insufficiencies, which indicates potential for increasing referrals by addressing these deficits. a streamlined patient referral system from clinic to library could be beneficial. research hypothesis vol. 32 no.1 fall/winter 2020 introduction library & knowledge services in the gerald tucker memorial medical library supports the clinical, research, and educational programs of national jewish health (njh), a nationally recognized respiratory hospital in denver, colorado. one goal of the medical library’s mission is to provide health information resources for patients and their families. this goal reflects njh’s institutional mission of providing individualized care to patients and improving patient health outcomes by appropriately centering health care decisions around individual patient preferences and needs. njh provides opportunities for promoting patient engagement and health literacy through several avenues, including providing online access to health records, publishing consumer health information on conditions njh treats, and teaching courses on self-management of chronic disease. the medical library is a unique resource for patients, offering librarian-mediated searches on specific health topics, guidance on finding authoritative health information, and help finding provider-recommended literature. while patients may find out about library services through a variety of sources, of particular interest are direct referrals from patients’ clinical care teams. the clinical care team is defined here as physicians and clinical staff who have direct contact with patients and have opportunities to discuss patients’ care with them. the clinical care teams are in a position of authority, and patients are likely to value their directives, which can positively impact patients’ willingness to seek health information outside the office setting [1]. physicians or other members of the care team may not always refer patients to the medical library to further investigate inquiries or curiosities, and the reasons for this are unclear. this study surveyed clinical care team members at njh on their knowledge and referral frequency of medical library services oriented towards patients. literature review evidence shows that patients who are engaged in their healthcare experience more satisfaction and have more positive health outcomes than those who are not [2]. one of the elements of this engagement is for patients to be adept at gathering information on their own and incorporating it into their lives. higher health literacy rates have been associated with improved self-management of diseases and better medication compliance [3]. hospital libraries that provide consumer health information and services are well-positioned to improve the health information literacy skills of patients [3]. as a service that provides both access to consumer health information and librarian-mediated searches, the medical library can attend to patients’ individualized information needs. librarians can supply specific physician-recommended articles, books, or handouts to a patient. additionally, librarians have unique skills to lead patients through a reference interview, which often uncovers additional information needs. the medical library is not simply a dispensary of curated materials but a service research hypothesis vol. 32 no.1 fall/winter 2020 that can instruct and empower patients to engage in efficient and effective informationseeking behaviors. however, connecting patients to library services and resources is a challenge. in 2002, the national library of medicine introduced the information prescription (information rx) program as a possible facilitating practice. under the information rx model, the physician prescribes specific reading material to patients, who are then expected to obtain and read the recommended material, sometimes with librarian mediation. these programs had a defined curriculum where physicians were trained and then given specific tools (e.g., information prescription pads) to be used when giving patients resource recommendations [4]. among the positive outcomes associated with several pilot programs based on this model was the fact that physicians found value in library services and resources for their patients and thought they enhanced patient education and provider-patient communication [1, 3]. some programs were able to successfully create a referral pathway that included the library [5, 6]. while the outcomes of the programs showed that clinical care teams considered medical library resources valuable for patients, attempts to standardize a referral system from clinic to library have been complicated. for these early pilot programs, barriers to success included issues around physician adherence, patient participation, and the overall difficulty of sustaining collaborative efforts between several departments [7, 8]. some newer iterations of the information rx model include the integration of the electronic health record (ehr) system to replace the outdated paper prescription pad [9, 10]. in these models, the clinical care team puts in specialized requests for information for their patients, which librarians fulfill. these programs cited lengthy technical hurdles to overcome, though both reported on ultimately positive outcomes, where bonds between departments were strengthened and the quality and volume of valuable health information to patients was increased [9, 10]. the literature does not discuss libraries and hospitals that lack a formal program or designated referral pathways for patients from physician to library. in such situations, it is unclear whether clinical care teams know about available library services or recommend them to their patients. the aim of this study was to understand why clinical care teams do or do not refer their patients to the medical library, the frequency of referral, and what barriers and facilitators are present in the process. methods a census of njh clinical care teams was taken from october to december 2018. a survey was designed and administered online using redcap and piloted by a small group of physicians and nurses. the final survey was sent to 482 employees identified as providers or clinical staff who have direct contact with patients and opportunities to discuss patients’ care with them. an incentive was offered for each completed survey of a $5 donation made to the morgridge academy, an on-campus k-8 school at njh research hypothesis vol. 32 no.1 fall/winter 2020 serving children with chronic illnesses. from the population, 95 responses were collected, representing a 19.7% response rate. the survey featured multi-choice questions and likert-type scales to measure awareness of library services available to patients, the facilitators and barriers to referral, and the likelihood of future referral. the focus was on five key services that the library provides for patients: patient portal account set up and questions, access to health information related to a diagnosis, librarian consultations or expert searching, physical library resources unrelated to health information (e.g., computers, printers), and archives or legacy information related to the hospital (e.g., annual reports). numerical values were assigned to the frequency answers, ranging from 1 (“never”) to 6 (“almost every day”), and the mean scores were calculated for each service. demographic variables included gender, age, professional role, and years employed at the hospital. spearman correlations were used to determine the strength of relationships between familiarity with the services and how often respondents referred those services (rs). referral rate distributions were compared between job type groups using the kruskal-wallis test. this study was determined exempt by the irb at njh (hs-3203). findings over half of the respondents (53%) identified themselves as providers (physicians, physician’s assistants, or nurse practitioners), with the next largest group being licensed nursing staff (19%). medical assistants, social workers, and an “other” category composed the remaining group (14%). fourteen percent did not answer this demographic question. overall, self-reported referral rates were low, with 57% of respondents saying that they have never referred a patient to the medical library, and only 14% saying they refer patients at least once a month. the highest referral rates were for “quick” services—those that don’t require librarian expertise but are easy to remember and offer fast help to a patient. other more specialized services, such as archives research or librarian-mediated expert searches, had the lowest referral scores. across all services, the top two reasons clinical care teams referred patients to the library were that they felt the service adds value to the patient's visit and that the service complements or enhances the patient's access to health information. looking at the strength of relationships between familiarity with the services and service referral frequency, there was a positive correlation across all services (rs=0.78 for combined data, p<0.0001) (figure 1). research hypothesis vol. 32 no.1 fall/winter 2020 figure 1. relationship of clinical care team knowledge of library services and referral rate. (familiarity scale: 1=not familiar, 2=slightly familiar, 3=somewhat familiar, 4=moderately familiar, 5=very familiar; referral scale: 1=never, 2=less often than once a year, 3=a few times a year, 4=a few times a month, 5=at least once a week, 6=almost every day) the highest correlations occurred for librarian consults, health information, and patient portal help (spearman correlations=0.75 or 0.76, p<0.0001), and slightly lower correlations for archives and physical library space (spearman correlations=0.665 and 0.60, p<0.0001). a positive correlation existed between current referral rates and likelihood of future referral (rs=0.43 for combined data, p<.0001). the more familiar respondents were with these services, the more often they were likely to refer them to patients. additionally, those who referred in the past were more likely to say they would refer patients again. there was a marginally significant relationship between referral rate and job type (p=0.01), with providers having lower referral rates (figure 2). research hypothesis vol. 32 no.1 fall/winter 2020 figure 2. relationship between referral rate and job type. (referral scale: 1=never, 2=less often than once a year, 3=a few times a year, 4=a few times a month, 5=at least once a week, 6=almost every day) among respondents who had never referred patients, the top reasons were “not aware that library offered this service” (48%) and “don’t know how to make a referral to library” (20%); among the least selected reasons were “clinic wants to control information to patient” (2%) and “concerned about quality of information provided by library” (0%). top referral reasons were “complements or enhances the patient’s access to health information” (28%) and “adds value to the patient’s visit” (27%). there was no significant correlation found between how often a person referred and how long they have been working at the hospital. discussion conclusions the results suggest that lack of knowledge, rather than lack of interest and support, results in lower clinic referrals to the library. the correlation between knowledge of services and likelihood of referral demonstrates that providers who are aware of library services do refer patients. similarly, those currently referring are likely to make future referrals, suggesting these providers value library services. when asked why they refer patients, the care teams selected reasons that reflect their belief that the library has something of value to give to the patients. research hypothesis vol. 32 no.1 fall/winter 2020 the qualitative responses show agreement, linking the lack of referrals to marketing and procedural insufficiencies (rather than distrust or dislike of the services), which indicates potential for increasing referrals by addressing these deficits. respondents who had referred patients in the past demonstrated an understanding of what the library offers, citing librarian services and quiet space. one respondent stated, “i feel [visiting the library] is the best way for patients to do their research rather than online by themselves,” which speaks to the value placed on the specialty of librarian services and on the willingness of clinic staff to look at the library as a collaborator on patient health. many of the comments from those who had not referred patients were variations of the fact that they were not aware of the services or do not think about them over the course of a visit. some were not sure what differentiated library services from patient education already provided at a clinic visit. one respondent wrote, “nursing provides [institutionally branded medical information sheets] and the team often prints and gives an article to the family if they have something they want to share.” while there may be overlap between the kind of educational material patients receive from their care team and those a librarian may help them find, the difference between these two sources gets at the heart of encouraging health literacy. in the clinic setting, information is provided to the patient based on what the care team member wants the patient or family to know. in the library setting, the question always starts with what the patient wants to know. this idea of being able to articulate inquiries on behalf of one’s own health and to take steps to obtain relevant resources are key to empowering patients to take an active role in their health. marketing inherent in the idea of marketing the library is the idea of communicating not just services but the library as a collaborative partner in improving health outcomes for patients. for this to be done, the clinic must understand what the library provides as distinct from what they provide, and providers must appreciate how the interaction may add value to the patient’s visit. because there is some evidence that clinical teams may be distrustful of librarians to provide accurate health information, the survey asked respondents about this [4]. however, in this survey 0% of the respondents selected this as a reason why they hadn’t referred patients, so it is hopeful that the positive reputation of the medical library is good groundwork for growth. interestingly, no significant correlation was found between how often a person referred and how long they have been working at the hospital. while it was initially hypothesized that people who had worked at the hospital longer would retain knowledge about library services, this result suggests that marketing the library has done in the past has not necessarily been retained. more persistent communication with the clinic might be necessary in order to see sustained results. there may be some practical research hypothesis vol. 32 no.1 fall/winter 2020 steps the library can take to bridge the gap between interest and awareness in services. it seems evident that focusing only on one marketing avenue is not sufficient and that spreading information about library services using varied tactics would increase the chance that care teams would be aware of the services. as a starting point, printed material could be distributed at new employee orientation and also placed in key work areas, such as clinic workstations. librarians can also make a point to get on the agendas at key departmental meetings periodically, where they can remind faculty and staff about patient services available at the library. it can be advantageous to look for opportunities for integrated messaging across services. for instance, librarians are now invited to discussions about changes in patient portal processes, and in turn they are included in messaging from the information services department about where patients can seek portal setup help. referral system for those who answered that they had never referred a patient to the medical library before, the second most frequent response (20%) was that they were unclear about how to make such a referral. there is currently no formal referral system in place at the hospital; in order for a patient to get to the medical library via the clinical care team, the recommendation must come organically, based on the team member’s knowledge of the library’s services and a belief that the patient can be helped through a visit. if there is to be a collaboration across departments, a system is needed to support the workflows and tendencies already established in each area. outdated systems, such as the information rx program, rely on tools like prescription pads that may not even be in frequent use anymore. how can library services be streamlined into clinic workflow? one possibility is integrating medical library services into an ehr system, with the ability for providers to order librarian consults through the platform. some such systems are already in place, with providers able to request a library collaboration by entering an order in the ehr. librarians access these orders along with the medical record and can provide personalized information packets based on specific needs [9, 10]. an integrated system also presents the potential for charting and documenting library intervention, which could lead to the ability for larger data-driven studies looking at correlations between librarian interactions and patient outcomes. libraries that have successfully set up such a system note the importance of securing a champion on the physician side of things, who can help guide the project and assure its goals and outcomes are aligned with those of the clinical care teams [7, 10]. limitations and future research because this research was conducted only within one hospital setting, it is unknown whether these trends would be found widely among other medical libraries research hypothesis vol. 32 no.1 fall/winter 2020 that serve a patient population. the survey instrument developed could be used elsewhere, and it may be beneficial to replicate the research in other settings to see if findings match those of this study. one limitation of this research was the exclusive focus on hospital staff rather than patients themselves. the results reveal only the intentions and behaviors of the clinical staff and not the patients’ responses to referrals. future research could look at patient populations using medical library services and examine their reasons for seeking out the resources, and trace whether their visits are motivated by clinical care team referrals. a combination of surveys would paint a more complete picture of the referral pipeline, measuring the difference between referral rate and compliance. those medical libraries that have referral systems incorporated into ehrs may be an obvious place to pick up the ideas in this research while working with electronic data. this research has shown there is interest in clinical care teams collaborating with the medical libraries to enhance patient care. the main barriers to referrals seem to be lack of knowledge of library services and an unclear referral pathway, which provides libraries and hospitals specific areas to focus on for improving referral rates. acknowledgements thank you to the mla research training institute for its training, support, and encouragement to carry out this research. this project was made possible in part by the institute of museum and library services (re-95-17-0025-17). author contributions l.k. designed and carried out the research, contributed to the analysis of results, and wrote the manuscript; m.s. processed the data, performed analysis, aided in interpreting the results, and contributed to writing the manuscript. references 1. siegel er, logan ra, harnsberger rl, cravedi k, krause ja, lyon b, hajarian k, uhl j, ruffin a, and lindberg d.a.b. information rx: evaluation of a new informatics tool for physicians, patients, and libraries. inf serv use. 2006;26(1):1–10. doi: https://dx.doi.org/10.3233/isu-2006-26101 2. prey je, woollen j, wilcox l, sackeim ad, hripcsak g, bakken s, et al. patient engagement in the inpatient setting: a systematic review. j am med inform assoc. 2014;21(4):742–50. doi: https://dx.doi.org/10.1136/amiajnl-2013-002141 3. shipman jp, kurtz-rossi s, funk cj. the health information literacy research project. j med libr assoc. 2009;97(4):293–301. doi: http://dx.doi.org/10.3163/15365050.97.4.014 https://dx.doi.org/10.3233/isu-2006-26101 https://dx.doi.org/10.1136/amiajnl-2013-002141 http://dx.doi.org/10.3163/1536-5050.97.4.014 http://dx.doi.org/10.3163/1536-5050.97.4.014 research hypothesis vol. 32 no.1 fall/winter 2020 4. mcknight m. information prescriptions, 1930–2013: an international history and comprehensive review. j med libr assoc. 2014;102(4):271–80. doi: https://dx.doi.org/10.3163/1536-5050.102.4.008 5. calabretta n, cavanaugh sk. education for inpatients: working with nurses through the clinical information system. med ref serv q. 2004;23(2):73–9. doi: https://dx.doi.org/10.1300/j115v23n02_07 6. williams md, gish kw, giuse nb, sathe na, carrell dl. the patient informatics consult service (pics): an approach for a patient-centered service. bull med libr assoc. 2001;89(2):185–93. 7. huber jt, shapiroii rm, gillaspy ml. top down versus bottom up: the social construction of the health literacy movement. the library quarterly. 2012;82(4):429–51. doi: https://dx.doi.org/10.1086/667438 8. whitney w, keselman a, humphreys b. libraries and librarians: key partners for progress in health literacy research and practice. stud health technol inform. 2017;240:415–32. doi: https://dx.doi.org/10.3233/978-1-61499-790-0-415 9. hansen j. librarian consults through epic: new opportunities for collaboration and education. med ref serv q. 2019;38(3):293–9. doi: https://dx.doi.org/10.1080/02763869.2019.1623610 10. witman l. “information prescription” orders in the electronic medical record at uva: bringing our consumer health information service to the bedside and planning for outcomes research. j hosp libr. 2017;17(2):137–46. doi: https://dx.doi.org/10.1080/15323269.2017.1291049 https://dx.doi.org/10.3163/1536-5050.102.4.008 https://dx.doi.org/10.1300/j115v23n02_07 https://dx.doi.org/10.1086/667438 https://dx.doi.org/10.3233/978-1-61499-790-0-415 https://dx.doi.org/10.1080/02763869.2019.1623610 https://dx.doi.org/10.1080/15323269.2017.1291049 research section spotlight hypothesis, vol. 30, no. 1, fall/winter 2018 58 reflections on the 2018 mla annual meeting elizabeth kellermeyer, msls biomedical research librarian national jewish health denver, co kellermeyerl@njhealth.org i was honored to receive the travel award from the research section to help fund my attendance at the 2018 mla annual meeting in atlanta, georgia. while a lot of information and connections are available online, attending this conference in person is a reminder of the value in community. there are many intangible benefits to being able to meet new people and connect over shared values and exciting ideas. i was very strongly affected by the themes of diversity, inclusion, and social justice that emerged from the conference. atlanta, the birthplace of martin luther king, jr. and home of the center for civil and human rights, was a fitting setting for these ideas that ran as an undercurrent through many sessions, presentations, and discussions at the conference. while a general commitment to diversity is nothing groundbreaking, there was more urgency to the messaging this year—less of a friendly reminder to be welcoming and more of a call to action. in particular, the theme was prominent in this year’s janet doe lecture, presented by elaine r. martin, “social justice and the medical librarian.” in her riveting talk, elaine laid out an argument for medical librarianship developing a new professional orientation, one focused on recognizing societal injustices and their effect on access to information and healthcare and to be inspired to advocate on behalf of the patrons we serve. beverly murphy gave her first address as the incoming president. “in the 120th year of the association i am proud and honored to be the first african american president of the medical library association,” she stated. “i want to thank you….for giving me this honor and opportunity and in doing so, taking a positive stance, considering the world that we are living in today.” she emphasized that she wanted her time as president to be an extended conversation with all members, drawing strength from the collective expertise and knowledge of our members. both speakers extolled the importance of education and in being thoughtful and purposeful about reaching out to the next generation of medical librarians to ensure we build a community that is as diverse as the ones we serve. their messages are powerful and left me both humbled and inspired. moving forward, matters of diversity and inclusion can be big motivators for research. what challenges and disparities are we witness to and how can we bring these topics to the forefront of discussion? research that brings them research section spotlight hypothesis, vol. 30, no. 1, fall/winter 2018 59 into focus or that uses them as a backdrop against which we can understand other issues will be crucial in defining our role in librarianship in the coming years. the conference was an excellent place to collect information from disparate places about a particular topic. for instance, i have been learning more about the systematic review process and i was able to enhance my knowledge by taking a ce course on identifying clinical trials for systematic reviews, attending a sponsored lunch and learn about how the joanna briggs institute conducts systematic reviews, and comparing features of systematic review software by talking directly to vendors and seeing product demonstrations. while presenting my poster, “covidence vs. rayyan: a comparison of systematic review tools,” i had the opportunity to talk directly with other librarians with similar interests and questions, generating a lot of discussion and opportunities for continued collaboration. it was also a chance to meet with groups and people that i otherwise primarily know through online interactions. this included new committees i will serve on, an introduction to the rising stars program, and section meetings. i collaborated on my first research project this past year, “measuring impostor phenomenon among health sciences librarians,” led by jill barr-walker. our group presented initial findings at a program session and it was great to meet with my co-authors in person and to showcase our work to colleagues. one of the conference highlights for me was getting to hear about the research undertaken by medical librarians across the country. i am planning on doing more research in the upcoming year, and was accepted into the mla research and training institute to further that goal. the conference offered a first introduction to this year’s cohort during a session by project director susan lessick, who provided an overview and answered questions. of course, some of the most prominent discussions this year focused on the proposed mla restructuring. the communities strategic goal task force presented their proposed model of mla architecture, including the dissolution of sections and sigs as they currently exist. they are reimagining interests being organized around what are tentatively called “affinity groups” and “domains of practice” and discontinuing section dues. i was able to attend several forums—a general one, and two held at section meetings—where representatives from the task force fielded questions and engaged in discussion with members. it was very beneficial to attend these in-person, as the general atmosphere was starkly different between groups, and conversations about how the restructuring would work continued among members throughout the conference. this was an excellent opportunity for me to be able to hear diverse perspectives and gain an understanding about some of the history of mla and how the impending changes will affect different groups. research section spotlight hypothesis, vol. 30, no. 1, fall/winter 2018 60 i am grateful for the opportunity to attend this year’s annual conference; what i learned will help shape my research goals and inform my future interactions with mla and my colleagues. -- national library of medicine (us). digital collections [internet]. bethesda (md): three vignettes demonstrating light projection apparatus. in oculus artificialis teledioptricus, 1685-86. available from: http://resource.nlm.nih.gov/101435223 http://resource.nlm.nih.gov/101435223 art show paints a collaborative picture stark, paris, rogan, finnegan dopkins, and etter hypothesis, vol. 29, no. 2, fall/winter 2017 21 art show paints a collaborative picture: increasing engagement in a pharmacy and health sciences library rachel stark, ms, ahip health sciences librarian, senior assistant librarian sacramento state university sacramento, ca stark@sacstate.edu mickel paris, mlis health sciences librarian, assistant professor rite aid information commons university of the pacific stockton, ca mparis@pacific.edu ed rogan, pharmd assistant professor, thomas j . long school of pharmacy and health sciences university of the pacific stockton, ca erogan@pacific.edu mary-kate finnegan dopkins, mlis library services specialist rite aid information commons university of the pacific stockton, ca kdopkins@pacific.edu cassie etter, mts library services specialist rite aid information commons university of the pacific stockton, ca cetter@pacific.edu art show paints a collaborative picture stark, paris, rogan, finnegan dopkins, and etter hypothesis, vol. 29, no. 2, fall/winter 2017 22 abstract objective: to increase student, staff, and faculty engagement with the health sciences library. methods: faculty, staff, and students were encouraged to create pharmacy-themed artworks and submit them to the library for entry into an art show. a two-hour art show and gala were planned for library patrons to view art pieces submitted by faculty, staff, and students. participation data was collected during the event and compared to previous programming held at the health sciences library. results: the pharmart program determined a metric of demonstrable success would show as an uptick in participation; both physical and digital, when compared against previous library events. physical participation was measured through program attendance and art piece submission. participating demographics were represented by all target populations of faculty, staff, and students with a total increase of 52 participants (347% increase) over the previous event. engagement through social media showed a considerable increase in clicks (2224%), shares (1200%), and overall reach (24%) within the school community. conclusions: incorporating student, staff, and faculty art works into a library-hosted event increased participation in a library program compared to previous library programming. attendance and social media engagement increased considerably. faculty, librarians, and students were brought together to share a common creative interest and to promote a casual atmosphere in which cross-curricular ideas could be discussed among attendees. introduction the rite aid information commons (raic) is the health sciences library serving the university of the pacific thomas j. long school of pharmacy and health sciences. the school supports the academic success of students studying pharmacy, audiology, physical therapy, and speech language pathology. library events at the raic, such as the better hearing & speech month (bhsm) kickoff party, have traditionally had poor attendance, even with social media marketing and incentives such as raffle prizes. due to the lack of engagement, many event goals were unmet (for example, providing important information regarding diseases and other healthrelated subjects), prompting a reevaluation of library programming at the raic. research increasingly supports library programs that provide students the opportunity to actively participate in creative endeavors at the intersection of art and science in order to view different facets of established academic and clinical concepts [1, 2]. in collaboration with art show paints a collaborative picture stark, paris, rogan, finnegan dopkins, and etter hypothesis, vol. 29, no. 2, fall/winter 2017 23 pharmacy faculty, the raic hosted an art event and subsequent pop-up art gallery with the goals of 1) increasing engagement and collaboration with faculty and students and 2) improving patron interest in the library through creative and participatory library services. the event, entitled pharmart, was designed to encourage collaborative creative activity and stimulate discussion among artists and participants. since the library is a neutral space, the raic provided a welcoming environment for artistic expression, which is recommended for the exchange of cross-curricular ideas [3]. the exhibit also provided pharmacy students and faculty an opportunity to interact socially with students from other disciplines. participation was broadly solicited from faculty, staff, and students within the school of pharmacy and health sciences, and works from every physical art medium were welcomed. the theme of the art show was “knowing your pharmacist and the local history of pharmacy.” objective previous programming at the raic had low participation, with just 15 attendees at the better hearing & speech month (bhsm) kickoff event. the primary goal of pharmart was to positively increase student and faculty engagement in person and online at the raic by hosting an active, participatory library program. not only did pharmart improve event turnout to 67 attendees from bhsm’s 15 (347%), it also increased online engagement by 24 (1200%) facebook shares and reactions. the secondary goal was to explore if students, faculty, and staff would be interested in a library-hosted art program. this goal was facilitated by encouraging creative participation from nine faculty, staff, and students who submitted eleven works of art, as well as providing attendees with the opportunity to vote in the art show. literature review libraries are discovering methods to reshape space through art and aesthetics in order to connect with the community and draw in new users. a review of the literature reveals that art shows and galleries are created with much success not only in campus libraries, but in science and health sciences libraries as well. beals’ 2007 article on student art in library exhibitions details not only the process of setting up an exhibition program, but also the rationale [3]. in this often-cited article, beals explores the process of designating the exhibition space, creating submission procedures, and deploying publicity, among other important logistical considerations, for the university of tennessee art show paints a collaborative picture stark, paris, rogan, finnegan dopkins, and etter hypothesis, vol. 29, no. 2, fall/winter 2017 24 libraries. the process includes the creation of an inviting and visually appealing space, as well as providing an educational purpose that supports the library’s mission. similarly, the university of colorado invited the general community to explore non-traditional uses of library space by establishing an art gallery that showcases works by local and visiting artists [4]. science and health sciences libraries are also realizing the value of art in science learning. the albert r. mann library at cornell university, which serves agriculture, life science and human ecology programs, has created an art gallery that attempts to create interdisciplinary knowledge and communicate science through visual imagery, such as sculpture, drawing, photography, and illustration [5]. for the marston science library at university of florida, an art contest called “elegance of science” encourages faculty, students, and staff to compete for awards and recognition, while also providing campus art connoisseurs the opportunity to reflect and contemplate the discoveries of life [6]. the national library of medicine (nlm) has contributed to interdisciplinary learning innovation by creating the nlm traveling exhibition program that health science libraries can adopt to engage with users. previous high-interest themes include health science topics found in frankenstein and harry potter novels. outcomes of the nlm’s traveling galleries reveal that they positively change users’ perceptions of the library and bring more people through the doors [7]. additionally, art shows and galleries are increasingly proving to be both ice breakers and sandboxes for future collaboration between faculty and librarians. catalano et al. point out that disparate areas of the school, such as the health sciences and arts programs, all share something in common: the library [8]. college and university libraries that have indicated an explicit desire to increase outcomes of faculty-librarian connections and collaborations through art include: • university of akron science & technology library, which created a gallery of artistic journal covers of published faculty research [9] • fort lewis college reed library, which emphasized student success outcomes within the college’s mission [10] • himmelfarb health sciences library at george washington university medical center, where the library’s art show has flourished since 1979 and has become a popular annual tradition embedded within the culture of the campus [11] throughout the literature, the common denominator in libraries developing an art show or establishing an art gallery is to further the library’s mission, supported by the ideal of the library art show paints a collaborative picture stark, paris, rogan, finnegan dopkins, and etter hypothesis, vol. 29, no. 2, fall/winter 2017 25 as a service organization with eight functions described by frieda o. weise in her 2003 janet doe lecture as: a physical symbol of knowledge, an intellectual commons, a haven for research, a place for collaboration, an access point for information, a forum to teach, a functional and pleasant workplace, and an attractive gateway to the campus [12]. art shows can embody these functions within the library by combining art with science to provide novel learning experiences for students between disciplines and encourage collaboration between faculty and librarians, where beals states: “conversation is stimulated, participation is encouraged, and new experiences are gained.” [3]. methods in an effort to engage students, faculty, and staff in library programing, library staff working in conjunction with a school of pharmacy faculty member, held a pharmacy-themed art show designed to bring together the various health sciences communities served by the raic. the name pharmart was chosen for the pilot event to support american pharmacists month, with the majority of the marketing targeted to school of pharmacy faculty, staff, and students. submission requirements also included a local history of pharmacy theme, to encourage artists and viewers to connect their learning with the event. library staff released requests for art submissions on the health sciences campus during the summer of 2015 through social media accounts and print advertising displayed in campus libraries and health sciences buildings. the pharmart gala was scheduled for october 13th from noon to 1pm. the submission process asked that prospective artists submit information about their works through a google form, which was also accessible through a qr code created at kaywa.com. artists were informed through email communications from library staff as to when and where to submit their artwork during the event. as the event date approached, internal reminders about the event were emailed to faculty and staff. pharmacy faculty were asked to promote the event to their students. student library staff were trained to answer common questions about submissions, contest details, and the event, as well as encourage participation and attendance to anyone inquiring. the library decided to organize the event in the style of a gala in the hopes of drawing additional interest. to meet this end, the gala event organizers obtained a $500 budget from the university library to purchase food and drinks, prizes, and necessary exhibit staging for artwork and decorations. knowing food would be a popular inducement, funds were set aside for that purpose in addition to the library purchasing door and participation prizes for the top three art pieces to be awarded in the days following the pharmart art show. the prizes art show paints a collaborative picture stark, paris, rogan, finnegan dopkins, and etter hypothesis, vol. 29, no. 2, fall/winter 2017 26 selected were a one-terabyte external hard drive, a $25.00 amazon gift card, a university tote bag, and a $15.00 itunes card. these were allocated as first, second, third, and door prizes respectively. prizes were awarded through the use of raffle tickets given to attendees as they entered. mention of raffle prizes and free food played a part in the in-house and online marketing campaigns. during the event, the library remained open, however users were informed that the event might be disruptive. art pieces were placed on display with a card stating title, artist, and art medium on the day of the event and, during the event, artists were encouraged to stand near their art and discuss it with viewers. raffle tickets were given to pharmart exhibit attendees for a door prize and also used as voting ballots for patrons’ favorite art pieces. these tickets were tallied after the event day and prizes were awarded and ribbons placed. all art pieces were assigned numbers which were displayed prominently on the title card, which was used to record the vote selection. at the event, attendees were encouraged to mingle while enjoying the different art displays and then place a vote for the piece they liked best by dropping their ticket, with contact information and vote recorded on the back, in a secured box. after the show, votes were tallied and winners determined based on number of votes. the art remained on display for several weeks following the exhibition. the library determined the number of students, faculty, and staff who attended the event by conducting patron counts at the beginning, middle, and end of the event, and by asking all attendees to put their basic information on a ticket used to vote for the best art work. in order to exclude patrons who were in the library for more traditional use, such as studying, attendance counts for pharmart was checked against the number of tickets collected, removing library users who did not participate in the pharmart event from the final attendance count. results to determine if the pharmart program theme and marketing efforts netted a broader audience and increased online engagement, the library compared metrics collected from a prior event. first, the research team assessed attendance and social media ‘hits’ from pharmart to the same information recorded during an earlier event held at the library, the better hearing and speech month (bhsm) kickoff party. compared to data collected from the bhsm event in may of the same year, pharmart successfully increased the target demographic attendance in the library, and increased the library’s social media presence. art show paints a collaborative picture stark, paris, rogan, finnegan dopkins, and etter hypothesis, vol. 29, no. 2, fall/winter 2017 27 attendance at pharmart recorded an increase of 347% above bhsm. a total of 67 individuals attended the art show, improving the attendance numbers from the previous bhsm program, which drew only 15 attendees (table 1). over the course of the pharmart gala, 67 raffle tickets were handed out and 60 votes were received, indicating a willingness of attendees to participate and contribute to the event. all metrics for social media also indicated a marked increase (table 1). facebook metrics showed a gain in posts (+393.33%), shares/reactions (+1200%), cumulative reach (+24.07%), and click-throughs (+2223.53%). the library’s pharmart blog post hits experienced a gain of 23.08% over bhsm. better hearing & speech month may 2015 pharmart october 2015 total attendance 15 67 student 11 54 faculty 3 7 staff 1 6 blog post hits 26 32 facebook page posts 6 29 facebook shares/reactions 2 26 facebook cumulative reach 108 134 facebook click throughs 17 395 table 1. physical attendance and social media engagement additionally, it was demonstrated anecdotally that the target community was interested in participating in an art show. student staff workers regularly answered queries about the event in the month leading up to the gala and, by the date of the gala, the library had received a rich cross section of representation of art works in a variety of two-dimensional and threedimensional media. all 11 art pieces that artists submitted for the art show were approved and included in the event. the ratio of university patron types participating was diverse, with 4 students, 3 faculty and 2 staff comprising the cohort of 9 artists. furthermore, art was submitted not only from the school of pharmacy, but also from other areas of the university: of the 11 art submissions, 6 came from the school of pharmacy, 3 from the university library, 1 from computer engineering, and 1 from the business program. art show paints a collaborative picture stark, paris, rogan, finnegan dopkins, and etter hypothesis, vol. 29, no. 2, fall/winter 2017 28 discussion compared to previous library programming, the engagement metrics reveal that pharmart generated interest and participation amongst the faculty, staff, and students in the thomas j. long school of pharmacy and health sciences. the use of a common creative interest allowed these normally separate groups to engage in a more casual atmosphere that helped to remove some of the barriers experienced within a university setting. participants included members of the different departments and disciplines of the school. verbal feedback from participants provided during the event was positive and showed that the library can be a place for engagement as well as learning. the primary goal of increasing student and faculty engagement for an active, participatory library program was achieved. not only did raic notice improvement for in-person attendance, but online engagement and interaction increased as well. an art show, in particular, may help online and social media marketing of the library since it is a visual, aesthetic medium that complements online spaces. the secondary goal of exploring interest in a library-hosted art program revealed that interest is very good, as the ratio of art participants included members from the three targeted patron groups: students, faculty and staff. furthermore, the majority of art gala attendees participated in the art show itself through voting for their favorite art pieces. previous events at the library, such as bhsm, did not include active participation and were unsuccessful in generating engagement and participation. by finding a common creative interest and developing a program that brought all together, the library space transformed from the everyday usage as a study and snacking area to a collaborative engagement area, where students, faculty, and staff mingled, interacted and participated in a pop-up art gallery. limitations a major limitation is that the research primarily calculates the quantitative increase in patron engagement of pharmart within the library and online in social media but does not assess qualitative data. samples of qualitative data that would be relevant and helpful to discuss include whether patrons enjoyed the art show, found it useful, or learned something new. as only one pharmart event has been held, future shows could include qualitative assessments or interviews, and this data could be combined with engagement metrics to build a more complete picture on the quality of engagement with the attendees. art show paints a collaborative picture stark, paris, rogan, finnegan dopkins, and etter hypothesis, vol. 29, no. 2, fall/winter 2017 29 another limitation in the study concerns other events or factors that could influence attendance. the timing of events, for example, could be construed as potentially helping or hurting attendance. in this study, attendance at bhsm was held in the time leading up to midterm exams and also just before the release of speech therapy students for their summer break, whereas pharmart was held following mid-terms: as such, the difference could be construed to affect attendance. time of year could also have an effect, as students in spring semester may be more interested in outside activities, whereas fall semester weather may be more conducive to indoor activities and events. the seasonal difference could also affect online engagement if patrons are more interested in outdoor activities in the spring rather than the fall. future directions the success of the first pharmart show has led the raic to begin developing future programs involving art and examining how this active participation can help students develop skills for stress relief and confidence building. the use of art in pharmacy curricula is very limited but has shown to be helpful in other health science disciplines. multiple studies on observing the effects of art on student development and pedagogies have been conducted primarily in medical school programs, and can provide outcomes that encourage discussion and understanding as it relates to patient care [13]. for example, in one study, a cross-disciplinary approach for developing medical insight incorporated the use of visual arts with medical themes: following the evaluation of artwork, students self-reported the positive development of observation, decision-making, reflection, and confidence-building skills [14]. in another study, medical students who took part in an arts-based learning course noted enhancement of awareness in self, improvement in self-confidence and self-care, increased preparation for stress, and development of communication and team-building skills [15]. in a study where students created and presented original artwork based on interactions with individuals living with chronic diseases, creative art was found to impact personal growth, self-discovery, sense of community and collaboration skills [16]. medical education studies such as these can help inform the development of research in pharmacy education that incorporate interdisciplinary activities, forming a nexus between art and science that could help future pharmacists develop important self-awareness and community-building tools. recently revised education and accreditation standards in pharmacy now include interdisciplinary and metacognition elements and programs such as pharmart can help schools and colleges meet these requirements [17, 18]. at the raic, future research is planned to art show paints a collaborative picture stark, paris, rogan, finnegan dopkins, and etter hypothesis, vol. 29, no. 2, fall/winter 2017 30 determine the effect of art and creativity on professionalism, stress relief, and confidence in health professional students. references 1. frei, j, alvarez, se, alexander mb. ways of seeing: using the visual arts in nursing education. journal of nursing education. 2010;49(12):672-676. doi: 10.3928/01484834-20100831-04 2. jones ek, kittendorf al, kumagai ak. creative art and medical student development: a qualitative study. medical education. 2016;51:174-183. doi:10.1111/medu.13140 3. beals jb. student art in the library juried exhibition program. art documentation: bulletin of the art libraries society of north america. 2007;26(1), 56-58. 4. shipman jp, stoddart jm, peay wj, eccles ss. spaces that support redefined roles of academic health sciences librarians. journal of the medical library association. 2013;101(3), 179-184. doi:10.3163/1536-5050.101.3.006 5. raskin h. can an art gallery thrive in a science library? college & research libraries news. 2009;70(4), 226-229. 6. buhler ag, davis v. elegance of science: coordinating an annual scientific art contest at the library. college & research libraries news. 2010;71(5), 248-251. 7. auten b, norton hf, tennant mr, edwards me, stoyan-rosenzweig n, daley m. using nlm exhibits and events to engage library users and reach the community. medical reference services quarterly. 2013;32(3):266-289. 8. catalano m, essinger c, ferimer s, lewin-lane s, vaughn p. at the center of things: how an academic library built a bridge between art and science on campus. collaborative librarianship. 2017;9(2):90-96. 9. mccullough i. science outreach through art: a journal article cover gallery. issues in science and technology librarianship. 2015;81. doi:10.5062/f4cj8bgg 10. oliver a. strengthening on-campus relationships via an annual student art commission. journal of library innovation. 2012;3(2):89-104. art show paints a collaborative picture stark, paris, rogan, finnegan dopkins, and etter hypothesis, vol. 29, no. 2, fall/winter 2017 31 11. gomes a, palena-hall e, abate l, sullo e, levett p, wilcoxson m. connecting to our community: extending librarians' roles through collaboration. journal of hospital librarianship. 2011;11(2):165-174. 12. weise f. being there: the library as place. journal of the medical library association.2004;92(1):6-13. 13. lake j, jackson l, hardman c. a fresh perspective on medical education: the lens of the arts. medical education. 2015;49:759-772. doi:10.1111/medu.12768 14. edmonds k, hammond m. how can visual arts help doctors develop medical insight? international journal of art & design education. 2012;31(1):78-89. 15. de la croix a, rose c, wildig e, willson s. arts-based learning in medical education: the students’ perspective. medical education. 2011;45(11):1090-1100. 16. jones ek, kittendorf al, kumagai ak. creative art and medical student development: a qualitative study. medical education. 2017;51(2):174-183. 17. fjortoft n. the challenge of the accreditation council for pharmacy education’s standard four: identifying, teaching, measuring. american journal of pharmacy education. 2016;80(5):73. 18. acpe. accreditation standards and key elements for the professional program in pharmacy leading to the doctor of pharmacy degree. 2015. https://www.acpeaccredit.org/pdf/standards2016final.pdf . accessed september 1, 2017. https://www.acpe-accredit.org/pdf/standards2016final.pdf https://www.acpe-accredit.org/pdf/standards2016final.pdf hypothesis: failure hypothesis vol. 31 no.1 fall/winter 2019 library renovation: from setbacks to success elaine wells, ma, mls, ahip library director suny optometry new york, ny the journey started at a 2016 meeting of the college’s learning resources committee, which i chair as library director, a student representative casually commented that our library looked “dated”. in fact, we actually were up to date, with electronic resources, 24/7 remote access, printers, scanners, wireless, and a state-ofthe-art library management system. however, through the eyes of our young student, the library looked like something from the 1970s, and anyone familiar with that era will know that was not meant as a compliment. the student’s criticism prompted the dean of academic affairs, who sat in on the meeting, to ask when the library had had its last facelift. that was an easy one to answer as far as i knew…never. and i’ve been here over 20 years. our library staff includes two librarians, two paraprofessionals, and two administrative clerks. the staff member with the longest tenure has been here for 27 years, while our most recent hire has been here for seven years. given our small size, each staff member has an area of expertise and responsibility as well as deep knowledge and substantial institutional memory. being intimately acquainted with the library’s age-related shortcomings, none of the staff members was surprised at the comments made on the survey or in the focus group, and everyone understood why we needed change. just like that, a renovation was born. how hard could this be? i would query the students on what a suitable update might look like, get a budget, buy some new furniture, and go back to the business of being a librarian rather than an interior designer. spoiler alert: not so fast. background: getting it off the ground with student consensus our library serves approximately 400 professional students, as well as graduate students, didactic faculty, clinical faculty, and clinical and administrative staff in a 10,750 square foot, one floor facility. to get the ball rolling, i developed a survey designed to measure satisfaction (or more appropriately, “dissatisfaction”) with our facility. we had a response rate of approximately 20%, which exceeded our expectations. about a month later, we followed up the survey with a focus group. the group was limited to 20 attendees, and the invitation was extended to the entire college community with responses accepted on a first come first served basis. most responses came from our professional four-year professional students, perhaps motivated by the promise of a free lunch. although our library serves a varied constituency, our hypothesis: failure hypothesis vol. 31 no.1 fall/winter 2019 professional students are the most vested in the facility. they are the ones who make the library their home, spending most of their waking (and some of their sleeping) hours in the library. the online survey and the focus group comments showed remarkable consistency, and below was the “wish list” that emerged from their written and spoken comments: areas for quiet study private study rooms, study carrels, areas of the library designated as “quiet zones”; group study areas large tables, places to think out loud and share ideas, white boards, spaces to spread out; better lighting the simple fluorescent non-led lights were deemed too “yellow” by the students, and were not conducive to long-term studying; comfortable chairs the chairs we had for so many years were crimes against ergonomic design. they sagged in the middle, provided no back support, and perhaps worst of all, did not roll; natural light our library is fortunate to face one of new york city’s most beautiful parks. however, we had limited windows and large, hulking walls that seemed to block the sun; fewer dusty bound volumes like most bound journal collections, ours had become somewhat idiosyncratic. we had a strong core collection, which i promised the college community would maintain intact; however, we also had donated volumes, limited runs, suspiciously odd titles (the journal of soviet ultrasound? seriously?) with many bound journals duplicating content that was fully available online through several different aggregators; more soothing wall colors the 70s era “salmon” and “turquoise” was no longer cutting it; wired study tables during hours of peak use, our library was an obstacle course with student laptops, phones, and ipads plugged into walls and cords creating tripping hazards. getting started: following the survey and focus group, the first steps of the renovation were relatively simple. we opened up space by clearing out some ancient equipment, including a microform reader, slide projectors and, yes, a typewriter. we reduced the shelving by significantly cutting our reference collection and moving most of the books (at least those we were keeping) to the main collection, thus reducing the space they took and increasing their availability for use. we replaced as many print volumes as possible with digital resources. we weeded old volumes, and repurposed some furniture donated due to renovations on other floors. the first stage was easy, nondisruptive, and accomplished on a shoestring budget. it seemed that we were on to hypothesis: failure hypothesis vol. 31 no.1 fall/winter 2019 something here! we even presented on the topic of the synergistic effect of reducing the print collection and opening up space. [1] the more progress we made through simple steps, the more we saw the immense opportunities for improvement. when we (and the administration and students) got a taste of change, we decided to take the deep dive. the students’ requests were both reasonable and long overdue. before my eyes, the renovation was morphing from a few cosmetic changes to a total overhaul of the library’s physical and intellectual content. the words “go big or go home” rolled through my brain. i will confess that there were a few times during the renovation process when “go home” seemed vastly preferable to “going big.” we were soon to discover that the more ambitious our goals, the bigger the potholes in the road ahead. our challenges fell into five main categories: visualizing the new space, developing consensus among the library staff, doing the heavy physical and intellectual work of reducing our bound journal collection to accommodate the new shelving configuration, undertaking the physical challenge of deshelving and re-shelving, and finally, remembering that despite physical chaos, our job is to keep the information flowing. [1] reimagining our space: since visual arts, interior design, and spatial relations are not part of my skill set, i got permission to hire a design firm with a proven track record in planning beautiful and functional libraries. i advised our consultants at the outset that we wanted our students to be partners in the renovation process, which meant involving the students in every decision we made. what we did right: we needed mechanisms for plugging our students into the process, with the goal of creating eye-catching information-gathering tools that even the most sleep-deprived student could not ignore. i created a special “branded” email conference with its own name and logo (a paint roller) and periodically sent out questions, updates, and reminders of what was happening. in return, the students sent back ideas, questions, and thank you notes. when it came to selecting carpeting and wall colors, we created design boards listing color and pattern options, and also displayed our interior designers’ suggestions for fabrics and furniture colors. we placed them at the entrance to the library with a voting sheet so students could select their own preferences. this generated a great deal of enthusiasm and a sense of competition, as students checked in to see if their specific color combinations were “winning.” we even invited some students to come along on our furniture buying field trip. however, studying took precedence as it always does, so they trusted us with the decision. the students were a great asset. we hypothesis: failure hypothesis vol. 31 no.1 fall/winter 2019 listened to them. they were honest and decisive, and appreciated being “heard.” we were so passionate about their involvement that we explained the process via poster presentation. [1] where we went wrong: the students were amazingly adaptable during the process, even when noisy drilling, demolition of the circulation desk (along with the unexpected discovery of ancient asbestos under the desk), and plastic covers draped over everything radically changed their physical environment. what they were not so crazy about was the fact that the quiet study furniture (the study carrels) took way longer to arrive than the tables. since parts of the library were inaccessible, and except for a few study rooms all of the space was open, studying tended to be a noisy endeavor. the students were sitting in close proximity to one another and combined with the stress of exams, getting on each other’s’ nerves. they were surprisingly uncomfortable with asking each other politely to quiet down, and urged me to become the infamous “shushing librarian.” my influence in that vein, however, only went so far. i got desperate emails about the noise level, with requests to please do something about it. finally, i appealed to the dean of student affairs and asked him to approach it from a “professional ethics” standpoint. it worked to some degree, and there was a collective sigh of relief when the study carrels arrived just in time for exams. what we learned: i had no idea how long furniture took to arrive, or how long it would take to assemble. i also did not realize how edgy students get when they feel their studying styles are compromised the takeaway: carve out space, even if it’s small, for different studying styles. know in advance where the quiet zones and the group study zones will be, post signs, and send email reminders. invest in an earplug dispenser they’ll love it! [2] involve the library staff in all decisions to promote buy-in what we did right: no one likes surprises in the workplace, so i shared every piece of information the minute i received it. all floor plans and furniture/color schemes were presented to staff. i announced scheduled work as soon as i knew the timing. staff selected and approved the floor plan and the decorating scheme with minor disagreements. we took a furniture buying trip and picked out the new furnishings together, sitting in a variety of chairs and couches to see which ones promised to be the most comfortable for hours of hypothesis: failure hypothesis vol. 31 no.1 fall/winter 2019 studying. to be sure that i was covering all the bases, the selection committee included a representative from student affairs (who knew the students’ tastes and could advocate for their comfort) and an employee from facilities familiar with the kind of fabrics that can withstand shoes, food, and sleeping. since we were losing six full journal stacks in the bound journal section (almost 50% of what we had) it was clear that substantial recycling needed to be done. the intellectual work behind that decision was intense, but i developed a “decision grid” that evaluated the following with respect to each bound journal title: ● shelf space occupied ● importance to our core collection ● dates of coverage ● preceding and succeeding titles when relevant ● frequency of use via interlibrary loan and/or document delivery ● availability online via aggregators i was confident about the analysis done and the decisions made, and it was clear that many of our bound journals had overstayed their welcome. where we went wrong: i seriously underestimated how unsettling change, even positive change, would be for a group of people who had inhabited, worked in, and managed the same space for over 20 years. our old circulation desk had a place for everything and everything was in its place. staff felt that that the new desk would mean inadequate storage and a slightly modified interpersonal space when interacting with patrons, differences that led to concerns about comfort and circulation errors. removal of 50% of our bound journals meant saying goodbye to old friends (albeit dusty ones) that staff had put great effort into binding and organizing. they weren’t just books they represented hard work that now somehow seemed irrelevant. our journal collection had been scrupulously organized and maintained, even surviving a move to a new building 19 years ago. having to remove the journals quickly and without a detailed storage plan was extremely disconcerting to staff members who had invested so much of their time to making sure everything was in the proper order. not having an upfront plan for reshelving them was even more unnerving. in addition, staff was justifiably concerned about a possible lengthy period of unavailability. what we learned: it is easy for library staff to think of the library as our own space. we have arranged it in a way that was functional for us and over the last 20 years it had become our extended home. the truth is, however, it is not ours; it is public space. it needs to be responsive not only to what our patrons want now, but as much as possible to what hypothesis: failure hypothesis vol. 31 no.1 fall/winter 2019 they will want until the next renovation comes along. fostering a vision for the process while understanding everyone’s deep sense of ownership of the space proved far trickier than i had ever imagined. the takeaway: plan openly, work transparently, share information, and involve staff in decisions. remember, though, that while many voices will be heard, only one will make the decision; making that clear at the outset will prevent conflict and keep the process moving forward. finally, work through the discomfort of change until you reach your new comfort zone. [3] clearing the shelves for stack reconfiguration background: the most substantial reconfiguration of space in the library involved the bound journal stacks, which were eliminated for the purposes of opening up study space and creating a sight line from the south side of the library to the north. while this article focuses upon the physical work that took place, it is incumbent upon me to give a nod to the intense intellectual work also involved in weeding our collection. our small staff, lack of redundancy, and other priorities meant that that particular job fell to me, another decision i perhaps should have made differently. our in-house construction staff scheduled the demolition and reconfiguration of our stacks to be done by outside contractors. before that could happen, we had to remove the journals from the shelves, a task that ideally required physical strength and sufficient knowledge of the journal section to keep them in order during the move. i quickly discovered that those two skill sets did not intersect among library staff members and that removing heavy volumes is not for the faint of heart…or back. we were fighting the clock to get the shelves cleared so that the work could begin on time. students had no desire to do the heavy lifting, even with promises of copious amounts of pizza. our incredible facilities/housekeeping staff stepped in to clear the shelves to keep us on schedule and earn a bit of overtime pay. clearing the bound journals from the shelves went smoothly, thanks to the discipline, reliability, smarts and stamina of our housekeeping staff. the post-removal organization of afore mentioned bound journals? not so much. what we did right: we worked closely with our facilities department to schedule noisy work at minimally disruptive times and our housekeeping staff/journal removal crew worked like a well-oiled machine. for individuals who do not work in the library, our crew was remarkably organized and knew the importance of keeping boxes and their contents in hypothesis: failure hypothesis vol. 31 no.1 fall/winter 2019 order. thanks to overtime work from our facilities staff, the shelves were cleared and the work was on schedule. where we went wrong: my intellectual content analysis showed me that at least half of the bound journals would not be returning to the stacks, and they should have been removed before the process started. time was not on our side, however, and that would have meant a preliminary de-shelving project that simply wasn’t an option. in addition, there was no way of keeping pace with our facilities helpers who were emptying shelves at record speed. despite trying to photograph and record what went into each of the more than 2,000 boxes the best we could do was remove the journals in alphabetical order and number the boxes sequentially, a plan that was distinctly inadequate in the eyes of our serials manager. the library has no onsite storage, another obstacle that we should have thoroughly discussed before creating 2,000 boxes in need of storage. the boxes were temporarily stored in an empty anatomy lab (a feat of moving in and of itself, again carried out by our crack housekeeping crew). however, summer turned to fall and the anatomy lab was about to be pressed into use for teaching so the boxes returned, piled anywhere in the library they could fit in alcoves, study rooms, around the perimeter, and on top of workspaces. what we learned: make sure construction adheres to your schedule and supports the intellectual content decisions. there are many departments involved in even a small library renovation facilities, student affairs, housekeeping, and administration to name a few. the integrity of the library’s collections and services is a priority though; there will be interruptions and detours, but this is about creating a better library, and that goes beyond the cosmetic changes and the physical facility. we were lucky in that the boxes, despite traveling throughout the building, ended up in reasonably good numerical order. if they hadn’t, putting the journal collection back together would have been like assembling a jigsaw puzzle…without the fun. the takeaway: don’t let the proverbial tail wag the dog. assess your resources, interrupt other staff work when you need help with a time-sensitive project, and don’t let anyone underestimate what it takes to get the library ready for all this. oh, and when someone opens a box that has spent time in an anatomy lab and shouts “what is that?” well, you might want to think before looking. hypothesis: failure hypothesis vol. 31 no.1 fall/winter 2019 [4] the students will supply the physical labor for re-shelving background: once the stacks were re-configured, we faced the task of getting the journals we were keeping back on the shelves. those that were leaving us would be going into recycling bins; however, with no time to do the weeding first, all of the journals remained packed away in the boxes, so the weeding would have to happen simultaneously with the re-shelving. what we did right: despite the journals being packed in 2,000 unlabeled boxes, and their migration from the library to the anatomy lab and back again, we did manage to maintain the boxes in pretty good sequential order. where we went wrong: difficult as it as to comprehend this now, i did not give nearly enough thought to the re-shelving process. first, because i underestimated what a big job it would be, and secondly, because i seriously did not consider the physical strength required. looking back, i am not quite sure how that escaped me, but it did. we have hale and hearty young students, always hungry for pizza and work/study paychecks, and i thought they would take it on. they seemed to be the perfect candidates. what i failed to consider is that they can make the same amount of money by checking out books at the circulation desk so the number of students volunteering to re-shelve journals was absolutely zero. our housekeeping help, so adept at clearing the shelves at a record pace, could not be expected to not only re-shelve the journals in order, but leave out the ones that were being recycled and leave room for others to grow. many of our journals had similar names and titles that were difficult to read. the risk of re-shelving something we did not want, or worse, recycling something we did want, was significant. what we learned: after discussing my conundrum with some colleagues who had recently renovated, i decided to pay a professional to get the bound journals back in order. this expense was not in the original budget, but the cost of living with the boxes, inaccessible journals, and disgruntled staff and patrons was far greater. under the guidance of our serials manager, we made a simple, color coded list: 1) journals to keep were indicated in black, 2) journals to recycle were indicated in red, 3) journals needing room to grow were indicated in green. the professional team dealt with our 2,000 boxes and our less than perfect packing scheme and had the journals back on the shelves in two days. hypothesis: failure hypothesis vol. 31 no.1 fall/winter 2019 the takeaway: time is money and inaccessibility causes frustration. admit when you need help, and don’t be shy about asking for more money than you planned for. what comes next? phase iii of our renovation will start in the late spring of 2019. we will transform one under-utilized computer lab into a quiet study area with carrels and a small computer lab into a conference/meeting room, outfit the six study rooms with monitors and floor to ceiling whiteboards, replace our old office furniture, and build a café. i believe, and hope, that the hardest parts, and most of the decision-making are behind me. [5] remember that the library is more than the renovation background: the renovation seemed all-consuming, especially since i was completely in the dark about how it would go. a renovation is so visible, everyone has questions and opinions; on certain days, i avoided crowded elevators because i know someone (or several people) would ask any of these dreaded questions: so what’s going on with the renovation? when will you be done? when are you getting rid of all those boxes? these questions were especially problematic since, as a librarian, i assiduously avoid the “i don’t know” answer despite sometimes not really knowing. what i did right: i accepted that managing the renovation put me squarely outside my comfort zone; however, i have never been afraid of learning something new. i asked questions, i made mistakes, i talked to colleagues, and i realized what a vast source of expertise is out there if i just looked for it. i trusted my instincts, and when i needed to make an executive decision, even an unpopular one, i made it. and i lived to tell about it all. i also remembered that through it all, we ran a library. we never closed (even when we had to step over carpeting glue and rope off parts of the facility), we never shut down interlibrary loan, even if every single article wasn’t available every single day, and no book was ever off limits. everyone got every piece of information he/she needed. thanks to the efforts of our digital and technical services librarian, we are surviving a migration to a new library services platform a digital change to complement our physical library renovation. what i did wrong: occasionally, i forgot some of the above. hypothesis: failure hypothesis vol. 31 no.1 fall/winter 2019 the takeaway: as challenging as the experience has been, always focus on the outcome. when i look out of my office window and see the beautiful carpeting, the natural light, the students lounging and studying on the comfortable furniture instead of jockeying for limited space, some with earplugs studying quietly and some in groups learning noisily, i know we did something good. my thanks go to the library staff for being so good at their jobs and so dedicated to this place, to the administration for supporting us every step of the way with funding and encouragement, and to the students, who helped plan a wonderful new library while putting up with some drilling and hammering along the way. coming full circle: as i mentioned, a student representative to the learning resources committee was the catalyst for the renovation. it was my goal to involve the students from start to finish, and to create a library that would not only serve them now but leave space for growth. they were honest and decisive and encouraging and they were grateful, occasionally sending an email my way just to say thanks. just this past week, an accreditation team visited our college. they had a private session with the students, who are not known for holding back when they have something to say. faculty were not allowed at that meeting, but in my own meeting with the accreditors, one of the team members told me that the students had been effusive in their praise of our efforts on their behalf. their exact words, as noted by our evaluator were: “our voices are always heard.” despite all the missteps, conflict, and headaches, that was what i had planned all along. references [1] wells e, locascio j. if you build it they will come: re-imagining our space, re-thinking our collection. presentation at: metropolitan library council annual meeting; january, 2016; new york, ny; jan, 2017. editorial foster and perryman hypothesis, vol. 30, no. 1, fall/winter 2018 3 revitalizing hypothesis erin d foster, msls co-editor, hypothesis data services librarian ruth lilly medical library indiana university school of medicine indianapolis, in carol l perryman, phd co-editor, hypothesis associate professor school of library and information studies texas woman’s university denton, tx hypothesis has existed for over 30 years the first issue was released in 1987 with at least one issue published per year since its debut. with this kind of legacy, the journal has experienced change at multiple levels ranging from leadership (both editorial and sectionlevel) to the logo. as co-editors, it has been our privilege to steward the journal through a process of revitalization and repurposing, which we recently had an opportunity to summarize when applying for the section project of the year award. while hypothesis is part of the research section’s “normal operational programming”, efforts made in the past year have arguably extended beyond the standard operating procedures and, with encouragement from leaders of the research section, we threw our hats into the ring! in past editorials, we discussed the submission system now used by contributors (https://www.mlanet.org/e/sx/in/eid=70) as well as the work taken to standardize journal policy, format, and content categories (https://www.mlanet.org/page/hypothesis). for this issue, hypothesis piloted peer review checklists for both research and non-research project/program description papers. these checklists aim to define the peer review criteria for these article types as well as guide peer reviewers in the process of reviewing. we received excellent feedback from our awesome reviewers on these forms and hope to have them up on our public site within the new year so hypothesis contributors can see the criteria used to evaluate their submissions. this issue also contains content that demonstrates how hypothesis is dedicated to supporting the dissemination of information about research beyond just the results. in our “research project spotlight”, angela spencer details a research project to explore health https://www.mlanet.org/e/sx/in/eid=70 https://www.mlanet.org/page/hypothesis editorial foster and perryman hypothesis, vol. 30, no. 1, fall/winter 2018 4 sciences library closings in the united states and invites readers interested in participating in the project to reach out to her. heather n. holmes introduces a new column, hypothesis: failure “dedicated to failures, lessons learned, or mistakes that mattered”. finally, we recognize the recipients of the research section’s mla 2018 meeting stipend scholarships as well as the contributed paper/poster awardees in our annual “research section spotlight”. thank you to those to who contributed to this issue, in particular those who peer reviewed the three (!) research or project/program description articles we published this issue. we appreciate your prompt and thoughtful feedback! past issues of hypothesis can be found on the research section homepage (https://www.mlanet.org/p/cm/ld/fid=503) and, as always, we welcome any comments, feedbacks, or questions at mlarshypothesis@gmail.com. enjoy the issue! carol & erin -- national library of medicine (us). digital collections [internet]. bethesda (md): surgeon general's reference library (1948). available from: http://resource.nlm.nih.gov/101445631 https://www.mlanet.org/p/cm/ld/fid=503 mailto:mlarshypothesis@gmail.com http://resource.nlm.nih.gov/101445631 research hypothesis vol. 32 no.1 fall/winter 2020 expanding a single-institution survey to multiple institutions: lessons learned in research design and deployment erin r.b. eldermire, mls head, flower sprecker university library cornell university ithaca, ny kristine m. alpi, mls, ahip university librarian oregon health & sciences university library portland, or suzanne fricke, mls, ahip animal health sciences librarian washington state university pullman, wa andrea c. kepsel, mls, ahip health sciences educational technology librarian michigan state university lansing, mi erin e. kerby, msi veterinary medicine librarian university of illinois urbana, il hannah f. norton, msis, ahip chair, health science campus library university of florida gainesville, fl jessica r. page, mlis head, veterinary medicine library ohio state university columbus, oh abstract objective: creating generalizable knowledge across institutions is a step beyond a successful local research project. the purpose of this article is to share the process and lessons learned from expanding a survey tool developed and piloted at a single veterinary college to its deployment at multiple veterinary colleges in the united states and canada. problem: little guidance exists on expanding a survey developed for a single institution to distribution to health professions students across multiple institutions. research hypothesis vol. 32 no.1 fall/winter 2020 methods: in june 2016, the first author of the survey contacted librarians from veterinary colleges to explore a possible multi-institution study to investigate student behaviors and perceptions around scientific information. librarians from 29 institutions initially expressed interest. those at 15 institutions participated in initial planning, and eight elected to distribute the survey. of these, seven submitted for irb review at their own institution and one institution facilitated the distribution of the survey under the original institution’s irb exemption. findings: the irb submission process and requirements varied by participating institution. mean time from submission to approval was 10 days (range: 2-31 days). several changes were made to the survey based on the recommendations of participating librarians, ranging from simplifying the method of survey distribution to modifying specific questions to make them meaningful across institutions. because participating institutions did not have synchronized academic calendars, the survey distribution took a staggered approach between institutions based on irb review and varying institutional processes. conclusions: expanding even a simple irb-exempt survey from one institution to others requires careful consideration of local practices, attention to differences in the irb process, and ethical considerations for recruiting students where librarians serve as instructors or hold other positions of influence. attempts to standardize recruitment messaging and survey questions for generalizable results required compromise by the librarian researchers at participating institutions. introduction educators struggle to know if research conducted at one institution will translate to similar results at other institutions with differing curricula and cultures. to develop a generalizable understanding of how veterinary students retrieve, evaluate and apply information from scientific papers, a librarian researcher solicited interest and feasibility of administering a survey, developed and piloted at cornell university, across multiple institutions. the purpose of this report is to share the process and lessons learned from expanding a survey developed and deployed at a single veterinary college to multiple veterinary colleges in the united states and canada. findings from the survey research itself were published in volume 107, issue 4 of the journal of the medical library association [toews, oct 2019] [1]. literature review little literature exists to inform conducting a multi-institutional survey to assess health professions students’ information behaviors. a search of pubmed in march 2020 research hypothesis vol. 32 no.1 fall/winter 2020 for articles with “library” in the affiliation field and the publication type of “multicenter study” yielded few studies involving practicing librarians studying populations at multiple institutions. three of these surveys address the practices of health professionals [2,3] or medical photographers [4]. each of these have only two authors rather than broader authorship representing the other participating institutions. perhaps one of the bestknown multicenter studies by librarians with authorship from multiple institutions is the marshall et al [5] study of physicians, residents and nurses on the value of library and information services in patient care. we did not find any multicenter studies by librarians researching students in the health professions. outside of the librarianship domain, a small number of studies address the implementation of multi-institutional research involving medical students as participants and discuss the varied landscape of institutional review board (irb) review processes at included institutions. mealer et al [6] provide a helpful summary of human subjects research and irbs. sarpel et al [7] outline experiences in interfacing with seven irbs for a coordinated research project. a strategic approach to submitting to institutional irbs is advised [8]. one approach that was successful in oermann et al’s [9] study across 10 institutions was pursuing a deferral agreement, in which irbs from different institutions defer to an external irb’s decision. whether pursuing novel research or through coordinated survey efforts such as the acrl project outcome for academic libraries [10] or ithaka s+r [11], participation in multi-institutional human research efforts will require librarians at each institution to pursue irb approval. regardless of the details, flexibility and communication are paramount as the specifics of a multi-institutional research study may introduce novel situations to navigate [8,12]. research timeframes can quickly expand and diverge when working across different colleges and curricula [12]. methods in may 2016, the first author presented initial findings from a survey piloted at cornell university at the medical library association annual meeting [13]. this survey investigated student behaviors and perceptions about reading scientific information. through this presentation, she solicited interest from potential collaborators to explore a multi-institutional study using the same survey. on june 17, 2016, the first author sent a message via two email discussion lists that host veterinary librarians (vetlib-g@vt.edu and vetlibs@lists.mlanet.org) to solicit further interest. librarians from 29 institutions, representing veterinary libraries in austria, australia, canada, ireland, the united states, and zimbabwe, expressed interest in participating. cornell hosted a webex [14] meeting to discuss this project on july 7, 2016. during the meeting, librarians at 15 institutions participated in initial planning discussions about timing, the role of librarians in the study, changes needed to survey questions, irb requirements, promotional strategies, and the use of qualtrics [15] mailto:vetlib-g@vt.edu mailto:vetlibs@lists.mlanet.org research hypothesis vol. 32 no.1 fall/winter 2020 software for survey distribution. the recording was distributed via a onedrive for business [16] file. a message bounced back that recipients might not be able to access the files attached, so the recording was resent as a link to a shared box [17] account. after the meeting, librarians at 17 (59%) institutions decided not to administer the survey. reasons included poor timing, staff changes, curriculum changes, lack of interest from institutional leaders, and lack of time to devote to such a project. afterwards, four (14%) additional institutions were unable to continue their collaboration due to leaves (planned and unplanned) or changes in work prioritization. librarians representing the institutions that remained were all in the united states or canada. the first author used a consensus-driven approach to make decisions and take next steps. from her initial solicitation message sent to email discussion lists to the date of first survey launch, seven email updates to participating individuals were sent and 105 days elapsed. between each email update there were a number of individual or follow-up emails or phone calls. ultimately, librarians at eight institutions (28% of those who initially expressed interest) distributed the survey: librarians at seven institutions submitted for irb approval or exemption at their own institution and one facilitated the distribution of the survey under the original institution’s irb exemption. the survey was distributed at participating institutions between september 14 and october 26, 2016. during the survey design and distribution process, participating librarians logged their experience with irb submission and survey distribution for future reference and comparison. this study reviewing and summarizing those experiences was determined to be exempt by the michigan state university institutional review board (msu study id study00003868). after the original research was published, participating librarians reviewed their notes and described modifications that needed to be made to the irb and survey process at their own institutions, and questions and challenges that arose during the process. the narratives were anonymized and composites were created to highlight general themes and challenges. we discussed the possibility of talking with non-participants about their reasons for non-participation, but decided not to pursue this line of investigation due to time constraints. findings the multi-institutional survey, like the pilot survey, included a 30 short-answer or likert-type response structure, and an embedded article for participants to read and then respond to additional questions. the survey instrument can be seen in the supplemental files of eldermire et al [1]. adapting the survey to be distributed to students across multiple institutions required an iterative process with participation from each institution’s librarian. the adaptations, described below, impacted the survey implementation and research outcomes, and likely contributed to a lower participation rate than the pilot survey. research hypothesis vol. 32 no.1 fall/winter 2020 minor changes to the survey content were made to clarify language to participants in multiple contexts. examples included adding a drop-down menu for participants to indicate which school they were attending, and contact information of local institutional leads for questions or concerns about the survey. other changes were more significant. unlike the pilot survey, no incentives for participation were given for the multi-institutional study. librarian researchers felt that offering incentives would lead to unneeded logistical complexities as well as funding source challenges. in addition, the embedded article included in the survey was changed to an open-access article so that copyright restrictions or authentication requirements did not add complexity. although both studies focused on metacognition, the pilot survey used an article on the impact of teaching metacognitive learning strategies on academic performance [18], while the multi-institutional survey article focused on the impact of deliberate practice on the development of clinical skills [19]. the different embedded articles limited comparison between the pilot and multiinstitutional surveys. thus, we did not compare the embedded article analysis between the pilot and multi-institutional surveys. approval or exemption and project information required by institutional irbs varied between participating institutions. three irbs approved or exempted a customized version of the cornell university protocol with minor changes. three others required the local researcher to include additional information, such as timelines for how students would receive the survey invitation, details on data storage, and when and how students would be reminded about the opportunity to participate in the survey. one irb required a copy of cornell’s irb exemption and assurance that ip addresses would not be collected for surveys answered via an anonymous link to move forward in their approval/exemption process. partnering institutions in multi-state programs, of which there was one, relied on exemption by the primary program institution. the mean time from submission to approval or exemption was 10 days with a range of 2 to 31 days. irb processes also impacted the distribution method used. the initial plan was to administer the survey using the qualtrics mailer from the first author’s account, with the ability to send targeted reminders to non-responders. however, we determined that this was not possible without providing student emails to the primary administering institution, which did not meet the confidentiality needs at some institutions whereas the original survey design broke the survey into two parts, allowing participants to leave the survey and return to it later, the fact that some institutions required distribution of the survey with an anonymous link meant that their participants could not easily leave and return to the survey. therefore, the survey was adapted to encourage participants to take it in one session. variance between college administrations also impacted how the survey was distributed to participants. some institutions did not require direct approval for librarians to contact students about a research study, and the librarian researcher was able to research hypothesis vol. 32 no.1 fall/winter 2020 send a survey invitation directly to cohort email lists or through the qualtrics mailer. other institutions required permission from deans of the respective veterinary colleges, did not allow direct emails to student lists from librarians, or restricted the number of times students could receive reminder messages to take the survey. one administration requested that the survey be approved by the american association of veterinary medical colleges (aavmc), however, when contacted, the aavmc declined to do so stating that it was beyond the capacity of their organization. survey fatigue was a stated concern for at least one institution. some institutions allowed only one reminder to be sent, others allowed two or more. due to the variety of processing times for administrative and irb approval/exemption, as well as different academic calendars at participating institutions, the survey distribution dates at each institution were staggered and did not occur on a particular date. while this worked on a rolling basis, it meant that by the time the survey was released at some institutions, students were already in the midst of the semester and were less likely to participate in the survey due to course load, time restraints, and other similar reasons. we collected data for most of the institutions in the cornell university instance of qualtrics. for these, the first author distributed institution-specific results to the coauthor at each institution. data collected by other institutions in their own instance of qualtrics were shared with the first author to be aggregated with the group data in the analysis phase. discussion conducting this multi-institutional research project took collaboration and patience while the authors iteratively adapted a survey from one institution, sought institutional assurances, and administered it in heterogeneous contexts. throughout this process we learned lessons we believe will help those considering conducting a multi-institutional study using a survey instrument. first, we advise careful consideration of survey content and platform functionality. when adapting a survey that was created for one institution to be used at multiple institutions, intensive communication between researchers and survey instrument testing may be necessary. adapting the survey required each researcher to iteratively evaluate the survey with their institutional environment in mind, suggest changes that could be accommodated by all participating institutions, and field test and reassess the updated instrument until all were comfortable with the ultimate product. second, as research involving human subjects, irb policies and expectations varied between institutions. variations can occur in the time elapsed from submission to approval or exemption, as well as in the materials or information required for review. the first author’s expectation that materials from one irb exemption could be applied to other institutions’ irb processes with minor edits proved not to be the case, as several research hypothesis vol. 32 no.1 fall/winter 2020 other irbs required much more detailed information. furthermore, irb and equivalent review boards in other countries may have varying protections for human subjects or processes for seeking approval or exemption [20]. one size does not fit all. if we were to implement this project again, we would expect variation in irb processes at each institution. each irb is composed of individuals who are operating in a unique institutional environment, so variation and change are to be expected. additionally, irb processes have changed since we sought approval for our multi-institutional study. if we were to embark on a similar multi-institutional study at the same institutions now, we would start by examining the irb process. since the implementation of this study, the irb landscape has changed significantly, including the requirement of a single irb for multi-institutional studies [21]. as this policy trickles down, organizations have adapted. for example, the national institutes of health adapted a single irb policy for multi-site research starting in 2018 [22]. we may also consider a deferral agreement, as suggested by the literature, for a more strategic approach. third, we found that administrative considerations and expectations varied between institutions. similar to other institutional variations encountered in this process, administrators such as deans, department heads, and other leaders had differing expectations on how students could learn more about the survey, survey dissemination, and security of participant information. the outcome of such variation translated to differences in how the survey was distributed at each institution, introducing additional variability to the participation and outcomes across institutions. attempts to standardize recruitment messaging and survey questions for generalizable results required compromise by the researchers at participating institutions. related, participating librarians varied in faculty status; however, that status was not reported by any participating librarian as influencing the priority that survey distribution received at each institution. fourth, managing a multi-institutional project required additional layers of adaptability beyond the challenges of project management at a single institution. typical project management tasks, such as clear and persistent communication from the leader–both with the group and with individuals–and final decision making to move the group forward were essential. project management across multiple institutions required additional accommodations, such as scheduling discussions across time zones, establishing an online file-sharing platform that all researchers could access and contribute to, and in our specific case, integrating an open access article into the survey to be distributed across institutions rather than negotiating copyright restrictions or institutional links to full-text that would require authentication by participants. had participants from additional time zones or countries participated, the group may have needed to exercise more flexibility in meeting arrangements, file sharing tools, or other details that we did not encounter in conducting this study. research hypothesis vol. 32 no.1 fall/winter 2020 an additional participant protection issue to be addressed would be to consider a system that can be trusted to safely store survey results for a multi-institutional survey. some institutions specified that they did not want survey responses to be stored at cornell university. storing survey data in the cloud with restricted access by certain individuals may be a viable option, but the online survey tools used may introduce other data storage considerations. for example, each of three popular survey tools— qualtrics, survey monkey, and google forms—have different access, storage and company policy environments. we suggest careful consideration of the survey tool and implications for data access and storage. further, the language of our consent process did not permit the data to be made publicly available—this is something we recommend considering in future studies. at the time of our survey, there were significant concerns about survey fatigue and stress among students frequently surveyed for assessment of teaching, assessment of learning, and research. attitudes and processes surrounding educational research with veterinary students have likely evolved since we pursued this research in 2016. for example, institutions that did not have or were in the process of developing a survey review process for any surveys to be disseminated to the students in 2016 may have set up a formal process, such as a committee established at the ohio state university for conducting surveys [23]. in the case of one participating institution, there was a plan in 2016 to set up a formal process, but since few requests came in, as of march 2020 the process remains to submit surveys for review and approval by the associate dean and director of academic affairs. in planning for another survey, we would attempt to clear review processes well enough in advance so that surveys could be distributed on the same timeline at all institutions, limiting differences in response rates due to timing. in conclusion, expanding even a simple irb-exempt survey from one institution to others requires careful consideration of local practices, attention to differences in the irb process, and ethical considerations for recruiting students where librarians serve as instructors or hold other positions of influence. administering a survey to participants across multiple institutions will require ample time, cooperation, and flexibility amongst the researchers involved. the survey timeframe, delivery method, communication with participants and survey content were all impacted by the process required to deliver the survey to a multi-institutional context. even though these confounders were a factor in our research, we see great value in conducting studies across institutional boundaries to broaden understanding and conclusions beyond the unique environment of one institution. acknowledgements we thank the anonymous peer reviewers for offering suggestions that strengthened this manuscript. research hypothesis vol. 32 no.1 fall/winter 2020 author contributions authors include erin r. b. eldermire, kristine m. alpi, suzanne fricke, andrea c. kepsel, erin e. kerby, hannah f. norton and jessica r. page. all authors secured irb exemption/permission from their respective institutions to administer the survey referenced in this manuscript, administered the survey, and contributed substantially to writing the manuscript. in addition: • erin eldermire led the effort • kris alpi made substantial revisions to the manuscript • suzanne fricke made significant suggestions for the inclusion of concepts to the manuscript • andrea kepsel sought irb exemption for this study references 1. eldermire erb, fricke s, alpi km, davies e, kepsel ac, norton h. information seeking and evaluation: a multiinstitutional survey of veterinary students. j med lib assoc. 2019 may; 107(4):515-526. doi: https://doi.org/10.5195/jmla.2019.674 2. ajuwon ga, olorunsaye jo. knowledge, access and usage pattern of hinari by researchers and clinicians in tertiary health institutions in south-west nigeria. afr j med med sci. 2013 mar;42(1):97–106. 3. cheng gy, lam lm. information-seeking behavior of health professionals in hong kong: a survey of thirty-seven hospitals. bull med libr assoc. 1996 jan;84(1):32–40. 4. addison fj, kehoe ma. an assessment of risk associated with the photography of microbiological materials. j audiov media med. 2004 sep;27(3):102–6. doi: https://doi.org/10.1080/01405110400007726 5. marshall jg, sollenberger j, easterby-gannett s, morgan lk, klem ml, cavanaugh sk, oliver kb, thompson ca, romanosky n, hunter s. the value of library and information services in patient care: results of a multisite study. j med libr assoc. 2013 jan;101(1):38–46. doi: https://doi.org/10.3163/1536-5050.101.1.007 6. mealer m, flynn l, ironside p, spurlock d. accelerating multisite research in nursing education: navigating institutional review board regulatory issues. j nurs educ. 2017 feb 1;56(2):65–8. doi: https://doi.org/10.3928/01484834-20170123-02 7. sarpel u, hopkins m, more f, yavner s, pusic m, nick m, song h, ellaway r, kalet a. medical students as human subjects in educational research the importance of responder bias. med educ online. 2013 jan1;18(1):19524. doi: https://doi.org/10.3402/meo.v18i0.20757 8. north s, giddens j. lessons learned in multisite, nursing education research while studying a technology learning innovation. j nurs educ. 2013 sep 13;52(10):567–73. doi: https://doi.org/10.3928/01484834-20130313-02 https://doi.org/10.5195/jmla.2019.674 https://doi.org/10.1080/01405110400007726 https://doi.org/10.3163/1536-5050.101.1.007 https://doi.org/10.3928/01484834-20170123-02 https://doi.org/10.3402/meo.v18i0.20757 https://doi.org/10.3928/01484834-20130313-02 research hypothesis vol. 32 no.1 fall/winter 2020 9. oermann mh, hallmark bf, haus c, kardong-edgren se, mccolgan jk, rogers n. conducting multisite research studies in nursing education: brief practice of cpr skills as an exemplar. j nurs educ. 2012 jan;51(1):23–8. doi: https://doi.org/10.3928/01484834-20111130-05 10. acrl project outcome [internet]. [cited 2020 mar 30]. available from: https://acrl.projectoutcome.org/ 11. ithaka s+r [internet]. [cited 2020 mar 30]. available from: https://sr.ithaka.org/ 12. foronda c, alhusen j. lessons learned from an international, multisite simulation research study. nurse educ. 2016 aug;41(4):199–203. doi: https://doi.org/10.1097/nne.0000000000000242 13. eldermire, e. and davies, e. (2016, may). reading behavior of veterinary students: how big (or small) is their picture? medical library association conference. toronto, canada. 14. webex [internet]. [cited 2020 mar 30]. available from: https://www.webex.com/video-conferencing 15. qualtrics [internet]. [cited 2020 mar 30]. available from: https://www.qualtrics.com/ 16. microsoft onedrive [internet]. [cited 2020 mar 30]. available from: https://products.office.com/en-us/onedrive/onedrive-for-business 17. box [internet]. [cited 2020 mar 30]. available from: https://www.box.com/home 18. cook e, kennedy e, mcguire sy. effect of teaching metacognitive learning strategies on performance in general chemistry courses. j chem educ. 2013 aug 13;90(8):961–7. doi: https://doi.org/10.1021/ed300686h 19. duvivier rj, van dalen j, muijtjens am, moulaert vr, van der vleuten cp, scherpbier aj. the role of deliberate practice in the acquisition of clinical skills. bmc med educ. 2011 dec 6;11:101. doi: https://doi.org/10.1093/med/9780199366149.003.0005 20. millum j. canada’s new ethical guidelines for research with humans: a critique and comparison with the united states. can med assoc j. 2012 apr 3;184(6):657–61. doi: https://doi.org/10.1503/cmaj.111217 21. hodge jg, gostin lo. revamping the us federal common rule: modernizing human participant research regulations. j am med assoc. 2017 apr 18;317(15):1521–2. doi: https://doi.org/10.1001/jama.2017.1633 22. single irb policy for multi-site research | grants.nih.gov [internet]. [cited 2020 jan 29]. available from: https://grants.nih.gov/policy/humansubjects/single-irb-policy-multisite-research.htm 23. conducting surveys [internet]. office of academic affairs, the ohio state university. 2017 [cited 2020 mar 30]. available from: https://oaa.osu.edu/conductingsurveys https://doi.org/10.3928/01484834-20111130-05 https://acrl.projectoutcome.org/ https://sr.ithaka.org/ https://doi.org/10.1097/nne.0000000000000242 https://www.webex.com/video-conferencing https://www.qualtrics.com/ https://products.office.com/en-us/onedrive/onedrive-for-business https://www.box.com/home https://doi.org/10.1021/ed300686h https://doi.org/10.1093/med/9780199366149.003.0005 https://doi.org/10.1503/cmaj.111217 https://doi.org/10.1001/jama.2017.1633 https://grants.nih.gov/policy/humansubjects/single-irb-policy-multi-site-research.htm https://grants.nih.gov/policy/humansubjects/single-irb-policy-multi-site-research.htm https://oaa.osu.edu/conducting-surveys https://oaa.osu.edu/conducting-surveys research project spotlight spencer hypothesis, vol. 30, no. 1, fall/winter 2018 6 health sciences library closings: overview and background angela spencer, ahip librarian st. luke’s hospital saint louis, mo angela.spencer@stlukes-stl.com many libraries across the united states have closed and other libraries are facing closure. a recent study of health sciences library closings [1] found that from 1989-2006 approximately 23-34% of u.s. health sciences libraries closed. furthermore, between 2011 and 2015, 613 libraries closed, for an average of 115 closings per year [2]. but why? what was the rationale for these closings? was the decision based solely on economic reasons, or were there other underlying factors? how will providers who lack library services fill their information needs and how can the closing of additional libraries be prevented? a group of health sciences librarians, herein known as the health sciences library study group (hslsg) representing both academic and hospital librarians was informally organized in 2017 to address these issues. the hslsg believes there is a lack of substantive data on why libraries are closing. thus, during the 2018 annual meeting of the medical library association (mla) held in atlanta, georgia, the hospital libraries section, and the outreach and marketing special interest group organized a special content session to share information, and gather ideas and information from interested mla members. the hslsg’s goal is to seek funding to conduct a comprehensive, substantive study to enable the healthcare community to understand why u.s. health sciences libraries are closing, and what the consequences of the closings will be. we also seek to prevent the closing of additional health sciences libraries. some of the questions we hope to answer are: 1. why are health sciences libraries being closed? a. what was the rationale for the closing? b. was the decision based solely on economic reasons, or were there other underlying motives? 2. what will be the effect of closing libraries on healthcare providers, learners, and patients? 3. how will providers who lack library services fill their information needs? research project spotlight spencer hypothesis, vol. 30, no. 1, fall/winter 2018 7 4. how are models of library services and medical information delivery evolving without a library? the project’s overarching goal will be to gather information to define a new strategic vision for the roles and responsibilities of the practicing health sciences librarian and build on the national library of medicine’s strategic plan, 2017-2027. we are looking for people who can work with us on research, data analysis, and statistics. to participate, please contact angela spencer, angela.spencer@stlukes-stl.com or angspencer730@gmail.com works cited 1. schwartz dg, elkin pl. health sciences library closings: a context sensitive pilot study. studies in health technology and informatics, 2017;241: 21-27. doi: 10.3233/978-161499-794-8-21. 2. thibodeau pl, funk cj. trends in hospital librarianship and hospital library services: 1989 to 2006. journal of the medical library association: jmla, 2009;97(4): 273. doi: 10.3163%2f1536-5050.97.4.011. national library of medicine (us). digital collections [internet]. bethesda (md): early medical library, from oeconomus prudens et legalis,1722. available from: http://resource.nlm.nih.gov/101438987 mailto:angela.spencer@stlukes-stl.com mailto:angspencer730@gmail.com original article perryman hypothesis, vol. 30, no. 1, fall/winter 2018 21 credit due: multiple author attribution for interdisciplinary informatics research groups carol l perryman, phd co-editor, hypothesis associate professor school of library and information studies texas woman’s university denton, tx “author” is increasingly an archaic term: more and more people want to know what it is a person has contributed to a project and not just that they have authored a paper about the findings” [1]. objective: to understand issues related to multiple authorship in interdisciplinary working groups, and to identify best practices for authorship attribution for a multidisciplinary group. problem: research groups comprised of individuals from diverse disciplines need to identify their own internal agreement and process for authorship attribution. methods: limited literature review findings and conclusions: resources are described and considerations for interdisciplinary working groups are suggested. introduction author attribution and ranking for multi-author publication has long been an issue of concern, leading to author statements in peer reviewed journals that spell out the nature of contributions, institutional review board (irb) guidelines, and even institution-wide ethics statements. however, such standards pay less attention to concerns of interdisciplinary groups. member expertise in a working group conducting informatics research, for example, may include data curation and knowledge management, literature retrieval and synthesis, statistical modeling, examination of data from very different perspectives (e.g., economic or network analysis). issues relevant to author attribution, including author list ranking, are examined here, from the perspectives of authorship benefits and ethical concerns, concluding with a brief listing of resources and recommendations for action. medical librarians, who are increasingly involved in supporting and performing interdisciplinary research, may be informed by practices in other disciplines. background the informatics research group (irg) at texas woman’s university consists of nearly 20 individuals (faculty, staff, and students from both graduate and undergraduate programs) from different disciplines, including math and computer science, business, nursing, occupational therapy, health studies, and library and information studies. collaborators original article perryman hypothesis, vol. 30, no. 1, fall/winter 2018 22 also include a number of librarians and staff from the institution’s office of sponsored research, who directly support research methods, analysis, and grant activities. the group has already obtained several internal grants and completed a number of poster and conference presentations, as well as having a systematic review and several research papers in process. the focus of research efforts at present is the analysis of a large dataset obtained from a regional foundation, whose for-profit arm collects, cleans, and disseminates data to over 90 regional hospital administrators for decision support and reporting, as well as making it available to external researchers. our particular research area is emergency department utilization by individuals diagnosed with schizophrenia, with questions about health disparities and the intent to model the paths of care for individuals in order to identify bottlenecks and opportunities for intervention. a very incomplete listing of expertise represented by irg group members includes health systems, data analysis, statistical modeling, management and security, literature and systematic review searching, mental health, digital divide, and community informatics. the interdisciplinary nature and large size of the group is new to most members, presenting a valuable opportunity to build trust and collaborative partnerships between people working in different areas, presenting several challenges. these include collaborating across disciplinary boundaries with attention to discipline-specific professional communications for research output, including author ranking and content type attributions. variation between disciplinary norms is a particular problem encountered by the irg. for some members, publication in particular journals will not count toward tenure, even if they have made substantive contributions, and the same is true for author order. these are potential anti-motivational factors that should play a part in the group’s consideration of possible publication venues. while initially an informal agreement was made between members that all would be listed on every work disseminated, concern about the ethical issues involved have resulted in a need to examine the situation and how we might best address it in future work. as an example, a large working systematic review subgroup is led by a faculty member from nursing, with members comprised of faculty from library and information studies, math and computer science, and staff from the university’s library (one of whom serves as the expert guide). listed authorship for the eventual publication, per earlier agreement, would include all members of the larger group, yet not all members have had any role in production. upon discussion, some group members felt that inclusion of every person as a contributor to every product of the group is an important factor in encouraging ongoing participation original article perryman hypothesis, vol. 30, no. 1, fall/winter 2018 23 in a new effort. in addition, the argument was put forth that work products are clearly the result of group meetings where members have contributed to discussions about ongoing research, and all group members have been asked for editing feedback for submitted works. however, there are questions to be addressed. the central question of this limited review is the identification of best practices for authorship attribution for a multidisciplinary group. sub-questions include the following: are there levels of collaborative contribution that should be identified? is contribution to discussion about research products or editing commentary sufficient to warrant inclusion in the author list of the products? if so, how shall contributions to each product be identified and recognized? does the nature of our collaboration warrant inclusion of every group member’s name in every poster, presentation, published research paper, and other disseminated product? in other large workgroups, members may also have wrestled with these questions, but if they share a discipline, questions may be relatively simple to address because journals in medicine, for example, may follow similar standards. methods in order to understand the issues and to identify discipline-specific practices in attribution for interdisciplinary research groups, limited literature searches were performed in google scholar to retrieve materials addressing multi-author attributions in research literature across multiple disciplines. next, the most pertinent articles retrieved during the initial literature retrieval phase were entered into the web of science database to find citations and their citing articles, and all were examined to identify common words or phrases. these were used to construct a search statement restricted to the title field (ti) only (shown in table 1), in order to limit retrieval in the interest of time. citations for pertinent articles were examined and articles retrieved in order to understand issues involved in attribution and differing disciplinary practices. as this work is not a systematic review, use of a second reviewer was not needed. search statement results (((ti=(author* or multiauthor* or multiauthor* or co-author*)) and (ti=(interdisciplin* or collaborat* or interprofess* or multidisciplin*))) not (ti=(authority or authoriz* or authoris*))) and language: (english) indexes=sci-expanded, ssci, a&hci, cpci-s, bkci-s, esci timespan=all years the set was further limited to 2000-2018 380 225 table 1. search statement, web of science, 10/4/2018. original article perryman hypothesis, vol. 30, no. 1, fall/winter 2018 24 exclusion criteria items focused primarily on the topics listed: ● network analysis ● bibliometric analysis ● impact ranking ● literary studies focused on author collaborations ● mapping studies examining author citedness in a discipline or topic ● collaborative authorship not focused on the issue of authorship attribution, i.e. international collaborative research and publication. inclusion criteria items addressing: ● trends in collaborative authorship ● policies of particular journals or association statements ● ethics of author attribution reviewing titles only, 48 items were identified. abstracts (or, in situations where these were unavailable, the item introduction) were then reviewed based on the above criteria. those that did not clearly address the question of authorship attribution were not considered, resulting in 32 items. this subset of articles (available upon request) was reviewed, framing the review. findings and limited review of literature the issue of author attribution in work groups has increasingly been discussed as a concern needing exploration and agreement within professions (particularly in medicine), but also more broadly in the social sciences [2] and problematically in the sciences, where there has been corresponding growth in research misconduct [3]. in 1993, a letter to the editor published in bmj described a growing trend of multiple authorship in 15-year increments from 1935-1990. comparing medical and scientific papers, there was a fourfold increase in multi-authorship over the period for medical literature, compared to a twofold rise for scientific papers [4]. while early on, some condemned the practice as unethical, others have found that due to trends in the complexity of research, multiple contributors were increasingly necessary. at the same time, the ethical expectation of ‘substantive’ contribution to justify author attribution has been expressed: an alternative explanation for our findings is that science has become less of a cottage industry and now requires larger collaborative groups. if this is so multiple original article perryman hypothesis, vol. 30, no. 1, fall/winter 2018 25 authorship is justifiable. clearly, all authors should have played a substantive part in the project reported, otherwise authorship becomes meaningless (pp.1345) [4]. other justifications for group authorship include the need to collect large amounts of input from multiple locations (for medical research, with patients recruited for increasingly large studies); the growing involvement of students in research; and growing subspecialization involvement in interdisciplinary research has required collaboration and subsequent recognition [5]; the ability to contribute from a distance thanks to web-based document preparation and distance conferencing; and for top publications in particular fields, increased competition [6][7]. however, problems have arisen from the practice. “ghost authors” is a term referring to those who have contributed to the work of research but were never recognized in the final product, whereas “honorary authors” are those named in recognition of their support of work efforts, who have not otherwise contributed to the work [8]. in the early 1980s, a scandal arose from the publication of research whose authors had made no real contribution at emory and harvard universities [9], resulting in a recommendation for change by leaders in the international committee of medical journal editors (icmje). sometimes referred to as the vancouver protocol, the standards are used by many universities and across most medical disciplines [10]. while icmje is now widely cited as a standard for literature in medical disciplines, other disciplines vary in their approach, and the path has not been smooth. in disagreeing with the icmje proposal and pointing out that bibliographic databases such as pubmed do not differentiate authorship order in terms of contribution, cappell argued in the pages of jmla that mentors should merit inclusion on the author list as the last author named [11]. cooper, then jmla editor, contended that the practice might encourage coercion, and pointed to the existence of demands by editors (undoubtedly including icmje among them) that authors spell out unique contributions of each and stated that those who had not “contributed significantly to the execution of the project and the writing of the manuscript” (pp.365) should instead be named in an acknowledgment [12]. author ranking conventions have changed over time, a reality that argues in favor of spelled-out attributions at least in research group process documentation and in preparation for a publishing environment lacking universal standards [13]. authorship of research publications is recognized as a benefit to authors for a number of reasons. among them are: contribution to a body of knowledge, personal achievement, evidence of intellectual effort, enhancement of professional reputation, contribution to academic promotion and tenure, research funding, and recognition among professional colleagues [5]. in many disciplines, rank placement in publication is of central importance original article perryman hypothesis, vol. 30, no. 1, fall/winter 2018 26 in tenure and review, based on the agreed-upon expectation that placement denotes level of responsibility and participation. it is important to recognize that on an interdisciplinary team, some will not receive credit in review or tenure evaluation if they are not listed among first, second, or third authors in works disseminated, or if work is not published in particular journals recognized by the discipline. neither problem is addressed by current standards across disciplines. in some disciplines (e.g., math and physics), authors are commonly listed in alphabetical order by surname [14]. a recent survey of institutions across the united states in biomedical engineering, biology, and bioengineering disciplines (n=102, or 18% response rate) asked participants to rank contributions in terms of perceived importance, with the result that “time spent conducting experiments, coming up with a hypothesis, analyzing data, and writing the manuscript were selected as the four most important criteria for both determining one’s authorship status and rank” [15]. other contributions named, in ranked order, were total time spent, uniqueness of techniques (particular techniques such as gis could rank here), quality of contribution to the manuscript, background/literature review, editing/proofreading, applying for funding, and coding. identification of individual contributions is of particular importance to the promotion and tenure process, but here there tends not to be any universal standard. according to klein and falk-krzensinski [16], decisions may often be made at the academic component level, further supporting the need agreed-upon within-group standards. criteria often rests upon promotion and tenure standards, which may not have dealt with increasingly interdisciplinary work, or provide only vague guidance (pp.1057). however, it may not be desirable or possible to achieve complete standardization, due to well-established promotion and tenure criteria within disciplines and institutions, meaning that research groups bear primary responsibility for arriving at their desired practice of attribution. correct attribution has been called a ‘public responsibility’ in that participants represent their work to readers as the product of author investigations [5]; therefore, each member of the group is responsible as a representative of the integrity of works published or otherwise disseminated, and willing to respond to challenges based on its content. as well, finding agreement on collaborative attribution is non-trivial to the ongoing success of the research group, since […] determining who should be listed as the authors of a publication, and in which order, could be often critical to the overall success of a research collaboration. publication is a major product of a collaboration. successful, agreeable determination of publication authorship can increase the likelihood of converting a one-time collaboration into a long-lasting research team [17]. original article perryman hypothesis, vol. 30, no. 1, fall/winter 2018 27 considering that group members bring unique expertise to projects, the issue of critical evaluation of publications, including understanding the authority of contributors with regard to specific contributions made, may be of primary importance to sustainable research collaborations. if, for example, one or more group members perform statistical analysis or other forms of data analysis, it seems commonsensical to attribute this part of a work to the individuals involved not only for credit, but for follow-up: errors or questions can thus be brought to the attention of the appropriate people. at the same time, all named authors have a responsibility to be familiar with the work overall, and to participate in the editing and revision processes. by far, the most widely adopted attribution set of standards, based upon this limited review, appears to be the documentation provided by the icmje. the literature of most disciplines has not specifically addressed the issue of interdisciplinary, multi-author attribution, perhaps due to existing tenure and promotion standards, but many follow similar policies in publication. itself adapted from earlier guidelines, the current icmje statement provides support for decision making, without mention of interdisciplinary author teams. icmje recommends the following 4 criteria for inclusion as an author: substantial contributions to the conception or design of the work; or the acquisition, analysis, or interpretation of data for the work; and drafting the work or revising it critically for important intellectual content; and final approval of the version to be published; and agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved [10]. in addition, the organization specifically recommends that the following not constitute rationale for authorship inclusion: “acquisition of funding; general supervision of a research group or general administrative support; and writing assistance, technical editing, language editing, and proofreading” [10]. those who have contributed to the work but do not meet the 4 criteria should be recognizedwith an acknowledgment. specific wording is suggested for acknowledgments: those whose contributions do not justify authorship may be acknowledged individually or together as a group under a single heading (e.g. "clinical investigators" or "participating investigators"), and their contributions should be specified (e.g., "served as scientific advisors," "critically reviewed the study proposal," "collected original article perryman hypothesis, vol. 30, no. 1, fall/winter 2018 28 data," "provided and cared for study patients", "participated in writing or technical editing of the manuscript"). because acknowledgment may imply endorsement by acknowledged individuals of a study’s data and conclusions, editors are advised to require that the corresponding author obtain written permission to be acknowledged from all acknowledged individuals. [10]. the council of science editors (cse) discusses the issue at some length, providing examples of author attributions for large working groups, and recommendations to editors, bibliographic database editors, and working groups [18]. recognizing that the current lack of standardization leads to problems such as those previously identified, cse also points out that problems with miscalculations of citation statistics and retrieval due to mis-citing authorship are likely, and suggests the inclusion of attribution to group affiliations as shown in the following options (bolding is the author’s): smith sq, suzuki y, mann jt, schulze kt, deangelo c, davis c, jones kj, cunningham tl, snyder mj, gutierrez am; and the generic coalition group. a randomized trial of chemoradiotherapy of esophageal cancer. j onc dis. 2004;183:1763-1770. smith sq, suzuki y, mann jt, schulze kt, deangelo c, davis c, jones kj; generic coalition group. a randomized trial of chemoradiotherapy of esophageal cancer. j onc dis. 2004;183:1763-70. aside from the help provided by the icmje, the american psychological association has also recognized concerns with attribution, and provides useful decision support tools for inclusion and ranking that may be particularly helpful if students are contributing to research efforts. their rubric, entitled ‘authorship determination scorecard’ breaks down and assigns a point value to activities from conceptualizing to the submission process, with writing being more detailed. a second helpful tool is their ‘authorship tie-breaker scorecard’ [20], which provides an even more specific listing of possible contributions, including literature searching, irb document preparation, and document or data management processes. conclusion and recommendations it is a reality that publication requirements will differ based on the venue for publication, and that an interdisciplinary team represents diverse scholarly expectations for promotion and tenure. because of this, there is a need to be flexible, and to set guidelines for discussion, rather than rules. the concerns identified do call for discussion and original article perryman hypothesis, vol. 30, no. 1, fall/winter 2018 29 agreement, preferably at the start of a project. like all research, planning early in the process – before data analysis even begins – benefits the long-term success of the project (or in the case of an interdisciplinary group, the ongoing viability of the group itself), leading to trust based on clear expectations. coming to agreement at the start does not preclude the need to revisit the group’s guidelines; it is also very likely that as work evolves, group members may also grow in understanding of the factors involved in project planning and output management. any collaborative efforts that may lead to formal dissemination, including abstracts, presentations, and full research papers submitted for publication, should consider the following issues, regardless of the requirements of the venue. 1. agreement between group members should be reached before research begins, and should be documented. 2. clear differentiation of contributions in publications. records identifying contributions should be kept to support review, promotion and tenure efforts of members. 3. awareness of review, promotion and tenure criteria needs to be the responsibility of each of the affiliated group members. 4. members should also be responsible for representing their own needs when considering venues for publication of research papers, in particular. 5. direct authorship should be attributed only for substantive contributions, including agreed-upon types of input. 6. if author attribution is ranked, agreed-upon ranking criteria should be used and documented (level of contribution, type of contribution, or other). 7. while publication processes will usually ask for a corresponding author, it will help to also identify specific contact information for authors who ‘own’ responsibility for unique components, such as analysis methods that require considerable expertise. 8. creation and inclusion of a template paragraph recognizing the entire group is encouraged, or if possible, agreement on wording to be included in the authorship listing. for those interested in more in depth exploration of this topic, recommended reading includes klein and falk-krzesinski [16] whose review of author attribution is focused on tenure and promotion at academic institutions. the article also includes a list of recommendations and discussion of multiple medical association and journal standards. also worth review are the recommendations of both the icmje [10], the american original article perryman hypothesis, vol. 30, no. 1, fall/winter 2018 30 psychological association [19,20], and the council of science editors (cse) [18]. a special issue on interdisciplinary research assessment published in 2006, in the journal research evaluation [21] may be informative, adding yet another layer of complexity to the issues with its titular focus on interdisciplinary research evaluation. for a succinct discussion of some of the basic issues, see chapter 3 of the freely downloadable monograph published by the national academies of science, which is devoted to the topic [22]. limitations and further research needed this document makes no claims to generalization because the literature review and thus the subsequent discussion of issues involved are incomplete. the search process was nonexhaustive due to time limitations and the vagaries of natural language searching. there was no attempt to reproduce the searches by another person. future research specific to interdisciplinary research groups might involve efforts to identify and compare publication statements as well as citation counts based on author ranking. as well, examination of professional associations such as engineering, chemistry, and more might be informative. the present work was limited to the use of web of science and google scholar, and those disciplines more pertinent to the informatics research group at one institution, and in the interest of time, allowed to stop there. in performing interdisciplinary informatics research where the establishment or creation of a workable research data environment requires considerable effort, the question also remains whether these efforts constitute author-level contributions. this single example demonstrates the need for further discussion for both interdisciplinary and informatics research. ensuring that credit due is given to those who increasingly are moving from support roles to research collaboration is an issue for our profession. acknowledgments credit to the twu interprofessional research group, members of which discussed the problem and reviewed the draft of this article. works cited 1. haak ll. persistent identifiers can improve provenance and attribution and encourage sharing of research results. inf services & use. 2014;34(1/2):93-96. doi:10.3233/isu140736 2. cronin b. hyperauthorship: a postmodern perversion or evidence of a structural shift in scholarly communication practices? j assoc inf sci technol. 2001;52(7)5:58-569. 3. schoenherr jr. social-cognitive barriers to ethical authorship. frontiers psychol. 2015;6. doi: 10.3389/fpsyg.2015.00877. original article perryman hypothesis, vol. 30, no. 1, fall/winter 2018 31 4. onwude jl, staines a, lilford rj. multiple author trend worst in medicine. bmj. 1993;306(6888):1345. 5. bennett dm, taylor dm. unethical practices in authorship of scientific papers. emerg med australas. 2003;15(3):263-270. 6. macfarlane b. (2017). tackling the challenges of academic co-authorship. <http://authorservices.taylorandfrancis.com/custom/uploads/2017/09/coauthorshipwhite-paper.pdf >. 7. macfarlane b. the ethics of multiple authorship: power, performativity and the gift economy. stud high educ. 2017;42(7):1194-1210. doi: https://doi.org/10.1080/03075079.2015.1085009. 8. flanagin a, carey la., fontanarosa pb, phillips sg, pace bp, lundberg gd, rennie d. prevalence of articles with honorary authors and ghost authors in peer-reviewed medical journals. jama, 1998;280(3):222-224. 9. smith j. gift authorship: a poisoned chalice? bmj. 1994;309(6967):1456. 10. international committee of medical journal editors (icmje). recommendations for the conduct, reporting, editing and publication of scholarly work in medical journals. <http://www.icmje.org/recommendations/browse/roles-and-responsibilities/definingthe-role-of-authors-and-contributors.html#three >. 11. cappell ms. equal authorship for equal authors: personal experience as an equal author in twenty peer-reviewed medical publications during the last three years. j med libr assoc. 2016;104(4):363–364. doi:10.3163/1536-5050.104.4.022. 12. cooper d. editor's response. j med libr assoc. 2016;104(4),365. doi: http://10.3163/1536-5050.104.4.023. 13. hagen nt. harmonic publication and citation counting: sharing authorship credit equitably – not equally, geometrically or arithmetically. scientometrics. 2010;84:785. doi: https://doi.org/10.1007/s11192-009-0129-4. 14. frandsen tf, nicolaisen j. what is in a name? credit assignment practices in different disciplines. j informetrics. 2014;4:608–617. doi: https://doi.org/10.1016/j.joi.2010.06.010. 15. kassis t. (2017). how do research faculty in the biosciences evaluate paper authorship criteria? biorxiv, 069468. doi: http://dx.doi.org/10.1101/069468doi. 16. klein jt, falk-krzesinski hj. interdisciplinary and collaborative work: framing promotion and tenure practices and policies. res policy, 2017;46(6):1055-1061. doi: https://doi.org/10.1016/j.respol.2017.03.001. 17. stvilia b, hinnant cc, wu s, worrall a, lee dj, burnett k, burnett g, kazmer m, marty pf. toward collaborator selection and determination of data ownership and publication authorship in research collaborations. libr & inform sci res. 2017;39(2):5-97. doi: https://doi.org/10.1016/j.lisr.2017.03.004. https://doi.org/10.1016/j.lisr.2017.03.004 https://doi.org/10.1016/j.lisr.2017.03.004 https://doi.org/10.1016/j.lisr.2017.03.004 original article perryman hypothesis, vol. 30, no. 1, fall/winter 2018 32 18. council of scientific editors. (2017). cse recommendations for group-author articles in scientific journals and bibliometric databases. <https://www.councilscienceeditors.org/resource-library/editorial-policies/cse policies/approved-by-the-cse-board-of-directors/cse-recommendations-for-groupauthor-articles-in-scientific-journals-and-bibliometric-databases/>. 19. american psychological association (1985a). authorship determination scorecard. <http://www.apa.org/science/leadership/students/authorship-determinationscorecard.pdf>. 20. american psychological association (1985b). authorship tie-breaker scorecard <http://www.apa.org/science/leadership/students/authorship-tie-breakerscorecard.pdf>. 21. special issue on interdisciplinary research assessment. res eval. 2006;15(1). 22. national academy of sciences, n. a. (2009). authorship and the allocation of credit, in on being a scientist: a guide to responsible conduct in research. national academies press (us):35-38. doi: https://10.17226/12192. web of science reference list (after exclusion) is available from the author. national library of medicine (us). digital collections [internet]. bethesda (md]. sudden breaking up of a consultation: weighty arguments on both sides! when doctors disagree, who shall decide. ,1722. available from: http://resource.nlm.nih.gov/101444440 http://resource.nlm.nih.gov/101444440 hypothesis: failure hypothesis vol. 32 no.1 fall/winter 2020 challenges of the two-year manuscript: engaging in reflective practice with a manuscript about reflective practice jolene m miller, mls, ahip director, assistant professor mulford health science library the university of toledo toledo, oh in january, one of my manuscripts was published in the journal of the medical library association [1]. i was thrilled that the manuscript was published by my firstchoice journal; i was not thrilled that it took me over two years to write and submit it. after reflecting on my process, i realized that if there is a concept of writing management (similar to time management), i was sorely lacking those skills. given that my research focused on reflective practice, sharing my reflective process to better understand my challenges with the two-year manuscript with the readers of hypothesis is a natural next step. natural next steps are not necessarily easy ones. “why would i want to tell anyone that i failed, let alone publish it in a journal with substantial readership? i don’t want my colleagues, and especially not my boss, to know that my project didn’t work, so no way would i want to tell the world” [2]. indeed, why would anyone do such a thing? i cannot deny that i am more than a little uncomfortable sharing the results of my reflection after all, i want you to think that i am competent and have it together all the time. that said, i hope that this column will help other librarians who encounter similar writing management problems, allowing them to intervene sooner rather than later. since i used reflective practice to review my writing process and will be talking about the process in this column, i’d like to share a bit about reflective practice before delving into my writing woes. reflective practice is intentionally doing what we all do naturally: thinking about situations (often those with surprising or disappointing outcomes) to figure out what happened. it is different from ruminating, when someone replays a negative situation in their head repeatedly but never reaches the point of figuring out what went wrong and how to make improvements. the goal of reflective practice is to identify and make changes that can improve future performance including doing something differently, thinking about something or someone differently, or updating understanding of one’s role and/or identity. reflective practice has three basic steps [3-5] though many models of reflective practice add additional steps, questions, and alternative perspectives: (1) describe the event (what happened? what did i do? what did others do? how was i feeling? what was i thinking?); (2) reflect on the event (what really happened? how might others have perceived it? what if i had done y instead of x?); and (3) identify potential changes for future situations (do i need to change a process? do i need to change how i approach hypothesis: failure hypothesis vol. 32 no.1 fall/winter 2020 a situation? do i need to change how i think about a person, a situation, or my role as a librarian?). i do not always use a model for reflection, but when i do, it is graham gibbs’ model. this model was developed as a “full structured debriefing to the stages of the experiential learning cycle” that teachers can use with their learners [6]. it works for reflective practice because reflection helps us learn from our experiences. the gibbs model expands the basic three steps described above to six steps: description, feelings, evaluation, analysis, conclusion, and action plan. this model underpins the analysis of my writing challenges. it is my hope that not only will readers learn how to be more expedient when preparing manuscripts, but that they will also see how a model of reflective practice can be applied to library work. using intentional reflection to improve professional practice can be an individual pursuit through pondering and/or writing alone or it can be communal by having conversations with a peer, coach, mentor, or a small group. i started individual reflective work about eighteen months into the research project, when it felt like i would literally never finish the manuscript. at about the same time, i began talking with trusted others: a colleague, my therapist, and my spiritual director. each had a different perspective, which led to different observations and suggestions. my colleague approached my situation from perspective of a tenured librarian who has experienced this themselves. they were able to share what worked for them. my therapist approached it from a time and stress-management perspective, with the stress management skills being more helpful than i expected. my spiritual director approached it from a values-based view, inviting me to consider why i was doing research and writing for publication in the first place in order to clarify my motivation. this column moves my reflection to a broader community. i am sharing the experience with you in hopes that it will help you. i am also hoping that you might share your observations with me in return. the gibbs reflection model in action description the first step in the gibbs model is a non-judgmental description of the situation. it always includes one’s own perceptions and understandings, but it can also include others’ feedback and external documentation such as evaluation forms. in spring of 2017, i surveyed health science librarians using an online questionnaire about their engagement in reflective practice and what they identified as the benefits of and barriers to engaging in reflective practice. it took two years to write up the results and submit the manuscript for publication. i would work regularly on it for a time, then i would set it aside for weeks or months as urgent library projects took priority. when i returned to the manuscript, it took time to remember where i was in the project and where i was headed. my reflective conversations provided additional information to consider, such as the hypothesis: failure hypothesis vol. 32 no.1 fall/winter 2020 role that stress played in the situation and how i had been treating the manuscript as one more hoop for the tenure process, rather than as an opportunity to better understand our profession. feelings one of the benefits of the gibbs model is that it explicitly asks about emotions to provide insight into a situation. remembering the feelings experienced during a situation and paying attention to those feelings while reflecting invariably enriches my results while ignoring the emotions experienced in a situation leads to missed cues that could have helped me understand what happened. on the other hand, if a situation was highly emotional, i might need to give myself some space before reflecting, so emotions do not derail the reflection. while working on the two-year manuscript, my feelings varied. there were times early on when i felt great. in hindsight, i was feeling great because i was regularly spending time in a local coffee shop working on the manuscript (very cool and academic). during that time, however, i had no clue how little progress i had to show for my time. when i finally compared my progress to the amount of time that had passed, i felt frustrated with the project and angry with myself. in this situation, feelings helped me identify the aspect of the process that needed my attention. my therapist was particularly helpful at the frustration and anger part of my reflection. with careful application of self-compassion, i was able to see beyond where i was to where i wanted to be a submitted manuscript. evaluation the first two stages of the gibbs model are purely descriptive. at the evaluation stage, judgment finally plays a role: what was good or bad about the experience, in terms of the overall experience and in terms of specific parts of the experience? in my case, while i was pleased by the positive outcome of a published manuscript, i was not happy with my performance, specifically that it took so long to prepare the manuscript. for this column, i am reflecting on what i perceived to be a negative aspect of the situation: i didn’t manage my writing well. reflective practice can also center on positive aspects and situations. analysis the analysis step of this model is where deeper exploration of the situation takes place. in previous steps, information was gathered and judgements made. this step looks at the information to determine why some things went well and why other things didn’t go so well. to quote gibbs, “what was really going on?” [6]. hypothesis: failure hypothesis vol. 32 no.1 fall/winter 2020 as i looked back over my writing process, both while i was working on the manuscript and when it was completed, i identified one primary factor contributing to the slow speed of my process: writing management. specifically, i had difficulty staying focused on and engaged with my writing in the midst of competing work and home demands. my professional situation is not unique; we all have more projects than we can reasonably fit into the workweek. i have responsibilities in a variety of areas: providing information and research support to my liaison areas; coordinating information and access services at my library; dealing with administrative issues, including shepherding the library renovations that took place while i was writing; representing my library in university libraries meetings; assisting with program accreditation; committee service; etc. on top of work demands, i had purchased a new home that required minor renovations and, at the same time, i had an old home that needed work to be ready for market. more often than i anticipated, these work and personal responsibilities led me to put my writing aside, sometimes for months at a time. this obviously delayed the completion of the manuscript, but the delay was compounded by the fact that for every break, i had to take extra time to review my work and remember where i was in the project. conclusions in this step, the gibbs model invites conclusions as a result of the previous steps. the model calls for both general conclusions (such as the challenges librarians face in terms of managing their writing) as well as those that are more personal (the challenges i personally face and how i think and work). while it is difficult to make general conclusions in this situation, i can tentatively say that given the natures of librarianship and of scholarly writing, completing manuscripts can be challenging for librarians. personally, my reflection made it obvious to me that relying solely upon internal resources, such as finding time for writing and relying upon my memory to help me pick up where i left off, was not enough to help me manage my writing in an effective way. i needed to identify one or more external tools to support my writing. in other areas of my work life, i use external tool such as a calendar for my schedule and my bullet journal [7] to keep on top of tasks and projects. perhaps something similar would help with writing management. action plan in the action plan step, gibbs invites us to identify what needs to change (if anything) based on the results of analysis and conclusions. at this point, it is important hypothesis: failure hypothesis vol. 32 no.1 fall/winter 2020 that the action plan reflects the conclusions so that selected strategies will target the actual issues and not assumed ones. what i learned through this reflection is that i cannot rely solely on what i had been doing: writing when i had time and assuming that i would remember my thought process when i paused my project. as noted in the conclusion section, i needed one or more external tools to keep me on track. i implemented two actions. first, i started a weekly “write on site” session at my library to help support student and faculty writing including my own. for an hour each week, the library offers a drop-in session for students and faculty. following the pomodoro technique® [8], we have two 25-minute work sprints, with a 5-minute break between. “write on site” provides informal accountability. participants do not report on progress toward writing goals as would happen in writing groups, but each week, we show up and write. when i show up and write, sometimes i end up with a “shitty first draft” [9] of a section, and other times, i end up with a solid piece of writing. for me, the most important benefit of this program is staying engaged with my writing. i never go more than a week without working on it (unless there is a global pandemic; in 2020, i am cutting myself some slack, and you should cut yourself some too). this regular engagement reduces wasted time reviewing the project to remember where i was and what i was thinking. another tool that i adopted near the end of the project was a research log. research logs (or diaries) have long been used in qualitative research as a way to help researchers clarify assumptions and biases that may have an impact on the research, improving transparency [10]. the goal for my research log was to capture where i was in my writing process, including what i expected my next actions to be. initially, my use of the log was haphazard, with scattered brief notes to myself about the project or ideas for future projects, with little or no follow-up. i am now using it in an intentional way to better capture my process with narrative descriptions of research ideas, summarizing the direction of the project to make it easier when i return to it, and reflecting on my research and writing process using guiding questions (table 1). as you can see from the questions, i’ve incorporated some gibbs-inspired questions in order to identify issues before they cause larger problems. i write my research log longhand because it physically slows me down, which leads to a mental slowing down. the mental slow-down allows me to think more deeply about my project. why did i decide on these two interventions? i selected “write on site” because i had been searching the web for something to help me stay focused on writing. when i found the “shut up and write!” post on the blog the thesis whisperer [11], i was intrigued by the idea of getting together to write every week. we scheduled a time and advertised it to campus. usually there were a hypothesis: failure hypothesis vol. 32 no.1 fall/winter 2020 couple of us, but before the library closed due to the pandemic in march 2020, there were three of us getting together regularly. in terms of the research log, this was something i adapted from a technique that i promote to my physician assistant students when they begin their scholarly projects. while i retained part of what i recommend for the students (jotting down thoughts and ideas during the research process), i generated the questions shown in table 1 to help me focus on the information that i felt would help my writing the most. table 1. guiding questions for my research log future plans while “write on site” and the research log have been helpful, there are additional tools i would like to explore. one is a mental shift i need to treat writing projects like any other project. i mentioned above about how i use my bullet journal to manage projects, but i have yet to apply the tool to writing in a meaningful way. also, i would like to increase my accountability to others beyond the informal accountability of “write on site”. a colleague and i tried email accountability for a couple of months, but our technique (sending each other emails once a week with a report on what we did and what our goals were for the upcoming week) was not particularly helpful for either of us. participating in a writing accountability group that met regularly might be helpful, not only report on progress on writing projects but also to discuss writing challenges and solutions and to celebrate achievements [12-14]. given that so many of us are now familiar with an assortment of videoconferencing software, the time might be right to explore an online medical/health science librarian writing accountability group using one of these platforms. meta-reflection one thing that i value about reflective practice is its iterative nature. as i write this column, and again as i make revisions, i am engaging in meta-reflection – reflecting on my reflection – and understanding more about the situation. as i reread what i wrote in the analysis section, i see that i gave away my agency: “responsibilities led me to put my writing aside.” i can’t help but wonder how much of an impact that this thought pattern had on my two-year manuscript. how much time did i not claim for writing because my brain was telling me i was too busy? i will need to pay attention in the • what did i accomplish during this work period? • where am i headed with this project? • what is my next action? • what challenges am i encountering? • what might be the reason for these challenges? • how can i overcome these challenges? • how could i prevent them in the future? hypothesis: failure hypothesis vol. 32 no.1 fall/winter 2020 future for that way of thinking, in order to explore it and make a more informed choice about how i am spending my time. conclusion despite the challenges with writing the two-year manuscript, i was pleased to discover how much i enjoy doing research and enjoy scholarly writing. engaging in reflective practice about the manuscript and implementing changes to my writing process has resulting in positive changes. since the two-year manuscript was accepted in june 2019, i have authored this column and coauthored a paper published in july 2020. my initial choice of tools seems beneficial for now. future reflection may suggest other areas and changes for improvement. librarians who are struggling with writing management may want to consider using reflection (individually or in conversation with one or more others) to uncover the underlying issues in order to select the best strategies for their situations. participating in “write on site” sessions may sound like a great idea, but for someone whose writing challenges are structuring a document or wrestling with grammar, working with a writing tutor might be more appropriate solution. once issues have been identified, it is easier to look for solutions or ask colleagues for solutions. why would someone admit their failure publicly like i did in this column, besides the obvious reason to help readers better understand writing management and reflective practice? i am reminded of those reflective conversations with my spiritual director why do research and write for publication at all? it is important for me as a health science librarian to do what i can to learn more about what we do and how we can do it better. in this way, perhaps i am moving the profession forward a little bit by helping librarians learn to apply reflective practice in and beyond their writing practice. i believe in this enough to overcome my discomfort and write this column. ps i am interested in taking the reflection even further: what questions would you ask me about my initial writing process that might help me understand the situation more clearly? is there something that i didn’t consider? if you’ve found yourself in a similar situation, what helped you? references 1. miller jm. reflective practice and health science librarians: engagement, benefits, and barriers. j med libr assoc. 2020 108(1):17-28. https://dx.doi.org/10.5195/jmla.2020.777. 2. holmes hn. coming soon! hypothesis: failure. hypothesis [internet]. 2018 fal-win [cited 2019 oct 1]; 30(1):5. <https://www.mlanet.org/d/do/13566>. 3. constable g. reflection as a catalyst in the development of personal and professional effectiveness. in: mantell a, scragg t, editors. reflective practice in social work. 5th ed. thousand oaks, ca: sage publications; 2019. p. 63-79. http://www.mlanet.org/d/do/13566 http://www.mlanet.org/d/do/13566 http://www.mlanet.org/d/do/13566 hypothesis: failure hypothesis vol. 32 no.1 fall/winter 2020 4. driscoll j. chapter 2, supported reflective learning: the essence of clinical supervision? in: john driscoll, editor. practising clinical supervision: a reflective approach for healthcare professionals. 2nd ed. philadelphia, pa: elsevier; 2007. p. 27-50. 5. jasper m. beginning reflective practice. 2nd ed. andover, hampshire; 2013. chapter 1: what is reflective practice?; p. 1-31. 6. gibbs g. learning by doing: a guide to teaching and learning methods. oxford, uk: oxford centre for staff and learning development; 1988 [cited 12 mar 2020]. <https://shop.brookes.ac.uk/product-catalogue/oxford-centre-for-stafflearningdevelopment/books-publications/ebooks/learning-by-doing-a-guide-toteaching-andlearning-methods-by-graham-gibbs-ebook>. 7. carroll r. the bullet journal method: track the past, order the present, design the future. new york, ny: portfolio/penguin; 2018. 8. cirillo f. the pomodoro technique [internet] [rev. 2019; cited 23 mar 2020]. <https://francescocirillo.com/pages/pomodoro-technique>. 9. lamott a. bird by bird: some instructions on writing and life. new york, ny: anchor; 1995. chapter 3, shitty first drafts; p. 20-6. 10. ortlipp m. keeping and using reflective journals in the qualitative research process. qualitative report [internet]. 2008 dec [cited 2020 feb 20];13(4):695705. <https://nsuworks.nova.edu/tqr/vol13/iss4/8>. 11. mewburn i. shut up and write [internet]. 2011 jun 14. [cited 2020 aug 13]. <https://thesiswhisperer.com/shut-up-and-write/> 12. houfek jf, kaiser kl, visovsky c, barry tl, nelson ae, kaiser mm, miller cl. using a writing group to promote faculty scholarship. nurse educ. 2010 janfeb;35(1):41-5. doi: https://dx.doi.org/10.1097/nne.0b013e3181c42133. 13. pennamon re, moss lj, springer si. scholarly writing groups: a source of support for early career counseling faculty and doctoral students. vistas online [internet]. 2016 [cited 2020 mar 20]:article 72. <https://www.counseling.org/knowledgecenter/vistas/by-year2/vistas-2016/docs/default-source/vistas/article_72_2016>. 14. skarupski ka, foucher kc. writing accountability groups (wags): a tool to help junior faculty members build sustainable writing habits. j fac dev [internet]. 2018 sep [cited 2020 mar 20];32(3):47-54. <https://www.hopkinsmedicine.org/fac_development/career_path/skarupski-wagsarticle.pdf>. welcome back! the phrase most suitable for our inaugural issue on a new platform is ‘deceptively simple.’ it could equally be called ‘what have we done,’ or ‘we’re not in kansas anymore.’ because we most decidedly are not. we’ve been on quite a journey of discovery as we worked with ted polley at the indiana university–purdue university indianapolis (iupui) to set up hypothesis (and note, its subtitle is now journal of the mla research caucus) in this new virtual space. thank you to the iupui university library, which provides the ojs (open journal systems) platform, and to ted, who has been there for a million questions, and whose sage advice and experience has been so crucial to this lengthy process. to members of the mla research caucus, whose enthusiastic support has been what kept us going through thick and thin and etc., thank you as well! you’re invited to check out our new about the journal page to see what we’ve got going on: readily available content and instructions, indexing information (did you know when you publish with us, you’ll be indexed in cinahl and other databases?), and our brand new, shiny submission portal. options here now include several new types of content, which we’re excited to tell you about below. perhaps most exciting is that we’ll now have the ability to provide dois for journal content and spread the word about our articles using the individual pdfs generated by publication. up to this point, finding a particular piece meant downloading the entire publication, and scanning through its pages no more! we will soon be working to import all back issues to this new platform, as well. following publication of this issue, we’ll be seeking additional participation in putting together issues, including peer review, copyedit, and promotional activities to encourage growth, increased use, and more content submissions. introducing the inaugural hypothesis: failure! column conceptualized and edited by heather holmes, this is content well overdue. years past, an outgoing mla president called for sharing our stumbles and bumbles, based on the ancient notion that those who fail to learn about the past are doomed to repeat it. failure is not failure in the hide-your-head and never emerge sense. after you get over the initial despair and frustration, it’s informative, educational, sometimes (eventually, wincingly) humorous in a word, important to our authentic sharing in an ever-evolving profession. if you have failures (and lessons learned) you’d like to share, we would love to know about it. contents of this issue the goals of the hypothesis: failure! column are described above, and in heather holmes’ introduction, but it’s hard to say too much about this column. mention to others of this new effort finds an immediate response of ‘it’s about time!’ and agreement about the importance of sharing our lessons learned. our first column submission, library renovation: from setbacks to success, is an account from elaine wells (suny optometry) of lessons learned amid a library renovation something that many of us can relate to, and learn from. http://www.ulib.iupui.edu/digitalscholarship/people/tedpolley http://www.ulib.iupui.edu/digitalscholarship/people/tedpolley https://journals.iupui.edu/index.php/hypothesis/about https://journals.iupui.edu/index.php/hypothesis/about/submissions https://doi.org/10.18060/23802 https://doi.org/10.18060/23802 https://doi.org/10.18060/23802 https://doi.org/10.18060/23629 https://doi.org/10.18060/23629 as we do every year, we recognize the wonderful research undertaken and presented by mla members at the annual meeting. the mla 2019 research caucus awards include research on: emerging roles for academic, medical librarians; burnout amongst health information professional; nursing librarianship and distance education; wellness behaviors amongst health sciences librarians; a systematic review of institutional repositories; and the use of e-books by medical students and residents. for the first time, we have recognized the creators of research-related training with the new mla 2019 research caucus research training quality awards. offerings were evaluated for quality of content, topical currency, and fit to the audience. we awarded three trainings this year one on the use of metrics in research evaluation, another on community health assessment tools, and a third on the involvement of libraries/librarians in offering research data services to their communities. angela spencer (saint louis university), elizabeth mamo (rochester regional health), and brooke billman (college of american pathologists) have authored the benchmarking study of hospital libraries, updating our previous understanding of hospital closures in a volatile healthcare environment, and including information about staffing, services, and budgets. research roadmap part 2: disseminating your research, authored by katherine akers (wayne state university), kris alpi (oregon health & science university), jon eldredge (university of new mexico), heather holmes (medical university of south carolina), kim powell (emory university), and margaret hoogland (university of toledo) concludes this 2-part publication with advice about sharing your work at conferences and the process of research writing and submitting that work for publication. it’s been a year of quality training efforts! margaret hoogland shared her thoughts about the imls-funded research training institute experience in a year in review: mla rti fellow reflection. look for reflections authored by other rti fellows in future issues. in summary as editors of hypothesis: the journal of mla’s research caucus, we hope you enjoy this issue and join us on this journey: submit, read, and share what you find in our pages. erin d. foster, msls co-editor, hypothesis data services librarian ruth lilly medical library indiana university school of medicine indianapolis, in carol l. perryman, phd co-editor, hypothesis associate professor school of library and information studies texas woman’s university denton, tx https://doi.org/10.18060/23672 https://doi.org/10.18060/23827 https://doi.org/10.18060/23805 https://doi.org/10.18060/23805 https://doi.org/10.18060/23693 https://doi.org/10.18060/23630 https://doi.org/10.18060/23630 research section news erin d. foster hypothesis, vol. 29, no. 1, spring/summer 2017 20 mla 2017 debrief the librarian’s role in reproducibility of research symposium held saturday, may 27th from 1-5pm this symposium aimed to educate attendees about the important roles for librarians and information professionals at their home institutions to raise awareness about and support research reproducibility. attendees learned more about reproducibility from keynote speaker shona kirtley, senior research information specialist for the equator network at the centre for statistics in medicine, university of oxford, as well as a panel of librarians and information professionals who are involved in enhancing reproducibility. at the conclusion of the event, attendees better understood the basics of reproducibility, were empowered to create strategies for how they might find partners, and get involved in reproducibility of research at their individual institutions. more information – including speaker slides, handouts, and additional resources can be found on the symposium website: http://mlasymposium.libguides.com/2017 #meddatalibs special content session held sunday, may 28th from 3-4:25pm the research section (rs) and the medical informatics section (mis) co-hosted a book group discussion on the new mla publication: the medical library association guide to data management for librarians, edited by lisa m. federer. michelle bass, rs chair-elect, teamed up with mis chair-elect, patricia gogniat, to moderate this session. they facilitated an engaging conversation of the book and its contents. overall, it was a successful showing for the first book group discussion held at an mla conference! additionally, those interested in following discussions about medical data librarianship online, should follow and make use of the twitter hashtag #meddatalibs! research section business meeting held sunday, may 29th from 7-8:55am for meeting notes, please see the mla research section’s webpage on mlanet.org. http://mlasymposium.libguides.com/2017 http://www.mlanet.org/p/cm/ld/fid=503 research section news erin d. foster hypothesis, vol. 29, no. 1, spring/summer 2017 21 research section member activities @ mla 2017 the information for these papers and posters was gathered as part of a pre-conference survey of mla 2017 research section activities sent out by chair, michelle b. bass. this is not a complete listing of research section member contributions to mla 2017. contributed papers: presenter(s) presentation title session akers, katherine amos, kathleen the value of case studies in evidence-based librarianship: an examination of the journal of the medical library association daring to dream: facilitating scientific communication akers, katherine increasing clinical scholarship: promoting the publication of medical case reports daring to dream: facilitating scientific communication henderson, margaret providing support for an interdisciplinary research group with a multidisciplinary informationist team: is it effective? librarians as innovative collaborators: doing what it takes to form new partnerships vardell, emily health insurance information needs: how librarians can help dreaming of a healthier community: librarians doing research yatcilla, jane hammering out the details: what can an online bibliometrics engine tell us about research in animal-assisted therapy? bibliometrics in action research section news erin d. foster hypothesis, vol. 29, no. 1, spring/summer 2017 22 foster, erin d. enhancing accessibility of rare disease literature for researchers and patients technology dreamin': new applications for current and emerging technologies henderson, margaret adding to the evidence: a survey of faculty research data management practices and needs data librarians: daring to move beyond traditional roles ascher, marie t. martinez, ophelia moy, fred dare your students to debate: an innovative and collaborative approach to teaching evidence-based medicine daring to design: librarian roles in curriculum foster, erin d. hosting a science hack day...and you can too! lightning talks session 3 -- poster sessions: presenter(s) poster title session galati, marianne varman, beatriz brown, rebecca the publisher and editor regretfully retract this publication session 1 hoogland, margaret clinical apps: gateway to accessing health information session 1 rosenzweig, merle the anatomage table: an innovative approach to anatomy education session 1 johnson, emily rural information connection: an ipad mini lending program to rural student physicians session 1 brigham, tara coloring your "art" out: outcomes of offering coloring materials in targeted hospital staff areas session 1 research section news erin d. foster hypothesis, vol. 29, no. 1, spring/summer 2017 23 sakmar, kristen bullers, krystal hanson, ardis howard, allison orriola, john polo, randall dare you look the predator in the eye? how librarians are addressing the nightmare of questionable publication practices session 1 fitterling, lori medlib q-bank anyone? creating a question bank of medical informatics information literacy test questions for librarians session 3 powell, kimberly searching by grant number: analysis of web of sciences and pubmed search results session 3 spencer, angela j. eldredge, jonathan d. roles for librarians in systematic reviews over time session 3 fitterling, lori voices in the choir: librarians helping to move osteopathic research forward session 4 rosenzweig, merle san francisco plague of 1900-1904: economics, politics, and racism session 4 grabowsky, adelia using reflection and peer discussion to increase engagement of master’s in nursing students during library orientation session 4 -- original article theis-mahon, menard, schmillen, and stark hypothesis, vol. 30, no. 1, fall/winter 2018 8 defining a project proposal to enhance the medical library association’s annual meeting through session-level assessment: the exploration of the 2017-2018 rising star cohort nicole r theis-mahon, mlis, ahip liaison librarian to the school of dentistry and health sciences libraries collections coordinator health sciences library university of minnesota minneapolis, mn theis025@umn.edu laura m menard, mls, ahip assistant director for medical education and access services ruth lilly medical library indiana university school of medicine indianapolis, in lmenard@iu.edu hanna schmillen, mlis, ahip subject librarian for health sciences ohio university libraries athens, oh schmille@ohio.edu rachel k stark, ms, ahip health sciences librarian university library california state university, sacramento sacramento, ca stark@csus.edu objectives: associations and organizations rely on feedback from membership to assess conferences, programs, and meetings. the medical library association (mla) utilizes postconference assessment to get an overall evaluation of the meeting. while this informs future meeting planning, it does not provide targeted assessment data about the perceived quality and relevance of sessions, papers, or posters. incorporating session-level, just-in-time feedback would further engage meeting attendees and ensure relevance of the meeting to the membership. methods: the 2017-2018 mla rising star cohort investigated the interest in and use of sessionlevel, just-in-time feedback at conferences of seven peer associations. a five-question survey to original article theis-mahon, menard, schmillen, and stark hypothesis, vol. 30, no. 1, fall/winter 2018 9 gauge mla member interest in session-level feedback was distributed in february 2018. the survey was only available to current mla members and advertised on the mla blog, distributed to section and sig, state, and select mla chapter lists. live polling was also conducted at the may 22, 2018, mla rising star project proposal presentation. results: the cohort received responses from five peer associations and only three are using some form of session-level, just-in-time assessment at their conferences. the february 2018 mla membership survey yielded 157 responses. 94% of respondents (n=147) had attended a mla meeting and 72% of respondents agreed that they would find session-level assessment valuable. respondents indicated that they would be interested in receiving feedback from attendees about the application of their session’s content, whether attendees learned something new, and if their session met expectations. of attendees at this may 22, 2018, project proposal presentation, 97% agreed that they would value the opportunity to provide session-level, and 91% indicated that as a presenter would find attendee feedback useful. conclusion: the investigation by the 2017-2018 mla rising star cohort indicated an interest in session-level, just-in-time feedback for mla’s annual meetings. introduction for the past seven years, the medical library association’s (mla) rising stars program has provided members with the opportunity to familiarize themselves with the association, develop leadership skills, and contribute their insights and ideas. usually this has manifested in the form of an individual yearlong project that investigated a need in the association. the project’s ultimate goal was to have each member of the cohort contribute to mla in a tangible way. the format for the 2017-2018 mla rising star project was a departure from previous years’ projects. this new approach was based on feedback from previous cohorts, who expressed a desire to work more closely and collaboratively with one another. the change sought to address and encourage collaboration, cohort development, and investment in mla. instead of being presented with a question to investigate the 2017-2018 cohort was tasked with identifying and creating a project proposal that was both forward thinking and would support mla’s current and future needs. the intent of this approach was that if mla, or any group within the association, decided to implement the project, it would be relevant and could be easily adapted to the association’s needs and values. the cohort initially drafted two project proposals: one that addressed assessing diversity and inclusion, and a second to investigate the mla competencies for lifelong learning and professional success. after receiving feedback from the mla rising star faculty and other key stakeholders, one proposal was selected to expand upon. the final proposal, “just-in-time assessment at a national conference: increasing member engagement and assisting with professional development,” was an exploration of integrating an electronic assessment option for session-level content at the mla annual meeting. for this proposal, original article theis-mahon, menard, schmillen, and stark hypothesis, vol. 30, no. 1, fall/winter 2018 10 the cohort defined just-in-time assessment as an approach that would allow mla annual meeting attendees to provide feedback on session-level presentations, such as posters or papers, throughout the duration of the conference, at their convenience, and while the content of the presentations was fresh. the intent of the proposal was to build upon existing member involvement with the association and offer a mechanism to increase feedback and participation with the annual meeting, its presenters, and content. a secondary goal of this project was to identify avenues for membership to engage with the mla competencies for lifelong learning and professional success. the cohort proposed that options for constructive, session-level feedback should be available through several platforms including the mla website, the mla annual meeting app, kiosks at the meeting, or through other means. literature review there is little published in the library and information science (lis) field specific to session‐level, just‐in‐time feedback at lis conferences and meetings. the closest example was the 2010 evidence based scholarly communication conference in albuquerque, new mexico, which piloted an approach to conduct real-time peer review by conference attendees on presentations [1]. session‐level assessment is present at conferences and annual meetings outside of the lis fields, including medical professional conferences [2]. there is a rich corpus of literature on assessment and standard approaches to assessing the qualities of meetings or conferences. many assessment approaches include a post‐ meeting evaluation distributed to attendees. while an overall evaluation of the conference is helpful to the association, it does not always capture the quality or value of individual sessions, nor does it provide a means where individuals improve on their practice. by implementing this proposal, mla would be in the forefront of library associations when it comes to peer review and assessment at conferences. the proposal would also align mla with other professional associations outside of lis, as demonstrated in the published literature. information professionals and librarians value lifelong learning and should embrace the skills and practice of receiving feedback [3]. this is relevant in our approaches to information literacy instruction, the competencies that librarians utilize, and the desire to improve as information professionals. the gap in the lis literature about session-level assessment at conferences reveals that a proposed just-in-time assessment mechanism would be an innovative way to benefit the association, increase professional development, provide an opportunity to engage with mla’s new professional competencies for lifelong learning and professional success, and improve engagement at the mla annual meeting. original article theis-mahon, menard, schmillen, and stark hypothesis, vol. 30, no. 1, fall/winter 2018 11 methods to better understand assessment in the lis and other comparative fields, the cohort identified a set of peer associations to determine what they were doing in respect to justin-time, session-level assessment at their conferences and annual meetings. the cohort, in consultation with their mentors, compiled a list of seven associations that were peers to mla. the final list of peer associations included the special library association (sla), chicago association of law libraries, association of college & research libraries (acrl), european association of health information and libraries (eahil), american medical informatics association (amia), american library association (ala), and the charleston conference. the cohort developed six questions to share with contacts at each association. the questions below were sent via email: 1. does your organization conduct session-level assessment at annual conferences/meetings? 2. if yes, how does your organization assess sessions? a. paper form after the session/presentation b. webform that is shared with attendees/participants c. session-level assessment integrated in the overall conference assessment d. other 3. what questions do you ask to attendees? 4. what is the average completion rate of session-level assessment? 5. how do presenters and attendees perceive session-level assessment? 6. how does your organization use this feedback in planning future meetings? what have you learned from this feedback? the cohort developed a short, five-question survey for the mla membership to gauge interest in the notion of session-level, just-in-time assessment (appendix). the first question asked if respondents had attended a mla in person or online. responses were collapsed into “yes” or “no.” the following questions offered a likert scale (strongly agree, agree, neutral, disagree, strongly disagree). in our analysis, the categories “strongly agree” and “agree” were collapsed into “agree” and “neutral”, “disagree” and “strongly disagree” were collapsed as “other.” an open-ended question at the end of the survey provided an option for respondents to share additional thoughts about the ability to provide sessionlevel feedback on paper and panel presentations, lightning rounds, posters, and special content sessions. the survey was available to the mla membership in late february 2018 and was open for two weeks. it was administered through mlanet and advertised on the mla blog, section and special interest group (sig) listservs, state listservs, and to some mla chapter listservs. participant responses were anonymous. original article theis-mahon, menard, schmillen, and stark hypothesis, vol. 30, no. 1, fall/winter 2018 12 responses to the mla membership survey were analyzed and tabulated for each question. none of the questions were required and respondents could choose which questions to respond to about their perceived value in providing and receiving feedback. descriptive statistics were used to calculate frequencies. comments to the open-ended question were analyzed and a thematic analysis was conducted to identify major themes. this was done by author one [ntm] identifying major themes and the second author [lm] reviewing these themes. results assessment approaches by peer associations to assess the viability of just-in-time, session-level feedback the rising star cohort investigated feedback approaches of peer associations to determine 1) if associations were utilizing this assessment strategy at their annual meetings and conferences and 2) if it was successful. seven associations were contacted about their use of session-level feedback. the cohort received responses from five associations and only two, acrl and sla, reported using mobile technology for just-in-time feedback at their annual meetings (table 1). a third association, eahil, offered an optional paper form for feedback on oral presentations and keynote speakers. amia and the charleston conference did not respond. association just-in-time assessment session-level assessment mobile app interested acrl x x x x sla x x x x eahil x x (paper format) x chicago association of law libraries x table 1. assessment of peer associations and use of session-level assessment acrl encourages conference session attendees to rate sessions on a 1-5 scale and add comments to a text box within their mobile conference app. overall, about 30% of acrl attendees utilize the session-level feedback mechanism. the feedback is primarily used by the acrl conference planning committee and sometimes presentation specific comments are shared with participants. original article theis-mahon, menard, schmillen, and stark hypothesis, vol. 30, no. 1, fall/winter 2018 13 sla planned to implement session-level feedback at their june 2018 conference. the association is interested in gathering data for in-depth assessment of individual sessions, presenters, and posters within their mobile conference app. eahil and the chicago association of law libraries expressed interest in just-in-time, session-level feedback for their conferences and meetings. however, neither association currently has plans or a timeline to implement this form of assessment. mla membership surveys the february 2018, mla membership survey yielded 157 responses and a response rate of 5%. the cohort did not have access to the mla membership list and needed to rely on distribution lists that they had access to. survey results revealed that 94% (n=147) of respondents have attended an mla annual meeting in person, online, or both. this number is higher than the 2017 mla executive director’s report, which stated that 38% of mla members attend the 2017 annual meeting [4]. although there is a low response rate to the mla membership survey, these may be viewed as representative of mla members who participate in an annual meeting. when survey respondents were asked if they would find it valuable to “provide immediate, presentation-level feedback (on for example, paper and panel presentation, lightning rounds, posters, and special content sessions)” 72% (n=114) of respondents agreed or strongly agreed (table 2). value n % strongly disagree 3 2% disagree 4 3% neutral 32 20% agree 74 47% strongly agree 40 25% don’t know/not applicable 4 3% table 2. value in providing feedback when asked if, as a presenter, there would be value in “hav[ing] the option of receiving presentation-level feedback (on, for example, paper and panel presentations, lightning rounds, posters, and special content sessions)” 82% (n= 128) of respondents either agreed or strongly agreed (table 3). original article theis-mahon, menard, schmillen, and stark hypothesis, vol. 30, no. 1, fall/winter 2018 14 value n % strongly disagree 1 1% disagree 0 0% neutral 16 1% agree 69 44% strongly agree 59 38% don’t know / not applicable 12 8% table 3. value in receiving feedback survey respondents were asked to select from a list of feedback options that they would desire as a session-level presenter. respondents could check more than one answer. possible options for session-level feedback included: whether attendees learned something from my session, how participants think this may apply what they learned, and the level of attendee engagement with the session. participants had the option of selecting multiple answers for this question. respondents were primarily interested in knowing “whether participants learned something from my session” or “how participants think they might apply what they learned at my session” (table 4). type of feedback desired n % none 5 1% other (please describe) 21 4% met participant expectations 123 25% participate engagement in session 74 15% application of session content 130 27% learned something 132 27% table 4. types of presenter feedback desired* original article theis-mahon, menard, schmillen, and stark hypothesis, vol. 30, no. 1, fall/winter 2018 15 participants were given the option to share their thoughts about the ability to provide feedback on paper and panel presentations, lightning rounds, posters, and special content sessions at mla’s annual meeting in an open-ended question. one primary category emerged from these comments: feedback for personal or professional use and improvement. other comments were about the mla annual meeting and not specific to this project proposal. respondents welcomed the notion of feedback to improve their professional practice and felt that these comments would improve their mla meeting submissions. they felt that feedback would allow for transparency between the presenter, mla, and annual meeting attendees. one participant summed this up nicely: “we can't improve without the ability to receive honest, timely feedback.” the 2017-2018 cohort also conducted a live audience poll at the may 22, 2018, project proposal presentation. attendees responded to two questions using their laptops or mobile devices and participant responses were anonymous. of attendees at the presentation session, 97% (n= 31) “agreed” or “strongly agreed” that “as an attendee of the mla annual meeting, i would find it valuable to be able to provide immediate, presentation-level feedback (on, for example, paper and panel presentations, lightning rounds, posters, and special content sessions)”(figure 1). when asked, “as a presenter at the mla annual meeting, i would find it valuable to have the option of receiving presentation-level feedback (on, for example, paper and panel presentations, lightning rounds, posters, and special content sessions)”, 91% (n=33) “agreed” or “strongly agree” that they would want the option (figure 1). figure 1. value of session-level feedback (results of an audience poll) original article theis-mahon, menard, schmillen, and stark hypothesis, vol. 30, no. 1, fall/winter 2018 16 discussion constructive feedback and assessment is commonplace in education and professional development due to the desire to improve and evolve. feedback should not be avoided, but rather encouraged for personal and professional development. it offers external views, recognizes or interprets ways to improve past performances for the future, shows genuine interest and appreciation of work, and gives encouragement, affirmation, and support to build confidence and competence [5]. individuals providing the feedback and those receiving it enter into a partnership with the overall goal to improve and further develop a program or outcome [5], meaning peer-to-peer feedback is mutually beneficial. constructive feedback and peer assessment has been used in medical education, including hospitals, to assess practitioner performance and skills, communication skills, and the physician patient relationships [6]. it is argued that peer assessment “can be valuable as a formative assessment method” [7] and that self-assessment allows individuals to “focus on aspects of their work that seem to be problematic, forming a more objective lens for selfassessment (and promoting performance awareness)” [8]. library and information science professionals value self-assessment and constructive feedback as it provides an opportunity for them to develop, improve, and assess their impact. they "look to professional partners, broad contexts, and campus priorities with which to engage" [9]. furthermore, “in a profession focused on lifelong learning, the skill of accepting [and giving] feedback should be an area of continual improvement” [3]. lis professionals already embrace the desire to refine professional skills and develop new programs, initiatives, or services based on the needs, expectations, and feedback from their users and institutions. mla members indicated a desire to both provide and receive feedback to improve as professionals and to assist peers in improving their research, presentations, and skillset. mla annual meeting presenters recognize the potential of session-level feedback to assess whether attendees learned something and are interested in applying programs or services. presenters also recognize the opportunity to assess the overall impact of their presentations. receiving peer feedback from other lis professionals should only further develop this culture of continual improvement and context for demonstrating impact. for this project, the cohort investigated the use of session-level assessment at conferences. the use of mobile technology is key to this type of assessment. mobile apps are increasingly used to enhance conferences and educational settings [10] and have the potential for real-time engagement [2]. the use of this technological platform makes sense since nearly nine in ten americans own a smartphone and rely on these devices to access or interact with content [11]. mci usa, a telecommunications company that supports vent original article theis-mahon, menard, schmillen, and stark hypothesis, vol. 30, no. 1, fall/winter 2018 17 needs, offers functionality to engage with meeting attendees through surveys and questions within a mobile app. mla, like other professional associations, creates and manages both a conference website and mobile app for annual meeting attendees to learn about session offerings, manage schedules, and interact with conference content [12]. currently, mla only assesses its annual meeting through a survey link that is shared with attendees after the meeting and focuses on the event overall. building an assessment into the annual meeting app and website has the potential to gather feedback on a specific portion, or subset, of the annual meeting and increase overall feedback. the integration of an assessment function has the potential to maintain attendee engagement and participation with conference content and provide an opportunity for the association to gather feedback as attendees participate in events or content. the 2017-2018 rising stars project proposal presents a novel approach for gathering feedback at library and information conferences and meetings. the investigation of peer associations showed that while acrl, sla, and eahil conduct just-time-assessment there is range of questions asked, methods, and approaches employed in session-level assessment. the integration of a just-in-time feedback mechanism into the mla mobile app aligns with the trends of peer institutions and is a functionality of interest to the membership. further data is needed to identify the optimal integration of session-level, just-in-time feedback at the mla annual meeting. limitations although the cohort tried to be as comprehensive as possible in exploring evidence supporting this proposal, some limitations exist. lack of relevant information in the literature, a low survey response rate from the mla membership, and potential challenges in implementation of the proposal are all issues that were considered throughout the process. there is little-to-no direct evidence in the lis literature pertaining to online or app-based session-level assessment at conferences. the cohort therefore broadened the parameters of the literature search to include conferences outside of the lis field, as well as the value in peer-to-peer feedback. this was valuable in shaping the proposal; however, it may lack specificity and direct applicability to the proposed assessment approach at mla annual meetings. the cohort is cognizant of this limitation and views the possible implementation of this proposal as an opportunity for mla to contribute to the scholarly conversation by publishing on assessment efforts. the mla member survey that was developed and administered by the cohort yielded valuable information on member perceptions of the proposed project. however, the original article theis-mahon, menard, schmillen, and stark hypothesis, vol. 30, no. 1, fall/winter 2018 18 survey was not be shared directly with the mla membership and achieved a response rate of 5% (n=157). since the survey was advertised on the mla blog and shared on lists that the cohort had access to the results represent a self-selected sample [13]. as the project moves forward, the cohort recommends maintaining open communication with membership for feedback and suggestions. as this year’s cohort was tasked with developing and presenting a proposal, they discussed but did not directly address limitations of the proposed feedback mechanism. given the fact that many sessions include multiple presenters, the proposed feedback system may need to be modified for ease of use and clear interpretation of individual feedback results. each presenter may or may not be comfortable receiving feedback, and therefore the cohort recommends that the mechanism be opt-in for presenters. this project proposal offers an additional approach to gather feedback from session-level content and engage members at the mla annual meeting. conclusion the 2017-2018 rising star cohort believes that this project would benefit mla as an association, its members as stakeholders, and participants in the annual meeting as professionals. the cohort is currently exploring options for piloting just-in-time, sessionlevel assessment. the project proposal mirrors assessment approaches used by other professional associations at events and meetings. the survey of the mla membership and feedback received at the mla rising star presentation indicates there is interest in pursuing the proposal. the potential to leverage feedback from the membership using existing mla platforms is significant, and could have far-reaching impacts for engagement with and interest in mla annual meeting offerings. in the literature and in conversations with peer associations, the cohort found that similar approaches to assessment might improve professional development, competence, and engagement within an association or professional associations. works cited 1. eldredge jd, phillips he, kroth pj. real-time peer review: an innovative feature to an evidence-based practice conference. med ref serv q. 2013 32(4):412-23. doi:http://dx.doi.org/ 10.1080/02763869.2013.837690. 2. mci usa. engaging and activating event attendees: a comprehensive guide to executing successful meetings and events in the digital age [internet]. [cited 28 jul 2018]. availabe from http://association.mci-group.com/wp-content/uploads/2016/08/engagingactivating-event-attendee.pdf 3. dorney e. zen and the art of constructive criticism [internet]. in the library with the lead pipe. 2012. [cited 28 jul 2018]. available from: original article theis-mahon, menard, schmillen, and stark hypothesis, vol. 30, no. 1, fall/winter 2018 19 http://www.inthelibrarywiththeleadpipe.org/2012/zen-and-the-art-of-constructivecriticism/. 4. baliozian k. mla executive director’s report. presented at: mla ‘17, the 117th medical library association annual meeting; seattle, wa; may 30, 2017. 5. petress k. constructive criticism a tool for improvement. coll stud j. 2000 34(3):475478. 6. falchikov n. peer feedback marking: developing peer assessment. innov educ train int. 2006 32(2):175-87. 7. dochy f, segers m, sluijsmans d. the use of self-, peer and co-assessment in higher education: a review. stud high ed 1999 24(3):331-50. 8. reinholz d. the assessment cycle: a model for learning through peer assessment. assess eval high education. 2015 41(2):301-15. 9. hinchliffe lj. professional development for assessment: lessons from reflective practice. j acad libr. 2015 41(6):850-2. 10. wittich cm, wang at, fiala ja, mauck kf, mandrekar jn, ratelle jt, beckman tj. measuring participants' attitudes toward mobile device conference applications in continuing medical education: validation of an instrument. j contin educ health prof. 2016 winter;36(1):69-73. doi: http://dx.doi.org/ 10.1097/ceh.0000000000000031. 11. smith a. record shares of americans now own smartphones, have home broadband [internet]. washington, d.c.: pew research center. jan 2017. [cited 28 jul 2018]. available from: http://www.pewresearch.org/fact-tank/2017/01/12/evolution-of-technology/ 12. medical library association. download the mla ’18 app. 2018. [cited 28 jul 2018]. available from: http://www.mlanet.org/p/cm/ld/fid=1414 13. zhang y. using the internet for survey research: a case study. j am soc inf sci. 2000 51(1):57-68. doi: http://dx.doi.org/10.1002/(sici)1097-4571(2000)51:1<57::aidasi9>3.0.co;2-w appendix questions shared with mla members would you like to be able to evaluate the sessions that you attend at mla annual meetings? would you like to get feedback on your paper, poster, or lightning round talk? research shows that feedback is an essential part of the process of becoming a more effective and engaging presenter. however, there currently isn’t a way for presenters at mla annual meetings to receive feedback on their papers, posters, lightning talks, panel presentations, or special content sessions. the mla rising stars cohort has created a survey to find out your thoughts on adding a session-level component to mla annual meeting feedback. we plan to use the results of this survey to inform a proposal to the mla board on how to improve the mla annual meeting experience for attendees and presenters. http://www.pewresearch.org/fact-tank/2017/01/12/evolution-of-technology/ http://www.mlanet.org/p/cm/ld/fid=1414 http://dx.doi.org/10.3163/1536-5050.96.4.011 https://doi-org.ezp2.lib.umn.edu/10.1002/(sici)1097-4571(2000)51:1%3c57::aid-asi9%3e3.0.co;2-w https://doi-org.ezp2.lib.umn.edu/10.1002/(sici)1097-4571(2000)51:1%3c57::aid-asi9%3e3.0.co;2-w original article theis-mahon, menard, schmillen, and stark hypothesis, vol. 30, no. 1, fall/winter 2018 20 this five-question survey will take only a couple of minutes to complete. all responses are anonymous. the information we collect from the survey will be shared in a future post and may be used by mla if our proposal is adopted. 1. i have attended an mla annual meeting in person or online ● yes: in-person ● yes: online ● yes: both in-person and online ● no: neither 2. as an attendee at the mla annual meeting, i would find it valuable to be able to provide immediate, presentation-level feedback (on, for example, paper and panel presentations, lightning rounds, posters, and special content session) strongly disagree disagree neutral agree strongly agree 3. as a presenter at the mla annual meeting, i would find it valuable to have the option of receiving presentation-level feedback (on, for example, paper and panel presentations, lightning rounds, posters, and special content sessions): strongly disagree disagree neutral agree strongly agree 4. as a presenter at the mla annual meeting, i would find value in receiving the following types of feedback: (select all that apply) ● whether participants learned from my session ● how participants think they might apply what they learned at my session ● what the level of participant engagement with my session was ● to what extent my session met participant expectations ● other:__________________ ● none of the above 5. is there anything else you want to share with us about the ability to provide feedback on paper and panel presentations, lightning rounds, posters, and special content sessions? -- volume 29, no 2 fall/winter 2017 hypothesis the journal of the research section of the medical library association hypothesis, vol. 29, no. 2, fall/winter 2017 1 volume 29, no 2 fall/winter 2017 hypothesis the journal of the research section of the medical library association editorial what’s our story? ………………………..…………………... 4 carol l perryman and erin d foster guest editorial welcome from research section chair .………………..... 6 michelle b bass original articles rising stars research projects 2016-2017: action research to improve mla’s communities ……….…... 8 gregg a stevens, tony nguyen, rachel c lerner, and phill jo art show paints a collaborative picture: increasing engagement in a pharmacy and health sciences library ……………………………………………………...... 21 rachel stark, mickel paris, ed rogan, mary-kate finnegan dopkins, and cassie etter research section spotlight “librarian’s role in reproducibility of research” 2017 symposium: research section stipend winner reflection ........................................................................... 32 kimberly r powell mla 2017 research section research awards ……...... 39 linear regression from wikimedia commons, the free media repository https://commons.wikimedia.org/wiki /file:linear_regression.svg hypothesis (issn 1093-5665) is the official journal of the research section of the medical library association. hypothesis is indexed in the cumulative index to nursing and allied health literature™ (cinahl). hypothesis is available online at http://www.mlanet.org/p/cm/ld/fid=503. https://commons.wikimedia.org/wiki/file:linear_regression.svg https://commons.wikimedia.org/wiki/file:linear_regression.svg http://www.mlanet.org/p/cm/ld/fid=503 hypothesis, vol. 29, no. 2, fall/winter 2017 2 editors erin d. foster, msls data services librarian ruth lilly medical library indiana university school of medicine edfoster10@gmail.com carol l. perryman, phd associate professor school of library and information studies texas woman’s university cperryman@twu.edu editorial board abby adamczyk, mlis liaison librarian for medicine drexel university libraries ala99@drexel.edu krystal bullers, mls emerging technologies librarian, pharmacy liaison shimberg health sciences library university of south florida health kbullers@health.usf.edu ayaba logan, mlis, mph research and education informationist musc libraries medical university of south carolina (musc) loganay@musc.edu ex officio michelle b. bass, phd, msi, ahip research section chair, section council alternate representative population research librarian lane medical library stanford university school of medicine mbbass@stanford.edu brooke l. billman, mlis research section chair-elect; program chair medical librarian college of american pathologists bbillma@cap.org marie ascher, ms, mph, ahip research section immediate past chair nominating committee chair section council rep lillian hetrick huber endowed director health sciences library new york medical college marie_ascher@nymc.edu margaret hoogland, mls, ahip research section community manager assistant professor, clinical medical librarian health science library mulford health sciences library university of toledo mahoogla@gmail.com hypothesis, vol. 29, no. 2, fall/winter 2017 3 about the publication hypothesis (issn 1093-5665) is the official journal of the research section of the medical library association. the journal is open access, and is indexed in cinahl (cumulative index to nursing and allied health literature), liss (library and information science source), and lista (library, information science & technology abstracts). hypothesis is a peer-reviewed journal whose purpose is to provide research section and other mla members with an outlet for research and research-related content: works in progress, announcements, and other related content. submission deadlines hypothesis is published online twice per calendar year: spring/summer with a submission deadline of april 15 and fall/winter, with a submission deadline of july 15. submission criteria to submit an article or other item for consideration, contributors should ensure that the following criteria are met. we welcome inquiries about possible submissions. reference style hypothesis follows the mla style manual for references. consult citing medicine: the nlm style guide for authors, editors, and publishers, 2nd edition, for further information and examples. hypothesis content types • research papers -peer reviewed • programs and/or projects (non-research) -peer reviewed • other (e.g., announcements, letters to the editors) – reviewed by editor(s) peer review is based on how well submitted work follow evidence-based guidelines for research, which will be made available on the research section webpage. if you are a new researcher, you are encouraged to contact a research mentor through mla (see professional development / find or become a mentor, or http://www.mlanet.org/p/cm/ld/fid=45). copyright all works in hypothesis are licensed under a creative commons attribution 4.0 international license. authors own copyright of their articles appearing in hypothesis. readers may copy articles without permission of the copyright owner(s), as long as the author(s) and the medical library association are acknowledged in the copy and the copy is used for educational, not-forprofit purposes. for any other use of articles, please contact the copyright owner(s). http://www.mlanet.org/p/cm/ld/fid=256 http://www.nlm.nih.gov/citingmedicine/ http://www.nlm.nih.gov/citingmedicine/ http://www.mlanet.org/p/cm/ld/fid=503 http://www.mlanet.org/p/cm/ld/fid=45 https://creativecommons.org/licenses/by/4.0/ https://creativecommons.org/licenses/by/4.0/ editorial perryman and foster hypothesis, vol. 29, no. 2, fall/winter 2017 4 what’s our story? and how can you help tell it? carol l. perryman, phd co-editor, hypothesis associate professor school of library and information studies texas woman’s university denton, tx erin d. foster, msls co-editor, hypothesis data services librarian ruth lilly medical library indiana university school of medicine indianapolis, in perhaps the most important goal of a publication is to clearly state its purpose. it’s obvious in our journal’s subtitle, but hypothesis exists to provide research section and other mla members with an outlet for research and research-related content, including works in progress, announcements, and other related content. are we a younger sibling to the journal of the medical library association (jmla)? perhaps so, but what does that mean for mla members doing research? we hope it means that you understand this is your publication, here to enhance our shared aims through conversations about all aspects of research. we’re here for research articles, but also for poster abstracts, articles about doing research (including failures, explorations of research methods, continuing education about research… and so much more). we’re here to celebrate successes and share funding opportunities, too. consider this your invitation to join the hypothesis conversation and help shape our future direction! to help with that, we are thrilled to announce a new submission process and platform, beginning immediately. you can now submit articles and other content using the socious platform; this system is in beta so we appreciate your help (and patience!) testing it with your submissions: http://www.mlanet.org/e/sx/eid=70. we’ve also set up an evidence-based ‘checklist’ for authors to use that should help you review your work before you upload it, which will be available soon on the research section webpage. in addition to the new submission system, you may have noticed our updated publication information on page 3 of this issue. it http://www.mlanet.org/e/sx/eid=70 http://www.mlanet.org/p/cm/ld/fid=503 editorial perryman and foster hypothesis, vol. 29, no. 2, fall/winter 2017 5 includes an expanded list of where hypothesis is indexed, new submission deadlines for our biannual issues, our copyright statement, and more details on the types of content we accept plus the level of review they receive. this information will soon be listed on both public facing and members only pages of the research section’s webpages on the mla website. please do not hesitate to get in touch with questions about possible content or other issues at our new hypothesis email address, mlarshypothesis@gmail.com.* so what’s our story this issue? our content kicks off with a welcome from michelle bass, current chair of the research section, in which she has a specific ask for research section members who have served in section leadership positions over the years. our articles include project work by mla’s rising star 2016-2017 cohort to, “provide mla leadership and the communities task force with actionable recommendations to support the changing needs of its members” as those needs relate to mla section & sig memberships. this is followed by an article discussing the work of the university of the pacific’s rite aid information commons to host an in-house art show made possible through partnership with their institution’s thomas j. long school of pharmacy and health sciences. from there, we move to research section news with kimberly r. powell – recipient of a research section stipend to attend the 2017 “librarian’s role in reproducibility of research” symposium – giving a wonderful summary of the symposium and highlighting key takeaways on reproducibility and transparency. we round out with a listing of reseach section members’ papers and posters that received awards at mla 2017 – congratulations to all first place, second place, and honorable mention recipients! finally, what’s coming up content-wise for our spring/summer 2018 issue? well, we need your help with that! are you a hospital librarian? or do you collaborate with hospital librarians? we’d love to have our first issue of 2018 focus on the great research and/or project work done in this area of medical librarianship. have something you’d like to share? send it our way using our shiny new (beta) system: http://www.mlanet.org/e/sx/eid=70. until next time cheers and happy new year! carol & erin -- * immeasurable thanks to margaret hoogland and kate corcoran for their input and hard work in updating webpages and setting up the submission site. we appreciate your help in making hypothesis a better publication! mailto:mlarshypothesis@gmail.com http://www.mlanet.org/e/sx/eid=70 peer reviewed article hypothesis , vol. 35, no. 1, 2023 some like it quiet: using a heat map to study space use in a health sciences library will deana aresearch and data services librarian, temple university libraries, temple university, philadelphia, pennsylvania, will.t.dean@gmail.com objective: this method is used to identify preferred seating locations within the library. approach: this method involves counting where patrons are situated in the library at different times of the day, at different times of the year, and using that data to generate heat maps. data type(s) used: the data consisted of locations and counts of patrons around the library at different time points. strengths: provides a granular look at space use in the library. limitations: it is difficult to know why patrons prefer different sections of the library. overview as with all libraries, health science libraries want to know how their patrons use their space so changes can be made to help create a welcoming and useful experience. to keep track of library use, our library regularly counts the total number of patrons in the library and the number on each floor throughout the day. there is no record, however, of where on each floor patrons choose to study, read, and congregate. nor are there any records of how those behaviors vary over the course of an academic school year, a week, or hour by hour. a literature review of space use studies in academic and health sciences libraries was conducted and served as the basis for the work described in this article. khoo1,3 and gullikson2 described studying space use in academic libraries using heat maps, and this option was chosen for several reasons. first, it allowed for collecting granular data about how patrons were using the space at a particular time instead of relying on those who are motivated to respond to surveys or participate in focus groups. additionally, heat maps are easy for the viewer to parse and give immediate information about how different spaces are used. library staff also wanted to create visualizations of how the library was used as part of a 10-year review of the library’s operations, and the data collection could be conducted using existing staff time and operations. the method described below makes it possible to quickly generate heat map images from any number of time points by loading data and variables into adobe photoshop. since data was collected four times a day, five days a week, over the course of two weeks, we wanted to generate heat maps for each time point – forty times overall – with minimal manual effort. this process allows users to generate a heat map for each of those time points quickly and easily and one can modify the data if needed without any manual image manipulation. methods for this study were adapted from khoo1 and gullikson2 such as recommendations on how to divide up sections of the library space into zones that could be analyzed together. 1 mailto:will.t.dean@gmail.com peer reviewed article hypothesis , vol. 35, no. 1, 2023 example in the fall of 2018, a health sciences library in a large, urban research university conducted a space use study by creating a series of heat maps. the intent was to determine usage levels of different areas of the library daily for two one-week periods at the beginning and end of the semester. working with library circulation staff, a plan was developed to modify pre-existing patron counts to record the exact patron seating locations during specific times throughout the day. using the methods described by khoo,1 each floor of the library was divided into zones based on furniture groupings and separation barriers. to gather more granular data, we decided to follow gullikson’s2 example and count the occupied seats in each zone as well. using modified paper floor plans, which detailed the different zones and seats on each floor, we tested the data collection process in late november 2018. library staff counted patrons by marking each occupied seat in some zones and tallying the total number of patrons in others. after feedback from library staff, the floor plans were modified to improve usability and include orientation instructions. the library’s regular weekday hours of operation are 7:30am-10pm. staff counted library patrons four times a day every weekday for one week at 9:30am, 12:30pm, 3:30pm, and 5:30pm. these data collection times were chosen so the data from different dates could be compared easily and to fit with existing staff counts of patrons. although our library is open in the evening (and overnight during finals weeks) and on weekends, counts could not be done at those times due to staffing limitations. the first set of data was collected during a week near the end of the semester with an expected high occupancy rate. the second set of data was collected during a traditionally low occupancy week following the winter break. data was transferred from the paper templates into spreadsheets that mimicked the layout of the floor plans, as outlined in gullikson.2 calculations were performed to derive an occupancy percentage based on a hypothetical 100 percent occupancy. to create the heat maps, we used adobe photoshop4 and instructions from rich media5 to overlay color-coded boxes on the library floor plans. occupancy percentages were translated into colors using a gradient of blue (low occupancy) to red (high occupancy) based on those used in khoo3 and gullikson.2 results studying the use of space in a library over two weeks does not give a complete picture of how the space is used throughout the entire academic year. however, using heat maps allowed us to identify some interesting trends. table 1 illustrates the total use of the library by floor during both the high occupancy week and the low occupancy week. in general, use of the library was slightly more than double during the high occupancy week compared to the low occupancy week. the second floor was the most used area of the library, regardless of the time of day. compared to the low occupancy week, the first and second floors were nearly twice as 2 peer reviewed article hypothesis , vol. 35, no. 1, 2023 popular as the high occupancy week, and the lower level was nearly three times as popular. the heat maps made these differences easily visible. table 1. occupancy counts and percentages from two one-week periods high occupancy week count occupancy percentage low occupancy week count occupancy percentage 2nd floor 1,854 20.74% 846 9.46% 1st floor 743 14.04% 417 7.87% lower level 1,145 17.04% 457 6.80% total 3,742 17.85% 1,720 8.21% as seen in the heat map below (figure 1), the lower level was quite popular during the high occupancy week, with some surprising findings. the study tables in zones 6, 7 and 8 were very popular. our assumption was that this was due to their proximity to the only sources of natural light on this level. however, the carrels in zone 2 which are furthest from all natural light sources – were almost as popular, if not more so, during the same time period. additionally, patrons seemed to prefer more secluded areas of the floor with less foot traffic, such as near the stacks. figure 1 clearly shows the study rooms in zone 3 and carrels in zone 2 were used more often than other areas on the lower level. perhaps the lower level is more popular during times of intense studying overall due to its seclusion and the large number of seats located near natural light. figure 1: lower level during high occupancy time period 3 peer reviewed article hypothesis , vol. 35, no. 1, 2023 figure 2, illustrating the data on the lower level from the low occupancy week at the beginning of the semester, appears much cooler (more blue areas) than in figure 1. by comparing the two figures, the difference in colors clearly illustrates how the same library space is populated at different times of the year. of particular interest is that zone 3 was popular during both times of the year indicating a strong preference for this location. figure 2: lower level during low occupancy period using heat maps also made it clear that some high-traffic areas were never popular. zone 11 contains a large circular staircase that runs through all three floors with nearby seating. that seating was not popular in either week, perhaps due to the disruptions from people using the stairs. one of the key benefits of using heat maps is the ability to easily identify which areas of a library are popular or unpopular just by noting how ‘hot’ (bright green/yellow) or ‘cool’ (dark blue) an area is colored. analyzing the maps and the underlying data also identifies other potential topics for more study and consideration. the key limitation of the heat map method is understanding why a particular space in the library is popular or unpopular with patrons. given this limitation, the heat maps generated in this study will be used to help inform space use decisions, but only in conjunction with direct feedback from library users gathered via focus groups and advisory groups. 4 peer reviewed article hypothesis , vol. 35, no. 1, 2023 this method did not answer all our questions, and often raised more, but provided a visual way to easily understand library space usage across multiple time points. further reading given lm, leckie gj. (2003). “sweeping” the library: mapping the social activity space of the public library. library & information science research. 2003;25:365–385. https://doi.org/10.1016/s0740-8188(03)00049-5. nitecki da. (2011). space assessment as a venue for defining the academic library. the library quarterly. 2011;81(1):27-59. https://doi.org/10.1086/657446. schmidt a. (2016). data-driven design. library journal. 2016;141(6):26. http://lj.libraryjournal.com/2016/04/opinion/aaron-schmidt/data-driven-design-the-userexperience. references 1. khoo m, rozaklis l, hall ce, kusunoki d, rehrig m. heat map visualizations of seating patterns in an academic library. iconference 2014 proceedings, jan. 2014. https://doi.org/10.9776/14274. 2. gullikson, s, meyer, k. collecting space use data to improve the ux of library space. weave, journal of library user experience. 2016;1(5). http://dx.doi.org/10.3998/weave.12535642.0001.502. 3. khoo mj, rozaklis l, hall c, kusunoki d. (2016). “a really nice spot”: evaluating place, space and technology in academic libraries. college & research libraries. 2016;77(1):51-70. https://doi.org/10.5860/crl.77.1.51. 4. adobe. adobe photoshop user guide [internet]. create data-driven graphics. https://helpx.adobe.com/in/photoshop/using/creating-data-driven-graphics.html [cited 27 apr 2021]. 5. rich media creative services. using variables in photoshop with external data source. http://www.richmediacs.com/user manuals/rmcs ps training/using%20ps%20variables /usingvariablesinps external.html [cited 8 aug 2021]. 5 https://doi.org/10.1016/s0740-8188(03)00049-5 https://doi.org/10.1086/657446 http://lj.libraryjournal.com/2016/04/opinion/aaron-schmidt/data-driven-design-the-user-experience http://lj.libraryjournal.com/2016/04/opinion/aaron-schmidt/data-driven-design-the-user-experience https://doi.org/10.9776/14274 http://dx.doi.org/10.3998/weave.12535642.0001.502 https://doi.org/10.5860/crl.77.1.51 https://helpx.adobe.com/in/photoshop/using/creating-data-driven-graphics.html http://www.richmediacs.com/user_manuals/rmcs_ps_training/using%20ps%20variables/usingvariablesinps_external.html http://www.richmediacs.com/user_manuals/rmcs_ps_training/using%20ps%20variables/usingvariablesinps_external.html invited commentary hypothesis vol. 33 no.1 fall/winter 2021 writing structured abstracts for conference submissions abby l. adamczyk, mlis, ahip mla research caucus chair 2021-2022 graduate medical education librarian scott memorial library thomas jefferson university philadelphia, pennsylvania http://orcid.org/0000-0001-7653-295x abby.adamczyk@jefferson.edu if you are new to academic conferences and see that a call for proposals requests the use of a structured abstract format, you may be tempted to use journal article abstracts as a model for your own writing. however, conference abstracts serve a different purpose than those for scholarly journal articles. a journal article abstract serves to summarize the contents of the full manuscript in order to convince readers to read the full article. meanwhile, conference abstracts need to stand alone as a brief representation of your work. a conference abstract first aims to convince conference planners to select your work for inclusion. second, an accepted conference abstract also tells attendees what to expect from your paper or poster if they attend your session. third, an accepted abstract also may be used by judges when considering conference research awards. and finally, the conference proceedings may be used by future researchers compiling systematic reviews. all these purposes require a slightly different framing than the abstract you might write as the summary of a full manuscript. when designing a conference abstract, you have a few brief sentences to argue that your research project is valuable and worth the time of these conference attendees. this article briefly highlights key considerations for each section of a structured abstract for a conference research submission. for more detailed information about structured abstracts, see “the structured abstract: an essential tool for researchers” [1]. objectives/hypothesis/research question(s): for research, this section is best summarized as a question. if you already have the results, this may require taking a step back to a time before you knew the answer to the question. for example, the statement "to show that a live demonstration improves student assignment scores compared to handouts alone" can be rewritten as a question "does providing students with a live demonstration in addition to a pdf handout improve assignment scores compared to handouts alone?" if you cannot explain the purpose of your project with a question or hypothesis, you likely are not writing a research abstract. consider submitting a program description (or alternative) instead. use this section to argue that your question is worth considering, and that the answers to this question will be of interest to attendees of the conference. that may require different framing depending on the topic of the conference. http://orcid.org/0000-0001-7653-295x mailto:abby.adamczyk@jefferson.edu invited commentary hypothesis vol. 33 no.1 fall/winter 2021 methods: explain what you did (or are planning to do) and why. you want to make sure to explain how the thing you are counting will answer your question. in the case of the research question example, we need to compare assignment scores. but which groups are we comparing? two different years of the same course, two class sections in the same year, more classes, or two assignments in the same course? each one of those introduces its own bias that will need to be considered. demonstrate to the conference planners and attendees that you will be able to answer your question with this methodology. results: numbers, numbers, numbers! this is where all of your numbers belong. not just the comparison of, for example, an average assignment score of 9/10 points in one group vs. 6/10 points in the other. but also, how many students were in the class that you were evaluating? was this in one class or 20 classes? if you used another methodology, what was the response rate to your survey? present the data in a way that makes it clear how it answers your research question. include as much information as possible within word counts limits. often, when submitting a conference abstract, a project is in progress and may not yet have results. in that case, note that "if accepted, these data will be shared at the annual meeting." be aware that in these cases, it is likely that further scrutiny will be paid to the objectives and methods sections, heightening the importance of conveying the purpose of your research and why your proposed methods are appropriate for answering your research question(s). discussion/conclusions: tell the audience what your results mean in the broader context. did the data answer your question? did the data give you the answer you expected? if not, consider reasons why not. what other questions do you have now? what should be repeated? why will this be of interest to the conference audience? since you will have a limited word count, use the abstract to focus on the conclusions that will be most interesting to the conference audience. again, you may submit before having results to frame conclusions around. in this case, say something like "discussion of results will be shared following completion of data collection. results will be interpreted based on comparisons to existing literature on the topic." key points: • conference abstracts are a tool to advocate for your project’s inclusion in a conference. • the abstract is a stand-alone document, since there is no manuscript to read for more information. • include as much information as possible within the word count limits, especially your data. invited commentary hypothesis vol. 33 no.1 fall/winter 2021 • an abstract without a question/hypothesis and data might not be a research abstract. bibliography bayley l, eldredge j. the structured abstract: an essential tool for researchers. hypothesis 2003 spring; 17 (1): 1, 11-13. http://journals.iupui.edu/index.php/hypothesis/issue/view/1604 http://journals.iupui.edu/index.php/hypothesis/issue/view/1604 hypothesis , vol. 34, no. 1, 2022 giving new life to our failures heather n. holmes, mlis, ahip, associate editora, aassociate director of libraries, musc libraries, medical university of south carolina, charleston, south carolina, , holmesh@musc.edu in the fall/winter 2019 issue of hypothesis: research journal for health information professionals we published our first hypothesis: failure! article. that article talked about pitfalls of a library renovation and plenty of lessons learned along the way. the next issue’s column addressed using reflective practice after (coincidentally) suffering writing woes while writing an article about reflective practice. now, for our 2022 issue i’m pleased to share the article, a pilot project exploring low-tech collaborative board gaming on student attitudes toward interprofessional education, that discusses a relatively simple method of student engagement but like many things in life, timing is everything and perhaps rolling out this initiative during a world-wide pandemic did not set it up for success. the authors give a great background on the project and all the reasons it should have been successful, but they provide thoughtful reasoning on why they are submitting this as a failure. reading the article made me think a bit more about what my vision has been for this column because while this month’s article is reported as a statistical failure, the authors did recognize some anecdotal success with their project. when i first proposed this column to then editors, carol perryman and erin foster, i told them i wanted to have a place for libraries and librarians to report failures, lessons learned, or mistakes that mattered. that is and remains my primary focus and criteria for submissions to this column. however, i am a firm believer in using anecdotal “evidence” when appropriate, especially if there’s high probability of it offering or lending to something being successful. one could argue that much of evidence-based library and information practice (eblip) is anecdotal. while the numbers didn’t show success with the project in this month’s article, the self-reported feedback from the students who took part in it did feel there was benefit. there is more to this story as you will see when you read the article, but i appreciated the authors perhaps unintentionally making me, as the reader, think about other considerations within their failed study. i invite you to share your failures as a way of giving back to the profession what you’ve learned not to do the next time you try something. this column is not limited to health science libraries or librarians so please encourage your colleagues in other types of libraries to consider submitting an article. regardless of which type of library we work in, many of our practices, goals, and missions are the same. because of that, we can learn plenty of lessons from one another even if the failure wasn’t in a setting just like our own. heather n. holmes, mlis, ahip associate editor 1 https://www.orcid.org/ https://orcid.org/ mailto:holmesh@musc.edu guest editorial bass hypothesis, vol. 29, no. 2, fall/winter 2017 6 welcome from research section chair (2017-18 term) michelle b. bass, phd, msi, ahip population research librarian lane medical library stanford university school of medicine palo alto, ca i’m fairly new to the medical library association (mla) as my professional organization home but i was not shy about looking for ways to get involved. i joined as a student member in the fall of 2013 as i began my two-year assistantship at the taubman health sciences library at the university of michigan in tandem with my master of science in information degree at the university of michigan school of information. i was honored to be selected as an mla “new voice” at the 2015 austin meeting and took this as my cue to become more involved, and served as a member of the research imperative task force to populate the forthcoming health sciences librarian research guide page on the mla advocacy website. i was also encouraged by my colleagues at taubman, many of whom were active in the research section, to run for a leadership position. i did and here i am. my new leader status has left me a bit unsteady on my feet as i try to figure out what i should do as chair and brainstorming what i could do as an organization leader. a main goal of my term as chair of the research section is the production of a section procedure manual with delineation of roles and responsibilities for leadership positions as well as serving to guide the continued publication of hypothesis. i thank the current members of the research section leadership team for working with me on the manual and would like to further extend this call to the membership. have you held an elected position in the research section? did you work on hypothesis? do you have your notes? i’d love to hear from you to add your knowledge to the manual for future leaders to reference as we continue to expand the impact of the research section in the organization. please contact me by email (michellebbass@stanford.edu) and we can set up a time to chat or, if you’re in the bay area, i’m also open to field trips to meet in-person. mailto:michellebbass@stanford.edu guest editorial bass hypothesis, vol. 29, no. 2, fall/winter 2017 7 looking forward to a strong slate of programming from section members at the 2018 annual conference! sincerely, michelle b. bass, phd, msi, ahip -- awards hypothesis vol. 33 no.1 fall/winter 2021 2021 research caucus research awards congratulations to the winning research papers and posters from mla ’21 virtual! the mla research caucus is pleased to announce the winners for best research papers and posters presented at the mla 2021 virtual meeting. thank you to all the judges who volunteered their expertise to help select these deserving awardees both in the pre-judging phase and at the virtual conference. to learn more about the awards and selection process, visit the mla meeting poster and paper awards page from the research caucus community information pages at http://www.mlanet.org/p/cm/ld/fid=938 mla ’21 winning contributed papers first place health sciences librarians’ engagement in open science: a scoping review • kevin read – associate librarian, university of saskatchewan • dean giustini – ubc biomedical branch librarian, university of british columbia • ariel deardoff – data services librarian, ucsf library • lisa federer – nlm data science and open science librarian, national library of medicine • melissa l. rethlefsen executive director, health sciences library & informatics center at the university of new mexico health sciences center objectives: to identify health sciences librarians’ (hsls) engagement in open science (os) through the delivery of library services, support, and programs for researchers. methods: we performed a scoping review guided by arksey and o’malley’s framework and joanna briggs’ manual for scoping reviews. our search methods consisted of searching five databases (medline, embase, cinahl, lista, and web of science core collection), tracking citations, contacting experts, and targeted web searching. we used zotero to manage citations, and covidence for screening. to determine study eligibility, we applied predetermined inclusion and exclusion criteria, achieving consensus among reviewers when there was disagreement. finally, we extracted data in duplicate and performed qualitative analysis to map key themes. http://www.mlanet.org/p/cm/ld/fid=938 awards hypothesis vol. 33 no.1 fall/winter 2021 results: we identified 54 included studies after reviewing 8173 citations and 319 full text studies. research methods included descriptive or narrative approaches (76%), surveys, questionnaires and interviews (15%) or mixed methods (9%). publication types included case studies (46%), journal articles (39%), conference posters and abstracts (9%), book chapters (4%), and reports (2%). using foster’s open science taxonomy, we labeled studies with one or more of six themes: open access (54%), open data (43%), open science (24%), and open education, open source and citizen science (17%). key drivers in os were scientific integrity and transparency, openness as a guiding principle in research, and funder mandates making research openly-accessible. hsls engaged in advocacy for os in many ways, with most examples coming from academic institutions. hsls assumed key roles by advocating for and promoting os, and by collaborating on policy development, especially in support of oa and open data. conclusions: hsls play key roles in advancing os worldwide. however, more formal studies are needed to assess the impact of hsls’ engagement in os to determine best practices. future studies should identify os researchers’ needs, and evaluate the library service models best designed to meet them. hsls should promote broader adoption of os within their research communities, and develop strategic plans aligned with institutional partners. further, hsls can promote os by adopting more rigorous and transparent research practices of their own. future research should consider examining hsls’ engagement in os through social justice and equity perspectives. second place considerations for conducting evidence syntheses: evaluating the performance of various electronic methods for de-duplicating references • sandra mckeown – health sciences librarian, queen’s university • zuhaib m. mir – resident physician, queen’s university objectives: the objective of this study was to determine the accuracy and performance of commonly-used electronic methods for de-deduplicating references retrieved from searching multiple bibliographic databases. methods: a heterogeneous sample of references was obtained by conducting a similar topical search in medline, embase, psycinfo and cochrane central. manual abstraction was used to identify duplicate references among the search results and develop a gold standard for comparison. de-duplication methods awards hypothesis vol. 33 no.1 fall/winter 2021 included ovid multifile search, endnote desktop (versions x9 and 20), mendeley, zotero, covidence and rayyan. false negative and false positive duplicate references for each method were identified and then the accuracy, sensitivity, and specificity of each was calculated. results: default de-duplication settings in ovid multifile search and dedicated review software (covidence and rayyan) significantly outperformed reference management software. ovid, covidence and endnote 20 possessed the highest specificity (1.00, 1.00 and 0.998, respectively) for identifying duplicate references, while rayyan demonstrated the highest sensitivity (0.96). the accuracy of endnote desktop improved from the x9 version (0.76) to the newer 20 version ( 0.92). conclusions: this study highlights the strengths and weaknesses of commonly-used electronic methods for de-duplicating references and suggests strategies to avoid unintentionally removing eligible studies and introducing bias into syntheses. when selecting database platforms and utilizing software programs for the review process, de-duplication functionality and performance is an important consideration for improving the efficiency and quality of evidence syntheses. honorable mention health science librarians’ engagement in work-related reflection: results of a qualitative exploration of why they engage in reflective practice • jolene m. miller – director, mulford health science library, university of toledo objectives: published research on health science librarians use of reflection at work (reflective practice) tends to be results of action research and surveys. the current study fills a gap in the literature, exploring the following questions: as librarians engage in intentional reflection to improve work performance, what are they actually doing? why do they invest time and energy in reflection in the first place? the purpose of this qualitative study is to better understand how health science librarians experience and understand reflective practice. methods: in 2020, 18 health science librarians who use reflection at work were recruited using email invitations sent to medical/health science librarianship email distribution lists based in the us and canada. there were no limitations on how, to what extent, or how long participants had been using reflection. they were interviewed using a video conferencing platform about their use of reflection at work and what advice they would give to someone new to reflective practice. the interview transcripts were analyzed within a phenomenological framework for awards hypothesis vol. 33 no.1 fall/winter 2021 themes using maxqda 2020 software. (prior to recruitment, this study was reviewed and identified as exempt research by the university of toledo social, behavioral, and educational institutional review board.) results: several themes emerged from the analyses: (1) participants reported a variety of benefits from engaging in reflection at work; (2) the importance of intentionality in reflection; (3) the role of emotion in reflective practices; and (4) the wide range of reflective strategies used by participants. the primary themes regarding advice to people new to reflection: (1) start small; and (2) find what works best for you. conclusions: this project provides rich detail about the ways that health science librarians engage in using reflection at work. perhaps the most important finding is that librarians used reflection in ways that work best for them in terms of their workload, work environment, personality, and extent of experience with reflection. hopefully, this encourages health science librarians to consider using reflection to improve their work performance without fear that they are “doing it wrong.” mla ’21 winning contributed posters first place developing and conducting a dei language inclusivity assessment on a health science library’s website • jane morgan-daniel – community engagement and health literacy liaison librarian, health science center libraries, university of florida • matthew daley – web designer, health science center libraries, university of florida • mary e. edwards – liaison librarian, health science center libraries, university of florida • lori h. eubanks – office assistant, health science center libraries, university of florida • hannah f. norton – chair, health science center libraries, university of florida • nina stoyan-rosenzweig – senior associate in libraries, health science center libraries, university of florida objectives: the diversity, equity, and inclusion (dei) team at a university health science library sought to conduct a language inclusivity assessment of their library’s public-facing website. the project’s objectives were to 1) create a checklist for inclusive language, 2) review the library’s webpages using the checklist, 3) update awards hypothesis vol. 33 no.1 fall/winter 2021 the webpages accordingly, and 4) circulate the checklist to the university’s other libraries for potential use. language inclusivity was defined as “language that is free from words, phrases or tones that reflect prejudiced, stereotyped or discriminatory views of particular people or groups” (british columbia public service). methods: first, the team searched for literature relating to best practices for language inclusivity. nineteen useful resources were located, from which the team compiled a 32-item checklist, covering demographic information collected in webforms, as well as identification of gendered language, stereotypes, ableist language, and out-of-date terminology. with this checklist, the team reviewed the 33 webpages of the library’s website, not including libguides or links to external webpages. each webpage was assessed by one of five team members, who used a spreadsheet pre-populated with the checklist to record language that needed to be changed. the team also made a note of dei-related content that could potentially be added, based on the language inclusivity literature we had consulted before beginning the assessment. results: nineteen instances were recorded for which improvements needed to be made, encompassing the following topics: adding pronouns for employees who opt in; referring to invisible disabilities on our “users with disabilities” page; using student-first language instead of faculty-first; spelling out acronyms; and removing outdated language with negative connotations such as “earmarked”, as this term has been associated with enslavement. the team’s recommendations for website content additions included creating a dei statement; developing a code of conduct for the library’s online and physical spaces; and adding information about parking, assistive technologies, and emergency evacuation procedures to our “users with disabilities” page. conclusions: the website updates are currently underway and the checklist has been circulated to the six other university libraries as planned. the next stage of the project is to expand the website review to include the library’s archives pages; this poses the additional challenge of recognizing and appropriately delineating historical language where appropriate, so our library’s archivist has joined the project team. following completion of the archives pages, the dei team plans to use the checklist to begin assessing our library’s libguides, signage, and non-digital flyers and forms. second place the status of scholarly efforts of librarians in health literacy: a bibliometric analysis awards hypothesis vol. 33 no.1 fall/winter 2021 • alexandria q. wilson – research & learning services librarian, university of tennessee graduate school of medicine • courtney wombles – graduate research assistant, university of tennessee graduate school of medicine • robert e. heidel – associate professor of biostatistics, university of tennessee graduate school of medicine • kelsey l. grabeel – assistant director, university of tennessee graduate school of medicine objectives: over the past 20 years, health literacy has increasingly impacted patient care. recently, healthy people 2030 updated the definition to include personal health literacy as well as organizational health literacy. as information professionals, librarians can be leaders of helping to improve health literacy through their roles in the classroom, public library, or in the healthcare environment. in order to determine the scholarly impact librarians may have had on health literacy, researchers examined the status and characteristics of health literacy (hl) publications authored by librarians from 2000 to 2019. methods: bibliometrics, the statistical analysis of publications, is used to assess the indicators of productivity, collaboration, and impact or visibility of librarians in health literacy-related research. data was collected using the multidisciplinary database scopus and articles were screened for inclusion criteria using rayyan. through bibliometrics, quantitative analysis will be performed, including examination of number of citations, number of publications per year and per country, the most prolific authors and their affiliations, most prolific journals and journal impact factors, number of articles published in library science-related and non-library sciencerelated journals, the origin of the first author’s country, number of articles per research type, etc. the itsa (interrupted time series analysis) procedure in stata was used to assess the trends in publications preand post-2013. results: of 321 search results, 219 unique references met the inclusion criteria of librarian authorship. before 2013, the rate of hl publications increased at a rate of 1.30 annually (95% ci 0.97–1.62, p<0.001). after 2013, the number of hl publications increased at a rate of 4.18 publications (95% ci 3.71–4.65, p<0.001) annually with the peak in 2019 (n=32). journal of consumer health on the internet was the most prolific journal. the majority of included references were authored by at least two authors with a collaboration index of 2.58. in terms of impact, 79% of included articles were cited at least once. conclusions: in the last two decades, librarian involvement in hl publications has awards hypothesis vol. 33 no.1 fall/winter 2021 exponentially increased, most markedly in the years following 2013. librarian authors are highly collaborative, but few have published on health literacy multiple times. the visibility, volume, and collaboration efforts show how engaged librarians are in health literacy scholarship. further research is needed to examine the research and impactful efforts made by librarians that aren’t published. second place find the studies without the noise: validation of the performance of a pragmatically designed pubmed methodological filter • nancy firchow – senior librarian, california environmental protection agency, office of environmental health hazard assessment • melinda davies – associate research informationist, kaiser permanent center for health research objectives: comprehensive review searches can generate a high volume of results, including many non-empirical publications that present no original patient/participant research data and will be screened out during the title/abstract review phase. using a set of included studies from previously published systematic reviews (srs), we demonstrate the relative recall performance of a pragmatically-designed pubmed search filter, the non-empirical publication (nep) filter, intended to distinguish and exclude non-research publications from comprehensive searches and still retrieve publications of interest. methods: we identified 32 srs published in 2017 and indexed in pubmed which reported their included studies. the focus of the srs included treatment, epidemiology, diagnosis, and other. the srs included studies of a broad range of methodologies, including rcts, non-randomized clinical trials, observational studies, and case reports. we measured the ability of the nep filter to retrieve the pubmed-included studies included in these reviews. results: the 32 srs included a total of 1172 studies, of which 1062 were found in pubmed by known-item searching. the nep filter demonstrated a relative recall of 0.98 across all 32 reviews. relative recall for sr subsets was as follows: treatment srs 0.97; epidemiology srs 0.98; diagnosis srs 0.98; other srs 0.99 conclusions: the nep filter performs very well in retaining studies of interest using a relative recall method. it holds its high recall rate across srs of different types and across included studies of a broad range of methodologies. work is continuing to establish sensitivity, precision and reduction in number needed to read. awards hypothesis vol. 33 no.1 fall/winter 2021 honorable mention medical student question formulation skills development: a randomized controlled trial • jonathan d. eldredge – associate professor, university of new mexico • melissa schiff – professor, university of new mexico • jens langsjoen – hospitalist, university of new mexico objectives: eldredge et al. reported at mla 2020 on their quasi-experiment involving a new fac (focus, amplify, compose) rubric for assessing medical students’ evidence based practice (ebp) question formulation performance. this new rubric, when linked to complementary hands-on training, improved student scores. this study builds on the previous study by measuring whether the rubric alone, or when combined with a 25-minute intervention, will be more effective in teaching secondyear medical students on how to formulate effective ebp questions. methods: randomized controlled trial. 80 second-year medical students were randomized into either an intervention group or a control group prior to the beginning of the block. the authors administered a pre-test to all students that presented them with a clinical vignette and asked them to formulate an ebp question. following the pre-test, all students were shown the question formulation rubric and it was briefly explained. students in the intervention group were taught how to formulate ebp questions using the rubric followed by another 30 minutes of ebp search training. students in the control group only received the ebp search training in their small group labs. all 80 students took the post-test in which they formulated a question in response to a clinical vignette. statistical analysis will involve a two-sample paired ttest to test between-group differences. results: the authors collected their pre-test and post-test data for the intervention and the control groups during december 2020. the authors will report their findings at the mla 2021 annual meeting. conclusions: the authors will offer their conclusions during the mla 2021 annual meeting. mla ’21 winning contributed papers mla ’21 winning contributed posters column announcement holmes hypothesis, vol. 30, no. 1, fall/winter 2018 5 coming soon! hypothesis: failure heather n holmes, mls, ahip associate director of libraries musc libraries medical university of south carolina (musc) charleston, sc holmesh@musc.edu wait, what? a hypothesis failure? what is that? and why would i want to tell anyone that i failed, let alone publish it in a journal with substantial readership? i don’t want my colleagues, and especially not my boss, to know that my project didn’t work so no way would i want to tell the world. hypothesis is asking you to tell the world! from the time we are children we know that we learn by trial and error. sometimes those errors are pretty big, and even life changing. the key is that we’ve learned and we know that if we’re in that position again we should try a different approach so as to avoid another (or at least the same) mistake. you’d only walk through fire once, right? in our professional lives we tend to be more guarded; we do a really good job of celebrating our successes, but we tend to hide our failures or things that we didn’t do so well. then the question “how will we learn if we don’t share our failures?” arises. it is a question that i’ve asked many times, and also one that i’ve vowed to find a way for others to answer. this column has been several years in the making, but it is finally coming to fruition and many thanks to hypothesis co-editors carol perryman and erin foster, for providing a space for us to learn from projects that didn’t go so well. so what does this mean? beginning in 2019, hypothesis will have a regular column dedicated to failures, lessons learned, or mistakes that mattered. submissions will be expected to describe the research, including the methodology and objectives, provide a narrative about the experience, and lessons learned or what would be changed if given the opportunity to repeat it. writing style should be casual but informative. submissions will also be subject to peer review. we all have plenty to learn from each other regardless of the type of library that we’re in, so it is our hope that this column will be filled with submissions from all types of libraries. our colleagues from the american library association, special library association, academic & college research libraries, canadian health libraries association, and any others are encouraged to submit for publication. --invited commentary hypothesis vol. 33 no.1 fall/winter 2021 review of "one database at a time: pubmed for expert searchers and systematic reviews” alexandria q. wilson, msis, ahip assistant professor, research and learning services librarian preston medical library the university of tennessee graduate school of medicine knoxville, tennessee http://orcid.org/0000-0002-2649-8252 as the interest in systematic review expertise grows, the medical library association technology in education and systematic review caucuses came together to cosponsor a webinar series called “one database at a time” that examines and discusses how to use a different database for systematic searches. the technology in education caucus (tec) acknowledged the expertise available in the systematic review caucus and approached them about co-sponsoring the series. the caucuses hope to provide both expert searching and systematic review tips for health science librarians and informationists. the first session, focused on the pubmed database, was offered on march 23. moderated by margaret foster and led by panelists alyssa grimshaw, liz suelzer, and lateesa james, the session covered a variety of topics, from building your systematic search in pubmed to systematic review software. the session began with setting up a systematic search in pubmed. panelists recommended starting with mesh (medical subject headings). then, they recommended combining mesh terms with keywords to facilitate translating the strategy to databases that do not use mesh vocabulary and to catch any results that are not indexed with their mesh terms yet. another key piece of advice was to utilize “search details.” this allows searchers to see how pubmed translated the search. this even includes warnings if a search term does not map correctly or is pulling in irrelevant terms (e.g., the word “refusal” mapping to refuse or garbage). while this session mainly focused on pubmed, there were topics covered that anyone completing a systematic review could benefit from, even if they do not use pubmed for their systematic search. the next section covered the documentation process of the search and guidelines to follow, such as prisma and the equator network. once the search has been documented appropriately, panelists stressed the importance of using a citation manager, regardless of searching pubmed or not, to export and manage invited commentary hypothesis vol. 33 no.1 fall/winter 2021 those results. the pros and cons of citation managers endnote (clarivate analytics), mendeley (elsevier), and zotero were discussed. systematic review software options of covidence, rayyan, and distillersr were briefly introduced before a formal q&a session, although the moderator and panelists engaged in questions from the attendees throughout the webinar. the chat feature of the webinar and the q&a session were moderated by kay hogan smith and tara brigham. questions and answers brought up important discussions comparing pubmed to ovid/medline, signing up for search alerts versus re-running the entire search to update a systematic review, reproducibility concerns, and the “ideal” number of databases to search, if such a number exists. future sessions in this series will be determined via popular vote, and ovid/medline is planned for the second session. this webinar recording was edited by julia stumpff, who also led the advertising efforts. to watch the recording in full or access the transcript, please visit https://iu.mediaspace.kaltura.com/media/t/1_shw3x5ld. https://iu.mediaspace.kaltura.com/media/t/1_shw3x5ld research section spotlight hypothesis, vol. 30, no. 1, fall/winter 2018 33 research roadmap: understanding the research process (part 1) kristine m alpi, phd, mph, mls, ahip university librarian oregon health & science university library oregon health & science university portland, or krisalpi@gmail.com jonathan d eldredge, phd, mls, ahip evidence based and translational science collaboration coordinator health sciences library and informatics center university of new mexico new mexico, nm jeldredge@salud.unm.edu heather n holmes, mls, ahip associate director of libraries musc libraries medical university of south carolina (musc) charleston, sc holmesh@musc.edu katherine g akers, phd biomedical research/research data specialist shiffman medical library wayne state university detroit, mi katherine.akers@wayne.edu kimberly r powell, mlis research impact informationist woodruff health sciences center library emory university atlanta, ga krpowel@emory.edu margaret hoogland, mls, ahip assistant professor, clinical medical librarian mulford health sciences library university of toledo toledo, oh mahoogla@gmail.com research section spotlight hypothesis, vol. 30, no. 1, fall/winter 2018 34 *editor’s note: this piece has been split into two parts due to length. part 1 covers advice and guidance about doing research while part 2 (which will be in the hypothesis spring/summer 2019 issue) discusses how to share the products of your research. introduction since 1982, the mla research section has fostered discussion, shared advice, connected members to potential collaborators and research mentors, and kept its membership up-todate on library research efforts, particularly as they apply to a wide variety of academic life sciences and community health efforts. towards these endeavors, the 2017-2018 research section program planners proposed a special content session on understanding the research process for the 2018 mla annual meeting & exhibition in atlanta, ga. the session was envisioned as a panel discussion for an audience of new to intermediate librarian researchers that would offer advice around getting started, from formulating an idea, to study design, data collection, and finally dissemination and the publication process. other mla sections--leadership and management section, hospital library section, and educational media and technologies section--offered their support. panelists were recruited based on their previous involvement in continuing education and editorial work regarding library research, and several belonged to multiple sections sponsoring the session. in discussion with the research section program chair [billman] and the session moderator [powell], each panelist [akers, alpi, eldredge, holmes] selected to focus on an area most relevant to their work, with the recognition that all the presenters had expertise across the following topics comprising the presentation: • overview of the research process • finding and working with a research mentor • turning daily work into a research project • choosing research methods that fit you and your question • using technology and data collection methods efficiently • disseminating your research • understanding the publishing process “research roadmap: understanding the research process,” was presented on may 20, 2018, and 2018 mla annual meeting attendees can access the recorded session by logging into the online program with their registration information (https://www.eventscribe.com/2018/mla/). this account of the session is presented so those unable to view the recordings may benefit from the panelists’ advice and expertise. making a member-driven presentation https://www.eventscribe.com/2018/mla/ research section spotlight hypothesis, vol. 30, no. 1, fall/winter 2018 35 in planning the session, we wanted to respond to mla members’ questions about the research process. on april 5, 2018, billman posted a call for questions in the mla-focus, an email update sent to all mla members. questions received before the submission deadline were addressed in the content presented. the session incorporated two question and answer periods, one after the first two speakers and the second at the end. at the beginning of the session, a slide encouraged the audience to “submit your questions” using a tinyurl online survey or on cards. those not addressed by the presenters during the session have responses from our panelists in the appendix. the roadmap as presented overview of the research process we began by reviewing the evidence based practice process and the opportunity for research after the existing literature has been critically appraised, and before its application to the problem that spurred the initial search for evidence. moving on to the research process, we shared a four-step version for which we would provide our experiences: 1. formulate a research question (objective) 2. design an appropriate method to answer the question 3. interpret your results 4. communicate your results as we developed our expertise through being mentored and mentoring others, and because several of the questions we received were about finding mentors or collaborators, we addressed mentorship first. finding and working with a research mentor a mentor can be defined as “an individual with advanced knowledge, usually more senior in some regard, who is committed to providing upward career mobility and assistance for the protégé …” moreover, “the mentoring relationship has been characterized as an intense, sometimes intimate professional relationship devoted to providing social support and development for the protégé’s career.”[1] johnson observes that mentoring represents an enduring and reciprocal personal relationship in which the mentor serves as a role model while offering a safe environment for the protégé to consider new ideas. in this context, the mentor provides acceptance, protection, challenge and coaching for the protégé.[2] while many studies have investigated career or academic mentoring, few investigations have rigorously explored the specialized world of mentoring other professional colleagues in conducting research. and almost all investigations into research mentoring have research section spotlight hypothesis, vol. 30, no. 1, fall/winter 2018 36 focused upon faculty-student mentoring. eldredge has mentored more than 20 colleagues over the past two decades. a few of these colleagues have been new faculty members in his medical school whereas most have been library or information practitioners wanting guidance with research projects. these early experiences, coupled with limited time for mentoring individuals, led to sponsoring “the research mentor” column in hypothesis during 2008-2014. these columns featured topics about which his protégés needed recurring guidance ranging from creativity in research [3, 4, 5, 6], to defining authorship [7, 8], to the pragmatic aspects of research [9, 10]. they also included interviews with esteemed researchers [11, 12] and a reflection upon a past mentor’s long-term effect upon his own career [13]. additionally, the author participated in several of the university of new mexico’s mentoring institute’s research conferences [14]. roles of the protégé generalizing from these experiences, the protégé needs to practice reciprocity in the mentoring relationship. the research mentor brings a wealth of knowledge and skills on how to conduct a research project. the gains to the protégé are obvious. yet, what does the protégé bring to the mentoring relationship? inventorying the attributes possessed by the protégé might strengthen the relationship: 1. previous research education the mentoring relationship should not be confused with a tutoring relationship in which the mentor provides comprehensive one-on-one instruction on all aspects of research. the protégé should bring previous learning on research methods to the relationship. the protégé can learn research methods by taking or auditing courses at nearby or online academic institutions, such as those in subjects such as psychology, biology, or one of the social sciences. the new mla research training institute might provide an avenue for some protégés to gain research methods education and identify mentors, but this new, not yet evaluated, program does not offer the depth of training found in a graduate-level research course. 2. literature searching every research process involves literature searches in the formative stages due to the need to perform iterative literature searches until the final research question reflects the state of the existing research knowledge on the topic to be investigated. designing the research methodology provides another stage in which one or more literature searches will inform how the research project will unfold. finally, once the research is done, additional literature searching will assist in interpreting and discussing the results. librarians who are capable literature searchers can assist a research project with their unique skill sets. 3. organization research section spotlight hypothesis, vol. 30, no. 1, fall/winter 2018 37 librarians are legendary for their organizational skills, and many have learned project management skills that have direct application to the complex dimensions of a research project. librarians also can use bibliographic management tools such as zotero or endnote to organize the hundreds of references associated with any research project. a recently published scoping review involved elaborate bibliographies and analytics, highlighting the protégé’s contribution of an impressive skill set to the project [15]. 4. track record many librarians have contributed to research projects as part of their everyday work. these roles often involve literature searching and bibliographic management. while most involvement with research projects takes a more episodic form, some projects involve larger commitments. these projects form a track record of librarian involvement in past research projects that librarians can point to when approaching a research mentor for guidance. 5. commitment the potential protégé needs to bring a commitment to completing the research project to the mentoring relationship. experienced researchers know that many well-intended research projects are abandoned at various phases prior to completion. ideally, the protégé can present the potential mentor with a plan and timeline that will advance the research project through completion and communication to the profession through poster, presentation, or publication of the results. the multi-phased mla research agenda project [16] tested the commitment of all researchers involved yet when it reached completion, it provided a great service to our profession [17]. 6. creativity established researchers sometimes descend so deeply into the weeds of the research design and its careful implementation that we forget to consider the opportunities to lend some creativity to the project. a noteworthy example surfaced when one of the authors included three medical students in a randomized controlled trial on point-of-use tools. the three medical students expressed their creativity by producing a top-notch training video for participants in the study. in other ways, the students also added elements of creativity unrecognized by the veteran researchers and thereby improved project participation among providers [18]. 7. enthusiasm a few years ago, one of the author asked an audience of translational researchers what adjectives they would use to describe the research process. the adjectives offered were “tedious” and “detailed” and “comprehensive.” there are admittedly boring moments in any research project in which one can be tempted to walk away. toward the end of the project, however, excitement builds as results, particularly difficult to explain results, research section spotlight hypothesis, vol. 30, no. 1, fall/winter 2018 38 emerge in need of interpretation. a protégé with enthusiasm most likely will rely on it to muddle through the less exciting aspects of a research project. one applied informatics research study that comes to mind involved a lot of tedium, but the researchers all plowed through the hours of careful checking in this fidelity study to produce surprising results [19]. initiating the mentoring relationship we have focused on how the protégé seeking to recruit a research mentor needs to step outside the limits of her or his own perceptions to consider the perspective of the potential mentor(s). the mla mentoring/expertise directory (http://www.mlanet.org/page/mentoring) is one way to find mentors who have already indicated willingness to engage in mentoring relationships. in other cases, you may identify someone in the literature or at your institution with expertise you aspire to develop and there you will not know their willingness to mentor until you make the approach. it is up to the mentorship dyad to set mutually agreeable terms of the relationship. the respect is reciprocal, and it is also clear that the more experienced researcher can gain insight and satisfaction, and sometimes co-authorship from these relationships. in some cases, an enduring relationship develops that leads to new studies as collaborators. turning daily work into a research project there are many ways to turn daily work into a research project; it is a matter of looking and asking thoughtful questions about what work is being done. more specifically, it means looking at what we are, or perhaps should be, measuring. valuable research can be performed at many levels; not every question is best addressed by a large, multi-center nih-funded study. this is especially true in library research projects because much of what we measure and seek to improve is part of our daily work. one of the key pieces of evidence-based library and information practice (eblip) is applying local evidence to evidence in the literature, and to our own professional knowledge. considering the five principles of eblip as presented by koufogiannakis and brettle [20] may help us better understand ways we can turn daily work into research: articulate: a clear understanding of the problem or question must be reached. part of articulating the problem includes working out what is known already and why the information is needed, as well as ensuring that the problem is set in the appropriate context. assemble: evidence should be assembled from multiple sources that are the most appropriate to the question or problem at hand and should include research evidence, local evidence, and professional knowledge. http://www.mlanet.org/page/mentoring research section spotlight hypothesis, vol. 30, no. 1, fall/winter 2018 39 assess: evidence should be assessed for its quality (often known as appraisal or critical appraisal), determining what the evidence says as a whole. agree: determine a course of action and begin implementation of the decision. if working with a group, try to achieve consensus based on the evidence and organizational goals. adapt: evaluate the decision and how it has worked in practice. revisit goals and reflect on the success of implementation. applied research at the medical university of south carolina (musc) keeping these principles in mind and thinking through the work that we do daily at the library, a relatively simple research project grew out of updating libguides. we were looking at click stats on our guides and noticed some irregularities such as usage considerably different from what we expected. that started us in the first principle of eblip, asking (articulating) the question of why these irregularities were present. the research project developed from seeing the numbers and articulating the question. before we knew it, we had developed the project. ultimately, looking at the clicks in our libguides gave us an opportunity to assemble the evidence, assess it, and decide (agree) on what to do with what we learned. using what we learned, we decided on ways to reformat our guides and to reconsider placement of links based on perceived importance. if things we felt were important had been prominently placed, but were not getting many clicks, we moved them a little further down the page since fewer clicks seemed to indicate less importance to the user. conversely, when we found things placed lower getting a lot of use, we relocated them to a more prominent position. it is a very simple project, and one that we can easily adapt over time. we will revisit the statistics in the future to assess whether our repositioning made any difference. it is important to note that this simple project led to more questions being asked and therefore more research needing to be done. a secondary outcome (that we had not considered until we really looked at the data) was the realization that we needed to adjust our instruction sessions to include more guidance on navigating libguides, as well as accessing the resources therein. a follow-up study looking at data from and the questions being asked through libanswers is also being launched. the results will also factor into future libguides updates and we expect it to bring out even more questions and things to study. this is one small example of taking our everyday work and turning it into research. it is a project we know we will learn from, and we anticipate publishing our results to share our procedures with others who may consider a similar project. for another example from a different institution, see lindsay, oelschlegel & earl [21]. research section spotlight hypothesis, vol. 30, no. 1, fall/winter 2018 40 identifying and overcoming obstacles how do we find time to do research when we often feel like we do not have enough time to do our day-to-day job? further, how do we find time to do research when it is not expected of our position, or when our superiors are not supportive? this author [holmes] feels that the best way to achieve time for research is to simply prioritize. that means figuring out what can and cannot be shuffled, eliminated, or delegated from your daily work. it also means setting aside dedicated time in your schedule to get the research done, even if it is on saturday afternoon [holmes] or in thirty-minute blocks before work while others are sleeping [alpi]; see chronicle of higher education [22] for more ideas. you must set boundaries so that your dedicated time is not always interrupted or sacrificed, and you also must enforce the boundaries. if you are passionate enough about a topic and want to research it, then you must be willing to pursue it on your own time. there are many librarians who are either not in positions where there are expectations of publishing and presenting or who lack support from their superiors or even their colleagues, yet they want to be involved in advancing our profession. it is important for all library staff with questions to find their own way to pursue their interests, even if it is on their own time. in fact, most librarians spend a portion of their own time (and often their own money) to pursue their professional goals and interests. the results that will come from taking the time to work on projects will help your resume/curriculum vitae (cv) for future opportunities. another obstacle that we frequently encounter is lack of access to resources, whether it is software or people or something else. your mentor or your professional association can help you find these resources, but your own professional networking efforts will probably yield the biggest benefit. if someone’s research interests you, it is entirely appropriate to contact the researcher to say you would like to know more about the work or perhaps even suggest a collaboration. it helps to have a scholarly presence online (e.g. linkedin or google scholar profiles) so they can learn more about you before responding to your inquiry. collaboration is one of the keys to success, and it requires work on your end to make those meetings happen. collaborators can provide resources like software or expertise, but they can also end up serving as a mentor or a network to yet another person with whom you may be able to work with in the future. while a lack of understanding of the research process is an obstacle for early researchers, it is very much a learning opportunity. looking to the literature is one of the best ways to get an introduction to the process, and again working with a mentor can help with guidance. educational opportunities are increasing, and the mla’s research and evidence based practice curriculum committee is in the process of building an education curriculum for learning about research. a final suggestion for learning about the research process, and a general way to get more involved in research, is to engage with research section spotlight hypothesis, vol. 30, no. 1, fall/winter 2018 41 your institutional review board (irb)/ethics committee. if you are going to study your users or analyze educational output from previous courses for a presentation or publication to the outside world, you will need to have your proposed project reviewed by the irb. understanding the background of your irb members or looking into becoming an irb member yourself may be an excellent engagement opportunity to learn about research at your institution, and how you might find a partner. no matter your role, you will learn much more about the process, and when the time comes for your own research with human participants, you will know what you have to do to prepare your own submission for approval. always be sure to check with your irb prior to starting your research so you will know if what you propose will qualify as research or be considered quality improvement. having this information at the beginning of your work will save you a tremendous amount of time later and will be important if you plan to present or publish your findings! in summary, there are many ways to turn daily work into research. it is a matter of taking time to really look at the work you are doing, deciding what is measurable, and considering if those measurements can or will make a difference to how we work in the future. technology is changing how we do our jobs almost overnight, so it is important that we continue to make ourselves and our work relevant. that cannot be done without producing evidence, so we must find ways to work together to meet our goals and foster our professional interests. choosing research methods that fit you and your question there is not a “right” choice of research method for any question. as researchers at any level of experience, we work on a continuum of what may help us address our question and be feasible to execute. there are several characteristics of a proposed study to consider in deciding among methods—your question, the participants, timing, and capacity. the first consideration is your research question—what do you want to know or learn? research questions are also not set in stone. you may start with a very broad research question and realize that you need to focus it in order to begin working on it. participants are an extremely important component as you have varying levels of access to the insights they can provide. consider whether you can learn what you want to know from direct engagement with people (clients, students, library staff, etc.), artifacts representing people (assignments, electronic health records), or documents (cataloging records, reports, published articles, etc.). you as the researcher are also a participant, bringing your experience and current point of view to the study design, your interpretation of the results, and the conclusions you make. in qualitative research, you must be explicit about this in terms of sharing your contextual bias and subjectivity—for more see preissle [23]. research section spotlight hypothesis, vol. 30, no. 1, fall/winter 2018 42 timing and capacity often intertwine, and they tie back into your choice of participants. for example, you may be interested in interviewing a lot of people, but you do not have sufficient capacity to perform a lot of interviews, so you choose to do a survey or analyze existing records. research always takes much longer than we propose, so usually we come up with an initial time estimate and then double or triple it. you must include time and resources to pilot test any methodology. piloting surveys is common best practice, but you also need to pilot interview protocols, observation checklists, data extraction forms, and any other tool that would be used to gather or analyze data. if you do not feel you have time to pilot the tool, then you probably do not have sufficient time for the research. pilot testing does not have to be with the target participants, but should be with participants who are not involved in the research and are relatively similar to your intended population. many research projects are cross-sectional, meaning they measure a slice of a population at a single point in time, typically a one-time survey administration or observation. for more robust evidence of change over time, particularly in educational research, it would be ideal to perform longitudinal research, repeating measures on the same population over multiple points in time. however, tracking participants over time can be challenging. methodologies have inherent characteristics and limitations, but they are also more or less effective in the hands of experienced practitioners. look at studies that have addressed questions similar to yours in the library/information science literature and other social science fields such as education, psychology, management, etc. if the literature on a topic is from case studies or single-institution surveys, consider expanding the view with other study types or by including additional institutions for cross-institutional comparisons. if you are interested in applying a method and need assistance, methods mentorship can be internal or external to your institution or field. you do not have to seek the most renowned experts; it may be easier learn from those with slightly more skill or experience who can describe their approaches and refer as needed to those with greater expertise. the next section discusses a few common methodologies you may wish to consider. common methodologies in research a. document review and analysis analyzing published documents is a very comfortable research strategy for librarians and can be a study on its own or part of a larger mixed methods study. examples include a) evaluating information quality and readability; b) bibliometrics of publications or networks; and c) systematic or scoping reviews. you may evaluate materials produced by your own institution or those produced by others. if using your own materials, be aware of your potential bias and consider a collaborator or second evaluator not involved in the production of the materials being evaluated. standard and well-documented methodology research section spotlight hypothesis, vol. 30, no. 1, fall/winter 2018 43 for the evaluation or data extraction such as a codebook is very important for reproducibility! this is an area where the analysis can easily be either one-time or longitudinal, looking at annual compilations/comparisons. b. observational methods observational studies have different units of analysis, such as individuals, groups of people, or things. observations in libraries may be easy to facilitate since libraries are public places, whereas observations in other areas may require permission to access that environment. the potential impact of being observed can be very high in an environment where the observer stands out to those being observed. it may take several passive observation periods to reduce that effect. having an observational protocol and recording form that has been pilot tested in a similar environment is helpful to make sure observers are consistent. in addition to what is being observed that has been thought of, it is often help to note other things you notice which may turn out to be confounders, things that could explain the situation, but are not what you planned to study. there is value in observing cohorts or comparison groups, whether contemporary or historical, so that you have a sense of whether the observed phenomenon is consistent across groups or may have been an outlier specific to one group. this question of generalizability of findings is also an issue when gathering survey or interview data. c. gathering data from people: surveys and interviews the primary limitation to surveys and interviews is that only certain people participate and therefore findings may not represent the broader population from which you recruited your participants. this is one reason you report on the baseline population you attempted to survey when you calculate the response rate. for example, a survey of medical students may have a response rate of 50% (which would be considered very good). however, if all the participants are female, the survey is not representative of your population if half of your students are male, and no females participated. one way to understand a phenomenon more deeply and gain multiple points of view is to gather data on the same issue directed to multiple types of respondents and then look at correlations between respondents, e.g. asking instructors, students, and library staff about perceptions of same issue. surveys and interviews introduce recall challenges for participants in terms of whether they remember accurately what they did to report on it for the survey or whether those that had a negative experience are more likely to remember it and respond accordingly. it is important to understand whether respondents truly understand what you are asking. ways to address this are to use previously studied and validated measures, work on developing your question-writing skills, and use piloting combined with cognitive interviewing when you are writing new questions that may have never been asked in your study population. research section spotlight hypothesis, vol. 30, no. 1, fall/winter 2018 44 d. combining multiple methods case study research often combines multiple methods to add validity. a recent case study by north carolina state university doctoral candidate j.j. evans included a pilot study followed by four types of data gathering and analysis: 1) demographic survey; 2) semistructured interviews; 3) research activity log; and 4) documents & investigator’s research journal. case studies often include other types of document analysis to provide historical and geographic context for the case or can include and draw conclusions from multiple cases. for more on case studies, see a forthcoming editorial in the journal of the medical library association. e. interventional (“experimental”) methods we have opportunities for interventional studies in library practice. although educational studies come most often to mind, these do not have to be measuring real-time interactions with people. for example, studying the impact of changes in policy that affect circulation, collections, etc. can be done using analysis of existing records before and after the period in which the change was made. preand post-test evaluations in a population that have received an intervention are common; they require attention to the effect of testing itself as it is possible that taking the first test is what primed the person to do better on the second test and the intervention did not produce the effect. another option is historical control groups, if they are sufficiently like the current group. there may be an opportunity to do a group or individual randomized controlled trial in a course with the same instructor or different instructors. for fairness, consider offering a crossover design so that the control group eventually gets the intervention if the intervention is shown to be valuable or even neutral. as you involve multiple investigators in delivering the interventions, following the protocol to be sure you are measuring the impact of the same intervention becomes very important. f. combining data from multiple institutions one of the questions we received prior to the presentation was about our experience with multiple institution studies to strengthen confidence in our findings. designing a study to include multiple institutions has several positives. it may be that you would not have had the capacity to do your study alone if the partner institutions had not provided expertise or funding. having larger potential populations can may help with finding significant effects by reducing type ii (beta) errors. this would mean you do not find a statistically significant effect when there is an effect, because you do not have enough power to detect a difference. this is usually due to a low number of participants or the effect is smaller than you anticipated. or it may allow enough participants to identify subpopulations affected differently by what you are studying. it may also improve generalizability since effect is measured at multiple institutions if similar results are seen across the institutions. finally, it allows you to comment on the reproducibility of methodological tools and of findings. research section spotlight hypothesis, vol. 30, no. 1, fall/winter 2018 45 challenges now for the challenges—the three most salient are timing, instrument development, and ethics approvals. collaborators may find it difficult for all members to be available at the same time, even in terms of scheduling meetings in different time zones. timing also comes into play with accessing participants, particularly if the institutions are on different calendars or structures. for example, if you want to reach nursing students in their third week of classes, that could be weeks apart at different institutions. it can also be difficult to get multiple investigators to agree on the same protocol or survey questions. although there is the possibility of adding unique questions at some institutions, lengthening an instrument can affect response rates, jeopardizing the larger survey participation or introducing other variability. this can be addressed through joint instrument development and selecting very similar institutions for the collaboration where question wording would not need to be greatly changed. irb/ethics board differences across institutions may also cause delays, as many irbs will not accept the judgment of another irb (although that is slowly changing). in our experience getting a multi-site study approved by multiple irbs, some had a separate minimal application form for studies anticipated to be exempt, while others required a full application but then the study went through expedited review. this meant that many of the questions asked by the full irb application form had never been considered by the research collaborators at the other institutions. using technology and data collection methods efficiently 1. audience and timing impact choice of methods one you have considered your question, participants, timing, and personal capacity related to potential methods, it is time to focus on the audience and their timing and your institutional capacity. who is the audience for your anticipated findings? what kind of evidence will convince them? if your question is to respond to the needs of administrators, it may need to address return on investment in addition to the effect itself. it may not be enough to say something has an effect if you do not also include the costs of achieving that impact. questions that you are trying to answer to impact researchers or clinicians who are used to evidence-based practice and high-quality research literature may need to have control or comparison groups for them to trust in your findings. if you pursue research to build a portfolio for your library promotion and tenure committee, be sure you understand how they will value individual versus collaborative efforts, as that may shape whether you execute a study on your own and acknowledge others versus offering the opportunity for co-authorship with greater involvement. in addition to the audience, consider whether the research is time-sensitive or timespecific. for example, is there a limited window of time in which the study is possible (e.g., preand post-construction) or are the findings to be the basis of a time-sensitive decision? research section spotlight hypothesis, vol. 30, no. 1, fall/winter 2018 46 do you have deadlines to produce publications for promotion? these can all lead you to choose methods that can be planned and executed quickly although almost any methodology can be expedited if you have enough help or funds and the ethics committee/irb finds the study to be exempt. some examples of exempt research are anonymous surveys, passive observation of public behavior without collection of identifiers, and retrospective chart review. even if you have help or funding, it matters whether it is consistently available throughout, or in chunks, or at the time of day you need. an example is whether to use student labor, your own labor, or funds to hire external people to transcribe interviews. for observational studies, it might be whether you can do the observations at the time needed such as evenings or weekends or offsite daytime, or whether you need to hire or train others to do the observations, or you decide to rely on self-report data since no one is available to do the observations. 2. anonymous vs. confidential one of the most common questions that you must clarify for the irb submission is whether the participants will be anonymous or confidential, and this often determines whether a study will or will not be exempt from further review. your study design, the risk to participants, and the safeguards you must place on your data and whether you can share it often depend on getting this distinction correct in your consent documentation. if data collection is truly anonymous, you do not know who participated and hopefully no one could figure it out from your data. if it is confidential, you know the participants (people/institutions) and you take steps to conceal their identities. there are benefits and drawbacks to both approaches. choosing to collect anonymous data is generally an easier write-up for the irb application, since there are fewer risks of identity exposure and fewer data protections to describe. however, if your data is anonymous, you cannot follow up with respondents about their responses, do longitudinal studies, or join the data with other data sources that contain identifiers if you decide to expand the scope of your research project. even anonymous surveys require data protection considerations. for example, to be anonymous you must not track or download the ip addresses of participants which are often automatically gathered by online survey systems such as survey monkey or qualtrics. if you recruited by sending surveys to individual emails through the software system mailing functions, you may also need to have the system unlink or mask the responding email addresses. survey tool functions are discussed later, with some shown in table 1. if questions are specific and the population studied is small enough, people may be identifiable from their responses, and this may essentially turn your study into a confidential one where you need to redact identifying information from open-ended questions or only share aggregated responses for questions that have over five respondents in a certain category. confidential data collection involves assigning participants a unique identifier which is maintained securely and separately from research data. because people are identifiable research section spotlight hypothesis, vol. 30, no. 1, fall/winter 2018 47 in-person, by voice, and image, methodologies such as individual and focus group interviews and audio or video analysis can usually only be treated as confidential data. 3. analysis planning as part of the research design consider the relationship between your questions, your analysis skills, and your research team members. if you have a question with multiple factors and competing explanations, and you intend to gather and analyze quantitative data you will need some statistical expertise. it is much better to discuss your questions and data collection on the front end of the project with a statistician for statistical consulting so that you can come to a good understanding of the number of participants you will need to reach a meaningful conclusion and what type of data you might collect that allows you to test your question most robustly. if your research team includes students or library student workers or volunteers, consider what they can learn about data collection and analysis, and what expertise or capacity they already have. for example, while it may be more efficient to pay a transcription service to create transcripts of video interviews, if you have time and that is something you want the students to gain experience with you may decide to have them do some of the transcriptions. in planning your research timeline, design, and budget, it is important to consider the tools you have available and your familiarity with using them and what types of output they produce. a common choice is deciding on what online survey tool to use (see table 1). if you have designed and pilot tested your survey on paper, thinking about what question types will make it easy for participants to answer questions. it can be frustrating when the type of question you made is not available in one of the free tools. be sure to pilot test the online version of your survey to make sure it works properly, and that the resulting data is captured successfully. it is worth finding out whether you have institutional access to a tool such as qualtrics or redcap, as that will also make it possible to share surveys across institutions to facilitate collaboration. you also want to test question types by answering them in your survey tool to make sure that the output will be easy to analyze. for example, if you use a multiple checkbox function in google forms, the data will be in the same column in the spreadsheet separated by a delimiter and you will have to then use a function to separate those responses and group them. while the native google display will show the aggregated data, you will not be able to pull out the data that way. we compare the features of the most common online survey tools (qualtrics, redcap, google forms, survey monkey) available to us as university-based health sciences librarians in table 1. survey tool free account subscription account notes research section spotlight hypothesis, vol. 30, no. 1, fall/winter 2018 48 google forms (drive.google .com) includes <15 mb of storage unlimited surveys, responses, and questions >15 mb of storage month-to-month fee minimal data analysis available regardless of the account type offers no support. minimal types of questions available for questions with multiple response options, view individual, not group, responses to get the full picture. qualtrics (www.qualtr ics.com) 100 responses 10 outgoing emails 8 question types 1 active survey unlimited questions summary reports & filtering survey logic & randomization online reporting no expiration limits to features are based upon account type and subscription plan expiration occurs when subscription to product ends initially designed for business and market research surveys works for other types of research as well redcap (projectredc ap.org) one week trial 15 templates all collected or entered data is removed after the trial version concludes unlimited templates technical support for implementing and maintaining the product initially designed to support researchers and to encourage collaboration relatively easy to create a survey without using tutorials video tutorials and step-bystep instructions available in all versions survey monkey (surveymonk ey.com) create surveys with up to 10 questions or elements (i.e. question types, descriptive text, or images) collect up to 100 responses per survey unlimited questions and question types, descriptive text, images, etc. no limits to collecting numbers of responses learning curve is small. not easy to share unpublished surveys with non-subscribers for testing or editing purposes. support features are minimal. the knowledge base, while useful, is not easy to search table 1. comparison of select survey tools based on publicly available information as of may 2018. compiled by margaret hoogland. timing questions are more intensive for studies with a qualitative component, particularly analyzing any text or images or videos for your studies. will you ingest text into computer-assisted qualitative data analysis software tools (e.g. nvivo, atlas.ti, dedoose) for analyzing transcripts or other data resources, or will you annotate in word or by hand research section spotlight hypothesis, vol. 30, no. 1, fall/winter 2018 49 on print copies? if you have multiple coders involved, where will you maintain the code listing? multi-institutional studies have more complexity in terms of sharing data storage, access controls, agreeing on tools and preferences for types of analysis. regardless of the tools used, keep track of which members of the team performed which aspects of the research, as many journal publishers require a statement about how each author contributed to the research. learn about options for disseminating your research and how the journal article publication process works in the spring/summer 2019 issue of hypothesis. acknowledgements: this content was originally presented as a special content session "research roadmap: understanding the research process" at the medical library association (mla) annual meeting on may 20, 2018 co-sponsored by the research section, leadership and management section, hospital library section, and educational media and technologies section. the session was made possible by the coordinating efforts of brooke l. billman of the mla research section. we also thank the two anonymous peer reviewers for their thoughtful feedback. disclosure: heather holmes is the chair of the mla’s research and evidence-based practice curriculum committee. works cited 1. godshalk vm, sosik jj. mentoring and leadership. in: the handbook of mentoring at work: theory, research, and practice. ragins br, kram ke, eds. los angeles, ca: sage, 2007: 150. 2. johnson wb. on being a mentor. new york, ny: lawrence erlbaum associates, 2007: 2122. 3. the research mentor: creativity in research: an introduction. hypothesis 2010; 22 (2): 11-12. 4. eldredge j. the research mentor: creativity in research, part 2: classifying creativity. hypothesis 2011; 23 (1): 10-13. 5. eldredge j. the research mentor: creativity in research, part 3: age and creative potential. hypothesis 2012; 23 (2): 14-19. 6. eldredge j. the research mentor: interview with phillip eaton, m.d., on creativity in research. hypothesis 2012; 24 (1): 15-16. 7. eldredge j. the research mentor: authorship part one: defining the article author. hypothesis 2009; 21 (3): 11-14. 8. eldredge j. the research mentor: authorship part two: order of authors. hypothesis 2010; 22 (1): 8-11. 9. eldredge j. the research mentor: finding time for research. hypothesis 2008; 20 (2): 5-7. research section spotlight hypothesis, vol. 30, no. 1, fall/winter 2018 50 10. eldredge jd, ascher mt, holmes hn, harris mr. the research mentor: top-ranked research questions and systematic reviews. hypothesis 2013; 24(2): 19-20. 11. eldredge j. the research mentor: interview with library researcher jo dorsch. hypothesis 2008; 20 (3): 8-9. 12. eldredge j. the research mentor: interview with library researcher gary byrd. hypothesis 2009; 21 (1): 11-13. 13. eldredge jd. the research mentor: the long reach of a mentor’s influence. hypothesis 2014; 26 (1): 14-9. 14. university of new mexico. mentoring institute. conferences. available from: <https://mentor.unm.edu/conference/>. accessed 20 july 2018. 15. spencer aj, eldredge jd. roles for librarians in systematic reviews: a scoping review. j med libr assoc 2018;106(1):46-56. doi: 10.5195/jmla.2018.82. epub 2018 jan 2. 16. eldredge jd, ascher mt, holmes hn. an innovative model of evidence-based practice for other professions. j med libr assoc. 2015;103(2):100-2. 17. eldredge jd, ascher mt, holmes hn. an innovative model of evidence-based practice for other professions. j med libr assoc. 2015;103(2):100-2. 18. eldredge jd, hall lj, mcelfresh kr, warner td, stromberg tl, trost j, jelinek da. rural providers' access to online resources: a randomized controlled trial. j med libr assoc. 2016;104(1):33-41. 19. eldredge jd, kroth pj, murray-krezan c, hantak cm, weagel ef, hannigan gg. how accurately does the vivo harvester reflect actual clinical and translational sciences award-affiliated faculty member publications? j med libr assoc. 2015;103(1):19-21. 20. koufogiannakis d, brettle a. a new framework for eblip. in: being evidence based in library and information practice (pp. 11-18). london: facet publishing, 2016. 21. lindsay jm, oelschlegel s, earl m. surveying hospital nurses to discover educational needs and preferences. j med libr assoc 2017;105(3):226-232. doi:10.5195/jmla.2017.85. 22. how to make time for research and writing. chronicle of higher education. 2017 april 9. available from: <https://www.chronicle.com/article/how-to-make-time-forresearch/239724> accessed july 20, 2018. 23. preissle j. subjectivity statement. in: the sage encyclopedia of qualitative research methods. ed. lisa m. given. thousand oaks, ca: sage, 2008. 846. appendix questions from participants these questions came electronically from mla members prior to and during the session. we addressed questions received verbally during the session within the body of the article. prior to the session via mla-focus (n = 1) and email to research section and hospital libraries section email lists (n = 2) https://www.chronicle.com/article/how-to-make-time-for-research/239724 https://www.chronicle.com/article/how-to-make-time-for-research/239724 research section spotlight hypothesis, vol. 30, no. 1, fall/winter 2018 51 1. how can you move beyond research studies based primarily on your own work (which tends to be case studies), into doing research that will have bigger impact on the field? or put another way, how do you come up with research studies that will be impactful, but are still manageable to do within the confines of your job and institution? one way to move beyond your case study is to aggregate multiple case studies from similar institutions to strengthen the patterns of findings (see robert k. yin’s case study research and applications, 6th ed., 2018). another is to use your case study as pilot data for a different methodology that involves a control population or other attempts to capture or address confounding variables. 2. how to find collaborators, especially for hospital librarians that are solo or may have an idea, and don’t have time to pursue by themselves? 3. my question about research is how the heck can i do it as a solo librarian. addressing questions two and three together, one approach to advancing research as a solo librarian is to mentor a library science student doing research as part of a field experience in your library or to mentor student volunteers from other social science disciplines in your library. many students have research skills or requirements but struggle to come up with a meaningful question or real data, both of which you can provide. author alpi co-authors many of her research studies with student employees from diverse majors or library students doing field experiences. you can reach out to your local library school, state library association, or post your inquiry to the mla medical library education section (http://www.mlanet.org/page/section-landing-medicallibrary-education-section) to reach instructors of health sciences library students in online programs who may be interested. in a small library, choosing projects that can survive a longer timeframe or partnering with other areas of the institution with more capacity or the ability to advocate for answering the question, e.g. nursing research, continuing education, or quality improvement. during the session verbally or via the online form 1. how can you find a research mentor at an institution with promotion and tenure requirements similar to your own? the mla mentoring website allows you to look at the institution of the mentor, but beyond examining the title of the librarian for indicators of rank, does not offer any other information about those requirements. the website academic librarian status (https://academiclibrarianstatus.wordpress.com/) offers a list of academic institutions in the united states and elsewhere, sorted by the professional status of their librarians. although the professional school libraries may not always have the same requirements as the university libraries, this is a good starting point. mentors may have worked at multiple institutions with diverse requirements and therefore it is important to consider past experience beyond the mentor’s current institutional affiliation. http://www.mlanet.org/page/section-landing-medical-library-education-section http://www.mlanet.org/page/section-landing-medical-library-education-section https://academiclibrarianstatus.wordpress.com/ research section spotlight hypothesis, vol. 30, no. 1, fall/winter 2018 52 2. i’m gung ho about my research project but my more experienced research partners are not as excited. any advice on finding research partners who are excited to work with new researchers on their early projects? the first question that comes to mind is have you asked why they are not excited—it is the topic, the method, the role, the collaborators, or their own capacity? the piece of this article about what you offer mentors may provide some insight. if your institution is pressuring them to produce more advanced evidence, you may be better off finding peer collaborators or mentors outside your institution who do not feel the same pressure. you must be open to the idea that your research project is not as compelling to others as it is to you—check the literature to see how your proposed idea may be received if you decide to go forward. 3. what are your opinions of keeping daily stats of everyday work and do you know of ways to turn those daily stats into a story that highlight librarian/library impact? at nc state university we track public service daily work statistics and space usage using an open-source program called suma (https://www.lib.ncsu.edu/projects/suma) to gather data. it has built-in analytic tools. we track activities by user type and when we see changes or growth with a certain user population this invites us to dig deeper. for example, if we are doing more transactions with house officers, we then might interview a few to understand how we are impacting them. since intern/resident support is an important issue, telling a story of how we support them would resonate. we also tell library stories based on experiences, see https://www.lib.ncsu.edu/stories/supportingncsu-researchers-achieving-nih-public-access. 4. clinicians and academic faculty barely have time or interest for my short 7 question surveys, but the questions i have are best answered by surveys. how can i better engage my test audience? is there an alternative to surveys? one of the best ways to engage an audience is to have co-authors who represent that audience. can you co-author the survey with key opinion leaders or have them distribute the survey for you? if you have a library committee with members from your test audience, they might be able to assist. depending on the topic of your surveys, it may be easier to do a mini-focus group with representatives of the test audience at a meeting. feel free to contact author alpi with more details to receive more focused feedback. 5. when your paper is submitted, and you are about to revise and resubmit, can the author ask for new reviewers? probably not. when you are asked to revise and resubmit your manuscript, the editor of the journal most likely wants the original reviewers to re-evaluate the revised manuscript to ensure that your revisions have sufficiently alleviated their specific concerns. moreover, inviting new reviewers at this point would likely only prolong the peer review process and create more “hoops” for you to jump through. however, if you have serious concerns about a reviewer—perhaps you feel that they did not carefully read your manuscript or https://www.lib.ncsu.edu/projects/suma research section spotlight hypothesis, vol. 30, no. 1, fall/winter 2018 53 were unreasonably critical—you can privately express those concerns in an email or formal letter to the editor, who will take your concerns into consideration when weighing the reviewers’ opinions to arrive at a final decision regarding acceptance. national library of medicine (us). digital collections [internet]. bethesda (md): medline keyboard in the reading room [196?]. available from: http://resource.nlm.nih.gov/101445854 http://resource.nlm.nih.gov/101445854 volume 28, no 1 fall 2016 hypothesis the journal of the research section of the medical library association hypothesis, vol. 28, no. 1, fall 2016 1 volume 28, no 1 fall 2016 hypothesis the journal of the research section of the medical library association editorial introducing the new hypothesis christine marton……………………………………………4 original research an examination of research topics in the journal of the medical library association and the bulletin of the medical library association: quantifying the importance of research to medical librarians over time christine marton .............................................................6 customizing the ipad to support a clinical fellowship in radiology: a qualitative study antonio p. derosa and delia m. keating….…..……23 research section news mla 2016 annual meeting research award winners sandra de groote, jennifer lyon, terry henner, & kim powell ……………………………………………… 33 mla 2016 annual meeting research section programs and meetings christine marton and marie ascher………………...45 linear regression from wikimedia commons, the free media repository https://commons.wikimedia.org/wiki/ file:linear_regression.svg metro toronto convention centre by coolcaesar own work, cc by-sa 3.0, https://commons.wikimedia.org/w/ index.php?curid=33969198 hypothesis (issn 1093-5665) is the official journal of the research section of the medical library association. hypothesis is indexed in the cumulative index to nursing and allied health literature™ (cinahl). hypothesis is available online at http://www.mlanet.org/p/cm/ld/fid=503. https://commons.wikimedia.org/wiki/ https://commons.wikimedia.org/w/%20index.php?curid=33969198 https://commons.wikimedia.org/w/%20index.php?curid=33969198 http://www.mlanet.org/p/cm/ld/fid=503 hypothesis, vol. 28, no. 1, fall 2016 2 editors christine marton, phd adjunct instructor university of toronto faculty of information christine.marton@utoronto.ca erin foster, msls data services librarian indiana university ruth lilly medical library erdfost@iu.edu editorial board abby adamczyk, mls drexel libraries liaison to the coas departments of biology, chemistry, and bees ala99@drexel.edu brooke l. billman, ma medical librarian, governance services college of american pathologists bbillma@cap.org susan fowler, mls washington university school of medicine systematic review services coordinator fowler@wustl.edu krystal bullers, mls usf health shimberg health sciences library emerging technologies librarian, pharmacy liaison kbullers@health.usf.edu ayaba logan, mlis, mph research and education informationist musc libraries medical university of south carolina loganay@musc.edu carol perryman, phd assistant professor school of library and information studies texas woman's university cperryman@twu.edu all research section board members, ex-officio mailto:christine.marton@utoronto.ca mailto:erdfost@iu.edu mailto:ala99@drexel.edu mailto:bbillma@cap.org mailto:fowler@wustl.edu mailto:loganay@musc.edu hypothesis, vol. 28, no. 1, fall 2016 3 research section officers and chairs 2016-2017 chair marie ascher, ms, mph, ahip lillian hetrick huber endowed director, health sciences library new york medical college marie_ascher@nymc.edu chair-elect/program chair michelle beth bass, phd, msi science research services librarian university of chicago libraries science libraries university of chicago mbbass@uchicago.edu immediate past chair nominating committee chair section council rep emily mazure, msi biomedical research liaison librarian duke university emily.mazure@duke.edu secretary/treasurer tony nguyen, mlis, ahip technology and communications coordinator national network of libraries of medicine se/a region mentoring co-coordinator maureen clark, mhs, mlis, ahip, chis library of the health science university of illinois – chicago mentoring co-coordinator jennifer staley, mlis medical librarian university hospitals rainbow babies and children's hospital community manager margaret hoogland, mls, ahip assistant professor, clinical medical librarian, health science library mulford health sciences library university of toledo chair, awards committee kimberly r. powell, mlis life sciences informationist woodruff health sciences center library emory university education chair martha f. earl, msls, ahip, chis associate professor associate director, preston medical library university of tennessee membership committee chair beatriz g. varman, mlis liaison librarian houston academy of medicinetexas medical center library research agenda committee chair, international research collaboration chair jonathan eldredge, mls, phd, ahip evidence based and translational science collaboration coordinator health science library & informatics center university of new mexico strategic planning committee chair susan lessick, ma, mls, ahip, fmla distinguished librarian librarian emeritus university of california, irvine mailto:mbbass@uchicago.edu editorial hypothesis, vol. 28, no. 1, fall 2016 4 introducing the new hypothesis christine marton it is my great pleasure to welcome you to a new issue of hypothesis! as the profession of health sciences librarianship continues to undertake new roles – informationist, data librarian, research librarian – among others, it becomes more embedded in the biological and health sciences. working alongside clinicians, health sciences librarians are increasingly becoming researchers in their own right, both within their profession and within the professions in which they are embedded in their workplaces: universities, healthcare institutions, governments, and biopharma corporations. the new hypothesis reflects this reality, both in the composition of the new editorial board and in the topics of the research articles presented in this issue. like many of my mla colleagues, i have a dual background in health sciences and library and information science. as an academic and a former high school science teacher, i have taught many courses over a twenty-year period. i have also been actively involved in research in several disciplines and have utilized several approaches (experimental, observational, and textual) with several types of study designs (quantitative, qualitative, and mixed-methods). the members of the editorial board share with me these characteristics – they are both teachers and researchers. beyond the classic and still vital librarian roles of collection development and management, preparation of library guides, hiring and managing staff, and liaising with department heads, health sciences librarians act as instructors by teaching clinicians and clinical students how to effectively search the journal literature and distinguish quality information sources from less reputable publications. health sciences librarians act as researchers by performing literature reviews and co-authoring systematic reviews of the health sciences literature required by clinicians to write grant proposals and keep track of key findings in their fields. it is only appropriate that i take this opportunity to introduce my co-editor, erin d. foster, msls. erin foster is the newly appointed data services librarian at indiana university school of medicine's ruth lilly medical library. she is a former national library of medicine fellow where she spent time at the national library of medicine, on the national institutes of health campus in bethesda md, and oregon health & science university in portland or. erin received her editorial hypothesis, vol. 28, no. 1, fall 2016 5 master’s in library science at the university of north carolina, chapel hill and worked as a graduate research assistant at the university's health sciences library. she received her undergraduate degree at the university of california, santa cruz. her research interests include data curation and management practices within health sciences communities, the development of standards and technologies to enable public access to research, and the promotion of open science efforts. erin foster’s academic background and work experience exemplifies the new breed of health sciences librarian. please welcome her onboard! the new hypothesis editorial board also deserves a hearty welcome. i would like to introduce to you our hard working editorial board members, several of whom have been actively involved in reviewing the contributions to this issue. listed in alphabetical order, they are: abby adamczyk, brooke billman, krystal bullers, susan fowler, ayaba logan, and carol perryman. representing both the geographic range of the united states of america and the many roles undertaken by health sciences librarians and academics, they collectively offer a comprehensive knowledge base to this publication. i would also like to take the opportunity to thank the outgoing editorial board, who have contributed extensively to past issues of hypothesis: kristine alpi, leslie behm, ellen detlefsen, and jonathan eldredge. as well, thanks are due to the outgoing editorial team of brooke billman and emily mazure for the 2015 issue of hypothesis. in this issue of hypothesis, two original research articles are presented. both were reviewed twice by a minimum of three reviewers with the author names and institutional affiliations removed to minimize reviewer bias. the first article is a quantitative bibliometric study of research topics in the highest ranked health sciences librarianship journal – the journal of the medical library association, formerly known as the bulletin of the medical library association. the second article is a survey study of the use of ipad mini tablets pre-loaded with apps/resources/websites/tools by clinical fellows in the breast imaging service (department of radiology) at the memorial sloan kettering cancer center in new york city. as well, this issue features extensive coverage of the research award winners at mla’16 in toronto and the research section’s conference programs. i hope you find the content informative and are inspired to submit an article about your current research to hypothesis. as fall turns into winter and the polar vortex approaches, we wish you joy and happiness for the holiday season. we look forward to your participation in making hypothesis a success in 2017. original research marie t. ascher jonathan d. eldredge hypothesis, vol. 29, no. 1, spring/summer 2017 5 mla research section’s research agenda committee systematic review project: a status report marie t. ascher, ms, mph, ahip lillian hetrick huber endowed director health sciences library new york medical college valhalla, ny jonathan d. eldredge, mls, phd, ahip associate professor evidence based and translational science collaboration coordinator health science library & informatics center biomedical informatics research, training, and scholarship unit university of new mexico albuquerque, nm jeldredge@salud.unm.edu background ambitious intentions, however well-conceived and embraced, can take time to manifest. our elected and appointed mla leaders have encouraged us for over 20 years to integrate the best evidence into our professional practices. these broad-based investments at the policy level now are yielding tangible results. the first mla research policy, using scientific evidence to inform practice, emphasized applying research evidence when making decisions [1]. this 1995 policy statement, along with mla president rachel anderson’s 1997 inaugural address, helped spark the international evidence based library and information practice (eblip) movement that continues to thrive today [2, 3, 4]. in fact, most mla presidents since 1997 have called on us to integrate research evidence into our practices [5]. the newest mla research policy, the research imperative, called for the research section to articulate an mla research agenda [6]. the research section’s research agenda committee original research marie t. ascher jonathan d. eldredge hypothesis, vol. 29, no. 1, spring/summer 2017 6 conducted two delphi studies in 2008 and 2011 to identify the most important and answerable research questions facing the profession [7]. the second delphi study led the committee to develop guidelines for voluntary teams to create systematic reviews for assembling the best evidence to form the tentative answers to each of the 15 top-ranked research questions [8]. the teams were to act largely autonomously. the 15 teams, consisting of over 200 medical librarians worldwide have made progress, for the most part, in completing their systematic reviews [9, 10]. the principal benefits of these systematic reviews will be as evidence resources for answering these top-ranked questions in addition to acting as blueprints pointing to further research needed to build our knowledge base strategically. health sciences librarians have been integral members of systematic review teams outside librarianship since the 1990s, particularly in medicine [11, 12, 13, 14, 15]. many guidelines and experts on systematic reviews point to the need for librarians to serve on systematic review teams [16, 17, 18]. our own profession has produced over 90 systematic reviews on subjects related to library and information practice [19]. these systematic reviews frequently confirm the recurring observation that our own knowledge base lacks sufficient amounts of rigorous research evidence. a large percentage of our evidence also resides within the gray literature rather than the peer reviewed literature, which poses challenges to identification and critical appraisal. this paper provides a status report on the mla research section’s research agenda committee systematic review project. there has not been a comprehensive report on the project since the 2015 open forum held at the annual meeting of the medical library association in austin, tx, where representatives from all active teams provided reports [20]. to date, teams have continued to vary in their rate of progress. some teams have moved through the process relatively quickly while others are regrouping. an evaluation of the overall project experience from the point of view of the participants is forthcoming. method & results to complete this inventory of progress, the fifteen team leaders were surveyed as to their current progress and to report any research outputs to date. the status categories they had to original research marie t. ascher jonathan d. eldredge hypothesis, vol. 29, no. 1, spring/summer 2017 7 choose from represent discrete phases of the systematic review process: 1) not started or regrouping; 2) very early: question clarification or earlier; 3) early: search strategy development 4) mid: screening abstracts; 4) late mid: screening articles; 5) later: data extraction and analysis; 6) nearing completion: manuscript preparation; 7) complete and published. all teams with current leadership responded. the remaining two were described by the first author of this paper as “not started.” there is one team currently without an assigned team leader and has stalled at square one as of this writing. results of this inquiry are reported in table 1. table 1. team leaders’ reported team progress three of the fifteen teams have completed their reviews and published articles. six other teams are late in the systematic review process, two teams are in the middle, and four teams are very early in the process. three teams are either regrouping or currently without leadership. the loss of leadership has been an issue for several teams throughout this ambitious project. table 2 lists status by team and research outputs, including publication and presentation information. 3 1 0 1 1 4 2 3 0.0% 5.0% 10.0% 15.0% 20.0% 25.0% 30.0% not started very early early mid late mid later nearing complete original research marie t. ascher jonathan d. eldredge hypothesis, vol. 29, no. 1, spring/summer 2017 8 the original questions that emerged from the second delphi study [7] are listed although teams were tasked with reworking their questions to make them suitable for conducting a review. table 2. team progress and research outputs team # 1. question: there are still a number of relevant questions from the 2008 research agenda, but to me this is most critical: "what is the quantifiable evidence that the presence of a librarian, not just information resources, improves patient outcomes, increases research dollars, improves student outcomes (e.g., better board scores), or increases hospital intelligence (e.g., if the top hospitals have access to hospital librarians/libraries)?" status: complete and published presentations: perrier l, farrell a, weiss a, lightfoot d, aaronson e, connor e, epstein brown ha, muellenbach jm, allee n, ayala p, kenny t, constantinescu t, brigham t. effects of librarianprovided services in health care: a systematic review. [contributed paper.] proceedings of the canadian health libraries association/association des bibliothèques de la santé du canada. 2014 june 16-20. montreal, qc, canada. j can health libr assoc. 2014; 34(2):92-93. https://journals.library.ualberta.ca/jchla/index.php/jchla/article/view/22905/17060 publications: perrier l, farrell a, ayala ap, lightfoot d, kenny t, aaronson e, allee n, brigham t, connor e, constantinescu t, muellenbach j, epstein ha, weiss a. effects of librarian-provided services in healthcare settings: a systematic review. j am med inform assoc. 2014;21(6):1118-24. https://www.ncbi.nlm.nih.gov/pmc/articles/pmc4215058/ note: this paper was the winner of the 2015 ida and george eliot prize which is given to the authors of the work published in the preceding calendar year that has been judge most effective in furthering medical librarianship. team # 2. question: is there a significant difference in patient outcomes (or research output or educational outcomes) between institutions with and without libraries? status: very early – question clarification https://journals.library.ualberta.ca/jchla/index.php/jchla/article/view/22905/17060 https://www.ncbi.nlm.nih.gov/pmc/articles/pmc4215058/ original research marie t. ascher jonathan d. eldredge hypothesis, vol. 29, no. 1, spring/summer 2017 9 team # 3. question: what is the added value libraries bring to education, research, and patient care in the health sciences and health care fields? even if it is not possible to quantify benefits, documenting qualitative research results rigorous enough to stand the scrutiny of administrators and researchers would be of great value. status: later data extraction and analysis team # 4. question: low health literacy can result in medication errors, noncompliance of treatment regimes, poor health outcomes and even death. what is the role of the medical librarian with health care providers, community organizations, local public libraries and members of the public to improve health literacy among entire communities? status: later – data extraction and analysis presentations: klem m, devine pj, el-khayat ym, gutzman ke, knehans a, mills tn, oren ga, perryman cl, saleh aa, unno zp, vardell e. librarians and health literacy: a scoping review. presented at: medical library association annual meeting. 2015 may 14-20. austin tx. http://www.mlanet.org/d/do/1924 team # 5. question: what are the information needs of practicing physicians and other health care workers? the 1985 covell article is still heavily cited but was published way back in 1985. the information environment has changed dramatically. we need to update that study in lite of new educational strategies, resources, technology and social networks. status: nearing completion manuscript preparation team # 6 question: the explosion of information, expanding of technology (especially mobile technology), and complexity of healthcare environment present medical librarians and medical libraries opportunities and challenges. to live up to the opportunities and challenges, what kinds of skill sets or information structure do medical librarians or medical libraries are required to have or acquire so as to be strong partners or contributors of continuing effectiveness to the changing environment? status: nearing completion – manuscript preparation presentations: http://www.mlanet.org/d/do/1924 original research marie t. ascher jonathan d. eldredge hypothesis, vol. 29, no. 1, spring/summer 2017 10 anderson pf, bickett s, doucette j, herring pr, kammerer j, kepsel a, lyons t, mclachlan s, tonnison i, wu l. tools for building our information future: emerging technologies vital to medical libraries” health library group conference, chartered institute of library and information professionals. 2014 july 24-25. university of oxford, u.k. anderson pf, bickett s, doucette j, herring pr, kammerer j, kepsel a, lyons t, mclachlan s, tonnison i, wu l. “tools for building our information future: emerging technologies vital to medical libraries.” [poster.] annual meeting medical library association. 2014 may 16–21. chicago, il. note: this poster won an 2014 mla research section, research award: posters, honorable mention anderson pf, bickett s, doucette j, herring pr, kammerer j, kepsel a, lyons t, mclachlan s, tonnison i, wu l. future technological practices: medical librarians’ skills and information structures for continued effectiveness in a changing environment. 2014 may 16–21. annual meeting medical library association, chicago, il. anderson pf, bickett s, doucette j, herring pr, kammerer j, kepsel a, lyons t, mclachlan s, tonnison i, wu l. tools for building our information future: emerging technologies vital to medical libraries. 2014 september 19. hli/hla joint conference 2014. state library of victoria, melbourne, victoria, australia. https://www.alia.org.au/sites/default/files/hla%20news-conference2014.pdf. anderson pf, bickett s, doucette j, herring pr, kepsel a, lyons t, mclachlan s, shannon c, wu l. developing a replicable methodology for automated identification of emerging technologies in healthcare. annual meeting of the medical library association. 2016 may 13–18. toronto, canada. team # 7. question: does what we do matter? longer form: do the resources we provide materials, reference services, and educational offerings make a difference to our customers save lives, shorten length of stay, improved educational outcomes, increase research dollars, improve research results? status: late mid screening articles https://www.alia.org.au/sites/default/files/hla%20news-conference2014.pdf original research marie t. ascher jonathan d. eldredge hypothesis, vol. 29, no. 1, spring/summer 2017 11 presentations: glynn la, sakmar k, lalla nj, berry r, kim c, geldenhuys pr, lawton a, siebert jl, mcclurg c, clemans-taylor l, gadd k, ettien a. the value to and impact of health sciences libraries and information services on academic and clinical practices: a systematic review. [paper.] annual meeting of the medical library association. 2014 may 16–21. chicago, il. team # 8. question: how do we provide information support in a clinical world that functions based on electronic medical records systems and other similar informatics platforms and tools. what is the library's role, if any, in providing preclinical education with respect to informatics applications like electronic medical records systems? status: mid screening abstracts team # 9. question: do health sciences libraries and librarians have any measureable (statistically significant) positive impacts on consumer health, the outcomes of medical care, the productivity of biomedical researchers and the knowledge obtained by graduates of biomedical and health sciences training programs, and at what total cost? status: later data extraction and analysis presentations: henderson me, crum ja, fatkin kj, gagnon m-m, nguyen t, taylor m, and vrabel m. do health sciences libraries and librarians have an impact on the cost of health care and research? a systematic review. [poster.] annual meeting of the medical library association. 2015 may 14-20. austin, tx. http://scholarscompass.vcu.edu/libraries_present/43/ team # 10. question: how best to objectively document library/librarian impact on the 'bottom line' (time, money saved, shorter length of stay, roi for expensive electronic resources, support training programs/magnet status, funded research support, etc.)? status: completed and published presentations: collins, p. counter-measures: a systematic review of the measurement tools used to demonstrate the impact of libraries in a clinical setting. [presentation.] health libraries group. 2016 sept 15-16. scarborough, united kingdom. http://www.cilip.org.uk/sites/default/files/documents/pam_collins.pdf. http://scholarscompass.vcu.edu/libraries_present/43/ http://www.cilip.org.uk/sites/default/files/documents/pam_collins.pdf original research marie t. ascher jonathan d. eldredge hypothesis, vol. 29, no. 1, spring/summer 2017 12 madden a. "you say tomayto…”: alternative viewpoints on international collaboration. [presentation.] health libraries group 2014 july 24. oxford, united kingdom. https://www.cilip.org.uk/sites/default/files/documents/anne%20madden.pdf publications: madden a, collins p, mcgowan s, stevenson p, castelli d, hyde l, desanto k, o'brien n, purdon m, delgado d. 2016. demonstrating the financial impact of clinical libraries: a systematic review. health info lib j. 2016; 33(3):172-189. http://onlinelibrary.wiley.com/doi/10.1111/hir.12151/abstract team # 11. question: as a profession, how do we measure our impact in our environment—be it clinical or academic—in such a way that it influences the decision makers in our institutions? [i "stole" this from the previous study, but i think that it is still the most important question facing us.] status: not started – regrouping team # 12. question: does the intervention/instruction/assistance of a professional medical librarian have a long term impact on the information seeking behaviors of health care professionals? status: not started – regrouping team # 13. question: what are the most effective instructional methods for teaching informatics/knowledge management/ebp within health sciences curricula? status: complete and published presentations: pannabecker v, dennison cc, holyoke an, farrell a, machel v, marton c, o'brien kk, swanberg sm, thuna m. creative convergence: conducting a systematic review project through cross-institutional, distance collaboration. [presentation.] quintessential joint chapter meeting of the medical library association. 2014 oct 14. denver, co. https://vtechworks.lib.vt.edu/handle/10919/50634 holyoke an, dennison cc, farrell a, machel v, marton c, o'brien kk, pannabecker v, swanberg sm, thuna m. systematically assessing methods used by librarians to teach https://www.cilip.org.uk/sites/default/files/documents/anne%20madden.pdf http://onlinelibrary.wiley.com/doi/10.1111/hir.12151/abstract https://vtechworks.lib.vt.edu/handle/10919/50634 original research marie t. ascher jonathan d. eldredge hypothesis, vol. 29, no. 1, spring/summer 2017 13 evidence-based practice: what works best. [presentation.] quintessential joint chapter meeting of the medical library association. 2014 oct 14. denver, co. dennison cc, farrell a, gore g, swanberg sm, machel v, marton c, o'brien kk, pannabecker v, thuna m. effectiveness of instructional methods used by librarians for teaching evidence based practice: a systematic review. canadian health libraries association annual meeting. 2014 june 20. montreal, quebec, canada. farrell a, dennison cc, gore g, holyoke an, machel v, marton c, o'brien kk, pannabecker v, swanberg sm, thuna m. an initiation to systematic review. hawaii-pacific chapter of the medical library association annual meeting. 2014 april 4. honolulu, hi. publications: swanberg sm, dennison cc, farrell a, machel v, marton c, o'brien kk, pannabecker v, thuna m, holyoke an. instructional methods used by health sciences librarians to teach evidencebased practice (ebp): a systematic review. j med libr assoc. 2016 jul; 104(3): 197–208. https://www.ncbi.nlm.nih.gov/pmc/articles/pmc4915637/ team # 14. question: in medical schools where librarians are included in the curriculum, do the students have a greater degree of information literacy than students in schools where librarians are not part of the curriculum? status: not started regrouping team # 15. question: what skills and knowledge must librarians possess in order to be able to design tools to help researchers visualize, mine, and otherwise manage large and complex data gathered during both quantitative and qualitative research? status: later data extraction and analysis presentations: boden c, adamczyk a, ambriz l, billman bl, booth a, clark e, engwall k, johnson r, millernesbitt a, morris m, woznica a. librarian knowledge and skills of tools for visualizing, mining and managing large and complex research data: a systematic review. [poster.] annual https://www.ncbi.nlm.nih.gov/pmc/articles/pmc4915637/ original research marie t. ascher jonathan d. eldredge hypothesis, vol. 29, no. 1, spring/summer 2017 14 meeting of the medical library association. 2015 may 14-20. austin, tx. http://eventscribe.com/2015/mla/twitterposter.asp?posterid=38894 miller-nesbitt a, boden c, adamczyk a, ambriz l, billman bl, booth a, clark e, engwall k, johnson r, morris m, woznica a. applying the best-fit framework to systematic review data extraction. [presentation.] 7th qualitative and quantitative methods in libraries international conference (qqml2015). 2015 may 26-29. paris, france. morris m, boden c & nesbitt-smith a. (2014). distributed collaborative virtual systematic reviewing: a blueprint for the future? [presentation.] canadian health libraries association annual meeting, june 16-20, 2014, montreal, qb. https://journals.library.ualberta.ca/jchla/index.php/jchla/article/view/22906/17075 discussion fifteen teams conducting systematic reviews on the top-ranked research questions met with varying success. one of the teams completed their systematic review in about a year and their paper was the 2015 winner of the medical library association’s ida and george eliot prize, awarded annually for a work that has been judged most effective in furthering medical librarianship [21]. others have moved more slowly, but most are on track to completion. the project has resulted in presentations at meetings in the us, australia, france, canada, and the uk and three important systematic review papers have been published so far [21, 22, 23] with several in the pipeline. future work of the committee involves a centralized web resource summarizing the outcomes of the project as well as an overall evaluation of the project to inform other such potential endeavors. although the actual overall time of completion has exceeded preliminary expectations, the project is continuously yielding valuable information and will continue to be a landmark in health sciences librarianship research. references 1 dalrymple pw, bastille jd, bradley j, dee cr, humphreys bl, marshall jg, weller ac, webb re. using scientific evidence to improve information practice. chicago, il: medical library http://eventscribe.com/2015/mla/twitterposter.asp?posterid=38894 https://journals.library.ualberta.ca/jchla/index.php/jchla/article/view/22906/17075 original research marie t. ascher jonathan d. eldredge hypothesis, vol. 29, no. 1, spring/summer 2017 15 association, 1995. [cited 10 march 2017]. <http://www.mlanet.org/p/do/sd/topic=574&sid=2948>. 2 anderson rk. inaugural address. proceedings, ninety-seventh annual meeting medical library association, inc. seattle, washington may 24-28, 1997. bull med libr assoc. 1998 jan;86(1):137-9. pubmed pmid: 16018060; pubmed central pmcid: pmc226342. 3 eldredge j. evidence-based librarianship; a commentary. hypothesis. 1997 fall; 11(3): 4-7. 4 eldredge jd. evidence-based practice. in: introduction to health sciences librarianship. edited by sandra wood. binghamton, ny: haworth press, 2008: 245-69. 5 eldredge jd. integrating research into practice. journal of the medical library association 2016 oct; 104 (4): 333-7. 6 medical library association. the research imperative: the research policy statement of the medical library association. appendix 4: action plan. section one [internet]. chicago: medical library association, 2007 [cited 10 march 2017]. <http://www.mlanet.org/p/do/sd/topic=574&sid=2947 >. 7 eldredge jd, ascher mt, holmes hn, harris mr. the new medical library association research agenda: final results from a three-phase delphi study. j med libr assoc. 2012 jul;100(3):214-8. doi: 10.3163/1536-5050.100.3.012. pubmed pmid:22879811; pubmed central pmcid: pmc3411260 8 ascher mt, eldredge jd, holmes hn, harris mr. the mla research agenda: appraising the best available evidence. proposed guidelines to the mla research section executive committee. working paper. 2012 nov 2. [unpublished]. [cited 10 march 2017]. <http://repository.unm.edu/handle/1928/24634>. 9 ascher mt, holmes hn, eldredge jd. addressing the mla research agenda questions: where are we now? poster. international congress of medical librarianship/medical library association annual meeting. boston, ma. may 4-8, 2013. 10 holmes hn, ascher mt, eldredge jd. the mla research agenda systematic review project. open forum. medical library association annual meeting. austin, tx. may 17, 2015. 11 mulrow cd. rationale for systematic reviews. bmj. 1994 sep 3; 309(6954):597-9. pubmed pmid: 8086953; pubmed central pmcid: pmc2541393. original research marie t. ascher jonathan d. eldredge hypothesis, vol. 29, no. 1, spring/summer 2017 16 12 dickersin k, scherer r, lefebvre c. identifying relevant studies for systematic reviews. bmj. 1994 nov 12; 309(6964):1286-91. pubmed pmid: 7718048; pubmed central pmcid: pmc2541778. 13 hunt dl, mckibbon ka. locating and appraising systematic reviews. ann intern med. 1997 apr 1; 126(7):532-8. pubmed pmid: 9092319 14 beverley ca, booth a, bath pa. the role of the information specialist in the systematic review process: a health information case study. health info libr j. 2003 jun; 20(2):65-74. pubmed pmid: 12786905. 15 harris mr. the librarian's roles in the systematic review process: a case study. j med libr assoc. 2005 jan; 93(1):81-7. pubmed pmid: 15685279; pubmed central pmcid: pmc545126. 16 institute of medicine. finding what works in health care: standards for systematic reviews. washington, dc; national academies press; 2011. 17 foster mj. an overview of the role of librarians in systematic review: from expert searcher to project manager. j eur assoc health inf libr. 2015;11(3):3-7. 18 metzendorf mi. why medical information specialists should routinely form part of teams producing high quality systematic review: a cochrane perspective. j eur assoc health inf libr 2016; 12 (4): 6-9. 19 koufogiannakis d. lis systematic reviews [internet]. jan 23 2012. [cited 17 march 2017]. <http://lis-systematic-reviews.wikispaces.com/welcome >. 20 eldredge j, ascher m, holmes hn. mla research agenda systematic review project team updates presentation. annual meeting of the medical library association. 2015 may 17. austin, tx. 21 perrier l, farrell a, ayala ap, lightfoot d, kenny t, aaronson e, allee n, brigham t, connor e, constantinescu t, muellenbach j, epstein ha, weiss a. effects of librarian-provided services in healthcare settings: a systematic review. j am med inform assoc. 2014;21(6):1118-24. <https://www.ncbi.nlm.nih.gov/pmc/articles/pmc4215058/>. 22 madden a, collins p, mcgowan s, stevenson p, castelli d, hyde l, desanto k, o'brien n, purdon m, delgado d. 2016. demonstrating the financial impact of clinical libraries: a systematic review. health info lib j. 2016; 33(3):172-189. <http://onlinelibrary.wiley.com/doi/10.1111/hir.12151/abstract>. original research marie t. ascher jonathan d. eldredge hypothesis, vol. 29, no. 1, spring/summer 2017 17 23 swanberg sm, dennison cc, farrell a, machel v, marton c, o'brien kk, pannabecker v, thuna m, holyoke an. instructional methods used by health sciences librarians to teach evidencebased practice (ebp): a systematic review. j med libr assoc. 2016 jul; 104(3): 197–208. <https://www.ncbi.nlm.nih.gov/pmc/articles/pmc4915637/>. --peer reviewed article hypothesis , vol. 35, no. 1, 2023 an introduction to statistics for librarians (part two): frequency distributions and measures of central tendency caitlin j. bakker, mlis, ahipa adiscovery technologies librarian, dr. john archer library & archives, university of regina, regina, saskatchewan, canada, https://www.orcid.org/0000-0003-4154-8382 , caitlin.bakker@uregina.ca in part one of this column, the different types of data were discussed.1 understanding the type of data is essential to interpreting them. if the type of data isn’t correctly identified, it’s not possible to answer some fundamental questions accurately. one of these fundamental questions is “what’s the average value?” this is often the building block for more advanced statistical tests. in statistical terms, this question is asking us for the central tendency of the data. the central tendency is a single value that represents the midpoint of the data set. it tells us what is “average” or “normal” in the data set. there are three different ways to measure central tendency: mode, median, and mean. the measure chosen will depend on the type of data and the distribution of that data. frequency distributions frequency distributions describe how often values occur in a dataset. these may be presented in tables, but are also often visualized as histograms. a distribution can be normal, or it can be skewed, either positively or negatively. a normal distribution is called a bell curve because it is symmetrical. although there may be outliers, meaning extreme positive or negative values, these are relatively comparable and the majority of data clusters evenly around a center point. when a distribution is skewed, it means there are some particularly high or low values that make the distribution asymmetrical. figure 1: skewed and normal distributions in figure 1, distribution b is a bell curve or normal distribution, whereas distribution a is left or negatively skewed, and distribution c is right or positively skewed. left 1 https://www.orcid.org/0000-0003-4154-8382 https://orcid.org/ mailto:caitlin.bakker@uregina.ca peer reviewed article hypothesis , vol. 35, no. 1, 2023 skewness or negative skewness refers to the fact that some extreme low outliers, or negative values, are present and those negative values are what is causing the asymmetry. the inverse is true of right or positive skewness, where some particularly high outliers are causing asymmetry. distribution matters because the measure of central tendency–that is, the way the midpoint is calculated–depends upon the distribution of the data. if we use the wrong measure of central tendency, the result will show the midpoint of the data to be either higher or lower than what is accurate. measures of central tendency table 1 describes the three measures of central tendency. table 1. measures of central tendency and data types mode median mean the most frequently occurring value the middle value the average of all values nominal x ordinal x x interval x x x ratio x x x mode the mode refers to the most frequently occurring value in the dataset. the benefit of the mode is that we can use this with any type of data, including nominal data. for example, if we are conducting a review of a library’s chat reference service, the chat service may begin by prompting users to identify their patron type (e.g., student, staff, faculty). this would be nominal data. the mode here would be whichever value occurs most frequently. if the chat service is used by 40 students, 30 staff, and 20 faculty, the mode would be “student.” if we review a library’s chat reference service and look at how long each chat interaction was during one day, we would be looking at ratio data. we might have the following data: 1.5 minutes, 3 minutes, 3 minutes, 5 minutes, 7.5 minutes, 9 minutes, 10 minutes. the mode would be 3 minutes because this is the value that occurs most frequently. however, the most frequent value may not be truly representative of the data and its midpoint, as is the case in these examples. in the first case, while “student” may be the most common patron type, it still represents less than 45% of all patrons, so concluding that the “average patron” is a student overlooks the majority of chat reference patrons. in the case of chat reference time, an “average time” of 3 minutes is quite low when we look at it in the context of all of the data, given that only one of seven chat transactions was less than three minutes. median the median is the middle value when all of the values in the data set are arranged in ascending or descending order. using the above example of chat reference times, the median length of a chat reference interaction was 5 minutes, because 5 is the fourth of seven values. you can think of the median as the point at which 50% of the data will 2 peer reviewed article hypothesis , vol. 35, no. 1, 2023 be above that point, and 50% of the data will fall below that point, as shown in table 2. table 2. example calculation of the median 1 minute 3 minutes 3 minutes 5 minutes 7.5 minutes 9 minutes 10 minutes 50% of values below the median median value 50% of values above the median the median can be calculated for most types of data. however, because the median requires that data be ranked or placed in order, we cannot use this as a measure of nominal data because nominal data cannot be ranked. the median is less impacted by outliers. this makes it a helpful measure when the data are not normally distributed, meaning that the distribution is skewed either positively or negatively rather than being symmetrical. mean the mean is what people often think of when describing the average of a set of numbers. a mean involves adding all of the values in a dataset together, and then dividing it by the number of values. using the chat reference example, we would calculate a total of 39 minutes spent conducting chat reference, and divide that by the seven chat reference interactions to have a mean value of 5.6 minutes, as seen in figure 2. figure 2: example calculation of the mean to have an accurate mean, the data should be normally distributed. if the data skew right, meaning that there are some larger values as outliers, the mean will be higher than the median and will overestimate the midpoint. where the data skew left, meaning that there are some smaller values as outliers, the mean will be lower than the median and will underestimate the midpoint. alongside the mean, researchers will generally also present the standard deviation. the standard deviation is a measure of variance. the smaller the standard deviation, the more the data tends to be concentrated around the mean, or has less variation. a larger standard deviation means that the data are more spread out from the mean, or it has more variation. to learn more about standard deviations and their meaning, consult illowsky and dean.2 3 peer reviewed article hypothesis , vol. 35, no. 1, 2023 why does this matter? imagine the data in figure 3 represent responses from patrons at two different libraries. the patrons are asked to rate their satisfaction with library services on a scale from 1 to 6, where 1 means very unsatisfied and 6 means very satisfied. figure 3: patron satisfaction at two libraries to compare these two groups, we would compare a measure of central tendency. however, the measure we select would give us quite different results. in table 3, we see the various midpoints calculated based on the data in the visualization above. table 3. patron satisfaction at two libraries library a library b mean 4.1 2.5 median 4.5 2.0 mode 5 1 for library a, we have a mean of 4.1, a median of 4.5, and a mode of 5. when we try to express the “average” level of patron satisfaction with library services, if we define the “average” using the mean, we would say patron satisfaction was 4.1/6. however if we defined “average” as the median, we see that patron satisfaction is 4.5/6. because the mean doesn’t account for the fact that the data are left (negatively) skewed, using the mean would lead us to conclude that patrons are less satisfied with library services than may actually be the case. for library b, we have a mean of 2.5, a median of 2.0, and a mode of 1. if we were to describe patron satisfaction with library services using the mean, we would conclude that the average rating was 2.5/6. if we were to use the median, we would conclude that the average rating was 2.0. as opposed to library a, the data are right (positively) skewed in this distribution, which means that using the mean would lead us to conclude that patrons are more satisfied with library services than they may actually be. 4 peer reviewed article hypothesis , vol. 35, no. 1, 2023 these decisions about what is “average” matter both for individual groups, but also when we compare different groups. if we were to compare library a and library b using the means, we would conclude that the difference between the two groups was smaller than if we were to compare them using the medians. using the mode would make the difference look quite dramatic. the mode, the median, and the mean all represent the “average” value, but those values can be very different. these differences become more important as more advanced statistical tests, such as t-tests or anovas, are involved, because these tests compare different midpoints between and within groups and have certain assumptions about how the data are distributed. choosing a test that compares means when you have a skewed distribution might lead to inaccurate results and false conclusions, which can subsequently lead to poor decisions. in the next column in this series, more advanced statistical tests will be discussed, building on some of this foundational knowledge. further reading shafer d, zhang z. measures of central location: three kinds of averages. in: introductory statistics [internet]. saylor foundation; 2012 [cited 2022 dec 17]. available from: https://stats.libretexts.org/bookshelves/introductory statistics/ book%3a introductory statistics (shafer and zhang) references 1. bakker cj. an introduction to statistics for librarians (part one): types of data. hypothesis res j health inf prof [internet]. 2022 aug 24 [cited 2022 dec 17];34(1). available from: https://journals.iupui.edu/index.php/hypothesis/article/view/26428 2. illowsky b, dean s. measures of the spread of the data. in: introductory statistics [internet]. houston, tx: openstax; 2013 [cited 2022 dec 17]. available from: https://openstax.org/details/books/introductory-statistics 5 https://stats.libretexts.org/bookshelves/introductory_statistics/book%3a_introductory_statistics_(shafer_and_zhang) https://stats.libretexts.org/bookshelves/introductory_statistics/book%3a_introductory_statistics_(shafer_and_zhang) https://journals.iupui.edu/index.php/hypothesis/article/view/26428 https://openstax.org/details/books/introductory-statistics hypothesis , vol. 34, no. 1, 2022 the return of “the research mentor” column jonathan d. eldredge, phd, ahip, fmla, associate editora aprofessor and evidence-based practice librarian, health sciences library and informatics center, school of medicine, college of population health, university of new mexico, albuquerque, new mexico, https://orcid.org/0000-0003-3132-9450 , jeldredge@salud.unm.edu mentoring generally consists of a complex, multifaceted relationship between a more professionally experienced colleague and a less experienced colleague.1,2 while a mentor might be more mature biologically, there are plenty of exceptions in the everyday work life of information practitioners. between 2008 and 2014 the author edited “the research mentor” column as a regular feature in hypothesis. web analytics and direct feedback suggested that it was a popular column. the titles of individual columns reveal the subjects and tone of the 12 installments of this column: finding time for research. 2008 interview with library researcher jo dorsch. 2008 interview with library researcher gary byrd. 2009 the institutional review board (irb): a primer. 2009 authorship part one: defining the article author. 2009 authorship part two: order of authors. 2010 creativity in research: an introduction. 2010 creativity in research, part 2: classifying creativity. 2011 creativity in research, part 3: age and creative potential. 2012 interview with phillip eaton, m.d., on creativity in research. 2012 top-ranked research questions and systematic reviews. 2013 the long reach of a mentor’s influence. 2014 most of these columns were either authored or co-authored by the editor (je). the column editor sought to provide useful guidance for less experienced researchers in these columns. at the time he was one of only a few researchers in the ranks of the health information practitioners experienced with a diversity of study designs and with conducting numerous research projects. he stepped down from his column editorship mostly due to adverse circumstances at his own workplace that required his full attention. a succession of hypothesis editors asked him to resuscitate “the research mentor” column since 2014. the current hypothesis editor, margaret hoogland, prevailed in persuading him to re-launch the column. the new column will continue to offer advice to researchers. the column for this issue explains some fundamental concepts in research statistics. future columns might cover formulating research questions, selecting an appropriate research study design, aligning one’s own research with institutional priorities, assembling a research team, selecting a 1 https://orcid.org/0000-0003-3132-9450 https://orcid.org/ mailto:jeldredge@salud.unm.edu hypothesis , vol. 34, no. 1, 2022 research population, recruiting study participants, identifying grant sources, managing multisite research studies, preparing a conference poster, presenting a conference paper, selecting an appropriate journal to publish one’s research results, interacting with journal editors, handling feedback from editorial peer reviewers, and reaching as many potential readers as possible. if you would like to author a column on one of these topics or another possible topic, please email the column editor at jeldredge@salud.unm.edu jon eldredge, phd associate editor, university of new mexico. references 1. johnson wb. on being a mentor: a guide for higher education faculty. mahwah (nj): lawrence erlbaum associates; 2007. 2. ragins br, kram ke. the handbook of mentoring at work: theory, research, and practice. los angeles (ca): sage publications; 2007. 2 mailto:jeldredge@salud.unm.edu editorial hypothesis vol. 33 no.1 fall/winter 2021 welcome to our newest issue of hypothesis! we hope you enjoy reading the content in this issue and learning about the changes we have made in the past year. issue preview invited commentaries (wilson, carroll, adamczyk) in spring 2021, alexandria q. wilson wrote a review on the inaugural “one database at a time” webinar. this series of webinars is co-hosted by the technology in education caucus and the systematic reviews caucus. feel free to join either caucus and view all the webinars in this series on the technology in education caucus website. as a response to increased levels of anti-asian bias, hate, and violence, immediate past-chair alexander j. carroll released a statement of solidarity with asian, asianamerican, and pacific islander community members in march 2021. the statement links to the asian pacific american librarians association (apala)’s statement on these issues and provides important training resources for bystander intervention. finally, abby admaczyk, current chair of the research caucus, compiled guidance on how to write structured abstract research proposals for conferences. this information is useful in particular for new authors, who are writing structured abstracts for the first time. research article (morgan-daniel et al.) the university of gainesville’s health science center libraries (hscl) conducted a diversity, equity, and inclusion (dei) needs assessment to learn more about patrons’ perception of the current libraries’ climate as well as identify strengths, gaps, and areas to improve upon. hscl gained helpful information from this work and identified a number of next steps to improve their engagement with dei across their libraries. the process described in this article also provides an excellent template for other organizations looking to conduct this type of assessment. research caucus awards as is tradition, this issue includes the list of awards given out by the research caucus for mla 2021 conference research papers and posters. congratulations to those who received an award for their excellent research! we encourage all past and current research caucus award recipients to consider submitting articles to hypothesis. hypothesis archives all past issues (67 in total!) of hypothesis are now available in the archives section of the website. below are four facts margaret discovered about hypothesis and the research caucus while completing this work: https://www.mlanet.org/page/emts https://www.mlanet.org/page/emts https://www.mlanet.org/page/caucus-systematic https://orcid.org/0000-0003-0248-3811 https://orcid.org/0000-0001-7653-295x http://journals.iupui.edu/index.php/hypothesis/issue/archive editorial hypothesis vol. 33 no.1 fall/winter 2021 1. pre-2010, this was a newsletter for the research section and it was published 3 times a year. it contained annual meeting updates, a roster of members, and a list of research topics. 2. in 2010, the first mla annual meeting research awards criteria were established and the first award recipients were featured in hypothesis. 3. the conversion from newsletter to journal started in 2013 and the first official journal issue was published in 2014. 4. though the editorial team has turned over several times, the quality of publications remains consistent. we encourage you to browse our robust archive to learn more about the evolution of hypothesis from a newsletter to an open access journal. editorial board updates new members we welcome matthew (matt) cross, who graduates in december 2021 from texas woman’s university, as a new peer-reviewer co-lead. matt has served as a peer reviewer for a number of issues and we know his expertise will assist us in improving how hypothesis handles peer review. in january 2021, alexandria (alex) quesenberry wilson joined the social media team for hypothesis. with her experience working on the library and information science journal as a graduate student, alex brings a wealth of knowledge and experience for how we can continue to grow the journal. additionally, alex is an up-and-coming researcher, who was a recipient of an mla 2021 research caucus award. wishing them the best! we would like to thank kimberly (kim) powell for onboarding matt and assisting us in developing a structure for peer reviewing and workflow in ojs. we thank her for continuing to serve as a peer reviewer for hypothesis. additionally, in august 2020, amy reyes joined the inaugural social media team for hypothesis. amy worked closely with danielle aloia to create a distribution list, which the social media team will use to solicit new submissions and to publicize new issues. we thank you, amy, for your contributions and we wish you well. last (but not least!), from 2017-2021, erin d. foster has served as editor of hypothesis. erin was joined by carol perryman in 2018 as co-editor and in 2020, margaret hoogland was recruited to serve as co-editor while erin transitioned out of her role. erin has worked tirelessly to initially create an online submission for hypothesis in mlanet and then to migrate the journal from mlanet onto the open journal system https://orcid.org/0000-0002-9495-0976 https://orcid.org/0000-0002-2649-8252 https://orcid.org/0000-0003-3434-7689 https://orcid.org/0000-0001-6908-9849 https://orcid.org/0000-0001-7492-5944 https://orcid.org/0000-0002-9932-3605 https://orcid.org/0000-0002-9932-3605 editorial hypothesis vol. 33 no.1 fall/winter 2021 platform. if an issue cropped up at any point related to the website, workflow, etc, erin jumped in and fixed it. last fall, erin moved back to california and became the service lead for the university of california berkeley research data management program. we thank erin for her time and dedication to hypothesis. although we will miss her, we wish her all the best and we are thankful she has agreed to continue serving as a peer reviewer for hypothesis. if you have suggestions for how we can improve the journal -or if you have questions for us, we encourage you to contact the co-editors. your hypothesis co-editors, erin d. foster margaret a. hoogland carol l. perryman mailto:malrshypothesis@gmail.com peer reviewed article hypothesis , vol. 34, no. 1, 2022 an introduction to statistics for librarians (part one): types of data caitlin j. bakker, mlis, ahipa adiscovery technologies librarian, dr. john archer library, university of regina, regina, saskatchewan, canada, https://orcid.org/0000-0003-4154-8382 , caitlin.bakker@uregina.ca librarians usually aren’t statisticians. most of us haven’t taken a statistics course and might not feel comfortable talking about things like p-values and confidence intervals. at the same time, most of us also want or need to know that results are “real.” whether we’re assessing our own programs and services, conducting research, or reading scientific articles, it’s helpful to understand some of the basics of statistics. this series in “the research mentor” column will guide you through the process of identifying the data type, choosing a statistical test, and interpreting the results, but it is not meant to be comprehensive or conclusive. whenever possible, it’s best to find a trained statistician who you can consult with about your specific project. identifying the data type the first step to deciding what type of statistical test is appropriate is to think about what type of outcomes you are reviewing. there are two broad categories of data: categorical and continuous. within these categories, there are four types of data: categorical data can be classified as (1) nominal or (2) ordinal, while continuous data can be classified as (3) interval or (4) ratio. this table introduces the four different data types, which we’ll go over in more depth below. data type description examples nominal categorical data that can’t be ordered in any meaningful way. these are data without any quantitative value. eye color, hair color, type of car, ice cream flavor, nationality ordinal categorical data that can be ranked or ordered meaningfully. likert scales (most to least likely, strongly disagree to strongly agree), course grades (letter grades a, b, c, d, f), class rank interval continuous data that don’t have a natural zero. temperature in fahrenheit or celsius; mcat, gre, or sat scores ratio continuous data that have a natural zero. temperature in kelvin, height, weight, age, scores on a test (percentage correct) 1 https://orcid.org/0000-0003-4154-8382 https://orcid.org/ mailto:caitlin.bakker@uregina.ca peer reviewed article hypothesis , vol. 34, no. 1, 2022 categorical data types: nominal and ordinal categorical data are also called discrete and refer to outcomes where there are a fixed number of possible values. demographic data, such as gender, age group, or educational level are categorical data. there are two specific types of categorical data: (1) nominal and (2) ordinal. nominal data have no order, while ordinal data are ordered. for example, if you surveyed a group of medical students and asked them which specialty they were planning on pursuing, their answers would be nominal data because you couldn’t rank family medicine above surgery or below obstetrics. the answers are all different, but there is no rank or order. if you surveyed those same students and asked them to say whether they used certain library resources a lot, a little, rarely, or not at all, those answers would be ordinal because we could order them from most to least used. continuous data types: interval and ratio continuous data are numerical and can have any value between a minimum and maximum. questions correct on an exam, or the final percentage of correct answers would be continuous, since a student could get anywhere from 0% to 100%. like ordinal data, continuous data are ordered. they are also evenly spaced. the difference between 5% and 10% is the same as the difference between 10% and 15%. this is not true of ordinal data. the difference between “strongly disagree” and “disagree” on a likert scale may not be the same as the difference between “disagree” and “neutral.” there are two types of continuous data: (1) interval and (2) ratio. the difference between these two lies in what zero means for each. with ratio data, zero would mean an absence of that variable. this is called a “natural zero,” where zero means, “there isn’t any of this.” the example above of student scores on an exam would be ratio data. if a student scored 0%, it would mean that they got no questions right on the exam. in contrast, those same students’ mcat scores would be interval data since the possible scores for the mcat range from 472 to 528. absolute zero isn’t used as a reference point since getting no questions right wouldn’t result in a score of 0. therefore, zero has a different meaning in interval data than it does in ratio data. this means that while both ratio and interval data can be added and subtracted, only ratio data can be divided and multiplied. why does this matter? knowing the type of data is essential to choosing the right statistical test. trying to conduct a test designed for interval or ratio data on ordinal data would lead to incorrect results and possibly false conclusions. in the next column in this series, we’ll discuss which statistical tests may be right for which type of data. suggested readings bowers d. first things first – the nature of data. in: medical statistics from scratch: an introduction for health professionals. 4th ed. hoboken (nj): wiley; 2019. p. 3–14. 2 peer reviewed article hypothesis , vol. 34, no. 1, 2022 daniel ww, cross cl. measurement and measurement scales. in: biostatistics: a foundation for analysis in the health sciences. 10th ed. hoboken (nj): wiley; 2019. greenhalgh t. how to read a paper: statistics for the non-statistician. i: different types of data need different statistical tests. bmj. 1997 aug 9;315(7104):364–6. doi: 10.1136/bmj.315.7104.364. stewart a. types of data. in: basic statistics and epidemiology: a practical guide. 4th ed. london (england): crc press; 2016. p. 17–9. 3 editorial reviewed article hypothesis , vol. 34, no. 1, 2022 hello from the research caucus chair-elect rebecca carlson, mlsa ahealth sciences librarian and liaison to the eshelman school of pharmacy, health sciences library, university of north carolina at chapel hill, chapel hill, north carolina, https://orcid.org/0000-0002-4380-8435 , rcarlson@unc.edu, twitter i want to introduce myself as the new chair-elect for the research caucus and share my background, goals for the caucus, and thoughts on the future of library research. i’m currently a health sciences librarian and the liaison to the eshelman school of pharmacy at the university of north carolina-chapel hill, where i collaborate on education, research, and evidence-based practice initiatives. i teach and advise health sciences students on research methods and have co-authored more than 20 publications, including reviews, original research, and commentaries. i was also a fellow of the mla research training institute for 2021-2022. like many other medical librarians, i did not come into the profession with knowledge of or experience in conducting health sciences research. rather, i have a ba in english and my undergraduate thesis was a work of creative nonfiction. during and after my master of library science degree i gained experience in supporting others’ research aims as part of my role as a librarian, but did not start taking a more active collaboration role with research projects and conducting my own research until six years ago. i started with projects that were already a part of my work, adding preand post-assessments and user experience testing to expand existing library services, and pursued continuing education on research methodologies. then, over time, whenever i could not find studies that answered questions i had or information i needed, i began to consider these as opportunities for my own research. from there, i explored my developing research interests and began to set research goals for myself. my current research interests are broad and include interprofessional education outcomes, medical librarian impact on health sciences education and research, and emerging technologies for literature synthesis efficiency. each study i have conducted has challenged me and i have learned from each of them, and am still learning. as well, i had the opportunity to participate as a fellow in the medical library association research training institute (rti) in 2021-2022. throughout the past year, i have learned about research from the rti peer coaches, research mentors, speakers, and the other fellows and worked through the research process from beginning to end. as i worked on my rti research project, i realized i wanted to take what my experiences from that community and share them with an even broader group. over my three-year term on the research caucus leadership team, i hope to see the caucus continue to grow as a collaborative, inclusive community for mla members. the research caucus is already a home for medical information professions who are 1 https://orcid.org/0000-0002-4380-8435 https://orcid.org/ mailto:rcarlson@unc.edu https://twitter.com/carlson_unc editorial reviewed article hypothesis , vol. 34, no. 1, 2022 engaged in learning about, conducting, and teaching research, and i plan to continue to expand opportunities for continuing education and networking opportunities for researchers at all levels of experience. my goal is to build on the work of past and current caucus leaders as we strive to make it the hub for health sciences information professionals interested in or working on research to support, challenge, and learn from and with each other. the research caucus already offers a supportive, informal setting to share research ideas and findings through the reports from the field forums and i plan to add other times for discussion, mentoring, and collaboration. i would like to encourage early-career librarians and anyone not currently engaged in research collaboration or research leadership to join the research caucus, ask questions, explore opportunities, and think about the questions you face in your life and work that you could help answer. you do not need to be an expert in research methods and analyses to join or start a research project. it is never to early or late to grow your research skills, and you can learn from and share any results—successes or failures. 2 voices of experience hypothesis vol. 32 no.1 fall/winter 2020 sharing research data to comply with a journal policy: experience of a first-time depositor marianne d. burke phd, ma, ahip associate professor emerita university of vermont burlington, vt abstract background journals in health sciences increasingly require or recommend that authors deposit the data from their research in open repositories. the rationale for publicly available data is well understood but many researchers lack the time, knowledge, and skills to do it well, if at all. there are few descriptions of the pragmatic process a researcher author undertakes to complete the open data deposit in the literature. when the author’s manuscript for a mixed methods study was accepted by a journal that required shared data as condition of publication, she proceeded to comply despite uncertainty with the process. purpose the purpose of this study is to describe the experience of an information science researcher and first-time data depositor to complete an open data deposit. the study illustrates the questions encountered and choices made in the process. methods to begin the data deposit process, the author found guidance from the accepting journal’s policy and rationale for its shared data requirement. a checklist of pragmatic steps from an open repository provide a framework that the author used to outline and organize the process. process steps included organizing data files, preparing documentation, determining rights and licensing, and determining sharing and permissions. choices and decisions included which data versions to share, how much data to share, repository choice, and file naming. processes and decisions varied between the quantitative and qualitative data prepared. results the author deposited data in two datasets and documentation for each in figshare open repository, thus meeting the journal policy requirements to deposit sufficient data and documentation to replicate the results reported in the journal article and also meeting the publication deadline to include a data availability statement with the published article. conclusion this experience illustrated some practical data sharing issues faced by a librarian author seeking to comply with a journal data sharing policy requirement for publication of an accepted manuscript. both novice data depositors and data librarians voices of experience hypothesis vol. 32 no.1 fall/winter 2020 may find this individual experience useful for their own work and the advice they give to others. background journals in health sciences increasingly require or recommend that authors deposit the data from their research in open repositories. in october 2019, the journal of the medical library association (jmla) announced its policy to require authors to deposit deidentified research data for all original investigations and case report articles accepted for publication in october of 2019 [1]. although many library and information science journals recommend data sharing, jmla was the first and still possibly the only one to require it. the rationale for publicly available data—that it fosters scientific progress and enables replication and reproducibility of research—is well described in the literature and in the jmla editorial justifying its requirement [2]. data sharing workflows for researchers and for repositories are described at a conceptual level by austin et al [3] with little process detail. educational methods for teaching research data management methods to students and researchers are described in a review by corti and van den eyden. [4]. they recommend skills training and active learning methods but do not cover specific content on how to share data and what data to share. while most researchers support data sharing and agree with its value, studies show that many researchers have uncertainty and concerns with the pragmatic aspects of data sharing [5-7]. a 2018 survey by stuart et al of 7,000 researchers from various disciplines and experience levels found that 76% rated the importance of discoverable data highly, while 46% reported organizing data and presenting data as problematic. lack of time was a problem for 35%, and repository choice a problem for 33% [8]. a survey of clinical trial investigators, by tannenbaum et al, found that they spent a median of 18 hours (iqr 8 – 40) per data set shared [9]. a meta-synthesis of qualitative studies of researcher experience with data sharing found that “researchers lack time, resources and skills to effectively share their data in public repositories”[10]. i submitted a manuscript to jmla after its data sharing policy was announced, but prior to when it went into effect. technically i was not required to share the data for manuscript. but when my manuscript was accepted for the july 2020 issue, i decided to share the data from my research motivated by belief in the value of data sharing and by the journal requirement policy. like other researchers, i was uncertain about how to organize and prepare the data, and was apprehensive about the time it would take, but i decided to do it. the accepted manuscript (now a jmla article) is a mixed methods assessment of a clinical evidence technology based on a survey of 32 primary care providers (pcps) who participated in an earlier randomized trial concerning their use (or non-use) voices of experience hypothesis vol. 32 no.1 fall/winter 2020 of the technology during that trial, and interviews with 11 pcps in the intervention arm [11]. while advantages and obstacles to researcher data sharing are well described, few studies describe researchers’ data sharing experiences in detail. the purpose of this case study is to describe my experience as an information science researcher and first-time data-depositor, and illustrate the practical questions and issues that i encountered in the process. methods to begin the open repository deposit process, i reviewed the journal’s requirements. the jmla data sharing policy states ”the jmla requires authors of original investigation, case report, and special paper articles to (1) place the deidentified data associated with the manuscript in a repository and (2) include a data availability statement in the manuscript describing where and how the data can be accessed” [1]. the jmla editorial announcing the policy by akers et al elaborated on the requirements stating that “at least minimal data needed to support or replicate results”, and “documentation describing the contents of the data files” must be deposited [2]. a data availability statement, including a url or doi for the data, must be provided by the author and included with the published article. the editorial offered guidance for authors depositing data for the first time and recommended a selection of resources and references that could be consulted for help, including library-based web-sites and libguides, and webpages of open repositories and noted that help is available from health science libraries and librarians [2]. i sought practical guidance that provided specific data sharing information and advice, and not data sharing background or rationale from among the editorial references. three references that provided pragmatic methods for organization and presentation of data were washington university libraries’ webpage “preparing data for deposit” [12], the 2019 journal of escience librarianship presentations article entitled “best practices for data sharing and deposit for librarian authors”, by regina raboin et al [13] , and the inter-university consortium for political and social research (icpsr) guide which includes a section entitled “preparing data for sharing” [14]. i found clear steps to follow in the digital research materials repository (drmr) deposit website created by the washington university st. louis libraries (available at https://libguides.wustl.edu/drmr/dataprep) [12]. there, a checklist and a template readme file form are available as downloadable and printable tools to address the organization, presentation, documentation and other tasks. the checklist outlines four major steps and multiple itemized sub-steps. the wusl data research materials repository (drmr) checklist tasks are to organize your deposit, prepare documentation, determine deposit rights & licensing, and determine sharing and permissions. i used the drmr checklist as a process framework https://libguides.wustl.edu/drmr/dataprep voices of experience hypothesis vol. 32 no.1 fall/winter 2020 to move forward with the data deposit but revised it adding “review journal policy and requirements” as first step described above and a fifth task, “choose your data repository”. the process was recursive in that i worked on several checklist steps at once or went back and forth between them rather than proceeding in a consecutive workflow. [see figure 1] figure 1: author's process to deposit research data in an open repository to comply with journal policy. step 1: organize your deposit after reviewing the journal’s policy and guidelines, the first step, to “organize your deposit” seemed simple, but it was on this step that i encountered the most decision points and spent the most time. sub-items in this category included gathering and choosing the dataset files to share, naming files and labelling variables consistently, and transforming datasets and files to -non-proprietary file formats. the jmla policy names appropriate data types as “including but not limited to spreadsheets, text files, interview recordings or transcripts, images, videos, output from statistical software, and computer code or scripts”[2]. i had several of these data types and felt uncertainty concerning which versions of similar data files and how much of the total available data to prepare. voices of experience hypothesis vol. 32 no.1 fall/winter 2020 step 1a: data version owing to the mixed methods design of the research, there were both quantitative and qualitative data files, and version was an issue in each type. for the quantitative data, potential data files included raw deidentified survey response data output from redcap [15], and the same survey data cleaned of extraneous variables and processed for analysis from the stata statistical software [16]. raboin et al noted that either raw or processed data may be used [13] but recommend redcap export data because the system can automatically remove personal identifying information and provides a codebook or data dictionary that describes all variables. on the other hand, processed data, i.e. data cleaned of extraneous or administrative variables and possibly containing new variables generated and used in analysis, may more completely represent the reported results. jmla policy did not address this difference, but plos guidelines state, “authors do not need to submit the raw data collected if the standard in the field is to share data that have been processed.”[17]. i did not know of a library science standard but, in clinical and translational science, where i was a graduate student, the usual practice was to import deidentified raw data into the statistical package (stata) and clean it and sometimes generate new variables. for this reason, i chose the processed stata dataset to deposit. in addition, i edited the data dictionary exported from redcap and included it as a deposited file. for the qualitative data, there were two versions of the interviews: digital recordings and full transcripts. the digital recordings were difficult to edit and de-identify given my lack of technical skill and the possibility of voice recognition of the participants. i decided to deposit the deidentified transcripts, with the caveat they would need further review for deidentification and anonymization (see checklist step 4). i also considered preparation of the fully coded interview transcripts as analyzed in nvivo [18] for deposit. i found nvivo exported tables in a word processing format (.docx) difficult and timeconsuming to edit and convert into a text (.txt) document. in addition, this information would be redundant since many exemplar quotations and associated codes were included as supplementary material in the accepted manuscript. therefore, i decided not to deposit the complete coded interviews. step 1b: entire data or portion whether to share all existing data results or just those reported in the accepted manuscript was a decision point. according to the jmla policy “authors are expected to deposit at least the minimum amount of data needed to reproduce the results described in the manuscript” [1]. a similar policy by plos states, “authors do not need to submit their entire data set if only a portion of the data were used in the reported study” [17]. i considered depositing data from a baseline survey analyzed and reported in an earlier related journal article, but not yet shared [19]. the baseline data were not used or analyzed in the results in the current accepted manuscript. depositing the complete voices of experience hypothesis vol. 32 no.1 fall/winter 2020 data might be considered a better practice because the data are available sooner for review and secondary analyses. however, i decided to deposit only the data needed to reproduce the results of the manuscript to reduce the organization and documentation effort and save time. i plan to share all data from the earlier study in the near future. another task at the “organize your deposit” step was to convert all data files to non-proprietary formats. i did this for the chosen files without much difficulty. tabular survey data was exported from stata as comma separated values (.csv) files, and the interview transcripts were converted from word processing documents (.docx) to text (.txt) files for the deposit. the last task at this step was to create consistent labels for data headers, files, and variables. for me, dataset file naming was an iterative process and was only concluded with the final deposit. variable names and data column headers in the quantitative data were consistent as exported from stata and redcap as tabular data. step 2: prepare data documentation this step is the preparation of the metadata, the data descriptions of each data file, the names of files deposited and so on. the downloadable wusl readme (https://libguides.wustl.edu/ld.php?content_id=29108244) template form described earlier presented a complete list of information elements needed to be provided with the data. the form kept track of progress on decisions at each of the outlined steps creating an inventory of file choices, naming and documentation. information recorded on this form was drawn from every other step in the process and served as a complete source of documentation for the deposit. step 3: determine rights and licensing this step concerned the rights of others to reuse of the data. since this would be an open data deposit, i knew that a creative commons (cc) license was appropriate, but there are several types of cc licenses so this needed to be checked. license types are defined at https://creativecommons.org/licenses/. i chose the cc by license which lets others distribute remix and adapt your data as long as they credit the author for the original creation. i did not seek any copyright permissions for the deposit because my co-authors and i were the creators of the data. step 4: determine sharing and permissions this step involved assuring that authorization and consent to share data collected from participants in a study was obtained. for the study reported in the accepted manuscript, participants explicitly consented to their data being deidentified, aggregated, analyzed, and reported in a research journal in an irb approved consent form. deidentification of participants in the quantitative data was easily settled with the https://libguides.wustl.edu/ld.php?content_id=29108244 https://creativecommons.org/licenses/ voices of experience hypothesis vol. 32 no.1 fall/winter 2020 automated removal of personal information step that an investigator utilizes when exporting data from redcap. deidentification in qualitative data is more problematic [14]. studies have shown that even when data is deidentified by the removal of personal information, deductive disclosure i.e. logical supposition of who the source is by members of the same community is possible, so additional anonymization is needed [20]. i had not explicitly stated the likelihood of deposit of full transcripts of interviews in the consent information for provider participants, only that their answers to questions would be aggregated with those of others. i sent a sample transcript to the university of vermont human protections office for guidance. after review, the irb administrator ruled that transcripts deposit would be allowed because there were no direct identifiers and the “information would not be considered sensitive in nature.” i.e. they posed little risk of harm to the primary care provider participants [email communication to author from l. crain “re: research data sharing in repositories”, 31 jan 2020]. i chose to deposit the transcripts after removing additional possibly identifying clues such as clinic site and idiosyncratic phraseology. step 5: choose repository for repository choice, i found criteria to consider in the raboin presentation (slide 26) [13], journal recommendation, visibility through a doi, usability, and features such as embargoes, creative commons licensing, and formats accepted were meaningful to me. several general non-discipline specific open repositories were suggested by jmla. i reviewed a comparison grid of general data repositories at a harvard medical school website that included four of the journal-recommended repositories: dataverse, dryad, figshare and zenodo [21]. a librarian colleague who had deposited data, advised that figshare [22] was easy to use and provided prompts and mini-tutorials. without having a strong preference, i chose figshare. when exploring the figshare website, i found simple instructions and few restrictions. i used trial and error methods to learn the repository’s features and began uploading the data files and documentation. for each dataset entered in figshare, there are prompts for metadata including title, authors categories, links to related keywords, data description, any restrictions on use of data, and sharing level. my prepared readme form contained most of the required information so i cut and paste the authors, titles, descriptions, file names, license and permissions into the figshare metadata boxes. figshare offered additional advice on naming and other metadata conventions to increase findability and i did rename files and datasets to increase findability. i entered separate documentation metadata for the qualitative and quantitative datasets based on my readme form. i named dataset authors based on who had participated in design and review of each set, and each dataset was described differently [23, 24]. i did voices of experience hypothesis vol. 32 no.1 fall/winter 2020 change some metadata and file names after publishing. this resulted in the version numbers being added to the dois. results i deposited data in two separate datasets in figshare, one for the quantitative survey dataset, https://doi.org/10.6084/m9.figshare.11893875.v3 and one for the qualitative https://doi.org/10.6084/m9.figshare.11893956.v1. the survey dataset contains metadata description, tabular processed data with 39 variables and 22 observations, and a data dictionary file that includes the variable names, corresponding survey questions, and response type and possible responses in a comma separated file. the qualitative dataset contains metadata description and the interview transcripts as a single text (.txt) data file. i spent at least 30 hours over three weeks reading polices, referring to links in library libguides, conferring with my institution’s data librarian and researcher colleagues, and performing the tasks in the checklist outline, making decisions, and navigating the figshare repository. i met the journal policy requirements to deposit sufficient data and documentation to reproduce the findings of the manuscript, and met the publication deadline for including the data availability statement with the published article. discussion my experience supports the finding of kim and stanton that authors’ belief in the value and benefits of data sharing combined with the open data sharing requirement of journals may motivate authors to share data, but the time and effort needed to complete the process may negatively impact the researchers’ actual participation in data sharing [5]. as anticipated, my data deposit experience was time-consuming and presented myriad decision-points without exact answers even with consulted sources. however, this was a first-time experience; i suspect that there will be fewer obstacles next time, although different data types and file sizes will certainly present different hurdles. i will utilize checklist and pragmatic tools again, and will be less hesitant to make decisions. my advice to researcher/authors who are novice data sharers is to use the most specific checklists and tools you can find or make your own. pragmatic advice and tools found on the websites supporting open data repositories such as the wsu, icpsr and figshare repositories may be the most useful. i believe it is better to publish the data even if some uncertainties linger. don’t let hesitation or imperfection prevent completion of the deposit. it is possible to revise metadata and upload and deposit revised data versions in repositories if that is needed. https://doi.org/10.6084/m9.figshare.11893875.v3 https://doi.org/10.6084/m9.figshare.11893956.v1 voices of experience hypothesis vol. 32 no.1 fall/winter 2020 implications for health science librarians support for the data sharing activities needs of researchers may not be fully developed in health sciences libraries. while many academic health science libraries provide some level of data services, according to a survey by ragon of academic health science library directors, 25% of those surveyed did not provide data services in their libraries and did not plan to do so in the future [25]. a survey of data librarians by federer revealed that most of them spend less than half of their work time on data services [26]. a taxonomy of skills needed by data librarians proposed in the same article did not include skills or experience in data sharing itself. i appreciated the advice and recommendations from the library website and the data librarian at my institution, but they answered few of my specific data sharing questions. library data services could emphasize pragmatic data sharing tools and tips at least as much as background and advocacy information on their research data management web-sites. librarysponsored teaching sessions that utilize group workshop formats and in-class case presentations of data sharing by researchers could provide a more practical learning experience for researcher authors. data librarians with experience sharing their own data might be better equipped to assist novice depositors (including library and information science researchers) with the problems and decision-points that arise in the process of data sharing. limitations this individual case study reflects only one data sharing experience. other researcher authors will find different issues, choices, and take-aways in their experience particularly if they are sharing data from randomized trials, other empirical designs, large data sets, or complex statistical analyses. nevertheless, this description of my experience may give some insight to the process overall. conclusions the burden placed on researcher authors when depositing data is significant. having experienced the process once, i now see where pitfalls and problems make open data sharing problematic for researcher authors. as more health science librarians gain experience depositing their own research data, they may provide more concrete and practical answers when assisting researcher/authors and thereby reduce author uncertainty and time spent in the data sharing process. this experience illustrated some practical data sharing issues faced by a librarian author seeking to comply with a journal data sharing policy requirement for publication of an accepted manuscript. both novice data depositors and data librarians may find this individual experience useful for their own work and the advice they give to others. voices of experience hypothesis vol. 32 no.1 fall/winter 2020 references 1. journal of the medical library association data sharing policy: journal of the medical library association; 2019 [may 20 2020]. available from: http://jmla.mlanet.org/ojs/jmla/about/editorialpolicies#custom-0. 2. akers kg, read kb, amos l, federer lm, logan a, plutchak ts. announcing the journal of the medical library association's data sharing policy. j med libr assoc. 2019 oct;107(4):468-71. doi: http://dx.doi.org/ 10.5195/jmla.2019.801 3. austin cc, bloom t, dallmeier-tiessen s, khodiyar vk, murphy f, nurnberger a, et al. key components of data publishing: using current best practices to develop a reference model for data publishing. international journal on digital libraries. 2017 2017/06/01;18(2):77-92. doi: http://dx.doi.org/ 10.1007/s00799-016-0178-2 4. corti l, van den eynden v. learning to manage and share data: jump-starting the research methods curriculum. international journal of social research methodology. 2015 2015/09/03;18(5):545-59. doi: http://dx.doi.org/ 10.1080/13645579.2015.1062627 5. kim y, stanton jm. institutional and individual factors affecting scientists' datasharing behaviors: a multilevel analysis. journal of the association for information science and technology. 2016; 67:776–99. doi: http://dx.doi.org/ http://dx.doi.org/10.1002/asi.23424 6. tenopir c, rice nm, allard s, baird l, borycz j, christian l, et al. data sharing, management, use, and reuse: practices and perceptions of scientists worldwide. plos one. 2020;15(3):e0229003. doi: http://dx.doi.org/ 10.1371/journal.pone.0229003 7. federer lm, lu yl, joubert dj, welsh j, brandys b. biomedical data sharing and reuse: attitudes and practices of clinical and scientific research staff. plos one. 2015;10(6):e0129506. doi: http://dx.doi.org/ 10.1371/journal.pone.0129506 8. stuart d, baynes g, hrynaszkiewicz i, allin k, penny d, lucraft m, et al. practical challenges for researchers in data sharing. springer nat doi: https://doi org/106084/m9 figsh are. 2018;59750(11):v1 9. tannenbaum s, ross js, krumholz hm, desai nr, ritchie jd, lehman r, et al. early experiences with journal data sharing policies: a survey of published clinical trial investigators. ann intern med. 2018 oct 16;169(8):586-8. doi: http://dx.doi.org/ 10.7326/m18-0723 10. perrier l, blondal e, macdonald h. the views, perspectives, and experiences of academic researchers with data sharing and reuse: a meta-synthesis. plos one. 2020;15(2):e0229182. doi: http://dx.doi.org/ 10.1371/journal.pone.0229182 11. burke md, savard lb, rubin as, littenberg b. barriers and facilitators to use of a clinical evidence technology in the management of skin problems in primary care: insights from mixed methods. j med libr assoc. 2020 jul 1;108(3):428-39. doi: http://dx.doi.org/ 10.5195/jmla.2020.787 voices of experience hypothesis vol. 32 no.1 fall/winter 2020 12. digital research materials repository: preparing data for deposit [internet].washington university st. louis. the libraries. [cited 20 aug 2020]. available from: https://libguides.wustl.edu/drmr/dataprep. 13. raboin r, plutchak t, palmer l, goldman j. best practices for data sharing and deposit for librarians. journal of escience librarianship presentations [internet]. 2019; 2020 (may 2020). available from: https://escholarship.umassmed.edu/jeslib_presentations/1 14. guide to social science data preparation and archiving. phase 5 preparing data for sharing [internet]. icpsr: inter-university consortium for political and social research [cited 2020_8_20]. available from: https://www.icpsr.umich.edu/files/deposit/dataprep.pdf. 15. harris pa, taylor r, thielke r, payne j, gonzalez n, conde jg. research electronic data capture (redcap)--a metadata-driven methodology and workflow process for providing translational research informatics support. j biomed inform. 2009 apr;42(2):377-81. doi: http://dx.doi.org/10.1016/j.jbi.2008.08.010 16. statacorp: stata statistical software: release 14 college station, tx: statacorp lp; 2015. 17. data availability [policy]: plos journals; 2019 [updated december 5, 2019; cited 2020 2020_3_27]. available from: https://journals.plos.org/plosone/s/data-availability. 18. nvivo qualitative data analysis software version 12 ed: qsr international pty ltd.; 2018. 19. burke m, littenberg b. effect of a clinical evidence technology on patient skin disease outcomes in primary care: a cluster-randomized controlled trial. j med libr assoc. 2019 apr;107(2):151-62. doi: http://dx.doi.org/ 10.5195/jmla.2019.581 20. tsai ac, kohrt ba, matthews lt, betancourt ts, lee jk, papachristos av, et al. promises and pitfalls of data sharing in qualitative research. soc sci med. 2016 nov;169:191-8. doi: http://dx.doi.org/ 10.1016/j.socscimed.2016.08.004 21. harvard biomedical data management: best practices & support services for research data lifecycles: harvard university; [updated 2018_07_062020__9_9]. available from: https://datamanagement.hms.harvard.edu/repositories. 22. figshare [internet]. london: figshare; 2013 [cited september 15, 2020]. available from: https://figshare.com/ 23. burke m, littenberg b. use of a clinical evidence technology for skin disease in primary care: clinician survey data [dataset]. figshare 2020. 24. burke m, savard l, rubin a, littenberg b. use of a clinical evidence technology for skin disease in primary care: clinician interviews [dataset]. figshare; 2020. 25. ragon b. alignment of library services with the research lifecycle. j med libr assoc. 2019 jul;107(3):384-93. doi: http://dx.doi.org/ 10.5195/jmla.2019.595 26. federer l. defining data librarianship: a survey of competencies, skills, and training. 2018. 2018 2018-07-02;106(3):10. doi: http://dx.doi.org/ 10.5195/jmla.2018.306 voices of experience hypothesis vol. 32 no.1 fall/winter 2020 hypothesis, vol. 27, no. 1, winter 2015 3 articles a look at the scholarly output of the medical library association research section terrie r. wheeler, amls1; noga yaniv, msc2; and ruth e. fenske, phd3 the research section (rs) has guided the medical library association (mla) specifically, and the field of health sciences librarianship more generally, in the practice of seeking evidence to inform decision making and practice. the best evidence comes from welldesigned research projects. the section has mentored many health sciences librarians in designing research projects, in conducting research according to the scientific method, and in publishing their results. mla has adopted a research agenda and it relies on the section to carry out that agenda. further, the research section has played a role over the years in advancing the scholarship of the organization. for example, the mla has embraced scholarly publishing by setting high standards in its own publication, the journal of the medical library association. major articles in the journal are research based, require a structured or academic abstract, and generally consist of objectives, methods, findings, results and conclusions. objective while the research section has been critical in advancing the scholarly communication within the field of health sciences librarianship, rarely has the section highlighted its own role in scholarly output. that is the purpose of this abbreviated study. the authors were curious to discover what scholarly publications trends might exist among rs members. some questions to answer were: 1) which journals were most often selected for publication by rs members? 2) were any of the works highly cited? 3) did rs members engage in self-citation? 4) have any rs members built a body of work on a certain topic or topics that have affected the profession at large? 5) which rs members are most prolific? 6) does citation mapping indicate that some rs members have had a national or international impact? 7) are any rs member papers cited often enough to place them in the top ten percentile [1] of all papers published by discipline and year of publication? 8) does the research section encourage collaboration with authors not in the section, or, do rs members co-author primarily with other rs members? the authors wanted to answer these questions by looking at the scholarly output of the rs members, and ran the search strategy that was created in web of science on april 6, 2014. the search parameters included 1970 to date (april 6, 2014) and all document types. the strategy yielded 1,100 publications by rs members. methods one author downloaded the 2013 2014 roster of members of the research section from the mla website. the authors used surnames, first and middle initials when known, and organizational affiliations to identify the scholarly output of the section. thomson reuters’ core, formerly thomson reuters’ web of knowledge, was searched from 1970 to date. of the 258 section members, 160 published at least one publication that was indexed by the thomson-reuters core during this period. as the names that were downloaded from the section membership file did not always include middle initials, the authors took extra care to review articles by subject matter or affiliation to ensure that as often as possible they had selected the correct records for inclusion. the authors checked webpages and curriculum vitae as necessary to ensure accuracy as well as used the name disambiguation tool in thomson-reuters’ core, along with checking certain records for full names in pubmed. after executing the search strategies on each current member in the section, the search strategies were combined into a single set and the final search strategy was run. findings authors ran a citation report on the resulting citations (figure 1). there were 142 papers written in 1995 but in 1996 there was a drop to 18 papers. beginning in 1996 there was an increasing trajectory of papers over time. closer analysis of the 1995 papers reveals that patricia reavis, barbara epstein and lynn piotrowicz authored 115 book reviews that appeared in the december, 1995 issue of psychiatric services, which accounted for this unusually large productivity in one year. peer-reviewed hypothesis, vol. 27, no. 1, winter 2015 4 the citations to rs member articles in each year also reveal a steady upward trend which is generally found in sets of papers. there was a marked increase in 2012 (875) and 2013 (1078). this was a combination of more papers published in 2012 and of certain papers, such as papers 1, 2 and 3 in the top ten highly cited papers (figure 2) having a high citation rate during that period. the times cited for the entire set in figure 1 is 7,580, or approximately 194.36 citations per year. papers published by rs members that are indexed in web of science have an h-index of 36. an overview of the hindex and the reliability of such indices can be found in an article by alonso et al [2]. to find the ten journals in which members of the research section most frequently published, we analyzed the set by source title. the top journals were:  journal of the medical library association (222 papers),  the bulletin of the medical library association [precedes the jmla] (148 papers)  library journal (139 papers). several members have authored book reviews for library journal, which accounted for this higher number.  psychiatric services (115 book reviews)  neurologist (33 papers)  library trends (22 papers)  the journal of family practice (17 papers)  online cdrom review (15 reviews)  reference and user services quarterly, formerly rq (15 papers)  college and research libraries (14 papers) the web of science subject categories most frequently used to identify papers in this set were as follows: information science and library science (718 papers), public environmental and occupational health (134 papers), health policy services (126 papers), psychiatry (123 papers), and computer science information systems (79 papers). the nine most prolific members, according to the web of science data, during that time frame were barbara epstein (129 papers), margaret henderson (46 papers), linda smith (43 papers), kay wellik (43 papers), joanne gard marshall (38 papers), pru dalrymple (35 papers), lucretia mcclure (34 papers), ellen detlefsen (32 papers), and scott plutchak (31 papers). the research section’s most influential authors, according to number of citations, were carol lefebvre with two papers in the top ten, totaling 1,239 citations; shandra protzko with a single paper in the top ten (454 citations); and michele tennant, who was co-author on the sixth, seventh and tenth of the top ten most highly cited papers, (394 citations). by far the most highly cited paper is the 1994 bmj article on systematic reviews coauthored by carol lefebvre. lefebvre’s two papers in the top ten established the librarian’s role in systematic reviews, as michele tennant’s three papers influenced thinking on the librarian’s role in the field of bioinformatics. not only was joanne marshall’s rochester study in the top ten, but her body of work throughout her career established best practices for identifying the value of information provided by librarians. according to percentile ranking [3] in the social sciences, papers one, two and three of the top ten significantly outperformed their peer publications in citation frequency during 2014, falling into the top 1% of cited papers in their field in their first ten years after publication. these findings were verified with thomson-reuters’ essential science indicators™. essential science indicators tracks papers whose citation counts place them in the top 1% of their discipline over a ten-year period after publication. figure 1. citation report of citation activity on rs members’ papers by default, a web of science citation report shows the latest 20 years of citations in a view. one can click on another view to see a graph with all years of citations. a look at the scholarly output, continued wheeler, yaniv, and fenske hypothesis, vol. 27, no. 1, winter 2015 5 section members were conservative with regard to selfcitation. the corpus of papers was cited 7,927 times, with a total of 446 self-citations. this comprised 5.75% of total citations, far below the 30% rate of expected selfcitation suggested in earlier literature [4], or the between 10% and 20% found by a study in 2012 [5]. each of the top ten most highly cited papers influenced the field of library and information science. lefebvre’s paper on systematic reviews, the most highly cited article, demonstrated international influence in the citation histogram shown in figure 3. research section members developed research protocols that enhanced the field of medical information science through publication. two very interesting findings related to the top ten most highly cited articles included the amount of co-authorship done by rs members with other healthcare professionals and the amount of publications in more widely read medical journals. five of the article titles in figure 2 initially looked like false drops, but deeper investigation into the author strings indicated that in each case there was an rs member listed as a co-author. this finding continued throughout the set. often section members were co-authoring with their clients on medical topics as part of the research team. an example of this collaboration was kay wellik’s work with mayo clinic arizona neurology residents on their critical appraisal topics for grand rounds presentations [6]. she also coauthored a series in neurologist of over 30 critical appraisal articles written by the residents. dolores judkins co-authored 13 of the 17 papers in journal of family practice with colleagues from other health disciplines. as collaborators and co-authors, these rs figure 2. ten most highly cited publications authored by research section members, according to the web of science citation report a citation report shows citations for the last five years, followed by a total of all citations to a work, followed by the average citations per year. the second column from the right is the citation data relevant to our discussion. a look at the scholarly output, continued wheeler, yaniv, and fenske hypothesis, vol. 27, no. 1, winter 2015 6 figure 3. citation histogram from thomson-reuters web of science™ shows carol lefebvre’s paper on systematic reviews cited 1071 times by authors in many different asian, eastern and western european, and north american countries. a look at the scholarly output, continued wheeler, yaniv, and fenske hypothesis, vol. 27, no. 1, winter 2015 7 members demonstrated their collegial relationship to other health professionals as they increased their scholarly production and expanded the corpus of our profession’s knowledge. as for collaboration with other mla members, the study authors found that some research section members did co-author with fellow section members, but there was no indication that the section played in role in those collaborations. limitations this study was limited to the papers authored by research section members that were indexed in thomson-reuters’ web of science. since the author disambiguation tools currently available also limited this study, much of the disambiguation was done by hand or by checking websites. timeframe was limited to the period from 1970 to april 6, 2014. to accurately reflect rs membership, the roster of members for the year 2013 2014 was used. this limited membership inclusion criteria to those who had paid memberships in 2014. conclusions this brief study of the publishing habits of the research section revealed that section members added to the corpus of knowledge supporting evidence-based practice. it demonstrated that rs members appear cognizant of not over utilizing self-citation. research section members had respectable citation rates with a few in the very highest percentile ranking and they had papers that influenced thought internationally. the most frequent publication outlets were mla publications. since some rs members published in medical or more general information science journals, this indicated further opportunities to take rs member research beyond jmla to increase awareness of section research and scholarship to the larger medical community. the authors hope that this brief study revealed some of the value to be found, as well as some of the inherent limitations, in using bibliometric analysis. references 1. bornmann l, leydesdorff l, mutz r. the use of percentiles and percentile rank classes in the analysis of bibliometric data: opportunities and limits. journal of informetrics. 2013;7(1):158-165. doi: 10.1016/j.joi.2012.10.001. 2. alonso s, cabrerizo fj, herrera-viedma e, herrera f. h-index: a review focused in its variants, computation and standardization for different scientific fields. journal of informetrics. 2009;3(4):273 -289. doi: 10.1016/j.joi.2009.04.001. 3. bornmann l, marx w. how good is research really? measuring the citation impact of publications with percentiles increases correct assessments and fair comparisons. embo reports. 2013;14(3):226-230. doi: 10.1038/embor.2013.9. 4. seglen po. citation rates and journal impact factors are not suitable for evaluation of research. acta orthopaedica. 1998;69(3):224-229. doi: 10.3109/17453679809000920. 5. wallace ml, larivière v, gingras y. a small world of citations? the influence of collaboration networks on citation practices. plos one. 2012; 2011;7 (3):e33339. doi: 10.1371/journal.pone.0033339. 6. demaerschalk, bart, wingerchuk, dean, wellik, kay, budavari, adriane, carter, jonathan, dodick, david. there is merit to evidence based neurology training in residency: p01.006. neurology. 2006;66 (5):a16. author affiliations 1 director, samuel j. wood library & c.v. starr biomedical information center, weill cornell medicine, new york, ny; tew2004@med.cornell.edu 2 performance data analyst, hospital quality institute, sacramento, ca; ynoga@umich.edu 3 associate librarian emeritus, john carroll university, university heights, oh; rfenske@jcu.edu a look at the scholarly output, continued wheeler, yaniv, and fenske mailto:tew2004@med.cornell.edu mailto:ynoga@umich.edu mailto:rfenske@jcu.edu of interest carol l. perryman hypothesis, vol. 29, no. 1, spring/summer 2017 18 introducing ‘of interest’ carol l. perryman, phd co-editor, hypothesis associate professor school of library and information studies texas woman’s university denton, tx the role of a librarian is to make sense of the world of information. if that’s not a qualification for superhero-dom, what is? nancy pearl, as quoted in the seattle times, july 10, 2003. we hypothesize (pun, of course, intended) that librarians really are super-heroes, putting out small-, mid-, and large-scale fires while dancing between flames, giving our time to keeping up with change. sharing those experiences, both good and bad, is the best part of practice and research – and making sense of it all is something best done by superheroes! we’re delighted to announce the start of a new column intended to highlight in-progress library research, and to share information of interest. publishing your planned or in-progress work in hypothesis can increase participation and interest in your study, and encourage connections with others working in the same area – the very essence of our community. after your research is complete, consider publication in hypothesis, a peer-reviewed publication that’s indexed in cinahl. but there’s more: if you had a poster at mla, consider submitting a summary. if you see an announcement you think might be of interest to your colleagues, send us a link and a brief annotation. if you have upcoming research and are interested in promoting it through hypothesis, please send us the following information: • research project working title • investigator(s) information: name, institution, position title • contact information for the principal investigator • research project description (250-350 words) • additional information: survey deadline, project website, etc. • if you are conducting survey research, you must have obtained human subjects approval, and will need to include a brief statement to that effect. of interest carol l. perryman hypothesis, vol. 29, no. 1, spring/summer 2017 19 ‘of interest’ contents will not be limited to dissemination of survey-based research; equallength summations of planned research in the area of health science librarianship are welcome. other information considered for publication in the column includes brief summaries or annotated links to research of interest. use of the structured abstract format for research reports is encouraged, but not mandated, and we will be happy to respond to your questions. hypothesis is published biannually by the mla research section. items to be included should be sent to the co-editors by march 15 for the spring/summer issue and september 15 for the fall/winter issue. -- editorial hypothesis vol. 32 no.1 fall/winter 2020 erin d. foster, msls co-editor service lead, research data management program university of california, berkeley berkeley, ca margaret hoogland, mls, ahip co-editor assistant professor, clinical medical librarian mulford health science library university of toledo toledo, oh carol l. perryman, mlis, phd co-editor associate professor school of library and information studies texas woman’s university denton, tx we at hypothesis want to start by saying that we hope you and your families, friends, colleagues, and all loved ones are doing well (or as well as can be expected) during this time. as co-editors, we acknowledge that this has been a time of great challenge for those in our community and beyond. in june, research caucus chair alexander carroll released a message of solidarity in support of our black, indigenous, and people of color (bipoc) library community members. the statement charged research caucus members to recognize “how [our] systems and services reflect and reinforce systemic inequality, and [ask] what can we do to reform those systems to create more equitable libraries, institutions, and communities?” we strongly encourage you to read this statement (published in this issue as an invited commentary) if you have not had the opportunity. hypothesis welcomes submissions (across any of our submission types) that actively engage with these questions and contribute to efforts to dismantle systems of oppression within the (medical) library community and our society at large. in addition to the ongoing fight for social justice, many within our community have been, and continue to be, affected by the global pandemic. library budgets across http://journals.iupui.edu/index.php/hypothesis/about/submissions editorial hypothesis vol. 32 no.1 fall/winter 2020 institutions have been cut and many of us know colleagues/friends who have lost their jobs. in such a time of uncertainty and stress, hypothesis wants to publish research or perspectives (i.e., voices of experience) that highlight the work and value of information professionals now more than ever. finally, cheers to our second issue published on the ojs platform! we continue to improve our processes from submission to publication to best highlight the work of our authors and showcase the amazing research of health sciences librarians/information professionals. we are grateful for those that submit manuscripts and, as always, for our readership. read on to learn more about recent (exciting) journal developments and to get brief overviews of this issue’s content. welcoming our new co-editor, margaret hoogland we’re delighted to welcome co-editor, margaret hoogland! employed at the university of toledo libraries as the clinical medical librarian, margaret formerly served as the mla research section community manager (with a job description that starts with ‘other duties as assigned’ and goes on from there), margaret’s deep knowledge of mla as an organization brings great value to this publication, and her involvement in the efforts of getting hypothesis indexed and uploading back issues continues to be invaluable to us. research caucus 2020 research awards as always, we are excited to showcase those that received research awards for excellent research papers or posters presented at the annual mla meeting. this year the research caucus also awarded its biannual award for jmla best research paper for a paper published in jmla between 2017-2019. congratulations to these awardees and all the great research conducted! new column methods moment also new to hypothesis is methods moment, an edited column focused on research methods. the column is the brain child of nina exner, virginia commonwealth university research data librarian, who contributed a piece for the inaugural column on case studies, providing the benefit of her considerable research experience. feel free to contact nina with suggestions if there’s a research methodology you’d like to see explored or if you, yourself would like to write a focused piece on a methodology to help others benefit from your own experience. learn about submission criteria here. methods moment case studies since librarians often earn their mls with little or no exposure to actually performing research, it’s no surprise that our most frequently used method is the survey. http://journals.iupui.edu/index.php/hypothesis/about/submissions http://journals.iupui.edu/index.php/hypothesis/about/submissions https://journals.iupui.edu/index.php/hypothesis/about/submissions editorial hypothesis vol. 32 no.1 fall/winter 2020 learn with column editor nina exner about other methods to enhance your practice and publication record. and if you’re not aware of it, mla has a number of research mentors to help get your own research accomplished, whether you’re new to librarianship or just new to a method of research. voices of experience sharing research data to comply with a journal policy marianne d. burke enumerates lessons learned for the benefit of others (either novices to the process or current data librarians) entering the territory of open data policy compliance, particularly for lis research during an era where it’s all still being developed. hypothesis: failure challenges of the two-year manuscript the honesty of author jolene m. miller in reflecting on the writing process encourages our own thoughts with a well-written overview of reflective methods, enhancing this tale of a journey toward publication. research expanding a single-institution survey to multiple institutions erin r.b. eldermire, et al, discuss the challenges of getting not only institutional review board approval to do a study but also how research approval policies and permissions for distributing studies vary by location. despite these challenges, the team feels that multi-institutional studies are worthwhile endeavors. research factors affecting clinical referrals to the medical library liz kellermeyer, who writes about the results of her 2018 medical library association research training institute project, and matthew strand collaborated to investigate barriers preventing health providers from referring patients to the library. of the population surveyed, the primary barrier was lack of knowledge about the library services and ways to refer patients to the library. kellermeyer and strand acknowledge that these are identified barriers for a single hospital, but future studies could investigate barriers at standalone hospitals or hospital systems. additionally, hospital libraries could explore the capability of making library referrals an option within the electronic health record system. introducing our new hypothesis team members we’ve been growing! the hypothesis team now consists of our editorial team, but also a peer review team headed by kimberly powell, a social media team led by danielle aloia and amy reyes, a copyediting team led by andrea ball and heather healey, and a number of contributing team members see the full list of team members on our journal site (link). http://journals.iupui.edu/index.php/hypothesis/about/editorialteam editorial hypothesis vol. 32 no.1 fall/winter 2020 back issue import and call for tech team members hypothesis now has the first two published issues from 1987 and 1988 on the journal website! we hope to make more issues available, and even individual articles, where appropriate. we plan to consider ways for streamlining our indexing processes. another goal is to get hypothesis listed in the directory of open access journals. to be consistent and to achieve these goals, we are forming a tech team, which will work directly with margaret hoogland. if anything in the above description calls your name, please email mlarshypothesis@gmail.com to schedule a brief meeting (15-30 minutes) with the co-editors. in summary as co-editors, we hope you enjoy vol. 32, no. 1 of hypothesis: journal of the research caucus! join us on this journey: submit, read, and share what you find in our pages. mailto:mlarshypothesis@gmail.com welcoming our new co-editor, margaret hoogland research caucus 2020 research awards new column methods moment methods moment case studies voices of experience sharing research data to comply with a journal policy hypothesis: failure challenges of the two-year manuscript research expanding a single-institution survey to multiple institutions research factors affecting clinical referrals to the medical library introducing our new hypothesis team members back issue import and call for tech team members research section awards de groote, lyon, henner, and kim powell hypothesis, vol. 28, no. 1, fall 2016 42 honorable mention authors: rick wallace, ahip – assistant director, quillen college of medicine library; nakia woodward – sr. clinical librarian, quillen college of medicine library title: piecing together the mosaic of rural clinician information practices over a twenty year period abstract: objectives: the purpose of this study is to determine how the information practices of rural clinicians in fifteen counties have changed over the last twenty years. this data is needed to design programs to meet the information needs of the population. methods: this study is a longitudinal cross-sectional study. a validated survey methodology was used to gather data at a specific point in time. physicians’ names were gathered from the state licensing verification database and librarians’ personal knowledge. advanced practice and registered nurses were identified from a list from the state center for nursing. the questionnaires were sent by mail with a self-addressed stamped return envelope with a cover letter explaining the purpose of the survey. returned surveys were accepted for a 6 week period. the physicians surveyed were the complete population of a fifteen county area and nurse/nurse practitioners were a random sample of the population. previous iterations were done in 1998 and 2009. results: in 1997, names of physicians and nurses in 17 rural tennessee counties were obtained. a random sample (p=.05) was surveyed for a total of 707-(357 rns and 350 mds). in 2009, this exact procedure was replicated with slight adjustments to the survey instrument (477 rns and 312 mds=789). eighteen percent (125) of the information questionnaires were returned in 1997. in 2009, sixteen percent (124) of usable surveys were returned. in 2015, one hundred thirty useful surveys were collected. clinicians were measured as to information barriers, resources, access points, smartphone use, and practice demographics. conclusions: we need to be out in the community improving access to health information. a longitudinal, cross-sectional study is a good methodology to map progress and trends. making changes in the community is hard. hospital purchases by large corporations affect outreach opportunities. research section awards de groote, lyon, henner, and kim powell hypothesis, vol. 28, no. 1, fall 2016 43 best hospital paper/poster authors: sarah sutton – clinical librarian, university hospitals of leicester title: has the distribution and role of clinical librarians in the united kingdom changed in the last ten years? abstract: objectives: to establish if the number of clinical librarians (cls) has changed over the eleven years since the first clinical librarian survey was conducted in 2004. to investigate whether newer roles such as embedded librarians or informationists have been established. to see if there is geographical variation in such roles and to gain data to enable further networking of these post holders. methods: a questionnaire was sent to the uk email discussion lists, also blogged and tweeted. the questionnaire was matched to the one used in 2004, so that trends could be identified. results: the results of the questionnaire were analysed. in 2004 26 librarians responded to the original survey and identified themselves as working as cls (although the actual job title may have been different) in 2015 47 respondents identified as cls, with an additional 32 identifying as outreach librarians, 9 as embedded librarians and 2 as informationists. the 2004 cls were largely full time, with 92% in this category, in 2015 the split was 58% full time and 42% part time. conclusions: clinical librarians have increased in numbers in the last ten years but so have other roles that mimic or build on the clinical librarian role. in 2004 clinical librarians were largely full time, in 2015 the part time roles have increased dramatically. a map will be included in the actual poster showing geographical distribution of the different roles. research section awards de groote, lyon, henner, and kim powell hypothesis, vol. 28, no. 1, fall 2016 44 best jmla research paper (2014-2015) authors: julie m. glanville, msc, associate director; steven duffy, pgdip, senior information consultant; rachael mccool, bsc, research consultant; danielle varley, msc, research assistant; york health economics consortium, university of york,, united kingdom title: searching clinicaltrials.gov and the international clinical trials registry platform to inform systematic reviews: what are the optimal search approaches? citation: j med libr assoc. 2014 jul;102(3):177-83. doi: 10.3163/1536-5050.102.3.007. abstract: background: since 2005, international committee of medical journal editors (icmje) member journals have required that clinical trials be registered in publicly available trials registers before they are considered for publication. objectives: the research explores whether it is adequate, when searching to inform systematic reviews, to search for relevant clinical trials using only public trials registers and to identify the optimal search approaches in trials registers. methods: a search was conducted in clinicaltrials.gov and the international clinical trials registry platform (ictrp) for research studies that had been included in eight systematic reviews. four search approaches (highly sensitive, sensitive, precise, and highly precise) were performed using the basic and advanced interfaces in both resources. results: on average, 84% of studies were not listed in either resource. the largest number of included studies was retrieved in clinicaltrials.gov and ictrp when a sensitive search approach was used in the basic interface. the use of the advanced interface maintained or improved sensitivity in 16 of 19 strategies for clinicaltrials.gov and 8 of 18 for ictrp. no single search approach was sensitive enough to identify all studies included in the 6 reviews. conclusions: trials registers cannot yet be relied upon as the sole means to locate trials for systematic reviews. trials registers lag behind the major bibliographic databases in terms of their search interfaces. implications: for systematic reviews, trials registers and major bibliographic databases should be searched. trials registers should be searched using sensitive approaches, and both the registers consulted in this study should be searched. research section programs mla16 marton and ascher hypothesis, vol. 28, no. 1, fall 2016 45 title research section programs at mla’16 authors christine marton marie ascher introduction this year’s annual conference was co-organized by the medical library association and the canadian health libraries association/ association des bibliothèques de la santé du canada. mosaic’16 was held in toronto, ontario at the metro toronto convention centre. although the weather was unseasonably cold, attendees enjoyed a wide range of programming and entertaining social events in downtown toronto, canada’s largest city and financial capital. the research section offered two section programs. the first program focused on systematic reviews and the role of the librarian while the second program focused on professional communication skills for presenting and publishing research. beyond the search was identified as one of the highest ranked conference programs in the july 7, 2016 issue of mla-focus. beyond the search: expanding the role of the librarian in the systematic review process session program 3 monday, may 16, 2016 1:00 p.m.–2:25 p.m. room 104c moderator: marie t. ascher, lillian hetrick huber endowed director, health sciences library, new york medical college–valhalla speakers: marie t. ascher, margaret j. foster, ahip, mark maceachern, and whitney a. townsend direct link: http://eventscribe.com/2016/mla/twitterpres.asp?pres=135613 http://eventscribe.com/2016/mla/twitterpres.asp?pres=135613 research section programs mla16 marton and ascher hypothesis, vol. 28, no. 1, fall 2016 46 librarians are increasingly involved with systematic reviews, most frequently as creators of the exhaustive search strategy as recommended by the institute of medicine guidelines, and also as organizers, archivers, and writers of the search method component. in this session a distinguished panel of systematic review experts will present strategies and tools for expanding the role of the librarian "beyond the search," to the project and data management aspects of the next steps during systematic review processselection, appraisal, and data extraction. this session will dispel the notion that librarians are only suited for the search component of the systematic review process by describing some of the methods to select and assess studies, extract data, and present results.. topics that will be covered include: * overall systematic review process * quality assurance through project management principles * screening and selection of studies, including how to calculate inter-rater agreement * risk of bias assessment tools, focusing on validated tools * data extraction tools and techniques * presentation of results and archiving data the goal of this session is to provide participants with an increased understanding of these components to move forward to expanding their roles in the overall systematic review process. professional communication skills: publishing and presenting your research tuesday, may 17 3:00 p.m. 4:25 p.m. room 203d moderator: patrick mclaughlin technical information specialist, medlars management section, national library of medicine, bethesda, maryland moderator: ariel deardorff assessment and data management librarian, nlm associate fellow, university of california, san francisco, ucsf library, san francisco, california https://www.eventscribe.com/2016/mla/searchbykeyword.asp https://www.eventscribe.com/2016/mla/searchbykeyword.asp research section programs mla16 marton and ascher hypothesis, vol. 28, no. 1, fall 2016 47 speakers: jacqueline wirz, natalie clairoux, joey nicholson, carole m. gilbert, ahip, fmla, cari markley, and i. diane cooper, ahip this program brings together leaders from the profession to share professional communication strategies. the session is divided into two communication areas: presentations and publishing. the presentation component focuses on how librarians can improve their presentation and public speaking skills. the publishing component focuses on how to write for publication, identify publication venues, and successfully navigate the publication process. the emphasis of this panel discussion is effective communication strategies for sharing information. panelists include jackie wirz (research data specialist, oregon health and sciences university), natalie clairoux (biomedical librarian, university of montreal / université de montréal), joey nicholson (education and curriculum librarian, new york university health sciences library), diane cooper (editor, journal of the medical library association), carole gilbert (editor, journal of hospital librarianship), and cari merkley (co-editor, journal of the canadian health libraries association / journal de l’association des bibliothèques de la santé du canada). metro toronto convention centre, south entrance, downtown toronto customizing the ipad derosa and keating hypothesis, vol. 28, no. 1, fall 2016 26 title customizing the ipad to support a clinical fellowship in radiology: a qualitative study authors antonio p. derosa, mds, mlis, ahip1 delia m. keating, md2 objective to determine the value of providing fellows in the breast imaging service (department of radiology) with ipad mini tablets pre-loaded with apps/resources/websites/tools, to test whether a tablet is a viable delivery platform for supporting the research needs of fellows. literature review there is little evidence in the published literature describing use of a tablet to support the general needs of a clinical group in the hospital setting. clinical information needs can range from bibliographic management support, research methods, literature searching, keeping current with research in the field, and workflow and knowledge organization support, to name a few. most of the literature consists of using tablets/ipads for educational purposes in various settings, both clinical and educational [1-5] the most closely related research found to the content of this study was the sharpe et al. study from 2013, which focuses on delivering a “radiology resident ipad toolbox” to the residents at the university of colorado [6] the approach taken in the sharpe study is similar to our study in that the investigators come up with a collection of resources to support the needs of the residents. the major difference is that the sharpe study investigators anonymously surveyed the residents three months after program implementation. our plan includes surveying the fellows preand post-clinical rotation. customizing the ipad derosa and keating hypothesis, vol. 28, no. 1, fall 2016 27 research questions what is the value of supplying fellows with ipad mini tablets pre-loaded with productivity apps, pertinent e-books, diagnostic tools, health information resource pathways, and current awareness resources? does a tablet with selected resources enhance the fellows experience during a one-year clinical rotation in a radiology service? methods the approach to gathering data to answer these questions consisted of preand post-clinical rotation survey questionnaires, administered via surveymonkey online. during the annual fellows orientation, the research informationist introduced the ipad and provided an overview of the various clinical, productivity, and research tools available—at which time the first questionnaire was administered to garner initial impressions on the ipad and its resources. the post-survey was administered during the exit interview after the fellows’ rotation was complete. all fellows (n=6) completed the preand post-questionnaire as a mandatory component of their fellowship. all fellows agreed to participate in this irb-waived study. survey results were analyzed using qualitative techniques by both the research informationist and the attending radiologist who oversees the education program for rotating fellows and residents. a qualitative analysis approach and survey design was decided upon as the most appropriate method for garnering anecdotal feedback from fellows and creating a narrative of survey results. see prefellowship survey for questions asked (post-fellowship survey grammar was adjusted to reflect the past tense, but all questions remained the same). customizing the ipad derosa and keating hypothesis, vol. 28, no. 1, fall 2016 28 pre-fellowship survey: 1. at first glance, the resources provided on the ipad (apps, websites, tools, journals, etc.) seem to meet my productivity and workflow needs. strongly agree agree undecided disagree strongly disagree 2. based on personal importance and what you think will be the top items you will use each day, please rank the following resources on the ipad (1=most important, 7=least important). ebooks ejournals databases productivity tools (includes email, pdf annotation app, and microsoft office apps) web browser internal msk webpages clinical medical librarian (your information specialist contact) 3. generally, how often do you think you will use the ipad and its resources? once a day more than once a day once a week more than once a week monthly 4. how do you think you will most use the ipad in general or the resources available? please select all that apply. i will not use the ipad or its resources change approach to a particular patient and/or future patient(s) facilitate sharing/discussing information with colleagues presenting research or publish apply for grant funding revision of clinical pathways, practice guidelines, policies or procedures customizing the ipad derosa and keating hypothesis, vol. 28, no. 1, fall 2016 29 results comparing the preand post-rotation survey results determined future perspectives on continuing an ipad program under the responsibility of the research informationist and assisted in prioritizing the selected resources and identifying future training opportunities. as the survey results suggested, 100% (n=6) of the respondents either agreed or strongly agreed that the resources/apps provided on the ipad met their productivity or workflow needs during their rotation. this was a change from the responses pre-rotation where 83% of fellows agreed/strongly agreed and 16% were undecided at the time. most free-text comments to this question consisted of the usefulness of the ipad for referencing the american college of radiology’s bi-rads clinical e-book which provides standardized breast imaging findings terminology, report organization, assessment structure and a classification system for mammography, ultrasound and mri of the breast [7]. when asked about why type of resources were most important for their clinical/research needs on the ipad, e-books and productivity tools (including ms office and pdf annotation app) received the most votes with 83%. the top resources reported during the pre-rotation survey for this question included productivity tools, web browsing, and internal web pages, all at 83%. when questioned about prospective usage of the ipad versus actual usage of the device, the responses varied from preto post-rotation. before beginning their fellowship, 67% of respondents thought that they would use the ipad more than once a day. upon completion of their fellowship, only 16% reported using the ipad and its resources more than once a day. the final question to the survey consisted of how fellows thought they would most use the resources or the ipad, generally. pre-rotation survey responses indicated that 83% would use the ipad and its resources for sharing/discussion with colleagues, as well as, for general research or publishing. post-rotation answers showed that the ipad was most used for presenting and/or general research/publishing (50%), as well as, revision of clinical pathways (50%). customizing the ipad derosa and keating hypothesis, vol. 28, no. 1, fall 2016 30 discussion although the survey results indicated that the ipads were used less frequently than originally anticipated by the fellows, the weighted importance of the resources provided and the impact of using the ipad throughout their fellowship was positive and well-received overall. noteworthy findings include the shift in initial thoughts on reasons for ipad use versus how the ipads were actually used during the fellowship. specifically, the shift from a more general-purpose research focus to a more clinical focus in terms of usage shows an opportunity for supporting more clinical research needs of the fellows. this could include assistance with gathering the highest level evidence for difficult/interesting patient cases, and/or providing guidance in organizing clinical data for analysis and interpretation. by examining the survey responses, changes or enhancements can be made to the resources/apps offered on the ipad for future fellowship cohorts. for instance, key ebooks can be made more prominent on the device as this type of resource ended up being used frequently (and voted highly important) by the fellows. outlook on the use of the liaison research informationist services (literature search, project guidance, bibliographic management tool support, general training/consult, etc.) remained the same both preand post-rotation. the services provided by the research informationist can be adapted to be more relevant and supportive of the actual research and clinical endeavors undertaken by the fellows. this can take the form of training on targeted resources (such at the acr bi-rads) and key research services offered by the library (such as scholarly communication and getting published). enhancing the informationist/library component of the fellows’ research support services by way of the ipad, could have a positive impact on the fellows’ experience with the mobile device. limitations the customization of the ipad was well received among fellows in this pilot study but had certain limitations. where it fell short was its inability to handle file transfer/storage of documents. fellows reported consistently reverting back to using more traditional technology like a laptop or desktop computer to store and access large files. this is an inherent limitation of the ipad. customizing the ipad derosa and keating hypothesis, vol. 28, no. 1, fall 2016 31 conclusions this survey study has revealed a positive impact, value, and experience on having a customized ipad during a clinical fellowship. data gathered suggest that the ipad was a time-saver and provided in-context and in-the-moment resources. a noticeable upswing in research consultation and informal training requests were observed, although adapting to the changing needs of the fellows to offer more targeted training would be beneficial. including more clinical researchfocused resources/apps to the ipad would help to bring more value to the ipad program as fellows reported using the ipad to revise clinical pathways and support their clinical work. future directions more research in the area of providing technology solutions to clinical focus areas is needed to determine the lasting impact and value of this type of service. the pilot ipad program described in this study can be expanded to other clinical groups at other institutions. librarians and informationists are in a unique position to facilitate and disseminate knowledge in a variety of mediums and formats. using the collective skills of the information and knowledge management profession could lead to added benefits of offering technology solutions to support the research and patient care needs of targeted clinical groups. references 1. brown j, mccrorie p. the ipad: tablet technology to support nursing and midwifery student learning: an evaluation in practice. computers, informatics, nursing. 2015;;33(3):93-98. doi: 10.1097/cin.0000000000000131 2. george, p., newey, c. r., & bhimraj, a. (2015). the tablet device in hospital neurology and in neurology graduate medical education: a preliminary study. neurohospitalist, 5(1), 15-21. doi: 10.1177/1941874414548802 3. gerstle, t., hassanein, a. h., & eriksson, e. (2015). ipad integration in plastic surgical training: optimizing clinical efficiency, education, and compliance with the health insurance portability and accountability act. plast reconstr surg, 135(1), 223e-225e. doi: 10.1097/prs.0000000000001014 customizing the ipad derosa and keating hypothesis, vol. 28, no. 1, fall 2016 32 4. mccreadie, s. r., & mcgregory, m. e. (2005). experiences incorporating tablet pccs into clinical pharmacists' workflow. j healthc inf manag, 19(4), 32-37. 5. soma, d. b., homme, j. h., & jacobson, r. m. (2013). using tablet computers to teach evidence-based medicine to pediatrics residents: a prospective study. acad pediatr, 13(6), 546-550. doi: 10.1016/j.acap.2013.05.028 6. sharpe, e. e., 3rd, kendrick, m., strickland, c., & dodd, g. d., 3rd. (2013). the radiology resident ipad toolbox: an educational and clinical tool for radiology residents. j am coll radiol, 10(7), 527-532. doi: 10.1016/j.jacr.2013.02.007 7. american college of radiology. (n.d.). retrieved august 09, 2016, from http://www.acr.org/quality-safety/resources/birads author affiliations 1 research informationist, memorial sloan kettering cancer center library, new york 2 radiologist, breast imaging service (department of radiology), memorial sloan kettering cancer center, new york editorial board 2016 profile: ayaba logan, mlis, mph ayaba logan earned her bachelor of art in sociology and public health from spelman college. she went on to pursue a master of public health from the university of michigan, and a master of library and information studies from the university of alabama. mrs. logan is interested in qualitative research, systematic reviews, citation analysis, assessment and evaluation, data visualization, and instruction. currently, ayaba is a research and education informationist at the medical university of south carolina under the leadership of shannon jones, med, mis, mls. ayaba has joint appointments with the department of public health sciences (college of medicine) and college of nursing. as an informationist/librarian, ayaba is heavily involved with conducting systematic and other reviews and library instruction evaluation/assessment. lastly, ayaba enjoys time with her hubby, their daughters, and traveling abroad. http://www.acr.org/quality-safety/resources/birads hypothesis, vol. 27, no. 1, winter 2015 8 report on the hypothesis and research section communication survey kristine m. alpi, mls, mph, ahip1 and brooke l. billman, ma, ahip2 the research section (rs) of the medical library association (mla) has published hypothesis, an open access journal formerly structured as a newsletter, since summer 1987. in order to plan for the future of rs’ research-related communications and strategies, the rs executive board and the hypothesis editorial board decided to seek information from the health sciences information (hsi) community about their engagement with and interest in publishing in hypothesis. objective the purpose of this investigation was to gain insight from the hsi community about the current usage of the rs open access journal hypothesis and possible improvements that would meet the needs of those interested in learning more about hsi research. methods members of the hypothesis editorial board developed an online survey to gather information from mla members and others. it was reviewed by the rs executive board, but as the survey was anonymous and meant for section quality improvement, no institutional review board review was pursued. the link to the 20-24 question online survey in survey monkey was distributed on october 7, 2015 to the rs email list, announced in the october 15 mla focus, and linked from the mla webpage hosting hypothesis content. we are unsure whether it was forwarded to other lists. the online survey was open until november 1, 2015. results the survey was completed by 53 individuals. while distribution via lists is uncertain, using the mla membership of approximately 3,000 as a denominator, the response rate was approximately 0.02%. of respondents providing demographics, 98% (n=42) were current mla members and 61% (n=27) were current or past rs members. forty (98%) were based in the united states. although most respondents were in academic health science centers (n=27, 64%) or a college/ university (n=8, 19%), five hospital librarians (12%), one working outside of libraries (2%), and one retiree (2%) also responded. no library or ischool faculty or students responded, nor did any corporate, public or specialty librarians. responses were received from four (10%) librarians with three or few years of experience, sixteen (38%) with 4-10 years, twelve (29%) with 11-20 years, and ten (24%) with more than 21 years. multiple responses were permitted for many non-demographic questions resulting in percentages exceeding 100%. in descending order for frequency, the most common avenues for becoming aware of hypothesis were the rs member email list (n=20, 42%), mla focus (n=10, 21%), other mla publication or email list (n=4 , 8%), lista (n=2, 4%), cinahl (n=1, 2%), and forwarded messages from other lists (n=1, 2%). no one reported finding it searching the web, and ten respondents (21%) were unaware of hypothesis. additional avenues described in comments included being told by rs members in person at the mla meeting (n=1) and when accessing the structured abstract writing guidelines published in hypothesis used for mla meeting submissions (n=2). several of the questions were intended for the respondents who were aware of or read hypothesis. when asked about frequency of reading it when it was published 2-3 times per year, only ten respondents (19%) read every issue, with an additional eleven (21%) reading at least one, but not all, issues per year. surprisingly, 32 (60%) of the 53 respondents said they never read it; this presumably includes the ten who were not aware of it. of the 17 indicating how much of each publication of hypothesis they read, most (n=8, 47%) read around half of each issue while 29% (n=5) read less than half and 24% (n=4) read more. twenty respondents indicated why they read hypothesis. percentages for each offered reason are shown in figure 1 and equal more than 100% as multiple responses could be selected. the three “other” responses are summarized as 1) reading about various research methods, 2) gaining knowledge about methods and techniques in medical librarianship research, and 3) using it as a forum to publish with research mentees. readers were asked to rate the usefulness of content published in hypothesis on a scale from no opinion/notapplicable (0), not very useful (1), somewhat useful (2), to very useful (3) in three contexts. after removing no opinion responses, the average score for usefulness to the respondents’ current position or educational program (n=11) was 2.0 (sd 0.4) and the average usefulness for a desired position (n=10) was similar at 1.9 (sd 0.7). usefulness to respondents’ research whether related to position or not (n=12) averaged 2.3 (sd 0.7). many types of content have appeared or could appear in hypothesis. respondents indicated the likelihood of reading each of the following content types on a scale ranged from would not read (0) to very likely to read (3). table 1 shows the mean likelihood of reading this content hypothesis, vol. 27, no. 1, winter 2015 9 figure 1. why respondents read hypothesis, n=20 report on the hypothesis, continued if presented in hypothesis. almost all content types were rated at least somewhat likely to be read (2.0). those most often rated very likely to be read were articles reporting original research (n=24, 56%) or articles on research methods and strategies (n=26, 62%). content suggested under “other” were 1) failed research lessons learned, 2) reviews of library research-related textbooks, and 3) summaries of new technology applications to library science. another content question asked whether informal peerreviewed project write-ups would be of interest. as currently envisioned, these would be geared towards librarians who are doing projects and producing outcomes, and perhaps even changing their practices based on their results. the intent is to be able to share outcomes, lessons learned, successes, and failures so that those in the hsi community can learn from one another. of the 46 respondents who answered this multichoice question, more than half would like to read writeups in hypothesis (n=27, 59%) while the idea of reading these in an online, blog-type format and not having to wait for an issue of hypothesis was preferred (n=32, 70%). only two responded that they would not be interested in reading these regardless of format. the open-ended question “what would make you more likely to read hypothesis?” generated twelve responses which were classified into discoverability characteristics (n=7) and content characteristics (n=5). ideas shared related to discoverability included:  more awareness and communication via email (tocs, specific reference to what hypothesis is)  announcements in existing channels like mla focus or mla news or tweeting about new issues/article,  individual indexing of articles for inclusion in google scholar and being easier to find on the web. content suggestions offered diverse opinions. one suggested reporting research that has been through irb review (expedited or otherwise), another suggested featuring photography by one librarian in each issue, and two others’ commented about the practice of research and publishing (shared in the respondents’ own words):  “having my colleagues submit content about what really happens with our research efforts in libraries of all sizes not just academic medical center libraries.”  “any journal that understands that librarians are, first and foremost, customer service agents would pique my attention. that journal would cater to real-world issues. journals proposing that librarians are on par with actual scientists/researchers seem always to struggle with the fact that submissions aren't "scientific" enough….” the average likelihood of contributing the content types to hypothesis appears in table 1. the scale ranged from would not contribute (0) to extremely likely (3). mean scores all fell between not very likely (1) to somewhat likely (2). the types of content with the most respondents alpi and billman hypothesis, vol. 27, no. 1, winter 2015 10 table 1. likelihood of reading or contributing content to hypothesis, n=41-43. reading likelihood mean score contributing likelihood mean score articles reporting original research by health sciences librarians/ information professionals (hslip) 2.5 1.7 articles on research methods or strategies that have been/could be used by hslip 2.5 1.6 summaries of research conferences or training opportunities 2.2 1.1 summaries of original health informatics and librarianship dissertations and theses. 2.0 1.0 abstracts of award-winning presentations at mla annual and chapter meetings 2.0 1.2 announcements of research conferences or training opportunities 2.0 1.3 announcements of research funding opportunities 1.6 1.2 indicating somewhat or extremely likely to contribute were articles reporting original research (n=24, 56%) or articles on research methods and strategies (n=26, 62%). other content contribution issues mentioned were original research with a research mentee and the desire only to report summaries of original research which is planned for publication in a peer-reviewed journal. for the previously mentioned informal peer reviewed project write -ups, half of respondents (n=21, 50%) would be interested in sharing these in hypothesis while one-third (n=14, 33%) would be interested in sharing via an online, blog-type format and not waiting for an issue of hypothesis. others (n=17, 40%) expressed that they might be interested, but don’t have a project like that in the works, or that they publish projects like that elsewhere (n=3, 7%). when asked whether they would follow an online forum to discuss publications read in hypothesis, a majority said maybe (n=26, 54%) or yes (n=8, 17%) while fourteen (29%) indicated they would not. there were no majorities on preferred comment features of an online forum such as required sign-in (n=14, 30%), full name (n=17, 36%) or first name display (n=7, 15%), and editorial review (n=8, 17%). many respondents (n=18, 38%) had no opinion on features of such an online forum. of the 43 respondents who answered whether or not they had considered publishing in hypothesis, three respondents (7%) indicated they had published and two (5%) were considering it. the vast majority (n=38, 88%) had neither considered submitting or had submitted an article. when asked how important a list of specific characteristics was in making a decision to submit work, more than half of respondents included peer-reviewed (n=32, 74%), no author fees (n=30, 70%), and online submission system (n=22, 51%) as being very important. average scores for each characteristic are provided in table 2. the preferred publication frequency of hypothesis was twice a year (n=18, 43%) followed closely by more than twice a year (n=16, 38%), with eleven (26%) preferring on an ongoing basis as content arises. only two suggested annual publication. two questions in the survey were presented only to rs members. the first asked about the process and amount of a possible stipend for the hypothesis editor. almost half (n=13, 45%) indicated a majority vote of rs members should be required to approve spending dues money on a stipend for the hypothesis editor regardless of the amount. only three (10%) felt it should be up to the editor whether to request a stipend for editing hypothesis. in terms of the amount, the responses are listed in order of frequency:  a stipend up to $100 per issue seems reasonable. (n=14, 48%)  a stipend up to $100 per year, regardless of the number of issues, seems reasonable. (n=5, 17%)  no stipend should be offered. (n=3, 10%) five usable comments were offered. two respondents raised questions to inform the process— is a stipend typical; is there a standard for this type of work; what is the level of effort on the part of the editor; and is a stipend needed to attract a responsible editor? the other three comments related to the amount, suggesting 1) it should be tied to the overall mla operating budget, 2) a recommendation of $250 as more reasonable considering all the work involved, and 3) that it seems reasonable but not knowing whether $100 an issue or $100 a year would be needed. report on the hypothesis, continued alpi and billman hypothesis, vol. 27, no. 1, winter 2015 11 table 2. potential authors’ interest in various publication characteristics average rating (0=not important, 1=not very important, 2=somewhat important, 3=very important) response counts peer reviewed 2.7 43 no author fees 2.6 43 indexed by pubmed 2.4 43 online submission system 2.3 43 time from acceptance to publication 2.3 43 open access 2.2 43 download statistics available 1.9 42 online peer review system 1.8 42 indexed by lista 1.8 42 indexed by cinahl 1.7 43 public commenting available 1.0 42 responses for the second question (n=29) about how members feel about hypothesis as part of the research section activities are listed in order of frequency of response below:  i would like to see more opportunities for members to be involved in hypothesis. (n=18, 62%)  i am proud that our section produces a peer reviewed online journal. (n=16, 55%)  i have been disappointed in the absence of hypothesis issues in the recent past. (n=9, 31%)  i don’t give any thought to hypothesis. (n=7, 24%)  hypothesis is redundant with other online journal or newsletter publications. (n=2, 7%)  hypothesis is redundant with other avenues of section communication available. (n=1, 3%) five respondents provided comments for the research section to consider. two related to awareness of hypothesis, with one having learned about hypothesis years ago but having forgotten about it until this survey and the other having not stumbled upon hypothesis in spite of being an rs member for years. two comments related to the various features of hypothesis. for one respondent, the importance of various features in choosing to publish depended on the type of publication; for original research indexing and peer review is very important, but for how-to type method articles, conference summaries, or critically appraised topics, those characteristics are not as important. the fifth comment is quoted here directly: “in theory i like hypothesis, but it doesn't give enough back for the effort put into it…lack of pubmed indexing, inability to see distinct articles without looking at the whole issue, etc…it seems too much like a newsletter and nothing like a journal.” finally, the majority of respondents (n=33, 75%) preferred to learn about the survey results from an mla focus announcement. additional avenues were posting to the mla rs section list (n=22, 50%) or linked from the rs website (n=19, 43%). six respondents (14%) preferred the eblip list. discussion a limitation of this study is that it only describes the usage of hypothesis by respondents who were primarily mla members in the united states. the online usage data for the rs website which includes hypothesis shows that only 34% of page views are from the united states with the next highest countries as china (22%), great britain (7%) and italy and the russian federation with 4% each. this suggests an international audience for hypothesis and other rs communications that did not participate in this survey. as no responses were received from library or ischool faculty or students, it is unclear whether hypothesis has value for educational activities outside of its primary audience of practicing librarians. it is not surprising that there is a lack of awareness of hypothesis since the last issue published was summer 2014. while it seems that there is an audience for many types of content based on the likelihood of reading scores, it is less certain whether there are sufficient potential authors to provide this content given the likelihood to contribute. report on the hypothesis, continued alpi and billman hypothesis, vol. 27, no. 1, winter 2015 12 hypothesis is an mla rs communication tool and ultimately decisions about the value of the journal related to the work involved in its production are internal. the rs leadership intends to use the feedback gathered from this survey to improve rs communication of research-related content to a variety of audiences. the findings will be disseminated through mla focus and email list postings with a link to this specific article in hypothesis. presenting the findings in this manner allows for both the indexing and discoverability requested by survey respondents, but also shares our questions and methodology with other organizations planning to evaluate their communication strategies. acknowledgements thank you to all the respondents who took the time to participate in the survey. the hypothesis editorial board and other rs members who contributed to developing this survey were kristine m. alpi, brooke l. billman, leslie behm, ellen detlefsen, jonathan eldredge, and heather holmes. author affiliations 1 director, william rand kenan, jr. library of veterinary medicine, north carolina state university, raleigh, nc; kmalpi@ncsu.edu 2 azhin & special projects librarian, university of arizona health sciences library, university of arizona, tucson, az; brooke.billman@ahsl.arizona.edu report on the hypothesis, continued alpi and billman mailto:kmalpi@ncsu.edu mailto:brooke.billman@ahsl.arizona.edu research hypothesis vol. 31 no.1 fall/winter 2019 benchmarking study of hospital libraries angela spencer, mals assistant professor, health sciences reference librarian saint louis university st. louis, mo elizabeth mamo, mls library director regional health rochester, ny brooke l. billman, mlis, ahip senior manager, governance operations college of american pathologists northfield, il abstract objectives: to assess the current landscape of hospital libraries by collecting benchmarking data from hospital librarians in the u.s. and other countries. since the last mla benchmarking survey in 2002 hospital libraries have faced significant changes including downsizing, position and library elimination, and hospital mergers. this survey will provide information to inform the development and implementation of effective advocacy for hospital libraries. methods: a web-based, anonymous survey was designed to collect information from hospital librarians representing stand-alone hospitals and hospital systems. the 57question survey was distributed via select listservs, targeting the us and canada but open to any country. the topic areas covered hospital/health system, library, and library staff demographics; library characteristics and scope of service; interlibrary loan and document delivery; library funding; and library budget. hospital library benchmarking surveys, including the previous mla surveys, were reviewed and applicable questions were added. results: there were a total of 180 respondents, but the total number of responses for each question varied. select results are as follows: of the responding libraries, 67.2% were part of a hospital system; 24.4% had merged with or were bought by another hospital or health system and, of those, 77.1% had acquired 1-5 hospitals in the last 10 years; 77.9% were not for profits; over half (55.2%) had <5,001 fte in the organization; 56.9% had one library; 47.7% had 1 fte librarian, 34.9% had 2-5; 82.1% did not or were not able to use social media; 60.7% didn’t have strategic plans; 66.1% belonged to a consortium; 48.2% provided up to 250 search requests a year; 66.3% did not receive funding outside of their organization; 32.5% had budgets for print books totaling less than $1,000; 30.1% had budgets, excluding salaries, of less than $100,000 and 9.7% had budgets over $1m. research hypothesis vol. 31 no.1 fall/winter 2019 conclusions: these findings contribute to the field’s knowledge of hospital library demographics as well as the services provided. the results suggest implications for hospital librarians regarding staffing levels and the depth of services within their unique settings, especially within the context of rapidly expanding health systems. introduction in a time of increased health care system mergers and hospital library closures, library staff are looking for ways to advocate for the retention of services, resources, personnel, and, sometimes, the mere existence of the library. compared to other libraries, those in hospitals face unique challenges in that they are evaluated based on their ability to generate revenue. since hospital libraries are not usually revenuegenerating, they are often under scrutiny and the value of the library and its services are questioned. this is especially true when the hospital operating budget is cut or the organization is involved in a healthcare merger, either of which may lead to library closures. according to kaufmannhall, in 2017 alone there were 115 health system mergers and acquisitions in the united states [1]. hospital libraries within health systems that have merged are faced with integrating library services and resources across multiple sites. this is a task that often requires more library staff and funding, not less [2]. as these hospitals grow in services and size, library operations don’t grow with them and physical space may be downsized [3] and library locations may close. although there does not exist a single, comprehensive resource that tracks library closures, a review of the literature reveals a disturbing trend. over a decade ago the mla vital pathways project found a decline in the number of hospitals with libraries, from 44% in 1989 to 30% in 2006 [4]. a more recent attempt to determine the rate of closures was a study in which the docline membership database was queried and it was found that 613 nn/lm member libraries had closed from 2011-2015 [5]. harrow et al compared the 2007 edition of the library and book trade almanac with the 2017 edition and noted a 32.7% decrease in medical libraries, including hospital libraries, from 2,055 in 2007 to 1,384 in 2017 [6]. previous hospital benchmarking studies are now outdated, especially with the increase in health care mergers and library closures. the aim of this study was to collect data about hospital library characteristics, services, populations served, and budgets. this information will provide library staff and hospital administrators with current data to inform their plans for library services. library staff can also use this information when advocating for the needs of their libraries. research hypothesis vol. 31 no.1 fall/winter 2019 history of hospital library benchmarking hospital library benchmarking studies have been conducted since the early 1960s, beginning with a survey by the american hospital association’s division of research which collected information from a sample of hospitals in their registry. select data was analyzed and it was recommended that improvements were needed for libraries to function successfully [7]. the american medical association, in partnership with case western reserve university and the american hospital association, issued three surveys (1969,1973, 1979) of health sciences libraries which resulted in the publication of the directory of health science libraries in the united states [8-10]. each edition of the directory provided a listing of libraries by state and name. the directories contained a breakdown of libraries by sponsoring organizations and by number of staff. crawford analyzed results of the three surveys to determine patterns, growth, or decline. overall there was in increase in the number of libraries in hospitals in the 10-year period [11]. by the 1980s, hospitals were experiencing an increase in health care costs and institutional mergers. wos surveyed hospitals that were part of multi-site systems; data illustrated the impact of mergers on library collections, staffing, and budgets [12]. in 1990, hospital libraries in michigan provided survey responses for two time periods and it was concluded that these libraries were adversely affected by mergers with noted reductions in staff, hours, and resources [13]. the author called for the ongoing collection of hospital library data. also in 1990, the american hospital association surveyed registered hospitals in the united states [14]. two years later, glitz conducted a survey of hospital libraries in the pacific southwest medical library region, comparing the data to the 1989 version of the same survey, noting decreases in libraries and staff [15]. in the late 1990s mla observed an increase in closures of hospital libraries and established the mla benchmarking network. dudden described the development of a task force to create a benchmarking tool, issue surveys, and share the results with mla members [16]. surveys were distributed in 2002 and 2004 and select data were presented in publications, conference proceedings, and made available on an interactive website. mla members could select parameters applicable to their situation and obtain data to share with their administrators. for example, bertolucci used the data to increase her library’s book budget [17] and fama used the data to revise her library’s hours of operation [18]. the mla hospital libraries section standards committee used the data to develop and publish a formula for library staffing, which was included in the 2002 mla hospital libraries standards [19]. meanwhile, mla’s support evolved into the task force on vital pathways for hospital libraries. the task force distributed a new survey in 2005. thibodeau compared the results of the 2005 survey to the 1989 american hospital association survey results and noted a decrease in staffing [4]. thibodeau also noted the need to track the status research hypothesis vol. 31 no.1 fall/winter 2019 of hospital libraries as there wasn’t a way to accurately quantify closures. later that decade vanmoorsel distributed a survey of hospital library staffing and compared the results to the mla standards for hospital libraries, finding a negative variance from the mla standard [20]. similarly, ducas surveyed canadian hospital libraries in 2012 and compared the results to the canadian hospital library association (chla) standards for library staffing and found that many libraries did not have a master’s prepared librarian, or adequate staffing levels according to the chla standards [21]. methods in august 2017 an online survey using surveymonkey was distributed in order to collect and assess hospital library benchmarking data. a three-member planning team developed and pilot-tested the survey instrument (appendix a), consisting of 57 questions with integrated skip logic. the survey was reviewed and determined to be exempt from human subject review by the institutional review board at rochester regional health on july 19, 2017. the call to participate in the survey was sent to various health science librarian online distribution lists including medlib-l and the mla hospital libraries section on august 16, 2017 and the survey closed on september 15, 2017. reminders were sent during this period. identifying information, such as ip address, respondent name, and organization name, was not collected. the survey indicated that just one library representative from each hospital or hospital system should fill out the survey. for open-ended questions the constant comparative method was used; two authors independently coded the responses and resolved discrepancies through discussion [22]. a total of 180 respondents participated in the survey. results institutional characteristics of the total responses, 91.6% were from the united states, 7.3% from canada, and 1.1% from new zealand. one-third (32.8%) of respondents indicated they were from a hospital that is not part of a health system while two-thirds (67.2%) were. about thirty percent (30.1%) were from a system with 2–5 hospitals, 12.5% with 6–10, 6.8% with 11–15, and 18.2% with more than 15. the majority of respondents (30.5%) indicated they were from hospitals/hospital systems that had 201–500 beds, while 16.7% had 200 or fewer beds, 20.7% had 501–1,000, 14.4% had between 1,001-2,000 beds, and 17.8% had more than 2,001 beds. just 8.7% of hospitals/health systems had 1,000 or fewer full time equivalent (fte) employees while 46.5% had 1,001–5,000 fte, 16.9% had 5,001–10,000 fte, 20.9% had 10.001–50,000, and 6.9% had over 50,001 fte. most (91.0%) of the respondents were from teaching hospitals. twenty-four percent (24.4%) of the respondents’ hospital or hospital systems had merged or been acquired by another organization in the last 10 years. of the locations that have research hypothesis vol. 31 no.1 fall/winter 2019 acquired hospitals, 77.1% had acquired 1–5 sites, 10.8% acquired 6–10 sites, and 12.1% had acquired 11 or more sites. library characteristics over half (56.9%) of all respondents had 1 library within their hospital/hospital system, 32.8% had 2–5 libraries, 5.2% had 6–10, and 5.2% had more than 10 libraries. figure 1 is a cross tabulation of the number of libraries and the number of hospitals and figure 2 is a cross tabulation of number of libraries and number of beds. half (50.0%) of respondents indicated that their library supported 1 hospital while 31.2% supported 2–5 hospitals, 9.4% supported 6–10, and 9.4% supported more than 10. the number of fte librarians ranged from fewer than 1 to more than 15: 5.8% had fewer than 1 fte, 50.6% had 1 fte, 35.5% had 2–5, 2.9% had 6–10, and 4.6% had more than 10. one respondent (0.6%) was not an employee but a volunteer. figure 3 is a cross tabulation of librarian fte and number of sites supported. half (52.0%) of libraries did not have support staff while 39.3% had 1–3, 6.9% had 4–7, and 1.7% had 8 or more staff. research hypothesis vol. 31 no.1 fall/winter 2019 the department(s) the library/libraries reported to included education (e.g., graduate medical education, nursing education) (44.1%), administration (41.3%), research hypothesis vol. 31 no.1 fall/winter 2019 information technology (6.1%), human resources (5.6%), research services (4.5%), and other departments (6.7%). the majority of libraries (60.7%) did not have a strategic plan and 78.8% did not have a formal marketing or communications plan. almost all (97.1%) respondents had physical library space. many (40.6%) respondents did not have a library website. of the respondents who did, the following departments managed it: library (50.0%), marketing (29.0%), information technology (16.0%), public affairs (3.0%) and other (2.0%). most (91.7%) of respondents had an internal library website or intranet. of those respondents, the following departments managed it: library (66.2%), information technology (22.1%), and marketing (11.7%). off-site access to electronic resources was provided by the hospital (39.2%), the library (34.5%), both the hospital and library (12.9%), 10.5% provided no access off-site, and 2.9% were unsure. most (84.4%) libraries did not use social media. most (80.7%) library staff did not have access to the electronic health record while 9.4% had full access and 9.9% had access only to the module that allows staff to provide links to resources (e.g., via the infobutton feature.) populations served most health care professionals are served by the library with respondents indicating they serve nurses (94.6%), physicians, (94.0%), pharmacists (92.2%), allied health staff (89.8%), residents (78.3%), and other clinical staff not already categorized (88.6%). others served include administration/management (92.2%), students (85.5%), researchers (76.5%), patients (61.4%), faculty (59.0%), visitors (56.6%), and other nonclinical staff not already categorized (83.1%). of the respondents who provided services to residents, 63.6% served up to 100 residents, 31.0% served 101–500 residents, and 5.4% provided services to over 500 residents. a cross tabulation of librarian fte and number of residents served is available in figure 4. of the respondents who provided services to fellows, 89.4% served up to 100 fellows and 10.4% served over 100 fellows. research hypothesis vol. 31 no.1 fall/winter 2019 library services respondents selected services provided by their libraries. “other” responses were coded and included in table 1. table 1. services provided by respondents services % literature searching 100.0% interlibrary loan 98.8% teaching/instruction 87.4% cataloging 85.0% intranet/internet pages 72.0% meeting space 60.5% patient education 52.7% consumer health 52.1% research hypothesis vol. 31 no.1 fall/winter 2019 publishing support 45.5% systematic reviews 32.3% libguides 30.5% cme 23.4% clinical medical librarian program/patient care rounding 22.2% media services 11.9% data management 12.0% internal review board 9.0% social media 5.4% the number of search requests respondents received each year ranged from up to 100 (20.5%), 101–250 (27.7%), 251–500 (21.1%), 501–750 (10.2%), 751–1,000 (10.2%), and more than 1,000 (10.2%). figure 5 is a cross tabulation of the number of searches and hospital/system fte and figure 6 is a cross tabulation of the number of searches and bed count. most (86.7%) of respondents received consumer health questions and, of those who did, 57.3% received up to 25 questions a year while 21.0% received 26–100 questions and 21.7% received over 100 questions a year. a third (34.1%) of respondents maintained the hospital/health system archives. research hypothesis vol. 31 no.1 fall/winter 2019 respondents provided the following number of articles from their collection to those within their organization in one year: up to 250 (22.5%), 251–500 (6.9%), 501– 1,000 (18.1%), 1,001–2,500 (15.6%), 2,501–5,000 (11.9%), and over 5,000 (25.0%). research hypothesis vol. 31 no.1 fall/winter 2019 figure 7 is a cross tabulation of document delivery requests and hospital/hospital system fte. articles requests filled each year via interlibrary loan (ill) were as follows: up to 100 (21.3%), 101–250 (17.7%), 251–500 (20.1%), 501–1,000 (15.2%), 1,001– 2500 (13.4%), 2,501 or more (12.2%). figure 8 is a cross tabulation of the number of ill requests and hospital/hospital system fte. research hypothesis vol. 31 no.1 fall/winter 2019 two thirds of respondents (68.1%) spent up to $1,000 a year on article requests, excluding copyright fees, 24.1% spent between $1,001–$5,000, and 7.8% spent over $5,000. most respondents (83.0%) spent up to $500 a year on copyright fees. the response to whether the library was a loansome doc provider was almost split with 45.2% indicating they were. almost half of the libraries (47.9%) borrowed up to 10 physical books a year from other libraries, while 34.5% borrowed 11–50, and 17.6% borrowed more than 50 per year. most respondents (71.7%) lent up to 10 books a year to other libraries. library budgets budgets, excluding staff salaries, varied greatly with 12.2% up to $50,000, 17.9% from $50,001–$100,000, 19.2% from $100,001–$200,000, 28.8% from $200,001– $500,000, 12.1% from $500,001–$1,000,000, 5.8% from $1,000,001–$2,500,000, 2.6% from $$2,500,001–$5,000,000, and 1.3% over $5,000,000. figure 9 is a cross tabulation of library budgets and hospital fte. research hypothesis vol. 31 no.1 fall/winter 2019 for staff salaries, 11.5% of respondents had a total budget for staff salaries, excluding benefits, of up to $50,000 while 34.6% had budgets from $50,001–$100,000, 33.3% from $100,001–$250,000, 13.5% from $250,001–$500,000, and 7.1% over $500,001. figure 10 is a cross tabulation of library salary budget, excluding benefits, and hospital ftes available. research hypothesis vol. 31 no.1 fall/winter 2019 over a third (39.6%) of respondents had a journal budget of up to $50,000, 22.0% from $50,001–100,000, 22.0% from $100,000–250,000, 8.2% from $250,001– 500,000, 5.1% from $500,001–$1,000,000, and 3.1% over $1,000,000. research hypothesis vol. 31 no.1 fall/winter 2019 figure 11 is a cross tabulation of journal budgets and hospital ftes. for ebook budgets, 22.8% spent up to $500; 19.0%, $501–5,000; $18.4%, $5,001–20,000; 12.0%, $20,001–50,000; 16.5%, $50,001–100,000; and 11.4% were over $100,000. the majority (26.1%) of respondents spent $50,001–100,000 on database budgets (which excluded point of care tools) while 21.5% spent up to $10,000, 16.4% spent $10,001–25,000, 15.0% spent $25,001–50,000, and 20.9% spent more than $100,000. almost a third of respondents (32.5%) had a print book budget of up to $1,000, 30.7% had a budget of $1,001–5,000, 20.6% had a budget of $5,001–$10,000, and 16.3% had a budget of over $10,000. most libraries (66.1%) belonged to a consortium and, of those, three-quarters (74.8%) allocate up to 25% of their budget to consortium purchased products while 14.4% allocate 26–40%, 8.1% allocate 41–60%, and 2.7% allocate more than 60%. over a third of respondents (35.6%) did not have a point of care tool or did not pay for one. most libraries (66.3%) do not receive funding outside of their organization. one quarter (26.3%) received donations, 18.1% generated funds from ill, 16.9% received grants, 4.4% generated funds from non-systematic review mediated searches, and 0.6% generated funds from a systematic review service. research hypothesis vol. 31 no.1 fall/winter 2019 discussion we were unable to calculate the survey sample size, and therefore response rate, as there is no source documenting the current number of hospital libraries in existence. we found we could not compare our response rates or survey results to dudden [16] or thibodeau [4] as we asked participants to coordinate one survey response for each organization (as compared to previous studies in which any library staff representative could participate). requesting one respondent per organization was done in an attempt to minimize duplicate data. the inability to compare our data to previous studies is indeed problematic; this speaks to the difficulty in collecting data representative of the current landscape while minimizing the possibility of duplicative responses. related to organizational responses, we found that some hospital libraries, despite being a member of a health care system, do not work closely with others from the same system and don’t have information (e.g., library characteristics, budget, resources, search statistics) about the other locations. respondents commented that health care systems have libraries with partially or entirely separate budgets and function autonomously. it is unclear if some responses reflect one or multiple hospitals. the landscape of hospital librarianship is changing so it was difficult to hypothesize what the survey data would reveal. we presumed there would be a steady increase in budgets as the size of the organization increased but this wasn’t indicated (see figure 6). we also speculated there would be a correlation between the number of librarians and the number of hospitals supported. results show that there are many solo librarians supporting single sites as well as multiple sites, in some cases a single librarian is supporting 10 or more sites (see figure 3). this data reflects the results of the solo census study by mclaughlin et al [23]. the mla hospital libraries section standards committee published standards for hospital libraries in 2008 which included staffing [24]. in 2009, van moorsel studied a sample of hospitals and found that the majority did not comply with recommended staffing levels in their libraries [20]. those standards have not been updated in over a decade and there are no current recommendations to provide guidance on the ratio of number of librarians to staff or hospitals supported. it is of interest that very few libraries had a strategic plan (39.3%) and even fewer had a communications or marketing plan (21.2%). a surprisingly low percentage of libraries (59.4%) had a website and, of those who do, just 50.0% of library can manage it themselves. most (91.7%) respondents had an internal library website or intranet presence but just two-thirds (66.2%) of the staff could manage it themselves. the data on marketing plans and web presence speaks to the barriers that hospital libraries face in promoting their services or interacting with populations served. although it may be difficult to persuade hospital information services of the need for a web presence, the need to create strategic communications and marketing plans presents an opportunity research hypothesis vol. 31 no.1 fall/winter 2019 for educational resources and courses to be created to assist hospital librarians in understanding the benefit of and creating these plans. it was surprising to discover that 35.6% of hospital librarians do not have or do not pay for a point of care tool, such as uptodate or dynamedplus. it may be that some librarians successfully eliminated costly point of care tools from their budgets or that another department pays for access to the tool. the authors speculate whether the respondents who pay for a point of care tool also pay for the majority of knowledgebased resources at their organization, even resources unique to one department or those that may not be typically thought of as library resources. limitations we identified limitations that resulted from the design of the questions. “document delivery” and “interlibrary loan” were used to distinguish between articles supplied to users from one’s collection as compared to resources requested from other libraries’ collections, respectively. respondents’ comments indicated confusion over the use of these terms. in the past, when collection formats were mostly print, there were distinct budget lines for journals and books. for the resource budget questions, it was difficult for respondents to provide accurate estimations on the total spend for resource types as some have a combination (e.g., journals, books, databases) within a single resource. consequently, it is difficult to draw conclusions as to how much is spent on each type of resource. related to resources, the question about point of care tools was worded so that one selection included both “does not have a point of care tool” and “does not pay for a point of care tool” whereas they should have been separate options. categorizing the open-ended responses for the question about library staff participation on committees was not possible as each organization has unique committee names. it was difficult to understand the scope or charge of the committees without having additional information. future efforts the largest takeaway from this study is that there is a need for a concerted effort to understand baseline hospital library data in an era of increasing health care mergers and overall changes. only once this data is collected can we begin to understand the current landscape and how to advocate for hospital libraries. capturing data about these libraries will be challenging, as was demonstrated in this survey, and future iterations of the survey will need to address a way to collect data about health care system libraries, both those that work closely with one another and those that function independently. in order to do this, respondents may need to indicate which institutions they represent which is something not done in this survey in order to retain anonymity of respondents due to the sensitive nature of the data (especially regarding budgets). research hypothesis vol. 31 no.1 fall/winter 2019 current efforts at data collection include mla’s hospital libraries section census task force, which is charged with determining the number of hospital libraries presently in existence and the number that have closed since 2017. once that information is collected, we suggest sending a benchmarking survey on a determined schedule (even creating a large-scale longitudinal study to set baseline data and track changes), making sure there is consistency in the questions and response choices. this will ensure we have comparable data, which should be readily available for use with administrators and for hospital library planning and advocacy. it would be helpful to understand other changes in the hospital library profession such as staffing number changes (e.g., reduction of professional staff, mergers, retirements with or without a rehire), and changes in service responsibilities (to help determine if librarians are doing more with less). it would also be of interest to explore whether closures have impacted services in other hospital libraries or even academic libraries. with limited budgets and varying access to resources, it’s possible some health systems are experiencing an environment of information inequity, or a gap in access to library resources [25]. studies are needed that explore the shortand longterm effects of closures and mergers and their impact not only on library resources, services and budgets but also on access to information for health care providers as well as any impact on patient care outcomes. to provide the most benefit, access to future survey data for institutional comparison, such as in an online platform in which hospital library staff can input their information to see how their organization compares, would be of great benefit. this paper presents selected results from our survey. data associated with this article are available in the open science framework at https://doi.org/10.17605/osf.io/5va74. note that identifying information, such as location within each country, has been removed. references 1. kaufman hall & associates llc. 2017 in review: the year m&a shook the healthcare landscape. chicago, il; 2018. 2. martin hj, delawska-elliott b. combining resources, combining forces: regionalizing hospital library services in a large statewide health system. j med libr assoc. 2015;103(1):44-8. 3. gibson ds, hernandez m, draemel a. destination library: validating the importance of physical space. j hosp libr. 2015;15(3):251-61. 4. thibodeau pl, funk cj. trends in hospital librarianship and hospital library services: 1989 to 2006. j med libr assoc. 2009;97(4):273-9. 5. jones b, honour j. looking at mcr library closures: whats happening and why? plains to peaks post: a publication of the national network of libraries of medicine, midcontinental region. 2015;13(4):9-11. 6. harrow a, marks la, schneider d, lyubechansky a, aaronson e, kysh l, et al. hospital library closures and consolidations: a case series. j med libr assoc. 2019;107(2):129-36. research hypothesis vol. 31 no.1 fall/winter 2019 7. giesler rh, yast ht. a survey of current hospital library resources. hospitals. 1964;38:55-7. 8. schick fl, crawford s. directory of health sciences libraries in the united states, 1969. chicago: american medical association; 1970. 9. crawford s, dandurant g. directory of health sciences libraries in the united states, 1973. chicago: american medical association; 1974. 10. rees amcs. directory of health sciences libraries in the united states, 1979. cleveland: case western reserve university; 1980. 11. crawford s. health science libraries in the united states: iii. hospital health science libraries, 1969-1979. bull med libr assoc. 1983;71(1):30-6. 12. wos m, oddan l. effect of institutional mergers on hospital libraries. bull med libr assoc. 1987;75(1):34-5. 13. stevens sr. impact of changing health care economics on michigan hospital libraries: report of a survey. bull med libr assoc. 1990;78(2):140-5. 14. wakeley pj, foster ec. a survey of health sciences libraries in hospitals: implications for the 1990s. bull med libr assoc. 1993;81(2):123-8. 15. glitz b, lovas i, flack v. the changing status of hospital libraries 1984 to 1989: characteristics and services in region 7 of the national network of libraries of medicine. bull med libr assoc. 1992;80(2):179-84. 16. dudden rf, corcoran k, kaplan j, magouirk j, rand dc, smith bt. the medical library association benchmarking network: development and implementation. j med libr assoc. 2006;94(2):107-17. 17. bertolucci yr, van houten l, balogh l. riding the wave from benchmarking to bucks. in: fulda po, satterthwaite rk, editors. proceedings, 103rd annual meeting medical library association, inc; may 4, 2003; san diego, ca: j med libr assoc; 2003. p. 117-54. 18. fama j. “but we made the ‘easy’ cuts last year!” staffing the library. in: fulda po, satterthwaite rk, editors. proceedings, 104th annual meeting medical library association, inc; may 25, 2004; washington, dc: j med libr assoc; 2005. p. 143-88. 19. gluck jc, hassig ra. raising the bar: the importance of hospital library standards in the continuing medical education accreditation process. bull med libr assoc. 2001;89(3):272-6. 20. van moorsel g. analysis of compliance of hospital libraries with the medical library association staffing standard: examination of the current state of the industry and reconsideration of the standard. j hosp libr. 2009;9(3):273-85. 21. ducas a, demczuk l, macdonald k. results of a survey to benchmark canadian health facility libraries. j can health libr assoc. 2015;36(1):3-10. 22. glaser bg, strauss al, strutzel e. the discovery of grounded theory: strategies for qualitative research. new york: aldine de gruyter; 1967. 23. mclaughlin l, spencer a, zeblisky k, liszczynskyj h, laera e. solo census: demographics, duties, needs and challenges. j hosp libr. 2018;18(2):127-35. 24. hospital libraries section standards committee, bandy m, doyle jd, fladger a, frumento ks, girouard l, et al. standards for hospital libraries 2007: hospital libraries section standards committee. j med libr assoc. 2008;96(2):162-9. 25. kraft m. the donut hole of library access. 2018 jun 18. in: krafty librarian [internet]: michelle kraft. [cited 2019 jun 20]. available from: http://www.kraftylibrarian.com/the-donuthole-of-library-access/ angela spencer, mals assistant professor, health sciences reference librarian saint louis university invited commentary hypothesis vol. 33 no.1 fall/winter 2021 message of solidarity to research caucus members* alexander j. carroll, msls, ahip librarian for steam research jean and alexander heard libraries vanderbilt university nashville, tennessee https://orcid.org/0000-0003-0248-3811 alex.carroll@vanderbilt.edu dear rc members, for members of the research caucus who are members of the asian, asian-american, or pacific islander communities, i join the chorus of voices expressing solidarity with you this week. the leadership of the research caucus affirms and echoes the asian pacific american librarians association (apala)’s recent statement, which condemns the rise in anti-asian hate crimes across the united states over the past year. to those of us wondering what they can do to get involved and show up for our asian, asian-american, and pacific islander community members, i would encourage you to learn how you can speak out and step in when you see anti-asian bias, harassment, and violence happening in your local communities. if you’re looking for places to start, i would recommend hollaback!’s training resources on bystander intervention, available here: https://www.ihollaback.org/bystander-resources/. if you want to learn more, they are also offering several synchronous online training programs over the next several weeks, which you can register to attend for free here: https://www.ihollaback.org/harassmenttraining/. beyond bystander training, it is critically important that as medical librarians we continue to interrogate how our professional practices may reinforce systemic racism in ways that can contribute to acts of racialized violence. when drafting a statement of solidarity to our black, indigenous, and people of color (bipoc) colleagues this past summer, i encouraged members of the research caucus to use research to ask questions about systemic inequities within medical libraries. i renew that charge to all of us today: how do our systems and services reflect and reinforce systemic inequality, and what can we do to reform those systems to create more equitable libraries, institutions, and communities? these questions are challenging; asking them will require courage and will lead to discomfort. but if we want to live up to the spirit of “i am mla,” which mla past president beverly murphy describes as “the collective understanding...that it is up to us, as members and as volunteers, to do what needs to be done for our association,” we must rise to meet that challenge. to our asian, asian-american, and pacific islander colleagues: we see you, and we hear you. the research caucus is committed to fostering a safe and equitable community for all our members, and if we can support you in this time, please know that we are here for you. https://orcid.org/0000-0003-0248-3811 mailto:alex.carroll@vanderbilt.edu https://www.apalaweb.org/apala-statement-against-anti-asian-violence/ https://www.apalaweb.org/apala-statement-against-anti-asian-violence/ https://www.ihollaback.org/bystander-resources/ https://www.ihollaback.org/harassmenttraining/ invited commentary hypothesis vol. 33 no.1 fall/winter 2021 sincerely, alexander j. carroll, msls, ahip mla research caucus chair 2020-2021 *solidarity statement initially disseminated to mla research caucus members on march 19, 2021. appendix author questionnaire you are receiving this survey because you presented either a poster or paper at an annual meeting of the medical library association (mla) during 2012 or 2014. this survey is part of mla’s research training institute and will determine the dissemination of research and projects presented at the annual conferences. this survey will take ten minutes or less. your answers will be confidential. survey responses will be stripped of personally identifiable information. please consider the list of your abstracts in the email sent to you when responding. thank you for your participation! if you need to reference the meeting programs, you can do so at mla's website. this survey was reviewed by the university institutional review board and was declared to be exempt. 1. did any of your mla posters or presentations from 2012 and 2014 result in a published article or have been submitted publication? o yes o no skip to q42: if did any of your mla posters or presentations from 2012 and 2014 result in a published article or... = no display this question :if did any of your mla posters or presentations from 2012 and 2014 result in a published article or... = yes 2. which abstract(s) has been published or submitted for publication? note: there may be blank fields in this question. ▢ ${e://field/abstract1} ▢ ${e://field/abstract2} ▢ ${e://field/abstract3} ▢ ${e://field/abstract4} display this question: if which abstract(s) has been published or submitted for publication? note: there may be blank field... = ${e://field/abstract1} 3. where did you publish this abstract (${e://field/abstract1})? if you published in more than one source, consider the most recent publication. o peer-reviewed journal o non-peer-reviewed journal o other ________________________________________________ o i have submitted this abstract for publication. display this question: if where did you publish this abstract (${e://field/abstract1})? if you published in more than one s... = peer-reviewed journal or where did you publish this abstract (${e://field/abstract1})? if you published in more than one s... = non-peer reviewed journal or where did you publish this abstract (${e://field/abstract1})? if you published in more than one s... = other 4. please provide the full name of the journal or other source that published your paper. ________________________________________________________________ 5. what year was your paper published? ________________________________________________________________ display this question: if where did you publish this abstract (${e://field/abstract1})? if you published in more than one s... = i have submitted this abstract for publication. 6. at what stage is your paper submission? o paper was accepted, publication pending. please write in which journal accepted your paper: ________________________________________________ o still awaiting a response from the editor. o paper was rejected. display this question: if which abstract(s) has been published or submitted for publication? note: there may be blank field... = ${e://field/abstract1} 7. select the primary reason you chose to pursue publication for this abstract: ${e://field/abstract1} o to receive merit increments, tenure, and/or promotion due to my research activities o share results with a broader audience o contribute to my profession's evidence-base o my research or project was novel o i am (formally or informally) expected to participate in research as part of my job. o my coauthors and/or colleagues encouraged me to publish o to build a professional reputation for myself o to demonstrate the impact of my library or the profession o other, please explain: ________________________________________________ 8. what is the secondary reason you chose to pursue publication? o to receive merit increments, tenure, and/or promotion due to my research activities o share results with a broader audience o contribute to my profession's evidence-base o my research or project was novel o i am (formally or informally) expected to participate in research as part of my job. o my coauthors and/or colleagues encouraged me to publish o to build a professional reputation for myself o to demonstrate the impact of my library or the profession o no other reason o other, please explain: ________________________________________________ note: these questions are then repeated for each abstract the participant said was published in question 2. q42 what were your credentials at the time of the conference you presented at (2012 or 2014)? if you presented at both conferences, answer for 2014. check all that apply. o mls or equivalent o non-mls master's degree o phd o professional degree: md, rd, rn, pa, pharmd, do, etc. o other, please explain: ________________________________________________ o none of the above hypothesis , vol. 34, no. 1, 2022 completing the transition from newsletter to journal margaret a. hoogland, mls, ahipa aclinical medical librarian and associate professor, university libraries, the university of toledo, toledo, ohio, https://orcid.org/0000-0002-9932-3605 , margaret.hoogland@utoledo.edu, twitter background from 1987-2002, hypothesis served as the newsletter for the medical library association (mla) research section. news of the section began to appear toward the end of each issue in the new millennium. articles became more prominently displayed in the front of each issue with volume 16, number 1, 2002. in 2005, the word ‘journal’ was added to hypothesis for the first time and the title changed to, “hypothesis : journal of the mla research section.” the newsletter included top research questions, members of the section, and short but noteworthy pieces by up-and-coming leaders. in 2012, editor diane cooper1 and the research section leadership began transitioning hypothesis into a journal. peer reviewed articles were clearly identified to distinguish these submissions from those articles reviewed by one of the editors. in the hypothesis archives, volume 27, number 1, was the first issue to include digital object identifiers (dois) for individual articles2. despite being indexed in cinahl, lissa, and lista; hypothesis accepted submissions via email until 2018, when then co-editors erin d. foster (foster) and carol l. perryman (perryman) worked with mla headquarters to create an online submission system hosted on the mla website. in 2020, foster and perryman transitioned hypothesis to its current website, which is hosted by the indiana university purdue university open journal system (ojs) platform. shortly after publishing the first issue in summer 2020, they invited community manager of the research caucus, margaret a. hoogland (hoogland), to serve as co-editor. transitioning to the open journal system website while a basic structure existed in ojs, it became clear that hoogland’s curiosity and enjoyment of technology could be used to streamline some of the existing processes and to implement new ojs features. below is a table describing some of the major changes we implemented during june 2020 february 2022: 1 https://orcid.org/0000-0002-9932-3605 https://www.orcid.org/ https://orcid.org/ mailto:margaret.hoogland@utoledo.edu https://twitter.com/hoogland_ma https://journals.iupui.edu/index.php/hypothesis/issue/view/1601 https://journals.iupui.edu/index.php/hypothesis/issue/archive hypothesis , vol. 34, no. 1, 2022 table 1: timeline of changes made to ojs platform to improve editorial team processes. time frame accomplishment august october 2020 created email templates to improve communication among members for the editorial team, peer reviewers, and authors. november 2020 published second issue in ojs. november 2020-february 2021 added back issues to ojs website. february 2021-february 2022 created peer review forms that could be added to author notification emails. created an area on the website describing the available submission categories. editors revisited descriptions and divided submission categories by review type. created and implemented use of online peer review forms for all categories. created and linked pdf copies of peer reviewer forms to facilitate taking notes. after publishing the second and entirely ojs based issue of hypothesis in november 2021, foster stepped down as co-editor. in january 2022, hypothesis published its short-term and long-term goals on the website and started soliciting regular input from the research caucus leaders. in late february 2022 after 5 years of hard work and many contributions such as crafting the voices of experience submission category and providing valuable feedback on our peer review forms, perryman stepped down as co-editor and plans to devote more time to conducting and participating on research projects and to working with the texas women’s university library and information sciences student journal. planning for the future: march 2022-present starting in march 2022, the editorial team reassessed the existing structure, discussed next steps, and recruited new section editors (research – jonathan eldredge, failure – 2 https://journals.iupui.edu/index.php/hypothesis/peer_reviewed hypothesis , vol. 34, no. 1, 2022 heather holmes, and methods moment – nina exner). from 2008-2013, most issues of hypothesis included a column, called “the research mentor,” authored by jonathan eldredge. after an 8-year hiatus, “the research mentor” is back and we thank jonathan eldredge for agreeing to serve as column editor. change and transitions frequently result in turnover and so in mid-march 2022, heather healy decided to step down as lead copy editor. we thank heather for her input, assistance, hard work and for working with and transitioning duties to andrea ball and our newest copy editor, christine rigda, through the inaugural august issue. to assist in getting hypothesis ready for pubmed central inclusion in the upcoming years, we created a latex template3 for each submission category. hoogland will be training and then transitioning duties to our new production editor, alexandria quesenberry wilson, who previously served on the hypothesis marketing team. in july, hypothesis added a new subtitle, “research journal for health information professionals” and new logo to our website, twitter, and facebook accounts. we encourage you to follow and like our posts on social media. in 2022-2023, expect to see and to hear more about hypothesis on social media, in your caucuses, on your listservs, and at your state, national, regional, and local conferences. we intend to provide candid and encouraging yet thoughtful feedback for all submitters. we invite authors to submit their manuscripts regardless of how much or little experience they have with writing, research, and publishing. recognizing that not everyone has employer support for participating in research or for completing scholarly activities, hypothesis provides opportunities to interested caucus members to serve as peer reviewers or as members of the marketing team. volunteering in any capacity improves your understanding of journal requirements, research, and knowledge of the publishing process. if you would like to serve as a peer reviewer, which involves reviewing a minimum of one article per issue, or as a member of the marketing team, consider emailing hypothesisj@protonmail.com. we hope you enjoy our new look and first issue as hypothesis : research journal for health information professionals. acknowledgements i thank ted polley for his patience with my questions and curiosity with ojs related questions and issues. i appreciate jon eldredge’s support and wisdom – i know it will continue. i would also like to thank my colleagues in the university libraries at the university of toledo, who listened patiently and provided feedback, as we implemented these changes. references 1. cooper id, eldredge j, woodson d. our past and our future. hypothesis [internet]. 2012 [cited 11 may 2022];23(2). available from: https://journals.iupui.edu/index.php/hypothesis/issue/view/1622/608 3 https://twitter.com/mlahypothesis?s=20 https://www.facebook.com/groups/87918304357 mailto:hypothesisj@protonmail.com https://journals.iupui.edu/index.php/hypothesis/issue/view/1622/608 hypothesis , vol. 34, no. 1, 2022 2. hypothesis: research journal of health information professionals. 2015;27(1). accessed 11 may 2022. available from: https://journals.iupui.edu/index.php/hypothesis/issue/archive 3. the latex project [internet]. [cited 11 may 2022]. available from: https://www.latex-project.org/. 4 https://journals.iupui.edu/index.php/hypothesis/issue/archive https://www.latex-project.org/ peer reviewed article hypothesis , vol. 35, no. 2, 2023 methods for evaluating database coverage stacy brodya, rachel brillb, robyn butcherc areference and instructional librarian, himmelfarb library, the george washington university, washington, district of columbia, https://orcid.org/0000-0002-2817-3849 , sbrody98@gwu.edu breference and instructional librarian, himmelfarb library, the george washington university, washington, district of columbia, https://www.orcid.org/0009-0005-8517-5401 , rgbrill@gwu.edu cresearch information specialist, canadian agency for drugs and technologies in health, ottawa, canada,https://www.orcid.org/0000-0003-2385-5629 , robynb@cadth.ca purposes: in this article, we describe two methods to evaluate database content coverage, illustrating their use with the example of evaluating databases for coverage of medical education literature. method names: though several methods for evaluating database content coverage have been described, the two used herein, referred to as the reference list search method and the journal method, use traditional librarian skills and tools and can be used across a variety of topic areas. description: in this paper, we describe two approaches to evaluating database coverage. for each method, we use the example of evaluating databases for their coverage of medical education literature. in the reference list search method, we search for references from comprehensive reviews or bibliographies to determine their presence in databases. in the journal method, we describe the indexing status and coverage of key publications in databases. strengths: these methods use established librarian skills and tools and can be applied to a wide variety of topics. limitations: the results of each method depend on the quality and comprehensiveness of the original source material: comprehensive reviews and bibliographies for the reference list search method and publication lists for the journal method. information professionals who conduct clinical evidence syntheses often refer to cochrane for general guidance when selecting databases for searches.1 campbell and other organizations provide guidance for selecting databases for other topic areas.2,3 more detailed evidence regarding database coverage of specific topics could assist searchers more effectively than the broad guidance currently available. cochrane prescribes core databases to search for reviews of medical interventions. in other areas of research, lists of potentially relevant databases are provided, but decisions of which and how many databases to search are left to the searcher and review team. with little evidence available on the coverage of recommended databases and their utility in reviews, 1 https://www.orcid.org/0000-0002-2817-3849 https://orcid.org/ mailto:sbrody98@gwu.edu https://www.orcid.org/0009-0005-8517-5401 https://orcid.org/ mailto:rgbrill@gwu.edu https://www.orcid.org/0000-0003-2385-5629 https://orcid.org/ mailto:robynb@cadth.ca peer reviewed article hypothesis , vol. 35, no. 2, 2023 decisions may feel subjective. selecting databases is only one part of the larger framework of decision-making processes in designing a search strategy.4 current, relevant evidence to support that decision could facilitate more efficient, effective searching. evaluating databases for comprehensiveness of domain-specific content can provide one piece of evidence to help guide those decisions. several methods for evaluating database completeness have been described in the literature.5,6 we selected two methods to review based on our prior experience evaluating database coverage.7 they are the reference list search method, in which searches are conducted using a gold standard set of references on a topic, and the journal method, which relies upon the indexing status of relevant journals. these two methods rely on traditional library skills and tools and can be implemented across a variety of subject areas. please note that this paper only addresses methods for assessing database coverage, not for evaluating database functionalities.8,9 to illustrate how each method is used to evaluate database coverage, we used the topic of medical education. reference list search overview the reference list search method consists of five steps: selecting evidence sources, extracting references, searching databases, calculating proportions retrieved, and comparing proportions across databases (figure 1). in step one, bibliographies, systematic reviews, or other evidence sources are selected for relevance to the research topic, comprehensiveness, and quality. in step two, references are extracted from these sources. these references represent the “reference standard”. next, known item searches, using title, author and keyword, or other combinations, are conducted to determine the presence of these references in the databases of interest. in step four, the proportion of relevant references retrieved is calculated for each database by dividing the number of retrieved references by the total number of references in the reference standard. in step five, the proportions are compared across databases to identify those with greatest coverage. figure 1: the five steps of the reference list search method for database evaluation. when using the reference list approach, criteria should be applied to the initial selection of systematic reviews or bibliographies. for instance, butcher et al. describe criteria for selecting systematic reviews to capture references on covid-19 interventions, diagnosis, and prediction.7 to be eligible, a systematic review had to a) “[report] searching at least four databases” and b) include a “minimum of five . . . studies . . . published or posted in 2020” or, if including fewer than five studies, “all must have been published or posted in 2020”.7 more generally, systematic reviews should provide at least one full search strategy and reference appropriate conduct and reporting guidelines. this helps readers evaluate how comprehensive and representative the result sets of these reviews may be. as more recent systematic reviews report following prisma-s, selection criteria can be updated.10 2 peer reviewed article hypothesis , vol. 35, no. 2, 2023 when using this approach, it is important to consider the inclusion/exclusion criteria described in evidence sources. for example, if a reference standard is created from systematic reviews that exclude non-english language articles, conference abstracts, and qualitative research, the database coverage evaluation will reflect those parameters. the composition of the reference list dictates the utility of results. alternative approaches have been used to create the reference standard. for example, collected works from specific evidence synthesis groups may provide a ready source for developing a reference set. whiting et al. drew from systematic reviews with searches conducted by the centre for reviews and dissemination at the university of york.11 alternatively, royle, bain, and waugh searched for relevant articles and consulted an expert on their usefulness.12 with the subject matter expert, they were able to include gray literature, research in progress, meeting abstracts, and conference proceedings. however, this approach requires considerable time and subject matter expertise. the results of the reference list search method can include raw numbers and proportions of references retrieved by known item searches, per database. results can also be presented as references retrieved beyond those found in a core database. for instance, butcher et al. calculated “the number of records found in [a] database and not found in pubmed”.7 similarly, hartling et al. “recorded how many of the studies not indexed in medline were indexed in each of the additional databases” and further evaluated whether articles uniquely found in databases other than medline significantly impacted the results of systematic reviews.13 for different topic areas, researchers may use databases other than medline as the standard of comparison. our experience with the reference list search we applied the reference list search method to evaluate databases for their comprehensiveness for medical education searches. to create the reference set, we used beme (best evidence in medical education) reviews. the beme collaboration is a group under the auspices of amee, an international association for health professions education. the beme collaboration has provided guidance for conducting reviews in medical education and has published reviews on their website. (note: during the conduct of our research, their site moved behind a paywall.14,15 [published reviews (archive.org)].) figure 2: data validation and automatic formatting in excel facilitated screening of reviews first, we selected relevant, comprehensive evidence sources. following butcher et al., we screened reviews using the following inclusion criteria: systematic type review, medical education focus, minimum three databases searched, and clearly reported search strategy.7 we conducted screening in excel, using validation rules to guide data entry (figure 2). next, we recorded citation information; the numbers of databases searched and studies included; 3 https://amee.org/amee/publications/publications/beme_guides/amee/publications/bemelanding.aspx?category=beme_guide&hkey=315583c4-5d6a-42c9-8f66-b647bf1ef75d https://amee.org/amee/publications/publications/beme_guides/amee/publications/bemelanding.aspx?category=beme_guide&hkey=315583c4-5d6a-42c9-8f66-b647bf1ef75d http://web.archive.org/web/20221130150407/https:/bemecollaboration.org/published+reviews/ peer reviewed article hypothesis , vol. 35, no. 2, 2023 and whether (yes-no) the review was systematic, on topic, and presented clear search strategies. when the beme reviews site moved behind a paywall, the supplementary materials, search strategies and lists of included studies were not consistently posted on publisher sites accompanying the articles. therefore, we had to use the internet archive’s wayback machine to locate supplementary materials originally found on the beme reviews site. as screening continued, we started saving supplementary materials reporting search strategies and, when available, lists of included references to ensure we had consistent access to data. following screening, we planned to extract included references to a citation manager; deduplicate; and search for references by doi, author and title fragments, or combinations thereof. data would be analyzed for proportion found per database and references retrieved beyond those found in pubmed (“additional yield”).11 at the time of this writing, we have paused screening beme reviews. we may return to screening, looking for supplementary materials on the wayback machine, as necessary. alternatively, we may curate a list of systematic reviews published in journals in the annotated bibliography of journals for educational scholarship, created by the aamc-regional groups on educational affairs. we encountered this resource during a webinar for authors seeking guidance on publishing medical education research (scholarly publishing webinar series aamc). while beme has an explicit focus on systematic reviews for medical education, other journals may include similar publications. limitations of the reference list search it may be important to consult subject matter experts to review the list of selected sources, as well as information professionals with evidence synthesis experience to assist in developing selection criteria and screening evidence sources. time may be a limiting factor. for example, extracting and searching for items in the reference standard can take much time, depending on the size of the set. newer librarians and library students could be involved in this task to distribute the workload. when determining whether to use this method, it is important to consider the availability of high-quality, comprehensive, relevant evidence sources. the reference standard, its size and composition, is at the core of this method. criteria used in the evidence sources will be reflected in the reference set. for instance, if reviews only included randomized controlled trials, articles published in the last ten years, and articles written in the english language, these criteria are reflected in the composition of the reference standard. the potential influence of these criteria should be described in reporting the results of this method. journal method overview the journal method for evaluating databases for completeness begins with developing a list of journals pertinent to the topic, reviewing database documentation to determine whether those journals are indexed in each database, and calculating and comparing proportions of relevant journals indexed across databases (figure 3). 4 https://www.aamc.org/media/38166/download?attachment https://www.aamc.org/about-us/mission-areas/medical-education/scholarly-publishing-webinar-series https://www.aamc.org/about-us/mission-areas/medical-education/scholarly-publishing-webinar-series peer reviewed article hypothesis , vol. 35, no. 2, 2023 figure 3: the four steps of the journal method for database evaluation. the success of the journal method depends on the list of journals examined in much the same way that the success of the reference list method depends on the selection of a strong reference standard. various tools can be used to select journals, including subscription-based resources like web of science and ulrich’s, and freely available tools such as the scopus source list and citescore.7,12,16 in addition to subject relevance, impact factors may be considered in building a journal list. some disciplines may have bibliographies or core title lists, such as the annotated bibliography of journals for educational scholarship, that could be used in this type of work. our experience with the journal method for comparison purposes, we used the same topic of medical education to illustrate how to use the journal method. to start, we generated a list of high-impact journals in the category “education, scientific disciplines” from journal citation reports (jcr). we also identified journals commonly encountered when responding to medical education-related reference questions. we found that not all journals with articles relevant to health sciences education are in jcr; of those that are, not all are categorized under “education, scientific disciplines”. some are, for instance, categorized under a specialty, e.g., pediatrics. in our sample, only 17 of 23 (74%) relevant journals in jcr are categorized under “education”. this issue demonstrates that the categorization schema of tools involved in journal selection must be considered when conducting this work. we then collected data in excel, using data validation tools and a readme tab to guide data entry (figure 4). for each journal, we collected the year the publication started, category in jcr, journal impact factor (jif), peer review information, and indexing status in the databases of interest. figure 4: excel data validation tools, for instance, input messages, as pictured above, helped improve consistency in data entry. we analyzed 26 journals across eight databases and found the highest number of journals were cited in pubmed and indexed in scopus, 26 and 24, respectively. furthermore, 21 5 https://www.scopus.com/sources https://www.scopus.com/sources https://www.aamc.org/media/38166/download?attachment peer reviewed article hypothesis , vol. 35, no. 2, 2023 journals were indexed in medline and ten were included in pubmed central. we noted lower representation of health sciences education journals in eric (4) and education source (8). no journals were uniquely indexed in eric and education source. conversely, two journals were found only in pubmed and pubmed central, reinforcing the importance of searching pubmed at the very least and the use of pubmed as a relative standard. we concluded that, while all databases may be included for thorough education-related searches, omitting eric and education source when time is limited is not likely to impact the comprehensiveness of a medical education literature search. after completing the journal analysis, we encountered additional sources for journal lists. the annotated bibliography of journals for educational scholarship, created by the aamc-regional groups on educational affairs, may have been a more comprehensive list for this project. any future iteration of this project could use this list. limitations of the journal method in our experience with the journal method, the categorization schema of tools used to develop the journal list is important to consider and report. for the topic of medical education, relying on jcr categorizations alone would have led us to miss journals that we, in our professional experience, understand to be relevant to this topic. furthermore, when collecting data on indexing status, it is important to note selective indexing, as not all databases index all journals “cover to cover”. for instance, a database may selectively index certain topics or publication types (e.g., eric).17 overall, this method was less time-consuming than the reference list method, provided evidence to support decision-making, and relied on traditional librarian skills and tools. comparing and contrasting the reference list search and journal methods the journal method is appropriate when time is limited and publishing on the topic is highly concentrated in a few key journals. when the topic is more complex and articles might be published in a diverse range of journals, we recommend using the reference list method. lessons learned & practical tips the two methods described rely on established librarian skills, e.g., searching, and tools, e.g. excel and citation managers. overall, we found the journal method faster and easier. the data could also be used to assist patrons in selecting journals for publication. developing the initial journal list was perhaps the most challenging piece. when preparing for database evaluations, plan for data collection and reuse. conducting sample searches can be helpful in anticipating which pieces of data to collect. data validation tips and in-sheet data entry instructions help ensure consistency in data collection and save data collectors time in having to switch between the entry document and an instruction sheet (figures 2 and 4). finally, consider the potential for additional work. manage data for reuse and provide sufficient documentation to enable reuse by your team or others, for instance in bibliometrics projects. conclusion these methods can address practical challenges in selecting databases and provide evidence to support those decisions in evidence synthesis teams. we hope these methods empower librarians to conduct research using traditional librarian skills and resources. 6 https://www.aamc.org/media/38166/download?attachment https://www.aamc.org/media/38166/download?attachment peer reviewed article hypothesis , vol. 35, no. 2, 2023 though we have framed this article in terms of evidence synthesis searching, we invite our colleagues to comment on its applicability for purchasing and subscriptions decisions, identification of resources for research guides, and other practical uses. additional resources for database evaluation research • search smart: https://www.searchsmart.org/ – “a free tool to give you the confidence to search the best databases available” – filter 88 databases by coverage (topic, record type), access, search functions – see: “a free online guide to researchers’ best search options”18 • online databases: advanced search techniques and strategies for graduate students in music and music education: https://researchguides.case.edu/c.php?g=17746&p=99778 – libguide by case western reserve university, kelvin smith library – describes 6 quick ways to evaluate database content and 10 items for evaluating interfaces • gusenbauer m. search where you will find most: comparing the disciplinary coverage of 56 bibliographic databases. scientometrics. 2022;127(5):2683–745.19 – gusenbauer uses a keyword approach and the 26 categories in the scopus all science journal classification systems to compare subject coverage across databases • bibliographies and other evidence sources for reference set development – annotated bibliography of journals for educational scholarship • library carpentries tidy data for librarians: basic quality assurance and control – lesson on data validation, conditional formatting, and other excel tools for quality control in data entry credit stacy brody: conceptualization, formal analysis, data curation, project administration, writing original draft rachel brill: formal analysis, data curation, writing review & editing robyn butcher: writing review & editing references 1. higgins j, lasserson t, chandler j, tovey d, thomas j, flemyng e, et al. methodological expectations of cochrane intervention reviews (mecir) [internet]. london: cochrane; 2022 feb [cited 2020 dec 11]. report no.: version february 2022. available from: https://community.cochrane.org/mecir-manual 2. kugley s, wade a, thomas j, mahood q, jørgensen ak, hammerstrøm k, et al. searching for studies: a guide to information retrieval for campbell systematic reviews. campbell syst rev. 2017 jan;13(1):1–73. 7 https://www.searchsmart.org/ https://researchguides.case.edu/c.php?g=17746&p=99778 https://www.aamc.org/media/38166/download?attachment https://librarycarpentry.org/lc-spreadsheets/04-quality-control.html https://librarycarpentry.org/lc-spreadsheets/04-quality-control.html https://community.cochrane.org/mecir-manual peer reviewed article hypothesis , vol. 35, no. 2, 2023 3. aromataris e, munn z. jbi manual for evidence synthesis. adelaide, australia: joanna briggs institute; 2020. 4. clark jm, beller e, glasziou p, sanders s. the decisions and processes involved in a systematic search strategy: a hierarchical framework. j med libr assoc [internet]. 2021 jul 20 [cited 2023 jul 12];109(2). available from: http://jmla.pitt.edu/ojs/jmla/article/view/1086 5. ritchie sm, young lm, sigman j. comparison of selected bibliographic database subject overlap for agricultural information. issues in science and technology librarianship [internet]. 2018 jun 15 [cited 2022 dec 6];(89). available from: https://journals.library.ualberta.ca/istl/index.php/istl/article/view/1727 6. tenopir c. evaluation of database coverage: a comparison of two methodologies. online rev. 1982 jan 1;6(5):423–41. 7. butcher r, sampson m, couban rj, malin je, loree s, brody s. the currency and completeness of specialized databases of covid-19 publications. j clin epidemiol. 2022 jul;147:52–9. 8. gusenbauer m, haddaway nr. which academic search systems are suitable for systematic reviews or meta-analyses? evaluating retrieval qualities of google scholar, pubmed, and 26 other resources. res synth methods. 2020 mar;11(2):181–217. 9. bethel a, rogers m. a checklist to assess database-hosting platforms for designing and running searches for systematic reviews. health inf libr j. 2014 mar;31(1):43–53. 10. rethlefsen ml, kirtley s, waffenschmidt s, ayala ap, moher d, page mj, et al. prisma-s: an extension to the prisma statement for reporting literature searches in systematic reviews. syst rev. 2021 jan 26;10(1):39. 11. whiting p, westwood m, burke m, sterne j, glanville j. systematic reviews of test accuracy should search a range of databases to identify primary studies. j clin epidemiol. 2008 apr;61(4):357–64. 12. royle p, bain l, waugh n. systematic reviews of epidemiology in diabetes: finding the evidence. bmc med res methodol. 2005 jan 8;5:2. 13. hartling l, featherstone r, nuspl m, shave k, dryden dm, vandermeer b. the contribution of databases to the results of systematic reviews: a cross-sectional study. bmc med res methodol. 2016 sep 26;16(1):127. 14. haig a, dozier m. beme guide no. 3: systematic searching for evidence in medical education–part 2: constructing searches. med teach. 2003 jan;25(5):463–84. 15. haig a, dozier m. beme guide no 3: systematic searching for evidence in medical education–part 1: sources of information. med teach. 2003 jan;25(4):352–63. 16. elsevier. citescore [internet]. 2023 [cited 2023 jul 18]. available from: https://www.elsevier.com/solutions/scopus/how-scopus-works/metrics/citescore 17. education resources information center (eric). eric selection policy [internet]. washington, dc: u.s. department of education; 2022 may [cited 2023 jul 19]. available from: https://eric.ed.gov/pdf/eric selection 2022.pdf 18. gusenbauer m. a free online guide to researchers’ best search options. nature. 2023 mar 23;615(7953):586–586. 19. gusenbauer m. search where you will find most: comparing the disciplinary coverage of 56 bibliographic databases. scientometrics. 2022;127(5):2683–745. 20. schaefer n, morgan-daniel j. comparison of databases for complex evidence syntheses on education for the health professions. med ref serv q. 2023 jul 3;42(3):240–59. 8 http://jmla.pitt.edu/ojs/jmla/article/view/1086 https://journals.library.ualberta.ca/istl/index.php/istl/article/view/1727 https://www.elsevier.com/solutions/scopus/how-scopus-works/metrics/citescore https://eric.ed.gov/pdf/eric_selection_2022.pdf hypothesis , vol. 35, no. 2, 2023 staying relevant in the era of budget cuts and artificial intelligence margaret a. hoogland, mls, ahipa aclinical medical librarian and associate professor, university libraries, the university of toledo, toledo, ohio, https://orcid.org/0000-0002-9932-3605 , margaret.hoogland@utoledo.edu back in the stone ages of the early 2000s, i attended a panel session on career paths. at least once, each panelist found an occasion to state “i never learned that in library school!” at that time, the panelists predicted a turnover in current library administrators within 5 years. they also thought e-books would be the norm and that google or google scholar, at least in academic settings, would put our profession out of business. fast forward to present day and career path panelists might say, “i never learned how to do programming in library school.” or, “it’s not possible to do my job because of declining enrollment, budget cuts, etc.” my responses, when asked about my career or getting through a murky time, fall into three categories: stay agile, community connections, and finding the bright spot. stay agile take a moment to consider the development and launching of the electronic health record (née electronic medical record) and the field of informatics. both continue to grow and change, but in time librarians potentially could work with other departments to add connections to library resources in the electronic health record. further, respected researchers in our field are lecturers and – in some cases – directors of informatics programs. the use of artificial intelligence (ai) could improve how we provide support and reduce the labor-intensive aspects of some services1-4. similarly, our field does acknowledge that google and google scholar are not going away. depending on the setting, google scholar could be the go-to information source for a busy health care provider or a student on rotation5-6. as publishers and authors, it is important to keep up with trends, especially when it comes to adjusting our requirements for citations, the data availability statement, and use of ai within submissions7. so rather than viewing new or emerging technologies as threats, perhaps consider more carefully current and potential use of such products. community connections tightening budgets, loss of personnel, and increasing responsibilities make it essential to establish and maintain a rapport with the community you support. for many, this translates into saying “yes” and spending time in meetings, which might not always directly apply to our work. these meetings, however, do get your name and face out in 1 https://orcid.org/0000-0002-9932-3605 https://www.orcid.org/ https://orcid.org/ mailto:margaret.hoogland@utoledo.edu hypothesis , vol. 35, no. 2, 2023 the community and could result in a follow-up email, appointment, or a pre-meeting instant message that leads to something more promising. when people ask me to take on a new role, join a committee, or recommend me for something, however, two questions come immediately to mind: what can i bring (e.g., prior experience, skills, etc.) to this role? what will this role give me or my community? having tenure gives me flexibility, but it also means increased responsibilities within my own college. recent conversations with colleagues have helped me realize that developing community connections takes courage, frequently hurts, and that it is almost never fun to say “no.” by thoughtfully considering these questions and protecting my time by saying “no,” i meet deadlines, reduce the possibility of burnout, and get the most out of projects. employing such tactics also opens the opportunity for me to provide better support within my community and to assist my colleagues in the university libraries. finding the bright spot regardless of how tough, murky, or impossible a situation may seem, surrendering is the only unacceptable response. it is ok to get frustrated, angry, or even yell. temporarily retreating, strategizing, and then returning for round two of the discussion or debate over the next steps also works. such tactics provide clarity and improve our ability to assess and to navigate or sometimes avoid future circumstances. hypothesis is a bright spot for me and each issue brings satisfaction and enjoyment. the editorial team and i hope you enjoy reading the articles featured in this issue. if you want to be in the loop about journal happenings and see the new issue table of contents alerts in your inbox, consider registering as a reader. margaret a. hoogland, mls, ahip, she, her, hers; september 2023 orcid id: https://orcid.org/0000-0002-9932-3605 email: margaret.hoogland@utoledo.edu acknowledgements i thank our editorial team, peer reviewers, kelly thormodson, and hypothesis open fora attendees for carving out time to share your thoughts, ideas, time, and talents with me. references 1. watkins e. ai and scholarly publishing a (slightly) hopeful view. the scholarly kitchen. 2023 may 10; 1–6. https://scholarlykitchen.sspnet.org/2023/05/10/guestpost-ai-and-scholarly-publishing-a-slightly-hopeful-view/ 2 https://journals.iupui.edu/index.php/hypothesis/user/register https://orcid.org/0000-0002-9932-3605 mailto:margaret.hoogland@utoledo.edu https://scholarlykitchen.sspnet.org/2023/05/10/guest-post-ai-and-scholarly-publishing-a-slightly-hopeful-view/ https://scholarlykitchen.sspnet.org/2023/05/10/guest-post-ai-and-scholarly-publishing-a-slightly-hopeful-view/ hypothesis , vol. 35, no. 2, 2023 2. klein n. ai machines aren’t “hallucinating.” but their makers are. the guardian. 2023 may 8;1–3. https://www.theguardian.com/commentisfree/2023/may/08/aimachines-hallucinating-naomi-klein 3. abbasgholizadeh rs, légaré f, sharma g, archambault p, zomahoun htv, chandavong s, et al. application of artificial intelligence in community-based primary health care: systematic scoping review and critical appraisal. j med internet res. 2021;23(9):e29839. doi:10.2196/29839 4. alkaissi h, mcfarlane si. artificial hallucinations in chatgpt: implications in scientific writing. cureus. 2023;10.7759/cureus.35179. doi:10.7759/cureus.35179 5. eldredge jd, hall lj, mcelfresh kr, warner td, stromberg tl, trost j, et al. rural providers’ access to online resources: a randomized controlled trial. j med libr assoc. 2016;104(1):33-41. doi:10.3163/1536-5050.104.1.005 6. duong d. time to embrace dr. google? cmaj. 2021 apr 19;193(16):e571–2. 7. lacey bryant s, bridgen r, hopkins e, mclaren c, stewart d. nhs knowledge and library services in england in the digital age. health info libraries j. 2022;39(4):385-91. doi:10.1111/hir.12457 3 https://www.theguardian.com/commentisfree/2023/may/08/ai-machines-hallucinating-naomi-klein https://www.theguardian.com/commentisfree/2023/may/08/ai-machines-hallucinating-naomi-klein appendix a. roles and responsibilities of research team members collects consent & completes citi training pre-protocol training not required principal investigator (pi) this role is responsible for being the point person with the irb for the protocol from submission-to-completion of the final report. this role collects consent and is listed on study recruitment information. consultant they address and speak to specific aspects or issues related to a project. if they view collected data, it must first be de-identified. co-principal investigator(s) (co-pi) they are listed on the study recruitment information and assist in drafting the irb protocol. statistical or data experts they will assist with creating the data collection instrument, but they analyze and perform appropriate statistical tests on de-identified data. research assistant(s) they may or may not assist with study recruitment, but they report to the pi or co-pi for instructions and tasks. they could collect consent and serve as notetakers for focus groups. additionally, they could assist with transcribing and analyzing interviews. non-research personnel these folks may perform any or all of the above roles. this is an ideal role for someone, who is new to doing research and might not have the time or interest in completing pre-protocol training requirements. t h e j o ur n a l o f the re se ar ch sec t i o n o f the med ic a l l i b r ar y as s oc i a t i o n winter 2015 volume 27, issue 1 hypothesis hypothesis, vol. 27, no. 1, winter 2015 1 hypothesis (issn 1093-5665) is the official journal of the research section of the medical library association. hypothesis is indexed in the cumulative index to nursing and allied health literature™ (cinahl). hypothesis is available online at http:// www.mlanet.org/p/cm/ld/fid=503. articles a look at the scholarly output of the medical library association research section* terrie r. wheeler, noga yaniv, and ruth e. fenske ..... 3 report on the hypothesis and research section communication survey kristine m. alpi and brooke l. billman ........................... 8 research section news section chair’s column emily s. mazure........................................................... 13 new award for best jmla research paper with an mla author – inaugural 2014 winning authors! aileen mccrillis and kristine m. alpi ............................ 14 mla 2014 annual meeting research award winners, chicago, il diane cooper, sandra de groote, and aileen mccrillis 15 mla 2015 annual meeting research award winners, austin, tx sandra de groote, jennifer lyon, and terry henner .... 20 mla 2015 annual meeting research section programs and meetings, austin, tx ............................................. 25 *peer-reviewed cover art: sabbatia angularis, annin & smith, illustration from american medical botany, by jacob bigelow, 1820. from the images from the history of medicine database by the national library of medicine. winter 2015 volume 27, issue 1 hypothesis t he jo ur na l o f t he re sear ch sec t i on o f t he med ica l l i brar y a ssoc ia t io n http://www.mlanet.org/p/cm/ld/fid=503 http://www.mlanet.org/p/cm/ld/fid=503 hypothesis, vol. 27, no. 1, winter 2015 2 interim editors editorial board section officers & executive committee kristine m. alpi, mls, mph, ahip kristine_alpi@ncsu.edu leslie m. behm, ahip behm@msu.edu brooke l. billman, ma, ahip brooke.billman@ahsl.arizona.edu jonathan eldredge, mls, phd, ahip jeldredge@salud.unm.edu ellen detlefsen, dls ellen@sis.pitt.edu chair emily s. mazure, msi, ahip emily.mazure@duke.edu chair-elect, program committee chair marie ascher, ms, ahip marie_asher@nymc.edu immediate past chair, nominating committee chair, section council representative merle rosenzweig, amls oriley@umich.edu secretary/treasurer martha f. earl, msls, ahip, chis mearl@mc.utmck.edu awards committee chair sandra de groote, mlis, ahip sgroote@uic.edu continuing education chair, email list manager leslie behm, ahip behm@msu.edu government relations liaison elaine g. powers, msls epowers@vcom.vt.edu international research chair, research agenda committee chair jonathan eldredge, mls, phd, ahip jeldredge@salud.unm.edu membership committee chair beatriz varman, mlis beatriz.varman@library.tmc.edu mentoring committee co-chairs maureen clark, mhs, mlis, lcpc mdclark@uic.edu susan lessick, mls, ma, ahip slessick@uci.edu jennifer staley, mlis jennifer.staley@uhhospitals.org strategic planning committee chair susan lessick, mls, ma, ahip slessick@uci.edu web site editor brooke l. billman, ma, ahip brooke.billman@ahsl.arizona.edu brooke l. billman, ma, ahip brooke.billman@ahsl.arizona.edu emily s. mazure, msi, ahip emily.mazure@duke.edu mailto:kristine_alpi@ncsu.edu mailto:behm@msu.edu mailto:brooke.billman@ahsl.arizona.edu mailto:jeldredge@salud.unm.edu mailto:ellen@sis.pitt.edu mailto:emily.mazure@duke.edu mailto:marie_asher@nymc.edu mailto:oriley@umich.edu mailto:mearl@mc.utmck.edu mailto:sgroote@uic.edu mailto:behm@msu.edu mailto:epowers@vcom.vt.edu mailto:jeldredge@salud.unm.edu mailto:beatriz.varman@library.tmc.edu mailto:mdclark@uic.edu mailto:slessick@uci.edu mailto:jennifer.staley@uhhospitals.org mailto:slessick@uci.edu mailto:brooke.billman@ahsl.arizona.edu mailto:brooke.billman@ahsl.arizona.edu mailto:emily.mazure@duke.edu editorial reviewed article hypothesis , vol. 35, no. 2, 2023 medical library association and special library association 2023 hybrid meeting research awards lindsay blake, mlis, ahipa aclinical services coordinator, uams library, university of arkansas for medical sciences, little rock, arkansas, https://www.orcid.org/0000-0002-8234-8611 , leblake@uams.edu the mla research caucus is pleased to announce the winners for best research papers and posters presented at the mla/sla 2023 hybrid meeting held in detroit, michigan. congratulations to all of the award winners! thank you to all the judges who volunteered their expertise to help select these deserving awardees both in the prejudging phase and during the meeting. to learn more about the awards and the selection process, visit the research caucus website. mla/sla 2023 annual meeting contributed paper awards 1st place osteopathic medical student attitudes towards research: consumers or creators? authors: molly montgomery, director of library services, idaho college of osteopathic medicine library; austin cuttone, marisa helm, mindy hoang, nahleh koochack, morton joshua, and nicholas rincon, medical students, idaho college of osteopathic medicine. 2nd place “you need to see it”: attitudes and preferences of health sciences librarians regarding assessment of evidence-based medicine via an objective structured clinical examination. authors: caitlin plovnick, lead, education and curriculum integration and joey nicholson, chair and director, new york university health sciences library, nyu langone health. 2nd place a scoping review on diversity, equity, and inclusion initiatives for health and medical library workers. authors: jane morgan-daniel, community engagement and health literacy librarian, university of florida health sciences center; amy taylor, medical librarian, houston methodist education institute, houston methodist hospital; xan goodman, health sciences librarian, university of nevada las vegas university libraries; and chloe hough, reference librarian, tampa bay regional campus library, nova southeastern university. mla/sla 2023 annual meeting contributed poster awards 1st place classifying data management plan guidance using dmptool. 1 https://www.orcid.org/0000-0002-8234-8611 https://orcid.org/ mailto:leblake@uams.edu http://www.mlanet.org/p/cm/ld/fid=938 editorial reviewed article hypothesis , vol. 35, no. 2, 2023 authors: lucy carr jones, research data and scholarly communications librarian, claude moore health sciences library, university of virginia and lauren phegley, research data engineer, penn libraries, university of pennsylvania. 2nd place open but hidden: open access gaps in national science foundation funded publications as posted online in the nsf public access repository. authors: kimberly r. powell, research impact informationist, woodruff health sciences center library; jenny townes, open access librarian, robert w. woodruff library, emory university; and fred rascoe, georgia tech library, georgia institute of technology. honorable mention existence of a library and degreed librarian at ancc magnet recognized hospitals. author: louisa verma, information professional, portland cement association. 2 an examination of research topics in bmla and jmla marton hypothesis, vol. 28, no. 1, fall 2016 6 title an examination of research topics in the journal of the medical library association and the bulletin of the medical library association: quantifying the importance of research to medical librarians over time author christine marton, phd1 introduction historically, librarians have undertaken observational studies of their users, specifically, of their preferences for and uses of specific information source formats, online databases, and information technologies. user satisfaction studies of library collections and services have also been popular. with the advent of desktop computing, computer networks and online databases in the latter half of the twentieth century, research studies of all kinds and across all disciplines have become more numerous. in today’s information society, research has grown into an integral activity no longer restricted to academics and research institutes. how has this trend affected the profession of health sciences librarianship? has the number of research projects in this discipline increased in recent years? what study designs are being utilized? as the foremost professional association for health sciences librarians and informationists in the united states of america, the medical library association (mla) sets priorities for its membership and more broadly, for the profession. in recent decades, mla has begun to prioritize research. in 1995, mla developed a research policy statement, using scientific evidence to improve information practice, which called on health sciences librarians to be proactive in creating, managing, and using scientific evidence [1]. in 2008, mla published the research imperative [2]. based on semi-structured interviews with key stakeholders, including editors of the journal of the medical library association, the research policy emphasizes evidence-based library and information practice. six themes were identified: creation of a research culture, domains of research, research skills set, roles of stakeholders, challenges, and measurement of progress. in 2008 and 2011 the research agenda committee of the research section of mla identified the an examination of research topics in bmla and jmla marton hypothesis, vol. 28, no. 1, fall 2016 7 fifteen most important researchable questions facing our profession. [3, 4] more recently teams have been conducting systematic reviews to determine the state of knowledge in these 15 areas. [5,6]. clearly, there is increasing recognition within the profession of health sciences librarianship of the importance of the role of librarians as researchers and evidence based practice. the journal of the medical library association (jmla) is an open access, peer-reviewed journal published quarterly by the medical library association. current and previous full-text issues are freely available on pubmed central. previously known as the bulletin of the medical library association (bmla), this journal serves as a reputable publication of research studies about health sciences librarianship. both major citation indexes, elsevier’s scimago and thomson reuters’ journal citation reports rank the jmla higher than comparable health sciences librarian journals in their respective lists of journal impact factors for library and information sciences journals. of the 209 journals in the library & information science (lis) subject category indexed by the scopus database, the scimago journal & country rank ranked the jmla in thirty-eighth position in 2015 with an sjr of 0.726, ahead of health information and libraries journal (43; 0.650); evidence based library and information practice (56, 0.586); medical reference services quarterly (65; 0.500); and journal of hospital librarianship (125; 0.217). [9] of the 86 journals in the lis subject category indexed in the journal citation reports database in 2015, the jmla was ranked 37 with a journal impact factor of 1.084, while health information and libraries journal was ranked 50, with a journal impact factor of 0.712. the other aforementioned health sciences librarianship journals are not indexed in the journal citation reports database. clearly, the jmla is the foremost publication for health sciences librarianship; articles published in this journal serve as a strong representation of the published research in this discipline. [7-9] objective this quantitative study seeks to determine whether health sciences librarians have undertaken more research activities in recent years by measuring the occurrences of research-related words in all issues of the peer-reviewed journal publication of the medical library association and comparing the percentage of occurrences (prevalence) between the earlier publication, bulletin of the medical library association and the current publication, journal of the medical library association. an examination of research topics in bmla and jmla marton hypothesis, vol. 28, no. 1, fall 2016 8 literature review several studies on librarianship research have been conducted by library science academics and academic librarians. these studies generally utilize a restricted time period, such as one year or five years, and examine a small number of library and information science (lis) journals. content analysis and bibliometrics are popular study designs. in their content analysis of librarianship research, koufogiannakis, slater and crumley examined the lis journal literature for a one-year period in 2001. of the 217 lis journals reviewed, they included 107 journals in their study, of which 91 contained relevant data. of the 2,664 journal articles examined, they classified 807 as research articles. the top lis journals for research in 2001 were the journal of the american society of information science and technology (jasist); scientometrics; information processing & management; college & research libraries; bulletin of the medical library association (tied with journal of library administration); libraries and culture; journal of documentation; and journal of information science (tied with journal of academic librarianship). according to the authors’ classification scheme of six subject domains, “information access and retrieval” contained the greatest number of research articles and the most frequently published type of research was descriptive research. non-experimental research, such as surveys, was found to be much more prevalent than experimental research. [10] slutsky and aytac reviewed science librarianship research from 2008–2012 in four lis journals: health information & libraries journal (hilj); journal of the medical library association (jmla); issues in science & technology librarianship (istl); and science & technology libraries (stl). they analyzed the texts of 574 articles and classified them as either research or non-research. non-research articles were removed from the analysis. slightly more than half of all articles were classified as research (n=311; 54.2%). study variables included authorship, affiliation, type of research, research topic, and data collection and data analysis techniques. bibliometric data analysis revealed that there has been a dramatic growth in research in these lis journals for the years 2008-2012. the majority of research papers featured quantitative study designs; qualitative studies comprised only 10% of the research articles examined. quantitative data analysis overwhelmingly consisted of descriptive statistics (88.7%). the most popular study designs were survey, content analysis, citation analysis, and interviews. there was a significant difference among these four different publications with respect to the location of the study, context of research, research approach, and statistical analysis. as well, overall authorship was an examination of research topics in bmla and jmla marton hypothesis, vol. 28, no. 1, fall 2016 9 highly collaborative; almost three-quarters of the research articles were written by two or more authors. of interest, the jmla contained the greatest number of research articles, followed by hilj, istl, and stl. this finding lends support to the presumption that jmla is a key journal, not only for health sciences librarianship research but for lis research more broadly. [11] in a more recent study, slutsky and aytac conducted a bibliometric analysis of research articles published in the aforementioned stem librarianship journals, istl and stl, over a ten-year period (2005-2014). they found a greater number of research articles in istl but higher scopus citation metrics for stl. as well, the most frequent topic in stl was “bibliometrics and citation analysis” while in istl it was “libraries and librarianship,” with “library resources” as the foremost topic. there were more author collaborations in istl than stl. of interest, jmla was ranked eighth in the list of top 25 lis journals cited in issues in istl and thirteenth in the top 25 lis journals cited in stl. [11] clearly, jmla is regarded as a reputable publication by science and technology librarians. [12] from these three bibliometric and citation analysis studies of the lis journal literature, it is apparent that bmla and its successor, jmla, are well regarded research publications in the field of library and information science. analyzing the prevalence of research studies and study designs in these two publications is a valid method for determining whether there has been an increase in research productivity in health sciences librarianship over time as well as an increase in the diversity of study designs beyond surveys and interviews. method to draw this comparison, the frequency (counts) and prevalence (percentages) of articles including words and phrases that are commonly used to describe research studies and methodologies were obtained from all issues of the bmla, which was published from 1911 to 2001 (volumes 1-89), and its successor, jmla, which has been published since 2002 (volumes 90-104). all issues are available online in full text from pubmed central (pmc). these two journals were queried using the search interface on the pmc webpage for the journals’ archive (www.ncbi.nlm.nih.gov/pmc/ journals/93/). http://www.ncbi.nlm.nih.gov/ an examination of research topics in bmla and jmla marton hypothesis, vol. 28, no. 1, fall 2016 10 search statements began with the first topic, which is the name of the journal: bulletin of the medical library association and its successor, journal of the medical library association. the journal name was entered as the first search term in the search box: "bulletin of the medical library association"[journal] for one set of searches and "journal of the medical library association : jmla"[journal] for the other set of searches. next, the first search term was paired with a second concept, beginning with the search term research, in double quotation marks, which was added to subsequent searches. for the second concept, the search term was entered in separate searches with either the field limit [all fields] or the field limits [title] or [abstract]. thus, two searches were conducted for each second concept. the [all fields] searches are broad: they capture the search terms as they appear in any part of the full text documents. the [title] or [abstract] field limits restrict the searches to those two bibliographic fields, thereby improving the precision of the searches. preliminary searches were conducted in late august, 2016. the original searches and new searches with additional second concept terms were conducted in early october, 2016 and also in mid-november 2016, when the fourth and final 2016 issue of jmla was added to the journal repository site on pubmed central. tables 1-2 in the results section below list all of the search statements and their corresponding number (counts and percentages) of search results. the searches progressed from broad to narrow as the search terms for the second concept became more specific. searching all issues of both bmla and jmla provides quantitative measures of all researchrelated words. the number of search results serves as a basic measure of the pervasiveness of research in this publication, and more broadly, in the field of health sciences librarianship. search term frequencies and percentages can be tabulated to facilitate pairwise quantitative comparisons that indicate whether there has been an increase in published research activities in health sciences librarianship over time. the use of inferential statistics, more specifically, the paired t-test, enables the calculation of the statistical significance of difference in means of search results for jmla in relation to its predecessor, bmla. an examination of research topics in bmla and jmla marton hypothesis, vol. 28, no. 1, fall 2016 11 the search results data were first entered into two comprehensive tables in a microsoft word document: one table for the [all fields] searches and a second table for the [title] or [abstract] field limited searches. all table columns were copied to an excel spreadsheet and the spreadsheet was then imported to the statistical package, spss (version 11). the appropriate inferential statistic, the paired t-test, was calculated to determine the strength and significance of the difference in means for the search results on research-related terms between bmla and jmla for (1) [all fields] searches and (2) [title] or [abstract] searches. percentages, instead of counts, were compared to take into account the larger number of journal issues for bmla in relation to jmla, which is a function of the greater timespan for the publication of bmla, and concomitant greater number of articles. unlike prior bibliometric and content analyses studies on the pervasiveness of research in the field of librarianship, the context of these terms was not investigated. thus, all occurrence of research-related terms in bmla and jmla are included irrespective of whether they were published in research articles or appear in editorials, reviews and other non-research columns of bmla and jmla. this is an acceptable approach because the objective is not to categorize the topics of research, nor to investigate characteristics of authorship or citation patterns, but solely to determine whether an increase in (1) research and (2) study designs has occurred in the profession of health sciences librarianship from 2002 onward, when bmla was renamed jmla. restricting the searches to the [title] or [abstract] fields improves, to some extent, the relevance of search results because of the greater specificity and relationship to topicality than the broad [all fields] searches. results the initial search, “bulletin of the medical library associaton”[journal] retrieved 7256 results while the initial search, "journal of the medical library association : jmla"[journal], retrieved 1498 search results. the greater publication timespan of bmla is accountable for this large difference in number of search results on journal title. an examination of research topics in bmla and jmla marton hypothesis, vol. 28, no. 1, fall 2016 12 general research search terms sr pmc search statement: bmla # % pmc search statement: jmla # % 1 "bulletin of the medical library association"[journal] alternatively: "bulletin of the medical library association"[journal] or "bull med libr assoc"[journal] 7256 100.0 "journal of the medical library association : jmla"[journal] alternatively: "journal of the medical library association"[journal] or "j med libr assoc"[journal] 1498 100.0 2 "bulletin of the medical library association"[journal] and ("research"[all fields]) 2941 40.5 ("journal of the medical library association : jmla"[journal]) and ("research"[all fields]) 1101 73.5 3 "bulletin of the medical library association"[journal] and ("study"[all fields] or "studies"[all fields]) 2945 40.6 ("journal of the medical library association : jmla"[journal]) and ("study"[all fields] or "studies"[all fields]) 996 66.5 4 "bulletin of the medical library association"[journal] and ("study design"[all fields]) 32 0.4 ("journal of the medical library association : jmla"[journal]) and ("study design"[all fields]) 75 5.0 table 1a: number of search results from pubmed central queries of bmla and jmla, all fields queries – general research search terms in the following sub-sections of the results section, the second concept, research-related, is operationalized as search terms for quantitative study designs (table 2a,b); systematic review and meta-analysis (table 3a,b); mixed-method study designs (table 4a,b), and qualitative study designs (table 5a,b). an examination of research topics in bmla and jmla marton hypothesis, vol. 28, no. 1, fall 2016 13 when the search term "research" is added and the [all fields] field limit is employed, the number of search results is somewhat more than halved to 2941 search results for the bulletin of the medical library association and 1101 search results for the journal of the medical library association. more importantly, the percentage is 40.5 for bmla and research versus 73.5 for jmla and research, almost a doubling of term occurrences. (table 1a) for the comparable [title] or [abstract] searches, the increase is even more dramatic: 4.6% for bmla and research versus 18.8% for jmla and research, a four-fold increase in term occurrence (table 1b). for this set of searches on general research terms, the mean number of search results for all fields bmla searches is 27.2 while the mean number of searches for all fields jmla searches is 48.3. for title or abstract searches, the mean number of search results for bmla searches is 3.3 while for jmla searches it is 13.1. thus, there is an increase over time for occurrence of general research terms. sr pmc search statement: bmla # % pmc search statement: jmla # % 1 "bulletin of the medical library association"[journal] 7256 100.0 "journal of the medical library association : jmla"[journal] 1498 100.0 2 "bulletin of the medical library association"[journal] and ((“research"[title] or "research"[abstract])) 332 4.6 ("journal of the medical library association : jmla"[journal]) and ((“research"[title] or "research"[abstract])) 281 18.8 3 "bulletin of the medical library association"[journal] and ((“study"[title] or "study"[abstract]) or ("studies"[title] or “studies"[abstract])) 379 5.2 ("journal of the medical library association : jmla"[journal]) and ((“study"[title] or "study"[abstract]) or ("studies"[title] or “studies"[abstract])) 301 20.1 4 "bulletin of the medical library association"[journal] and (("study design"[title] or "study design"[abstract]) or ("study designs"[title] or "study designs"[abstract])) 2 0 ("journal of the medical library association : jmla"[journal]) and (("study design"[title] or "study design"[abstract]) or ("study designs"[title] or "study designs"[abstract])) 5 0.3 table 1b: number of search results from pubmed central queries of bmla and jmla, title or abstract queries – general research search terms an examination of research topics in bmla and jmla marton hypothesis, vol. 28, no. 1, fall 2016 14 quantitative research search terms sr pmc search statement: bmla # % pmc search statement: jmla # % 1 "bulletin of the medical library association"[journal] and ("quantitative"[all fields] or "quantify"[all fields]) 286 3.9 ("journal of the medical library association : jmla"[journal]) and ("quantitative"[all fields] or "quantify"[all fields]) 182 12.2 2 "bulletin of the medical library association"[journal] and ("quantitative design"[all fields]) 0 0 ("journal of the medical library association : jmla"[journal]) and ("quantitative design"[all fields]) 0 0 3 "bulletin of the medical library association"[journal] and ("survey"[all fields] or "surveys"[all fields]) 1615 22.3 ("journal of the medical library association : jmla"[journal]) and ("survey"[all fields] or "surveys"[all fields]) 609 40.7 4 "bulletin of the medical library association"[journal] and ("questionnaire"[all fields] or "questionnaires"[all fields]) 655 9.0 ("journal of the medical library association : jmla"[journal]) and ("questionnaire"[all fields] or "questionnaires"[all fields]) 215 14.4 5 "bulletin of the medical library association"[journal] and ("bibliometric"[all fields]) 49 0.7 ("journal of the medical library association : jmla"[journal]) and ("bibliometric"[all fields]) 80 5.3 6 "bulletin of the medical library association"[journal] and ("citation analysis"[all fields]) 82 1.1 ("journal of the medical library association : jmla"[journal]) and ("citation analysis"[all fields]) 85 5.7 7 "bulletin of the medical library association"[journal] and ("content analysis"[all fields]) 30 0.4 ("journal of the medical library association : jmla"[journal]) and ("content analysis"[all fields]) 46 3.1 table 2a: number of search results from pubmed central queries of bmla and jmla, all fields queries –quantitative research search terms an examination of research topics in bmla and jmla marton hypothesis, vol. 28, no. 1, fall 2016 15 the second set of research terms examined pertain to quantitative study designs. the search results are presented in table 2a and table 2b. according to creswell, a key characteristic of quantitative study is determinism. “examining the relationships between and among variables is central to answering questions and hypotheses through surveys and experiments. the reduction to a parsimonious set of variables, tightly controlled through design or statistical analysis, provides measures or observations for testing a theory. objective data results from empirical observations and measures. validity and reliability of scores on instruments, additional standards for making knowledge claims, lead to meaningful interpretations of data.”[13] in tables 2a and 2b, several quantitative study designs are examined in relation to the occurrence of appropriate search terms in bmla and jmla. to begin, general terms, quantify and quantitative design, are utilized. then, the searches are narrowed to specific quantitative methods, such as survey and questionnaire. lastly, even narrower terms are employed, such as citation analysis. although the term quantitative design is not to be found in the search results, there are many search results for surveys and questionnaires, and smaller numbers for bibliometric(s), content analysis(es) and citation analysis(es). for each quantitative method, the percentage of search results for jmla exceeds that of bmla. the mean number of search results for bmla all fields searches on this set of quantitative study designs is 5.4 while the mean number of search results for jmla all fields searches is 11.6. for the title or abstract searches, bmla searches have a mean of 0.8 while jmla searches have a mean of 2.4. these statistical findings demonstrate an increase in the number of occurrences of quantitative research-related terms over time. an examination of research topics in jmla and bmla marton hypothesis, vol. 28, no. 1, fall 2016 16 sr pmc search statement: bmla # % pmc search statement: jmla # % 1 "bulletin of the medical library association"[journal] and (("quantitative"[title] or "quantitative"[abstract]) or ("quantify"[title] or “quantify"[abstract])) 27 0.4 ("journal of the medical library association : jmla"[journal]) and (("quantitative"[title] or "quantitative"[abstract]) or ("quantify"[title] or “quantify"[abstract])) 20 1.3 2 "bulletin of the medical library ssociation"[journal] and (("quantitative design"[title] or "quantitative design"[abstract]) or ("quantitative designs"[title] or “quantitative designs"[abstract])) 0 0 ("journal of the medical library association : jmla"[journal]) and (("quantitative design"[title] or "quantitative design"[abstract]) or ("quantitative designs"[title] or “quantitative designs"[abstract])) 0 0 3 "bulletin of the medical library association"[journal] and (("survey"[title] or "survey"[abstract]) or ("surveys"[title] or “surveys"[abstract])) 262 3.6 ("journal of the medical library association : jmla"[journal]) and (("survey"[title] or "survey"[abstract]) or ("surveys"[title] or “surveys"[abstract])) 147 9.8 4 "bulletin of the medical library association"[journal] and (("questionnaire"[title] or "questionnaire"[abstract]) or ("questionnaires"[title] or “questionnaires"[abstract])) 86 1.2 ("journal of the medical library association : jmla"[journal]) and (("questionnaire"[title] or "questionnaire"[abstract]) or ("questionnaires"[title] or “questionnaires"[abstract])) 34 2.3 5 "bulletin of the medical library association"[journal] and (("bibliometric"[title] or "bibliometric"[abstract]) or ("bibliometrics"[title] or “bibliometrics"[abstract])) 17 0.2 ("journal of the medical library association : jmla"[journal]) and (("bibliometric"[title] or "bibliometric"[abstract]) or ("bibliometrics"[title] or “bibliometrics"[abstract])) 20 1.3 6 "bulletin of the medical library association"[journal] and (("citation anaysis"[title] or "citation analysis"[abstract]) or ("citation analyses"[title] or “citation analyses"[abstract])) 13 0.2 ("journal of the medical library association : jmla"[journal]) and (("citation analysis"[title] or "citation analysis"[abstract]) or ("citation analyses"[title] or “citation analyses"[abstract])) 28 1.7 7 "bulletin of the medical library association"[journal] and (("content analysis"[title] or "content analysis"[abstract]) or ("content analyses"[title] or “content analyses"[abstract])) 7 0.1 ("journal of the medical library association : jmla"[journal]) and (("content analysis"[title] or "content analysis"[abstract]) or ("content analyses"[title] or “content analyses"[abstract])) 9 0.6 table 2b: number of search results from pubmed central queries of bmla and jmla, title or abstract queries –quantitative research search terms an examination of research topics in jmla and bmla marton hypothesis, vol. 28, no. 1, fall 2016 17 systematic reviews and meta-analyses a systematic review provides a comprehensive analysis of all existing primary studies on a very well-defined research question. it evaluates the methods employed in these studies, summarizes the results, presents important findings, identifies reasons for differences in findings across studies, and identifies the limitations of current knowledge. combining the results mathematically through the use of statistical methods that sumarize all of the findings from the primary studies is referred to as a meta-analysis. [14] sr pmc search statement: bmla # % pmc search statement: jmla # % 1 "bulletin of the medical library association"[journal] and ("systematic review"[all fields] or "systematic reviews"[all fields]) 19 0.3 ("journal of the medical library association : jmla"[journal]) and ("systematic review"[all fields] or "systematic reviews"[all fields]) 287 19.2 2 "bulletin of the medical library association"[journal] and ("meta analysis"[all fields] or "meta analyses"[all fields]) 31 0.4 ("journal of the medical library association : jmla"[journal]) and ("meta analysis"[all fields] or "meta analyses"[all fields]) 95 6.3 table 3a: number of search results from pubmed central queries of bmla and jmla, all fields queries – systematic review and meta-analysis search terms for the all fields searches of this set of research terms, the mean value of bmla searches is 0.35 while the mean value of jmla searches is 12.8. for the title or abstract searches, the mean value of bmla searches is 0.1 while the mean value of jmla searches is 1.4. these statistical findings demonstrate an increase in the number of occurrences of these two research-related terms: systematic review(s) and meta-analysis(es) over time. an examination of research topics in jmla and bmla marton hypothesis, vol. 28, no. 1, fall 2016 18 sr pmc search statement: bmla # % pmc search statement: jmla # % 1 "bulletin of the medical library association"[journal] and (("systematic review"[title] or "systematic review"[abstract]) or ("systematic reviews"[title] or “systematic reviews"[abstract])) 5 0.1 ("journal of the medical library association : jmla"[journal]) and (("systematic review"[title] or "systematic review"[abstract]) or ("systematic reviews"[title] or “systematic reviews"[abstract])) 35 2.3 2 "bulletin of the medical library association"[journal] and (("meta analysis"[title] or "meta analysis"[abstract]) or ("meta analyses"[title] or “meta analyses"[abstract])) 6 0.1 ("journal of the medical library association : jmla"[journal]) and (("meta analysis"[title] or "meta analysis"[abstract]) or ("meta analyses"[title] or “meta analyses"[abstract])) 7 0.5 table 3b: number of search results from pubmed central queries of bmla and jmla, title or abstract queries – systematic review and meta-analysis search terms mixed methods research mixed methods study designs include both quantitative and qualitative methods. either the quantitative or the qualitative component can be dominant, or they can co-exist equally. creswell provides a useful matrix that illustrates the four decisions required to select a mixed methods approach. the use of a theoretical framework; the implementation sequence of quantitative and qualitative data collection; the priority given to quantitative and qualitation data collection and analysis, and the integration of quantitative and qualitative findings are key considerations for mixed methods designs. [15] an examination of research topics in jmla and bmla marton hypothesis, vol. 28, no. 1, fall 2016 19 sr pmc search statement: bmla # % pmc search statement: jmla # % 1 "bulletin of the medical library association"[journal] and ("mixed method"[all fields] or "mixed methods"[all fields]) 0 0 ("journal of the medical library association : jmla"[journal]) and ("mixed method"[all fields] or "mixed methods"[all fields]) 21 1.4 2 "bulletin of the medical library association"[journal] and ("triangulation"[all fields]) 2 0 ("journal of the medical library association : jmla"[journal]) and ("triangulation"[all fields]) 17 1.1 table 4a: number of search results from pubmed central queries of bmla and jmla, all fields queries – mixed methods search terms sr pmc search statement: bmla # % pmc search statement: jmla # % 1 "bulletin of the medical library association"[journal] and (("mixed method"[title] or "mixed method"[abstract]) or ("mixed methods"[title] or "mixed methods"[abstract])) 0 0 ("journal of the medical library association : jmla"[journal]) and (("mixed method"[title] or "mixed method"[abstract]) or ("mixed methods"[title] or "mixed methods"[abstract])) 5 0.3 2 "bulletin of the medical library association"[journal] and (("triangulation"[title] or "triangulation"[abstract]) or ("triangulate"[title] or "triangulate"[abstract])) 0 0 ("journal of the medical library association : jmla"[journal]) and (("triangulation"[title] or "triangulation"[abstract]) or ("triangulate"[title] or "triangulate"[abstract])) 2 0.1 table 4b: number of search results from pubmed central queries of bmla and jmla, title or abstract queries – mixed methods search terms an examination of research topics in jmla and bmla marton hypothesis, vol. 28, no. 1, fall 2016 20 qualitative research search terms creswell characterizes qualitative research as broad, holistic, reflective and interpretative. [16] for this fifth and final set of searches on research-related terms in bmla and jmla, general terms for qualitative research, qualitative and qualitative design, are entered first, followed by more specific search terms for individual qualitative approaches: interview(s), focus group(s), critical incident(s), and phenomenology. for the all fields searches, the mean value for bmla searches is 2.3 while the mean value for jmla searches is 8.3, an almost four-fold difference. for the title or abstract searches, the mean value for bmla searches is 0.1 while the mean value for jmla searches is 0.85, an eight-fold difference. these statistical findings demonstrate an increase in the number of occurrences of qualitative research term over time. sr pmc search statement: bmla # % pmc search statement: jmla # % 1 "bulletin of the medical library association"[journal] and ("qualitative"[all fields] or “descriptive”[all fields]) 518 7.1 ("journal of the medical library association : jmla"[journal]) and ("qualitative"[all fields] or “descriptive”[all fields]) 343 22.9 2 "bulletin of the medical library association"[journal] and ("qualitative design"[all fields]) 0 0 ("journal of the medical library association : jmla"[journal]) and ("qualitative design"[all fields]) 1 0.3 3 "bulletin of the medical library association"[journal] and ("interview"[all fields] or "interviews"[all fields]) 375 5.2 ("journal of the medical library association : jmla"[journal]) and ("interview"[all fields] or "interviews"[all fields]) 261 17.4 4 "bulletin of the medical library association"[journal] and ("focus group"[all fields] or "focus groups"[all fields]) 48 0.7 ("journal of the medical library association : jmla"[journal]) and ("focus group"[all fields] or "focus groups"[all fields]) 119 7.9 5 "bulletin of the medical library association"[journal] and ("critical incident"[all fields] or "critical incidents"[all fields]) 34 0.5 ("journal of the medical library association : jmla"[journal]) and ("critical incident"[all fields] or "critical incidents"[all fields]) 12 0.8 6 "bulletin of the medical library association"[journal] and ("phenomenology"[all fields] or "phenomenological"[all fields]) 5 0.1 ("journal of the medical library association : jmla"[journal]) and ("phenomenology"[all fields] or "phenomenological"[all fields]) 4 0.3 table 5a: number of search results from pubmed central queries of bmla and jmla, all fields queries – qualitative research search terms an examination of research topics in jmla and bmla marton hypothesis, vol. 28, no. 1, fall 2016 21 sr pmc search statement: bmla # % pmc search statement: jmla # % 1 "bulletin of the medical library association"[journal] and (("qualitative"[title] or "qualitative"[abstract]) or ("descriptive”[title] or “descriptive"[abstract])) 33 0.5 ("journal of the medical library association : jmla"[journal]) and (("qualitative"[title] or "qualitative"[abstract]) or ("descriptive”[title] or “descriptive"[abstract])) 50 3.3 2 "bulletin of the medical library association"[journal] and (("qualitative design"[title] or "qualitative design"[abstract]) or ("qualitative designs”[title] or “qualitative designs"[abstract])) 0 0 ("journal of the medical library association : jmla"[journal]) and (("qualitative design"[title] or "qualitative design"[abstract]) or ("qualitative designs”[title] or “qualitative designs"[abstract])) 0 0 3 "bulletin of the medical library association"[journal] and (("interview"[title] or "interview"[abstract]) or ("interviews"[title] or “interviews"[abstract])) 35 0.5 ("journal of the medical library association : jmla"[journal]) and (("interview"[title] or "interview"[abstract]) or ("interviews"[title] or “interviews"[abstract])) 52 3.5 4 "bulletin of the medical library association"[journal] and (("focus group"[title] or "focus group"[abstract]) or ("focus groups"[title] or “focus groups"[abstract])) 4 0.1 ("journal of the medical library association : jmla"[journal]) and (("focus group"[title] or "focus group"[abstract]) or ("focus groups"[title] or “focus groups"[abstract])) 17 1.1 5 "bulletin of the medical library association"[journal] and (("critical incident"[title] or "critical incident"[abstract]) or ("critical incidents"[title] or “critical incidents"[abstract])) 1 0 ("journal of the medical library association : jmla"[journal]) and (("critical incident"[title] or "critical incident"[abstract]) or ("critical incidents"[title] or “critical incidents"[abstract])) 1 0.1 6 "bulletin of the medical library association"[journal] and ((“phenomenology"[title] or "phenomenology"[abstract]) or ("phenomenological"[title] or "phenomenological"[abstract])) 0 0 ("journal of the medical library association : jmla"[journal]) and ((“phenomenology"[title] or "phenomenology"[abstract]) or ("phenomenological"[title] or "phenomenological"[abstract])) 1 0.1 table 5b: number of search results from pubmed central queries of bmla and jmla, title or abstract queries – qualitative research search terms an examination of research topics in jmla and bmla marton hypothesis, vol. 28, no. 1, fall 2016 22 summary of findings progressively narrowing the searches through the addition of the second concept, the researchrelated terms, predictively reduces the number of search results, with more specific search terms reducing the number of search results considerably. moreover, when the second search concept is restricted to [title] or [abstract], there is a five to ten-fold reduction in the number of search results in comparison to the [all fields] searches. the higher percentage of research-related search results for jmla searches relative to bmla searches is found consistently for all searches, for both the [all fields] searches and the [title] or [abstract] searches. clearly, there is a marked increase in general and specific research-related terms in jmla relative to its predecessor, bmla. paired t-tests conducted in the statistical package, spss (version 11) demonstrate that these differences are statistically significant for both the all fields searches (t=4.092, df=19, p=.001) and the title or abstract searches (t=2.615, df=19, p=.017). overall, the quantitative data analysis of occurrences of research-related terms indicates that there is an increase in published research activity in the health sciences librarianship profession over time, with the largest noticeable increase for systematic reviews. discussion conducting searches within issues of the bulletin of the medical library association and the journal of the medical library association reveal several interesting findings concerning the prominence of research in this leading health sciences librarianship journal over time, as measured by percentage of search terms. first, the word research appears in abundance in both bmla and jmla. the word study and its plural, studies, is also frequently found in bmla and jmla. the large number of search results suggests a prominent role for research in health sciences librarianship. paired t-tests comparising bmla searches to jmla searches demonstrate statistically significant increases in occurrences of research-related terms over time for general research terms, quantitative research terms, qualitative research terms, and mixed-methods research terms. all research-related search terms are mentioned more frequently in jmla than bmla suggesting that health sciences librarianship research has increased since 2002, which supports a finding by slutsky and aytac concerning the increase in research in librarianship. as well, a greater diversity of study designs are being utilized. an examination of research topics in jmla and bmla marton hypothesis, vol. 28, no. 1, fall 2016 23 surveys are popular. surveys are commonly questionnaires. however, they can also take the form of interviews that utilize a pre-determined set of questions and list of responses. well-known qualitative methods, such as open-ended interviews and focus groups are also popular. the predominance of quantitative, non-experimental designs, confirms findings from earlier research on lis journals conducted by koufogiannakis, slater, and crumley and by slutsky and aytac. less well known qualitative study designs, such as the critical incident technique and phenomenological approaches, are infrequently found in both bmla and jmla. systematic reviews, a form of secondary research, has gained considerably in visibility. the significant increase in mention of systematic reviews is an interesting trend that is reflective of the current interest in evidence-based medicine (ebm) and evidence-based practice (ebp). limitations there are several limitations associated with this type of study. first, number of search results is a very basic quantitative measure that is largely devoid of context. this method can be refined by searching each individual issue of bmla and jmla and counting the number of search results per search term. as well, the occurrence of each search term in individual articles would represent a further refinement of this measure and determine to some extent the context in which these research-related terms are utilized. second, health sciences librarians have other publication options, such as health information and libraries journal, medical reference services quarterly and journal of hospital librarianship, to name but a few journal titles in this discipline. as well, they can publish their research in health sciences journals or information science journals. thus, a quantitative study that focuses solely on counting the number of search results for research-related terms in one journal, albeit a prominent journal in its field, will only capture a fraction of the research studies published in the journal literature by health sciences librarians. future directions future studies to investigate this research topic further include replication studies utilizing the research method employed in this study for investigation in other health sciences librarianship journals to determine if the same trends are present. as well, this research method could be utilized in future studies but with a more fine-grained approach to searching for research-related an examination of research topics in jmla and bmla marton hypothesis, vol. 28, no. 1, fall 2016 24 terms in specific issues of bmla and jmla. additional research-related search terms could be considered for inclusion, such as thematic analysis, inductive analysis, theoretical framework, and inferential statistics, among others. another avenue to explore is authorship metrics in relation to study design. for example, wheeler, yaniv and fenske determined the most influential authors who are members of the mla research section, according to number of citations in the web of science citation report. carol lefebvre’s articles on systematic reviews were in the top ten papers in the lis discipline, suggesting a relationship between citation count and study design. [17] author affiliation is also of interest. for example, hardin and stankus reported the institutional affiliations of the published academic science, engineering, agricultural, and medical librarians over a ten-year period (2000-2010). the lis journals examined were science & technology libraries; issues in science and technology librarianship; journal of agricultural and food information; journal of the medical library association; and medical reference services quarterly. the top us affiliations in this set of lis journals were: illinois, purdue, texas, penn state and cornell. [18] exploring relationships between author h-index, author affiliation, article citation count, journal ranking, and study design may yield interesting findings. references 1. grefsheim sf, rankin ja, perry gj, mckibbon ka. affirming our commitment to research: the medical library association’s research policy statement: the process and findings. journal of the medical library association. 2008;96(2):114-120. doi: 10.3163/1536-5050.96.2.114. 2. dalrymple pw, bastille jd, bradley j, dee cr, humphreys bl, marshall jg, weller ac, webb re. using scientific evidence to improve information practice. chicago, il: medical library association, 1995. 3. eldredge jd, harris mr, ascher mt. defining the medical library association research agenda: methodology and final results from a consensus process. journal of the medical library association.2009;97(3):178-185. doi: 10.3163/1536-5050.97.3.006. 4. eldredge jd, ascher mt, homes jn, harris mr. the new medical library association research agenda: final results from a three-phase delphi study. journal of the medical library association. 2012;100(3):214-218. doi: 10.3163/1536-5050.100.3.012 5. ascher mt, eldredge jd, holmes hn, harris mr. 2012. medical library association. research section. research agenda committee. mla research agenda: appraising the best available evidence. http://repository.unm.edu/handle/1928/24634 https://dx.doi.org/10.3163%2f1536-5050.96.2.114 https://dx.doi.org/10.3163%2f1536-5050.97.3.006 https://dx.doi.org/10.3163%2f1536-5050.100.3.012 https://scholar.google.com/scholar?oi=bibs&cluster=11436103200852211256&btni=1&hl=en https://scholar.google.com/scholar?oi=bibs&cluster=11436103200852211256&btni=1&hl=en http://repository.unm.edu/handle/1928/24634 an examination of research topics in jmla and bmla marton hypothesis, vol. 28, no. 1, fall 2016 25 6. eldredge jd, ascher mt, holmes hn. medical library association research section. research agenda committee. mla research agenda: systematic review project team updates presentation. mla annual meeting. may 17, 2015. http://repository.unm.edu/handle/1928/27127 7. scimago. (2016). about us. november 8, 2011. <http://www.scimagojr.com/aboutus.php> 8. scimago. (2016). journal rankings: library and information sciences. november 8, 2011. <http://www.scimagojr.com/ journalrank.php?category=3309> 9. 2015 journal citation reports® social sciences edition (thomson reuters, 2016) 10. koufogiannakis d, slater l, crumley e. a content analysis of librarianship research. journal of information science. 2004;30(3):227-239. doi: 10.1177/0165551504044668 11. slutsky b, aytac s. publication patterns of science, technology, and medical librarians: review of the 2008-2012 published research. science & technology libraries. 2014;33(4):369-382. doi: 10.1080/0194262x.2014.952486 12. slutsky b, aytac s. bibliometric analysis and comparison of two stem lis journals: science & technology libraries and issues in science & technology librarianship (2005–2014). science & technology libraries. 2016;35(2):152-171. doi: 10.1080/0194262x.2016.1171191 13. creswell jw. research design: qualitative, quantitative, and mixed methods approaches. 2nd ed. thousand oaks, ca: sage publications; 2002. p. 153. 14. garg ax, hackam d, tonelli m. systematic review and meta-analysis: when one study is just not enough. clinical journal of the american society of nephorology. 2008, 3(1): 253–260. doi: 10.2215/cjn.01430307. p. 253. 15. creswell jw. research design: qualitative, quantitative, and mixed methods approaches. 2nd ed. thousand oaks, ca: sage publications; 2002. p. 182. 16. creswell jw. research design: qualitative, quantitative, and mixed methods approaches. 2nd ed. thousand oaks, ca: sage publications; 2002. p. 211. 17. wheeler tr, yaniv n, fenske re. (2015). a look at the scholarly output of the medical library association research section. hypothesis. 2015;27(1):3-7. 18. hardin a, stankus t. the affiliations of u.s. academic librarians in the most prominent journals of science, engineering, agricultural, and medical librarianship, 2000-2010. science & technology libraries. 2011;30(2): 143-156. doi: 10.1080/0194262x.2011.575285 author affiliations 1 adjunct instructor, faculty of information, university of toronto, toronto, ontario; christine.marton@utoronto.ca http://www.scimagojr.com/aboutus.php http://www.scimagojr.com/%20journalrank.php?category=3309 https://scholar.google.com/citations?view_op=view_citation&hl=en&user=gi865cuaaaaj&cstart=20&sortby=pubdate&citation_for_view=gi865cuaaaaj:u-x6o8ysg0sc http://dx.doi.org/10.1080/0194262x.2014.952486 http://dx.doi.org/10.1080/0194262x.2016.1171191 http://dx.doi.org/10.1080/0194262x.2011.575285 mailto:christine.marton@utoronto.ca hypothesis, vol. 36, no. 1, 2024 the pre-submission checklist for hypothesis authors margaret a. hoogland*, mls, ahip; she, her, hers;a, aassociate professor and clinical medical librarian, university libraries, the university of toledo,toledo, ohio, https://orcid.org/0000-0002-9932-3605 , margaret.hoogland@utoledo.edu cite as: hoogland ma. the pre-submission checklist for hypothesis authors. hypothesis. 2024; 36(1). doi:10.18060/27712 hoogland. all works in hypothesis are licensed under a cc by-nc 4.0 deed attribution-noncommercial 4.0 international. authors own copyright of their articles appearing in hypothesis. readers may copy articles without permission of the copyright owner(s), as long as the author(s) are acknowledged in the copy, and the copy is used for educational, not-for-profit purposes. for any other use of articles, please contact the copyright owner(s). 1 https://www.orcid.org/ https://orcid.org/ mailto:email https://creativecommons.org/licenses/by-nc/4.0/ https://creativecommons.org/licenses/by-nc/4.0/ hypothesis, vol. 36, no. 1, 2024 this editorial provides a list of five things author(s) should consider before submitting to hypothesis: 1. avoid plagiarism and copyright infringement. the instructions to authors outline what hypothesis expects of submissions, which include non-original image(s) in a manuscript. if an author requires clarification or would benefit from additional information, it is best to touch base with authors who have experience with the journal or to email the editor. when considering plagiarism, the editor consults with associate editors, expert colleagues, and members of the editorial team. hypothesis values the input of all editorial team members, peer reviewers, and fellow information professionals. ultimately, the journal editor sets the policy, takes responsibility for the final decision, and shares the determination with the author. 2. include the appropriate number of relevant references. the editor expects to see a minimum of three references for each non-brief report submission and anticipates seeing the article which inspired the project, or the original article for historical submissions. articles on a new or emerging field (e.g., artificial intelligence, new use of mobile devices) should cite the most current literature. if the topic is new or emerging in the field of library and information science, then an author might need to consult published articles outside of the library and information science field. providing context for how the topic relates to the field of library and information science could also be useful for colleagues. 3. endeavor to place citations in your manuscript so as not to distract the reader. although the citation requirements and styles strive to be straightforward, it takes familiarity with the content and lots of practice to seamlessly include citations in the manuscript. for this editor, this entails planning to write multiple drafts (1-3 minimum – including this editorial!), asking for internal feedback from readers with multiple perspectives and experiences, and carving out sufficient time for the entire process. 4. finalize authors and author order as you prepare to submit the manuscript. determining authorship and author order continues to be a challenge. partway through a project, author(s) might need an extended break; they suddenly have limited time to work on the project; or they might completely step off a project. for these reasons, authors should consider waiting to determine authorship and order of authors as they prepare to submit a manuscript. in 2009 and 2010, the hypothesis research mentor columns outlined criteria for authorship and acknowledgement according to the international committee for medical journal editors (icjme)1-2.the credit (contributor roles taxonomy)3 statement outlines ways to identify contributions of team members on multi-authored submissions. hypothesis asks authors to complete the credit statement, because it encourages author teams to assess what they did 2 https://journals.iupui.edu/index.php/hypothesis/authorinstructions https://journals.iupui.edu/index.php/hypothesis/briefreport hypothesis, vol. 36, no. 1, 2024 on a project and this could contribute to determining author order. this editor only questions authorship and author order if a submission contains an incomplete or confusing credit statement or if an author submits a complaint. 5. the timeline from submission-to-notification varies! the publication process includes multiple stages. as a first-time author or when submitting to a new journal, the delay in processing time can be frustrating and sometimes a cause for concern. outlined below is a draft submission timeline for hypothesis submissions: stage 1:submission completed by the editor, duration: 7-14 days • the editor reviews the submission and either asks the author(s) to make some corrections (e.g., figure files are indicated in text but not uploaded into the system or references are missing), rejects the submission if it does meet the aims and scope of the journal, or moves the submission to the next stage. stage 2:review completed by the peer reviewers, associate editors, editor duration: 28-70 days • the editor assigns three peer reviewers, who have two-to-four weeks (depending on the submission date and category) to review and provide feedback. • upon reviewing the comments from peer reviewers, the editor checks with the associate editor, who might choose to add comments or review a copy of the revised submission, and then notifies the author(s). • the editor could suggest a date for uploading revisions and the response(s) to reviewers, if the author(s) hope to include the article in a specific issue, or leave the turnaround time up to the discretion of the author(s). the editor tries to give author(s) 4-6 weeks for completing the revisions and response to reviewers. • upon reviewing the revised manuscript and response to reviewers, the editor and associate editor either confer and request additional revisions from the author or the submission moves to the next stage. stage 3: copy editing completed by the copy editing team, duration: 28 days • the editor assigns two copy editors to review each submission and to ensure it flows smoothly from one paragraph to the next, citations conform to the hypothesis requirements, and make comments about punctuation as needed. copy editors get 4-6 weeks for each submission. • the author(s) get up to 2 weeks to review and to respond to the copy edited version. 3 hypothesis, vol. 36, no. 1, 2024 • once both groups are satisfied, the copy editors upload the revised version onto the journal website and the submission moves to the final stage. stage 4: production completed by production editor or editor, duration: 14 days. • take the approved copy edited version, figures, images, etc., and create the galley proof in the hypothesis latex template. • authors have up to 1 week to review and to request the production editor make minor edits (e.g., punctuation is missing; correct this typo; please create appendixes instead of having the table be in-text). • after completing the minor edits, the production editor uploads a pdf file of the “galley.” • the production editor assigns the digital object identifier (doi) for the article. stage 5 publication day completed by production editor or editor, duration: 3-6 hours on publication day, the production editor creates the issue by: • creating an entry for the new volume and issue number in the system. • scheduling each article for publication in the upcoming issue. • the production editor publishes the issue but does not notify any member of the editorial team. • the production editor confirms that each article doi is active and working. • the production editor tells the editor that the issue is ready. stage 6: marketing the issue completed by the marketing team and the editor,duration: 1 hour • editor performs a final check to ensure each article doi is working. • editor shares the news with the editorial team. • the marketing team and the editor share the news via listservs and on social media. the hypothesis editorial team wishes you a happy spring and hopes you enjoy the march issue. 4 hypothesis, vol. 36, no. 1, 2024 acknowledgements i thank wade lee-smith, julia stumpff, andrea l. ball, and jill turner for providing thoughtful and candid feedback. references 1. eldredge j. authorship part one: defining the article author. hypothesis: the journal of the research section of mla. 2009; 21(3):11–4. available from: https://journals.iupui.edu/index.php/hypothesis/issue/view/1618 2. eldredge j. authorship part two: order of authors. hypothesis: the journal of the research section of mla. 2010; 22(1):8–11. available from: https://journals.iupui.edu/index.php/hypothesis/issue/view/1619/605 3. brand a, allen l, altman m, hlava m, scott j. beyond authorship: attribution, contribution, collaboration, and credit. learn pub. 2015 apr 1; 28(2):151–5. doi:10.1087/20150211 5 https://journals.iupui.edu/index.php/hypothesis/issue/view/1618 https://journals.iupui.edu/index.php/hypothesis/issue/view/1619/605 https://dx.doi.org/10.1087/20150211 peer reviewed article hypothesis, vol. 37, no. 1, 2025 library research skills for pre-health students: partnering with the pre-health advising office john carey, ma, mls, ahip-sa, ajatshatru (a.j.) pathak, mls, mphb aassociate professor, head, health professions library, hunter college, city university of new york, new york, ny https://orcid.org/0000-0001-9891-5142 , jca0022@hunter.cuny.edu bassociate professor, health and informatics librarian, health professions library, hunter college, city university of new york, new york, ny, apathak@hunter.cuny.edu cite as: carey j and pathak aj. library research skills for pre-health students: partnering with the pre-health advising office. hypothesis. 2025;37(1). doi:10.18060/28649 carey, pathak. all works in hypothesis are licensed under a cc by-nc 4.0 deed attribution-noncommercial 4.0 international. authors own copyright of their articles appearing in hypothesis. readers may copy articles without permission of the copyright owner(s), as long as the author(s) are acknowledged in the copy, and the copy is used for educational, not-for-profit purposes. for any other use of articles, please contact the copyright owner(s). 1 https://orcid.org/0000-0001-9891-5142 https://orcid.org/ mailto:jca0022@hunter.cuny.edu mailto:apathak@hunter.cuny.edu https://creativecommons.org/licenses/by-nc/4.0/ https://creativecommons.org/licenses/by-nc/4.0/ peer reviewed article hypothesis, vol. 37, no. 1, 2025 abstract introduction: this brief report describes a library initiative to deliver health professions-specific information literacy instruction to students in a college-wide health professions advising program at hunter college, a four-year institution within the city university of new york (cuny) system. the pre-health advising office “assists students on their path to becoming a doctor, dentist, optometrist, podiatrist, veterinarian, and select allied health fields, such as physical therapy (pt), physician assistant (pa), pharmacy, and occupational therapy. pre-health students . . . receive advising, skills development, access to programs and resources, application planning and review, and guided support.1" many of these students come to hunter as transfers from two-year programs within the cuny system or at other institutions; others may be freshmen, post-baccalaureate, or non-degree seeking students. in connection with a college-wide task force designed to increase retention and graduation rates among transfer students, the hunter college libraries formed a transfer committee to examine how the library can better support transfer students. in the fall 2022 semester, after discussion with the director of pre-health advising, this committee reached out to librarians at the hunter health professions library (hpl) to conduct information literacy sessions for all students in the health professions advising program. methods: hpl librarians offer a “two-shot” series of workshops via zoom every semester. scheduling and other information for each 80-minute session is tracked in the library’s libcal software and a zoom link is provided by the director of the pre-health advising office. the first workshop covers basic library resources and services, developing a research question, and search strategies in databases. the second workshop, delivered one to two weeks later, focuses on advanced topics such as literature reviews, evidence synthesis, and systematic review tools and methods. the librarians provide a libguide on the hunter college libraries website which summarizes and expands upon the content of both sessions. results: informal feedback has been positive. the pre-health advising office now requests the two sessions every semester. while the size of cohorts has varied from semester to semester, these workshops consistently reach much larger numbers of students than do typical library instruction sessions. the authors presented results from this program at the annual meeting of the liberty chapter of the medical library association in october 2024. discussion: these findings will be of interest to health sciences librarians who provide instruction and research support to students pursuing admission to graduate health professions programs, including transfer students. the workshops offered in conjunction with the pre-health advising office greatly increased the outreach of hpl librarians’ instruction efforts. these workshops can also have an enduring impact as they address skills and concepts that will remain relevant for these students at varying stages of their academic careers or future employment; to the authors’ knowledge, no other examples exist of workshops covering systematic review methods for pre-health students. further study will be needed to assess how or whether the content taught in these sessions has contributed to student success within the health professions advising program. 2 peer reviewed article hypothesis, vol. 37, no. 1, 2025 introduction this brief report describes a library initiative to deliver health professions-specific information literacy instruction to students in a college-wide advising program at hunter college, a four-year institution within the city university of new york (cuny) system. the pre-health advising office offers counseling, resources, and guided support to students who intend to apply to graduate or professional programs in medicine or allied health fields1. many of these students come to hunter as transfers from two-year programs within the cuny system; others may be freshmen, post-baccalaureate, or non-degree seeking students. in the fall 2022 semester, after discussions between the director of pre-health advising and the library, librarians at hunter’s health professions library (hpl) began to conduct information literacy sessions for all students in the advising program. methods hpl librarians offer a “two-shot” series of workshops via zoom every semester. scheduling and other information for each 80-minute session is tracked in the library’s libcal software and a zoom link is provided by the director of the pre-health advising office. the librarians determined the content of each session in consultation with the director but thus far these discussions have not included assessment of the program. the first workshop covers basic library resources and services, developing a research question, and search strategies in databases. the second workshop, delivered one to two weeks later, focuses on advanced topics such as literature reviews, evidence synthesis, and systematic review tools and methods. the librarians provide a libguide on the hunter college libraries website which summarizes the content of both sessions and expands upon topics such as library access and retrieving full text; interlibrary loan; citation management tools; and citation styles. results while the librarians have yet to conduct any formal assessment of the program, informal feedback has been positive. the pre-health advising office now requests the sessions every semester. while the size of cohorts has varied, these workshops consistently reach much larger numbers of students than do typical library instruction sessions (see table 1). the authors presented results from this initiative at the annual meeting of the liberty chapter of the medical library association in october 2024. table 1. workshop attendance by academic semester semester fall 2022 spring 2023 fall 2023 spring 2024 summer 2024 fall 2024 students attending per session (n) 356 51 271 113 71 341 3 peer reviewed article hypothesis, vol. 37, no. 1, 2025 discussion these findings will be of interest to health sciences librarians who provide instruction and research support to students pursuing admission to graduate health professions programs, including transfer students. costs of replicating this program elsewhere would include the time and labor of instruction librarians, as well as resources likely already available at many institutions such as online meeting software and a content hosting platform such as libguides. these workshops greatly increased the outreach of hpl librarians’ instruction efforts. they can also have an enduring impact as they address skills and concepts that will remain relevant for these students at varying stages of their academic careers or future employment; to the authors’ knowledge, no other examples exist of workshops covering systematic review methods for pre-health students. further study is needed to assess how or whether these sessions have contributed to student success within the advising program. credit statement john carey: investigation, writing—original draft preparation, project administration. ajatshatru (a.j.) pathak: conceptualization, methodology, investigation, writing—reviewing and editing. references 1. pre-health advising office. about the pre-health program [internet]. new york: hunter college pre-health advising office; 2024 [cited 2024 nov 8]. available from: https://www.hunter.cuny.edu/prehealth/prospective-undergraduate-students/about-pre-healthprogram 4 https://www.hunter.cuny.edu/prehealth/prospective-undergraduate-students/about-pre-health-program https://www.hunter.cuny.edu/prehealth/prospective-undergraduate-students/about-pre-health-program peer reviewed article hypothesis , vol. 34, no. 1, 2022 from the ground up: building a culture of research in an academic library jenessa m. mcelfresh, mlis, ahipa, maggie albro, mlisb, k. megan sheffield, ms, mlisc asystematic review service coordinator, senior research and learning services librarian/associate professor, health sciences library, university of tennessee health science center, memphis, tennessee, https://orcid.org/0000-0002-1633-4633 , jmcelfre@uthsc.edu bagriculture and natural resources librarian/assistant professor, pendergrass agriculture and veterinary medicine library, university of tennessee knoxville, knoxville, tennessee, https://orcid.org/0000-0002-0538-3501 , malbro@utk.edu cdata services librarian/associate librarian, university libraries, clemson university, clemson, south carolina, https://orcid.org/0000-0002-4030-2674 , msheff@clemson.edu background: this article discusses the motivations, benefits, and actions taken to create and support a culture of research among library employees at a carnegie research 1 university library system. despite substantial pressures to publish and engage in library science research, many academic librarians lack the training or support to build their own research agendas. this is particularly worrisome as tenure requirements and professional trends require librarians to be knowledgeable and productive in their research dissemination. experience: at clemson university libraries, librarians responded to this problem by adapting evidence-based solutions from scientific and medical disciplines to fit their own research empowerment needs. the process of building a productive culture of research at clemson libraries was a multi-year endeavor that took considerable initiative, planning, and flexibility. discussion: each of the initiatives required adaptation to the knowledge levels and existing culture in the libraries. organizers had to be willing to make changes and attempt new approaches as the organization evolved and comfort with the research process grew. some initiatives, like writing circle, were disbanded, while others, like journal club and research roundtable, continue to develop with high levels of engagement. takeaways: the success of this progressive approach to fostering a thriving culture of research at clemson libraries is evident through the rates of participation, dissemination, and satisfaction of library employees in addition to the expansion of the libraries’ engagement with research in outside departments and disciplines. librarians who seek to enhance their own institution’s research culture should consider implementing some or all of these initiatives to jumpstart their efforts, including writing circles, journal clubs, and research roundtables. 1 https://www.orcid.org/0000-0002-1633-4633 https://orcid.org/ mailto:jmcelfre@uthsc.edu https://orcid.org/0000-0002-0538-3501 https://orcid.org/ mailto:malbro@utk.edu https://orcid.org/0000-0002-4030-2674 https://orcid.org/ mailto:msheff@clemson.edu peer reviewed article hypothesis , vol. 34, no. 1, 2022 background: librarians have been conducting and disseminating research throughout the history of the profession, yet many struggle with research undertakings without ample support and a robust knowledge base from which to grow. at many colleges and universities, librarians are eligible for tenure; for example, an academic librarian position that is hired as “tenure track” implies that the position will be annually renewed and is expected to undergo a rigorous peer review process of their achievements after several years of work. achieving tenure typically comes with a promotion and a more permanent contract. though specific criteria vary among institutions, a common metaphor for tenure is a three-legged stool in which the legs are librarianship, research (expressed in terms of grants, scholarly articles, book chapters, professional conference presentations, and similar venues), and service (expressed via committee work, peer review services, or other activities in service to the profession). this often differentiates tenure-track librarians from their non-tenure-track counterparts, who may have no research or service components to their positions. academic librarians may feel greater pressure to publish their own original research to maintain their employment and achieve tenure, a pressure many other academic disciplines colloquially call “publish or perish.” this stress may be compounded by a lack of research methods training in their professional graduate programs. this article seeks to explore how one academic research library set out to enhance their culture of research to advance the comfort of librarian researchers and advocate to the institution the strengths of the library faculty. as the library system at a large public, land-grant institution, clemson libraries faced the challenge of re-envisioning themselves as a major research library when clemson university became a carnegie research 1 (r1) institution in 20161. to support this development, the university’s clemsonforward strategic plan, implemented in fall 2016, set forth a goal to “increase annual production of nationally recognized forms of scholarship by 50 percent”2. for the libraries, this meant increased support for all research conducted by university affiliates while increasing the research productivity of libraries’ faculty. in accordance with this goal in 2017, the libraries guidelines for reappointment, tenure, and promotion were changed to include scholarship and creative activities, otherwise known as research output, as a required element for consideration for any of these faculty movements. with all of these developments, libraries employees, both faculty and staff, found themselves in the unique position of needing to increase their engagement with research in order to both support the institution’s needs and to ensure their own professional success. as with any change, implementation took time. faculty librarians could not become expert researchers publishing in top journals overnight, and knowledge and experience gaps had to be identified and addressed to increase the entire department’s research output. one such gap was the lack of internal support for new and experienced faculty members conducting research. in other colleges and departments, already-tenured faculty can be relied upon for mentoring and knowledge-sharing for inexperienced faculty members expected to conduct research. in this situation, however, tenured libraries faculty were largely inexperienced with conducting most forms of scientific research and had little experience with the methodologies, software, and publishing requirements that untenured faculty needed the most guidance with. most libraries 2 peer reviewed article hypothesis , vol. 34, no. 1, 2022 faculty, tenured and untenured, had never taken a research methods course that pertains to library and information studies, and those that had taken such a course had experience pertaining to their other degrees in other fields. the lack of internal research support became readily apparent after a new wave of libraries faculty members came on board. between 2017 and 2020, the libraries welcomed 12 tenure-track faculty members who all would be evaluated using the new research requirements. in the absence of tenured faculty support, newer libraries faculty members sought guidance from each other and developed informal networks of research support. while a faculty mentoring committee was formed in 2019, most new libraries faculty members had already started in their roles by that time, and the program of mentoring to provide untenured faculty with structured career support, including with research, has yet to be fully implemented in 2021. in addition to the lack of internal experience with scientific research and large numbers of new libraries faculty, there was also an obstacle of pervasive imposter syndrome throughout the libraries3. though many had assisted, supported, and collaborated with other faculty at the university, libraries faculty did not see themselves as researchers, or even at times faculty members, at the same level as their institutional peers. this manifested in widespread self-doubt in trying new methods, planning large projects, and even talking about or promoting research projects internally. all of this together highlights the stark need behind the initiatives explored in this case study—to increase the comfort with and willingness to engage in scientific research among libraries employees by increasing the internal culture of research. librarians as researchers academic librarians are not unique in that they are a community of practitioners who have crossed the bridge into academia. as can be seen in a study conducted with nurses in the uk, this transition from professional practitioner to educator requires a blending of the two cultures and a supportive community of practice to foster the skills required by the academy4. unfortunately, academic librarians often find their preparedness and support for research to be lacking, creating a culture where research is often met with trepidation. many librarians have taken some sort of course or workshop on research methodology, with a good proportion completing this training as part of their mlis coursework5. however, participating in a research methods course or workshop is not a predictor of whether a professionally employed librarian will conduct research6. for most tenure-track librarians, research and scholarship are one of several required components for tenure, thus motivating them to engage in research of some sort. requirements about the number and type of research outputs produced varies. some institutions provide specific numbers, while others only measure by vague categorizations such as “consistency” or “progress.” requirements for type of authorship (single, first author, co-authored, etc.) also vary between institutions5. these unclear tenure expectations are commonly faced by new faculty members, and they are a significant source of stress7. 3 peer reviewed article hypothesis , vol. 34, no. 1, 2022 while types of publications valued when working towards tenure varies, in most institutions, the peer-reviewed article is valued the highest as librarians and administrators place a significant level of value on the peer review process5. since many early-career academic librarians lack practice or formal training in writing and publishing for peer review, this process can be a barrier for those seeking tenure. as found by sassen and wahl, “with little grounding in research methodology, statistical analysis, and scholarly writing, many academic librarians are not prepared to fulfill the research and publication requirements for promotion and tenure criteria”8. librarians, as practitioner–researchers, need a thorough understanding of the research life cycle in order to support user needs, engage with developing trends in scholarly communications, and support the library’s tie to institutional strategic visions9. confidence plays a role in a librarian’s perceived ability to engage in research; though librarian confidence in ability to conduct a literature review is high, confidence in other discrete stages of the research project is much lower10. many librarians feel like they have the training to read and understand library research articles. similarly, many feel they received enough training in their lis degree coursework to conduct original research. however, many felt their capabilities and confidence relating to research writing would be bolstered with additional training11. despite this need, the availability of formal research support, such as additional training, is lower for librarians than for those in other fields5. early career librarians in particular “have limited access to formal or institution-sponsored research support and rely heavily on informal mentoring”5. this lack of support can be detrimental to a librarian’s willingness and ability to conduct research while working in a professional capacity12. while formal support is a significant factor in an academic librarian’s research approach, one of the biggest challenges librarians face in making progress toward their research agenda is finding the time to conduct research5, 10, 13. time and time again, academic librarians cite the lack of room in their schedule as a reason why research does not happen. even for those who seek to engage more in the research process, either by reading disseminated products or conducting their own studies, the lack of work time to do so serves as a substantial barrier to doing so and is reflective of other trends in the profession such as salary compression (when there is little difference in pay between employees of different status, experience, and rank) and under compensation (when employees are not provided with pay and benefits commensurate with their rank, workload, and experience). despite these barriers, the amount of research being produced by lis practitioners is steadily growing14. academic librarians seeking tenure and those with a passion for research are contributing to the growing body of lis research being published in a number of journals in the field. with proper support and dedication to reduce barriers, lis research and scholarship can continue to grow, and a culture of research productivity can blossom among librarians. fostering a research culture supporting and sustaining a culture of research requires understanding the peer and library management support needs, meeting the visions of library administration, and 4 peer reviewed article hypothesis , vol. 34, no. 1, 2022 developing library-wide engagement among employees9. librarians are unique in higher education due to the difference in their educational background, which is at odds with the level of scholarship they are expected to produce once on the tenure track15. while previous research studies have found that there is no correlation between whether librarians conduct research and whether they believe their educational backgrounds have prepared them as researchers, anecdotal evidence suggests there is much more that graduate programs in library and information science can do to equip degreed academic librarians to conduct research with their academic peers in other disciplines12. many librarians find a benefit to informal mentoring surrounding research because the emotional support and nonjudgmental relationship fosters a sense of safety and comfort that carries over into their feelings about performing research. formal mentoring has also been shown to be beneficial as the more experienced mentors are able to work with mentees to specifically tailor their programs of research to their institution’s tenure and promotion processes5. as demonstrated in nursing, mentoring relationships between experienced researchers and junior faculty can be mutually beneficial, including increasing productivity, providing clarity in establishing hierarchical relationships within the research team, incorporating developing trends in research focus to existing projects, helping to fulfill requirements for both tenure and promotion, and enhancing the stability and reputation of the institution16. having multiple support systems in place for librarian researchers may prove to be the most beneficial option. both peer and mentor support can positively influence early-career librarians in their journey to conduct research5. in addition to formal mentoring, research indicates two other types of collaboration that can be beneficial to new researcher librarians: collaboration while managing the research process and collaboration while writing research manuscripts15. according to wilkinson, these two types of peer interaction “alleviates the burden on each individual librarian while also providing an opportunity for a more experienced librarian to model successful research and writing strategies for a newer colleague”15. beyond producing outputs, these efforts contribute to the emotional needs of new librarians through “provid[ing] reassurance that other new library faculty members are having the same experiences and anxieties”15. these supportive collaboration models help “bolster confidence and foster a sense of belonging” for new librarians and novice researchers15. as demonstrated throughout the research, as librarians progress in their research process, particularly with enhanced support networks, their confidence in their own ability to conduct research grows5, 10. benefits to institutions and the profession fostering a supportive and productive culture of research in academic libraries has direct benefits to the institution and the profession. as reported in perkins and slowik’s study of the value of librarian research as perceived by library administrators, librarians who conduct research add value to the library, university, and profession through research outputs that enhance library procedures and services, grant funding, recognition and awards, relationship building on and off campus, fulfillment of tenure requirements, and engagement with other research in the field17. establishing a culture of research in an academic library can lead to outcomes such as increased librarian 5 peer reviewed article hypothesis , vol. 34, no. 1, 2022 understanding of the research process and greater appreciation of disseminating lis research results for the benefit of the profession in addition to increased research output by institutional librarian authors9. for academic libraries, the impact of librarians as research practitioners has a positive correlation to the success of core library functions. researcher librarians improve their librarianship by having a more informed practice, which can also lead to improved services for library users11. research trends such as evidence-based librarianship has a large impact on the success of the academic library as evidence-based library research can directly lead to necessary changes to library practice18. the practice of conducting research improves librarians’ problem-solving and decision-making skills, which can lead to better in-the-moment responses to unexpected library issues13. training librarians in research methodologies allows librarians to teach graduate level methodology courses, mentor students, and engage with researchers as peers at their university, particularly in qualitative methods, expanding the scope and possibilities for partnership in the institution19. as it impacts universities, librarians as practicing researchers help to form connections between departments and colleges. there is ample opportunity for collaboration between practitioners and other academic peers, in particular as library science research is highly collaborative with co-authorship reported in a majority of published articles5, 14. these collaborations help improve the standing of the library and library–researcher in many ways, including by demonstrating a contribution to the mission of the institution”13. as it impacts librarian–faculty relations, borrego and pinfield demonstrate that “carrying out research in partnership with faculty offers librarians the opportunity to gain valuable experience in how research operates13. this knowledge may help them provide better library research support services. while research demonstrates the many personal, institutional, and profession-wide benefits of nurturing librarians as researchers, the process of creating and sustaining a research culture in an academic library requires forethought, planning, and flexibility, as demonstrated through the experience at clemson university libraries. experience: writing circle one of the first attempts at introducing a research culture to clemson libraries was a writing circle. librarians were invited to participate in the circle where each would author their own article. other members of the circle were to provide extensive editing and assistance with writing to the degree that they would be listed as co-authors on each article. the goal was to complete the articles in eight weeks, and a schedule was created to provide a concrete timeline. finding librarians who were eager to participate in the writing circle proved to be a challenge. some librarians were uncertain about the relatively short turnaround for writing. others did not have projects prepared to write about. still more were concerned about their work being of high enough quality for publication. after much targeted outreach and many one-on-one conversations, four librarians committed to 6 peer reviewed article hypothesis , vol. 34, no. 1, 2022 participating in the writing circle the circle itself did not meet its original goals. during the writing process, one member realized their data was incomplete and not ready to be written about. another encountered continuity problems with surveys conducted at different dates that made statistical analysis unrealistic for the skills of the group; however, a conference presentation about the data proved successful with all members of the writing circle listed as co-authors. a different member completed an article but was not confident enough in the manuscript to ever submit it to a journal. the final member had their article accepted and published in a refereed journal. the group as a whole completed a conference presentation about their process. the ultimate goal of four published articles by the circle was not realized, but the circle did complete two conference presentations and one published article, which was still progress by a group so new to the requirement of research. many of the shortcomings of the writing circle can be distilled down to moving too quickly into trying to produce research outcomes. more groundwork was needed to get the librarians to a place where they saw themselves as researchers. if librarians identified as researchers, they would likely have been more interested in participating in a writing circle. though the writing circle was designed to help build confidence and provide group support for the writing process, many librarians were not yet in a place where they had the confidence to jump directly into publishing. had more time been spent building up librarians’ skills as researchers and writers prior to the implementation of a writing circle, the writing circle may have seen greater success. journal club as casual discussions with coworkers and the experiences from the writing circle revealed, the librarians needed more familiarity with the current library research being done in the field. to address this in a low-stakes way that would facilitate discussion, journal club was framed as a casual meeting over lunch where participants (who could be any faculty or staff member) just showed up to discuss the reading as they ate. the goal was to eliminate as many participation barriers as possible; for example, staff frequently might not get time away from their service points other than lunch but might still be interested in attending journal club. the journal club was not organized by the library administration or leadership, so it was seen as a social gathering of equals rather than just another presentation or committee meeting. the club was described in an email invitation as functioning similarly to a book club, in which everyone reads the book ahead of time, but with scholarly journal articles related to librarianship. there was no singular “chair” of the group; instead, members took turns “hosting” by selecting the article and preparing a couple of open-ended discussion questions in advance of the biweekly meetings. everything was organized on an openly available google doc whose link was distributed library wide. hosts also frequently brought treats for the group such as donuts or pizza, which was a big draw when meetings were held at lunchtime. one person did serve as coordinator by owning the google doc and scheduling meeting rooms, but these administrative tasks were often also handled by other journal club attendees. 7 peer reviewed article hypothesis , vol. 34, no. 1, 2022 soon, a core group of dedicated journal club attendees emerged. these attendees regularly read the articles and talked critically about the methods, data analysis, and writing of the selections. by doing so, these librarians became more confident in their abilities to judge research in their own field. many signed up to be peer reviewers for journals that matched their research interests. although initial journal club article selections focused on specific areas of librarianship, the group eventually began to shift to broader topics that transcended service areas such as work–life balance and social justice in librarianship. the group has even spawned a research group that is investigating publishing practices in lis journals. overall, the journal club has been successful in achieving its goals of getting people in clemson libraries to talk more openly about the process of doing and publishing research. however, the journal club saw dramatically decreased attendance when it was moved to a virtual format in march 2020 due to the covid-19 pandemic. due to this change, the journal club is currently on hiatus until the majority of library personnel have returned to working in the library building again. research roundtable following the success of journal club, the need emerged to bridge the gap between discussing research and supporting the research projects of attendees. recognizing this need, the research roundtable was formed as a monthly informal meeting to gather and discuss research. much like journal club, research roundtable sought to be accessible to all practitioners of research in the libraries with open meeting invitations, snacks, and loosely structured conversations. unlike the more frequent journal club meetings, research roundtable is a monthly meeting that gives participants time to reach their goals and report progress between meetings. in the earliest research roundtable gatherings, the meetings had a central theme, such as setting a research agenda, and the rotating host would provide information on the theme before leading a discussion on the topic. meeting norms were established early as a way to provide a safe space for attendees to speak honestly about their experiences, frustrations, and successes in their projects and were reviewed at the start of each meeting. attendees were welcome to participate in meetings as they desired, but a core group of participants were present monthly. while the themed meetings were successful, they did not provide the intended space for participants to dive deeper into their interests and projects. a new structure for research roundtable meetings was established in which every month featured a research captain (rc), who would partake in a series of interview questions regarding their current projects, frustrations, successes, and goals. to foster collegiality, an additional question was added in which the rc could share something that was currently bringing them joy, which could range from a television show, podcast, new snack, or other personal detail. this structure was successful in giving attendees the space to share their projects, learn from others, and provide accountability in moving forward in individual projects. while there are still special event-themed research roundtable events, often in collaboration with other committees or groups, the structured interview rc approach has been the most successful method to provide a 8 peer reviewed article hypothesis , vol. 34, no. 1, 2022 supportive and reflective space to foster research initiatives in the libraries. discussion: as experiences such as journal club and research roundtable have taken hold, the culture surrounding research in the libraries has begun to shift. most notably, librarians seem more comfortable talking about library research. prior to the establishment of journal club and research roundtable, there was hesitancy to state out loud any apprehension, confusion, or discomfort surrounding research, particularly since it became one of the areas on which librarians are evaluated. by providing a safe space to explore what research looks like, share experiences, and grow as a group, barriers to having honest conversations about scholarship have been broken down. in turn, this has led to increased willingness to participate in research because librarians have become more comfortable trying new methods and projects without fear of performing a study in a vacuum. having the experience to critique what has already been published has illustrated that not every project has to be perfect to be complete. taking time to dive deep into the literature published in journals that define the profession has reduced the intimidation felt by librarians seeking to publish as they realize they are capable of producing work of similar quality. seeing colleagues experience similar struggles and setbacks in their research has made it easier to talk about concerns with one’s own projects. research and writing has become a more approachable endeavor within the library, and colleagues are more willing to share both the highs and the lows of what they are working on. in addition to improved perception of research as a process within the libraries, collaborative research projects have increased in number. having a space to talk about research and share ideas has led to groups that have spun off from journal club or research roundtable to take on projects. people are able to connect with others who have similar research interests, where before it was more of a challenge to know what colleagues in other units were interested in researching. these collaborative groups have already produced several conference presentations, and manuscripts for articles are underway. takeaways: while the shift in culture surrounding research at the library has come a long way, there is still plenty of room to grow. one of the most notable areas for growth is among tenured faculty, who view the training, group learning, and sharing sessions as systems in place for their pre-tenure colleagues only. a large contingent of these tenured faculty members received tenure under the old faculty guidelines, which did not require research. as such, their research agendas take a different shape than those who were hired under the new guidelines. a barrier to effectively reaching this group of librarians is their reluctance to admit they have something to learn about research. the growing pains associated with new guidelines and a large cohort of pre-tenure faculty can make it uncomfortable for tenured librarians to admit they may not have the same or more knowledge and experience their new colleagues have. in order for the 9 peer reviewed article hypothesis , vol. 34, no. 1, 2022 libraries’ research culture to reach its fullest potential, this discomfort needs to be assuaged, and tenured librarians will need to join the conversations surrounding library research. the efforts described in this article (forming a writing circle, journal club, and research roundtable) were open to all faculty regardless of tenure status, but tenured librarians rarely participated. future efforts may target them more specifically; for example, the department chair has discussed requiring a “research agenda” as part of the regular annual activity review. future considerations given the opportunity, many changes could continue to advance the research culture at clemson libraries. first, the authors could get approval from their institutional review board (irb) to conduct a research project with human subjects; this would allow for a study to be done on the research agendas and research culture at clemson libraries directly. with some form of data to back up the findings, it would be easier to create a cohesive plan to continue to encourage clemson librarians to conduct research. second, one change going forward could be formal recognition of these efforts within the library. to date, activities like journal club and research roundtable are grassroots efforts spearheaded by untenured faculty. if these activities were adopted more widely and supported by the libraries’ administration, not only would engaging tenured faculty be easier but also leaders of these activities could receive recognition for their overlooked labor. although not discussed at length here, one weakness that faculty librarians have faced in the past is the perception of how their research compares to that done by other university faculty. along with formal recognition from library administration, promoting the research of library faculty at the same level as that of other university faculty helps demonstrate the legitimacy of librarian research and ideally situates library faculty alongside their institutional peers. author contributions conceptualization: m.a., j.m.m., k.m.s.; project administration: m.a., j.m.m., k.m.s.; writing – original draft: m.a., j.m.m., k.m.s.; writing – review and editing: m.a., j.m.m., k.m.s. references 1. cw staff. clemson moves into top tier of research universities [internet]. clemson world. 2013 may 1 [cited 31 aug 2021]. available from: https://clemson.world/clemson-moves-into-top-tier-of-research-universities/ 2. clemsonforward strategic plan [internet]. [updated mar 2019; cited 31 aug 2021]. available from: https://www.clemson.edu/provost/documents/clemsonforwardplan-updated32019.pdf 3. clark m, vardeman k, barba s. perceived inadequacy: a study of the imposter 10 https://clemson.world/clemson-moves-into-top-tier-of-research-universities/ https://www.clemson.edu/provost/documents/clemsonforwardplan-updated32019.pdf peer reviewed article hypothesis , vol. 34, no. 1, 2022 phenomenon among college and research librarians. coll res libr [internet]. 2014 may [cited 31 aug 2021];75(3):255-71. doi:10.5860/crl12-423 4. andrew n, ferguson d, wilkie g, corcoran t, simpson l. (2009). developing professional identity in nursing academics: the role of communities of practice. nurse educ today [internet]. 2009 aug [cited 31 aug 2021];29(6):607-11. doi:10.1016/j.nedt.2009.01.012 5. ackerman e, hunter j, wilkinson zt. the availability and effectiveness of research supports for early career academic librarians. j acad librariansh [internet]. 2018 sept [cited 31 aug 2021];44(5):553-68. doi:10.1016/j.acalib.2018.06.001 6. luo l. fusing research into practice: the role of research methods education. libr inf sci res [internet]. 2011 jul [cited 31 aug 2021];33(3):191-201. doi:10.1016/j.lisr.2010.12.001 7. eddy pl, gaston-gayles jl. new faculty on the block: issues of stress and support. j hum behav soc enviro [internet]. 2008 oct [cited 31 aug 2021];17(1-2):89-106 doi:10.1080/10911350802168878 8. sassen c, wahl d. fostering research and publication in academic libraries. coll res libr [internet]. 2014 jul [cited 31 aug 2021];75(4):458-91. doi:10.5860/crl.75.4.458 9. hall lw. (2018). from practice to research at flinders university library: sustaining a research culture. libr manag [internet]. 2015 [cited 31 aug 2021];39(8-9):615-24. doi:10.1108/lm-10-2017-0110 10. kennedy mr, brancolini, kr. academic librarian research: a survey of attitudes, involvement, and perceived capabilities. coll res libr [internet]. 2012 sept [cited 31 aug 2021];73(5):431-48. doi:10.5860/crl-276 11. apolinario rru, eclevia mr, eclevia jr. cl, lagrama erc, sagun kka. librarian as researcher and knowledge creator: examining librarian’s research involvement, perceived capabilities and confidence. paper presented at: ifla wlic 2014. proceedings of the ifla annual congress; 2014 aug 16-22; lyon, france. available from: http://library.ifla.org/id/eprint/961 12. kennedy mr, brancolini, kr. academic librarian research: an update to a survey of attitudes, involvement, and perceived capabilities. coll res libr [internet]. 2018 sept [cited 31 aug 2021];79(6):882-951 doi:10.5860/crl.79.6.822 13. borrego á, pinfield s. librarians publishing in partnership with other researchers: roles, motivations, benefits, and challenges. portal [internet]. 2020 oct [cited 31 aug 2021];20(4):655-75. doi:10.1353/pla.2020.0031 14. aytac s, slutsky b. published librarian research, 2008 through 2012: analyses and perspectives. collab librariansh [internet]. 2014 [cited 31 aug 2021];6(4):6. available from: https://digitalcommons.du.edu/collaborativelibrarianship/vol6/iss4/6/ 15. wilkinson z. rock around the (tenure) clock: research strategies for new academic librarians. new librar world. 2013 [cited 31 aug 2021];114(1/2):54-66. doi:10.1108/03074801311291965 16. morrison-beedy d, aronowitz t, dyne, j, mkandawire l. (2001). mentoring students and junior faculty in faculty research: a win-win scenario. j prof nurs [internet]. 2001 nov-dec [cited 31 aug 2021];17(6):291-6. doi:10.1053/jpnu.2001.28184 17. perkins gh, slowik aj. the value of research in academic libraries. coll res libr [internet]. 2013 mar [cited 31 aug 2021];74(2):143-58. doi:10.5860/crl-308 18. booth a. bridging the research-practice gap? the role of evidence based librarianship. new rev inf libr res [internet]. 2003 jul [cited 31 aug 2021];9(1):3-23. doi:10.1080/13614550410001687909 11 https://doi.org/10.5860/crl12-423 https://doi.org/10.1016/j.nedt.2009.01.012 https://doi.org/10.1016/j.acalib.2018.06.001 https://doi.org/10.1016/j.lisr.2010.12.001 https://doi.org/10.1080/10911350802168878 https://doi.org/10.5860/crl.75.4.458 https://doi.org/10.1108/lm-10-2017-0110 https://doi.org/10.5860/crl-276 http://library.ifla.org/id/eprint/961 https://doi.org/10.5860/crl.79.6.822 https://doi.org/10.1353/pla.2020.0031 https://digitalcommons.du.edu/collaborativelibrarianship/vol6/iss4/6/ https://doi.org/10.1108/03074801311291965 https://doi.org/10.1053/jpnu.2001.28184 https://doi.org/10.5860/crl-308 https://doi.org/10.1080/13614550410001687909 peer reviewed article hypothesis , vol. 34, no. 1, 2022 19. hansman ca. training librarians as qualitative researchers: developing skills and knowledge. ref librar [internet]. 2015 oct [cited 31 aug 2021];56(4):274-94. doi:10.1080/02763877.2015.1057683 12 https://doi.org/10.1080/02763877.2015.1057683 rising star research projects 2016-2017 stevens, nguyen, lerner, and jo hypothesis, vol. 29, no. 2, fall/winter 2017 8 rising stars research projects 2016-2017: action research to improve mla’s communities gregg a. stevens, msls, mst, ahip health sciences librarian health sciences library stony brook university stony brook, ny gregg.stevens@stonybrook.edu tony nguyen, mlis, ahip technology and communications coordinator national network of libraries of medicine, southeastern/atlantic region health sciences and human services library university of maryland, baltimore baltimore, md ttnguyen@hshsl.umaryland.edu rachel c. lerner, msls, ahip research and instruction librarian edward & barbara netter library quinnipiac university hamden, ct rachel.lerner@quinnipiac.edu phill jo, mlis assistant professor/ head of access services robert m. bird health sciences library university of oklahoma health sciences center oklahoma city, ok phill-jo@ouhsc.edu introduction started in 2010, the medical library association’s (mla) rising stars program has provided early career librarians with valuable leadership training. the 2016-2017 cohort of rising stars had a unique opportunity to experience a significant update of the program that included rising star research projects 2016-2017 stevens, nguyen, lerner, and jo hypothesis, vol. 29, no. 2, fall/winter 2017 9 projects based on one initiative, mla’s sections and special interest groups (sigs), as mla notes that less than half of mla members belong to a section or sig. by concentrating the rising stars’ efforts in one general area, the project results were designed to inform larger discussions within mla on the role of sections and sigs and supplement the work of the communities task force. the heart of the rising stars program is the mla project [1]. the purpose of the 2016-2017 projects was to add value to mla and provide real impact to the organization. projects were presented to the cohort as a form of action research, a disciplined inquiry that allows one to examine their practices, learn from them, and take action to effect positive change within the context of their own environment [2]. the cohort was provided a template to help organize the elements of the proposed projects, from the rationale and background information to the implementation and assessment. the intent of the completed projects was to provide mla leadership and the communities task force with actionable recommendations to support the changing needs of its members. the projects and the efforts of the task force ultimately help mla strengthen its communities, which is part of the organization’s three year rolling strategic plan. the four rising stars presented synopses of their projects in may 2017 to the section council at the mla annual meeting in seattle, wa. following the presentations, the rising stars participated in breakout groups with the section council representatives to discuss how mla sections and sigs could be improved to meet the needs of members. this article summarizes each of the individual rising stars projects, highlighting not only their work but also illustrating how these projects could potentially lead to more effective and meaningful mla communities. section council (sc) is comprised of 2 members of each section in mla generally the section chair and the past section chair. with the current 21 sections, sc is comprised of at least 46 members at any given point, including leadership positions. this research question could have taken a number of directions; however, it hinged on the stated purpose of sc: assess the structure, procedures, and operation of the mla section council and propose changes that better serve the council purpose and enable it to determine if it is successful in realizing its purpose rachel c. lerner rising star research projects 2016-2017 stevens, nguyen, lerner, and jo hypothesis, vol. 29, no. 2, fall/winter 2017 10 “the section council of the medical library association provides an opportunity for sections to participate more directly in the governance of the association by serving in an advisory capacity to the board of directors of the association. the section council also promotes interchange between sections and, with its counterpart, chapter council, between sections and chapters.” [3] as one of the main purposes of sc is advocacy for and governance of the sections, lerner began her project by reviewing all of the documentation surrounding sc activities in an attempt to discover what success might look like. after this period of review, lerner focused on one aspect of success: using data to advocate and facilitate informed decisions. while individual sections should not be directly compared with one another, there is value in understanding their longterm trajectories and efforts using a standardized set of data points. synthesizing this information would aid in advocacy efforts for more money, time, etc. as well as help sc advise the sections with up-to-date, easily accessible information. additionally, because the sections and sc rotate governance, centralized data collection and storage would facilitate smoother transitions for new leadership. objectives and methods the research plan for this project was threefold: lerner conducted a small qualitative survey of 10 years of past sc leadership (n=4) followed by telephone interviews of key mla stakeholders. additionally, she conducted a retrospective analysis and annotation of three years of mla board and sc meeting minutes; and, finally, reviewed and parsed the activity data from four years of section annual reports. results the survey, which had a 75% (n=3) response rate, revealed the following: sc would like the ability to track section membership and activity over time; they would like the learning curve for sc positions to be less impactful to the operations of sc; and they would like increased independence in decision-making processes. an interview with the then sc chair followed, along with informational calls to the sc webmaster and the mla director of membership and information services. these interviews helped to identify the information available for collection as well as the types of information that might be useful to sc. information readily available included: section membership numbers, operating budget, number of online discussion threads, and total amount of money raised. information that would be useful to rising star research projects 2016-2017 stevens, nguyen, lerner, and jo hypothesis, vol. 29, no. 2, fall/winter 2017 11 collect and centralized included: number of awards given and to whom, number of special content sessions contributed to mla annual conferences, number of nominations for chair and other positions, number of votes cast in elections, and number of mla news items and full speed ahead items contributed. some of this information is currently collected in the annual reports, but it is not centralized in one location. one would have to go to each report to gather the information. a retrospective analysis and annotation of board and sc meeting minutes revealed the type of information reported to sc, and in turn, what was reported to the mla board. additionally, this study revealed what decisions were handled internally within sc, and which were brought to the mla board. often, the board would ask for more information or data before coming to a consensus. finally, lerner compiled and systematically reviewed the section, sc, and mla annual reports in order to identify the confluences and dissonances of year-end reporting data and formats, and to gather ideas for standardized metrics. she discovered that information was being reported inconsistently and in a variety of formats, without a centralized repository for all data. recommendations in consideration of all information gathered and analyzed, lerner proposed that sc create a “metrics dashboard” as well as a standardized metrics collection form. lerner created a draft version of the dashboard for sc, which would be transparent to the membership and contain all data collected by sections and sc in the annual reports. this dashboard would be treated as a living document, with yearly review of metrics to verify or determine continued relevancy. considerable input and buy-in from sc would be required in order to implement a final version, including a brainstorming session wherein sc members would discuss specific metrics based on usefulness to both their sections and the council as a whole. considerations would include selection of metrics to help leaders make decisions; those to share with the mla board; and those that could further demonstrate impactful execution of goals set forth in annual reports. with the exception of time costs and the expense of providing a location at an annual meeting for a brainstorming session, this implementation would be financially null. a potential quagmire might result from an inclination of sections to compare themselves through quantitative metrics. this should be discouraged early and often, as each section is unique in their operating costs, membership, and deliverables. it should be communicated that decisions are never made on numbers alone they are just one point in a larger conversation. implementing a rising star research projects 2016-2017 stevens, nguyen, lerner, and jo hypothesis, vol. 29, no. 2, fall/winter 2017 12 standardized metrics form as an addendum to the current annual report template would be a fairly straightforward procedure. this form would not replace the current template; rather, it would augment what is currently used. this form would rely heavily on what is decided regarding the metrics dashboard, as all of the collected data would be included in the dashboard. the purpose of sc is to advocate to the mla board of directors on behalf of sections, and to promote and serve the projects of the sections. this project presents an opportunity for sc and sections to enhance both their advocacy and decision-making through judicious use of data collected. some, but not all, of this data is available through the mla headquarters. requiring each section to complete their forms and then adding that information to a central repository will reduce redundant data collection efforts and allow for re-use by other interested parties. finally, section members pay to be involved, and a more in-depth reporting of section activities as well as an increase in the transparency of those activities can demonstrate the value of their membership dues. at the time projects were assigned, mla accepted a long-term strategic action plan in high priority areas due to the significant and meaningful impact to deliver on the mission of the organization. the 21 mla sections were tasked to develop one to three multi-year goals and two to four strategies to achieve each of these goals to support the mission of mla and delivery of the organization’s strategic plan. mla provided sections a template to utilize to assist with the creation of their goals. objectives nguyen focused his research on developing a process that simplifies methods for sections to create strategic goals, improving their relevance to the mla membership. through these changes, sections could increase activity within their membership, recruit new members to support the strategic goals, allow for cross-collaboration between sections, and provide increased value to mla members. assess the relevance of the current (summer 2016) strategic goals of sections to mla members and the health information profession and propose an initiative to improve relevance tony nguyen rising star research projects 2016-2017 stevens, nguyen, lerner, and jo hypothesis, vol. 29, no. 2, fall/winter 2017 13 methods nguyen spent significant time reviewing the template provided by mla to its sections. after contacting each of the section leaders for their current strategic goals, he received 13 sets of goals from sections (approximately 62%). after reviewing what was received, he identified that each section utilized different formats in their strategic goals. at the time of review, only 2 of the strategic plans followed the mla current strategic plan structure (10% of all sections). in addition to reviewing all of the strategic plans, annual and mid-year reports from all sections submitted since 2014 were analyzed in order to identify goals and strategies, and the sections’ public pages on the mla website were explored to identify how they promoted their programs to mla members. results upon review of strategic plans, annual, and mid-year reports, nguyen identified similar interests, strategies, and goals from each section. unfortunately, sections may be unaware of these similarities as they might not read these reports, creating a duplication of effort that could be solved through cross-sectional collaboration. reviewing section’s public pages provided insight in how mla members may perceive the activities and benefits of spending additional money to join. of concern is that section public pages do not showcase the benefit of spending additional fees on top of their mla membership to join a specific section aside from access to a member directory, forum, and possibly a newsletter. members searching for value for their dollars may hesitate to spend additional money to join a section without knowledge of the section’s activities that could be identified within the section’s strategic plan. recommendations in light of these discoveries, nguyen proposed that sections might align their strategic plans more easily with mla’s strategic plan and submit goals in a more unified manner by utilizing an automated system. an automated system would help sections add their goals, strategies, and objectives more easily, assign point of contacts for each activity associated, and give section council and mla an opportunity to check up on and discuss the progress of specific activities. to assist with searchability, nguyen suggests including options that refer back to a specific mla goal the strategy would support. with updates and revisions planned for the mla competencies for lifelong learning and professional success, referring to these competencies rising star research projects 2016-2017 stevens, nguyen, lerner, and jo hypothesis, vol. 29, no. 2, fall/winter 2017 14 may help showcase how the strategies would support the professional development of mla members. with an automated system, the information will become searchable, increasing benefit to its membership. this would allow section leaders and members to search and identify other sections working on projects similar in nature to discuss collaboration. finally, in order to help promote sections and attract members, goals, strategies, and objectives made visible on the sections’ public pages would support increased membership and attract volunteers to help sections achieve their goals by promoting current projects that may interest new members. in consideration of the significant endeavor it would take to develop a database as described, potential challenges were considered that could be associated with implementation of the project plan. the first risk identified is the steep learning curve section leaders may have in developing strategic goals aligned to mla in general. as many members of the association may not be familiar with writing strategic goals, they may have difficulty developing strategic goals and may require training in order to write them. training would need to occur regarding strategic goals, utilizing an input form, and utilizing a searchable database which may take significant time to implement. there may be projects that sections do not want publicly available; for this, there would be an opt-out option. for a goal to be attractive to all members within the organization, it may be important to have goals and strategies that are visible only to section members. these goals may be available in section portals only available to members. the project plan outlined provides an opportunity for mla sections to be more closely aligned with the organization’s strategic plan. additionally, utilizing a database to identify what other sections are doing may help them learn from what others have accomplished or collaborate on projects with other sections. by connecting section goals to the public page, members will have an opportunity to understand what projects sections are working on which may improve membership and volunteerism within the group. the purpose of mla sections and sigs is to represent subject expertise in certain topic areas and to build communities of members based on their interests and specialties under the umbrella of mla. individual members can join sections and sigs based on types of libraries, assess how well the topical coverage of mla sections and sigs address the needs of mla’s primary audiences and propose an initiative to improve coverage phill jo rising star research projects 2016-2017 stevens, nguyen, lerner, and jo hypothesis, vol. 29, no. 2, fall/winter 2017 15 specific functions, and various purposes. members can join multiple sigs with no extra fees, and can join sections by paying annual fees ranging from $10.00 to $20.00. mla members can share and develop their knowledge and educational experiences by participating in sections and sigs, networking, and gaining leadership experience at the national level. according to a membership data analysis for sections and sigs, only a small portion of mla members participate in sections and sigs. as of september 2016, there were 21 sections and 25 sigs. membership within sections varied, ranging from 1% to 18% of the total of mla memberships. despite free membership, sigs ranged from 0.4% to 7% of the total mla memberships. even though mla offers sections and sigs on various topics and subject areas, less than 20% of mla members join them. objectives and methods based on the analysis of membership data, jo sought to assess the relevance of topical coverage of sections and sigs to mla’s primary audience. in addition to a membership participation analysis, jo conducted a short six question survey that was sent to section leaders such as the chair, chair-elect, past-chair, secretary, and sig conveners to assess the relevance of topic coverage and member satisfaction. the survey questions were sent to a total of 122 mla section and sig leaders over a 2 week period in december 2016. results based on the analysis, the highest participation rate of a section was in the hospital libraries section (n= 587) representing 18% of total mla members (n=3269). the clinical librarians and evidence-based healthcare sig had the most members (n=206), representing 6.3% of the total mla membership (n=3269). the survey response rate was 20% (n=25) for sections and 8% (n=10) for sigs. according to the survey results, 76% (n=19) of the section leaders were satisfied with current topical coverage and its relevance to mla; 80% (n=8) of sig conveners answered that they were satisfied with the topical coverage. respondents commented that topics were very similar between sections and sigs. they also mentioned that topics such as outreach, scholarly communication, research impact, emerging technologies, and instructional design need to be included in sections or sigs. the leaders consider joining a section or sig beneficial for individual members as it provides educational and funding opportunities, information sharing, networking, leadership, and rising star research projects 2016-2017 stevens, nguyen, lerner, and jo hypothesis, vol. 29, no. 2, fall/winter 2017 16 professional development, while gaining knowledge on specific topics. respondents shared potential barriers to joining sections and sigs as financial burdens, indirect benefits, lack of time, and being unaware of sections and sigs. respondents expressed that promoting the benefits and activities of sections and sigs is critical. while they mentioned that topical coverage is not a major issue, some commented that topics are overly broad, redundant, and mostly academic or institution-specific. one respondent said that some sections and sigs might consider merging since they cover similar topics. another commented that “there are too many sections and many sigs could be covered by sections.” results also indicated that sections and sigs should focus on enhancing their current membership. recommendations based on the survey results and review of sections and sigs from the mla website, jo recommends making sections and sigs, along with their activities and accomplishments, more visible to mla members through an improved presence on the mla website. mla members should be able to easily access the goals and benefits of sections and sigs. virtual meetups and social media can be used widely for different purposes as well. it is also recommended that mla host promotional sessions where sections and sigs can share their programs and activities and to have social events at annual meetings in order to increase their membership. additionally, a suggestion is made to offer specific internal benefits to section and sig members such as providing free continuing education courses if hosted by a section or sig they are a part of, and giving members paper and poster presentation opportunities at annual meetings in relation to their section and sig. it is recommended that mla conduct more detailed needs assessments and membership satisfaction surveys regularly to reflect the specific interests and needs of its members. in addition, mla could provide discounted membership fees for those who are active and join multiple sections; leaders of sections and sigs covering similar or overlapping topics could discuss either merging or collaborating with each other to develop activities of interest. these recommendations come with limitations, as other factors are involved in the relevancy of sections and sigs. even though many dedicated members consider sections and sigs as their “home” within mla, many struggle with recruiting new members and maintaining engagement with current members. active section and sig members have multiple roles in their own group rising star research projects 2016-2017 stevens, nguyen, lerner, and jo hypothesis, vol. 29, no. 2, fall/winter 2017 17 and find it time-consuming to improve the relevance of their community to mla members. analyzing results from the survey indicates that sections and sigs are important as they bring a sense of community and belonging for members at the national level. they are a place to share information by asking and answering questions from colleagues interested in a similar topic. as the project discovered, it is important to examine the values and benefits of sections and sigs and to continually communicate them to mla members. mla sigs, as more informal bodies with no membership requirements or annual dues, are created to support specialty areas in librarianship. because of their ephemeral nature and the changing popularity of some niche specialties, some of the groups have faced irregular levels of membership and participation in recent years. objectives in his research on this assigned topic, stevens focused on how sigs could increase membership in order to make participation more valuable to all members and create more meaningful member experiences. sigs might then be able to retain and add to their membership, increase activity, and ultimately provide greater value to mla members. methods in consultation with his project advisor, stevens created a qualitative survey with five openended questions addressing sig membership and activity. the survey was sent to the 38 sig conveners, based on mla records, in october 2016. of the 38 conveners, ten responded to the survey (26.3%). at the time of the survey, 25 sigs were in existence, accounting for 40% of the group responses. the responding sigs included groups with membership levels ranging from 12 to 211 members (mean of 74.8) as of december 1, 2016. group topical coverage for this cohort included groups formed around medical specialties (chiropractic, pediatric, vision science, department of veterans affairs), facets of health science librarianship (clinical librarians and evidence-based healthcare, libraries in curriculum, outreach and marketing, resource sharing), and social issues (health disparities, lgbtq). upon initial analysis of the results, assess sig membership requirements and guidelines and propose new requirements and guidelines gregg a. stevens rising star research projects 2016-2017 stevens, nguyen, lerner, and jo hypothesis, vol. 29, no. 2, fall/winter 2017 18 stevens contacted three conveners to arrange for follow-up interviews. only one was available, and the follow-up interview was conducted by phone in december 2016. results analysis of the responses pointed to three main concerns. six of the ten respondents (60%) expressed some concern over membership levels that were either flat or declining and many of the respondents indicated that they were unaware of any active recruitment or promotion to increase membership in their groups. the convener of one group stated that the group’s membership declined by about 75% after mla switched to the socious platform in 2015, which blocked sig participants who were not active mla members. secondly, the majority of the sigs reported that they are not active outside of the annual meeting (60% of respondents), so the value of these groups to its members is limited. finally, half of the respondents (50%) mentioned the potential for greater participation in the groups through the use of technology. one idea mentioned by many respondents was to hold online meetings, which would allow members unable to attend the mla annual meeting to participate in the group’s activities virtually. recommendations based on these responses, stevens created a two-fold pilot project proposal to address both the low membership levels and participation. the first part of the proposal involves the creation of a new basic level of mla membership, which would allow for access to sig resources but very few other mla resources. this would allow for non-mla members to access sig resources and participate in sig activities. however, this level of membership would not provide any of the other benefits of mla membership. this low level of membership might appeal to those interested in sig topics but who would not join mla due to cost. the “sig only” level of membership as proposed would have nominal annual dues of $20. this amount is comparable to the dues charged to join equivalent groups in mla’s peer organizations (american library association and special libraries association) and the funds would help to cover administrative costs. ultimately, the goal would be to convert many of the “sig only” memberships to full mla memberships, after “sig only” members learn the value of the content and experiences provided by mla membership. stevens proposed that the “sig only” membership dues could be applied to full mla membership. rising star research projects 2016-2017 stevens, nguyen, lerner, and jo hypothesis, vol. 29, no. 2, fall/winter 2017 19 the second part of the proposed pilot project involves encouraging the sigs to hold more activities online to address the survey concern that many sigs only hold a business meeting at the mla annual meeting but are inactive for the rest of the year. if a sig holds its annual business meeting online, prior to the mla annual meeting, it allows people to participate who are unable to travel to the meeting, allowing for greater participation. it provides the sig an opportunity to use its in-person meeting time for alternate activities related to its topic, such as round table discussions or guest speakers. this would mimic the blended or hybrid format used in academia, combining in-person and virtual elements. it also addresses two of the key challenges mentioned in the survey: low participation in the sigs throughout the year and the potential use of technology to make for more meaningful member experiences. this shift to online content has recently been tested by some sections and sigs. for example, the nursing and allied health resources section (nahrs) held its 2017 business meeting online prior to the mla annual meeting, and the outreach and marketing sig held its first virtual meeting in july 2017. with time, this could become a normal practice for mla communities. conclusion the concept of “action research” was foreign to the 2016-2017 rising stars. with the understanding of mla’s concept of this disciplined inquiry, the projects gave the cohort the opportunity to understand the inner workings of mla sections and sigs. through their assigned projects, the group members developed their research and problem-solving skills as they contributed toward mla’s strategic goal to strengthen member communities. however, there were challenges with the project implementation that could have improved and enhanced the cohort’s experience. after the group worked on their projects independently for several months, the participants shared their results in a class session, only to discover that each of their projects overlapped significantly. in retrospect, the cohort agreed that if they worked collectively as a team, the projects would have been more streamlined and possibly have reduced a duplication of effort. by working as a team, the cohort believed that they may have suggested different actionable items in their project proposals to mla and provided positive change to the organization. additionally, working on a group project would provide a better experience for the leadership program, because successful organizational projects are rarely conducted by one person in a bubble but rather a team. the cohort recommends that future rising star cohorts conduct group projects in lieu of individual ones. rising star research projects 2016-2017 stevens, nguyen, lerner, and jo hypothesis, vol. 29, no. 2, fall/winter 2017 20 an additional challenge to the research projects was the difficulty in obtaining mla data and records. archived mla documents are housed at the national library of medicine (nlm) and unavailable in a digital format. the only way to access this information is to visit nlm and conduct research at their facility. newer mla records are digital, but the group faced some difficulty locating information and at times relied on the collective knowledge of past members and leaders. a more streamlined, transparent, and extensive data management system would have improved the process to locate and utilize the background material necessary to conduct the research projects. despite these challenges, the individual projects provided the four rising stars significant opportunity to learn new skills, such as writing strategic goals, conducting applied research, and creating a pilot project plan. additionally, the action research projects provided deep insight into the inner workings of mla, section council, sections, and sigs which would not have been learned by the cohort otherwise. in creating their project proposals, each member utilized their strengths to develop actionable items based on their own research findings. in the end, the group believes that the project findings and ideas presented to both section council and the communities task force can strengthen mla’s sections and sigs and make them more valuable to the membership. references 1. medical library association. mla rising stars [internet]. chicago: medical library association; 2017. [cited 2017 aug 1]. available from: http://www.mlanet.org/p/cm/ld/fid=44. 2. crothers k. action research. in: research starters: education (online edition) [internet]. ipswich (ma): great neck publishing; 2015. [cited 2017 aug 1]. available from: ebscohost: http://search.ebscohost.com/login.aspx?direct=true&db=ers&an=89164051&site=edslive&scope=site. 3. medical library association. section council procedure manual [internet]. chicago: medical library association; 2017. [cited 2017 aug 21]. available from: http://www.mlanet.org/p/do/sd/sid=871&fid=1658&req=pdfdirect. -- http://www.mlanet.org/p/cm/ld/fid=44 http://search.ebscohost.com/login.aspx?direct=true&db=ers&an=89164051&site=eds-live&scope=site http://search.ebscohost.com/login.aspx?direct=true&db=ers&an=89164051&site=eds-live&scope=site http://www.mlanet.org/p/do/sd/sid=871&fid=1658&req=pdfdirect art show paints a collaborative picture stark, paris, rogan, finnegan dopkins, and etter hypothesis, vol. 29, no. 2, fall/winter 2017 21 art show paints a collaborative picture: increasing engagement in a pharmacy and health sciences library rachel stark, ms, ahip health sciences librarian, senior assistant librarian sacramento state university sacramento, ca stark@sacstate.edu mickel paris, mlis health sciences librarian, assistant professor rite aid information commons university of the pacific stockton, ca mparis@pacific.edu ed rogan, pharmd assistant professor, thomas j . long school of pharmacy and health sciences university of the pacific stockton, ca erogan@pacific.edu mary-kate finnegan dopkins, mlis library services specialist rite aid information commons university of the pacific stockton, ca kdopkins@pacific.edu cassie etter, mts library services specialist rite aid information commons university of the pacific stockton, ca cetter@pacific.edu art show paints a collaborative picture stark, paris, rogan, finnegan dopkins, and etter hypothesis, vol. 29, no. 2, fall/winter 2017 22 abstract objective: to increase student, staff, and faculty engagement with the health sciences library. methods: faculty, staff, and students were encouraged to create pharmacy-themed artworks and submit them to the library for entry into an art show. a two-hour art show and gala were planned for library patrons to view art pieces submitted by faculty, staff, and students. participation data was collected during the event and compared to previous programming held at the health sciences library. results: the pharmart program determined a metric of demonstrable success would show as an uptick in participation; both physical and digital, when compared against previous library events. physical participation was measured through program attendance and art piece submission. participating demographics were represented by all target populations of faculty, staff, and students with a total increase of 52 participants (347% increase) over the previous event. engagement through social media showed a considerable increase in clicks (2224%), shares (1200%), and overall reach (24%) within the school community. conclusions: incorporating student, staff, and faculty art works into a library-hosted event increased participation in a library program compared to previous library programming. attendance and social media engagement increased considerably. faculty, librarians, and students were brought together to share a common creative interest and to promote a casual atmosphere in which cross-curricular ideas could be discussed among attendees. introduction the rite aid information commons (raic) is the health sciences library serving the university of the pacific thomas j. long school of pharmacy and health sciences. the school supports the academic success of students studying pharmacy, audiology, physical therapy, and speech language pathology. library events at the raic, such as the better hearing & speech month (bhsm) kickoff party, have traditionally had poor attendance, even with social media marketing and incentives such as raffle prizes. due to the lack of engagement, many event goals were unmet (for example, providing important information regarding diseases and other healthrelated subjects), prompting a reevaluation of library programming at the raic. research increasingly supports library programs that provide students the opportunity to actively participate in creative endeavors at the intersection of art and science in order to view different facets of established academic and clinical concepts [1, 2]. in collaboration with art show paints a collaborative picture stark, paris, rogan, finnegan dopkins, and etter hypothesis, vol. 29, no. 2, fall/winter 2017 23 pharmacy faculty, the raic hosted an art event and subsequent pop-up art gallery with the goals of 1) increasing engagement and collaboration with faculty and students and 2) improving patron interest in the library through creative and participatory library services. the event, entitled pharmart, was designed to encourage collaborative creative activity and stimulate discussion among artists and participants. since the library is a neutral space, the raic provided a welcoming environment for artistic expression, which is recommended for the exchange of cross-curricular ideas [3]. the exhibit also provided pharmacy students and faculty an opportunity to interact socially with students from other disciplines. participation was broadly solicited from faculty, staff, and students within the school of pharmacy and health sciences, and works from every physical art medium were welcomed. the theme of the art show was “knowing your pharmacist and the local history of pharmacy.” objective previous programming at the raic had low participation, with just 15 attendees at the better hearing & speech month (bhsm) kickoff event. the primary goal of pharmart was to positively increase student and faculty engagement in person and online at the raic by hosting an active, participatory library program. not only did pharmart improve event turnout to 67 attendees from bhsm’s 15 (347%), it also increased online engagement by 24 (1200%) facebook shares and reactions. the secondary goal was to explore if students, faculty, and staff would be interested in a library-hosted art program. this goal was facilitated by encouraging creative participation from nine faculty, staff, and students who submitted eleven works of art, as well as providing attendees with the opportunity to vote in the art show. literature review libraries are discovering methods to reshape space through art and aesthetics in order to connect with the community and draw in new users. a review of the literature reveals that art shows and galleries are created with much success not only in campus libraries, but in science and health sciences libraries as well. beals’ 2007 article on student art in library exhibitions details not only the process of setting up an exhibition program, but also the rationale [3]. in this often-cited article, beals explores the process of designating the exhibition space, creating submission procedures, and deploying publicity, among other important logistical considerations, for the university of tennessee art show paints a collaborative picture stark, paris, rogan, finnegan dopkins, and etter hypothesis, vol. 29, no. 2, fall/winter 2017 24 libraries. the process includes the creation of an inviting and visually appealing space, as well as providing an educational purpose that supports the library’s mission. similarly, the university of colorado invited the general community to explore non-traditional uses of library space by establishing an art gallery that showcases works by local and visiting artists [4]. science and health sciences libraries are also realizing the value of art in science learning. the albert r. mann library at cornell university, which serves agriculture, life science and human ecology programs, has created an art gallery that attempts to create interdisciplinary knowledge and communicate science through visual imagery, such as sculpture, drawing, photography, and illustration [5]. for the marston science library at university of florida, an art contest called “elegance of science” encourages faculty, students, and staff to compete for awards and recognition, while also providing campus art connoisseurs the opportunity to reflect and contemplate the discoveries of life [6]. the national library of medicine (nlm) has contributed to interdisciplinary learning innovation by creating the nlm traveling exhibition program that health science libraries can adopt to engage with users. previous high-interest themes include health science topics found in frankenstein and harry potter novels. outcomes of the nlm’s traveling galleries reveal that they positively change users’ perceptions of the library and bring more people through the doors [7]. additionally, art shows and galleries are increasingly proving to be both ice breakers and sandboxes for future collaboration between faculty and librarians. catalano et al. point out that disparate areas of the school, such as the health sciences and arts programs, all share something in common: the library [8]. college and university libraries that have indicated an explicit desire to increase outcomes of faculty-librarian connections and collaborations through art include: • university of akron science & technology library, which created a gallery of artistic journal covers of published faculty research [9] • fort lewis college reed library, which emphasized student success outcomes within the college’s mission [10] • himmelfarb health sciences library at george washington university medical center, where the library’s art show has flourished since 1979 and has become a popular annual tradition embedded within the culture of the campus [11] throughout the literature, the common denominator in libraries developing an art show or establishing an art gallery is to further the library’s mission, supported by the ideal of the library art show paints a collaborative picture stark, paris, rogan, finnegan dopkins, and etter hypothesis, vol. 29, no. 2, fall/winter 2017 25 as a service organization with eight functions described by frieda o. weise in her 2003 janet doe lecture as: a physical symbol of knowledge, an intellectual commons, a haven for research, a place for collaboration, an access point for information, a forum to teach, a functional and pleasant workplace, and an attractive gateway to the campus [12]. art shows can embody these functions within the library by combining art with science to provide novel learning experiences for students between disciplines and encourage collaboration between faculty and librarians, where beals states: “conversation is stimulated, participation is encouraged, and new experiences are gained.” [3]. methods in an effort to engage students, faculty, and staff in library programing, library staff working in conjunction with a school of pharmacy faculty member, held a pharmacy-themed art show designed to bring together the various health sciences communities served by the raic. the name pharmart was chosen for the pilot event to support american pharmacists month, with the majority of the marketing targeted to school of pharmacy faculty, staff, and students. submission requirements also included a local history of pharmacy theme, to encourage artists and viewers to connect their learning with the event. library staff released requests for art submissions on the health sciences campus during the summer of 2015 through social media accounts and print advertising displayed in campus libraries and health sciences buildings. the pharmart gala was scheduled for october 13th from noon to 1pm. the submission process asked that prospective artists submit information about their works through a google form, which was also accessible through a qr code created at kaywa.com. artists were informed through email communications from library staff as to when and where to submit their artwork during the event. as the event date approached, internal reminders about the event were emailed to faculty and staff. pharmacy faculty were asked to promote the event to their students. student library staff were trained to answer common questions about submissions, contest details, and the event, as well as encourage participation and attendance to anyone inquiring. the library decided to organize the event in the style of a gala in the hopes of drawing additional interest. to meet this end, the gala event organizers obtained a $500 budget from the university library to purchase food and drinks, prizes, and necessary exhibit staging for artwork and decorations. knowing food would be a popular inducement, funds were set aside for that purpose in addition to the library purchasing door and participation prizes for the top three art pieces to be awarded in the days following the pharmart art show. the prizes art show paints a collaborative picture stark, paris, rogan, finnegan dopkins, and etter hypothesis, vol. 29, no. 2, fall/winter 2017 26 selected were a one-terabyte external hard drive, a $25.00 amazon gift card, a university tote bag, and a $15.00 itunes card. these were allocated as first, second, third, and door prizes respectively. prizes were awarded through the use of raffle tickets given to attendees as they entered. mention of raffle prizes and free food played a part in the in-house and online marketing campaigns. during the event, the library remained open, however users were informed that the event might be disruptive. art pieces were placed on display with a card stating title, artist, and art medium on the day of the event and, during the event, artists were encouraged to stand near their art and discuss it with viewers. raffle tickets were given to pharmart exhibit attendees for a door prize and also used as voting ballots for patrons’ favorite art pieces. these tickets were tallied after the event day and prizes were awarded and ribbons placed. all art pieces were assigned numbers which were displayed prominently on the title card, which was used to record the vote selection. at the event, attendees were encouraged to mingle while enjoying the different art displays and then place a vote for the piece they liked best by dropping their ticket, with contact information and vote recorded on the back, in a secured box. after the show, votes were tallied and winners determined based on number of votes. the art remained on display for several weeks following the exhibition. the library determined the number of students, faculty, and staff who attended the event by conducting patron counts at the beginning, middle, and end of the event, and by asking all attendees to put their basic information on a ticket used to vote for the best art work. in order to exclude patrons who were in the library for more traditional use, such as studying, attendance counts for pharmart was checked against the number of tickets collected, removing library users who did not participate in the pharmart event from the final attendance count. results to determine if the pharmart program theme and marketing efforts netted a broader audience and increased online engagement, the library compared metrics collected from a prior event. first, the research team assessed attendance and social media ‘hits’ from pharmart to the same information recorded during an earlier event held at the library, the better hearing and speech month (bhsm) kickoff party. compared to data collected from the bhsm event in may of the same year, pharmart successfully increased the target demographic attendance in the library, and increased the library’s social media presence. art show paints a collaborative picture stark, paris, rogan, finnegan dopkins, and etter hypothesis, vol. 29, no. 2, fall/winter 2017 27 attendance at pharmart recorded an increase of 347% above bhsm. a total of 67 individuals attended the art show, improving the attendance numbers from the previous bhsm program, which drew only 15 attendees (table 1). over the course of the pharmart gala, 67 raffle tickets were handed out and 60 votes were received, indicating a willingness of attendees to participate and contribute to the event. all metrics for social media also indicated a marked increase (table 1). facebook metrics showed a gain in posts (+393.33%), shares/reactions (+1200%), cumulative reach (+24.07%), and click-throughs (+2223.53%). the library’s pharmart blog post hits experienced a gain of 23.08% over bhsm. better hearing & speech month may 2015 pharmart october 2015 total attendance 15 67 student 11 54 faculty 3 7 staff 1 6 blog post hits 26 32 facebook page posts 6 29 facebook shares/reactions 2 26 facebook cumulative reach 108 134 facebook click throughs 17 395 table 1. physical attendance and social media engagement additionally, it was demonstrated anecdotally that the target community was interested in participating in an art show. student staff workers regularly answered queries about the event in the month leading up to the gala and, by the date of the gala, the library had received a rich cross section of representation of art works in a variety of two-dimensional and threedimensional media. all 11 art pieces that artists submitted for the art show were approved and included in the event. the ratio of university patron types participating was diverse, with 4 students, 3 faculty and 2 staff comprising the cohort of 9 artists. furthermore, art was submitted not only from the school of pharmacy, but also from other areas of the university: of the 11 art submissions, 6 came from the school of pharmacy, 3 from the university library, 1 from computer engineering, and 1 from the business program. art show paints a collaborative picture stark, paris, rogan, finnegan dopkins, and etter hypothesis, vol. 29, no. 2, fall/winter 2017 28 discussion compared to previous library programming, the engagement metrics reveal that pharmart generated interest and participation amongst the faculty, staff, and students in the thomas j. long school of pharmacy and health sciences. the use of a common creative interest allowed these normally separate groups to engage in a more casual atmosphere that helped to remove some of the barriers experienced within a university setting. participants included members of the different departments and disciplines of the school. verbal feedback from participants provided during the event was positive and showed that the library can be a place for engagement as well as learning. the primary goal of increasing student and faculty engagement for an active, participatory library program was achieved. not only did raic notice improvement for in-person attendance, but online engagement and interaction increased as well. an art show, in particular, may help online and social media marketing of the library since it is a visual, aesthetic medium that complements online spaces. the secondary goal of exploring interest in a library-hosted art program revealed that interest is very good, as the ratio of art participants included members from the three targeted patron groups: students, faculty and staff. furthermore, the majority of art gala attendees participated in the art show itself through voting for their favorite art pieces. previous events at the library, such as bhsm, did not include active participation and were unsuccessful in generating engagement and participation. by finding a common creative interest and developing a program that brought all together, the library space transformed from the everyday usage as a study and snacking area to a collaborative engagement area, where students, faculty, and staff mingled, interacted and participated in a pop-up art gallery. limitations a major limitation is that the research primarily calculates the quantitative increase in patron engagement of pharmart within the library and online in social media but does not assess qualitative data. samples of qualitative data that would be relevant and helpful to discuss include whether patrons enjoyed the art show, found it useful, or learned something new. as only one pharmart event has been held, future shows could include qualitative assessments or interviews, and this data could be combined with engagement metrics to build a more complete picture on the quality of engagement with the attendees. art show paints a collaborative picture stark, paris, rogan, finnegan dopkins, and etter hypothesis, vol. 29, no. 2, fall/winter 2017 29 another limitation in the study concerns other events or factors that could influence attendance. the timing of events, for example, could be construed as potentially helping or hurting attendance. in this study, attendance at bhsm was held in the time leading up to midterm exams and also just before the release of speech therapy students for their summer break, whereas pharmart was held following mid-terms: as such, the difference could be construed to affect attendance. time of year could also have an effect, as students in spring semester may be more interested in outside activities, whereas fall semester weather may be more conducive to indoor activities and events. the seasonal difference could also affect online engagement if patrons are more interested in outdoor activities in the spring rather than the fall. future directions the success of the first pharmart show has led the raic to begin developing future programs involving art and examining how this active participation can help students develop skills for stress relief and confidence building. the use of art in pharmacy curricula is very limited but has shown to be helpful in other health science disciplines. multiple studies on observing the effects of art on student development and pedagogies have been conducted primarily in medical school programs, and can provide outcomes that encourage discussion and understanding as it relates to patient care [13]. for example, in one study, a cross-disciplinary approach for developing medical insight incorporated the use of visual arts with medical themes: following the evaluation of artwork, students self-reported the positive development of observation, decision-making, reflection, and confidence-building skills [14]. in another study, medical students who took part in an arts-based learning course noted enhancement of awareness in self, improvement in self-confidence and self-care, increased preparation for stress, and development of communication and team-building skills [15]. in a study where students created and presented original artwork based on interactions with individuals living with chronic diseases, creative art was found to impact personal growth, self-discovery, sense of community and collaboration skills [16]. medical education studies such as these can help inform the development of research in pharmacy education that incorporate interdisciplinary activities, forming a nexus between art and science that could help future pharmacists develop important self-awareness and community-building tools. recently revised education and accreditation standards in pharmacy now include interdisciplinary and metacognition elements and programs such as pharmart can help schools and colleges meet these requirements [17, 18]. at the raic, future research is planned to art show paints a collaborative picture stark, paris, rogan, finnegan dopkins, and etter hypothesis, vol. 29, no. 2, fall/winter 2017 30 determine the effect of art and creativity on professionalism, stress relief, and confidence in health professional students. references 1. frei, j, alvarez, se, alexander mb. ways of seeing: using the visual arts in nursing education. journal of nursing education. 2010;49(12):672-676. doi: 10.3928/01484834-20100831-04 2. jones ek, kittendorf al, kumagai ak. creative art and medical student development: a qualitative study. medical education. 2016;51:174-183. doi:10.1111/medu.13140 3. beals jb. student art in the library juried exhibition program. art documentation: bulletin of the art libraries society of north america. 2007;26(1), 56-58. 4. shipman jp, stoddart jm, peay wj, eccles ss. spaces that support redefined roles of academic health sciences librarians. journal of the medical library association. 2013;101(3), 179-184. doi:10.3163/1536-5050.101.3.006 5. raskin h. can an art gallery thrive in a science library? college & research libraries news. 2009;70(4), 226-229. 6. buhler ag, davis v. elegance of science: coordinating an annual scientific art contest at the library. college & research libraries news. 2010;71(5), 248-251. 7. auten b, norton hf, tennant mr, edwards me, stoyan-rosenzweig n, daley m. using nlm exhibits and events to engage library users and reach the community. medical reference services quarterly. 2013;32(3):266-289. 8. catalano m, essinger c, ferimer s, lewin-lane s, vaughn p. at the center of things: how an academic library built a bridge between art and science on campus. collaborative librarianship. 2017;9(2):90-96. 9. mccullough i. science outreach through art: a journal article cover gallery. issues in science and technology librarianship. 2015;81. doi:10.5062/f4cj8bgg 10. oliver a. strengthening on-campus relationships via an annual student art commission. journal of library innovation. 2012;3(2):89-104. art show paints a collaborative picture stark, paris, rogan, finnegan dopkins, and etter hypothesis, vol. 29, no. 2, fall/winter 2017 31 11. gomes a, palena-hall e, abate l, sullo e, levett p, wilcoxson m. connecting to our community: extending librarians' roles through collaboration. journal of hospital librarianship. 2011;11(2):165-174. 12. weise f. being there: the library as place. journal of the medical library association.2004;92(1):6-13. 13. lake j, jackson l, hardman c. a fresh perspective on medical education: the lens of the arts. medical education. 2015;49:759-772. doi:10.1111/medu.12768 14. edmonds k, hammond m. how can visual arts help doctors develop medical insight? international journal of art & design education. 2012;31(1):78-89. 15. de la croix a, rose c, wildig e, willson s. arts-based learning in medical education: the students’ perspective. medical education. 2011;45(11):1090-1100. 16. jones ek, kittendorf al, kumagai ak. creative art and medical student development: a qualitative study. medical education. 2017;51(2):174-183. 17. fjortoft n. the challenge of the accreditation council for pharmacy education’s standard four: identifying, teaching, measuring. american journal of pharmacy education. 2016;80(5):73. 18. acpe. accreditation standards and key elements for the professional program in pharmacy leading to the doctor of pharmacy degree. 2015. https://www.acpeaccredit.org/pdf/standards2016final.pdf . accessed september 1, 2017. https://www.acpe-accredit.org/pdf/standards2016final.pdf https://www.acpe-accredit.org/pdf/standards2016final.pdf research section spotlight hypothesis, vol. 29, no. 2, fall/winter 2017 32 “librarian’s role in reproducibility of research” 2017 symposium: research section stipend winner reflection kimberly r. powell, mlis life sciences informationist woodruff health sciences center library emory university atanta, ga krpowel@emory.edu introduction at the 117th annual meeting and exhibition of the medical library association (mla) in 2017, eight sections of the association jointly sponsored a symposium entitled librarian’s role in reproducibility of research. the four hour symposium was held saturday, may 27th as part of the mla pre-conference activities. shona kirtley, knowledge and information manager for the equator network at the centre for statistics in medicine at the university of oxford, served as the keynote speaker. invited panelists discussed their roles in initiatives aimed to reduce the research waste caused by irreproducible reporting of scientific efforts within the published literature. the four-hour symposium concluded with a hands-on brainstorming activity that asked each of the attendees to propose and reflect on increasing the reproducibility of science. the following is a summary of the information shared and a reflection on the brainstorm suggestions made at the librarian's role in reproducibility of research symposium. a libguide for the event, including agenda and speaker slides are located at http://mlasymposium.libguides.com/c.php?g=584462&p=4036194. defining the reproducibility crisis in march 2012, a commentary on the reproducibility of preclinical cancer studies was published in nature. this report by the company amgen highlighted the disappointing success rate of translating basic science findings into clinical therapeutics, especially with regards to cancer studies. amgen researchers conducted a review of 53 published studies finding only six (11%) of the results could be replicated [1]. this endorsed an earlier report from the pharmaceutical http://mlasymposium.libguides.com/c.php?g=584462&p=4036194 research section spotlight hypothesis, vol. 29, no. 2, fall/winter 2017 33 company bayer that from a sample set of 67 published drug reports, only 20-25% were reproducible [2]. in response to these concerning observations, the lancet launched a series of papers that discuss increasing the value and reducing research waste in the published literature, ultimately becoming the lancet’s reduce research waste and reward diligence (reward) campaign in 2015. in 2016, nature published survey results from 1,576 researchers, 90% of whom affirmed a reproducibility crisis within the published literature [4]. in 2015, the u.s. national science foundation (nsf) released recommendations authored by the subcommittee on replicability in science. among other findings, this report noted that many terms used to discuss the ability to replicate or generalize a study were applied inconsistently. the report thereby offered the following definition: “reproducibility refers to the ability of a researcher to duplicate the results of a prior study using the same materials and procedures as were used by the original investigator (…) reproducibility is a minimum necessary condition for a finding to be believable and informative.” [3] this differs slightly from replicability, defined by the report as “the ability of a researcher to duplicate the results of a prior study if the same procedures are followed but new data are collected” [3]. in contrast, the nature survey of researchers asked if respondents were able to reproduce results in a “similar experimental system” which “may include slight variations in methods or materials” [4]. moving forward, as funders, publishers, researchers, librarians, and other stakeholders work to formulate strategies to address the concern, it will be important to work from a standard definition of the problem. many approaches would conceivably impact both reproducibility and replicability; however, when discussing the scale of the crisis, and proposing focused solutions, it will be important to note the difference. policies: what are funders and publishers doing to support reproducible research? proposed solutions for increasing the reproducibility (and replicability) of the published literature can typically be broken into two main approaches: those that target data reporting and those that target process reports, or methodologies. selective data sharing was cited as one of the top factors contributing to the crises by respondents of the nature survey. selective reporting may occur when authors publish a clean story, leaving out factors such as replicates that did not meet expectations, outliers, or statistical tests that did not show desired research section spotlight hypothesis, vol. 29, no. 2, fall/winter 2017 34 results. data sharing initiatives help address these selective pressures by requiring all underlying data of published summaries and visualizations be made available. this creates options for timely, independent verification and may limit exaggerated reporting. however to address the reproducibility crises rather than just re-use initiatives, more rigorous sharing of research design and data collection methods are also required. funders of research have put together recommendations and guidelines in both these areas as seen by the nsf data sharing policy (https://www.nsf.gov/bfa/dias/policy/dmp.jsp) and the nih rigor and reproducibility web portal (https://www.nih.gov/research-training/rigorreproducibility). additionally, organizations such as the center for open science (cos) (https://cos.io/) in the u.s. and the enhancing the quality and transparency of health research (equator) network (http://www.equator-network.org) in the u.k. also provide resources to address the process of sharing scientific findings. these organizations provide frameworks and guidelines for conducting and reporting reproducible scientific efforts. guidelines and toolkits for reporting common study types are linked on the equator network, while cos provides a framework for project management support, training, and updates on ongoing projects looking at field specific reproducibility issues. journal publishers have also responded to the call for increased reproducibility, though most have focused on data sharing strategies. science, nature, public library of science (plos), and proceedings of the national academy of sciences (pnas) have all established data sharing or data accessibility policies for their authors. authors in these journals are expected to share the data and statistics underlying their findings, preferably in a public domain repository. the sharing of such data allows for both reproducibility and confirmatory actives as well as the reuse of data for new discoveries. journal publishers focusing on the increased rigor of published methodologies have done so primarily by exploring new publication types. the journal of visualized experiments (jove) has been a leader in this area, developing a publication module of instructional videos aimed to better communicate experimental processes. as a panelist at the 2017 mla reproducibility symposium, jove co-founder and ceo, moshe pritsker, noted that the classic structure of the journal article has remained largely unchanged since the first publication in 1665 and called emphatically for detailed methods as a stand-alone publication model [5]. reinforcing his call, another panelist shared an example of the importance of robust materials reporting when a study found that variations in software version, workstation type (mac or pc), and mac https://www.nsf.gov/bfa/dias/policy/dmp.jsp https://www.nih.gov/research-training/rigor-reproducibility https://www.nih.gov/research-training/rigor-reproducibility https://cos.io/) http://www.equator-network.org/ research section spotlight hypothesis, vol. 29, no. 2, fall/winter 2017 35 operating system all had a significant impact on the analytic readings of 30 mri scans [6]. when variables such as computer type and operating system are shown to play a role in reproducible data analysis, there is a clear need for more robust reporting than just product names. registered reports offer another emerging publication type in support of reproducibility. these are detailed study designs which undergo the peer-review process separate from the final study analysis. registered reports describe key methodologies such as experimental model, data collection instruments, and statistical methods used in the proposed analysis. these are then reviewed for rigor and reproducibility before data are collected. currently, cos lists 52 journals which have adopted the publication of registered reports in some capacity. journals such as royal society open science and bmc biology will then provisionally accept results for publication, contingent upon adherence to registered study design, regardless of study outcomes. this slow shift in accepted publication types represents a positive cultural shift in the scientific community by placing emphasis on rigorous scientific processes rather than focusing solely on novel, positive results. library services: how can librarians support a culture of reproducible research? as kirtley emphasizes in her 2016 the lancet commentary, librarians are well posed to be part of the answer to managing the current reproducibility crisis [7]. academic and medical librarians are familiar partners in the research lifecycle, from developing robust and comprehensive search strategies, to selecting a journal and assisting with data sharing and management plans. librarians who work on systematic review projects are intimately familiar with the challenges of adapting complex methods to new database environments. by attuning specific knowledge and services to the language and needs of addressing reproducibility, librarians are equipped to serve as advocates and partners. even traditional roles such as collection development and access training can be easily adapted to addressing reproducibility as it relates to growing the awareness, availability, and utilization of new publication types. panelists at the 2017 mla reproducibility symposium shared unique ways librarians at their institutions have contributed to creating reproducible research. cynthia hudson-vitale, at the university of washington in st. louis, works with the institute for clinical and translational sciences to establish a framework for reproducible methods when working with electronic health records (ehrs). librarians collaborated with researchers to identify 103 variables needed to ensure the reproducibility of ehr analysis: beginning with stating a clear, focused hypothesis research section spotlight hypothesis, vol. 29, no. 2, fall/winter 2017 36 through tracking query language with version notes and access dates, identifying statistical tests and packages, and reporting such specifics with standard documentation. all 103 variables would be needed in any resulting publications or data codebooks or the analyses could not be reliably replicated, regardless if the raw de-identified data is available and accessible. bart ragon from the university of virginia and kristi holmes from northwestern university, echoed the vital importance for librarians to offer collaborative support for open science and data management. to further explore actionable roles and services, participants at the symposium completed a hands-on exercise in which each person was asked to propose a specific idea for librarian involvement and collaboration. ideas were then anonymously scored for how well the proposal resonated for an individual’s library and institution. collected from over the 30 responses, the highest scoring suggestions are listed below. while specifics of each suggestion were not discussed at the symposium, one possible interpretation of the proposal follows each participant idea. • host “reproduce-my-research” events such outreach or training events may take many different approaches. the idea seems to suggest giving researchers and students a formalized setting to reflect on and engage with each other specifically around how to improve the scientific reporting of their manuscripts. as this was the high scoring suggestion, it clearly resonated with many librarians who saw inspiration in this event title within their institutional outreach even without further details. • offer training to students and early career researchers many libraries already provide training opportunities to gain a better understanding and engage with resources, publishers, and services. refocusing or adopting specific language to target reproducibility concerns may be an easy adaptation for services already in place. • incorporate into the researcher workflow again, many libraries already have collaborations or specific services targeting various aspects of the research lifecycle. emphasizing these services as essential for addressing the reproducibility crisis may further campus collaborations and refine librarian roles. • establish data management best practices research section spotlight hypothesis, vol. 29, no. 2, fall/winter 2017 37 when working with data management strategies, many initial approaches supported by libraries may have been reactive to specific funder mandates. by proactively establishing best practices in line with robust and reproducible science, librarians are natural champions of more accessible research and open data. • educate librarians by seeking out learning and professional networking opportunities such as the 2017 reproducibility symposium, librarians are educating themselves about the problem, understanding researcher frustrations and emerging policies, and collaborating as a profession to explore targeted services. opportunities to brainstorm and share challenges and success stories are invaluable for the development of robust and innovative services. • collaborate with institutional offices collaboration is implied in many of the top-scoring suggestions. continued outreach to researcher groups and campus partners will be essential to the recognition of librarians as part of the solution. • provide high-quality, reproducible search results for those librarians working on systematic reviews or providing in depth reference support, this is an opportunity to lead by example. providing details of the search query along with specifics such as database coverage, search date and applied filters, raises subtle awareness of the necessity of these details. this may then provide the opportunity for discussion of similar essentials when reporting results. the majority of these high scoring responses reflect activities and services already underway, with some variation, at many academic libraries. by educating librarians and raising awareness among researchers in the specific areas of reproducible research, many library services can be easily adapted to address the reproducibility crises. hosting additional symposiums and workshops around the topic will encourage other librarians to share the specific services and outreach initiatives that have been successful. research section spotlight hypothesis, vol. 29, no. 2, fall/winter 2017 38 references 1. begley cg, ellis lm. drug development: raise standards for preclinical cancer research. nature. 2012;483(7391), 531-533. 2. prinz f, schlange t, asadullah k. believe it or not: how much can we rely on published data on potential drug targets? nat rev drug discov. 2011;10(9), 712-712. 3. bollen k, cacioppo jt, kaplan rm, krosnick ja, olds jl, dean h. reproducibility, replicability, and generalization in the social, behavioral, and economic sciences. 2015. retrieved from https://www.nsf.gov/sbe/sbe_spring_2015_ac_meeting_presentations/bollen_report_on_re plicability_subcommitteemay_2015.pdf 4. baker m. 1,500 scientists lift the lid on reproducibility. nature. 2016;533(7604), 452-454. doi:10.1038/533452a 5. pritsker m. the science reproducibility crisis: how librarians can solve it and why they should care? paper presented at the the librarian's role in reproducibity of research symposium, seattle, wa. 2017. 6. gronenschild ehbm, habets p, jacobs hil, mengelers r, rozendaal n, van os j, marcelis m. the effects of freesurfer version, workstation type, and macintosh operating system version on anatomical volume and cortical thickness measurements. plos one. 2012;7(6), e38234. doi:10.1371/journal.pone.0038234 7. kirtley s. increasing value and reducing waste in biomedical research: librarians are listening and are part of the answer. the lancet. 2016;387(10028), 1601. doi:http://dx.doi.org/10.1016/s0140-6736(16)30241-0 -- https://www.nsf.gov/sbe/sbe_spring_2015_ac_meeting_presentations/bollen_report_on_replicability_subcommitteemay_2015.pdf https://www.nsf.gov/sbe/sbe_spring_2015_ac_meeting_presentations/bollen_report_on_replicability_subcommitteemay_2015.pdf http://dx.doi.org/10.1016/s0140-6736(16)30241-0 research section spotlight hypothesis, vol. 29, no. 2, fall/winter 2017 39 mla 2017 research section research awards congratulations to the winning research papers and posters from mla ’17! the mla research section is pleased to announce the winners for best research papers and posters presented at the mla 2017 annual meeting in seattle, wa. thank you to the 57 judges who volunteered their expertise to help select these deserving awardees. to learn more about the awards and selection process, visit the research section website at http://www.mlanet.org/p/cm/ld/fid=938. contributed papers 1st place authors: christy jarvis, ahip, head of information resources and digital initiatives, university of utah, salt lake city, ut; melissa rethlefsen, ahip, deputy director / associate librarian, spencer s. eccles health sciences library, salt lake city, ut title: daring to dive deep into the citation data: going head to head with scihub abstract: objectives: to analyze the extent to which the library is fulfilling the information needs of its patrons in light of recent data showing heavy usage of scihub in [our metro city] and specifically to gain insight into the percentage of referenced resources and content areas that were not made available to researchers through library funded subscriptions. methods: using scopus, we identified papers published by the institution’s health sciences faculty in each of the previous 5 years. papers co-authored by researchers at other institutions were excluded in order to focus the analysis solely on information resources available to scholars at the target institution. the citations from the resulting set of publications were extracted to spreadsheets, where they were subjected to a data normalization process. incomplete or obviously erroneous references were removed. the remaining set of citations was sorted alphabetically by journal title, then secondarily by citation year. using a combination of data sources, including publisher entitlement reports, catalog records, and previously downloaded holdings reports, we compared each citation to the library’s collection at year of http://www.mlanet.org/p/cm/ld/fid=938 research section spotlight hypothesis, vol. 29, no. 2, fall/winter 2017 40 citation to determine if access to the cited resource was provided and paid for by the library. results: published literature authored by university of utah health sciences faculty between 2012 and 2016 yielded 119,794 citations for analysis. the libraries had print or electronic access to 99,298 (82.89%) of these cited resources. another 9,220 (7.7%) were accessible from open access platforms, leaving only 11,084 (9.25%) citations that needed to be obtained from other sources, such as interlibrary loan, pay-per-view, or illegitimate sites such as sci-hub. of the cited, but not provided, literature, 26% comes from backfile content whereas 74% comes from more recently published work. identified collection gaps include the disciplines of neurology, cardiology, and oncology. conclusion: a comprehensive evaluation of health sciences author citations over a 5-year period demonstrated that the vast majority were available from library-funded resources, thereby suggesting that the library’s collection has been adequately meeting the needs of researchers and scholars. this study provides valuable insight into faculty information-seeking behaviors and has implications for future collection development, service offerings, and funding priorities within the library. further study is needed to explore correlations between library-funded access to cited resources and other university metrics such as faulty recruitment and grant funding, as well as to investigate purported sci-hub activity in light of our findings. 2nd place authors: joanne marshall, ahip, fmla, research professor, school of information and library science chapel hill, nc; amber wells, doctoral graduate, dept of sociology, chapel hill, nc; kathel dunn, associate fellowship program director, national library of medicine, bethesda, md; joyce backus, associate director for library operations, national library of medicine, bethesda, md title: the role of medline in patient care: results of a secondary analysis of the value of libraries study abstract: objectives: what role does the medline database play in relation to other information resources that are available to health care providers? what is the role of medline in positively impacting patient care? since health care providers use multiple information resources in research section spotlight hypothesis, vol. 29, no. 2, fall/winter 2017 41 providing patient care, what is the specific role of medline? methods: a previous survey on the use of health information resources for patient care obtained 16,122 responses from 56 hospitals in the u.s. and canada. the study asked respondents to indicate resources used in answering specific clinical questions. on average, respondents reported using 3.5 resources to answer their question. this analysis used advanced descriptive statistics and regression analysis to examine the specific information resources used and how they were used in combination with one another. the use of more resources was associated with more changes made to patient care and increased avoidance of adverse events. medline was more likely to be among the resources consulted than any other information resource except journals. the analysis reported in this paper provides new insights into how medline is used in conjunction with other resources to answer clinical questions. results: our additional analysis of the value study data found that medline and online journals were the two most frequently used information resources. respondents reported using an average of 3.5 resources when seeking additional information related to a specific patient care decision making situation. using more information resources was associated with improved clinical decision making and a higher probability of making changes to patient care and avoiding adverse events. medline was most likely to be among the combinations of information resources used by physicians, residents and nurses. conclusion: medline continues to be a key information resource for health care providers as they seek answers to patient care questions. the medline database is also used in the preparation of many other specialized health information resources and point of care information tools. since health professionals use multiple information resources, libraries and librarians continue to have an important role in providing access to and supporting the use of a wide range of information tools. honorable mention authors: tanja bekhuis, ahip, principal scientist, tcb research & indexing llc, pittsburgh, pa title: training in support of data-driven research: a qualitative study of library workshops in top national institutes of health-funded and national science foundation-funded universities research section spotlight hypothesis, vol. 29, no. 2, fall/winter 2017 42 abstract: objectives: in an age of data-driven research and competitive funding, libraries help their patrons acquire new skills. to do so, some offer workshops about knowledge discovery, analysis, and management of various kinds of data. the primary objective of this qualitative study was to explore the nature of workshops offered by libraries to support data-driven research in top nihand nsf-funded universities. additionally, we developed a catalog of workshops, and indexed the resources and thematic content. methods: to identify top-funded universities, we used nih research portfolio online reporting tools and the nsf budget internet information system. from corresponding websites, we extracted information on 99 workshops offered by health sciences libraries (n=5) and main libraries (n=5) in schools funded by nih and nsf, respectively. workshop title, duration, and description were catalogued by library and source of federal funding for the university. we used nvivo 11 pro (qsr international) for qualitative data analysis and textract® (texyz) to semi-automate indexing the content of the catalog. themes were first identified in textual patterns and then were refined by an analyst. thematic overlap was described across funding source. additionally, we identified themes unique to each subset. results: main libraries in nsf-funded schools offered 36% more workshops than health sciences libraries in nih-funded schools (57 vs 42). overall workshop duration ranged from 1 to 16 hours in a bimodal distribution (1st mode = 1 hour; 2nd mode = 3 hours). the distribution of duration for nsf schools differed from the nih distribution. we identified 15 main themes overall: statistical programming and data visualization occurred most often, and finding funds for research and open science least often. thematic distributions varied with funding source. for example, bioinformatics occurred most often in the nih-funded subset and statistical programming in the nsf subset. for each subset, 20 most informative indexing terms were identified after sorting and discretizing into 7 quantiles. top indexing terms included: data visualization, pathway analysis, and data management (nih schools); data analysis, data management plan (dmp), and python (nsf schools). conclusion: a catalog of workshops organized by university funding source and library, along with 2 indexes (resource and subject), will be publicly available. the analytical results, as well as index content, yield insights regarding workshop coverage. implications for strategic planning and development of library workshops in support of data-driven research will be discussed. research section spotlight hypothesis, vol. 29, no. 2, fall/winter 2017 43 contributed posters 1st place authors: angela spencer, manager, c. alan mcafee md medical library, chesterfield, mo; elizabeth laera, medical librarian, brookwood baptist health, birmingham, al; halyna liszczynskyj, director, library services, st. elizabeth medical center, utica, ny; louise mclaughlin, information specialist, woman's health sciences library, baton rouge, la; kathy zeblisky, medical library manager, phoenix children's hospital, phoenix, az title: solo librarians: demographics, duties, needs, and challenges abstract: objective: to obtain data on how many librarians classify themselves as solo librarians within a medical/hospital setting. solo librarians constantly face challenges to maintain and expand services vital to their users. by quantifying their number and needs, a stronger voice can be developed. methods: a ten question survey using surveymonkey was sent to various medical library related listservs of interest to solo librarians. results: 383 surveys were returned, the majority from hospital and academic librarians. other settings include clinics, organizations, research institutions and veteran’s institutions. duties showed the variety of hats a solo can wear. duties included: reference, interlibrary loan, teaching, committee work, website development, marketing, creating policies/procedures, writing grants, archives, informatics and other work. the “best challenges” question was the most insightful into what the needs are for solos. major challenges included: funding/budget, awareness/visibility, time management, value/roi/proving your worth, staffing, space, promotion/marking/outreach, professional development, technology and organizational mergers. discussion: the full survey results quantify the size of the solo librarian population, and the contributions and challenges they face working in solo settings. this data can contribute useful information to discussions on best ways to support, educate, inform and advocate for this research section spotlight hypothesis, vol. 29, no. 2, fall/winter 2017 44 population. conclusion/next steps: solo librarians are faced with similar financial, marketing and operational challenges regardless of setting. we hope to encourage peers to share their challenges and concerns and work with nn/lm and mla to educate them about solo librarians’ needs and concerns so that we can sustain our future. 2nd place authors: nicole theis-mahon, liaison to the school of dentisty & hsl collections coordinator, university of minnesota, minneapolis, mn; shanda hunt, public health library liaison & data curation specialist, health sciences libraries, minneapolis, mn title: my doctor said what!? identifying and assessing online health information resources abstract: objectives: health information consumers look to the internet to find answers to questions about their health or that of a loved one. we conducted a study to identify where individuals find online health information, how they use it, and what they think is missing. results from this study are being used to make recommendations of how to improve services to this population. methods: the university of minnesota health sciences libraries conducted a cross-sectional study of adults in august 2016. the survey instrument was adapted from the ehealth literacy scale (eheals) and the patient activation measure (pam-13), administered electronically on tablets at the minnesota state fair, and took approximately six minutes to complete. convenience sampling yielded a total of 281 participants. analysis of descriptive statistics and statistics to explore relationships between variables were conducted using r, and a qualitative analysis of one survey item was conducted using nvivo. results/conclusion: preliminary results show that a majority of participants use a search engine, such as google, webmd, or the mayo clinic website, to locate online health information. while most respondents were confident in their ability to evaluate the health resources they find online, only half identified indicators of quality health information. this result was confounded by the high number of participants who were health providers. participants identified personalization of and interactivity with health websites as highly desirable. research section spotlight hypothesis, vol. 29, no. 2, fall/winter 2017 45 honorable mention authors: hannah norton, interim fackler director, associate university librarian, university of florida, gainesville, fl; mary edwards, reference & liaison librarian, health science center library, gainesville, fl; ariel pomputius, health sciences liaison librarian, health science center library, gainesville, fl; michele tennant , interim fackler director, health science center libraries, gainesville, fl title: tracking tech trends: studying patron technology use through annual surveying abstract: objectives: at an academic health sciences library serving students, faculty, and staff across a wide variety of disciplines, studying library patrons’ technology use, particularly in areas of mobile technology, provides necessary information on intersection points for library services. administering a similar survey annually for five years generates a holistic view of patrons’ technology needs and preferences over time. methods: beginning in 2012, the university of florida (uf) health science center library (hscl) began administering a 16-question survey designed by the university of southern california norris medical library to address technology use of health professional students and faculty and their interest in related library services. for three years we participated in a multiinstitution implementation of this survey; when the collaboration ended, we continued to administer the survey at uf. while some questions have been modified over time for clarity or changes in available technology, many are consistent across the five years of survey implementation, allowing analysis of trends over time in use of specific technologies and service needs at our institution. results: smartphone ownership among survey respondents is nearly universal (ranging from 87.6% to 95.7% over the past 5 years), and a majority of respondents also own a tablet (from 51.1% to 70.2%). while respondents were likely to check library hours, use medical apps, and use library electronic resources from their smartphone or tablet, they reported being unlikely to friend or follow the library on facebook or twitter or send a call number from the catalog. one simple change implemented in response to survey results was to add the library’s hours to the “quick links” portion of the library’s website; while the hours are featured on other parts of the site, the quick links are the most prominent portion of the site’s mobile version. likewise, when survey data indicated that respondents were highly interested in research section spotlight hypothesis, vol. 29, no. 2, fall/winter 2017 46 training on mobile device apps, the hscl developed a stand-alone workshop entitled “mobile resources for health.” trends that have not yet been explored further include respondents’ preference for print books for both academic (53.1% to 57.3%) and leisure (53.2% to 55.2%) reading, as compared to ebooks. conclusions: annual review of survey results has led to incremental changes in services offered. reviewing the aggregate data allows for more strategic consideration of future directions, with implications towards marketing the library’s resources, training development, and service development. -- about the publication submission deadlines submission criteria reference style hypothesis content types copyright what’s our story? and how can you help tell it? welcome from research section chair (2017-18 term) rising stars research projects 2016-2017: action research to improve mla’s communities introduction conclusion assess the structure, procedures, and operation of the mla section council and propose changes that better serve the council purpose and enable it to determine if it is successful in realizing its purpose rachel c. lerner assess the relevance of the current (summer 2016) strategic goals of sections to mla members and the health information profession and propose an initiative to improve relevance tony nguyen assess how well the topical coverage of mla sections and sigs address the needs of mla’s primary audiences and propose an initiative to improve coverage phill jo assess sig membership requirements and guidelines and propose new requirements and guidelines gregg a. stevens references art show paints a collaborative picture: increasing engagement in a pharmacy and health sciences library abstract introduction objective literature review methods results discussion limitations future directions references “librarian’s role in reproducibility of research” 2017 symposium: research section stipend winner reflection introduction defining the reproducibility crisis policies: what are funders and publishers doing to support reproducible research? library services: how can librarians support a culture of reproducible research? references -- mla 2017 research section research awards contributed papers 1st place 2nd place honorable mention contributed posters 1st place 2nd place honorable mention --editorial reviewed article hypothesis , vol. 34, no. 1, 2022 volunteer and ideas request dear readers: as the summer winds down and our next issue is weeks away from being published, we’d like to encourage you to submit ideas for what you would like to see in future issues. if you have ideas for an article or even a whole issue focus, we would love to hear from you: https://airtable.com/shrr8kxqvryzm6gcs we are also looking for interested individuals to volunteer for peer-reviewing, marketing, website maintenance and other opportunities. it’s a great way to meet new colleagues, improve your skill set, and contribute to the profession. volunteer here: https://airtable.com/shrb57mtvgykc76lx our journal can only be enhanced by involvement with the community. we look forward to hearing from you. marketing team 1 https://airtable.com/shrr8kxqvryzm6gcs https://airtable.com/shrb57mtvgykc76lx peer reviewed article hypothesis , vol. 35, no. 2, 2023 evaluation of a book club kit program amy weig-pickering, mlaa, jonathan d. eldredge, phd, fmlab alibrary information specialist 3, health sciences library and informatics center, university of new mexico, albuquerque, new mexico,https://orcid.org/0009-0006-9876-6312 , aweigpickering@salud.unm.edu bprofessor, health sciences library and informatics center, university of new mexico, albuquerque, new mexico,https://www.orcid.org/0000-0003-3132-9450 , jeldredge@salud.unm.edu background: book clubs consist of groups of people reading and discussing the same books. some book clubs are organized by public librarians. a far smaller number of health sciences librarians are involved with coordinating book clubs. objectives: this case report employed a survey and phone interviews to determine why a book club kit program organized by health sciences librarians did not generate more enthusiasm. lessons learned: the investigators found that direct phone interviews rather than a survey elicited the needed information. many public libraries, particularly those in rural areas, are directed by people not even possessing a master’s degree in library or information science. strategies for promoting use of the book club kits to these public librarians required different approaches. conclusions: through a deeper understanding of rural public librarians’ circumstances and needs, the investigators gained new insights on how health sciences librarians might possibly reach public librarians more effectively. background book groups are popular gatherings of people wanting to share their interest in reading books. mccook de la peña and bossaller depict them as “a long-standing phenomenon of american cultural life that now take[s] place in both real space and cyberspace.”1 they vary greatly in their format and context. some are organized by their participants spontaneously; others are more structured and hosted by a facilitator. some book groups borrow the books from a library while others rely on members to purchase their own copies. the ages of participants range from children to the elderly. book group discussions can take place in people’s homes, restaurants, community centers, or libraries. while most librarians associate book groups with public or school libraries, health sciences libraries also have hosted or organized book groups.2-4 the university of new mexico’s health sciences library and informatics center (hslic) is the only health sciences library in the state. one of our strategic goals was to “empower our community to engage with health information.” one strategy within that goal was to: “create a sustainable outreach program to promote health information literacy and health equity.” in order to support this strategic goal, we 1 https://orcid.org/0009-0006-9876-6312 https://orcid.org/ mailto:aweigpickering@salud.unm.edu https://www.orcid.org/0000-0003-3132-9450 https://orcid.org/ mailto:jeldredge@salud.unm.edu peer reviewed article hypothesis , vol. 35, no. 2, 2023 developed ten book club kits which included: eight copies of each book, book discussion questions, and health literacy handouts. our kits were based on the network of the national library of medicine’s (nnlm) “reading is healthy” book club model.5 the nnlm “reading is healthy” book club was created by a team of consumer health librarians who selected books to align with the priority health topics of its sponsoring organization, the nnlm all of us community engagement network. the nnlm reading club originally was promoted to public libraries as a popular and familiar “ready-to-use” health literacy program, and it was a success. they defined success as follows: “from a total collection of 21 featured health topics and 78 book titles, more than 6,300 readers nationwide discussed health and wellness. after a brief hiatus, the nnlm reading club resumed in 2022 with new health topics and titles.”6 our book club kits were publicized through sending a series of three direct emails to all 101 community and public library directors listed on the new mexico state library website.7 we thought the book club kits would be utilized since new mexico is a rural state and many of our libraries might not have sufficient resources to sponsor a book club. we offered the book club kits free of charge and included all the materials to conduct a book discussion group. ironically, we had an initial concern that the program would be so popular that we may not have enough materials to circulate. to promote the book club kits, we attended the native american special interest groupmeeting8 and the health literacy task force meeting.9 there was not much interest generated from those initial contacts, so we sent out emails to the 101 public and community libraries listed on the aforementioned new mexico state library website. unfortunately, the initial email generated only one library’s interest in checking out the book kits. this unexpected lack of response warranted further investigation. we were genuinely curious as to why more libraries were not taking advantage of the book club kits. was there something we could change so that more libraries would participate or was it something else that was preventing libraries from ordering a book club kit? method our method consisted of structured and unstructured interviews. we designed a five-question survey to guide the telephone interviews with library directors or staff at libraries who potentially might participate in the book club, after receiving approval from the institutional review board #22-270. we interviewed a total of 24 new mexico community and public librarians by telephone. we decided to reach out to these 24 libraries because they provided a cross section that represented rural/urban, large/small, public/community, and wide geographic spread throughout the state. also, several libraries did respond that they did not have any interest in the book club kits already, so we declined to interview them further. the first four interview questions only involved a yes or no response. the interviewers asked: 1. do you currently have a book club at your library? 2 peer reviewed article hypothesis , vol. 35, no. 2, 2023 2. do you have enough staff to facilitate a book club? 3. is your lack of participation due to the fact that materials are only available in english? 4. is it the choice of titles that discouraged you from ordering a book club kit? 5. in your opinion, what is the biggest reason as to why you did not take advantage of the book club kits? results talking with librarians directly made a huge difference. on more than one occasion, the library directors were the ones who picked up the phone and were generous with their time. the overall results summarized in table 1 demonstrate that 10 of these libraries are understaffed and are unable to conduct any type of book club. table 1. reasons for not responding explanations number of libraries lack of staff 10 programming is preplanned 4 did not see emails 4 lack of interest 2 would not participate in interview 2 do not have a book club 1 no book club experience 1 also, four libraries preplan their programs and do not have the ability to stray from that schedule. another four library directors reported not even seeing the original email, which might partially explain some lack of responses. two contacted librarians chose not to participate in the interview and one library did not have a book club. once there was an actual conversation with the various library directors and/or staff, three libraries expressed interest in possibly ordering book kits in the future. lessons learned 1. direct individual contact through the phone generated genuine responses and created interest in the book club kits. 2. a single mode of communication (example: email) is not always effective. 3. many people still prefer to communicate through conversation. 4. outside of the major cities in new mexico, many community and public libraries often do not have a professionally trained mls librarian with a broad perspective on the logistics and advantages of sponsoring a book club. 3 peer reviewed article hypothesis , vol. 35, no. 2, 2023 5. a book club kit program created by a national library that has proven successful in some states might require a different approach in other states. 6. public and community libraries are embedded in communities with interests that might not align with pursuit of a book club; or at least a book club revolving around health issues. 7. public and community libraries in rural areas have limited internet service, so phone rather than email might be the most reliable way to contact these sites. discussion overwhelmingly the results show that many public and community libraries around new mexico are facing staffing shortages.10 it appears there could have been more participation in the book clubs if there was either staff or volunteers to facilitate the program. in addition, the two libraries that stated they were unable to assist with this research could also be indicative of time constraints. in speaking with librarians on the phone, results showed that several did not receive or notice any communication regarding the book kits. by reaching out via phone to make more personal connections, we gained some traction for our book club kits. finally, the territorial governor of new mexico, lew wallace, once wrote, “every calculation based on experience elsewhere fails in new mexico”11 suggesting that a book club kits program created by a national library that has proven successful in other states might require a different approach in some states. the interview results reveal that the program itself may not be the problem, but other aforementioned factors that contributed to the lack of participation. references 1. mccook kathleen de la peña, bossaller js. introduction to public librarianship. third ed. chicago: ala neal-schuman; 2018. https://hslic-unm.on.worldcat.org/oclc/1047729276. accessed january 27, 2023 2. haley j, mccall rc, zomorodi m, de saxe zerdan l, moreton b richardson l. interprofessional collaboration between health sciences librarians and health professions faculty to implement a book club discussion for incoming students [published correction appears in j med libr assoc. 2019 oct 107(4):637]. journal of the medical library association. 2019;107(3):403-410. available from: https://doi.org/10.5195/jama.2019.536 3. stephenson pl, clever s, coady tr, ender d, heyd m, peth s. book clubs—outreach opportunities for hospital libraries. medical reference services quarterly. 2014;33(4):448-459. available from: https://doi.org/10.1080/02763869.2014.957096 4. timm, df, woodson d, jones d. book discussion course: timely topics for medical students. medical reference services quarterly.2014;33(2):147-156. available from: https://doi.org/10.1080/02763869.2014.897513 5. national library of medicine (us). national network of libraries of medicine. nnlm reading club. accessed april 20, 2023. https://www.nnlm.gov/nnlm-reading-club 6. ibid. 4 https://hslic-unm.on.worldcat.org/oclc/1047729276 https://doi.org/10.5195/jama.2019.536 https://doi.org/10.1080/02763869.2014.957096 https://doi.org/10.1080/02763869.2014.897513 https://www.nnlm.gov/nnlm-reading-club peer reviewed article hypothesis , vol. 35, no. 2, 2023 7. new mexico. state library. library locations and information. accessed april 20, 2023. https://nmstatelibrary.org/services-for-new-mexico-libraries/electronic-discussion-lists/ 8. new mexico library association native american special interest group. accessed august 10, 2023. https://nmla.org/nalsig 9. university of new mexico. hospital. health literacy office. accessed august 10, 2023. https://unmhealth.org/community/health-literacy.html 10. state of new mexico. state library. annual public library statistics fy2021. accessed april 25, 2023. https://nmstatelibrary.org/services-for-new-mexico-libraries/statistical-reports/ 11. mckee i. ”ben-hur” wallace: the life of general lew wallace. berkeley, ca: university of california press; 1947: 163. 5 https://nmstatelibrary.org/services-for-new-mexico-libraries/electronic-discussion-lists/ https://nmla.org/nalsig https://unmhealth.org/community/health-literacy.html https://nmstatelibrary.org/services-for-new-mexico-libraries/statistical-reports/ editorial reviewed article hypothesis , vol. 34, no. 1, 2022 mla 2022 annual meeting and jmla biannual research caucus research awards lindsay blake, mlis, ahipa, aclinical services coordinator, uams library, university of arkansas for medical sciences, little rock, arkansas, https://www.orcid.org/0000-0002-8234-8611 , leblake@uams.edu congratulations to the award-winning research papers and posters from mla ’22 hybrid as well as the biannual jmla research paper winner! the mla research caucus is pleased to announce the winners for best research papers and posters presented at the mla 2022 hybrid meeting. thank you to all the judges who volunteered their expertise to help select these deserving awardees both in the prejudging phase and during the conference. to learn more about the awards and selection process, visit the research caucus website. jmla biannual research article award exploring data management content in doctoral nursing handbooks. authors: rebecca raszewski, abigail h. goben, martha dewey bergren, krista jones, catherine ryan, alana steffen, susan c. vonderheid. citation: raszewski r, goben ah, bergren md, jones k, ryan c, steffen a, vonderheid sc. (2021). exploring data management content in doctoral nursing handbooks. j med libr assoc. 2021 apr; 109(2):248–257. doi:10.5195/jmla.2021.1115 mla 2022 annual meeting contributed paper awards 1st place adapting the national institute for health and care excellence (nice) organization for economic co-operation and development (oecd) countries’ geographic search filters (medline and embase, ovid): developing low-and middle-income countries’ (lmic) filters. author: lynda ayiku, information specialist. national institute for health and care excellence (nice), manchester, england, united kingdom. 2nd place beyond the conference: health sciences librarians’ motivations for publishing. author: rachel j. hinrichs, health sciences librarian. iupui university library, indianapolis, indiana. 3rd place analyzing changes in work-from-home policies for academic medical and health sciences librarians authors: david peterson, assistant professor, research and learning services librarian. preston medical library, knoxville, tennessee. matthew covey, university 1 https://www.orcid.org/0000-0002-8234-8611 https://orcid.org/ mailto:leblake@uams.edu http://www.mlanet.org/p/cm/ld/fid=938 https://doi.org/10.5195/jmla.2021.1115 editorial reviewed article hypothesis , vol. 34, no. 1, 2022 librarian. the rita and frits markus library at rockefeller university, new york, new york. janet crum, director. health sciences library, tucson, arizona. mla 2022 annual meeting contributed poster awards 1st place lgbtq+ health research guides: a cross-institutional pilot study of usage patterns. authors: gregg a. stevens, manager of education and clinical services. lamar soutter library, university of massachusetts chan medical school, worchester, massachusetts. francisco j. fajardo, information and instruction services librarian. herbert wertheim college of medicine medical library, miami, florida. martin morris, associate librarian. schulich library of physical sciences, life sciences, and engineering, montreal, quebec, canada. jessica berry, technical services librarian. kansas city university, kansas city, missouri. robin m. n. parker, evidence synthesis librarian. dalhousie libraries, halifax, nova scotia. katie d. mclean, librarian educator. nova scotia health, halifax, nova scotia, canada. 1st place news coverage of public health research: an exploratory study of topics, coverage, and open access status. authors: mirian ramirez rojas, research metrics librarian. ruth lilly medical library, indianapolis, indiana. rachel j. hinrichs, health sciences librarian. iupui university library, indianapolis, indiana. 3rd place cyberchondria and health information seeking: a scoping review. authors: amber burtis, assessment and user experience librarian. susan howell, cataloging and metadata librarian. southern illinois university. carbondale, illinois. 2 peer reviewed article hypothesis , vol. 35, no. 2, 2023 a graphic medicine reading experience for undergraduate nursing students: a brief report michelle nielsen otta aassistant professor, sciences/health sciences librarian, cullom-davis library, bradley university, peoria, illinois, https://www.orcid.org/0000-0002-2730-2928 , mnielsenott@bradley.edu health humanities, a growing field combining humanities and health sciences, aids healthcare students and providers in developing self-reflection, empathy, personal well-being, communication skills, and observational skills. developing these skills is important for students as they form their professional identity. one way to engage in health humanities and connect with faculty and students is through graphic medicine, which are illustrated narratives such as comics that involve health topics. as the science and health science librarian at a comprehensive university, i have been actively adding graphic medicine titles to the library’s collection which prompted me to investigate what effect reading a graphic medicine book has on undergraduate nursing students’ empathy and self-reflection skills. lgbtqia-focused efforts on campus and recent political debates about transgender healthcare informed my selection of the graphic medicine book first year out: a transition story by sabrina symington for this study. as there is currently no lgbtqia-focused nursing course available at bradley university, liaison contacts within the nursing department helped me identify the community health practicum course as a suitable setting with supportive instructors for this study. students were assigned to read first year out and answer a set of reflection questions i developed as the investigator. the library purchased an unlimited user copy of the e-book so students had immediate, no-cost, access to the title. the course instructors graded the student reflection assignments as complete or incomplete, and not on the content of the responses, to reduce bias. students voluntarily consented for their responses to be included in this study. faculty members removed identifying information before sharing the reflection documents. the student reflections were overwhelmingly positive in response to reading the book. i was pleasantly surprised since this was an extra assignment compared to previous iterations of this class. nursing students carry a very demanding course load and i was concerned that they might not take this assignment as seriously as their other assignments, but this was not the case. i was honored to read their carefully crafted reflections. initial findings showed that students related to the book’s main character in a variety of ways including expressing similar experiences with anxiety and body dysmorphia. 1 https://www.orcid.org/0000-0002-2730-2928 https://orcid.org/ mailto:mnielsenott@bradley.edu peer reviewed article hypothesis , vol. 35, no. 2, 2023 they also recognized shared traits with the main character such as determination and persistence. students expressed that the comic format was enjoyable and helped with their comprehension of the material. additionally, students found reading this book to be beneficial for their future as healthcare professionals, especially regarding patient interactions. students expressed many empathetic thoughts and feelings as they practiced self-reflection in this assignment. regarding future work, the community health practicum course will continue to include this reading experience and may add more graphic medicine titles to the course. i plan to partner with additional faculty members in the undergraduate nursing program and other health science programs to expand the use of graphic medicine titles in the curriculum to enhance student learning. the lead classroom faculty member and i will report more about this study in a forthcoming research article. 2 peer reviewed article hypothesis, vol. 36, no. 1, 2024 public health research in the news: an exploratory study of topics, coverage, and open access status mirian ramirez, mlisa, rachel j. hinrichs ms, mslsb, aresearch metrics librarian, ruth lilly medical librarian, indiana university school of medicine, indianapolis, indiana, https://www.orcid.org/0000-0002-6233-7480 , mirirami@iu.edu bhealth sciences librarian, university library, indiana university, indianapolis, indiana, https://orcid.org/0000-0003-0762-744x , rhinrich@iu.edu cite as: ramirez m, hinrichs rj. public health research in the news: an exploratory study of topics, coverage, and open access status. hypothesis. 2024;36(1). doi:10.18060/27264 ramirez, hinrichs. all works in hypothesis are licensed under a cc by-nc 4.0 deed attribution-noncommercial 4.0 international. authors own copyright of their articles appearing in hypothesis. readers may copy articles without permission of the copyright owner(s), as long as the author(s) are acknowledged in the copy, and the copy is used for educational, not-for-profit purposes. for any other use of articles, please contact the copyright owner(s). 1 https://www.orcid.org/0000-0002-6233-7480 https://orcid.org/ mailto:mirirami@iu.edu https://orcid.org/0000-0003-0762-744x https://orcid.org/ mailto:rhinrich@iu.edu https://creativecommons.org/licenses/by-nc/4.0/ https://creativecommons.org/licenses/by-nc/4.0/ peer reviewed article hypothesis, vol. 36, no. 1, 2024 abstract objectives: this study aims to identify and analyze the news outlets that reference research published by the richard m. fairbanks school of public health at indiana university-purdue university indianapolis (fsph). we analyzed the overall patterns of news sources, including the content type and overall coverage, the correlation with open access (oa) status, and the topics that received more attention in online news stories. we aim to contribute to a better understanding of the characteristics of news outlets that disseminate public health information, and the types of public health research referenced most often in the news. methods: we conducted a retrospective cohort study. the study population consisted of fsph authors, the exposure was publishing journal articles by fsph authors, and the outcome was news media attention. we compared the amount of media attention with other studies. we also compared open access journals versus commercially produced journals. we searched the scopus database using an affiliation search to locate and retrieve the journal articles. then we queried the altmetric explorer tool (altmetrics provider) for all the citations resulting from the scopus search. we conducted a bibliometric analysis of publications from authors affiliated with the school of public health over five years (2015–2019). the searches were conducted in december 2021. we used the identifiers and the titles to compile the online news mention data and identify the titles featured in news outlets. the journal articles with news mentions and the news outlets that reference them were analyzed. the data were combined, cleaned up, and analyzed using excel, and the subject analysis was produced using the vosviewer software. results: for the 2015–2019 time frame, 778 publications were published by the fsph. a total of 144 (18.5%) were highlighted in 2,079 online news stories from 655 news outlets. we analyzed the overall dataset of news outlets according to the source, content type, subject area, and country of publication. we also determined the attention likelihood of open access articles compared to those with closed access. we performed a co-occurrence analysis of the mesh terms the articles featured in news outlets. discussion: mentions in online news outlets can identify patterns and trends of attention and interest in research beyond academia. this bibliometric analysis helps to clarify the media outlets that distribute information on public health as well as the categories of research that are frequently featured in the media. the findings of this report, will help the library expand its research impact services and support the decision-making of leaders of public health schools for future publication and research strategy development directions. purpose and background alternative impact indicators (or altmetrics) provide quantitative and qualitative data that highlight online attention (e.g., social media coverage, mentions in news stories, citations on wikipedia) and influence of scholarly outputs. altmetrics offer opportunities and advantages over traditional citation metrics for assessing impact, especially for measuring the societal impact of research and public engagement with scholarly work1. in particular, news media-based altmetrics (mentions in news) can demonstrate the reach and attention the research received beyond academia, especially if it has a specialized subject focus or covers a 2 peer reviewed article hypothesis, vol. 36, no. 1, 2024 wide geographical area1–4. for medical and health sciences research, mass-circulation news sources offer a way to connect with and interact with a wide audience as part of knowledge translation and dissemination processes5. in this analysis, we look at the characteristics of the online news sources that covered research conducted by the researchers of the iu richard m. fairbanks school of public health at indiana university-purdue university indianapolis (fsph) as well as the open access status of the articles that were referenced in the news. the results of this study can help public health school leaders make decisions about future publication, knowledge translation, dissemination, and research strategy directions. several studies address online media attention, including blogs, social media, and others, to evaluate the degree of correlation of mentions or altmetric scores with the number of citations in various health sciences specialties6–12 or general scientific journals13. using bibliometrics and altmetric analysis, more studies have investigated the connection between citations and altmetric indicators. dumas-mallet et al14. analyzed the impact of english newspaper coverage from 6 countries on the citation counts of biomedical studies published in 38 different peer-reviewed journals with impact factors (ranging from jif 5 to 51). their findings evidence a positive association between newspaper coverage and the citation count of a scientific paper. further, palamar & strain7 analyzed traditional and social media coverage of manuscripts focusing on substance use and found that news and social media coverage were positively associated with a number of downloads and citations. according to their results, 13.9% of the articles were featured in news sources, with epidemiology papers receiving the most coverage. few studies have focused on the characteristics of the news sources that highlight research articles in mass media. one example is a paper by moorhead et al. that analyzed the characteristics of news sources featuring cancer research16. in this study, the authors reviewed cancer research articles mentioned in the news and classified the news sources as either top traditional news sites or top digital-native sites. their findings reveal that 1.88% of journal articles received at least one online news mention (totaling 642 mentions across 213 journal articles) and were cited by conventional news outlets much more frequently than digital-native and niche publications. according to the authors, the results suggest that prominent online news media outlets continue to under-represent research that focuses on cancer prevention and detection. open versus paywalled access to research articles may play a role in whether an article is mentioned in the news. there are several types of open access articles. gold open access means that the journal provides immediate open access to all its articles and may sometimes charge a fee to the authors17. green open access, on the other hand, means that a version of the publication is archived online, such as in a repository, while the final, published version of the article may or may not be open access17. hybrid gold open access articles are published in paywalled journals, but the authors pay a fee to make their article open access17. the literature has extensively discussed the citation advantage of open access publications over paywalled articles. according to a study by dehdarirad and didegah13, the likelihood that a paper will be shared on social media is significantly correlated with its open access status. they compared the social media exposure of open access versus non-open access, and gold against other types of open access articles in the life sciences and biomedicine fields. their 3 peer reviewed article hypothesis, vol. 36, no. 1, 2024 findings demonstrated that open access was strongly associated with a higher likelihood of a paper being mentioned on the analyzed social media platforms and that the percentage of open access papers mentioned on altmetric platforms was higher than that of non-open access articles. the open access articles had a greater average number of tweets, facebook postings, news posts, and blog posts compared to non-open access items. further, vadhera et al.18 found that open access publications published in orthopedic journals obtained noticeably more citations and social and news media attention than subscription-based studies. schultz19 investigated how much coverage open access scholarly articles from different disciplines get in the news media. the author found evidence that there is a positive correlation between the use of gold, green, and hybrid open access over paywalled articles; articles focusing on health received more news mentions after general sciences subject articles. it should be noted that in the united states, the national institutes of health (nih) open access policy20 plays a significant role in influencing the visibility and impact of research articles. any publication resulting from the nih-funded research is required to be made freely and publicly accessible through pubmed central within a year of the official publication date. to the best of our knowledge, there is no analysis of news media attention specific to a school of public health, which publishes a variety of research across public health domains. therefore, this study aims to identify and analyze the news outlets that mentioned research published between 2015 and 2019 by the iu richard m. fairbanks school of public health at indiana university-purdue university indianapolis (fsph) to answer the following research questions: 1. how many articles published by the fsph are mentioned in news stories? 2. what are the characteristics of news sources that mention the fsph’s public health research articles (type of news source, subject area, geographical scope, language of origin)? 3. what research topics covered by the fsph’s articles are most likely to be mentioned in news sources? 4. are open access articles published by fsph more likely to be referenced in news sources compared to those with paywalled access? to answer these questions, we conducted a retrospective cohort study that employed bibliometric analysis to identify the overall patterns of the news sources that reference or mention the articles: type of news outlet, country of publication, language, geographical coverage, and subject area. we also determined whether gold or hybrid gold open access scholarly articles were discussed more often in the news than subscription-access articles. our results will contribute to a better understanding of the characteristics of news outlets that disseminate public health information and the types of research mentioned most often in the news. methods and materials we conducted a retrospective cohort study. the population consisted of fsph authors, the exposure was publishing journal articles, and the outcome was news media attention. we compared the amount of news media attention with other studies. we also compared open access journals versus subscription journals. we conducted a bibliometric analysis, with a 4 peer reviewed article hypothesis, vol. 36, no. 1, 2024 focus on altmetrics (or alternative metrics), of publications from authors affiliated with the fsph over five years (2015-2019). we chose to analyze news articles published prior to the covid-19 pandemic as we suspected that the content of the research articles, open access status, and quantity of news mentions would likely change after 2020, and we wanted to establish a baseline. the data in this paper does not involve human subjects and is exempt from irb approval. data collection publications by researchers affiliated with the fsph over five years (2015-2019) were retrieved as raw .csv data files from the scopus database, which lists all authors’ full addresses. while scopus relies on various sources for indexing data and covers a wide range of disciplines and publications, it does not comprehensively index all relevant journals and publications in all fields. to collect the data, we queried the scopus database to find documents authored by researchers affiliated with the fsph; we developed an advanced search string using the affiliation field to catch variations of the school’s name in the affiliations used by the authors of the publications. queries were made using the advanced search option using the affil() search field code and restricted to the timeframe covered in the study: affil("fairbanks school of public health" or "fairbanks school of health") and (limit-to(pubyear , 2019 ) or limit-to(pubyear, 2018) or limit-to(pubyear, 2017) or limit-to(pubyear, 2016) or limit-to (pubyear, 2015)). the data was downloaded in december 2021. to identify the articles mentioned in news outlets, we used the altmetric explorer for academic librarians (https://www.altmetric.com/explorer/login), using the scholarly identifiers, dois, and pubmed ids available for all the publications collected in scopus. the altmetric explorer (academic librarians version) tool (https://www.altmetric.com) serve as an altmetric data aggregator that tracks mentions of research outputs using scholarly identifiers (e.g., dois, pubmed ids) of over 4,000 online news and mainstream media outlets around the world21. we used the import.io (https://www.import.io/) web tool (trial version), which extracts online data from websites, to extract the news that references the articles from the altmetric.com details page (image 1) for each article that identified news mentions. the data collected included the following fields: source name, news url, news headline/title, date, and altmetric.com url details page. data was exported to excel spreadsheets and further aggregated to generate a list of unique news sources. to determine the articles’ open access status, we used the data provided by scopus on open access status and the unpaywall tool22. we identified articles that were either gold or hybrid gold open access articles, which means that the article would be open access at the time of publication. green open access articles are made publicly accessible by archiving a copy of the article in a repository. however, there was no way for us to tell when the article had been deposited, so we could not be sure that the article was openly available when the news media were accessing it. 5 https://www.altmetric.com/explorer/login https://www.altmetric.com https://www.import.io/ peer reviewed article hypothesis, vol. 36, no. 1, 2024 image 1. overview of the details page showcasing newscoverage tracked by altmetric.com https://www.altmetric.com/details/4768439/news data analysis the analysis was based on the data from documents identified on altmetric.com using scholarly identifiers. to ensure the authors’ inclusion in the analysis, we examined the articles referenced in the news sources to double-check the authors’ institutional affiliation. one of the major tasks in the data analysis was to tag all the identified news sources with the various categories we created. to categorize and code the news sources, we created categories (table 1) that could group the type of news outlet, geographical scope, and subject area. to gather information we primarily rely on wikipedia entries related to news sources as they often contain detailed descriptions of various media outlets present in our dataset.) additionally we used the online ulrichsweb serials directory (https://ulrichsweb.serialssolutions.com/) to check information on the source. we coded each news source primarily using the descriptions of the sources found in the "about" section of the websites. the serial type was determined by the descriptions from the websites mentioned previously. the geographical scope was obtained mainly from the description of the source. the subject area was determined by analyzing the source’s main point of interest. the language was identified by navigating to the source’s website or reading the description (usually found under ’about’) provided then on its website; we used the iso 639.22323 language names for coding the sources. the list of news sources was examined by both researchers, who noted any discrepancies and attempted to reconcile them. table 1. categories were created for coding news sources: serial type, geographical scope, and subject area 6 https://www.altmetric.com/details/4768439/news https://ulrichsweb.serialssolutions.com/ peer reviewed article hypothesis, vol. 36, no. 1, 2024 serial type geographical scope subject area news website local/city arts, culture, and entertainment online magazine state business & economics online newspaper regional consumer health radio national general interest/mass circulation newspapers scholarly journal international trade journal politics website public health health policy science technology to create the topic analysis of the articles referenced in the news, we queried the pubmed database to find and collect the mesh terms of the articles. full pubmed records were exported to vosviewer software (version 1.6.17) to run the mesh term co-occurrence analysis. the results of the term co-occurrence analysis were synthesized in a chart with topics hierarchically structured, using the rawgraphs online tool (version 2.0). to generate the topic co-occurrence analysis, the following mesh categories were excluded: gender (female, male), organisms (humans, animals), and geographic-related terms (countries, regions). we used a chi-square test to determine if there was a significant association between gold or hybrid gold open access status and news mentions. p < 0.05 was considered as significant. results mentions in news outlets overall, a total of 778 articles were published by the fsph during 2015-2019 (figure 1). we identified a total of 144 (18.5%) documents that were referenced in 2,079 news articles from 643 news outlets (until the day of final search and extraction, december 2021). for articles referenced in the news, on average, each document was mentioned 14 times; 99 articles were mentioned more than once. among the top journals with the most articles mentioned in news stories with (greater than or equal to 100 mentions) are: circulation, nature genetics, jama network open, new england journal of medicine, nature, and alzheimer’s and dementia (table 2). yahoo/yahoo news is the news source that mentioned the most articles (table 3), with 78 news stories identified. 7 peer reviewed article hypothesis, vol. 36, no. 1, 2024 figure 1: distribution of articles published for the period 2015–2019: comparison of the 778 documents published vs. the 144 (18.5%) documents mentioned in the news. 8 peer reviewed article hypothesis, vol. 36, no. 1, 2024 table 2. top 20 journals publishing the most mentioned articles, published by the fsph, in the news. journal title number of number of citation mentions in news articles count circulation 255(12.2%) 2 133 nature genetics 203 (9.7%) 6 834 jama network open 169 (8.1%) 1 33 new england journal of medicine 129 (6.2%) 3 379 nature 127 (6.1%) 2 396 alzheimer’s and dementia: translational research and clinical interventions 100 (4.8%) 1 58 radiology 68 (3.2%) 1 20 cancer epidemiology biomarkers and prevention 61 (2.9%) 1 13 nature communications 61 (2.9%) 7 548 hepatology 57 (2.7%) 1 20 jama journal of the american medical association 52 (2.5%) 2 142 american journal of industrial medicine 48 (2.3%) 1 5 journal of infectious diseases 47 (2.2%) 1 61 mbio 45 (2.1%) 1 63 hypertension 45 (2.1%) 1 98 environmental health: a global access science source 39 (1.8%) 1 69 sexual health 39 (1.8%) 1 45 american journal of clinical nutrition 39 (1.8%) 2 94 journal of the american medical informatics association 38 (1.8%) 8 196 jama pediatrics 24 (1.1%) 2 69 9 peer reviewed article hypothesis, vol. 36, no. 1, 2024 ta bl e 3. to p 10 ne w s so ur ce s th at m en tio ne d th e m os ta rt ic le s pu bl is he d by th e fs ph . n am e of n ew ss ou rc e u r l su bj ec ta re a c ou nt o f n ew ss to ri es y ah oo ! o ry ah oo ! n ew s ne w s. ya ho o. co m / g en er al in te re st /m as s ci rc ul at io n 78 (3 .8 % ) m sn w w w .m sn .c om g en er al in te re st /m as s ci rc ul at io n 56 (2 .7 % ) m ed ic al x pr es s m ed ic al xp re ss .c om m ed ic in e 47 (2 .3 % ) e ur ek a le rt ! w w w .e ur ek al er t.o rg sc ie nc e te ch no lo gy 43 (2 .1 % ) m ed ic al h ea lth n ew s w w w .m ed ic al he al th ne w s. ne t c on su m er h ea lth 43 (2 .1 % ) t he n ew a ge (z a ) w w w .th en ew ag e. co .z a g en er al in te re st 39 (1 .9 % ) /m as s ci rc ul at io n ph ys ic ia n’ s b ri efi ng w w w .p hy si ci an sb ri efi ng .c om m ed ic in e 31 (1 .5 % ) h ea lth m ed ic in e he al th m ed ic in et .c om m ed ic in e 28 1. 3% ) m ed ic al n ew s to da y w w w .m ed ic al ne w st od ay .c om m ed ic in e 27 (1 .3 % ) t he m ed ic al n ew s w w w .n ew sm ed ic al .n et m ed ic in e 23 (1 .1 % ) 10 peer reviewed article hypothesis, vol. 36, no. 1, 2024 news sources characteristics the news sources vary by serial type, subject area, geographical scope, and language. most of the news sources identified are news websites, followed by web versions of general interest/mass-circulation newspapers, mainly with a national geographical scope. over half of the news sources fall under the "news websites" or "online newspapers" categories (figure 2). we also identified that specialized news sources focusing on medicine, science technology, consumer health, and public health health policy perspectives represent 26% of the news sources, which is considerably low compared to those with general interest or mass circulation types at 62% (figure 3). the percentage of trade journals, scholarly journals, and online magazines referencing the articles represents 16% of the news sources. the most common geographical scope of the news sources is national (44%), followed by international scope (figure 4). the country of origin of the news sources includes the united states, the united kingdom, australia, and india, and 81.03% corresponding to english-only sources, 4.04% to multiple languages (including english), and near 15% to a non-english language source (figure 5 and table 4). figure 2: distribution of news sources by type 11 peer reviewed article hypothesis, vol. 36, no. 1, 2024 figure 3: distribution of news sources by subject area figure 4: distribution of news sources by geographical scope of news sources 12 peer reviewed article hypothesis, vol. 36, no. 1, 2024 fi gu re 5: d is tr ib ut io n of ne w s so ur ce s id en tifi ed ac ro ss 45 co un tr ie s 13 peer reviewed article hypothesis, vol. 36, no. 1, 2024 table 4. distribution of news sources by language language total news sources percentage english 521 81.03% multiple languages 26 4.04% german 20 3.11% french 14 2.18% italian 12 1.87% spanish 12 1.87% dutch 7 1.09% portuguese 7 1.09% russian 4 0.62% chinese 4 0.62% swedish 3 0.47% hungarian 2 0.31% finnish 2 0.31% norwegian 2 0.31% greek 2 0.31% polish 1 0.16% japanese 1 0.16% danish 1 0.16% arabic 1 0.16% hindi 1 0.16% grand total 643 100.00% topics of fsph articles mentioned in news the identified research papers mentioned in the news covered a wide range of topics. the data in figure 6 represents the nodes of mesh terms with equal or greater than 8 co-occurrences in articles mentioned in news sources. we grouped the nodes in a hierarchical structure classified by broader mesh categories: diseases, phenomena and processes, epidemiological methods, and age groups. the size of the nodes is given by the number of co-occurrences of the mesh terms assigned to the articles featured in the news. the most common topics of articles referenced in the news were genome-wide association study, genetic predisposition to disease, risk factors, and skin neoplasms; and within age groups, middle age and adults were mostly assigned. 14 peer reviewed article hypothesis, vol. 36, no. 1, 2024 fi gu re 6: m es h te rm s w ith gr ea te rt ha n or eq ua lt o 8 co -o cc ur re nc e in ar tic le s by th e fs ph m en tio ne d in n ew s so ur ce s. 15 peer reviewed article hypothesis, vol. 36, no. 1, 2024 figure 7: distribution of articles mentioned in news according to gold/hybrid and gold open access status open access status of articles published by the fsph with number of news mentions overall, 202 documents (26%) out of 778 have gold or hybrid gold open access status. only 32 (15.8%) of the open access articles were featured in the news, whereas 112 (19.4%) of the paywalled articles were mentioned (figure 7). using a chi-square test, we found no significant association between gold/hybrid gold open access status and being mentioned in the news (n = 778; x = 1.29, df = 1, p = .26). discussion we found that 18.5% of articles published by the fsph from 2015-2019 were referenced in the news. this percentage is in line with many prior studies examining the rates of media coverage of scholarly literature, ranging from 13.9% to 17%19,24,25. many of those same studies used different criteria for identifying research articles, such as high impact journals19, a sample of cancer research articles selected from the medline database25, or united states government-funded research16,24. our study, on the other hand, focused on the research output of one school of public health, which produces a wide range of public health research in areas such as health policy, health services, environmental health, epidemiology, genomics, and behavioral health. that our percentage of articles mentioned in the news is similar could mean that faculty from the fsph publish in high impact journals, have government funding, or are writing on topics that generate media interest. based on the topic analysis of research highlighted in the news, the most common mesh categories were epidemiologic methods and certain diseases, skin neoplasms being the most prominent. this finding is in line with other studies that found that epidemiological studies gain more news mentions than non-epidemiology studies18,19, and that melanoma, a type of skin cancer, is the third most mentioned cancer type in the news16. we also found that the fsph’s research was most likely to be mentioned in general interest 16 peer reviewed article hypothesis, vol. 36, no. 1, 2024 news websites with a national united states-focused coverage. our category of news website refers to digital native news sources as opposed to traditional newspapers that have an online version such as new york times or wall street journal. the fact that news websites have overtaken online newspapers in our study as the dominant reporter of public health news suggests that researchers should be open to targeting online news organizations rather than focusing only on traditional newspapers. these online news organizations may be curating their content to specific groups, and so could be more likely to report on certain public health topics. for example, while 62% of the news sources were “general interest,” the remaining 38% of news sources reported on niche areas such as medicine (12%), science & technology (8%), business and economics (6%), and even arts, culture, and entertainment (3%). while most articles were discussed in english-language news sources, 20% were discussed in non-english or multi-language news sources. using news sources to demonstrate reach to a large, mass-interest audience or a smaller niche-audience could be beneficial to a researcher or public health school in determining research attention and impact. while national news sources were the most common in this study (44%), local news sources should not be overlooked when it comes to public health. we found that 18% and 13% of news sources covered local/city and state news, respectively. when considering research dissemination, audience size is important, but may not take into account local community health needs or demographics that may be more interested in certain types of research16. a pew research center study found that older americans, black adults, and those with a high school education or less are more likely to be interested in local news26. community health coalitions and public health practitioners working with nonprofit organizations or small local health departments may have limited access to paywalled research27, and could benefit from learning about new research from local news sources, especially if the research has local implications. future studies could consider what public health news is considered of local importance, and how local newspapers select and curate the health research news they include. a surprising finding of our study was that open access articles were not more likely to be mentioned than paywalled articles. this finding contrasts with many other studies that found that open access studies have higher mentions in the news and social media18,19,28. our study may not have had a big enough sample size to detect a difference. another reason could be that we only counted gold and hybrid gold studies as open access because we knew they would be open access at the time of publication. many of the studies that we labeled as paywalled could have been available through green open access methods such as a deposit in an institutional repository or pubmed central at the time of the news mention, but there was no way for us to determine that. in her analysis, schultz found that while green and hybrid open access articles together comprised 6.4% of her sample, they represented 23.3% of the articles with news mentions, whereas gold open access articles were 66.6% of the sample, but only represented 53.8% of the news mentions19. if green open access plays an important role in providing access to scientific articles for the news media, our study may have underestimated the association between open access and news mentions. in addition to this potential underestimation between open access and news mentions, our study had several other limitations. first, our study only included research from one school of public health rather than sampling from published public health articles. this means our results cannot be generalized to other public health schools or public health research in 17 peer reviewed article hypothesis, vol. 36, no. 1, 2024 general. however, it does demonstrate a replicable method for other schools of public health to track media attention to their own research. second, it is possible that we could have misclassified a news source, as it was sometimes difficult to determine the geographical scope of coverage, and the difference between a news website and online newspaper. third, while almetric.com has been found to have a high precision and acceptable recall for news mentions, it may have missed some news mentions that do not link to the research articles, mention the author or journal, or are in non-english media29. because of this, our estimate of 18.5% news coverage may be conservative. conclusion the results of this retrospective cohort study provide an overview of the patterns and trends of attention and interest in research beyond academia of articles published by a school of public health. this bibliometric analysis contributes to a better understanding of news outlets that disseminate public health information and the types of research mentioned most often in the news. this study’s insights are valuable for expanding the library’s research impact services by developing a pilot school-level report based on altmetrics for internal research departments and for schools of public health interested in tracking news mentions to their published research. acknowledgments the authors would like to thank altmetric.com for allowing them access to their data through the altmetric explorer for academic librarians for this study. references 1. thelwall m. measuring societal impacts of research with altmetrics? common problems and mistakes. j econ surv. 2021 dec;35(5):1302–14. doi:10.1111/joes.12381 2. kortelainen t. what is societal impact and where do altmetrics fit into the equation? 2019 dec 18;2(1):6. doi:10.29024/joa.21 3. kassab o, bornmann l, haunschild r. can altmetrics reflect societal impact considerations?: exploring the potential of altmetrics in the context of a sustainability science research center. quant sci stud. 2020 mar 10;1–18. doi:10.1162/qss_a_00032 4. ortega jl. how do media mention research papers? structural analysis of blogs and news networks using citation coupling. j informetr. 2021 aug;15(3):101175. doi:10.1016/j.joi.2021.101175 5. jaques h. get your research reported well in the news. bmj. 2011 jan 19;c6921. doi:10.1136/bmj.c6921 6. wang j, alotaibi nm, ibrahim gm, kulkarni av, lozano am. the spectrum of altmetrics in neurosurgery: the top 100 “trending” articles in neurosurgical journals. world neurosurg. 2017 jul 1;103:883-895.e1. doi:10.1016/j.wneu.2017.04.157 7. rosenkrantz ab, ayoola a, singh k, duszak r. alternative metrics (“altmetrics”) for assessing article impact in popular general radiology journals. acad radiol. 2017 18 https://dx.doi.org/10.1111/joes.12381 https://dx.doi.org/10.29024/joa.21 https://dx.doi.org/10.1162/qss_a_00032 https://dx.doi.org/10.1016/j.joi.2021.101175 https://dx.doi.org/10.1136/bmj.c6921 https://dx.doi.org/10.1016/j.wneu.2017.04.157 peer reviewed article hypothesis, vol. 36, no. 1, 2024 jul;24(7):891–7. doi:10.1016/j.acra.2016.11.019 8. jack ra, sochacki kr, morehouse ha, mcculloch pc, lintner dm, harris jd. correlation between quality of evidence and number of citations in top 50 cited articles on elbow medial ulnar collateral ligament surgery. orthop j sports med. 2018 apr;6(4):2325967118768216. doi:10.1177/2325967118768216 9. goyal dkc, galetta ms, fang t, mangan j, lendner m, vaccaro ar, et al. the impact of social media in orthopaedics. curr orthop pract. 2019 sep;30(5):419–22. doi:10.1097/bco.0000000000000788 10. kunze kn, richardson m, bernstein dn, premkumar a, piuzzi ns, mclawhorn as. altmetrics attention scores for randomized controlled trials in total joint arthroplasty are reflective of high scientific quality: an altmetrics-based methodological quality and bias analysis. j am acad orthop surg glob res rev. 2020 dec;4(12):e20.00187. doi:10.5435/jaaosglobal-d-20-00187 11. abdullah y, mathew aj, alokozai a, stamm ma, mulcahey mk. positive association between online attention and the bibliometric impact of shoulder instability publications. arthrosc sports med rehabil. 2022 aug 1;4(4):e1539–44. doi:10.1016/j.asmr.2022.04.023 12. kirloskar km, civilette md, rate wr, cohen as, haislup bd, nayar sk, et al. the 50 most impactful articles on the medial ulnar collateral ligament: an altmetric analysis of online media. sage open med. 2022 nov 24;10:20503121221129920. doi:10.1177/20503121221129921 13. huang w, wang p, wu q. a correlation comparison between altmetric attention scores and citations for six plos journals. van den besselaar p, editor. plos one. 2018 apr 5;13(4):e0194962. doi:10.1371/journal.pone.0194962 14. dumas-mallet e, garenne a, boraud t, gonon f. does newspapers coverage influence the citations count of scientific publications? an analysis of biomedical studies. scientometrics. 2020 apr;123(1):413–27. doi:10.1007/s11192-020-03380-1 15. palamar jj, strain ec. news and social media coverage is associated with more downloads and citations of manuscripts that focus on substance use. drug alcohol depend. 2021 jan 1;218:108357. doi:10.1016/j.drugalcdep.2020.108357 16. moorhead l, krakow m, maggio l. what cancer research makes the news? a quantitative analysis of online news stories that mention cancer studies. plos one. 2021 mar 10;16(3):e0247553. doi:10.1371/journal.pone.0247553 17. barnes l. green, gold, diamond, black – what does it all mean? [internet]. open book publishers blog. 2018 [cited 2023 jun 16]. doi:10.11647/obp.0173.0089 18. vadhera as, lee js, veloso il, khan za, trasolini na, gursoy s, et al. open access articles garner increased social media attention and citation rates compared with subscription access research articles: an altmetrics-based analysis. am j sports med. 19 https://dx.doi.org/10.1016/j.acra.2016.11.019 https://dx.doi.org/10.1177/2325967118768216 https://dx.doi.org/10.1097/bco.0000000000000788 https://dx.doi.org/10.5435/jaaosglobal-d-20-00187 https://dx.doi.org/10.1016/j.asmr.2022.04.023 https://dx.doi.org/10.1177/20503121221129921 https://dx.doi.org/10.1371/journal.pone.0194962 https://dx.doi.org/10.1007/s11192-020-03380-1 https://dx.doi.org/10.1016/j.drugalcdep.2020.108357 https:\dx.doi.org/10.1371/journal.pone.0247553 https://dx.doi.org/10.11647/obp.0173.0089 peer reviewed article hypothesis, vol. 36, no. 1, 2024 2022 nov;50(13):3690–7. doi:10.1177/03635465221124885 19. schultz t. all the research that’s fit to print: open access and the news media. quant sci stud. 2021 nov 5;2(3):828–44. doi:10.1162/qss_a_00139 20. national institutes of health. nih public access policy details [internet]. 2009 [cited 2023 apr 4]. available from: https://publicaccess.nih.gov/policy.htm 21. altmetric limited. our sources [internet]. altmetric. 2015 [cited 2023 feb 12]. available from: https://www.altmetric.com/about-our-data/our-sources-2/ 22. priem j, piwowar h. the unpaywall dataset. 2018;95760 bytes. 23. international organization for standardization. iso 639-2 language code list codes for the representation of names of languages (library of congress) [internet]. 2017 [cited 2023 jan 15]. available from: https://www.loc.gov/standards/iso639-2/php/code_list.php 24. maggio la, ratcliff cl, krakow m, moorhead ll, enkhbayar a, alperin jp. making headlines: an analysis of us government-funded cancer research mentioned in online media. bmj open. 2019 feb 1;9(2):e025783. doi:10.1186/s41073-017-0033-z 25. haneef r, ravaud p, baron g, ghosn l, boutron i. factors associated with online media attention to research: a cohort study of articles evaluating cancer treatments. res integr peer rev. 2017;2:9. doi:10.1186/s41073-017-0033-z 26. rosenberg s. older americans, black adults and americans with less education more interested in local news [internet]. pew research center’s journalism project. 2019 [cited 2023 apr 4]. available from: https://www.pewresearch.org/journalism/2019/08/14/olderamericans-black-adults-and-americans-with-less-education-more-interested-in-local-news/ 27. hunt sl, bakker cj. a qualitative analysis of the information science needs of public health researchers in an academic setting. j med libr assoc [internet]. 2018 apr 5 [cited 2023 mar 21];106(2). available from: http://jmla.pitt.edu/ojs/jmla/article/view/316 28. dehdarirad t, didegah f. to what extent does the open access status of articles predict their social media visibility? a case study of life sciences and biomedicine. j altmetrics. 2020 aug 12;3(1):5. doi:10.29024/joa.29 29. fleerackers a, nehring l, maggio la, enkhbayar a, moorhead l, alperin jp. identifying science in the news: an assessment of the precision and recall of altmetric.com news mention data. scientometrics. 2022 nov 1;127(11):6109–23. doi:10.1007/s11192-022-04510-7 20 https://dx.doi.org/10.1177/03635465221124885 https://dx.doi.org/10.1162/qss_a_00139 https://publicaccess.nih.gov/policy.htm https://www.altmetric.com/about-our-data/our-sources-2/ https://www.loc.gov/standards/iso639-2/php/code_list.php https://dx.doi.org/10.1186/s41073-017-0033-z https://dx.doi.org/10.1186/s41073-017-0033-z https://www.pewresearch.org/journalism/2019/08/14/older-americans-black-adults-and-americans-with-less-education-more-interested-in-local-news/ https://www.pewresearch.org/journalism/2019/08/14/older-americans-black-adults-and-americans-with-less-education-more-interested-in-local-news/ http://jmla.pitt.edu/ojs/jmla/article/view/316 https://dx.doi.org/10.29024/joa.29 https://dx.doi.org/10.1007/s11192-022-04510-7 hypothesis, vol. 37, no. 1, 2025 what should a first-time and emerging researcher know about publishing ethics? margaret a. hoogland, mls, ahipa aassociate professor and clinical medical librarian, university libraries, the university of toledo, toledo, ohio, https://www.orcid.org/0000-0002-9932-3605 , margaret.hoogland@utoledo.edu cite as: hoogland ma. what should a first-time and emerging researcher know about publishing ethics?. hypothesis. 2025;37(1). doi:10.18060/28783. hoogland. all works in hypothesis are licensed under a cc by-nc 4.0 deed attribution-noncommercial 4.0 international. authors own copyright of their articles appearing in hypothesis. readers may copy articles without permission of the copyright owner(s), as long as the author(s) are acknowledged in the copy, and the copy is used for educational, not-for-profit purposes. for any other use of articles, please contact the copyright owner(s). 1 https://www.orcid.org/0000-0002-9932-3605 https://orcid.org/ mailto:margaret.hoogland@utoledo.edu https://creativecommons.org/licenses/by-nc/4.0/ https://creativecommons.org/licenses/by-nc/4.0/ hypothesis, vol. 37, no. 1, 2025 as hypothesis begins recruiting first-time and emerging researcher submissions, i recalled my early publication experiences starting with an mlanews publication1 up through my first peer reviewed publication four years later in the journal of hospital librarianship2. when i read those early articles and conference abstracts, i can see how much my writing has changed and – i hope – improved over the years. back in 2014-2018, i wish i had known to ask the following questions: question: what does “a submission can only be considered by one journal at a time mean?” does this apply just to a manuscript? or does it apply to conference submissions as well? the submission checklist and author guidelines cover the main points a journal expects to see from its authors3. the acceptance rate can provide an idea of whether or not a journal gives any room for error (or misinterpretations) by an author. due to poor planning, for one submission i completed major revisions and response to reviewers the week of thanksgiving. as the hypothesis editor, i have chosen to communicate with author(s) about delays and inquire about sending notifications during a holiday. for the peer reviewers, i provide extra time from november 15-december 31st. if it becomes clear that we must push a submission to a future issue, then the author(s) will have extra time to make revisions. usually the delays are not the fault of the author(s) but rather recognizing and adjusting schedules to accommodate time needed by the editorial team. conferences are great opportunities to present ideas and to receive feedback from others. some conferences allow author(s) to submit conference abstracts, which have been submitted and even presented at a previous conference. on one occasion, i submitted an adjusted abstract to a conference and due to scheduling gave two different but very similar presentations back-to-back. some might have questioned if i was salami slicing, which is when you slice and dice content to make it “different enough” to justify giving multiple presentations4! as the presenter, you are at the mercy of the conference schedule – but i now limit the number of things i do at any given conference. unless each submission is on a separate topic or focuses on something introduced or referenced in passing in a previous presentation or poster, i will not submit a conference abstract. hypothesis states in many places the need for author(s) to confirm that the conference abstract has not been published elsewhere. if the conference abstract was presented but not published in any other journal or conference proceeding publication, this is an acceptable submission. if the conference abstract is stored in an institutional repository, authors should share this information at time of submission. to minimize the chance of duplicate publications, hypothesis will review publicly available conference proceedings from the journal of the canadian health libraries association, european association for health information and libraries, evidence based library and information practice, and lista. we might also do a cursory search of google scholar. questions: my promotion and tenure application requires x presentations and y peer reviewed publications. i really need to get something published soon. what suggestions do you have for me? 2 hypothesis, vol. 37, no. 1, 2025 as i was heading into my second year review for promotion with tenure, i had a rejected manuscript, a poster and panel presentation at a national conference, a state conference presentation, and an in-progress book chapter on my list of accomplishments. i was on the verge of panicking. during the holiday season, my parents and experienced researchers said, “you have sulked enough – get back on the horse!” then, in attempt to get me moving, they asked, “are you looking for a journal that works with newer authors?” these two questions led me to discovering the journal of hospital librarianship policy, which encourages submissions by newer authors, who could receive mentorship from editorial board members. question: i want to get experience as an author and researcher. what suggestions do you have for me? this is a tough one, because each person brings different experiences with them. i will tackle this by outlining my self-identified weak spots and steps i took to improve. writing when it came to scientific writing and learning to write in past-present tense, present tense, and active voice; i had a lot to learn. after conversing with tenured colleagues, i wrote book reviews, which allowed me to keep-up with trends and also add content to the existing library collections. writing a book chapter refined these skills even more as in one week i found and read materials, wrote, and revised about 3000 words. research study design and methods recognizing that i lack a basic knowledge of research design and methods, i looked for opportunities to collaborate with experienced researchers. at the end of a project, i have a pretty good idea of areas, which would benefit from further training. completing the critical appraisal institute for librarians enhanced my ability to analyze an article. reading the hypothesis research mentor article on statistical analyses and the more recent data bytes articles further improved my knowledge on the topic. they also made me realize that one of my studies will not be revised and submitted to a different journal, because the statistical analysis is wrong. of similar importance (and far easier to acknowledge), though, is that the project data is no longer current or of use to the university libraries, at the university of toledo. every researcher, director, project manager, and individual must develop the ability to recognize and to say, “this project, program, service, etc. no longer serves the purpose for which we designed it.” last fall, after being upset and discussing things with my dad about a situation, he said, “you have given this your best shot – it’s time to move on and to explore other options.” i hope, readers, you can find a person or people, who will be able to cut through the muddle, know you well, and say this to you. for me, giving up is something i avoid at all costs – i always end up feel defeated and fight the sense that i have let down the other team members. if it’s a solo endeavor, then i have to battle the feeling of failure. i am still – in many ways – working through the emotions from that decision, but the relief i felt was immediate. this confirmed to me that it was the right decision. 3 https://www.tandfonline.com/journals/whos20/about-this-journal#aims-and-scope hypothesis, vol. 37, no. 1, 2025 join or form a research, writing, etc. group i now get satisfaction from encouraging others to team up for research projects or to be a team member/facilitator, when i find a group with similar interests and availability. yes, it took me a while to figure out what i can bring to a team. it also took awhile for me to realize that my self-identification is usually on target. at the university of toledo, my collaborators are usually better than i am at knowing what skills i bring to a project. joining a group could assist in kickstarting a project or serve as a way to keep up momentum on a stalled project. tracking and productivity in fall 2019, i reconfigured my office white board to list all my ongoing research projects and things i needed to do. for a number of reasons (hello, covid!), this got out of date and i was just not using it effectively. i tried everything from to-do lists, writing a journal of what i did on a certain project (sometimes with hours spent on a task), and so forth. none of the above worked. on 02 january 2025, i cleaned off the white board and wrote the first five months of the year. then, i listed only the hard deadlines. instead of being discouraged or dreading what’s on the whiteboard, it’s now fun and satisfying to erase items on a semi-regular basis. tracking and productivity tools are lovely – but for me to use them, they have to demonstrate an easy way to track forward momentum. what’s hypothesis got for a new, emerging, or experienced author or researcher? read the update from the emerging researchers subcommittee report for more information about the upcoming special issue opportunity. for those interested in all things evidence syntheses, read the evidence syntheses subcommittee report. additional submission categories, which we think will capture the interest of all authors, will be available in the upcoming months and could be featured in the september issue. hypothesis welcomes submissions from new, experienced, or emerging researchers and authors. provided the content will be of use to the field of library and information science, our editorial team will support, mentor, and guide you every step of the way pre-submission to publication! be sure to browse the september editorial for topics more pertinent to mid-career and experienced researchers and authors. acknowledgments i thank and acknowledge alexandria q. wilson, lisa ennis, jennifer (jenn) monnin, terri gottschall, whose comments improved the design and content of this editorial. i cannot understate the importance and value i attribute to the many conversations with my parents, john l. and judy g. hoogland, who trained me to think scientifically, view red ink and comments and ways to improve my writing, and write. 4 hypothesis, vol. 37, no. 1, 2025 references 1. hoogland ma. using ibooks author to improve learning and interactivity. mla news. 2014;54(3):15. 2. hoogland ma. point of care products in health science classes. journal of hospital librarianship [internet]. 2018 oct 2 [cited 2025 jan 22];18(4):306–14. available from: https://www.tandfonline.com/doi/full/10.1080/15323269.2018.1509191 3. international committee of medical journal editors. preparing a a submission [internet]. available from: https://www.icmje.org/recommendations/browse/manuscriptpreparation/preparing-for-submission.html 4. brice j, bligh j, bordage g, colliver j, cook d, eva kw, et al. publishing ethics in medical education journals. acad med j assoc am med coll. 2009 oct;84(10 suppl):s132-134. 5 https://www.tandfonline.com/doi/full/10.1080/15323269.2018.1509191 https://www.icmje.org/recommendations/browse/manuscript-preparation/preparing-for-submission.html https://www.icmje.org/recommendations/browse/manuscript-preparation/preparing-for-submission.html hypothesis , vol. 35, no. 1, 2023 identifying and creating opportunities for growth margaret a. hoogland, mls, ahipa, aclinical medical librarian and associate professor, university libraries, the university of toledo, toledo, ohio, https://orcid.org/0000-0002-9932-3605 , margaret.hoogland@utoledo.edu in northwest ohio, the sun is beginning to shine, or at least peek through the clouds, on a more frequent basis. as the temperatures become more mild and new growth appears, we at hypothesis are pleased to share our march issue with you. editorial team updates it is my distinct pleasure to welcome our newest editorial team members: jennifer l. dean, who works at the university of michigan-flint, gary s. atwood , who works at the university of vermont, and michael greenlee, who works for the detroit area library network at wayne state university. jennifer and gary are joining the copy-editing team and michael will be assisting us with maintaining and improving the hypothesis website. introducing the brief reports submission category for the fall issue! as a new and emerging professional, the thought of putting together and then publishing an article seemed both intimidating and potentially insurmountable. my graduate work did not prepare me for entering academia or being in a tenure-track position. the brief reports submission category is our solution to these problems. to submit a brief report, an author will write and submit a structured manuscript of up to 500 words. the submission should include a separate attachment with title and author information. the format could resemble a failure, voices, research, or methods moment submission. we also welcome write-ups from previously presented (or not-presented or accepted) lightning talk, mini-talk, poster, oral abstract, or paper presentations. you can even write up a tool from a my favorite tool session! these submissions are peer-reviewed. pronouns in spring 2019, i observed people adding pronouns to their signatures and after their names in online meetings. when recruiting study participants or hosting events, i started including a field where attendees could share personal pronouns. in fall 2021, i began adding my personal pronouns to my email signature and this improved my communications and rapport with colleagues, faculty, staff, and even learners i support. during our november editorial team meeting, we discussed the feasibility of adding 1 https://www.orcid.org/0000-0002-9932-3605 https://orcid.org/ mailto:email https://www.orcid.org/0000-0002-9932-3605 https://orcid.org/ https://osf.io/prcz6/ hypothesis , vol. 35, no. 1, 2023 pronouns to submissions. we agreed to include them on the submission if authors choose to share this information. on the editorial team area of the journal website, we include this information upon request. many journals are not yet offering this opportunity. we hope our readers will recognize this as one way our journal supports diversity, equity, inclusion, and accessibility. marketing updates we encourage you to register as a reader with hypothesis, which keeps you in the loop, and ensures you are the first to know about any changes. starting in 2023, our marketing team will be posting news and updates on a regular basis. one year anniversary i thank you for your continued support of hypothesis and your patience with me this past year. it is both fun and challenging to serve as your editor. margaret a. hoogland, mls, ahip, she/her/hers; march 2023 2 https://journals.iupui.edu/index.php/hypothesis/about/editorialteam https://journals.iupui.edu/index.php/hypothesis/user/register peer reviewed article hypothesis , vol. 35, no. 2, 2023 evaluating the accuracy of scite, a smart citation index caitlin bakker, mlis, ahip-da, nicole theis-mahon, mlis, ahipb, sarah jane brown, mscc adiscovery technologies librarian, dr. john archer library, university of regina, regina, saskatchewan, canada, https://www.orcid.org/0000-0003-4154-8382 , caitlin.bakker@uregina.ca blibrarian liaison to the school of dentistry & health sciences collections coordinator, health sciences library, university of minnesota, minneapolis, minnesota, https://www.orcid.org/0000-0002-6913-5195 ,theis025@umn.edu clibrarian liaison to the college of pharmacy, health sciences library, university of minnesota, minneapolis, minnesota, https://www.orcid.org/0000-0001-7699-4417 , sjbrown@umn.edu abstract objectives: citations do not always equate endorsement, therefore it is important to understand the context of a citation. researchers may heavily rely on a paper they cite, they may refute it entirely, or they may mention it only in passing, so an accurate classification of a citation is valuable for researchers and users. while ai solutions have emerged to provide a more nuanced meaning, the accuracy of these tools has yet to be determined. this project seeks to assess the accuracy of scite in assessing the meaning of citations in a sample of publications. methods: using a previously established sample of systematic reviews that cited retracted publications, we conducted known item searching in scite, a tool that uses machine learning to categorize the meaning of citations. scite’s interpretation of the citation’s meaning was recorded, as was our assessment of the citation’s meaning. citations were classified as mentioning, supporting or contrasting. recall, precision, and f-measure were calculated to describe the accuracy of scite’s assessment in comparison to human assessment. results: from the original sample of 324 citations, 98 citations were classified in scite. of these, scite found that 2 were supporting and 96 were mentioning, while we determined that 42 were supporting, 39 were mentioning, and 17 were contrasting. supporting citations had high precision and low recall, while mentioning citations had high recall and low precision. f-measures ranged between 0.0 and 0.58, representing low classification accuracy. conclusions: in our sample, the overall accuracy of scite’s assessments was low. scite was less able to classify supporting and contrasting citations, and instead labeled them as mentioning. although there is potential and enthusiasm for ai to make engagement with literature easier and more immediate, the results generated from ai differed significantly from the human interpretation. introduction research is cited for a variety of reasons including contextualizing one’s own research, acknowledging foundational works, and critiquing or disputing previous research.1 despite the multitude of possible reasons for citations, they are typically viewed as an endorsement or an 1 https://www.orcid.org/0000-0003-4154-8382 https://orcid.org/ mailto:caitlin.bakker@uregina.ca https://www.orcid.org/0000-0002-6913-5195 https://orcid.org/ mailto:theis025@umn.edu https://www.orcid.org/0000-0001-7699-4417 https://orcid.org/ mailto:sjbrown@umn.edu peer reviewed article hypothesis , vol. 35, no. 2, 2023 indication of impact.2 in the context of a systematic review, a citation may also have a larger meaning, indicating that the data underlying the cited report have been included in the pool of data to be analyzed. understanding why a paper is being cited is far more complex than determining the number of times it has been cited, and often requires readers to find and assess the citation within the full text. artificial intelligence (ai) solutions have emerged in an attempt to provide a more nuanced understanding of a citation’s context and allow readers to quickly discern where there are supportive and divergent findings. these solutions have ranged from those that aim to expedite the systematic review process, including screening, extraction and appraisal,3 to large-scale text mining initiatives that seek to reinforce or contradict scientific claims.4 scite is one such tool. scite is a “smart citation index” that allows researchers to analyze the context of citations by locating a citation within the text and displaying and classifying the citation into one of three areas: supporting, mentioning, or contrasting.5 as of march 2022, scite had over one billion “smart citations” in its database and has continued to ingest articles via agreements with publishers and other content providers, including pubmed, unpaywall, and preprint servers.6 scite is available online via scite.ai, through a chrome browser extension, and through a zotero plugin. scite previously had a freemium model where users could access limited information, but has more recently moved to an exclusively subscription-based model. the paid version of scite includes publication reports that detail citation information for the article, the number of publications citing the article, references, an analysis of the citation statements in the sentence including the in-text citation as well as the preceding and following sentences, and a classification of the citation statement as supporting, mentioning, or contrasting. while the possibility of having a more nuanced representation of citations is an appealing one, scite’s value is predicated upon the assumption that the classification of these citations is an accurate one. to assess the accuracy of scite as a classification tool, we analyzed a sample of systematic reviews that cited publications that had been retracted. this study is part of a larger research project investigating the use of retracted publications in systematic reviews in the field of pharmacy, including the ways in which authors were using these retracted publications.7 the field of pharmacy was selected for its cross-sectional representation of research at a variety of different stages and research foci. pharmacy research extends from bench research, such as drug development, to clinical research with individual patients, and to environmental toxicology, while also intersecting with medical specialties and other health disciplines. systematic reviews synthesize the totality of the evidence on a given topic. this research method is commonly placed at the pinnacle of the evidence-based pyramid. it requires authors to thoroughly assess every included report or study to identify methodological issues or concerns, and to reflect that assessment in its findings. clinicians, students, and researchers often place a great deal of confidence in the findings of systematic reviews, and they might influence both patient care and future research. given the importance of literature to this research method, it would stand to reason that systematic review authors would engage more thoroughly with the literature. the uncritical incorporation of retracted publications into systematic reviews may undermine the value of the research method, as between 22% and 54% of retractions are due to methodological issues or concerns regarding data,8-12 meaning that these reported findings 2 peer reviewed article hypothesis , vol. 35, no. 2, 2023 may be invalid. previous research has found that the inclusion of retracted publications in systematic reviews can significantly alter the overall findings of the review.13 the challenge of retracted publications in systematic reviews is of growing interest, and is one that systematic review authors are cognizant of, as evidenced in guidance available through the cochrane handbook and mecir manual.14-16 despite the available guidance and growing awareness, the inclusion of retracted publications in systematic reviews is an ongoing concern. ai provides an opportunity to assess how retracted publications are used in systematic reviews and holds the potential to ultimately streamline the process of appraising evidence syntheses. our findings regarding the methodological quality of systematic reviews that cite retracted publications are described elsewhere.7 in completing this work, we simultaneously assessed scite’s ability to classify these citations, as the potential to automate and subsequently expedite identification of retracted publications could be of benefit to the evidence synthesis community. methods using data provided by retraction watch from the center for scientific integrity,17 we identified a sample of retracted publications in the field of pharmacy and then conducted known item searching in scopus and web of science core collection, including sci-expanded and ssci-expanded, to identify items which had cited these retracted publications. two reviewers screened each citing item to limit to evidence syntheses, including systematic reviews, meta-analyses, clinical practice guidelines, scoping reviews, and rapid reviews. the initial project, which resulted in this sample, is fully described elsewhere.7 a data extraction form was established in qualtrics to assess citation meaning as determined by scite and a human assessor. after piloting a subset of publications to ensure agreement, every reference in every systematic review was reviewed to assess the meaning of the citation. we conducted known item searching for every retracted publication in scite, and then reviewed the items citing the publication to identify the evidence synthesis in question. we then extracted scite’s assessment of the meaning of the citation, as well as the exact text of the citation. each reviewer recorded whether they agreed or disagreed with scite’s assessment, or where they were unsure of whether the assessment was appropriate. in cases where the reviewer was in disagreement or was unsure, they entered their independent assessment into the qualtrics form. all assessments were reviewed by the researchers collectively to ensure agreement and consistency. citations were classified as mentioning when the citation was included in passing but was not extensively discussed. the citation was classified as supporting if the publication was one of the reports included in the systematic review results without indication of its retracted status or significant critique of the research quality. the citation was classified as contrasting if the publication were described as retracted, or if concerns about the publication or underlying research were discussed. data were then extracted from qualtrics into a comma-separated file and were summarized using r 4.1.3.18 we assessed scite’s accuracy by calculating precision, recall, and f-measure, which are well-established quantifications of the accuracy of classification systems. precision, or positive predictive value, refers to the number of items that were classified with a meaning and truly had that meaning (i.e., the number of citations marked as supporting that truly were supporting). recall, or sensitivity, calculates the number of items classified with a particular meaning out of all items that had that meaning (i.e., the number of citations marked as supporting out of all of the supporting citations in the dataset). as rebala et al. note, “[e]ffectively, you want high precision as well as high recall; i.e., when the data is 3 peer reviewed article hypothesis , vol. 35, no. 2, 2023 positive class, the prediction is also positive, and when the prediction is positive, the actual class is also positive.”19 the f-measure is a score that includes both precision and recall and reflects overall accuracy. f-measures fall between 0 and 1, with higher scores indicating greater accuracy. results our initial sample included 1,396 retracted publications in the field of pharmacy. of these, 312 retracted publications were cited 32,559 times. screening to isolate evidence syntheses identified 324 references to retracted publications in 286 systematic reviews. eleven of these referenced articles were not found in scite, resulting in a set of 313 references. in 97 cases (31%) the referenced article was in scite, but the systematic review’s citation to it was not. as scite did not have access to the full text, 118 (37.7%) of the references were unclassified. of the 98 citations that were classified, scite found that 2 (2%) were supporting, 96 (98%) were mentioning, and 0 (0%) were contrasting. these findings are described in figure 1. figure 1: results of search and assessment in scite we manually coded the 98 references that were classified by scite to assess the accuracy of scite’s classification. we agreed with two citations scite had indicated were supporting. there were an additional 40 citations that scite misclassified which we assessed as supporting. while scite did not identify any contrasting citations, we identified 17 contrasting citations. finally, while we agreed with scite’s assessment of 39 citations as mentioning, we found 57 citations were marked as mentioning when they were in fact contrasting or supporting. 4 peer reviewed article hypothesis , vol. 35, no. 2, 2023 we calculated the precision, recall, and f-measure of each assessment (supporting, mentioning, and contrasting), which can be seen in table 1. table 1. precision, recall and accuracy of scite supporting contrasting mentioning precision 1.0 0.0 0.41 recall 0.05 0.0 1.0 f-measure 0.096 0.0 0.58 discussion scite shows promise in increasing the transparency of citations and has the potential to aid future researchers in understanding how authors use research, similar to the legal practice of shepardizing citations to determine how previous findings have been interpreted and applied. ai tools are rapidly developing to expedite the literature searching and appraisal process.3, 20-22. although scite’s solution is much less labor-intensive than reading each article to understand how it has utilized other literature, there are limitations to ai in contextualization since it may miss nuances easily discerned by human readers. in our sample of systematic reviews citing retracted publications, exclusive reliance on scite would have minimized the core problem: the supporting citation of retracted publications. scite found that the vast majority of citations in our sample were mentioning, while the remaining were supporting. this is broadly in alignment with scite’s overall assessments, as the creators indicate that “the average distribution of citation statements [is] 92.6% mentioning, 6.5% supporting, and 0.8% contrasting statements.”5 while this may be in alignment with scite’s overall assessment of literature, it diverged significantly from our assessment of meaning, as we determined that, of the 96 citations scite classified as mentioning, 40 were more appropriately classified as supporting and 17 were more appropriately classified as contrasting. while scite determined that the majority of citations should be classified as mentioning, human assessors disagreed and had higher rates of supporting and contrasting citations. in our sample, the overall accuracy of scite’s assessments was low. recall, or sensitivity, describes the number of citations classified with a particular meaning out of all citations that had that meaning. supporting and contrasting meanings had low recall, while mentioning had high recall. this means that scite was less able to classify supporting and contrasting citations as such, instead labeling them as mentioning. mentioning had high recall, meaning that, when citations were truly mentioning, scite recognized that they were mentioning. precision, or positive predictive value, describes the number of items classified with a meaning that truly had that meaning. supporting statements had high precision, meaning that the majority of citations classified as supporting were truly supporting. while scite may overlook supporting citations, it rarely misclassifies citations that are contrasting or mentioning as supporting. conversely, mentioning had relatively low precision, indicating that users should be less confident that citations marked as mentioning are truly mentioning rather than supporting or contrasting. scite’s tendency to classify the vast majority of citations as mentioning limits its overall utility. as stacy brody notes, “[a] high count of 5 peer reviewed article hypothesis , vol. 35, no. 2, 2023 mentioning citation statements may provide little more nuance or detail than a total citation count.”23 although scite may misclassify some citations, the larger challenge is missing content in scite’s database. in our sample, less than one-third of the citations were classified in scite. in 31% of cases, the citation was not represented at all, while in 38% of cases the citation was represented but marked as unclassified due to lack of full-text access. this is a significant barrier to use. it could be argued that, even if classified inaccurately, foregrounding that citations can have different meanings is of value to users. however, if this foregrounding occurs in only a minority of cases, the pedagogical value of the tool is minimized. while scite has established relationships with several large publishers,24 additional agreements with a broader range of publishers would enhance the utility and value of the tool. while this project focused on citations occurring within systematic reviews in the field of pharmacy, it occurs in the broader context of ai in systematic reviews and critical appraisal. almost every aspect of information seeking has an associated ai solution. ai tools and solutions have been developed to facilitate more efficient searching,25 to select relevant publications and disregard irrelevant publications,26 to extract study findings and characteristics,27 and to assess the quality of the publication.28 these advances have tremendous potential to expedite aspects of information seeking and allow time to be reallocated to critical engagement with and application of the literature. however, despite the potential of these tools, our findings highlight one context in which reliance upon ai would have led to skewed and inaccurate findings. limitations our research has several notable limitations. first, we focus on a sample of publications both within a discipline and using a specific study design. while scite is not programmed to perform differently based on study design or discipline, it is possible that studies using different publications would reveal different levels of accuracy. we also chose to focus on systematic reviews that had cited at least one retracted publication. retraction is a relatively rare and extreme publication state. as such, the specific citations included in our sample are, by nature of their retracted status, not indicative of the majority of publications. although we chose to focus on these outliers under the assumption that these publications would warrant stronger critique, this does limit the generalizability of our findings. finally, this work was done in conjunction with a larger research project, rather than as a standalone project. as such, the sample was derived through this larger work, rather than being selected for the sole purpose of assessing scite. other sampling methods may lead to different results. conclusions students, clinicians and researchers are increasingly seeking new technologies to enable rapid engagement with the scholarly literature. while ai holds tremendous promise, both in the completion of systematic reviews and in the critical appraisal of literature, reliance on ai in this context may be premature. this research emphasizes the 6 peer reviewed article hypothesis , vol. 35, no. 2, 2023 potential challenge of limited data sources and relatively low accuracy, both of which may cause challenges for adoption. references 1. garfield e. can citation indexing be automated? stat assoc methods mech doc symp proc. 1965;269:189–92. doi:10.1038/227669a0 2. white hd. citation analysis and discourse analysis revisited. appl linguist. 2004 mar 1;25(1):89–116. doi:10.1093/applin/25.1.89 3. blaizot a, veettil sk, saidoung p, moreno-garcia cf, wiratunga n, aceves-martins m, et al. using artificial intelligence methods for systematic review in health sciences: a systematic review. res synth methods. 2022 may;13(3):353–62. doi:10.1002/jrsm.1553 4. opscidia plateforme pour accélérer la veille technologique [internet]. 2022 [cited 2023 feb 10]. available from: https://www.opscidia.com/ 5. nicholson jm, mordaunt m, lopez p, uppala a, rosati d, rodrigues np, et al. scite: a smart citation index that displays the context of citations and classifies their intent using deep learning. quant sci stud. 2021 nov 5;2(3):882–98. doi:10.1162/qssa00146 6. scite. where do you get your articles from? [internet]. scite. [cited 2022 jun 28]. available from: https://help.scite.ai/en-us/article/where-do-you-get-your-articles-from-1vglydm/ 7. brown sj, bakker cj, theis-mahon nr. retracted publications in pharmacy systematic reviews. j med libr assoc. 2022 feb 11;110(1):47–55. doi:10.5195/jmla.2022.1280 8. moylan ec, kowalczuk mk. why articles are retracted: a retrospective cross-sectional study of retraction notices at biomed central. bmj open. 2016;6(11):e012047. published 2016 nov 23. doi:10.1136/bmjopen-2016-012047 9. wager e, williams p. why and how do journals retract articles? an analysis of medline retractions 1988-2008. j med ethics. 2011;37(9):567-570. doi:10.1136/jme.2010.040964 10. nair s, yean c, yoo j, leff j, delphin e, adams dc. reasons for article retraction in anesthesiology: a comprehensive analysis.raisons justifiant la rétractation d’un article en anesthésiologie: une analyse exhaustive. can j anaesth. 2020;67(1):57-63. doi:10.1007/s12630-019-01508-3 11. bozzo a, bali k, evaniew n, ghert m. retractions in cancer research: a systematic survey. res integr peer rev. 2017;2:5. published 2017 may 12. doi:10.1186/s41073-017-0031-1 7 https://www.opscidia.com/ https://help.scite.ai/en-us/article/where-do-you-get-your-articles-from-1vglydm/ peer reviewed article hypothesis , vol. 35, no. 2, 2023 12. chauvin a, de villelongue c, pateron d, yordanov y. a systematic review of retracted publications in emergency medicine. eur j emerg med. 2019;26(1):19-23. doi:10.1097/mej.0000000000000491 13. garmendia ca, nassar gorra l, rodriguez al, trepka mj, veledar e, madhivanan p. evaluation of the inclusion of studies identified by the fda as having falsified data in the results of meta-analyses: the example of the apixaban trials [published correction appears in jama intern med. 2021 mar 1;181(3):409]. jama intern med. 2019;179(4):582-584. doi:10.1001/jamainternmed.2018.7661 14. cochrane library. managing potentially problematic studies [internet]. cochrane database of systematic reviews: editorial policies. [cited 2023 feb 10]. available from: https://www.cochranelibrary.com/cdsr/editorial-policiesproblematic-studies 15.higgins jpt, lasserson t, chandler j, tovey d, thomas j, flemyng e, et al. mecir manual [internet]. london, england: cochrane; 2022 feb [cited 2023 feb 10]. available from: https://community.cochrane.org/mecir-manual 16. lefebvre c, glanville j, briscoe s, featherstone r, littlewood a, marshall c, et al. 4.s1 technical supplement to chapter 4: searching for and selecting studies. in: higgins jpt, thomas j, chandler j, cumpston m, li t, page m, et al., editors. cochrane handbook for systematic reviews of interventions version 63 [internet]. chichester, uk: john wiley sons, ltd; 2022 [cited 2023 feb 10]. available from: https://training.cochrane.org/handbook/current/chapter-04-technical-supplementsearching-and-selecting-studies 17. center for scientific integrity. retraction watch database [internet]. [cited 2022 jul 21]. available from: http://retractiondatabase.org/retractionsearch.aspx? 18. r core team. r: a language and environment for statistical computing [internet]. vienna, austria: r foundation for statistical computing; 2022. available from: https://www.r-project.org/ 19. rebala g, ravi a, churiwala s. classification. in: an introduction to machine learning [internet]. 2019 [cited 2022 jun 28]. p. 57–67. available from: https://doi.org/10.1007/978-3-030-15729-6 20. van de schoot r, de bruin j, schram r, zahedi p, de boer j, weijdema f, et al. an open source machine learning framework for efficient and transparent systematic reviews. nat mach intell. 2021 feb;3(2):125–33. doi:10.1038/s42256-020-00287-7 21. marshall ij, wallace bc. toward systematic review automation: a practical guide to using machine learning tools in research synthesis. syst rev. 2019 jul 11;8(1):163. doi:10.1186/s13643-019-1074-9 22. zhang y, liang s, feng y, wang q, sun f, chen s, et al. automation of literature screening using machine learning in medical evidence synthesis: a diagnostic test accuracy systematic review protocol. syst rev. 2022 jan 15;11(1):11. 8 https://www.cochranelibrary.com/cdsr/editorial-policies#problematic-studies https://community.cochrane.org/mecir-manual https://training.cochrane.org/handbook/current/chapter-04-technical-supplement-searching-and-selecting-studies https://training.cochrane.org/handbook/current/chapter-04-technical-supplement-searching-and-selecting-studies http://retractiondatabase.org/retractionsearch.aspx? https://www.r-project.org/ https://doi.org/10.1007/978-3-030-15729-6 peer reviewed article hypothesis , vol. 35, no. 2, 2023 doi:10.1186/s13643-021-01881-5 23. brody s. scite. j med libr assoc. 2021 nov 22;109(4):707–10. doi:10.5195/jmla.2021.1331 24. scite for publishers [internet]. scite.ai. [cited 2022 nov 12]. available from: https://scite.ai 25. wallace bc, noel-storr a, marshall ij, cohen am, smalheiser nr, thomas j. identifying reports of randomized controlled trials (rcts) via a hybrid machine learning and crowdsourcing approach. j am med inform assoc. 2017 nov 1;24(6):1165–8. doi:10.1093/jamia/ocx053 26. o’mara-eves a, thomas j, mcnaught j, miwa m, ananiadou s. using text mining for study identification in systematic reviews: a systematic review of current approaches. syst rev. 2015 jan 14;4(1):5. doi:10.1186/2046-4053-4-5 27. kiritchenko s, de bruijn b, carini s, martin j, sim i. exact: automatic extraction of clinical trial characteristics from journal publications. bmc med inform decis mak. 2010 sep 28;10(1):56. doi:10.1186/1472-6947-10-56 28. marshall ij, kuiper j, wallace bc. robotreviewer: evaluation of a system for automatically assessing bias in clinical trials. j am med inform assoc. 2016 jan 1;23(1):193–201. doi:10.1093/jamia/ocv044 author contributions caitlin bakker: conceptualization, data curation, investigation, writing original draft, writing review & editing; nicole theis-mahon: conceptualization, data curation, investigation, writing original draft, writing review & editing; sarah jane brown: conceptualization, data curation, investigation, writing original draft, writing review & editing 9 https://scite.ai peer reviewed article hypothesis, vol. 36, no. 1, 2024 case-based ebm instruction for osteopathic medical students: a case report laura lipke, ms, mlis, ahipa ahealth science librarian/nursing liaison, university libraries, binghamton university, binghamton, new york, https://www.orcid.org/0000-0003-3653-5843 , llipke@binghamton.edu cite as: lipke l. case-based ebm instruction for osteopathic medical students: a case report. hypothesis. 2024;36(1). doi:10.18060/27674 lipke. all works in hypothesis are licensed under a cc by-nc 4.0 deed attribution-noncommercial 4.0 international. authors own copyright of their articles appearing in hypothesis. readers may copy articles without permission of the copyright owner(s), as long as the author(s) are acknowledged in the copy, and the copy is used for educational, not-for-profit purposes. for any other use of articles, please contact the copyright owner(s). 1 https://www.orcid.org/0000-0003-3653-5843 https://orcid.org/ mailto:9. llipke@binghamton.edu https://creativecommons.org/licenses/by-nc/4.0/ https://creativecommons.org/licenses/by-nc/4.0/ peer reviewed article hypothesis, vol. 36, no. 1, 2024 abstract background: evidence-based medicine (ebm) is an important part of graduate medical education and a learning outcome necessary for preparing students to become effective residents. however, many medical educators and librarians struggle to offer ebm instruction that is contextualized, relevant, and engaging. case-based learning (cbl) can establish the relevance of ebm instruction sessions by using clinical scenarios that link theory to practice. experience: this case report describes how a college of osteopathic medicine (com) liaison librarian redesigned a large cohort, lecture-based learning approach to ebm instruction into interactive, relevant, and contextualized cbl labs. cbl labs with smaller student groups were designed to provide an active learning environment and encourage peer-to-peer learning. the cbl format has been previously applied to ebm instruction but not quantifiably evaluated. a modified rubric was created to provide a quantifiable measurement of student performance in future evaluations. discussion: first-year osteopathic medical students participated in cbl sessions, integrating ebm into their curriculum. the small groups facilitated personalized guidance, peer-to-peer learning, and critical thinking. clinical scenarios that were mapped to the students’ curriculum provided relevance to the learning experience. the cbl format allowed the librarian to support students individually and increased engagement compared to lecture-based learning. challenges included assessing the students’ prior ebm knowledge and the time commitment for a solo librarian. however, group assignments and a grading rubric helped to minimize the challenge. conclusion: the initial pilot testing of this instructional design format and rubric has shown promise in reaching the objectives of providing a relevant and active learning platform with quantifiable results. a future randomized controlled trial is planned to provide quantifiable evidence supporting cbl labs for ebm instruction when compared to lecture-based learning. 2 peer reviewed article hypothesis, vol. 36, no. 1, 2024 background the association of colleges of osteopathic medicine (aacom) and the accreditation council for graduate medical education (acgme) in 2015 outlined the competencies osteopathic medical students are expected to demonstrate such as, “locating, appraising, and assimilating evidence from scientific studies. . . using information technology”1-2. the ability to search for evidence is foundational to the clinical practice of evidence-based medicine and research. many students lack the skills needed to meet these competencies because most osteopathic medical schools have not fully integrated evidence-based medicine (ebm) instruction into their curricula due to the lack of faculty development, time and resources3-7. numerous studies have highlighted the struggles medical students face when implementing ebm skills. such struggles include the inability to develop well-formed clinically relevant questions, not being able to navigate medical databases, struggling to interpret search results, and an average poor grade in new residents’ core ebm skills5-8. there is also a significant lack of evidence supporting the ebm instruction methods within colleges of medicine (coms) in the united states. a current pubmed search demonstrated this lack of evidence in the results of only two studies. a 2010 study by laird et al. and a 2019 study by vaughan, et al. based on australian osteopathic students9-10. the association of academic health sciences libraries (aahsl) mapped the 2015 aacom competencies to the association of college and research libraries (acrl) information literacy framework to establish a baseline of library engagement and to create a cross-referencing map of existing ebm and information literacy documents11. this mapping supports the implementation of librarian-led ebm instruction. recent studies also demonstrate that librarian-led instruction improves question formulation and translation of search terms to productive searches because librarians use various pedagogical methods to promote independence in skill development6,12,13. as such, eldredge et al. suggest that librarians are “natural collaborators in teaching [ebm]"12. case-based learning (cbl) is an instructional method that provides students with clinically relevant patient scenarios and active learning opportunities within peer groups. students are provided with a clinical patient scenario relevant to their area of study and asked to apply their theoretical knowledge in a realistic application of ebm skills. by applying cbl to the ebm process of learning to develop an answerable question and acquire evidence to support this question, students can link theory to practice. students’ learning engagement also improved with cbl when compared to lecture-based learning14-17. cbl can also facilitate peer-to-peer interactions within the small group activities not found in lecture-based learning. this combination of peer interaction and active, relevant learning can enhance students’ engagement and comprehension of ebm methodology14,16. swanberg et al. presented a pilot integration of a librarian-led cbl approach to clinical question development and search strategy instruction. in their study, the effectiveness of the sessions was measured by student feedback in course evaluations and a faculty-generated rubric. the measures did not assess the students’ proficiencies with a validated instrument18. menard et al. measured the effectiveness of scaffolded ebm instruction using a modified version of the fresno test to assess student performance19. the proposed cbl instruction sessions to be discussed in this article combine swanberg et al.’s design and apply menard et al.’s modified version of the fresno test as a grading rubric. the combination of these 3 peer reviewed article hypothesis, vol. 36, no. 1, 2024 instructional methods will provide a quantifiable measurement tool to demonstrate that relevant teaching materials combined with peer learning opportunities are more effective than lecture-based learning. experience the medical library at this private midwestern college of osteopathic medicine has one librarian dedicated to the com. the university, across three separate campuses, has six liaison librarians and five staff. the university libraries serve the associated medical, health science, and dental programs through collection development, research assistance, information literacy, and ebm instruction. this com program did not have a liaison librarian for two years prior to the author’s appointment, and ebm was not addressed in any part of the com curriculum. the librarian initially approached the associate dean with the proposal of library instruction by referencing the dean’s prior scholarship which supported the inclusion of ebm instruction within coms to meet aacom competencies1,4,20. the associate dean agreed to allow the librarian one 60-minute session in the late fall to introduce ebm. the first session was conducted in a lecture-based format, with one librarian instructing an auditorium of approximately 160 first-year medical students. at this session, the librarian did not have the opportunity to interact with the students individually and was not able to incorporate active learning. these factors limited an interactive learning environment and the ability to measure student outcomes. the librarian decided to redesign the course from lecture-based to cbl which would incorporate the interactive and relevant learning that was missing in the lecture-based session. the librarian consulted the university’s teaching and learning center for assistance with the development of the course goals and objectives as well as the design of a cbl lab. the redesigned course was presented to the associate dean, who granted permission for the librarian to run a pilot of the course. the cbl session goals and course objectives included the aacom competencies expected by residency1. the overall learning goal(s) are to introduce first-year osteopathic medical students to the ebm process and demonstrate the applicability of this process to clinical practice using relevant clinical scenarios in an interactive, engaging environment. the learning objectives were as follows: 1. first-year osteopathic medical students will be able to construct a well-formed clinical question according to the criteria in the associated rubric, with a minimum of two points. 2. first-year osteopathic medical students will be able to acquire medical evidence according to the criteria in the associated rubric, with a minimum of two points. students at this com varied in their educational and professional experiences, from a basic undergraduate science degree to extensive medical and/or research experience. the librarian was unable to determine the degree to which any of the students had had prior information literacy or ebm instruction due to their varied backgrounds. to accommodate for these variations and unknowns, the librarian provided an introduction to ebm to the students before the implementation of the cbl sessions. this introduction was provided through a libwizard tutorial21. this tutorial guided students through the ask and acquire steps of the ebm 4 peer reviewed article hypothesis, vol. 36, no. 1, 2024 process and utilized a combination of videos and interactive activities to encourage participation. all students were assigned the tutorial through their learning management system (lms). they were provided instructions to complete the tutorial before the lab sessions and informed that the information within the tutorial was required for the lab sessions. students were instructed to upload the completion certificate provided by libwizard to receive credit for completing the tutorial. all students received the full amount of credit regardless of their performance. this detail was not provided to the students before the tutorial to encourage quality performance. the librarian worked with the medical education fellows from the com to design clinical scenarios aligned with the students’ current topics of study in the curriculum for the cbl sessions. this was done to ensure that the students would be knowledgeable about the disease and treatment presented and to further embed relevancy within the cbl assignment. (examples of the clinical scenarios can be found in (appendix a). the librarian divided the class of approximately 160 first-year osteopathic medicine students into smaller groups of 25-30 students for each 60-minute lab session. each lab group was then divided in to smaller working groups of 3-5 students each. each session began with a brief review of the pre-session tutorial and instructions on how to complete the in-lab assignment (appendix b). the students were instructed to complete the lab worksheet (appendix c) as a group and for each student to upload the completed worksheet into the lms. students were encouraged to ask questions throughout the session and they were provided feedback and prompts to encourage critical thinking. examples of prompts included suggestions to minimize the number of search terms, to focus on the most important search terms and to refer to background information resources (e.g., textbooks) when they required additional information about the clinical scenario. the grading rubrics (appendix d) were based upon the original fresno rubric developed by ramos et al. and modified based on the menard et al. study19,22. the fresno measurement was the rubric of choice because it has been validated and the modifications made in the menard et al. study fit the cbl instructional design for this com19,22. the original fresno rubric was employed to critique the students’ ability to design an answerable clinical research question based on the provided clinical scenario. in addition to the modifications made by menard et al.19, the fresno rubric was further modified to fit within the time constraints of the labs, such as limiting the search requirements to one database instead of three. the fresno rubric was also modified to grade the search strategy. the requirement to include field tags was necessary as the students had not received prior instruction regarding the inclusion of field tags. the pre-session tutorial focused only on question development and searching the pubmed database to provide focus on the basic search needs of this assignment. all student working groups were provided with a copy of the rubric and instructed as to how this rubric would be used to grade their submitted assignments. the liaison librarian taught all six cbl sessions and graded all assignments submitted to the lms. each student’s submission was also provided with detailed feedback as needed. students who performed poorly or required extensive feedback were encouraged to meet with the com librarian. students were allowed to revise their assignments if requested. 5 https://dx.doi.org/10.18060/27674 https://dx.doi.org/10.18060/27674 https://dx.doi.org/10.18060/27674 https://dx.doi.org/10.18060/27674 https://dx.doi.org/10.18060/27674 peer reviewed article hypothesis, vol. 36, no. 1, 2024 discussion cbl was implemented with a class of first-year osteopathic medical students to establish ebm instruction within the curriculum and provide a relevant, active learning experience linking theory to practice. the cbl lab design with small working groups of 3-5 students provided an opportunity for the librarian to work one-on-one with students as needed, fostered peer-to-peer learning, and presented a learning environment that encouraged critical thinking. by using curriculum-mapped clinical scenarios as examples for the practice of the ask and acquire portions of ebm, the students were provided with an opportunity to translate their academic knowledge to the clinical application of ebm skills. the instructor provided the students with clear assignment expectations per the modified fresno rubric. the modifications made to the original fresno rubric fit well within the 60-minute lab sessions and the baseline experience level of the students completing the lab assignment. one significant strength of the cbl design was that it allowed the librarian to observe student interactions and determine which students may require more assistance. the librarian also observed that the students appeared more engaged with the task when compared to lecture-based learning. examples of engagement included parsing of lab tasks, collaboration, and active debates regarding processes and interpretations of the data. the answers to the questions asked by the students during the lab sessions were shared and demonstrated by the librarian for the lab to learn as a whole group, which provided a larger aspect of peer-to-peer learning. there were also challenges associated with the adoption of cbl to teach ebm. one significant challenge was in designing the cbl labs. the amount of previous experience with ebm among the large group of students was unknown and therefore the course could not be designed to accommodate a higher level of knowledge if appropriate. while there is no simple solution to determine the degree of previous ebm instruction of a class, the peer-to-peer learning aspect of cbl encourages those with more experience to assist those with less. the smaller lab sessions also allow for one-to-one interactions between the librarian and the students that may require more in-depth assistance. a second challenge was that this type of course design requires significant time and dedicated commitment, which is often difficult for a solo librarian to manage. this librarian was able to prioritize the design and implementation of this course due to the support of the library director. because this program had not had a liaison librarian for a significant amount of time, both the librarian and the director recognized the vitality of instituting ebm instructional services into this curriculum. the administrator(s) granted the librarian approval to develop instruction services on an as needed basis and to prioritize these services over others. the librarian worked with the com faculty to schedule the lab sessions over a one-week period at the beginning of the spring semester. this schedule formatting allowed the librarian to focus solely on the lab sessions over other tasks and provided the librarian time to grade the assignments before the end of the semester. the librarian also had fewer assignments to grade due to the group work design of cbl. each small working group submitted one assignment and each group member received the same grade. each lab used the same clinical scenario, which allowed the librarian to address issues in the whole lab instead of individual scenarios. the use of the grading rubric also provided the ability to grade each group’s assignment 6 peer reviewed article hypothesis, vol. 36, no. 1, 2024 efficiently and guided the librarian to determine their actions with variations in the answers. with positive reflections from the initial run of this pilot program, the librarian designed a randomized controlled trial (rct) comparing the effectiveness of the lecture-based instruction sessions to the cbl sessions using the same assignment. this rct will be applied to future com cohorts upon receiving irb approval. changes envisioned for future sessions would be based on the results of the randomized controlled trial planned to compare the effectiveness of cbl to traditional lecture-based learning. takeaways many medical librarians are faced with the daunting task of providing ebm to large cohorts of medical students in one-shot sessions and growing numbers are required to do this on their own. the author’s desire to instill relevance to the ebm learning process and implement opportunities to interact more closely with the students inspired them to design case-based learning sessions. these small labs, mapped to the students’ curriculum with the assistance of the medical education fellows, provided relevant interactive learning opportunities where both the librarian instructor and the students could have fulfilling experiences. these experiences will be quantifiably measured through future studies comparing lecture-based learning to case-based learning. data availability statement there is no data associated with this article. references 1. basehore pm, mortensen lh, katsaros e, linsenmeyer m, mcclain ek, sexton ps, et al. entrustable professional activities for entering residency: establishing common osteopathic performance standards in the transition from medical school to residency. j am osteopath assoc. 2017 nov 1;117(11):712–8.doi:10.7556/jaoa.2017.137 2. accreditation council for graduate medical education (acgme). the program director guide to the common program requirements (residency) [internet]. 2021. available from: https://www.acgme.org/globalassets/pdfs/program-director-guide—residency.pdf 3. guralnick s, yedowitz-freeman j. core entrustable professional activities for entry into residency: curricular gap or unrealistic expectations? j grad med educ. 2017 oct;9(5):593–4.doi:10.4300/jgme-d-17-00559.1 4. linsenmeyer m, wimsatt l, speicher m, basehore p, sexton ps. status of entrustable professional activities (epa) implementation at colleges of osteopathic medicine in the united states and future considerations. j am osteopath assoc. 2020 nov 1;120(11):749–60. doi:10.7556/jaoa.2020.129 7 https://dx.doi.org/10.7556/jaoa.2017.137 https://www.acgme.org/globalassets/pdfs/program-director-guide---residency.pdf https://dx.doi.org/10.4300/jgme-d-17-00559.1 https://dx.doi.org/10.7556/jaoa.2020.129 peer reviewed article hypothesis, vol. 36, no. 1, 2024 5. mcclurg c, powelson s, lang e, aghajafari f, edworthy s. evaluating effectiveness of small group information literacy instruction for undergraduate medical education students using a preand post-survey study design. health inf libr j. 2015 jun;32(2):120-30. doi:10.1111/hir.12098 6. nicholson j, kalet a, van der vleuten c, de bruin a. understanding medical student evidence-based medicine information seeking in an authentic clinical simulation. j med libr assoc. 2020 apr;108(2):219–28. doi:10.5195/jmla.2020.875 7. pearlman re, pawelczak m, yacht ac, akbar s, farina ga. program director perceptions of proficiency in the core entrustable professional activities. j grad med educ. 2017 oct;9(5):588–92. doi:10.4300/jgme-d-16-00864.1 8. smith g, stark a, sanchez j. research and teaching: what does course design mean to college science and mathematics teachers? j coll sci teach [internet]. 2019 [cited 2021 nov 10];048(04). available from: https://www.jstor.org/stable/26901303 9. laird s, george j, sanford sm, coon s. development, implementation, and outcomes of an initiative to integrate evidence-based medicine into an osteopathic curriculum. j am osteopath assoc. 2010 oct;110(10):593–601. 10. vaughan b, grace s, gray b, kleinbaum a. engaging with evidence-based practice in the osteopathy clinical learning environment: a mixed methods pilot study. int j osteopath med. 2019 sep 1;33–34:52–8. doi:10.1016/j.ijosm.2019.09.001 11. brennan ea, ogawa rs, thormodson k, von isenburg m. introducing a health information literacy competencies map: connecting the association of american medical colleges core entrustable professional activities and accreditation council for graduate medical education common program requirements to the association of college research libraries framework. j med libr assoc. 2020 jul 1;108(3):420–7. doi:10.5195/jmla.2020.645 12. eldredge j, schiffma, langsjoen jo, jerabek rn. question formulation skills training using a novel rubric with first-year medical students. j med libr assoc. 2021 jan 7;109(1):68–74. doi:10.5195/jmla.2021.935 13. ma j, stahl l, knotts e. emerging roles of health information professionals for library and information science curriculum development: a scoping review. j med libr assoc. 2018 oct;106(4):432–44. doi:10.5195/jmla.2018.354 14. anderson h, studer ac, holm kn, suzuki a. a case-based active learning session for medical genetics resources. mededportal j teach learn resour. 2021 apr 1;17:11135. doi:10.15766/mep2374 − 8265.11135 15. kash mj. teaching evidence-based medicine in the era of point-of-care databases: the case of the giant bladder stone. med ref serv q. 2016;35(2):230–6. doi:10.1080/02763869.2016.1152148 8 https://doi.org/10.1111/hir.12098 https://doi.org/10.5195/jmla.2020.875 https://dx.doi.org/10.4300/jgme-d-16-00864.1 https://www.jstor.org/stable/26901303 https://doi.org/10.1016/j.ijosm.2019.09.001 https://dx.doi.org/10.5195/jmla.2020.645 https://dx.doi.org/10.5195/jmla.2021.935 https://dx.doi.org/10.5195/jmla.2018.354 https://dx.doi.org/10.15766/mep_2374-8265.11135 https://dx.doi.org/10.1080/02763869.2016.1152148 peer reviewed article hypothesis, vol. 36, no. 1, 2024 16. tayce jd, saunders ab, keefe l, korich j. the creation of a collaborative, case-based learning experience in a large-enrollment classroom. j vet med educ. 2021 feb;48(1):14–20. doi:10.3138/jvme.2019-0001 17. thistlethwaite je, davies d, ekeocha s, kidd jm, macdougall c, matthews p, et al. the effectiveness of case-based learning in health professional education. a beme systematic review: beme guide no. 23. med teach. 2012;34(6):e421-444. doi:10.3109/0142159x.2012.680939 18. swanberg sm, mi m, engwall k. an integrated, case-based approach to teaching medical students how to locate the best available evidence for clinical care. mededportal j teach learn resour. 2017 jan 19;13:10531. doi:10.15766/mep2374 − 8265.10531 19. menard l, blevins ae, trujillo dj, lazarus kh. integrating evidence-based medicine skills into a medical school curriculum: a quantitative outcomes assessment. bmj evid-based med. 2021 oct;26(5):249–50. doi:10.1136/bmjebm-2020-111391 20. lapinski j, sexton p. still in the closet: the invisible minority in medical education. bmc med educ. 2014 dec;14(1):171. doi:10.1186/1472-6920-14-171 21. lipke l. research question development with pico advanced search [internet]. libwizard tutorial presented at; 2022; a.t. still university. available from: https://atsu.libwizard.com/f/pico_advancedsearch 22. ramos kd, schafer s, tracz sm. validation of the fresno test of competence in evidence-based medicine. bmj. 2003 feb 8;326(7384):319. doi:10.1136/bmj.326.7384.319 9 https://dx.doi.org/10.3138/jvme.2019-0001 https://dx.doi.org/10.3109/0142159x.2012.680939 https://dx.doi.org/10.15766/mep_2374-8265.10531 https://dx.doi.org/10.1136/bmjebm-2020-111391 https://dx.doi.org/10.1186/1472-6920-14-171 https://atsu.libwizard.com/f/pico_advancedsearch https://dx.doi.org/10.1136/bmj.326.7384.319 appendix a: structural racism programming series end-of-year survey as we conclude this year's structural racism programming series, we're seeking feedback from both those who attended and those who did not attend this year to aid in future planning! we anticipate this will take no more than five minutes to complete. no identifying information is being collected in this evaluation. as a refresher, these events were offered this year [events listed below] did you attend any of the above events or watch the event recordings this year? yes or no ‘yes’ sends respondents to questions specifically for attendees, ‘no’ sends respondents to questions specifically for non-attendees questions for attendees: which event(s) did you attend or watch the recording of this year? (check all that apply): · [list event titles] how did you find out about the structural racism events? (check all that apply) · school website · email from coordinator of student services · friday school opportunities email · facebook event · newsletter · atrium tv · professor recommendation · other why did you attend these events? (check all that apply) · worked will with my schedule · interested in topics presented · applicable to my school/work/practice · other on a scale of 1 to 5, rate the following statements (1=strongly disagree to 5=strongly agree) attending events in this series: · increased my awareness/knowledge of structural racism overall · enhanced my awareness/knowledge of the specific impacts of structural racism on marginalized communities · provided opportunities for open dialogue regarding these issues · increased my comfort/ability to work with clients from diverse backgrounds · increased my awareness/knowledge of how structural racism impacts the mental health of my clients how likely are you to attend more structural racism events in the future (1= very likely to 5=not at all likely) questions for non-attendees: did you rsvp, but not attend or watch a recording of any of the events? yes/no/unsure if ‘yes’, for which of the following reasons did you decide to not attend/watch the session? (check all that apply) · lack of time · lack of energy · too many other priorities · topic not interesting · other questions for all respondents: which of the following reasons prevented you from attending more structural racism events this year? (check all that apply) · scheduling conflicts with the day/time · scheduled time works for me, but i have competing priorities · rather use break times to connect with friends or relax · not interested in topics covered · not applicable to my school/work/practice · was not aware of events · the event i was interested in attending was canceled · other what could be improved about the series to encourage future attendance? what topics or speaker recommendations do you have for future events? other questions or comments? hypothesis, vol. 37, no. 1, 2025 editorial board updates: evidence syntheses subcommittee alyssa grimshaw, mslis, mba, mpha, jennifer mckinnell, ma, mlisb, pamela pierce, mls, msc aclinical librarian, cushing whitney medical library, yale university, new haven, ct, alyssa.grimshaw@yale.edu bdirector of the health sciences library, mcmaster university, hamilton, ontario, canada, mckinn@mcmaster.ca cpublishing and data services manager and assistant professor, oregon health & science university, portland, or, piercepa@ohsu.edu cite as: grimshaw a, mckinnell j, and pierce p. editorial board updates: evidence syntheses subcommittee. hypothesis. 2025;37(2). doi:10.18060/28749 grimshaw, mckinnell, pierce. all works in hypothesis are licensed under a cc by-nc 4.0 deed attribution-noncommercial 4.0 international. authors own copyright of their articles appearing in hypothesis. readers may copy articles without permission of the copyright owner(s), as long as the author(s) are acknowledged in the copy, and the copy is used for educational, not-for-profit purposes. for any other use of articles, please contact the copyright owner(s). 1 mailto:alyssa.grimshaw@yale.edu mailto:mckinn@mcmaster.ca mailto:piercepa@ohsu.edu https://creativecommons.org/licenses/by-nc/4.0/ hypothesis, vol. 37, no. 1, 2025 this special issue explores emerging trends, challenges, and opportunities shaping the field of evidence synthesis. featuring cutting-edge insights, it highlights the transformative role of ai, the growing demand for comprehensive evidence reviews, and the need for enhanced training opportunities. additionally, it emphasizes research about the importance of guidelines and reporting standards in advancing the quality and transparency of evidence synthesis work. the issue also addresses critical challenges, such as the rise in retractions and their implications for the field. with a diverse range of content types—including brief reports, “voices of experience,” and research articles—this issue aims to provide perspectives that inform and inspire practitioners, researchers, and educators. we are especially interested in contributions from early-to-mid-career information professionals and student-led contributions as they shape the future of evidence synthesis. join us in charting the path forward for evidence synthesis! 2 editorial reviewed article hypothesis, vol. 36, no. 1, 2024 an introduction to statistics for librarians (part three): an introduction to statistical tests caitlin bakker, mlis, ahip-da, adiscovery technologies librarian, dr. john archer library, university of regina, regina, saskatchewan, canada, https://www.orcid.org/0000-0003-4154-8382 , caitlin.bakker@uregina.ca cite as: bakker c. an introduction to statistics for librarians (part three): an introduction to statistical tests. hypothesis. 2024;36(1). doi:10.18060/27969 bakker. all works in hypothesis are licensed under a cc by-nc 4.0 deed attribution-noncommercial 4.0 international. authors own copyright of their articles appearing in hypothesis. readers may copy articles without permission of the copyright owner(s), as long as the author(s) are acknowledged in the copy, and the copy is used for educational, not-for-profit purposes. for any other use of articles, please contact the copyright owner(s). 1 https://www.orcid.org/0000-0003-4154-8382 https://orcid.org/ mailto:caitlin.bakker@uregina.ca https://creativecommons.org/licenses/by-nc/4.0/ https://creativecommons.org/licenses/by-nc/4.0/ editorial reviewed article hypothesis, vol. 36, no. 1, 2024 data bytes is a new column derived from the statistics for librarians series of columns in research mentor1,2. the purpose of this column is to provide librarians and information professionals with brief introductions to specific dataand statistics-related topics in a way that is accessible and relevant for their research and practice. the intention of the column is not to cover all aspects of statistics or data, or to turn information professionals into statisticians. instead, the focus here is to help librarians and information professionals become savvier consumers of scholarly literature and to potentially enhance their own research practices. some statistical tests to consider in the previous two columns, the different types of data, central tendency, and frequency distributions were discussed1,2. in this column, we’re going to build on the previous two columns to consider some of the different statistical tests that are available. the intention of this column isn’t to state definitively which test should be used in any given situation, but rather to highlight how the topics from the previous two columns can be used to help you identify potential tests. the below table, based on the original work of samantha carlson and tanya hoskin3,4, is meant to introduce some tests to consider. this isn’t a comprehensive inventory of all tests and isn’t meant to definitively identify the “correct” test to use. the test you choose depends on the data that you have. beyond the type of variable you have for your outcome (sometimes referred to as the dependent variable), you should also think about how big your sample size is, how closely your sample represents the overall population, and whether a parametric or non-parametric test is more appropriate. parametric tests assume that the underlying data are normally distributed, or that the data forms a bell curve. parametric tests may also have other assumptions, such as requiring that the values of one observation aren’t related to or influenced by the values of another observation, which is known as independence of observations. before determining whether a parametric or non-parametric test is most appropriate, consider not only the distribution of the data but also confirm that all assumptions of the test are met. 2 editorial reviewed article hypothesis, vol. 36, no. 1, 2024 table 1. a selection of parametric and non-parametric tests for categorical and continuous outcome variables. scenarios to consider to help operationalize the above table, let’s look at some different scenarios in which it may be appropriate to conduct a statistical test and walk through the process of selecting tests for each. scenario #1: a librarian wants to determine if their class on health literacy is effective. at the end of the class, they give students a 10-question quiz and give one point per question. thinking back to the first column in this series,1 the librarian can determine that this is ratio (continuous) data because there is a natural zero, meaning that if a student were to get 0 out of 10, it would mean that there was an absence of any correct answers. there is only a single group, which limits the librarian’s options for statistical tests. statistical tests with a single group at a single point in time can be challenging, because it’s necessary to have some reference point to which we are comparing that data. the librarian investigates the underlying distribution of data to determine if the data is normally distributed. if so, the librarian could do a single sample t-test, which could be used to determine if the mean score on the quiz differed from either a known or hypothesized population mean. for example, if this was a quiz that had been developed elsewhere and the authors of the original quiz had published an article in which they reported the mean score on the quiz of a large, representative cohort, the librarian might use this as a hypothesized mean, or a value that is assumed to the mean score of a population. they could then compare the mean quiz scores for students who have participated in their class to that reference point. recalling the previous column on central tendency and distribution,2 we see that, because this test is comparing means, it would not be appropriate if the data were not normally distributed. if instead, the quiz scores skewed either positively or negatively, the librarian might instead decide to conduct a non-parametric test, such as a one-sample wilcoxon signed rank test. in a one-sample wilcoxon signed rank test, the librarian would be testing to see if there was a statistically significant difference between the sample’s median and the hypothesized median. for more information on the t-test and its alternatives, consult herzog, francis & clarke 5. 3 editorial reviewed article hypothesis, vol. 36, no. 1, 2024 scenario #2: that same librarian continues to teach the class on health literacy but starts introducing new modalities while continuing to give students the same 10-question quiz. after several years, they have quiz scores from students who participated in hybrid classes, in-person classes, and online classes. the librarian wants to know whether the modality of delivery has any impact on the quiz scores. the librarian is still assessing continuous data as we were in scenario 1, but there are now more than two groups to consider. because there are three different groups (hybrid, in-person and online), the librarian might consider doing an anova, or an analysis of variance, which is a parametric test. an anova can be either a one-way anova or a two-way anova depending on the number of independent variables, or factors, being included. in a one-way anova, there is one categorical independent variable (i.e., the teaching modality) and one continuous dependent variable (i.e., the quiz scores). the one-way anova shows whether there are any statistically significant differences in the means of the different groups. in comparison, a two-way anova can include two categorical independent variables. for example, the librarian may have also had two different lengths of classes, a 60-minute session and a 90-minute session. if the librarian wanted to assess whether the impact of both the teaching modality (e.g., hybrid, in-person, online) and the length of session (e.g., 1 hour, 1.5 hours) has any impact on quiz scores, they could conduct a two-way anova. the anova is comparing the mean scores and assumes that the underlying data are parametric. if the data were non-parametric, the librarian might consider conducting a kruskal-wallis test. to learn more about anovas and kruskal-wallis, consider reviewing sarty6. scenario #3: the librarian is now working with a new program and is adapting their teaching and assessment strategies. the librarian can’t give the same 10-point quiz in this program, but they are still offering course-integrated instruction. in addition to this class, the librarian has also developed an optional homework activity for students. the librarian wants to assess whether completing the optional homework activity has any impact on whether students pass or fail. there are now two groups (students who complete the homework, students who do not complete the homework) and a categorical outcome (passing or failing the course). if a parametric test is appropriate, the librarian could consider using a chi-square test to evaluate whether the distribution of the outcome (passing or failing) is significantly different between students who did and did not submit the homework. in a chi-square test, data would be organized into a table known as a contingency table, which shows the frequencies of observations for each combination of variables (i.e., students who complete the homework and pass and those who complete the homework and fail). a chi-square test compares the actual number of observed counts in the table to what you would expect to see if there was no relationship between the different variables. to learn more about chi-squared analyses, consult jarman7. however, rather than concluding that the homework activity has an impact on whether students pass or fail, we would instead be asking whether there is an association between submitting the optional homework and the likelihood of passing or failing the course. this is because, although we can establish whether there is some sort of relationship between the 4 editorial reviewed article hypothesis, vol. 36, no. 1, 2024 variables, we cannot establish a cause-and-effect relationship. in statistics, causation means that a change in the independent variable will cause a change in the dependent variable, or outcome. statisticians will often note that “correlation does not imply causation,” which emphasizes that the existence of a relationship between variables does not mean that causation has been established. in observational studies, it can be difficult to establish causation because there may be confounding variables (e.g., students who don’t submit the optional homework may have more commitments outside of school) and the researcher may not be able to manipulate variables directly (e.g., the homework is optional, so the librarian cannot force one group to do the homework while the other does not). interpreting results sometimes, the challenge with statistics can be understanding the results of the analysis. what exactly does it mean to be “statistically significant”? p-values are ubiquitous, but that doesn’t mean that they’re always interpreted correctly. to understand p-values, we need to understand two concepts: a null hypothesis and an alternative hypothesis. a null hypothesis is the assumption that there is no difference. for example, in scenario #2 above, the null hypothesis would be that there is no difference in the quiz scores from the different teaching modalities, and that any difference is just due to chance. an alternative hypothesis states that the null hypothesis is untrue (i.e., that there is a difference in the quiz scores from the different modalities). but what does this have to do with p-values? the p in p-value stands for probability. it describes the probability that, in a world where the null hypothesis is true, you would observe data like this. so, we assume that we live in a world where the teaching modality doesn’t have a relationship with the quiz scores, and then we look at the data from the quizzes and see how likely it is we would see those scores in that world. one important thing to remember is that a p-value doesn’t tell you that the alternative hypothesis is true–it’s not testing whether the teaching modality is related to quiz scores. instead, a small p-value indicates that it’s unlikely we would see this data if the null hypothesis were indeed true (i.e., if there was no difference). what exactly the threshold for a "small" p-value should be is controversial. the most widely used value is less than .05. this essentially means that, in a world where the null hypothesis is true, there’s less than a 5% chance of getting your results. although this cutoff point is standard, it’s also somewhat arbitrary. for more information about some of the history and controversy around p-values, consult amrhein, greenland mcshane8 and kennedy-shaffer9. the most important thing to keep in mind when we think about p-values is this: a low p-value isn’t proof, it’s evidence. a p-value of .001 does not definitively prove that something is absolutely true, it just means that it’s highly unlikely that we would see this data in a world where it’s not. conversely, a p-value of over .05 doesn’t mean that something isn’t real or doesn’t matter, it just means that it’s more likely. just as we have to be intentional about choosing which tests we use, we have to be cautious about our interpretations. references 1. bakker cj. an introduction to statistics for librarians (part one): types of data. hypothesis res j health inf prof. 2022;34(1). doi:10.18060/26428 5 https://dx.doi.org/ editorial reviewed article hypothesis, vol. 36, no. 1, 2024 2. bakker c. an introduction to statistics for librarians (part two): frequency distributions and measures of central tendency. hypothesis res j health inf prof. 2023;35(1). doi:10.18060/27162 3. carlson s. data analysis: making sense of the numbers. presented at: department of family medicine community health, university of minnesota; 2016; minneapolis, mn. 4. hoskin t. parametric and nonparametric: demystifying the terms. mayo clinic. https://www.mayo.edu/research/documents/parametric-and-nonparametric-demystifying-theterms/doc-20408960 5. herzog mh, francis g, clarke a. variations on the t-test. in: understanding statistics and experimental design: how to not lie with statistics. springer international publishing; 2019:51-59. accessed may 28, 2022. http://library.oapen.org/handle/20.500.12657/23029 6. sarty ge. introduction to applied statistics for psychology students. university of saskatchewan open press; 2022. accessed december 30, 2023. https://openpress.usask.ca/introtoappliedstatsforpsych/ 7. jarman kh. bunco, bricks, and marked cards: chi-squared tests and how to beat a cheater. in: beyond basic statistics: tips, tricks, and techniques that every data scientist should know. wiley; 2015:47-68. 8. amrhein v, greenland s, mcshane b. scientists rise up against statistical significance. nature. 2019;567(7748):305-307. doi:10.1038/d41586-019-00857-9 9. kennedy-shaffer l. before p < 0.05 to beyond p < 0.05: using history to contextualize p-values and significance testing. am stat. 2019;73(sup1):82-90. doi:10.1080/00031305.2018.1537891 6 https://dx.doi.org/10.18060/27162 https://www.mayo.edu/research/documents/parametric-and-nonparametric-demystifying-the-terms/doc-20408960 https://www.mayo.edu/research/documents/parametric-and-nonparametric-demystifying-the-terms/doc-20408960 http://library.oapen.org/handle/20.500.12657/23029 https://openpress.usask.ca/introtoappliedstatsforpsych/ https://dx.doi.org/10.1038/d41586-019-00857-9 https://dx.doi.org/10.1080/00031305.2018.1537891 methods moment hypothesis vol. 32 no.1 fall/winter 2020 case studies nina exner, phd, mls assistant professor / research data librarian virginia commonwealth university richmond, va overview case studies are common, but also commonly misunderstood. so, for this first methods moment column, let’s take an introductory look at the case study methodology. a case study is an approach that takes a wide view of a narrowly-defined circumstance. like all methods, there are many disciplinary variations and complexities to making a great case study. to keep an introductory focus, i will discuss three core elements of case studies: the case, the situation, and the triangulation of data. regardless of variations in jargon, these concepts are good starting places. consider a diagram of how these core elements might look in a research question: figure 1: diagram of the case, situation, and triangulation in a research question methods moment hypothesis vol. 32 no.1 fall/winter 2020 the “case” refers to the setting of a case study. a case is usually a single environment with clear boundaries where the research topic of interest is happening. it defines the people, places, and things that a researcher might consider (or not consider) to be part of the case. so, in the example above, we would only consider data to be in the case when that data interacts with the librarian/clinic partnership. a community member who visited the clinic and used resources recommended by the librarian is definitely relevant to the “case.” on the other hand, a peer librarian is probably not part of the “case,” because they are not involved with the librarian-clinic partnership. cases are often a specific place or organization. we often think of typical cases being a library or health center, but a case could also be a professional association, conference, or class. the “situation” refers to what is happening at that case. this is what case study research question(s) are trying to understand. research questions for case studies usually focus on what and how questions – what actually happened, and/or how it occurred. the situation might be a specific event, like a program or change in leadership, or ongoing situations. examples of ongoing situations for a case study might be the implementation of a particular team structure or information desk process. like the case, the best situations for a case study are ones that can be clearly defined. the “triangulation of data” has two parts: data sources and data analysis. the goal is to identify sources which can be combined (triangulated) to understand the situation. a rigorous case study requires a systematic approach to comparing multiple data sources in the case. we usually use the word triangulation to talk about the analysis phase, but it has implications for data collection, too. data collection and analysis must be planned together. data collection in a case study looks at how the situation is experienced from multiple perspectives in the case. because of the experience element, case studies predominantly should include qualitative data and may also include quantitative data. case study data sources can be any sources that tell us something about the situation. we call it triangulation because we are looking for common directions in perspectives on the situation, which is why it is crucial to include/pull from multiple data sources to improve perspective on the situation. thus, data analysis for case studies (which i am focusing on as “triangulation,” but you might also search for the related term “data integration”) organizes what the data sources have to say and then seeks commonalities. while case studies may also highlight data that contradict majority perceptions in the case, most of the analysis should focus on identifying commonalities or areas of agreement from the multiple data sources. qualitative coding approaches are a common way to do this, which involves extracting pieces of text or other data and giving them labels relevant to the research question. tables or spreadsheets are good ways to organize the analysis. analytical methods moment hypothesis vol. 32 no.1 fall/winter 2020 approaches should list pieces of text and add labels that can be compared between the data sources. when planning a case study, it is good to think about several issues: ● what situation does the research question ask about? research questions appropriate to case study methods tend to start with the words how or what, such as, “how does an academic library experience adding a community health satellite location?” ● what is the case? the case should have clear boundaries, so that researchers can clearly determine participants and documents that are or are not part of the case. examples might be specific locations experiencing the situation (“how do baz university and quux college libraries experience adding community health satellites?”), or a definable event (“how did the wubble center’s 2020 wubble institute experience receiving and addressing code of conduct violation reports?”). ● how does the study triangulate meaningful findings? to triangulate, the case will need to cross the perspectives of multiple data sources that all have useful but different perspectives on the situation. keep in mind that case studies as a methodological approach are different from (a) clinical case studies, in which clinicians analyze how a patient’s symptoms typify or contrast with the textbook expectation of the presentation of symptoms; (b) teaching case studies, which are real or mock summaries of a situation written to encourage discussion and enable learning; or (c) assessment case reports, which report assessment data and lessons learned from a program in order to measure personal effectiveness rather than rigorously study the situation or program (see also alpi & evans, 2019, in resource articles list, to understand more specifically jmla’s case reports format). if you decide to read further on case studies methodologically, be sure to look for the triangulation or integration of data. example of (mock) application reyes wants to understand the effect of a new interprofessional training institute at their hospital. they consider personal biases and (being a librarian) decide that evaluating impacts of librarian-related interprofessional projects might be a separate case report topic. nevertheless, they are still interested in a more rigorous study of the wider effects and experiences happening because of the institute. a case study fits this broader interest in studying a range of perspectives on a focused event. reyes decides to scope the case to clinician-trainees in the interprofessional institute who are piloting interprofessional teams. they decide that their research question is, “how do clinicians methods moment hypothesis vol. 32 no.1 fall/winter 2020 learning to work on interprofessional teams incorporate interprofessional work in patient care planning?” reyes then has to look for sources of information. they start with observing the institute’s training sessions. to understand the experiences of the interprofessional teams, reyes wants to interview trainees and ask about care planning. they initially try to interview everyone who is piloting interprofessional projects. reyes quickly finds that it is too hard to schedule interviews with everyone. they change to a short survey of institute trainees and their teams, plus two coffee-hour focus groups. reyes wants more perspectives in their data, so they decide to incorporate content analyses (looking at documents and identifying and categorizing text relevant to the research question, as mentioned above) of the project summaries that the trainees submitted to the institute. thus, their data sources are (1) training observation notes; (2) surveys; (3) focus groups; and (4) project documents. the data is collected into a series of files, and the files are numbered for anonymity. a key to the file numbers is stored separately. for analysis, reyes creates a triangulation spreadsheet. in the literature, they find a typology that says interprofessional education has outcomes on three levels: the individual’s learning, team dynamics, and patient-focused outcomes. another article suggests common categories of outcomes. therefore, for each datum of interest, reyes decides to have a space for personal, team, and care outcomes. they have a master list of terms from the literature to assign (technically, “code”) each event with an outcome category. the spreadsheet headers look like this: figure 2: headers of reyes' data triangulation matrix when reyes writes up their article, they decide to take a storytelling approach that describes the context of the hospital and the training institute in detail. they decide to organize the writing of codes in a story-like flow from personal outcomes to team outcomes to patient outcomes, to give the reader a sense of the impacts flowing from learning to action. there are many other ways they could choose to write the case study; storytelling is simply what comes most comfortably to their personal writing style. resources web content 1. axman, linnea. qualitative research methods, case study research. https://youtu.be/6sg0m8fn99e https://youtu.be/6sg0m8fn99e methods moment hypothesis vol. 32 no.1 fall/winter 2020 2. gibbs, graham. types of case study. part 1 of 3 on case studies. https://youtu.be/gqfoq7c4ue4. in playlist social research methods and design, https://www.youtube.com/playlist?list=plirezjzohkvxax8zzufusai4jdukeexwx 3. russell, pia. case study research, university of victoria libguides. https://libguides.uvic.ca/educationresearchmethods/education/researchmethods/boo ks/casestudy 4. van der ham, vanessa. analysing case studies | massey university. https://youtu.be/6sg0m8fn99e 5. whitaker, evans. case studies, ucsf guides at university of california, san francisco. https://guides.ucsf.edu/c.php?g=100971&p=655222 articles 1. alpi km, evans jj. distinguishing case study as a research method from case reports as a publication type. journal of the medical library association [internet]. medical library association; 2019; 107:1–5. doi: 10.5195/jmla.2019.615 <https://jmla.mlanet.org/ojs/jmla/article/view/615>. 2. baxter p, jack s. qualitative case study methodology: study design and implementation for novice researchers. qual.rep. [internet] 13(4):544–59. <https://nsuworks.nova.edu/tqr/vol13/iss4/2> 3. crowe s, cresswell k, robertson a, huby g, avery a, sheikh a. the case study approach. bmc med.res.methodol. [internet]. 2011;11:100. doi: 10.1186/14712288-11-100 <https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/1471-2288-11100> 4. flyvbjerg b. five misunderstandings about case-study research. qual inq [preprint version reposited arxiv] [internet]. 2006;12(2):219–45. doi: 10.1177/1077800405284363 <https://arxiv.org/pdf/1304.1186.pdf> 5. gibbert m, ruigrok w, wicki b. what passes as a rigorous case study? strateg. manag. j. [internet] [preprint version]. 2008;29(13):1465–74. doi: 10.1002/smj.722 <https://www.alexandria.unisg.ch/53678/1/gibbert%2c%20ruigrok%20%26%20kae s%20smj%202008%20manuscript.pdf> 6. hyett n, kenny a, dickson-swift v. methodology or method a critical review of qualitative case study reports. international journal of qualitative studies on health and well-being. informa healthcare; 2014; 9. doi 10.3402/qhw.v9.23606 <https://pubmed.ncbi.nlm.nih.gov/24809980/> 7. moseholm e, fetters md. conceptual models to guide integration during analysis in convergent mixed methods studies. methodol. innov. [internet]. 2017;10(2). doi: 10.1177/2059799117703118 <http://journals.sagepub.com/doi/10.1177/2059799117703118> https://youtu.be/gqfoq7c4ue4 https://www.youtube.com/playlist?list=plirezjzohkvxax8zzufusai4jdukeexwx https://libguides.uvic.ca/educationresearchmethods/education/researchmethods/books/casestudy https://libguides.uvic.ca/educationresearchmethods/education/researchmethods/books/casestudy https://youtu.be/6sg0m8fn99e https://guides.ucsf.edu/c.php?g=100971&p=655222 https://jmla.mlanet.org/ojs/jmla/article/view/615 https://nsuworks.nova.edu/tqr/vol13/iss4/2 https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/1471-2288-11-100 https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/1471-2288-11-100 https://arxiv.org/pdf/1304.1186.pdf https://www.alexandria.unisg.ch/53678/1/gibbert%2c%20ruigrok%20%26%20kaes%20smj%202008%20manuscript.pdf https://www.alexandria.unisg.ch/53678/1/gibbert%2c%20ruigrok%20%26%20kaes%20smj%202008%20manuscript.pdf https://pubmed.ncbi.nlm.nih.gov/24809980/%3e http://journals.sagepub.com/doi/10.1177/2059799117703118 methods moment hypothesis vol. 32 no.1 fall/winter 2020 8. oliver-hoyo m, allen d. the use of triangulation methods in qualitative educational research [internet]. nsta news. 2005. <https://www.nsta.org/publications/news/story.aspx?id=51319> 9. rashid y, rashid a, warraich ma, sabir ss, waseem a. case study method: a step-by-step guide for business researchers. int j qual methods [internet]. 2019;18. doi: 10.1177/1609406919862424 <http://journals.sagepub.com/doi/10.1177/1609406919862424> 10. yazan b. three approaches to case study methods in education: yin, merriam, stake. qual rep [internet]. 2015;20(2):134–52. <https://nsuworks.nova.edu/tqr/vol20/iss2/12/> 11. zucker dm. how to do case study research. in: teaching research methods in the humanities and social sciences [internet]. <https://scholarworks.umass.edu/nursing_faculty_pubs/2/> https://www.nsta.org/publications/news/story.aspx?id=51319 http://journals.sagepub.com/doi/10.1177/1609406919862424 https://nsuworks.nova.edu/tqr/vol20/iss2/12/ https://scholarworks.umass.edu/nursing_faculty_pubs/2/ hypothesis, vol. 37, no. 1, 2025 authors’ reply to rife et al: clarifying definitions and methods in assessing the accuracy of scite caitlin bakker, mlis, ahip-da, nicole theis-mahon, mlis, ahipb, sarah jane brown, mscc adiscovery technologies librarian, dr. john archer library, university of regina, regina, saskatchewan, canada, https://www.orcid.org/0000-0003-4154-8382 , caitlin.bakker@uregina.ca bevidence synthesis librarian & health sciences collections coordinator, health sciences library, university of minnesota, minneapolis, minnesota, https://www.orcid.org/0000-0002-6913-5195 , theis025@umn.edu clibrarian liaison to the college of pharmacy, health sciences library, university of minnesota, minneapolis, minnesota, https://www.orcid.org/0000-0001-7699-4417 , sjbrown@umn.edu cite as: bakker c, theis-mahon n, and brown sj. authors’ reply to rife et al: clarifying definitions and methods in assessing the accuracy of scite. hypothesis. 2025;37(1). doi: 10.18060/28651 bakker, theis-mahon, brown. all works in hypothesis are licensed under a cc by-nc 4.0 deed attribution-noncommercial 4.0 international. authors own copyright of their articles appearing in hypothesis. readers may copy articles without permission of the copyright owner(s), as long as the author(s) are acknowledged in the copy, and the copy is used for educational, not-for-profit purposes. for any other use of articles, please contact the copyright owner(s). 1 https://www.orcid.org/0000-0003-4154-8382 https://orcid.org/ mailto:caitlin.bakker@uregina.ca https://www.orcid.org/0000-0002-6913-5195 https://orcid.org/ mailto:theis025@umn.edu https://www.orcid.org/0000-0001-7699-4417 https://orcid.org/ mailto:sjbrown@umn.edu https://creativecommons.org/licenses/by-nc/4.0/ hypothesis, vol. 37, no. 1, 2025 we appreciate the time and attention rife et al. have spent considering our original paper, and in crafting their response1. we note with appreciation their encouragement of other researchers to perform external validations of their tool, and their acknowledgement of the potential biases of their own evaluation. in reflecting upon rife et al.’s statements1, we noted that there appear to be several misunderstandings and misstatements, and we appreciate the opportunity to provide clarification of our research and aims. rife et al. argue that we “restricted [our] analyses to citations of retracted works in systematic literature reviews, which artificially limits the types of statements that could be considered supporting or contrasting a specific claim.1” we agree that the decision to focus on systematic reviews is a notable limitation of our study, as is the decision to focus on retracted publications. these limitations, and their implications, were described in depth in the original article. in the limitations section of our article, we note the following: “our research has several notable limitations. first, we focus on a sample of publications both within a discipline and using a specific study design. while scite is not programmed to perform differently based on study design or discipline, it is possible that studies using different publications would reveal different levels of accuracy. we also chose to focus on systematic reviews that had cited at least one retracted publication. retraction is a relatively rare and extreme publication state. as such, the specific citations included in our sample are, by nature of their retracted status, not indicative of the majority of publications. although we chose to focus on these outliers under the assumption that these publications would warrant stronger critique, this does limit the generalizability of our findings. finally, this work was done in conjunction with a larger research project, rather than as a standalone project. as such, the sample was derived through this larger work, rather than being selected for the sole purpose of assessing scite. other sampling methods may lead to different results.2” we agree that focusing on a specific sample is a limitation of our study that could limit generalizability; however, we do not agree with the assessment of this as “artificially limit[ed].1” rather, we argue that our approach is indicative of one instance of information-seeking behavior. information-seeking behavior is defined as “the purposive seeking for information as a consequence of a need to satisfy some goal.3” it is inextricably tied to the specific needs and context of the user. as kuhlthau notes, “an information search is a learning process in which choices along the way are dependent on personal constructs rather than on one universal predictable search for everyone.4” we engaged with a tool based on our specific information need, which was grounded in an interest in a specific subject area and study design. rife et al.’s positioning of our approach as “artificially limited” is seemingly at odds with well-established principles and practices of information-seeking behavior. we believe that rife et al.1 misunderstood and mischaracterized our research aims and intent in several ways. in response to our findings, they present three examples of citations in systematic reviews outside our original set and state their assumption of how we would have classified the citation. first, we note that one of their examples appears to be included in error. the first example they present is gatto5 citing li et al6. we believe that rife et al. may be conflating li et al.’s 2014 paper6 with their 2013 paper of the same name7, which was retracted. the retraction notice for li et al.’s 2013 paper8 states that the authors requested the 2 hypothesis, vol. 37, no. 1, 2025 retraction due to errors they discovered in the paper following its publication. in this notice, the authors state that they would be publishing a corrected version of the manuscript, which is presumably what was subsequently published in 20148. as we have found no evidence that the 2014 paper6 was also retracted, it seems that gatto5 was not citing a retracted publication, and as such this example is not relevant. rife et al.’s language throughout their manuscript may lead readers to misinterpret their assumptions as fact, as they repeatedly state what we “would have” done1. these statements are problematic for several reasons. we are of the opinion that researchers should attempt to avoid positioning their hypotheses, assumptions, or beliefs as proven fact, and that to state unproven beliefs as fact is contrary to and undermines the principles of the scientific method. we are reminded of faraday’s assertion of the importance of “distinguish[ing] that knowledge which consists of assumption, by which i mean theory and hypothesis, from that which is the knowledge of facts and laws; never raising the former to the dignity or authority of the latter. . . 9” rife et al.’s positioning of their assumptions of our classification scheme as fact is especially problematic as their assumptions are incorrect. in their second example, they assert that we would have classified the following passage in zhang et al.10 as supporting: “some cross-sectional studies have shown a close association between thyroid disease and metabolic disorders (i.e., metabolic syndrome [mets] and its components) (11–13). mets is characterized by a cluster of abnormal metabolic parameters consisting of insulin resistance, central obesity, type 2 diabetes, impaired glucose tolerance, hyperinsulinemia, and dyslipidemia (14). the global prevalence of mets is between 11.6% and 62.5% (15).” (p. 2) (emphasis added by rife et al.1) rife et al. state that “[a]gain, because kaur does not mention the fact that zhang et al. was retracted, bakker et al. would classify this citation as supporting.1" we would like to clarify that zhang10 is not the retracted publication; kaur11 is the retracted publication. this is a passage from zhang’s 2021 paper citing kaur’s 2014 retracted publication, not kaur citing zhang. in addition to clarifying which publication is retracted, we would also like to clarify that rife et al.’s statement is wholly inaccurate1. this citation was not included in our original study, since it did not meet our inclusion criteria. however, after consulting zhang10, we confirmed that we would have classified this citation as mentioning, as it is by scite. this is based on our holistic evaluation of how the citation is functioning within the paper overall. in this case, the citation occurred in the introduction and functioned to provide context and background, rather than furthering the findings of that retracted publication. while one could argue that such a citation may be inappropriate, appropriateness of citation was not in the scope of our work. rife et al.’s misstatement leads us to believe that they may have fundamentally misunderstood our classification system1. rife et al. appear to consider our classification scheme to be a binary one, where supporting citations are determined to be such by the mere absence of the word retracted1. our original manuscript describes a three-part classification system rather than a binary classification system2 . we classified citations as supporting, contrasting, or mentioning, depending on how 3 hypothesis, vol. 37, no. 1, 2025 they functioned within the context of the citing article. we attempted to align this classification with what was proposed by nicholson et al. in their qss article12. in that article, nicholson et al. state that “scite focuses on the authors’ reasons for citing a paper" and that “[e]xtracted citation statements are classified into supporting, contrasting, or mentioning, to identify studies that have tested the claim and to evaluate how a scientific claim has been evaluated in the literature by subsequent research.12" in the context of evidence synthesis, our area of study, such evaluation may manifest in a number of ways, including the systematic assessment of potential biases and methodological issues through formalized risk of bias assessments, or through the identification of outliers through statistical tests, such as sensitivity and leave-one-out analyses, among others. the decision to include or exclude a publication, and the ways in which those publications are included, is a multi-faceted one. although authors consider whether the study meets eligibility criteria, they also may consider whether the methodological quality of the study is sufficient to include it in particular analyses, or if additional analyses should be conducted to account for the influence of any one study. in addition to applying inclusion and exclusion criteria, authors of evidence syntheses must evaluate and apply their scientific judgment to each individual report and its claims. to be included or excluded in an evidence synthesis is, in some cases, evidence of “how a scientific claim has been evaluated in the literature by subsequent research.12” our description of our three-part classification system was tailored to the core readership of hypothesis: health information professionals who are likely familiar with principles of evidence-based practice. we classified publications as supporting if they were included as reports in the evidence synthesis–that is to say, if they were included as component studies that had undergone the rigorous selection, appraisal, extraction and syntheses processes and were subsequently included as component data within the larger study. of note, not all included reports were considered supporting citations, as an included report whose methodology and findings were heavily critiqued, for example, would not have been classified as supporting. we classified citations as contrasting if they were treated as retracted, that is to say if the publication was treated as “contain[ing] such seriously flawed or erroneous content or data that their findings and conclusions cannot be relied upon,13” whether that was due to a concern with the report, such as findings that were inexplicably divergent from other comparable studies, or the underlying research, such as concerns raised about methodological issues or suspicions of misconduct. the majority of citations in our study were classified as mentioning, as they were by scite2. we understand that to those unfamiliar with evidence synthesis, our reference to “included reports” in evidence syntheses could have led to the misunderstanding that this simply meant any cited paper, as opposed to a study that had gone through an intensive review, appraisal and evaluation process to assess its quality and fit for purpose before being incorporated into the body of evidence being presented. we also recognize that our brief description of supporting, contrasting and mentioning citations omitted various nuances that we considered when assessing citations. while it had been our aim to be as succinct as possible rather than describing the extensive decision-making process behind each classification decision, we believe that such brevity may have posed challenges for some readers, for which we apologize. 4 hypothesis, vol. 37, no. 1, 2025 the third example rife et al. present, stavale14 citing tsukumo15, is a systematic review that “aimed to investigate the profile of medical and life sciences research retractions from authors affiliated with brazilian academic institutions.14" given that the scope of this project is focused specifically on identifying and analyzing retracted publications, we cannot speak to how we would have classified this citation as it is an entirely different situation and use case than those in which we were primarily interested. like many information professionals, we are intrigued and excited by technology and the opportunities for identifying, accessing and synthesizing an ever-increasing amount of information. tools such as scite, perplexity, research rabbit, primo research assistant and others have the potential to aid researchers in completing complex tasks, such as identifying gaps in the literature, synthesizing publication findings, and generating text. as noted in information literacy in the age of algorithms, “these mysterious black boxes can answer in seconds a question that formerly required hours in a library (though the answer may not necessarily be entirely accurate).16” such tools have the potential to–and arguably already are–revolutionizing the ways in which students and researchers access, use, and create information. the need for algorithmic literacy and critical discourse in the context of evidence synthesis and evidence-based practice has never been greater. as bender et al. note in their seminal paper on large language models: “the risks associated with synthetic but seemingly coherent text are deeply connected to the fact that such synthetic text can enter into conversations without any person or entity being accountable for it.17” in other words, an algorithm functions to take an input and produce an output but has no understanding of or accountability for the meaning of what is being ingested or produced. as a growing number of tools enter the marketplace, and existing tools are further developed, refined and integrated into everyday life, it is essential that information professionals actively engage in critical evaluation to ensure that those using the tool have full knowledge of its capabilities and its limitations. credit caitlin bakker: conceptualization, writing – original draft, writing review & editing; nicole theis-mahon: conceptualization, writing -original draft, writing review & editing; sarah jane brown: conceptualization, writing – original draft, writing review & editing references 1. rife sc, nicholson jm, uppala a, rosati d. reply to bakker et al.: assessing the accuracy of the scite citation classification system requires the same definitions to be used for training as for testing. hypothesis. 2025; 37(1). doi: 10.18060/28018 2. bakker c, theis-mahon n, brown sj. evaluating the accuracy of scite, a smart citation index. hypothesis. 2023;35(2). doi: 10.18060/26528 3. wilson td. human information behavior. informing sci. 2000;2(2):049–56. doi: 10.28945/576 4. kuhlthau cc. seeking meaning: a process approach to library and information services. 5 hypothesis, vol. 37, no. 1, 2025 2nd ed. westport, connecticut: libraries unlimited; 2004. 247 p. 5. gatto rg. molecular and microstructural biomarkers of neuroplasticity in neurodegenerative disorders through preclinical and diffusion magnetic resonance imaging studies. j integr neurosci. 2020;19(3):571–92. doi: 10.31083/j.jin.2020.03.165 6. li w, yu j, liu y, huang x, abumaria n, zhu y, et al. elevation of brain magnesium prevents synaptic loss and reverses cognitive deficits in alzheimer’s disease mouse model. mol brain. 2014;7(1):65–65. doi: 10.1186/s13041-014-0065-y 7. li w, yu j, liu y, huang x, abumaria n, zhu y, et al. elevation of brain magnesium prevents and reverses cognitive deficits and synaptic loss in alzheimer’s disease mouse model. j neurosci off j soc neurosci. 2013 may 8;33(19):8423–41. doi: 10.1523/jneurosci.4610-12.2013 8. author-initiated retraction: li et al., elevation of brain magnesium prevents and reverses cognitive deficits and synaptic loss in alzheimer’s disease mouse model. j neurosci. 2014 apr 16;34(16):5733–5733. doi: 10.1523/jneurosci.1265-14.2014 9. faraday m. xxiii. a speculation touching electric conduction and the nature of matter. lond edinb dublin philos mag j sci. 1844 feb;24(157):136–44. doi: 10.1080/14786444408644817 10. zhang c, gao x, han y, teng w, shan z. correlation between thyroid nodules and metabolic syndrome: a systematic review and meta-analysis. front endocrinol. 2021;12:730279. doi: 10.3389/fendo.2021.730279 11. kaur j. a comprehensive review on metabolic syndrome. cardiol res pract. 2014;2014:943162. doi: 10.1155/2014/943162 12. nicholson jm, mordaunt m, lopez p, uppala a, rosati d, rodrigues np, et al. scite: a smart citation index that displays the context of citations and classifies their intent using deep learning. quant sci stud. 2021;2(3):882–98. doi: 10.1162/qss_a_00146 13. barbour v, kleinert s, wager e, yentis s. guidelines for retracting articles [internet]. committee on publication ethics; 2009 sep [cited 2024 oct 16]. available from: https://publicationethics.org/node/19896 14. stavale r, ferreira gi, galvão jam, zicker f, novaes mrcg, oliveira cm de, et al. research misconduct in health and life sciences research: a systematic review of retracted literature from brazilian institutions. plos one. 2019;14(4):e0214272. doi: 10.1371/journal.pone.0214272 15. tsukumo dml, carvalho-filho ma, carvalheira jbc, prada po, hirabara sm, schenka aa, et al. loss-of-function mutation in toll-like receptor 4 prevents diet-induced obesity and insulin resistance. diabetes. 2007 aug;56(8):1986–98. doi: 10.2337/db06-1595 16. head aj, fister b, macmillan m. information literacy in the age of algorithms [internet]. 6 https://publicationethics.org/node/19896 hypothesis, vol. 37, no. 1, 2025 united states: project information research institute; 2020 jan [cited 2024 oct 24]. available from: https://projectinfolit.org/publications/algorithm-study 17. bender em, gebru t, mcmillan-major a, shmitchell s. on the dangers of stochastic parrots: can language models be too big?. in: proceedings of the 2021 acm conference on fairness, accountability, and transparency. virtual event canada: acm; 2021. p. 610–23. doi: 10.1145/3442188.3445922 7 https://projectinfolit.org/publications/algorithm-study peer reviewed article hypothesis, vol. 36, no.1, 2024 development of an empowering language assessment tool for type 1 diabetes gwen geiger wolfe, mph, mls*a, sally bowler-hill, ma, msis, pmpb, lisa m. acuff, mph, msis, ahipb ascience and engineering librarian, ku libraries, university of kansas, lawrence, kansas, https://www.orcid.org/0000-0002-3309-2488 , gwen.geigerwolfe@ku.edu, bmanager, administrative operations, health sciences library and informatics center, university of new mexico, albuquerque, new mexico, https://www.orcid.org/0000-0002-8371-0408 , sbowler-hill@salud.unm.edu, beducation and research librarian, health sciences library and informatics center, university of new mexico, albuquerque, new mexico, https://www.orcid.org/0000-0001-8427-4073 , lacuff@salud.unm.edu, cite as: geiger wolfe g, bowler-hill s, acuff lm. development of an empowering language assessment tool for type 1 diabetes. hypothesis. 2024;36(1). doi:10.18060/27668 geiger wolfe, bowler-hill, acuff. all works in hypothesis are licensed under a cc by-nc 4.0 deed attribution-noncommercial 4.0 international. authors own copyright of their articles appearing in hypothesis. readers may copy articles without permission of the copyright owner(s), as long as the author(s) are acknowledged in the copy, and the copy is used for educational, not-for-profit purposes. for any other use of articles, please contact the copyright owner(s). 1 https://www.orcid.org/0000-0002-3309-2488 https://orcid.org/ mailto:gwen.geigerwolfe@ku.edu https://www.orcid.org/0000-0002-8371-0408 https://orcid.org/ sbowler-hill@salud.unm.edu https://www.orcid.org/0000-0001-8427-4073 https://orcid.org/ mailto:lacuff@salud.unm.edu https://creativecommons.org/licenses/by-nc/4.0/ peer reviewed article hypothesis, vol. 36, no.1, 2024 abstract introduction: this brief report describes the early development of an empowering language assessment tool (t1d-elat) for patient education materials (pems) designed for people with type 1 diabetes (t1d), consisting of a checklist to assess the frequency of disempowering language. methods: the authors followed professional society guidelines to develop a checklist to quantify the use of disempowering words and phrases. we determined inclusion and exclusion criteria and grouped checklist terms into functional categories. we also collected descriptive characteristics about each pem, such as publication date, authorship, and country of origin, and created a codebook defining these terms and descriptive characteristics. results: the authors tested the checklist in two rounds and added nine term variations as a result. we removed the imperative can/can’t/cannot during testing because it was not consistently used as an imperative. discussion: pilot testing the checklist informed revisions to the checklist and study protocol. future phases of this research project will also explore the context in which disempowering terms are used in these materials. introduction this brief report describes the development of an empowering language assessment tool (t1d-elat) for patient education materials (pems) designed for people with type 1 diabetes (t1d) and their caregivers. the tool consists of a checklist to document the frequency of disempowering language usage in pems authored by academic, clinical, government, and health-related organizations. disempowering language in diabetes care and education includes the use of stigmatizing, blaming, and condition-first words and phrases, such as “diabetic,” “good/bad control,” or “must.” the authors anticipate future usage of the t1d-elat in academic and clinical settings or by any professional evaluating pems for community dissemination. background information for this research is available on the open science framework1 and in the journal of consumer health on the internet2. methods professional society guidelines on the use of language in diabetes care and education informed the development of the checklist3. the guidelines list disempowering words and phrases with empowering alternatives. the checklist includes disempowering language relevant to written pems and excludes terminology specific to clinical communications. the t1d-elat assesses disempowering language because empowering alternatives are many and various. empowering alternatives for the word “control” used as a noun, for example “good control,” include “an a1c less than 7%” or “glucose values within target,” which focus on factual terms that are neutral3. the authors grouped t1d-elat words and phrases into functional categories: control (verb or adjective), control (noun), diabetic (adjective or noun), imperatives, prevent, and test (verb 2 peer reviewed article hypothesis, vol. 36, no.1, 2024 or noun); and excluded medical terms, such as diabetic ketoacidosis. table 1 provides a complete list of t1d-elat terms. in addition to disempowering language, the checklist collects descriptive characteristics, including country of origin, authorship, audience, and publication date. the authors subsequently excluded the audience characteristic because the terminology in pems for people with t1d was similar for adults, parents, and children. finally, we created a codebook to define the disempowering terms and descriptive characteristics and guide the application of the tool. 3 peer reviewed article hypothesis, vol. 36, no.1, 2024 table 1. t1d-elat disempowering words and phrases by category category disempowering words and phrases control as verb or adjective controlled / uncontrolled well controlled / poorly controlled control your / control the/ controlling your control as noun gylycemic control / glucose control / blood sugar control poor control / bad control good control / tight control diabetic as adjective or noun diabetic person / diabetes patients diabetic / nondiabetic (*exclude diabetic ketoacidosis) imperatives should / shouldn’t/should not have to / need to must /must not prevent prevent / prevention / preventing/ preventable test as verb test blood glucose / test blood sugar / test your blood sugar / blood sugar tested test as noun blood test test strips results the authors pilot tested the checklist in two rounds: informally with pems about type 2 diabetes and formally with eleven pems about t1d. we added nine variations during formal testing: control your, controlling, blood sugar control, preventing, preventable, test blood sugar, test your blood sugar, blood sugar tested, and blood test. we removed the terms can, can’t, and cannot during testing because they occurred in multiple contexts, often not as imperatives. the authors presented results from the formal pilot study at the empirical librarians conference in march 20234. 4 peer reviewed article hypothesis, vol. 36, no.1, 2024 outcomes the authors did not count instances of disempowering words in organizational names like the centers for disease control and prevention during pilot testing; however, we included them in the revised checklist because they potentially impact people’s perceptions and experiences. after pilot testing, the authors used the revised checklist to assess a sample of twenty-nine publicly available pems, and we are currently engaged in descriptive analyses. the changing nature of language, such as the shift from condition-first to person-first language in diabetes care and education, is one limitation of the t1d-elat. in future phases of this research, the authors plan to explore the context in which pems use disempowering language. in summary, the t1d-elat offers a tool for assessing the real-world use of disempowering language in pems about t1d. author contributions gwen geiger wolfe: conceptualization, investigation, methodology, writing – original draft, and writing – review sally bowler-hill: conceptualization, investigation, methodology, writing – original draft, and writing – review lisa m. acuff: conceptualization, investigation, methodology, writing – original draft, and writing – review references 1. acuff l, geiger wolfe g, bowler-hill s. disempowering language in online patient education materials for people with type 1 diabetes: a summative content analysis. open science framework. 2023. available from: https://osf.io/wqh35 2. acuff l, geiger wolfe g, bowler-hill s. the language of type 1 diabetes: why it matters in online patient education. j consum health internet. 2023, mar 07. cited 2023 oct 20; 27(1):86-97. doi:10.1080/15398285.2023.2167424 3. dickinson jk, guzman sj, maryniuk md, o’brian ca, kadohiro jk, jackson ra, et al. the use of language in diabetes care and education [internet]. diabetes care. 2017 dec 1. cited 2022 aug 26; 40(12):1790–9. doi:10.2337/dci17-0041 4. eaker c, sergiadis a. empirical librarians conference. open science framework. 2023. available from: https://osf.io/7hjte 5 https://osf.io/wqh35 https://dx.doi.org/10.1080/15398285.2023.2167424 https://dx.doi.org/10.2337/dci17-0041 https://osf.io/7hjte peer reviewed article hypothesis, vol. 37, no. 2, 2025 collaborative endeavors: a team perspective on co-authoring systematic reviews with medical students megan dearmond, mslis, mfaa, sherli koshy-chenthittayil, phdb, amy el stone, phdc aassociate professor/systematic review services librarian, jay sexter library, touro university nevada, henderson, nevada, https://orcid.org/0000-0001-7048-1039 , mde_armo@touro.edu bdata analyst, office of institutional effectiveness, touro university nevada, henderson, nevada, https://www.orcid.org/0000-0002-1054-455x , skoshy-c@touro.edu cassistant professor of medical education, kirk kerkorian school of medicine, university of nevada las vegas, las vegas, nevada, https://www.orcid.org/0000-0002-9140-6510 , amy.stone@unlv.edu cite as: dearmond m, koshy-chenthittayil s, and stone ael. collaborative endeavors: a team perspective on co-authoring systematic reviews with medical students. hypothesis. 2025;37(2). doi:10.18060/28386 dearmond, koshy-chenthittayil, stone. all works in hypothesis are licensed under a cc by-nc 4.0 deed attribution-noncommercial 4.0 international. authors own copyright of their articles appearing in hypothesis. readers may copy articles without permission of the copyright owner(s), as long as the author(s) are acknowledged in the copy, and the copy is used for educational, not-for-profit purposes. for any other use of articles, please contact the copyright owner(s). 1 https://www.orcid.org/0000-0001-7048-1039 https://orcid.org/ mailto:mde_armo@touro.edu https://www.orcid.org/0000-0002-1054-455x https://orcid.org/ mailto:skoshy-c@touro.edu https://www.orcid.org/0000-0002-9140-6510 https://orcid.org/ mailto:amy.stone@unlv.edu https://creativecommons.org/licenses/by-nc/4.0/ peer reviewed article hypothesis, vol. 37, no. 2, 2025 abstract introduction: evidence synthesis (es) projects have been on the rise, providing a way to work collaboratively without needing to use a lab or meet in person. es projects are a good method for all involved to learn more about the subject, to foster meaningful collaborations among team members, and to establish and improve mentoring skills. the authors conceptualized, formulated, drafted, and published a diagnostic accuracy systematic review on rapid antigen tests used to detect covid-19. during the review, varied challenges arose, many unanticipated. to address them the authors and team members developed effective strategies that may benefit future es projects. experience: the article describes the experiences of the varied team members, which included medical students, a librarian, faculty, and a data analyst. the article addresses the collaborative process including defining roles and responsibilities. next, the authors describe laying the foundation of the project so that as team members were recruited, the project flowed smoothly. the team then reflects on the management of the project over multiple years. as with many es projects, the protocol needed to be revised, and the authors share their approach to these changes. discussion: the librarian’s collaboration with both students and faculty members yielded significant benefits, including improved mentoring skills, enhanced team coordination, and diverse perspectives gained through interdisciplinary collaboration. the challenges faced by navigating the literature landscape and managing the team are also described. takeaways: librarians are in a unique position to advocate for multi-disciplinary teams, incorporating a data analyst early on, and judiciously using software in evidence synthesis projects. these projects help build the backbone of medical literature. introduction there has been an uptick in publishing evidence synthesis projects such as systematic and scoping reviews1, particularly since the start of the pandemic in 2020, as evidenced by a search for the phrase ‘systematic review’ in the record titles in pubmed, which showed 73,559 results from 2015-2019 and 165,216 from 2020-2025. similarly, the search for the phrase “scoping review” in pubmed, showed 3,455 results for 2015-2019 and 25,406 for 2020-2025, (the search was conducted on jan 15, 2025). evidence synthesis (es) projects provide an avenue for students and faculty to publish, for mentorship (student/faculty, student/student, and faculty/faculty), and for all involved to learn more about the process of and participation on a team project. in a recent effort to write a systematic review on the diagnostic accuracy of rapid antigen tests for covid-19 detection (full review published in oct 20242; protocol published in 20213) a range of challenges were encountered, many of which were unanticipated, such as issues with published studies, challenges coordinating and training a large group, and deciding how to organize files and tasks. the authors developed methods to streamline the work that they feel may benefit a larger group. the faculty (six in total) on the team coordinated a large group (11 in total) of osteopathic medical students to complete this project. the methods used to keep the project moving forward included dedicated work time, piloting each step of the process, and assigning students specific numbers of papers to review. additionally, the team used collaborative spreadsheets with 2 peer reviewed article hypothesis, vol. 37, no. 2, 2025 google applications (alphabet, inc). here, we present some challenges of completing a systematic review of diagnostic test accuracy and some tips and tricks for mentoring a large group of students. systematic reviews offer a rigorous research methodology that contributes to improving practices and institutional research footprints. the cost of foundational science, translational, and clinical research continues to increase due to the cost and access to reagents, equipment, and specialized facilities. while the cost of journal subscriptions and article access continues to rise, the material costs for systematic reviews remain lower than lab or field-based biomedical research. this lower material cost is useful for institutions with more research demand than facilities. in addition to material costs, faculty time and expertise are critical resources. the human power (personnel hours) needed to complete an evidence synthesis project is similar to, if not more than, foundational, translational or clinical research but without requiring reagents, equipment, and specialized facilities. however, training faculty members in systematic review methodology can exponentially increase the number of people capable of doing systematic reviews as they continue instructing trainees (students, residents, fellows). while training doesn’t ensure high-quality research products, as is seen across many different fields, collaborations with library faculty can increase the integrity of evidence synthesis projects. these aspects of resource investment and the potential amplification of initial training and skills make systematic reviews an option to help meet research demand. institutions considering adding evidence synthesis projects to their research portfolio must consider training, labor, software, and article access before embarking on such endeavors. experiences the collaborative process: establishing roles and responsibilities librarian expertise in information sciences librarian expertise can be both broad and in-depth on evidence synthesis projects. this review didn’t require any conceptualizing or brainstorming related to the research question, but often, this is something librarians can do in a team meeting when assisting with searching for existing reviews and studies on a topic to help shape and direct the research question for evidence synthesis. our team had a clear question in the pird (population, index test, reference test, diagnosis of interest) framework, often used in diagnostic accuracy question development4. librarians can competently identify databases and resources appropriate to search for the research question and are familiar with what is available at their institution and the resources freely available, including identifying grey literature resources. developing and implementing search strategies for all resources is a prominent skill non-librarians think should be the role of librarians implementing evidence synthesis projects5-7. however, this time-consuming and valuable part of the project is one aspect of the librarian’s contribution. a librarian colleague refers to it as developing the data collection instrument since the search strategy determines the articles to be screened. 3 peer reviewed article hypothesis, vol. 37, no. 2, 2025 librarians are skilled and efficient at finding articles. additionally, librarians know citation software and can assist and train the team in managing citations using a specified software, such as endnote. for our project, this provided an opportunity to train students in article retrieval and citation management. the collaboration between the methods’ expert (librarian) and the more accessible subject matter expert (trainees) helped enhance the development of the search strategy. medical students’ clinical insights and research skills medical students provide a unique and diverse perspective on clinical topics as they are new to the field. they benefit from seeing clinical challenges with new eyes since they begin their journey to become clinicians. students are more exposed to research methods earlier in their educational journey than ever before. this shift is driven by the increasingly competitive environments where trainees and potential trainees need to distinguish themselves from other applicants at each level of education8,9 (undergraduate medical education and graduate medical education). two of the students that the librarian worked closely with mentioned that the time spent with the librarian helped them in a residency interview and with one or two board exam questions. this early exposure in their undergraduate careers allows them to develop and refine their research skill sets before they matriculate into medical school10,11. further, the technological and internet literacy many of today’s students have is a clear result of their lived experiences where the internet and computers have always been available12,13. these are invaluable skills to leverage as they may be able to develop new techniques and approaches that utilize technology in innovative ways. the spectrum of roles for medical students in writing a systematic review can range from data collection and entry to foundational team members who contribute intellectually and technically to the entire process. we empowered our student team to explore the range of potential roles within the project and find where they felt comfortable contributing. a few of our students stepped up into the foundational team member roles, where the project relied on their involvement to proceed. for example, one of our earliest students helped write the protocol as well as piloted the reference deduplication process, the title/abstract screening, the full-text screening, and the risk of bias process before helping teach the other students on the team. a different student was content to pull data from the selected articles but did not have the time to pilot or develop processes for the review. when embarking on a project such as this, it is important to help students define their roles and responsibilities. faculty roles of mentorship, guidance, and content expertise depending on the project’s goals, the spectrum of faculty roles can be as broad as those for medical students. the faculty’s main responsibilities are to promote student learning and collaboration, including accountability and professionalism, to ensure the integrity of the research, to ensure equitable treatment of all project team members, and to identify the needed expertise for the execution of the project. the faculty members need to take responsibility for the equitable treatment of all members of the project team. this entails both ensuring that the members know and uphold their responsibilities (accountability), and that team members interact in respectful ways (professionalism). these aspects should be specifically detailed in the working agreement 4 peer reviewed article hypothesis, vol. 37, no. 2, 2025 (see appendix 1). some trainees struggle with both aspects and can become unprofessional when they fail to see the value of multidisciplinary approaches or become overwhelmed with other activities. it is the faculty members’ responsibility to highlight the value that each team member brings to the project and emphasize that research is not done in a vacuum but in an intricate network of expertise and perspectives. the team recommends using written, living documents as practical advice for faculty members to handle these roles and responsibilities. these documents are collaboratively written by the faculty and students that detail the expectations, roles and responsibilities, and timelines for the project. the team recommends a living document so that updates can be made as the project or team changes, and the agreements within the document can be revisited and discussed at each project milestone. flexibility is a component of professional life and our agreements and roles within these projects should remain flexible. there are many different templates of these types of documents available, and the version selected should be tailored to the project and its personnel. when developing the roles and responsibilities for this project, the faculty members discussed the project, expectations, and timelines with each trainee individually and collaborated on desired contribution levels and expectations. in the end, the final working agreement (appendix 1) that was collectively agreed upon allowed accountability, professionalism, and timely project completion. data scientists’ roles in project design, and analysis methodology the data scientist should be incorporated throughout the project with specific roles in three main stages, the first being the protocol writing stage. they can help identify the type of data that needs to be collected to answer the review question. for example, in a diagnostic accuracy review, our data scientist recommended additional data for sub-group analysis beyond test accuracy values. they can design the data collection instrument so that the extraction process runs smoothly. for example, we modified our data collection instrument to use standardized terminology, which helped with the meta-analysis. the second stage is the data extraction process and meta-analysis. someone with a statistics background was essential for these steps of the review. the data analyst helped the team identify papers that did not report outcomes needed for meta-analysis. additionally, they made informed decisions on the types of statistical techniques implemented. for example, due to the high heterogeneity of the data, the data scientist guided the methodological shifts to the meta-analysis. the final stage is data presentation and sharing. the data scientist can ensure that technical methodology and results are well described in the manuscript. they can also help make data publicly available on repositories such as github. for our project, they were integral in the writing and submission of the manuscript. laying the foundation of the project a well-designed es project starts with a detailed protocol and a librarian-driven comprehensive search strategy. the protocol drafting and submission was headed up by the team librarian with assistance from two of the medical students. the medical students began with the pre-written template from jbi methodology (formerly known as joanna briggs institute, adelaide, australia4) and worked with the librarian to test, refine, and clarify the 5 peer reviewed article hypothesis, vol. 37, no. 2, 2025 research question, inclusion/exclusion criteria, and the results from the search strategy. in this step, the librarian served as the main mentor of the students and the project. the search strings for covid and for the rapid antigen tests were initially drafted by the librarian and then refined with the help of the students. they quickly learned that there were validated searches that existed for covid, which formed the basis for the protocol’s search strategy. this was tricky since most of the strategy was drafted in 2020, before the codification of covid terms. in january 2021, as part of the annual changes in mesh (medical subject headings) by the national library of medicine, many of the covid indexing terms were upgraded from being supplemental concepts to being mesh terms. due to this, the search had to be extensively revised with the new mesh terms. additionally, following press (peer-review of electronic search strategies) guidelines14 helped give structure to the search and its revisions. managing the project at its peak, the project had six faculty members and eleven students working on the project simultaneously. managing this project required coordination across the faculty and the students to maintain consistency with individuals and prevent unnecessary redundancies. throughout this project, the trainees and faculty were responsible for screening, critical appraisal, and data extraction. the faculty mentors kept the team on track and revised timelines as needed. the librarian was more heavily involved in retrieving full-text articles and screening. the data scientist served as a validator for the data extraction and performed the meta-analysis. several strategies contributed to the success of managing this project with so many individuals. the number one strategy employed to manage this project was to have a one-hour, optional but highly recommended, “working” meeting each week. as a team, we would meet virtually every monday evening to work for an hour on the project. this provided a dedicated time on each person’s schedule to contribute to the review, along with a level of accountability. additionally, this provided a time to address concerns, inconsistencies, and clarify the tasks of each step in the review process. the students knew that the faculty would be expecting them to attend, and the faculty expected each other to be present. the first few minutes of each meeting were used to review the current tasks, update everyone on the progress of the project, and address any confusion or concerns. the last few minutes of each meeting were used to address goals, whether ongoing or for the upcoming week; discuss issues that may impact upcoming efforts, such as upcoming exams for the firstand second-year medical students; and to summarize the progress done in the meeting so far. as a tool to keep the project moving forward, this standing meeting ensured that at least a few hours of work were completed weekly regardless of our other time constraints. this prevented the project from getting lost in the “back burner” of responsibilities that we have as faculty and that our students have for their academic performance. we utilized google’s collaborative software apps, gdrive15 and gsheets16, to manage the progress of our project. we recommend using open-source, collaborative, and cloud-based software as the home of your reviews-in-progress. this offers a low-cost alternative to expensive software packages. the google collaborative software apps, which are cloud-based, allowed our group to access the materials anywhere at any time and see real-time 6 peer reviewed article hypothesis, vol. 37, no. 2, 2025 updates to the process. for each step in the process of the review, we utilized a different, customized spreadsheet, details of which can be found in the published review2. these spreadsheets were generated by the faculty to be easy to use, easy to understand, and provide consistency across the project members. for example, we used prioritized drop-down boxes for our full-text screening that listed the criteria from most important reason to least important reason. each line included the name of the person working on the item and the outcome of their assessment. we instructed the students to use the drop-down like a list of yes or no questions such that the first “no” they encountered would be the selected reason for exclusion. this helped increase the efficiency of the process by only selecting one exclusion reason even if multiple reasons were present. this also allowed easy collation of the exclusion reasons when we compiled the tables for the required appendices that list the reasons studies were removed from the review. as we moved forward on our review, we trained the group on how to use each spreadsheet. while we had weekly meetings where we would discuss the methods to be used during the review process, there was rarely a week that everyone on the team could attend. to address this, we recorded the training discussions that occurred at the start of each new step. once completed, the recording was uploaded into a shared google drive folder for everyone to access. this accomplished several goals, the first being that the absent members had access to the same visual and oral directions as other team members. second, it allowed team members to go back and revisit the instructions as needed. finally, it generated a record of each step and how the steps were completed. the final key project management strategy that we used as part of this project was to loosely delegate tasks. the team discussed feasible metrics (e.g., number of article assessments) and deadlines. an example assignment to the students was to complete at least ten article assessments over the course of two weeks. we, however, did not assign specific articles to each student. instead, the students identified the ten assessments they wished to do in those two weeks. this strategy kept students accountable for the work but did not call out students when they completed their assigned tasks. occasionally, some students would be unable to finish the task by the deadline, but others would step in to help their peers. providing the students with some structure and some flexibility increased the confidence, happiness, and productivity of our students throughout the project. however, flexibility with deadlines did cause delays in the project timeline. revising our protocol as with any plan, our original published protocol did not survive its encounter with the systematic review itself. several of the faculty collaborating on this project had completed the jbi training together and had discussed the project at length during that initial phase. later, as we wrote the protocol with a few students, we further refined our thoughts. however, as more students were brought on, there became a distance between the ideation in the faculty’s minds and the students’ understanding of their tasks. many portions of our protocol required tweaking with minor adjustments to match the reality of the work. we also encountered larger hurdles that required more substantial revisions to our protocol. when we began working on our full-text screening, we quickly discovered that the phrasing 7 peer reviewed article hypothesis, vol. 37, no. 2, 2025 of our inclusion and exclusion criteria in our protocol did not match the goal of the research question. we had trained our students to follow the letter of the criteria, not the spirit. thus, when a student and a faculty member served as the two reviewers for a specific article, there was a discrepancy between them on whether the article met the inclusion criteria or not. as more of these discrepancies appeared, we discovered the difference in interpretation of the protocol. as this posed a significant, critical point in the applicability of our study, we needed to revisit the protocol and clarify our methods. the faculty members on the team discussed the specifics and then discussed it with the students in the next meeting. the number of discrepancies decreased after such meetings. changes made in the protocol were listed in the review as deviations from said protocol. this experience taught us the importance of precision in our language and our directions to the students, as well as the importance of flexibility in our thinking. shifting the criterion to better match our goal allowed many more studies to be included that would otherwise be excluded if we followed the letter of our protocol. this is important because we were able to garner additional information regarding our research question, increasing the power and strength of our study. it also demonstrated to the students that having this flexibility and documenting it is part of the research process. discussion this project was the team’s first major systematic review, and the librarian requested to be part of the entire process to gain a better footing and grounding in all steps involved in an es project. the team recognizes that it was a unique situation (being the first review and the pandemic), and it is unlikely that the members would do the same for following es projects. librarians do not commonly participate this heavily in a review since they are es methods experts and rarely have a background in the subject matter. this project enriched our understanding of librarian-medical student collaboration and highlighted the challenges in both the literature and multi-disciplinary collaborations. benefits of librarian-medical student collaboration taking the lead with the students on the protocol allowed the librarian to offer in-depth mentoring and training of the jbi methodology. it allowed for different teaching and training modalities with varying approaches of students, expertise, and skill levels. an unexpected benefit of having this time with a small group of students was that it allowed the librarian to have a greater understanding of medical students’ time constraints, the ways students use databases, think and frame questions, and their mindset with resources and research. it helped the librarian be a better liaison by getting to know the users of the resources more intimately. having less time now for hands-on small group training, our institution has subscribed to software (covidence) that assists with this and for the librarian to work with colleagues to develop tailored materials for different steps of the process to allow for a less hands-on approach. the librarian found in this project that working with students helped build their confidence and reflect on their understanding, which aided not only in the search but the writing of the protocol, and mentoring and training newly added students to the project. the students on this project were able to be more open and vulnerable with the librarian, than with other faculty. 8 peer reviewed article hypothesis, vol. 37, no. 2, 2025 collaborating does take more time in some respects but ultimately benefited our team since all involved had a strong understanding of the question and the content being sought. in contrast to our experience with this project, our librarians rely less on students and subject matter experts in developing searches. in their current projects, they usually meet with teams to discuss mesh definitions and review entry terms as well as look over initial results and how to refine or revise the search to improve either sensitivity or precision. other team members are often familiar with acronyms and synonyms unfamiliar to librarians and can often expand searches and assist with the effectiveness of the search. our institution has now implemented a tiered service model for es projects with two tiers of consultant or author. the consultant role would press the search developed by the team and provide feedback. the librarian in the author role would develop the search strategy and have another librarian press it. working on es projects with medical and health science students allows for in-depth collaboration and understanding of each other’s roles and expertise. it benefits both the librarian and the student. the librarian can offer encouragement, help connect with faculty, listen and support by bringing questions to the group. participating in es projects boosts confidence in interacting professionally and strengthens subject knowledge and skills in writing and working collaboratively. for our team, it allowed for equalizing roles and faculty dictating tasks, but also identifying ways for students to participate and motivated them to seek out ways to contribute, taking into account their capacity and time limits. challenges encountered in the literature one approach we used in this project was to modify a validated search. some experts note that by modifying a validated search, it is no longer valid. however, search building for an es project is an iterative and involved process. modifying a validated search increased the efficiency of drafting and formalizing the searches for this project. using a validated search as a starting point can be a valuable step in the search strategy development process. every es project’s search improves as we learn more about the methods and our skills increase. another challenge in our systematic review of diagnostic accuracy was the lack of diagnostic data in many of the published studies. we expected all diagnostic accuracy studies to adhere to the stard (standards for reporting of diagnostic accuray studies) guidelines17 published in 2015 and set international standards for publishing and reporting on diagnostic accuracy studies. many of the studies did not meet these standards, affecting the data extraction stage of the project. the data analyst on the project, being the second data extractor, noticed that while the medical and demographic data were extracted with accuracy, the numbers related to sensitivity and specificity were more difficult. the lack of adherence to stard guidelines17 made it hard to extract the underlying values for sensitivity and specificity. it required additional training for the students on how to obtain those values from the text of the paper as opposed to a table. challenges faced during collaboration as anyone who has worked with pre-clinical medical students can attest, they are quite busy 9 peer reviewed article hypothesis, vol. 37, no. 2, 2025 and have difficulty focusing on extracurricular activities such as research. to combat this, we utilized a large team of students so that the work was distributed across multiple students, reducing the amount of work any individual student needed to complete. our review took several years to complete (ideation in june 2020 to manuscript submission in july 2023), which meant that some of our students who joined the project early in the pandemic were graduating and moving on to their residencies before the review was completed. due to the length of the project, the search was run in 2021 as well as in 2022. additionally, we brought in new students who needed to be trained and mentored as the project continued. mentoring and training students is not an easy task, especially when you throw in the varying levels of experience the students bring into the project. it sometimes required individual meetings as well as additional videos for the students to be able to complete the project and their research experience successfully. there is potential for these types of projects to put an undue burden on librarians to mentor and train students in the es methodology. again, the authors emphasize the need to define roles and responsibilities at the start of the project and have a clearly defined es service model. there are students and faculty who are used to seeing librarians in a service role rather than as co-researchers and a full team member since their expertise and skill set are different from other faculty. the data scientist and librarian share this specialized research role on the team and sometimes need to discuss the intellectual contribution with other faculty and students, reminding them that it warrants rights and privileges on the project, such as authorship. takeaways this project has reinforced the importance of fostering successful partnerships. to accomplish this, we have the following recommendations. 1. clear roles and responsibilities in our project, we defined roles and responsibilities based on tasks to be completed. the librarians brought expertise in information science, search design, protocol development, and es methodology. trainees bought their clinical insights and were responsible for screening and data collection. the faculty held leadership and mentoring responsibilities along with subject matter expertise. the data scientist led the statistical methods of the es project. expectations for authorship were defined and expressed to individuals who joined the project. moving forward, students would have a role in developing the research question. one potential method that could be used is to have the students write 2-3 research questions on a topic based on the acronym frameworks (pico, pird, etc.). then each student would look through various databases to determine what work was done on their topics and refine their questions. finally, the team would collaboratively select a research question to pursue as a systematic review. having an interdisciplinary team for es projects, including a librarian, a subject matter expert, and a data scientist, is highly recommended. each team member will bring a fresh outlook to the review question, inclusion-exclusion criteria as well as conclusions about the meta-analysis. the students on the team will also obtain experience with multiple disciplines, thus enhancing their research experience. each discipline has 10 peer reviewed article hypothesis, vol. 37, no. 2, 2025 overlapping and unique roles within an es project, which need to be clearly defined. for many es projects, it would be desirable to have someone fulfill the role of the data analyst. the person fulfilling the data analyst role would ideally have working knowledge of statistical models used within the subject matter. for example, in our project, decisions needed to be made about how to address the heterogeneity of the data and the effect of outliers which was outside the expertise of the other team members. again, clarity of expectations from the individual fulfilling this role needs to be established early on. 2. ongoing communication and mutual respect for any project with an interdisciplinary team, it is important to have healthy and ongoing communication between the team members to successfully complete the project. for our team, the weekly meetings provided a forum to discuss the project. also, follow-up emails were used to summarize the meeting and ensure communication with the team members who missed the meeting. the team was actively encouraged to communicate in a way that suited them through inclusive language and various communication platforms. decisions need to be made at each stage of the review and cannot be made arbitrarily by one person otherwise, that could lead to bias. disagreements that occurred during screening involved discussions with different team members (trainees and faculty), where each team member was invited to share their opinion. errors were acknowledged and addressed irrespective of who made them. the selection of the team is critical to project success. the personalities of individuals and the team can radically impact the project’s success. expertise can be gained, but attitudes are harder to shift. it is vital to recognize each person’s contribution to the project and respect the other’s expertise and experience. the culture of mutual respect was established in the first meeting based on the faculty’s interactions with each other and the students. the team also acknowledged and celebrated each member’s worth as an individual, including achievements outside the project. conclusions our team encourages librarians who are interested in collaborative teams to be clear about their role and the team’s expectations. it can be helpful to have a service model outlined so that it can be referenced. overall, we had a good experience in completing our systematic review on the diagnostic accuracy of rapid antigen tests for covid-19. our success lies in our flexibility, clear communication, project management strategies, collaborative working spaces (live sessions and asynchronous), and commitment to high-quality, rigorous analysis and work (see figure 1). 11 peer reviewed article hypothesis, vol. 37, no. 2, 2025 figure 1. pillars of successful evidence synthesis projects. selection of a good team, regular communication, clear expectations of each team member, particularly the role of the librarian as a co-investigator, timelines and milestones, and collaborative software. credit megan armond: conceptualization, writing original draft, writing review & editing sherli koshy-chenthittayil: conceptualization, writing original draft, writing review & editing amy el stone: conceptualization, writing original draft, writing review & editing references 1. hoffmann f, allers k, rombey t, helbach j, hoffmann a, mathes t, et al. nearly 80 systematic reviews were published each day: observational study on trends in epidemiology and reporting over the years 2000-2019. j clin epidemiol. 2021;138:1-11. 2. hirabayashi e, mercado g, hull b, soin s, koshy-chenthittayil s, raman s, et al. comparison of diagnostic accuracy of rapid antigen tests for covid19 compared to the viral genetic test in adults: a systematic review and meta-analysis. jbi evid synth. 2024;22(10):1939-2002. 3. dearmond m, feliciano s, hirabayashi e, duus km, else ta, stone ael. diagnostic accuracy of rapid antigen tests for covid-19 compared to the viral genetic test in adults: a systematic review protocol. jbi evid synth. 2021;19(5):1148-56. 4. joanna briggs institute. jbi manual for evidence synthesis: jbi; 2024 2024. 5. mann m, editor the role of the librarian in the systematic review process: past, present, and future. qscience proceedings-the evolving health information landscape symposium 2021; 2022; qatar: qscience. 6. spencer aj, eldredge jd. roles for librarians in systematic reviews: a scoping review. j med libr assoc. 2018;106(1):46-56. 12 peer reviewed article hypothesis, vol. 37, no. 2, 2025 7. metzendorf mi, featherstone rm. ensuring quality as the basis of evidence synthesis: leveraging information specialists’ knowledge, skills, and expertise. cochrane database of systematic reviews. 2018(9). 8. clapp jt, gordon ek. selecting trainees: too much focus on predictive metrics, not enough on holistic review. med educ. 2022;56(2):139-41. 9. ferreira t. escalating competition in nhs: implications for healthcare quality and workforce sustainability. postgrad med j. 2024;100(1184):361-5. 10. savoca m, kernan k, scott c, bugallo m. early exposure to stem research as a foundational experience for stem careers. experiential learning and teaching in higher education. 2023;6(1):46-55. 11. murdoch-eaton d, drewery s, elton s, emmerson c, marshall m, smith ja, et al. what do medical students understand by research and research skills? identifying research opportunities within undergraduate projects. med teach. 2010;32(3):e152-60. 12. khulwa ca, luthfia a, editors. generation z students’ digital literacy on online learning readiness. 2023 11th international conference on information and education technology (iciet); 2023 18-20 march 2023. 13. zainab hazim i, ban hassan m, lina fouad j. computer literacy with skills of seeking for information electronically among university students. international journal of interactive mobile technologies (ijim). 2023;17(07):pp. 47-57. 14. mcgowan j, sampson m, salzwedel dm, cogo e, foerster v, lefebvre c. press peer review of electronic search strategies: 2015 guideline statement. j clin epidemiol. 2016;75:40-6. 15. google. google drive. drive.google.com: google; n.d. 16. google. google sheets. sheets.google.com: google; n.d. 17. bossuyt pm, reitsma jb, bruns de, gatsonis ca, glasziou pp, irwig l, et al. stard 2015: an updated list of essential items for reporting diagnostic accuracy studies. bmj. 2015;351:h5527. 13 drive.google.com sheets.google.com hypothesis, vol. 37, no. 2, 2025 introducing the new and emerging researchers submission category lorie a. kloda, phda avice-dean of libraries, mcgill university, montreal, qc, canada, https://orcid.org/0000-0003-1429-1497 , lorie.kloda@mcgill.ca cite as: kloda la. introducing the new and emerging researchers submission category. hypothesis. 2025;37(2). doi:10.18060/29393 kloda. all works in hypothesis are licensed under a cc by-nc 4.0 deed attribution-noncommercial 4.0 international. authors own copyright of their articles appearing in hypothesis. readers may copy articles without permission of the copyright owner(s), as long as the author(s) are acknowledged in the copy, and the copy is used for educational, not-for-profit purposes. for any other use of articles, please contact the copyright owner(s). 1 https://orcid.org/0000-0003-1429-1497 https://orcid.org/ mailto:lorie.kloda@mcgill.ca https://creativecommons.org/licenses/by-nc/4.0/ https://creativecommons.org/licenses/by-nc/4.0/ hypothesis, vol. 37, no. 2, 2025 fresh insights from new researchers i am delighted to introduce hypothesis readers to the nascent new and emerging researchers category. this new category reflects the importance to the profession of encouraging and supporting librarians to engage in research and to submit their work for publication. going forward, hypothesis welcomes submissions from new and emerging researchers with the intent to promote greater visibility and recognition to those at the outset of their research careers. there is never a better time to do research in librarianship than now. by this i don’t mean that this point in history is better than any other, only that it is no less good. in my own experience, i have sometimes been under the impression that all the important questions have been asked and answered. or that these important questions were best answered by someone else, some more qualified expert in method x or y or with population z. it can be daunting to come up with a focused question that feels new or original enough to pursue, let alone to design a study to answer such a question. in our field, librarians spend a significant amount of our work day encouraging students, clinicians, and researchers to search the literature for previous findings to inform not only their decisions but also their next projects. it’s only natural that some would conclude that everything that’s worth studying has already been studied, and that research should be left to those with more experience or more training. it’s all too easy to talk ourselves out of undertaking a new research project in these circumstances. i am therefore all the more impressed at the number of health sciences librarians who continue to ask new research questions, design studies to investigate them, and report on their findings and experiences. i am delighted that hypothesis is showcasing emerging researchers, who along with their day to day professional work, carved out the time to undertake research in lis and to disseminate it. in this issue in this issue of hypothesis are three papers authored by new emerging researchers in library and information studies. mary margaret thomas from university of iowa reports on original research using a qualitative approach that she developed as a participant at the mla research training institute. the pilot study contributes to what is known on the impact of mental health first aid training for librarians in academic health sciences and public libraries. christine neilson from university of manitoba conducted a content analysis on librarians’ relationship to knowledge synthesis work. her findings provide valuable insight to a perspective typically omitted from publications about knowledge synthesis work and complementary to reports on health researchers’ needs and behaviours with respect to this popular set of research methods. devon olson and sara b. westall present a report on their work at the university of north dakota of a needs assessment of users’ awareness about open access during oa week by way of a quiz competition. the paper provides an excellent description of an awareness campaign for open access as well as an assessment tool for evaluating its success and for identifying areas of further instruction. i hope these articles not only inform your thinking about health sciences librarianship, but 2 hypothesis, vol. 37, no. 2, 2025 inspire you in your own investigations, whether as a novice or a more experienced researcher, to ask questions and apply new methods to continue to inform the field. 3 peer reviewed article hypothesis, vol. 37, no. 2, 2025 publication pandemonium: an asynchronous online quiz competition for open access week devon olson, mlis, ahipa, sara b. westall, mlis, msb aresearch and education librarian, library resources, university of north dakota school of medicine and health sciences, grand forks, north dakota, https://www.orcid.org/0000-0002-1682-0199 , devon.olson.2@und.edu bne/se clinical campus librarian, university of north dakota school of medicine and health sciences, library resources, fargo, north dakota, https://www.orcid.org/0009-0001-1356-5823 , sara.westall@und.edu cite as: olson d & westall sb. publication pandemonium: an asynchronous online quiz competition for open access week. hypothesis. 2025;37(2). doi:10.18060/29028 olson, westall. all works in hypothesis are licensed under a cc by-nc 4.0 deed attribution-noncommercial 4.0 international. authors own copyright of their articles appearing in hypothesis. readers may copy articles without permission of the copyright owner(s), as long as the author(s) are acknowledged in the copy, and the copy is used for educational, not-for-profit purposes. for any other use of articles, please contact the copyright owner(s). 1 https://www.orcid.org/0000-0002-1682-0199 https://orcid.org/ mailto:devon.olson.2@und.edu https://www.orcid.org/0009-0001-1356-5823 https://orcid.org/ mailto:sara.westall@und.edu https://creativecommons.org/licenses/by-nc/4.0/ https://creativecommons.org/licenses/by-nc/4.0/ peer reviewed article hypothesis, vol. 37, no. 2, 2025 abstract open access week is an international event focused on issues of equity within the scholarly publishing system. at the university of north dakota library, we see a diverse range of attitudes towards oa. to build oa support and combat misinformation, we designed an oa week quiz competition, “publication pandemonium”, with an asynchronous format designed to include our clinical sites. this event was an overwhelming success despite a short turnaround, with impressive turnout never before seen in our library programming. introduction open access week is an international event focused on issues of equity within the scholarly publishing system1-4. at the university of north dakota library, we see a diverse range of attitudes towards open access (oa). to build oa support and combat misinformation, we designed an oa week quiz competition called “publication pandemonium” in an asynchronous format designed to include our clinical sites. this event was an overwhelming success despite being quickly launched, with impressive turnout never before seen in our library programming. experience our multiple choice, 14 question, online quiz was built using springshare’s libwizard survey tool. questions in the quiz were designed to vary widely in difficulty, ranging in scope from basic oa terminology, including data management, to current events knowledge5-10. the quiz was open to faculty, staff and students all oa week. we also created a companion libguide and oa week zine, covering all concepts and definitions required to correctly complete the quiz5-13 (see appendix 1). all companion resources were linked together. for example, qr codes on the zine linked to the quiz and libguide. 2 peer reviewed article hypothesis, vol. 37, no. 2, 2025 figure 1. cover of the oa week zine, featuring concepts covered in the quiz note: for full oa week publication pandemonium zine pdf, see university of south dakota scholarly commons repository we partnered with our associate dean of research and senior associate dean of education, medical, and faculty affairs to strategize content, outreach, and funding for prizes, as well as our graphic design department to create visual marketing materials. announcements were sent via school emails lists and newsletter, and on digital signage in the weeks preceding, during, and after the quiz was launched. advertising for the quiz contained contextual information about oa, graphics of prizes, timeline information, and hyperlinked or qr codes to the quiz. our advertising strategy was designed to reach all of our user groups. discussion our quiz had an overwhelming and diverse response rate of 54. 3 https://commons.und.edu/smhs-lp/11/ peer reviewed article hypothesis, vol. 37, no. 2, 2025 table 1: quiz participant demographics academic program or department/office average score total participants # of passing scores (<70%) program enrollment spring 2025 % of enrolled students participating with passing scores center for rural health 78.57% 1 1 n/a n/a medicine 75.65% 22 18 276 7% indigenous health 69.05% 3 2 54 4% occupational therapy 66.43% 10 6 206 3% medical laboratory science 60.71% 2 1 269 0% physical therapy 57.14% 8 4 318 1% alumni affairs office 50% 1 0 n/a n/a biomedical sciences 49.29% 3 2 37 5.41% public health 45.24% 3 0 51 0% education resources office 42.86% 1 0 n/a n/a note: academic programs are in bold font, while departments and offices are not in bold font. table 2: quiz-related resource usage metric count: month of august 2024 count: month of september 2024 count: month of october 2024 count: month of november 2024 count: month of december 2024 libguides views *the quiz was linked on this page 20 13 325 39 47 zine downloads *posted to the institutional repository in october 0 0 12 6 6 4 peer reviewed article hypothesis, vol. 37, no. 2, 2025 multiple methods were used to determine winners and increase participation and fairness of the quiz. we had a high score bracket as well as a participation bracket, which awarded prizes to the program with the highest percentage of enrolled student participants. figure 2: quiz winners’ brackets 5 peer reviewed article hypothesis, vol. 37, no. 2, 2025 table 3: correct and incorrect quiz response rate by question, most to least correct question incorrect (%) correct (%) 11 what are the four fair data management principles? 7.4 92.6 12 university of south dakota scholarly commons is a: 11.1 88.9 3 is "creating and implementing an indigenous evaluation framework" an open access article? 13.0 87.0 4 is the journal of maternal health, neonatology and perinatology an open access journal? 13.0 87.0 13 "double-dipping" in the publishing world refers to: 18.5 81.5 14 which of the following describes “platinum journals”? 24.1 76.0 2 which of the following is not a real creative commons license? 27.8 72.2 7 which journal publishers are currently implicated in a federal antitrust lawsuit? 27.8 72.2 10 which of the following is true about the nelson memo? 35.2 64.8 8 how much revenue in u.s. dollars did elsevier make in 2023? 46.3 53.7 9 which company had a higher profit margin in 2023 than elsevier? 48.1 51.9 5 which of the following is an open access database? 63.0 37.0 6 which of the following is a real collection of works on university of south dakota scholarly commons? 63.0 37.0 1 which of the following is not an aspect of open access publication? 66.7 33.3 note: for full questions and answers with rates of correct and incorrect answers, see appendix 1. takeaways the design of our event and quiz did not allow us to track whether responses were guesswork or the result of investigation and learning. however, the response data suggests a lack of oa conceptual knowledge and navigation skills. we will continue our response data analysis to help strategize instruction and outreach, and are planning follow-up workshops. it is possible email issues restricted our engagement. email lists initially excluded non-medical students and medical program residents have multiple emails to manage, which may explain their lack of participation despite a publication funding prize. we received little anecdotal feedback and are wary of drawing too many causal conclusions from a single event. however, in future, we hope to do more targeted messaging and data collection throughout, using focus groups to test question difficulty and relevance. we look forward to making this an annual event, increasing awareness and participation, creating additional tie-ins with programming such as workshops, and fun companion artefacts like zines and stickers. 6 peer reviewed article hypothesis, vol. 37, no. 2, 2025 credit statement devon olson: conceptualization, methodology, validation, resources, data curation, writing – original draft, writing – review & editing, visualization, project administration sara westall: conceptualization, methodology, resources, writing – original draft, writing – review & editing, visualization, funding acquisition references 1. butler la, matthias l, simard ma, mongeon p, haustein s. the oligopoly’s shift to open access: how the big five academic publishers profit from article processing charges. quantitative science studies [internet]. 2023 nov 1;4(4):778–99. doi:10.1162/qss_a_00272 2. fyfe a, coate k, curry s, lawson s, moxham n, røstvik cm. untangling academic publishing: a history of the relationship between commercial interests, academic prestige and the circulation of research. zenodo [internet]; 2017 may. doi:10.5281/zenodo.546100 3. langham-putrow a, bakker c, riegelman a. is the open access citation advantage real? a systematic review of the citation of open access and subscription-based articles. plos one [internet]. 2021 jun 23;16(6):e0253129. doi:10.1371/journal.pone.0253129 4. larivière v, haustein s, mongeon p. the oligopoly of academic publishers in the digital era. plos one [internet]. 2015 jun 10;10(6):e0127502. doi:10.1371/journal.pone.0127502 5. nelson, a. ensuring free, immediate, and equitable access to federally funded research. office of science and technology policy [internet]. 2022. [accessed 2025 jun 9]. https://www.whitehouse.gov/wp-content/uploads/2022/08/08-2022-ostp-public-accessmemo.pdf 6. singh chawla d. academic publishers face antitrust lawsuit. chemical & engineering news [internet]. 2024 sep 18 [accessed 2025 jun 9]. https://cen.acs.org/policy/publishing/academic-publishers-face-antitrust-lawsuit/102/i29 7. fair principles. [accessed 2024 october 20]. available from: https://www.go-fair.org/fair-principles/ 8. piwowar, h., priem, j., larivière, v., alperin, j. p., matthias, l., norlander, b., farley, a., west, j., & haustein, s. the state of oa: a large-scale analysis of the prevalence and impact of open access articles. peerj [internet]. 2018 feb;6:e4375. doi:10.7717/peerj.4375 9. smith, a. c., merz, l., borden, j. b., gulick, c. k., kshirsagar, a. r., & bruna, e. m. assessing the effect of article processing charges on the geographic diversity of authors using elsevier’s “mirror journal” system. quantitative science studies [internet]. 2021 dec;2(4):1123–1143. doi:10.1162/qss_a_00157 10. u.s. copyright office. more information on fair use; [accessed 2025 june 9]. available from: https://www.copyright.gov/fair-use/more-info.html 7 https://dx.doi.org/10.1162/qss_a_00272 https://doi.org/10.5281/zenodo.546100 https://dx.doi.org/10.1371/journal.pone.0253129 https://doi.org/10.1371/journal.pone.0127502 https://www.whitehouse.gov/wp-content/uploads/2022/08/08-2022-ostp-public-access-memo.pdf https://www.whitehouse.gov/wp-content/uploads/2022/08/08-2022-ostp-public-access-memo.pdf https://cen.acs.org/policy/publishing/academic-publishers-face-antitrust-lawsuit/102/i29 https://www.go-fair.org/fair-principles/ https://dx.doi.org/10.7717/peerj.4375 https://dx.doi.org/10.1162/qss_a_00157 https://www.copyright.gov/fair-use/more-info.html peer reviewed article hypothesis, vol. 37, no. 2, 2025 11. us national institutes of health. nih policy for data management and sharing. not-od-21-013. 2020 [accessed 2025 june 9]. available from: https://grants.nih.gov/grants/guide/notice-files/not-od-21-013.html 12. van noorden, r. open access: the true cost of science publishing. nature, 2013 march;495(7442):426–429. doi:10.1038/495426a 13. wall kimmerer, r. (2022). serviceberry: an economy of abundance. emergence magazine. october 26 2022. [accessed 2025 june 9]. https://emergencemagazine.org/essay/the-serviceberry/ 8 https://grants.nih.gov/grants/guide/notice-files/not-od-21-013.html https://doi.org/10.1038/495426a https://emergencemagazine.org/essay/the-serviceberry/ peer reviewed article hypothesis , vol. 35, no. 2, 2023 preparing for the new nih data management and sharing policy: outreach & engagement sara m. samuel, msia ainformationist, taubman health sciences library, university of michigan, ann arbor, michigan, https://orcid.org/0000-0001-6808-2514 , henrysm@umich.edu the national institutes of health (nih) released their new data management and sharing policy in october 2020 with the effective date of january 25, 2023. the taubman health sciences library (thsl) at the university of michigan established the research data core team (data team) to raise awareness and educate people about the policy. the data team, formed in july 2022, consists of three informationists who have experience with data management and sharing. the data team proceeded with an initial outreach phase followed by an engagement phase. description outreach was launched by updating a libguide with information about the new policy,1 creating a slide for digital screens, and producing a short video.2 an announcement was distributed to various newsletters on campus which referenced the libguide and video. included in the announcement was an offer to present to groups that would like to learn more. through this announcement and existing collaborations with the main library’s deep blue repositories and research data services, office of the vice president for research, and medical school’s office of regulatory affairs, the data team initiated the engagement phase. engagement was accomplished by providing online presentations, workshops, and office hours with a variety of groups on campus. for the engagement phase, a presentation was created to review the policy’s main points, emphasizing that a data management and sharing (dms) plan needed to be included in grant applications. the elements of a dms plan and answers to frequently asked questions were reviewed. the presentation incorporated new guidance from nih as it was released. the presentation was modified as needed to adjust the timing, or if there was a special interest in learning more about data repositories or how the policy applied to qualitative data. outcomes throughout the fall 2022 and winter 2023 semesters, the data team facilitated eight presentations, two workshops (one in collaboration with colleagues from the institutional data repository and one specifically for research administrators), and two office hours (in collaboration with medical school units). over 600 attendees came to either a presentation, workshop, or office hours event. many of the presentations were recorded and distributed, resulting in additional learners. attendees included faculty researchers and staff, research and project administrators, post-docs, and 1 https://www.orcid.org/0000-0001-6808-2514 https://orcid.org/ mailto:henrysm@umich.edu peer reviewed article hypothesis , vol. 35, no. 2, 2023 undergraduate and graduate students. feedback collected after one workshop indicated that it was helpful. e.g.: “i had attended another session a month ago, but this one answered my questions fully.” average libguide views increased to 997 per month after promotion began in august, up from 352 per month during the year prior. views peaked at 1,873 in january. four requests to evaluate dms plans were received directly after offering the service during a presentation. the data team created an evaluation template to effectively provide feedback on dms plans. the author spent approximately 30 hours in data team meetings, updating the libguide, presenting, and evaluating dms plans. additional time was spent preparing presentations which decreased over time as policy information stabilized. the data team anticipates continuing outreach and engagement after seeing how the nih responds to the first dms plans. acknowledgements special thanks to my colleagues on the data team, kate saylor and mark maceachern, for all their work in preparing for the new nih policy and for reviewing the manuscript. additional thanks to tyler nix for reviewing the manuscript and providing feedback. references 1. samuel, s. research guides: research data management (health sciences): nih data management & sharing plan [internet]. 2023 [cited 2023 jun 8]. available from: https://guides.lib.umich.edu/datamanagement/nihpolicy 2. quick overview: the new nih data management and sharing policy [internet]. 2022 [cited 2023 jun 8]. available from: https://lib.mivideo.it.umich.edu/media/t/1 q4i3f8mw 2 https://guides.lib.umich.edu/datamanagement/nihpolicy https://lib.mivideo.it.umich.edu/media/t/1_q4i3f8mw peer reviewed article hypothesis , vol. 34, no. 1, 2022 methods moment: the delphi method tenley sablatzky, mlisa amedical librarian, undergraduate medical academy, prairie view a&m university, prairie view, texas, https://orcid.org/0000-0001-5656-3206 , tnsablatzky@pvamu.edu, twitter method name(s): the delphi method description: the delphi method is a process used to arrive at a group opinion or decision by surveying a panel of experts. the method was developed in the 1950’s and 1960’s by the rand corporation and since then has been used in many fields, including library and information science. during a delphi study, selected experts respond to several rounds of questionnaires, and the responses are aggregated and shared with the group after each round. a delphi study relies on the idea that collective group responses are superior to individual responses. purposes: originally the delphi method was created as a systematic, interactive forecasting method relying on a panel of experts. since then, delphi studies have been used for many purposes: creating policy, establishing guidelines, and identifying trends. overview the delphi method is a survey technique used to gain a consensus of a panel of experts in the field through several rounds of questioning. the method was developed during the 1950s and 1960s by the rand corporation to prepare for national security threats during the cold war. since then, it has been used as a research method in a variety of fields, including marketing, advertising, education, and medical science. other fields, including library and information science, began incorporating the method into their research during the 1970s1. a typical delphi study consists of multiple rounds of written questionnaires sent to identified experts. after each round, the facilitator collects and reviews the data and distributes a summary report to each expert who then reviews the report and either agrees or disagrees with the other experts’ answers. this cycle continues until a consensus is reached. the justification for performing a delphi study lies in the theory that collective group responses are superior to individual responses. the theory suggests that even the most qualified, well-researched individual may have ideas that are not “the best”, but when considered as a collective, the best ideas tend to elevate to the top. niederberger and spranger’s 2020 article2 explains that in some fields, such as technical and natural sciences, the delphi method is used to analyze future developments; other fields, like the health sciences, use it for reaching a consensus. in library and information science, delphi studies have proven to be a popular research method; library science researchers have used delphi studies to answer questions, including what are the best practices for collection development or information literacy instruction3. for example, in 2009, a delphi study about the medical library association’s (mla) research agenda was published in the journal of the medical library association4. delphi studies are valid techniques even though they are considered a lower level of evidence than meta-analyses or correlation studies. because of this, delphi studies are less suited for research questions about intervention effectiveness and more suited for questions about practices or trends. often, the results of a delphi study are used to draft 1 https://orcid.org/0000-0001-5656-3206 https://orcid.org/ mailto:tnsablatzky@pvamu.edu https://twitter.com/tenleynadine peer reviewed article hypothesis , vol. 34, no. 1, 2022 figure 1: the step-by-step process of conducting a typical delphi study 2 peer reviewed article hypothesis , vol. 34, no. 1, 2022 white papers, techniques, or guidelines. they can also be used as a preface to another research method. carefully selecting a team of knowledgeable experts who are well-versed in at least one aspect of an issue is the most critical step of a successful delphi study. the team of experts does not need to be representative of every population, but the selected qualified participants should be knowledgeable of the issues and viewpoints related to the topic being studied. the strengths, weaknesses, and biases of the expert panel will determine the strengths, weaknesses and biases of the findings. okoli and pawlowski’s 2004 article designates the selection of experts as the most critical step of a delphi study. for their study, they carefully determined the research questions they were trying to answer and then designated four distinct groups of experts that would be beneficial to include in their study design: academics, practitioners, government officials, and officials at non-profit organizations5. the types of experts required varies depending on the research questions, but the initial determination of who will be part of the panel is critical to creating a study with minimal bias and viewpoints representative of the whole community. as the delphi method is a survey technique, bias is a common limitation that researchers need to be aware of and carefully consider. melander’s 2018 article explains that desirability bias is a major limitation of delphi studies. the article notes that desirability bias, or social-desirability bias, means survey respondents tend to answer questions in a manner that will be viewed favorably by others or lean towards a favored projected outcome6. a straightforward approach to minimizing desirability bias is simply to include survey questions asking the experts to rate the desirability of their projections. by doing so, the researcher is able to reflect on and account for desirability bias in the study as much as possible. the most common way of minimizing bias in the broader consensus is to ensure that experts are diverse and that they have been chosen based on their expertise and in reference to the research question(s). the best way to select a diverse group of qualified experts is for the researcher to take time at the start of their study to create a balanced knowledge resource nomination worksheet (krnw). a krnw is a helpful tool for identifying the necessary experts for a study (see figure 27). the first step, as defined by okoli and pawlowski, is to create a list of the necessary disciplines and skills, relevant organizations, and relevant literature needed for the research study5. the initial list is populated with the names of specific individuals at relevant organizations, or authors who have published relevant work on the subject matter. in the next step, the researcher should begin contacting individuals who have been identified. in addition to inviting these individuals to be part of the panel, the researcher may ask for the recommendation of additional experts. once a list of experts willing to serve on the panel has been created, the experts are then categorized based on their qualifications and expertise. the target size for an expert panel will vary depending on the expertise required, though sekayi and kennedy, in their 2017 article, suggest that twenty to thirty panelists is ideal for a delphi study. if more than thirty experts are used, then the process may become too unwieldy and repetitive to be of any real value8. the average delphi study consists of two or three rounds of surveys and typically takes four to six months to complete. 3 peer reviewed article hypothesis , vol. 34, no. 1, 2022 figure 2: the process of creating a knowledge resource nomination worksheet. adapted from okoli and pawlowski.7 the major strength of performing a delphi study, as noted in lund’s 2020 article, is that no one subject can dominate the conversation, as can sometimes happen in a focus group3. responses to delphi surveys that are less popular to the group of experts will be eliminated during the process, as opposed to a traditional survey where all responses are given equal weight. the major weakness, however, arises when the researcher conducting a delphi study does not want to eliminate responses that are less popular. lund explains that the researcher may consider these less popular responses to be important pieces of data because, while they are not common, still represent some of the total responses3. however, in a delphi study it is not ethical to use researcher’s personal preferences for a less popular piece of data to sway the trajectory of the study. the strength of a delphi study is the focus on consensus agreements among available experts. if a researcher needs to consider all viewpoints equally to adequately answer their research questions, a delphi study is not the ideal study design for the project. delphi studies are best used in situations where the collective group responses are superior to individual responses. 4 peer reviewed article hypothesis , vol. 34, no. 1, 2022 example rarian, an instructional librarian, needs to develop assessment guidelines to determine the success of their first-year information literacy class. being new to the job, they are unsure of the best way to measure the students’ learning. after doing some online research, rarian discovers that there are different instruction assessment methods used by librarians but no clear answer exists about which one is the most effective. rarian speaks to a colleague for advice on the best course of action. their colleague agrees that the research does not show a consensus on the best practices for assessment, and suggests that performing a delphi study could be a helpful starting place. being unfamiliar with the term, rarian asks for clarification, and their colleague proceeds to explain the basic mechanics of delphi study. rarian agrees that receiving input from a diverse panel of experts would be beneficial for determining the most common methods of assessment and reaching a consensus on the most effective methods. rarian and their coworker decide upon the following research question: what methods of assessment are most effective for information literacy instruction sessions? knowing that the selection of experts will be critical to the success of the study, rarian takes the time to prepare a knowledge resource nomination worksheet (krnw) to identify qualified experts. the first step is to determine how to categorize the backgrounds and subject areas required before selecting participants. rarian creates a chart listing the skills and disciplines necessary, and also notes libraries and educational organizations with potential experts. with the list in hand, they begin searching for and contacting individuals with the necessary skills and populating the krnw with names. rarian is also sure to ask the recruited experts to nominate additional experts who could be an asset to the project. once rarian has twenty-five confirmed experts to participate in the study, they begin drafting the first survey. for the first round of questions, rarian focuses on asking general questions to gain a broad understanding of the types of assessments the experts used and which they found most effective. as the surveys progress, the questions become more specific, based on the past responses. for example, after noting in the first round that a significant number of experts mentioned course surveys as an assessment method, they were sure to include questions in round two regarding the types of questions, survey length, and distribution method of the survey. rarian uses the strongest answers from the second round to create a summary report for the experts and a new questionnaire. the experts then rate and comment on the third set of assessment methods with guidance from the information in the report of the previous round. after the third survey, rarian has narrowed the questions enough to receive a consensus on the most effective form of assessment for library instruction. next, rarian writes up and further analyzes their findings by testing the selected assessment method in the classroom. after testing the results, they may find that further study or an additional research method is necessary. in some cases, this can lead to a second delphi study with a different panel of experts; in other cases, a different research method will be required to expand the research. resources delphi technique a step-by-step guide [internet]. project smart. project smart; 2021 [cited 2022 jul 11]. available from: https://www.projectsmart.co.uk/tools/delphi-technique-a-step-by-step-guide.php what is delphi method? what does delphi method mean? delphi method meaning, definition & explanation [internet]. youtube. audiopedia; 2017 [cited 2022 jul 11]. available from: https://www.youtube.com/watch?v=0ap77w8fkym 5 https://www.projectsmart.co.uk/tools/delphi-technique-a-step-by-step-guide.php https://www.youtube.com/watch?v=0ap77w8fkym peer reviewed article hypothesis , vol. 34, no. 1, 2022 delphi method [internet]. rand corporation. the rand corporation; [cited 2022 jul 11]. available from: https://www.rand.org/topics/delphi-method.html further reading: fletcher, a. j., marchildon, g. p. using the delphi method for qualitative, participatory action research in health leadership. international journal of qualitative methods. 2014 feb 1;13(1):1–18. hasson, f. research guidelines for the delphi survey technique. journal of advanced nursing. 2008 june;32(4):1008-1015. mozuni, m., jonas, w. an introduction to the morphological delphi method for design: a tool for future-oriented design research. she ji: the journal of design, economics, and innovation. 2017 dec 2017;3(4):303–318. rabina, d. reference materials in lis instruction: a delphi study. journal of education for library and information science. 2013 apr 1;108-23. townsend, l., hofer, a., lin hanick, s., brunetti, k., university of new mexico, university libraries. (2016). identifying threshold concepts for information literacy: a delphi study. communications in information literacy. 2016 june 6;10(1):1. references 1. ju b, jin t. incorporating nonparametric statistics into delphi studies in library and information science. information research: an international electronic journal. 2013 sep;18(3):n3. 2. niederberger m, spranger j. delphi technique in health sciences: a map. frontiers in public health. 2020 sep 22;8:457. 3. lund bd. review of the delphi method in library and information science research. journal of documentation. 2020 feb 28. 4. eldredge jd, harris mr, ascher mt. defining the medical library association research agenda: methodology and final results from a consensus process. journal of the medical library association: jmla. 2009 jul;97(3):178. 5. okoli c, pawlowski sd. the delphi method as a research tool: an example, design considerations and applications. information & management. 2004 dec 1;42(1):15-29. 6. melander l. scenario development in transport studies: methodological considerations and reflections on delphi studies. futures. 2018 feb 1;96:68-78. 7. okoli c, pawlowski sd. the delphi method as a research tool: an example, design considerations and applications. information & management. 2004 dec 1;42(1):15-29.] 8. sekayi d, kennedy a. qualitative delphi method: a four round process with a worked example. the qualitative report. 2017 oct 1;22(10):2755-63. 6 https://www.rand.org/topics/delphi-method.html research hypothesis vol. 33 no.1 fall/winter 2021 piloting a diversity, equity, and inclusion needs assessment to explore patron perceptions at a university health science library jane morgan-daniel, mlis, ma, ahip, corresponding author community engagement and health literacy liaison librarian assistant university librarian george a. smathers libraries university of florida health science center gainesville, florida https://orcid.org/0000-0003-4047-8669 morgandanie.jane@ufl.edu lauren e. adkins, mlis, ahip college of pharmacy liaison librarian george a. smathers libraries university of florida health science center gainesville, florida https://orcid.org/0000-0002-0914-2086 lauren.adkins@ufl.edu michele r. tennant, phd, mlis, fmla, ahip interim senior director, academic research consulting and services (arcs) george a. smathers libraries university of florida health science center gainesville, florida https://orcid.org/0000-0002-8439-6772 tennantm@ufl.edu hannah f. norton, msis, ahip department chair george a. smathers libraries university of florida health science center gainesville, florida http://orcid.org/0000-0001-8062-2763 nortonh@ufl.edu chloe hough, mlis, evening circulation supervisor george a. smathers libraries university of florida health science center gainesville, florida http://orcid.org/0000-0002-4865-3049 choughatuf@ufl.edu mary e. edwards, mlis, edd liaison librarian george a. smathers libraries university of florida health science center gainesville, florida meedwards@ufl.edu https://orcid.org/0000-0003-4047-8669 mailto:morgandanie.jane@ufl.edu https://orcid.org/0000-0002-0914-2086 mailto:lauren.adkins@ufl.edu https://orcid.org/0000-0002-8439-6772 mailto:tennantm@ufl.edu http://orcid.org/0000-0001-8062-2763 mailto:nortonh@ufl.edu http://orcid.org/0000-0002-4865-3049 mailto:choughatuf@ufl.edu mailto:meedwards@ufl.edu research hypothesis vol. 33 no.1 fall/winter 2021 matthew daley, mlis webmaster and designer george a. smathers libraries university of florida health science center gainesville, florida daleym@ufl.edu melissa l. rethlefsen*, msls, ahip executive director and professor health sciences library and informatics center university of new mexico albuquerque, new mexico https://orcid.org/0000-0001-5322-9368 mlrethlefsen@gmail.com *ms. rethlefsen is the former fackler director, health science center libraries, university of florida abstract objective: a diversity, equity, and inclusion (dei) needs assessment was conducted at a health sciences library. the objectives were to (1) quantitatively assess patrons’ attitudes towards the existing dei climate in the library and (2) qualitatively identify contextual factors influencing patrons’ perceptions. the purpose was to better understand patrons’ views of the dei climate in order to identify potential gaps, strengths, and areas of improvement within the library. population: the assessment was led by a dei team at the university of florida’s health science center libraries (hscl). hscl serves the university’s six health science colleges from two sites, gainesville and jacksonville. hscl gainesville was selected as the study location, due to higher annual footfall within the physical library. methods: two “happyornot” customer satisfaction machines were placed by library exits. twelve dei-related questions were displayed on the machines, one question per week, for the duration of 12 weeks total. additionally, a box for open-ended comments was placed by each machine. the survey used convenience sampling and was anonymous. findings: the total number of responses was 3,445, with 7 written comments. overall, respondents felt happiest with the library’s welcoming and home-like environment. the library was perceived by many as a physically safe space. most also felt that the library demonstrates a commitment to dei and that they are treated fairly, equitably, and with respect by library staff. areas of concern were a perceived lack of respect from other library visitors, as well as library services for patrons with disabilities and non-native english language speakers. mailto:daleym@ufl.edu https://orcid.org/0000-0001-5322-9368 mailto:mlrethlefsen@gmail.com research hypothesis vol. 33 no.1 fall/winter 2021 conclusions: the project enabled the hscl to gather important information concerning patrons’ perceptions of the library’s dei climate and their associated needs. the happyornot machines provided a straightforward and convenient method for patron feedback. however, limitations of the pilot included the inability to obtain participants’ demographic information as well as the anecdotal evidence indicating that some respondents did not read the questions. future plans therefore involve a follow-up online survey with detailed demographic questions and further use of the happyornot machines to conduct additional micro-assessments. keywords: diversity, equity, inclusion; climate assessment; needs assessment; medical libraries; health sciences libraries introduction in october 2018, the health science center libraries (hscl) at the university of florida (uf) formed a new diversity, equity, and inclusion team. the team was created by the hscl’s director in alignment with a new strategic direction for diversity, equity, and inclusion identified by uf’s george a. smathers libraries, of which the hscl is a part [1]. one of the team’s earliest priorities—falling under one of its stated goals to “assess and identify diversity, equity, inclusion, and justice opportunities within the hscl’s spaces”—was to understand patrons’ views of the diversity, equity, and inclusion (dei) climate in hscl and thus identify potential gaps, strengths, and areas of improvement. to do so, hscl piloted the use of two “happyornot” customer satisfaction machines to conduct a dei needs assessment, with the purpose of exploring the dei-related issues facing library patrons. it was anticipated that the results would inform the team’s future directions by opening a dialogue with patrons about their dei-related needs. the hscl has two physical library sites in florida, one on the main uf campus in gainesville and the other at a teaching hospital in jacksonville, called the borland library. the dei team has members from both sites and is composed of library faculty, staff, and student assistants. hscl gainesville was the location selected for this study. the library serves the university’s six health science colleges in addition to a number of associated centers and institutes and as such experiences a significantly higher level of foot traffic in the physical library throughout the year (e.g., in 2019 over 200,000 walk-in patrons visited hscl gainesville). hscl gainesville’s patrons include, but are not limited to, the health science center colleges’ faculty, staff, and students, as well as hospital patients and members of the public. the happyornot assessment launched on april 12, 2019, and ran until july 8, 2019. the study’s aim was to explore the deiresearch hypothesis vol. 33 no.1 fall/winter 2021 related issues currently experienced by hscl patrons in gainesville. its objectives were to (1) quantitatively assess patrons’ attitudes towards the existing dei climate in the library and to (2) qualitatively identify contextual factors influencing patrons’ perceptions. literature review assessing patron satisfaction in libraries the inspiration for the needs assessment was a study conducted by the university of central florida (ucf), in which happyornot machines were used alongside several other micro-assessments to elicit patron feedback regarding library services, events, resources, and policies [2]. the goal of the micro-assessments was to decrease the time taken to obtain feedback in comparison to more common methods of assessments such as surveys and focus groups. according to ucf, the happyornot machines were an effective method for gauging patrons’ opinions on library policies and resources. overall, the micro-assessments enabled the librarians to quickly obtain feedback from patrons, allowing them to create solutions more efficiently. happyornot machines are utilized globally to measure customer satisfaction in a variety of sectors such as retail, transportation, restaurants, education, healthcare, and human resources [3]. the machine terminals have four emoji buttons: a “very positive” green smiling emoji, a “positive” light green slightly smiling emoji, a “negative” light red slightly frowning emoji, and a “very negative” red frowning emoji. users can press an emoji button to answer questions on the terminal screen, thereby recording their experiences and responses in real time. when considering selecting happyornot as the data collection instrument for this study, the dei team had a number of initial concerns about the emojis and the color combinations of the buttons. these concerns related to whether the associated meaning of the buttons would be clearly conveyed to all patrons, including international students and people with color-blindness or autism. these concerns were allayed through a literature search pertaining to emojis and color as methods of communication. communicating emotion through emojis and color associations emojis consist of facial images that convey a variety of emotions in pictorial form and have been used by industries to gauge users’ satisfaction [4]. for example, jaeger et. al’s study demonstrates how emoji surveys were successfully used to measure customers’ satisfaction concerning new food and beverage products [5]. emojis have also been used for educational purposes; for example, a study by voss discusses the research hypothesis vol. 33 no.1 fall/winter 2021 use of emojis to teach children with autism how to better recognize emotions [6]. the dei team found this article encouraging, as it indicated that the meaning of happyornot emojis would be inclusive for most patrons. regarding color, many studies discuss the use of color to convey particular emotions or to assess participants' perceived emotions. a study of college students conducted by kaya and epps found that participants universally attributed the red to anger when assigning colors to emotional reactions. [7] in contrast, the color green was mainly associated with positive emotions due to reminding participants of nature. when seeking to obtain participant feedback, the literature illustrates that not all cultures and nationalities ascribe the same meaning to colors and emotions. a seminal study by saito found that people from japan and nearby asian countries often have a preference for the color white due to religious connotations [8]. study participants associated white with cleanliness, purity, harmony, beauty, and gentleness. similarly, a study conducted by hupka with participants from germany, mexico, poland, russia, and the united states found that the colors red and black were universally associated with anger [9]. even though studies have shown that various cultures and nationalities can hold different emotional associations for color, the association of green with the emotion of positivity and red with negativity is common across several cultures and nationalities [710]. after reviewing this literature, the dei team therefore felt comfortable moving forward with piloting the use of the happyornot machines for the needs assessment. dei climate assessments the development of the pilot questions stemmed from a search conducted for dei assessment literature in february 2019. first, the team looked for existing dei climaterelated surveys within the librarianship and health literature. the majority of these search results focused on surveying employees not patrons. for example, the literature highlights the recurring use of climatequal, which is an instrument frequently used within libraries to assess organizational climate through staff perceptions [11]. the search was therefore expanded to encompass university-level dei surveys, which are often accessible through google as grey literature in the form of reports. the google search yielded useful results. the first was case western reserve university’s climate survey, which used likert scale questions to assess how students and faculty perceived the environment for inclusion and diversity [12]. these questions pertained to participants’ general comfort levels, their experiences of discrimination, and whether they felt that the environment encourages diverse groups of people to interact. overall, the results illustrated “differences both among and within groups by gender, race/ethnicity, and international status” (p.1) [12]. another source of inspiration for the research hypothesis vol. 33 no.1 fall/winter 2021 happyornot questions was the university of michigan’s campus climate survey on diversity, equity and inclusion [13]. participants were asked to indicate their level of agreement with a number of statements, including those focused on institutional commitment to dei and on individuals being valued, treated with respect, and feeling they belong. the survey found that students with historically marginalized identities experienced the campus less positively [13]. at the time of planning the pilot, the dei team’s lead was a member of the medical library association’s (mla) diversity and inclusion task force and, as such, helped with question formation for an online general membership survey that took place in 2019 [14]. the survey’s purpose was to facilitate deeper understanding of the membership’s demographics and their feelings in relation to mla’s dei efforts. three questions from the survey were repurposed for the happyornot assessment: “i have felt welcomed and included at mla annual meetings”; “i feel mla has a strong commitment to diversity, equity, and inclusion”; and “i feel i am treated with respect in mla.” since the original searches in early 2019, the librarianship literature has expanded to include three new sources of relevance. the first is pionke’s article which concludes by emphasizing the importance of conducting recurring dei surveys, ideally every three years [14]. secondly, jones and murphy discuss how dei climate surveys benefit library administration through enabling leaders to assess organizational readiness for “maintaining and supporting an environment that is safe, inclusive, and welcoming” (p.79) [15]. finally, the wisconsin department of public instruction provides a reflection tool for libraries seeking to evaluate services on an ongoing basis through a dei lens [16]. the guide is a checklist that encourages organizations to reflectively consider their communities’ demographics, to ensure that every patron feels “safe, welcomed, and respected” throughout all library experiences. methods twelve closed-ended questions (table 1) were developed through team discussion and consensus, based on the recurring dei assessment themes discovered in the literature: welcome, respect, inclusivity, accessibility, physical and emotional safety, fair and equitable treatment, and organizational commitment to dei. the overarching purpose of these questions was to facilitate assessment of the library’s dei climate by evaluating patrons’ satisfaction with the library’s physical space, library services, and staff attitudes and behaviors. the questions were displayed on two happyornot machines, one question per week, for the duration of 12 weeks total. a comment box was placed by research hypothesis vol. 33 no.1 fall/winter 2021 each machine, so that patrons could provide open-ended written responses to the weekly questions if they chose to while still maintaining their anonymity. week 1 do you feel welcome at this library? week 2 do you feel you are treated with respect by this library’s staff? week 3 do you feel you are treated with respect by other visitors at this library? week 4 do you feel this library is an inclusive physical space? week 5 do you feel this library is a physically safe place for people of all backgrounds? week 6 do you feel this library is an emotionally safe space? week 7 are you satisfied with this library’s services for people whose native language is not english? week 8 do you feel this library reliably meets the needs of individuals with disabilities? week 9 do you feel this library is a welcoming environment for families? week 10 do you feel you are treated fairly and equitably at this library? week 11 do you feel that this library’s staff will take appropriate action in response to incidents of discrimination? week 12 do you feel this library demonstrates a strong commitment to diversity, equity, and inclusion? table 1 happyornot questions by week two happyornot machines were purchased by the hscl and placed by high-traffic library exits on the first and second floors. specifically, the smiley terminal, depicted in figure 1, was selected as this model is light-weight and portable, runs on batteries so is wireless, and has a simple interface encompassing a sleeve for printed questions. an additional advantage was that the data analysis process was expected to be mostly automated as these terminals are linked to a web-based software that compiles respondents’ feedback as it is submitted. in terms of initial machine set-up, the online software provided access to a personalized happyornot dashboard to which the 12 assessment questions were added. each research hypothesis vol. 33 no.1 fall/winter 2021 question had to be entered separately and the dates during which each question would be active had to be scheduled. this added a layer of complexity to the data collection process because, although the software was straightforward to use, the weekly question displayed on the actual happyornot terminals needed to be printed. this meant that the dei team needed to physically change out the printed question on the correct dates to ensure that patrons’ responses matched the correct question on the web-based dashboard. for the design of the printed questions, the dei team chose dark text on a white background. an explanatory sentence was added to each printout stating “hsc library staff want to better understand your experiences relating to diversity & inclusion in the library – please provide feedback to help us enhance services.” this design was replicated on the two comment boxes (figure 1). figure 1 image of the hscl’s happyornot smiley terminal and comments box after the study was approved as irb exempt (irb201900781), the pilot launched on april 12, 2019, and ran until july 8, 2019. an earlier timeframe was initially selected to coincide with the spring semester, but machine-ordering delays meant that the study continued into finals week and the beginning of the summer semester. the assessment was promoted through the library’s facebook and twitter accounts, through the website, and in-person by the desk staff. findings research hypothesis vol. 33 no.1 fall/winter 2021 the total number of responses received for all questions was 3,445 across both happyornot terminals, along with 7 written comments. the response rate was approximately 4.3% as around 81,000 visitors entered the library during the assessment period. patterns of gate count matched well with the number of responses over the study timeframe. the first two weeks of the assessment occurred during the last two weeks of the semester, and average gate count and response numbers were 13,979 and 724 respectively (19.3 visits per response). weeks 4 through 12 reflect the lower gate count seen during intersession and summer sessions weeks, with average gate count as 4,631 and response numbers averaging 184 (25.2 visits per response). finals week was the outlier, with 11,413 entrances and exits but only 342 responses (33.4 visits per response). quantitative results the quantitative results were automatically collated through the happyornot software, by number of responses per question and percentages of responses by question, for the four available button options on the machines. microsoft excel was used to filter the data by time period (weeks, days, hours) and by location within the library (first or second floor). overall, the team was excited to see that the majority of the responses received for all 12 questions was “very positive.” research hypothesis vol. 33 no.1 fall/winter 2021 figure 2 quantitative results by percentage of responses for weeks 1–12 the highest number of responses for the whole assessment were received for the first 2 questions posted, which were do you feel welcome at this library? (828 responses) and do you feel you are treated with respect by this library’s staff? (620 responses). figure 2 shows the responses for all 12 questions, broken down by the percentage of individuals answering for “very positive,” “positive,” “negative,” and “very negative.” the following questions had the highest percentages of responses for “very positive” and “positive”: research hypothesis vol. 33 no.1 fall/winter 2021 • do you feel you are treated fairly and equitably at this library? (combined 98% for “very positive” and “positive” from 91 responses) • do you feel this library demonstrates a strong commitment to diversity, equity, and inclusion? (combined 98% for “very positive” and “positive” from 106 responses) • do you feel you are treated with respect by this library’s staff? (combined 97% for “very positive” and “positive” from 620 responses) • do you feel this library is a physically safe place for people of all backgrounds? (combined 95% for “very positive” and “positive” from 259 responses) the questions with the largest percentage of “negative” and “very negative” responses were as follows: • do you feel you are treated with respect by other visitors at this library? (combined 10% “very negative” and “negative” from 342 responses) • are you satisfied with this library’s services for people whose native language is not english? (8% combined “negative” and “very negative” from 163 responses) • do you feel this library reliably meets the needs of individuals with disabilities? (7% “negative” or “very negative” from 261 responses) the results were also analyzed by the overall number of positive (“positive” combined with “very positive”) and negative (“negative” combined with “very negative”) responses per hour over the whole 12-week period (appendix a). the highest number of negative responses received coincided with the hours that had the most overall responses, which was around lunchtime between 12pm and 2pm. the same pattern held true for the highest number of positive responses received. an exception to this was the negative results often received between 7am and 9am, which we attribute to the unavoidable daily noise that the library cleaners make during this time period. in terms of the highest percentages of positive or negative responses per hour throughout the whole assessment period, participants responded most favorably at 2am, 3am, and 5am (100% “very positive” or “positive”) and most unfavorably at 4am and 6am (50-60%). however, due to the smaller number of overall responses at these times (2am, n=8; 3am, n=5; 4am, n=2; 5am, n=2; 6am, n=4), little weight can be placed on these findings. finally, the results were analyzed by the location of the terminals (appendix b). unsurprisingly, the machine at the main entrance of the library, on the first floor by the information services desk, collected 72% (2,488) of the total responses. during the assessment period, the second floor entrance received approximately 40% as much research hypothesis vol. 33 no.1 fall/winter 2021 traffic as the first floor; likewise, 28% (1,273) of the total responses were from the second floor. responses from the second floor were slightly more negative (combined 8.16% “very negative” and “negative” from 104 responses) than those from the first floor (combined 2.74% “very negative” and “negative” from 88 responses). qualitative results only 7 written responses were received via the comment boxes over the entire 12-week period of the survey. these qualitative results were thematically analyzed, with written responses divided into three categories: positive, mixed, and negative. three of the seven responses were extremely positive, suggesting that the library is perceived as a welcoming and homey place and that patrons appreciate the personal greetings they receive when they walk in from the desk staff: “best. library. ever. i love how the staff always says ‘hi’ when you walk in. i feel at home here.” (week 1) “love this library” (week 7) “yaaaaaaaaaaaa :)” (week 8) two of the written responses were mixed with one response again implying that the library is perceived to be a welcoming and safe space and the other referring positively to dei. at the same time, the mixed responses also commented on aspects of the library space that could be improved: “diversity is good but please do not eat snacks in [the] library” (week 1) “i really enjoy coming here, i feel welcome & safe. i feel sometimes it could be a little cleaner.” (week 1) the final two responses were negative. both were received during the week the hscl was experiencing problems with overnight access to the library via swipe-card (see the discussion section): “notify students of all 6 health colleges that they have 24 hour access and to use their ids when [the] library is closed.” (week 3) “larger warning and stop signs!” (week 3) research hypothesis vol. 33 no.1 fall/winter 2021 discussion implications the quantitative and qualitative findings suggested that patrons are happiest with the library’s welcoming and home-like environment; the library is perceived by many as a physically safe space; patrons largely feel the library demonstrates a commitment to dei; and patrons feel they are treated fairly, equitably, and with respect by library staff. of concern, some patrons expressed that they did not receive enough respect from other library visitors. the hscl has taken these responses seriously and has created a patron code of conduct, delineating expectations related to respect, language, and physical contact [17]. the code emphasizes to patrons that the library is a community space, welcoming and inclusive to all, and clearly communicates consequences for repeated violations. the team also created accompanying guidance documentation for staff regarding how to respond to specific instances of disruption and harassment as well as a three strikes policy to remove offenders. the pilot was administered during a time when the library experienced an ongoing malfunction in its overnight entrance to quiet study space, and it is possible that this contributed to patrons’ perceptions of lack of respect from other library users. during this timeframe, the library received numerous complaints about patrons causing noise and disruption through knocking on the doors and accidentally setting off the alarms. subsequently, the issue with the doors has been resolved. additionally, the library has been proactive in decreasing user error, by improving the visibility and readability of the door signage instructions for 24-hour library access and emailing all of the students from the six health science center colleges to remind them how to access the library outside of staffed hours. the library’s marketing team also produced a video tutorial on how to access the library after hours, which was distributed through the library’s social media pages [18]. other concerns revealed by the assessment suggest that the library needs to improve its services for patrons with disabilities and non-native english language speakers. the result for disabilities is unfortunately not surprising. the hscl encompasses three floors, with the main book collection and heavily used quiet study space located on the third floor and the information desk (check out, reserves, front-line information/reference) located on the first floor. there are no elevators in the library proper, and the building elevators and ada door buttons were unreliable at times during the assessment period. following the pilot, library administration advocated for research hypothesis vol. 33 no.1 fall/winter 2021 replacement of the ada door buttons as well as the creation of a wheel-chair accessible option to enter the hscl’s newly refurbished computer lab. both of these improvements have been accomplished. the dei team was initially surprised by the results of the language-related question. on reflection, uf’s health science center does have a large number of international graduate and professional students (12.01% of students from 33 countries) [19]. a large number of post-doctoral associates at uf are also international (41.9% of them are not u.s. citizens or nationals) [20]. although uf does not keep statistics on the number of languages spoken at the university, these figures on the international presence suggest that large numbers of the uf community do not speak english as their primary language. while the responses to the language question points to an area of concern, it is not possible, based on the lack of specificity in the question, to determine exactly what needs to be improved in this area. although hscl cannot make all resources and signage in all languages, it is certainly within the library’s purview to be knowledgeable about language resources and services on campus and to train staff to better and more patiently communicate with patrons whose native language is not english. while the overall feedback related to staff respect, comfort, equity and safety was positive, the team understands that ongoing success in dei requires ongoing effort. we have organized a number of trainings for staff (micro-aggression, bystander intervention, and many others) and will continue to do so. limitations while the library did implement a number of changes based on the happyornot results, there were multiple limitations that hindered the team’s ability to conclusively answer all of its questions. due to the structure of the happyornot machines, there was no collection of demographic data, which made it impossible to discern whether diverse population groups experience the library differently. as such, we do not know whether the negative responses came from specific population groups. as the machines were located in the physical library, this meant that individuals who did not visit the library in person were unable to participate (such as distance students or those who may feel uncomfortable coming to the library). since the purpose of the pilot was to assess patrons’ feelings about the library’s dei climate, patrons who do not feel welcome at the library being excluded could skew the results to the positive. another issue with the format of the pilot was anecdotal evidence indicating that patrons did not always read the questions and answer accordingly. desk staff noticed patrons not fully reading the prompts, or not reading them at all, before hitting the response research hypothesis vol. 33 no.1 fall/winter 2021 buttons. for example, a team member working the information desk at the first floor entrance observed multiple instances of patrons hitting one of the buttons after incorrectly or incompletely reading the prompt and voicing their mistake afterwards, i.e., not realizing that the question was asking about dei in the library. it was also observed that many patrons would repeatedly and sometimes excessively hit the “very positive” button without looking at the prompt when exiting the library. while this could indicate that the patrons are happy with the library in general, the unknown extent of repeat responses muddles the data regarding the specific questions posed. the disregard for the actual questions may have been a result of question fatigue or simply the fact that the happyornot format is generally associated with customer satisfaction, causing patrons to not look at the specific wording presented each week. happyornot’s website states that they are “the global leader in instant customer and employee satisfaction reporting.” if patrons are mainly familiar with seeing feedback machines in businesses and retail locations, it is possible that they would assume the question is asking about general satisfaction [3]. the timing of the assessment caused another barrier to participation. the pilot was launched toward the end of the spring semester, when the happyornot machines arrived at the library after several ordering delays; based on the pilot start date, the study continued into finals week and the summer semester. since many students opt not to take summer classes, a significant number leave gainesville from may to august. as such, the library receives a significantly lower footfall during the summer in comparison to the fall and spring. in 2019 the number of visitors to the library dropped from 59,239 in april to 21,567 in may, a 64% decrease. this drop in overall library visitation during the assessment period is reflected in the lower participation rate in the later weeks of the pilot. as mentioned, most responses were received during the first two weeks of the study, the time that coincided with the end of the spring semester. during the final 6 weeks of the pilot, the number of responses was 1,017, accounting for 30% of the total responses over the 12 weeks. while questions posed later in the assessment received fewer responses, this timeframe did allow the dei team to learn from users quickly and act accordingly, more so than if the study had been delayed to the fall semester. the dei team feels that the happyornot machines were useful for gauging patrons’ general opinions of the library and its services, thereby concurring with ucf who used this data collection instrument during their own micro-assessments [2]. however, happyornot machines are less useful for dei-specific needs assessments, due to the inability to collect data on participants’ demographics. it is the team’s opinion that when paired with demographic data, the pilot questions used in this study would likely reveal whether underrepresented groups experience the library’s dei climate differently to their research hypothesis vol. 33 no.1 fall/winter 2021 majority counterparts. the team therefore looks forward to gaining more detailed evidence through future iterations of this study. with those iterations, we also hope to compare our findings to case western reserve university and the university of michigan’s campus climate surveys [12, 13], in which participants with historically marginalized identities tended toward less positive experiences. future research as described previously, our experiences suggest that happyornot machines provide only a very cursory snapshot when considering specific issues within dei. it is unclear whether respondents consistently read the questions or were aware that the questions changed every week—either of these issues had the potential to affect data collection. of greater consequence was the inability to track who was answering and how, which made it impossible to determine whether underrepresented groups have different perceptions or experiences than their majority counterparts. in order to address some of the limitations of this pilot project, the hscl has developed an online survey that includes detailed demographic questions (age, gender identity, race, sexual orientation, native language, international status, first generation status, and caregiving responsibilities). the survey includes the original happyornot questions, augmented with new queries related to the inclusiveness of the library’s web presence and hscl’s inclusiveness to families, diverse religions, and other identities. the dei team looks forward to analyzing the results of this assessment of inclusivity and to responding to any deficits discovered. the team also plans to complete a local library-based environmental scan, stemming from the priorities identified through the pilot happyornot project and subsequent online survey. from the pilot’s results, we anticipate that the environmental scan will begin by focusing on current facilities and services for people with physical disabilities— identifying current resources and resource gaps and areas for improvement such as where policies need to be made, training conducted, or resources publicized. because the happyornot pilot results suggest services for non-native english language speakers and persons with disabilities are of concern, the dei team plans to conduct focus groups with such patrons to find out which specific improvements the library can make to its spaces and resources to be more inclusive. the hscl’s dei assessment efforts will be an ongoing process. although the happyornot system is not ideal for a detailed assessment, it is possible that making its intent (dei) and framework (the questions change) more apparent to patrons could yield results that better reflect their experiences and perceptions. as such, the team plans to research hypothesis vol. 33 no.1 fall/winter 2021 run a re-worked happyornot assessment in 2022, post-covid. in addition to an advertising campaign introducing the intent of the happyornot assessment, it may be beneficial to switch questions every two weeks rather than every week, as this may give patrons who visit the library less frequently a better chance to respond to each question. conclusion it is crucial for libraries to continuously work towards incorporating dei and social justice initiatives into their services for their communities while also as striving to become safe and welcoming spaces that celebrate the diversity of their patrons and staff. as evidenced by the literature, the majority of dei-related surveys conducted in libraries tend to assess dei climate through evaluating staff perceptions. however, it is also essential to evaluate patron perceptions on an ongoing basis through a dei lens [1516]. overall, the hscl dei team feels that this pilot project was fairly successful. the happyornot machines enabled us to gather important information concerning patrons’ perceptions of the library’s dei climate and their associated needs in relation to physical space, services, and staff attitudes and behaviors. the happyornot machines provided a straightforward and convenient method for patron feedback, resulting in an impressive number of responses. however, for a more nuanced analysis of a library’s dei climate, multiple data collection methods are recommended. in conclusion, it is essential to note that dei work is an ongoing process that takes time, effort, innovation, and—most importantly—the ability to listen to and act on the needs of the communities being served. author contributions conceived and designed project: mlr, jmd, hfn, md data collection: jmd, md, ceh, mrt, hfn, mlr, mee, lea data analysis: quantitative: hfn, mee; qualitative: jmd figures: hfn, mee, md written draft: jmd, lea, mrt, ceh, hfn article revision: hfn, mrt, mlr, jmd, mee, ceh, lea, md research hypothesis vol. 33 no.1 fall/winter 2021 final approval: jmd, ceh, hfn, lea, md, mee, mlr, mrt references 1. university of florida george a. smathers libraries. strategic directions [internet]. gainesville, fl: university of florida; 2018 dec [cited 1 mar 2021]. <https://uflib.ufl.edu/wordpress/files/2021/04/strategicdirections2018web.pdfhttps://librar y.ufl.edu/pers/documents/strategicdirections2018web.pdf>. 2. gillum s, williams n, gotschall t. a year of creative assessment: incorporating micro-assessments into the library [internet]. medical library association conference: chicago, il; 2019 mar 18 [cited 1 mar 2021]. <https://www.eventscribe.com/2019/mla/assets/pdf/mla19_paper_abstracts_march_1 8.pdf>. 3. happyornot [internet]. [cited 1 mar 2021]. <https://www.happy-ornot.com/en/smiley-terminal/>. 4. rodrigues d, prada m, gaspar r, garrido m, lopes d. lisbon emoji and emoticon database (leed): norms for emoji and emoticons in seven evaluative dimensions. behav res. 2017 march 31;50:392–405. doi: https://doi.org/10.3758/s13428-017-0878-6. 5. jaeger s, lee s, kim k, chheang s, jin d, ares g. measurement of product emotions using emoji surveys: case studies with tasted foods and beverages. food qual prefer. 2017 december; 62: 46–59. doi: https://doi.org/10.1016/j.foodqual.2017.05.016. 6. voss c, schwartz j, daniels j, kline a, haber n, washington p, tariq q, robinson t, desai m, phillips j, feinstein c, winograd t, wall d. effect of wearable digital intervention for improving socialization in children with autism spectrum disorder: a randomized clinical trial. jama pediatr. 2019 may 1;173(5):446–454. doi: https://doi.org/10.1001/jamapediatrics.2019.0285. 7. kaya n, epps hh. relationship between color and emotion: a study of college students. coll stud j. 2004 sep;38(3):396–405. 8. saito m. comparative studies on color preference in japan and other asian regions, with special emphasis on the preference for white. color res appl. 1996 feb;21(1):35–49. doi: https://doi.org/10.1002/(sici)1520-6378(199602)21:1<35::aidcol4>3.0.co;2-6. 9. hupka rb, zaleski z, otto j, reidl l, tarabrina nv. the colors of anger, envy, fear, and jealousy: a cross-cultural study. j cross cult psychol. 1997 mar;28(2):156–71. doi: https://doi.org/10.1177%2f0022022197282002. 10. yu hc. a cross-cultural analysis of symbolic meanings of color. chang gung journal of humanities and social sciences. 2014 apr 1;7(1):49–74. https://uflib.ufl.edu/wordpress/files/2021/04/strategicdirections2018web.pdfhttps:/library.ufl.edu/pers/documents/strategicdirections2018web.pdf https://uflib.ufl.edu/wordpress/files/2021/04/strategicdirections2018web.pdfhttps:/library.ufl.edu/pers/documents/strategicdirections2018web.pdf https://www.eventscribe.com/2019/mla/assets/pdf/mla19_paper_abstracts_march_18.pdf https://www.eventscribe.com/2019/mla/assets/pdf/mla19_paper_abstracts_march_18.pdf https://www.happy-or-not.com/en/smiley-terminal/%3e. https://www.happy-or-not.com/en/smiley-terminal/%3e. https://doi.org/10.3758/s13428-017-0878-6 https://doi.org/10.1016/j.foodqual.2017.05.016 https://doi.org/10.1001/jamapediatrics.2019.0285 https://doi.org/10.1002/(sici)1520-6378(199602)21:1%3c35::aid-col4%3e3.0.co;2-6 https://doi.org/10.1002/(sici)1520-6378(199602)21:1%3c35::aid-col4%3e3.0.co;2-6 https://doi.org/10.1177%2f0022022197282002 research hypothesis vol. 33 no.1 fall/winter 2021 11. association of research libraries. climatequal [internet]. washington, dc; [cited 10 mar 2021]. <https://www.climatequal.org/home>. 12. case western reserve university. 2010 2011 climate survey: results on common questions [internet]. cleveland, oh: case western reserve university; 2011 [cited 10 on mar 2021]. <https://case.edu/diversity/about/diversity-climate-survey>. 13. university of michigan. results of the 2016 university of michigan student campus climate survey on diversity, equity and inclusion [internet]. anne arbor, mi: university of michigan; 2016 [cited 10 on mar 2021]. <https://diversity.umich.edu/wpcontent/uploads/2017/11/dei-student-report-final.pdf>. 14. pionke, j. medical library association diversity and inclusion task force 2019 survey report. jmla. 2020 jul;108(3):503–512. doi: https://doi.org/10.5195/jmla.2020.948. 15. jones sd, murphy b. diversity and inclusion in libraries: a call to action and strategies for success. lanham, maryland, 2019. chapter 8: recruiting and retaining a diverse workforce; p. 75–88. 16. wisconsin department of public instruction. the inclusive services assessment and guide for wisconsin public libraries [internet]. madison, wi; 2019 jun [cited on 10 mar 2021]. <https://files.eric.ed.gov/fulltext/ed599993.pdf>. 17. university of florida health science center libraries. code of conduct [internet]. gainesville, fl: university of florida; 2021 apr [cited 1 apr 2021]. <https://library.health.ufl.edu/about-us/library-user-policies/code-of-conduct/>. 18. university of florida health science center libraries. in today's #howtolibrary post, we're reminding hsc students of their 24/7 access to our 2nd floor! we've been having some swipe card system issues, so if you're an hsc student and your card isn't working, drop by the 1st floor desk and we'll help get you sorted. happy studying! [internet]. 2021 jan 7 [cited 1 apr 2021]. <https://www.facebook.com/ufhscl/posts/5027652200608520>. 19. university of florida graduate school data management unit. re: question about grad students and professional students. email message to: michele tennant. 2021 mar 8, 11:15 a.m. [12 lines]. 20. university of florida. uf facts workforce [internet]. gainesville, fl: university of florida; 2021 mar [cited on 1 apr 2021]. <https://public.tableau.com/shared/d49ww6bx7?:display_count=y&:origin=viz_share_li nk&:embed=y&:showvizhome=no>. https://www.climatequal.org/home https://case.edu/diversity/about/diversity-climate-survey https://diversity.umich.edu/wp-content/uploads/2017/11/dei-student-report-final.pdf https://diversity.umich.edu/wp-content/uploads/2017/11/dei-student-report-final.pdf https://doi.org/10.5195/jmla.2020.948 https://files.eric.ed.gov/fulltext/ed599993.pdf https://library.health.ufl.edu/about-us/library-user-policies/code-of-conduct/ https://www.facebook.com/ufhscl/posts/5027652200608520 https://public.tableau.com/shared/d49ww6bx7?:display_count=y&:origin=viz_share_link&:embed=y&:showvizhome=no https://public.tableau.com/shared/d49ww6bx7?:display_count=y&:origin=viz_share_link&:embed=y&:showvizhome=no research hypothesis vol. 33 no.1 fall/winter 2021 appendixes appendix a: responses by hour research hypothesis vol. 33 no.1 fall/winter 2021 appendix b: percentage of responses by library floor editorial reviewed article hypothesis, vol. 37, no. 1, 2025 statistics for librarians: probability and odds caitlin bakker, mlis, ahip-da adiscovery technologies librarian, dr. john archer library, university of regina, regina, saskatchewan, canada, https://www.orcid.org/0000-0003-4154-8382 , caitlin.bakker@uregina.ca cite as: bakker c. statistics for librarians: probability and odds. hypothesis. 2025;37(1). doi:10.18060/28737 bakker. all works in hypothesis are licensed under a cc by-nc 4.0 deed attribution-noncommercial 4.0 international. authors own copyright of their articles appearing in hypothesis. readers may copy articles without permission of the copyright owner(s), as long as the author(s) are acknowledged in the copy, and the copy is used for educational, not-for-profit purposes. for any other use of articles, please contact the copyright owner(s). 1 https://www.orcid.org/0000-0003-4154-8382 https://orcid.org/ mailto:caitlin.bakker@uregina.ca https://creativecommons.org/licenses/by-nc/4.0/ https://creativecommons.org/licenses/by-nc/4.0/ editorial reviewed article hypothesis, vol. 37, no. 1, 2025 probability refers to how likely it is that an event would occur. it is fundamental to statistics, as it tells us the likelihood of particular outcomes and allows us to calculate risk, make predictions, and measure our level of certainty. probabilities are always between 0 and 1, where 0 would indicate that an event is impossible while 1 would indicate that it always happens. probabilities can be expressed as fractions, decimals, or percentages. while we may not formally calculate probabilities regularly, most people use probabilities on a daily basis, whether that’s looking at a weather forecast, trying to predict the stock market, or making decisions about medical treatment. understanding probability is crucial, both to research and to daily life. there are different types of probability and different ways of calculating probability depending on circumstances. this column will attempt to introduce some of these concepts. readers looking for a more in-depth resource may wish to consult kunin et al.’s seeing theory1, grinstead & snell’s introduction to probability2, or orloff and kamrin’s course, introduction to probability and statistics3. types of probability theoretical probability can be used when no experiment has occurred, and no observations have been made. it is probability that’s based on available data and logic and is essentially reflecting what is expected. for example, we can predict the probability of flipping a coin and it landing on heads, or rolling a die and it landing on a specific number. the formula for theoretical probability is: p(a) = the number of ways a can happen the total number of outcomes in the above formula, a refers to our event of interest, like a coin being flipped and landing on heads. while theoretical probability can be useful, it also has limitations. theoretical probability assumes somewhat idealized conditions, in which all outcomes have the same probability of occurring. in real life, this is often not the case. for example, the outcomes in medical trials may not all be equally likely. in these cases, we could not find the probability of an outcome without having further information or making assumptions. scientists may be able to use their expertise, combined with information about the specific disease and treatment, to make assumptions about probability in those cases. unlike theoretical probability, where we calculate the probability based on what is expected to happen, in empirical probability we instead focus on what actually happened. the formula for empirical probability is: p(a) = the number of times a occurred the number of observed occurrences let’s imagine that we have a library that has hosted a blind date with a book event. after the event, the library reviews data gathered from its gate counter to determine patron foot traffic and circulation statistics to see how many of the books circulated. they found that: 1. 1000 patrons visited the library 2. 165 patrons selected one item from “blind date with a book” 2 editorial reviewed article hypothesis, vol. 37, no. 1, 2025 3. 23 patrons selected two items from “blind date with a book” 4. 12 patrons selected three or more items from “blind date with a book” if we wanted to calculate the probability that a person visiting the library would also participate in the blind date with a book event, we could do so as follows: p(participation) = (165 + 23 + 12) 1000 = 200 1000 = 1 5 (or 0.2 or 20%) we can calculate that the probability a randomly selected patron would be a participant in the blind date with a book event is 20%. having this sort of data can help the library plan for future events by providing a baseline for assessment and planning. it’s important to note that, if the probability of participation is 20%, then the probability of non-participation is 80%. this is due to the complement rule, which states that the sum of the probabilities of all possible outcomes must equal 1 (or 100%). while empirical probability does not make the same assumptions that theoretical probability does, it has its own limitations, particularly when working with small data sets. for example, if we only observed five patrons entering the library, three of whom participated in the blind date with a book event, you might conclude that the probability of participation is 60%. as additional data is gathered, empirical probability will approach theoretical probability. this is known as the law of large numbers, which states that if you repeat an experiment a large number of times, the result obtained should be close to the expected value. one of the most famous examples of this was conducted by mathematician john kerrich, who flipped a coin 10,000 times, recording whether it landed on heads or tails each time4. the first three flips landed on tails. at that point, one would erroneously calculate that the probability of landing on tails was 100%. however, by the end of the 10,000 flips, kerrich had recorded 5,067 heads and 4,933 tails. the theoretical probability of landing on tails (49.33%) closely mirrors the theoretical probability of landing on heads (50%). dependence while we may want to calculate the probability of an individual event occurring, there are also times when we will need to calculate the probability of multiple events occurring. to understand how to calculate these probabilities, we need to know how the events relate to each other. independent events are those in which the probability of one event does not change the probability of the other. for example, we know that the probability that a library patron would participate in the blind date with a book event is 20%. based on previously gathered data, we also know that the probability that a library patron would access wifi is 95%. these two events are independent, since accessing wifi (or not) has no impact on participating in blind date with a book (or not). when calculating the probability for independent events, we use the multiplication rule: p(a and b) = p(a) x p(b) 3 editorial reviewed article hypothesis, vol. 37, no. 1, 2025 in this case, if we want to know the probability that a patron would both access the wifi and participate in the blind date with a book event, we can do so as follows: p(wifi and participation) = p(wifi) x p(participation) = 95 100 x 1 5 = 95 500 = 0.19 ultimately, we find that the probability of the patron participating in blind date with a book and accessing the wifi is 19%. dependent events are those that are affected by the outcome of other events. for example, at the blind date with a book event, the first patron comes into the event and selects a biography. if that item isn’t replaced with another biography, the original probabilities change. type of book original probability after the first patron mystery 10/25 = 0.4 = 40% 10/24 = 0.42 = 42% science fiction 10/25 = 0.4 = 40% 10/24 = 0.42 = 42% biography 5/25 = 0.2 = 20% 4/24 = 0.17 = 17% the previous event of a biography being selected impacts the probability of the other items being selected, making these dependent events. some dependent events are also what are called mutually exclusive events. mutually exclusive events are events that cannot happen at the same time. for example, a patron takes out a single book as part of the blind date with a book event and we want to know the probability that they would pick either a science fiction book or a biography. these are mutually exclusive events, since one book (presumably) couldn’t be both. with independent events, we calculate the probability that both happened (you can think of these as “and” events), while with mutually exclusive events, we can calculate the probability that either happened (“or” events). we can do this using the addition rule: p(a or b) = p(a) + p(b) in this case, if we want to calculate the probability that a randomly selected patron would choose either a science fiction novel or a biography, we could do so as follows: p(science fiction or biography) = 10 25 + 5 25 = 15 25 = 0.6 we find that there is a 60% chance that a randomly selected patron would choose either a science fiction novel or a biography. odds odds and probability are closely related, but not synonymous. they are both ways of expressing the likelihood of an event. we know that the probability of a patron participating in the blind date with a book event is 20%. but, if you were to say that the odds of a patron participating were 1/5, that would be incorrect. 4 editorial reviewed article hypothesis, vol. 37, no. 1, 2025 while empirical probability is the likelihood of an event occurring compared to the possible outcomes, odds are the probability of an event occurring dividing by the probability of it not occurring. we can calculate the odds for an event using the probability of that event: odds(a) = p(a) 1 − p(a) unlike probability, odds aren’t restricted to a 0 to 1 scale. because odds can be greater than 1, they are typically presented as fractions or decimals, rather than percentages. if the probability of participation is 20%, we could calculate the odds as follows: odds(participation) = 0.2 1 − 0.2 = 0.2 0.8 = 1 4 while it may strike you as strange that odds and probability are different, you can think of them as two different ways of expressing likelihood. with probability, we can say that there is a 20% chance that a patron would have participated in the event. with odds, we can say that for every patron that would have participated, four would not have. while both are communicating likelihood, they allow us to emphasize different things and can ultimately be used in different, more advanced calculations. references 1. kunin d, guo j, devlin td, xiang d. seeing theory. [cited 2025 jan 11]. available from: http://seeingtheory.io 2. grinstead cm, snell jl. introduction to probability [internet]. american mathematical society; 1997. [cited 2025 jan 11]. available from: https://open.umn.edu/opentextbooks/textbooks/21 3. orloff j, kamrin jf. introduction to probability and statistics. mit opencourseware; 2022. [cited 2025 jan 11]. available from: https://ocw.mit.edu/courses/18-05-introductionto-probability-and-statistics-spring-2022/pages/syllabus/ 4. kerrich je. an experimental introduction to the theory of probability. einar munksgaard; 1946. 5 http://seeingtheory.io https://open.umn.edu/opentextbooks/textbooks/21 https://ocw.mit.edu/courses/18-05-introduction-to-probability-and-statistics-spring-2022/pages/syllabus/ https://ocw.mit.edu/courses/18-05-introduction-to-probability-and-statistics-spring-2022/pages/syllabus/ hypothesis, vol. 37, no. 1, 2025 editorial board updates: emerging researchers subcommittee rachel j. hinrichs, ms, mslsa, michelle knecht, msis, mslb, priscilla l. stephenson, msls, msed, ahip, fmlac ahealth sciences librarian, indiana university, university library, indianapolis, in rhinrich@iu.edu bsenior medical librarian and head of medical health sciences collection and user services department, florida atlantic university schmidt college of medicine, boca raton, fl kebam@health.fau.edu cretired, james a. haley veterans’ hospital, tampa, fl, pstephenson9116@yahoo.com cite as: hinrichs rj, knecht m, and stephenson pl. editorial board updates: emerging researchers subcommittee. hypothesis. 2025;37(2). doi:10.18060/28750 hinrichs, knecht, stephenson. all works in hypothesis are licensed under a cc by-nc 4.0 deed attribution-noncommercial 4.0 international. authors own copyright of their articles appearing in hypothesis. readers may copy articles without permission of the copyright owner(s), as long as the author(s) are acknowledged in the copy, and the copy is used for educational, not-for-profit purposes. for any other use of articles, please contact the copyright owner(s). 1 mailto:rhinrich@iu.edu mailto:kebam@health.fau.edu mailto:pstephenson9116@yahoo.com https://creativecommons.org/licenses/by-nc/4.0/ hypothesis, vol. 37, no. 1, 2025 call for proposals, hypothesis feature issue: emerging lis researchers hypothesis is an international peer-reviewed and open access journal that provides a venue for original research and research-related content, including research studies and reflections on the process. the editorial board is seeking submissions from novice researchers, students, and newer authors for a special features issue. your project doesn’t need to be complete to make a submission. submissions will be accepted in the following categories: brief report: an extended abstract on your research project or program limited to 500 words. (this is a good option for describing a work in progress.) conference abstracts: did you present at a conference recently? submit your conference abstract for publication and editorial peer review. (this is a good option if you do not have time to write a full report.) methods moment: describe a study technique, method, or analysis. (this is a good option for describing the method you used in your research project.) failure: we learn most from our toughest moments. share your research failure and, in the process, explain the lessons learned or how you or your library has changed for the better. (this is a good option if your project or program didn’t turn out the way you hoped!) research: share the results of your case report, case study, survey, etc. (this is a good option if your project is complete, and you are ready to publish the results.) voices of experience: this is your chance to share your story about a project, program, or class. it should include a bit of the process (methods) you used, but the focus is on what you learned and sharing tips in the takeaways section of what others could learn from your experience. (this is a good option for a more reflective piece.) you may reach out to the hypothesis editor, margaret a. hoogland, at hypothesisj@protonmail.com with questions and/or possible mentorship. 2 https://journals.indianapolis.iu.edu/index.php/hypothesis/briefreport? https://journals.indianapolis.iu.edu/index.php/hypothesis/confabs https://journals.indianapolis.iu.edu/index.php/hypothesis/methodsmoment? https://journals.indianapolis.iu.edu/index.php/hypothesis/failure? https://journals.indianapolis.iu.edu/index.php/hypothesis/research? https://journals.indianapolis.iu.edu/index.php/hypothesis/voices? mailto:hypothesisj@protonmail.com peer reviewed article hypothesis, vol. 37, no. 1, 2025 rethinking systematic review assignment design in graduate health science education from librarians’ perspectives laura lipke, ms, mlis, ahipa, carrie price, mlsb ahealth science librarian, university libraries, binghamton university, binghamton, ny, https://orcid.org/0000-0003-3653-5843 , llipke@binghamton.edu bbiomedical librarian, nih library, office of research services, od, national institutes of health, bethesda, md, https://orcid.org/0000-0003-4345-3547 , carrie.price@nih.gov cite as: lipke l and price c. rethinking systematic review assignment design in graduate health science education from librarians’ perspectives. hypothesis. 2025;37(1). doi:10.18060/28463 lipke and price. all works in hypothesis are licensed under a cc by-nc 4.0 deed attribution-noncommercial 4.0 international. authors own copyright of their articles appearing in hypothesis. readers may copy articles without permission of the copyright owner(s), as long as the author(s) are acknowledged in the copy, and the copy is used for educational, not-for-profit purposes. for any other use of articles, please contact the copyright owner(s). 1 https://orcid.org/0000-0003-3653-5843 https://orcid.org/ mailto:llipke@binghamton.edu https://orcid.org/0000-0003-4345-3547 https://orcid.org/ mailto:carrie.price@nih.gov https://creativecommons.org/licenses/by-nc/4.0/ https://creativecommons.org/licenses/by-nc/4.0/ peer reviewed article hypothesis, vol. 37, no. 1, 2025 abstract background: this article examines the problematic phenomenon of faculty assigning graduate health science students systematic reviews as semester-based assignments while expecting a health science librarian to be a willing support system for those students. despite published conduct and reporting guidelines establishing that systematic reviews require a team, time, and methodological expertise, some faculty still turn to full systematic review assignments in the classroom. the authors propose applying cognitive load theory and chunking the systematic review process into manageable steps, allowing both faculty and students to better understand the required methodologies and to enhance educational outcomes. experience: the authors have often been invited to visit classes where faculty have required students to complete a full systematic review. these assignments often result in frustration among faculty, librarians, and students stemming from the faculty’s limited experience with the methodology and students feeling overwhelmed by the process. while the authors’ experiences with suggesting the adaptable assignments to faculty is limited at the time of this publication, it is the hope of the authors that by sharing these concepts with other health science librarians, the trend of adapting more appropriate review assignments will expand. discussion: through applying cognitive load theory and chunking principles to simplify the systematic review process, the authors propose approaches to systematic review research methods that can improve the educational process, ameliorate faculty workload, and enhance student learning outcomes. as a result, students will be more prepared in future research endeavors. challenges include faculty adoption and acceptance of different approaches to systematic review assignments, and a further burden on the librarians who support these types of assignments. takeaways: the authors aim to raise faculty awareness of proper systematic review methodologies by offering alternative assignments that enhance student learning outcomes and alleviate the librarian’s teaching burden. initial attempts to promote these adaptations have shown promise, with some faculty successfully revising their syllabi and capstone projects to align with the suggested modifications. future research will include webinars for faculty on systematic review methodologies, followed by ongoing evaluations of how these modifications impact the health science curriculum. background "i need you to visit my research methods class this fall. my students are going to be doing a systematic review and we need your help." what health science librarian has not received a similar invitation from faculty while feeling secretly resentful and frustrated at having to support a methodology that is not suited to the classroom? many health science programs encourage students to learn the skills they will need to perform research within their profession. to instill these research skills, many health science graduate program faculty assign systematic reviews as a semester-based individual or group project, and subsequently may invite a librarian to the classroom for support. academic librarians, especially health science librarians, are increasingly viewed as research collaborators given 2 peer reviewed article hypothesis, vol. 37, no. 1, 2025 their skills in information organization, database structure, and searching1-4. it is generally well known among librarians, however, that systematic reviews require an in-depth knowledge of the complex standardized methodology in order to produce the quality of evidence necessary for the domains of medicine and health. the authors have observed a lack of faculty knowledge about the full systematic review methods process. faculty inability to mentor students through a systematic review assignment impedes student learning outcomes and student success. by applying the educational principles of cognitive load theory (clt) and chunking, which entails breaking down large tasks into smaller ones, the authors recommend several alternative methods to provide opportunities for faculty to implement learning about the standardized methodology required of systematic reviews. enhanced knowledge could lead to positive outcomes for everyone involved: the faculty, the students, and the librarian. systematic reviews are often considered the highest quality of evidence due to their adherence to rigorous evolving published conduct and reporting standards5. consequently, reviews that do not adhere to the strict methodology cannot be considered trustworthy6-14. a recent flourish of published systematic reviews has demonstrated a general lack of knowledge regarding the rigorous methodology and reporting standards throughout the healthcare and medical fields8,9,11,15–19.this wave of inaccurately conducted and reported reviews has led to unreliable evidence, inaccurate estimates of treatment effectiveness, misleading conclusions, reduced applicability, and significant research waste11. similarly, due to the recent evolution of graduate education programs in the health science fields, many faculty report a lack of knowledge and skills needed to advise in such projects20–29. this shortfall of experience is supported in other studies that have also found a limited community of researchers who thoroughly understand the methodology of systematic reviews, and a majority of faculty who lack training in this methodology22,26–28,30,31. in scholarship arguing for and against systematic reviews as graduate projects, most authors agree that the lack of faculty knowledge and experience with these reviews is a major barrier26–28,31,32. scientific scholarship promotes skills of inquiry, analysis, critical thinking, and communication among healthcare professionals21. promoting research skills in the classroom provides students with the opportunity to learn about methodologies, to develop database search strategies, to appraise research, to synthesize evidence, and to translate knowledge into practice21,23,24,33,34. while all of these skills can be gained by completing a systematic review, this type of literature review introduces challenges, due to the complexity of the standardized methodology, the possible lack of mentor knowledge or experience, time constraints, and the group nature of such projects6,7,14,35,36. the assignment of systematic reviews has led to controversies in the scholarship supporting and opposing such assignments in health science graduate programs due to the lack of experienced and knowledgeable faculty to act as supervisors20,22,26–28,30-32,37. this has also resulted in mixed feedback from students and faculty who describe the work as demanding in terms of the time and knowledge required24. additionally, the final product of these review assignments rarely aligns with the standardized conduct and reporting standards. this significantly reduces the effectiveness of the overall exercise since the final review: 1) is most likely not publishable, and 2) has incorrectly influenced the students’ understanding of how a systematic review is conducted12,16,17,19. the ability to comprehend and apply the complex methodology of a systematic review can be the primary hurdle for students assigned this type of project. the process of learning this 3 peer reviewed article hypothesis, vol. 37, no. 1, 2025 methodology according to the standardized methods presented by regarded organizations such as the cochrane collaboration or jbi, compounded by the pressures of graduate research assignments, can lead to cognitive overload for students and significantly impact learning outcomes6,14,38. the basic premise of cognitive load theory (clt), “...is that human cognitive processing is heavily constrained by our limited working memory which can only process a limited number of information elements at a time.39” this theory postulates that when a learner is presented with an overload of new complex information, the working memory is unable to process the information and this leads to cognitive overload38,40,41. clt proposes an instructional design method known as “chunking” to reduce the cognitive load associated with learning new information. chunking reduces the complexity of an overall task, groups relevant information into an organized representation, allows time between learning episodes, and optimizes successful completion by advancing students through a simple to complex assignment process38,40–44. by chunking the aspects of learning the methodology process of a systematic review, the learner is able to merge prior knowledge into a cohesive unified framework and develop the expertise required to complete evidence-based research45,46. chunking can be incorporated into the instructional design of teaching the systematic review process, which can help reduce the cognitive overload reported by students. by chunking the steps of a systematic review, the student will be able to build upon previously learned research skills and progress toward the more complex elements of a review with more confidence and likelihood of success. the chunked assignments presented in this article will introduce students and faculty to the skills and methodology required of a full systematic review in steps, which can and should replace a full systematic review assignment based on the authors’ expertise with the methodology and their experience in a multitude of health science classrooms. experience the authors of this paper are academic health science librarians who have provided research support to undergraduate, graduate, and practicing healthcare professionals. librarian and information specialists, can specialize in various roles to support systematic reviews; inclusion of these specialists often leads to higher quality searching and reporting1,2,4. the authors have been invited to numerous graduate classes where “systematic reviews” have been assigned. often, the librarians are then expected to teach some kind of in-class instructional session and create learning materials or an instructional narrative to support the students in completing a semester-based systematic review. however, as noted earlier, full systematic reviews are not suitable for graduate students due to the guideline requirements, and the timeline of a systematic review is also not feasible within a typical semester. anecdotal experience of the authors and informal discussions with their librarian colleagues illustrate that these scenarios can often result in feelings of frustration for all parties. as a case in point, one of the authors was invited to a class where the students were required to complete a systematic review. the faculty requested that the prisma checklist and the prisma flow diagram be demonstrated and explained as part of the library instruction session. however, when the librarian demonstrated the prisma flow diagram, which captures total search results, duplicates, included records, and excluded records as they are moved through the systematic review screening process, the students became extremely distraught. upon delving deeper, the author realized that the students were not doing a 4 peer reviewed article hypothesis, vol. 37, no. 1, 2025 comprehensive literature review, and neither were they planning to document their searches. rather, the students had been directed by the faculty to handpick articles they felt were relevant to their topic, making the application of the prisma flow diagram and the prisma checklist impossible. the faculty member failed to realize this disconnect between their instruction and the assignment. the author left the classroom feeling distressed. due to multiple experiences like these, the authors wish to make recommendations that are better suited to the classroom, as well as to provide other academic health science librarians with examples to present when faced with similar circumstances. the assignments presented here were developed with the cochrane collaboration and jbi systematic review conduct guidelines, as well as the prisma reporting statement and its extensions, in mind6,14,18. even though the authors do not recommend that students be assigned or attempt to complete a full systematic review within a semester, they still believe that it is possible to provide students with research experiences that will assist them to produce true systematic reviews in the future6,14,35. the authors have applied their experience and knowledge working with graduate health science students and faculty to address the hurdles often presented with systematic review assignments and subsequent librarian collaborations. the listed assignments include a brief description, learning outcomes, and are presented in order of increasing complexity. the authors also recommend that these assignments be completed as a group. the systematic review process requires a team of experts, so by completing these assignments in a group setting, the students will learn how the team process reduces bias within the systematic review method6,14. the suggested assignments can be incorporated into a variety of pedagogical techniques as necessary and appropriate for the classroom environment. faculty and health science librarians may adapt these chunked assignments in the same way that they may have adapted a full systematic review assignment for the classroom; that is, with a rubric and curriculum to support the student, tailored to the needs of the discipline. because these assignments are more feasible than a full systematic review, the authors feel that faculty who adopt and adapt these practices may have more fruitful collaborations with librarians and better student academic outcomes. most health science librarians who have supported evidence synthesis are familiar with the ideas around literature review typologies, topic and protocol development, systematic searching and screening, risk of bias/critical appraisal, data extraction, synthesis, and peer review of academic manuscripts, enabling them to support such assignments more actively. table a. chunked assignment summaries assignment summary narrative review a narrative review allows students to learn the literature review process without the necessity of a comprehensive literature search, critical appraisal, data extraction, a team, or time. systematic review protocol a systematic or scoping review protocol contains many of the same reporting elements as the final review but can be performed as an independent exercise within a shorter amount of time. 5 peer reviewed article hypothesis, vol. 37, no. 1, 2025 systematic search and search methods conducting and reporting a documented and reproducible literature search helps students understand the foundation of a systematic literature review. risk of bias/critical appraisal critically appraising or determining the risk of bias of a set of relevant research requires that students become familiar with the standardized instruments, research design, and advanced critical thinking skills. data extraction having students extract data from relevant published studies requires them to adhere to pre-defined criteria and manage research data in a transparent way. qualitative synthesis qualitatively describing how the data answers the research question allows students to determine the strengths and weaknesses of the data, identifies evidence gaps, and compare the findings with existing scholarship. peer review peer review is a process all scholars will experience in their career, whether it is in the form of receiving or providing this feedback. conducting a peer review of another student’s work allows students to learn to provide gracious yet constructive feedback. the peer review can be completed with any of the alternative assignments suggested. systematized review while a systematized review does not meet the strict criteria for conducting and reporting a systematic review, it can work as an introduction to the process and be performed by an individual or group within a shorter time frame. while pairing assignments together is not necessary, suggested pairings have been provided dependent on the scope and goals of the research methods course. narrative review assignment a narrative literature review is an integrated analysis of the existing literature used to summarize a body of scholarship, draw conclusions about a topic, identify research gaps, justify the need for the research question, and develop a broad overview of a topic. by understanding the current state of the literature, one can show how new research fits into the larger corpus of literature. the purpose of a narrative literature review is to explain the background of the research on a topic; identify major themes, concepts, and scholars; and highlight critical gaps, points of disagreement, or flawed approaches. this type of literature review can also suggest new areas of research. a narrative literature review is often the opening section of many published articles and is a necessary pedagogical step to introduce the purpose of a systematic review. the complexity of the narrative literature review assignment can be modified depending on the course objectives, and steps in the process can be further chunked to meet timelines throughout the semester. 6 peer reviewed article hypothesis, vol. 37, no. 1, 2025 students can use a topic of their choice or a current assignment to complete a narrative literature review, which can be graded according to a rubric. a collaborating librarian can assist with topic development, resource selection, searching, and records management. learning objectives for the narrative review assignment include: • students will develop a research question. • students will search for literature to answer the research question. • students will distinguish appropriate databases for their topic of inquiry. • students will format and construct a literature review. • students will synthesize themes. • students will accurately cite sources. systematic review protocol assignment one of the initial steps for completing a systematic review is to develop and register a protocol. the purpose of a protocol is to provide a detailed plan for the review project, reduce the risk of bias within the review, and establish providence of the research question to be evaluated47. students can learn how to plan the steps and the workflow of a systematic review by following the prisma-protocol reporting guideline47. students may use a previously developed topic, or a topic related to a class assignment to complete the protocol assignment. the prisma-protocol checklist provides instructors with a grading rubric for the assignment47. a collaborating librarian can discuss different review types, help students plan and potentially execute their literature search, and support student knowledge on protocol development. learning objectives for the systematic review protocol assignment include: • students will develop a research question. • students will summarize the steps and methodology of the research process of a systematic review. • students will recognize and identify the resources available to develop a protocol according to established guidelines. systematic search and search methods assignment developing an appropriately sensitive and specific search strategy to ensure that the results are comprehensive for a systematic review is a complex task which often involves days of work for health science librarians. this process involves learning how to determine which search terms and databases are applicable to the research topic, identifying the appropriate controlled vocabulary within the chosen databases, and translating the search strategy from one database to another to ensure a systematic search6,14. in addition to the development and 7 peer reviewed article hypothesis, vol. 37, no. 1, 2025 translation of the search strategy, a systematic review requires that the search strategy be documented in enough detail to be reproducible48. documentation of the search strategy is done according to the prisma-search (prisma-s) extension standards and recorded in a prisma flow diagram13,48. developing a systematic and comprehensive search strategy is a major factor that distinguishes a systematic review from other types of reviews, and arguably an important foundation of the review49. students will conduct a comprehensive literature search in three to five bibliographic literature databases and report the search with the elements of prisma-s extension48. it is recommended to invite a health science librarian or information specialist to be embedded in the class to advise on searching comprehensively in academic databases for scholarly, peer-reviewed literature and documenting the search2,4. a collaborating librarian can assist students with selecting the best databases to search and provide tips for creating documented, reproducible searches. learning objectives for the systematic search and search methods assignment: • students will develop a research question. • students will demonstrate learned search skills through the successful retrieval of evidence. • students will identify optimal databases for evidence-based practice in health science. • students will understand how to document a reproducible search strategy. this assignment is best paired with the peer-review assignment. risk of bias assessment/critical appraisal assignment critical appraisal “is the process of carefully and systematically examining research to judge its trustworthiness, and its value and relevance in a particular context. it is an essential skill for evidence-based practice because it allows healthcare professionals to find and use research evidence reliably and efficiently.50” critical appraisal of the evidence to be synthesized is a vital step in the methodology of a systematic review because it can highlight the risk of bias in results, which can affect the quality of the evidence incorporated6,14. the process involves becoming familiar with standardized instruments and critical appraisal checklists, such as those from jbi and the critical appraisal skills programme (casp), learning the whys and hows of bias assessment and critical appraisal, and using critical thinking to determine the validity, reliability, and relevance of a study in relation to the research question12,50,51. start by providing the students with a set of key studies that can be appraised with the use of checklists or other tools. collaborating librarians can then assist with determining the value and quality of publications through recognized information literacy techniques. learning objectives for the risk of bias assessment/critical appraisal assignment: • students will use critical thinking and appraisal skills. • students will appraise critically through the use of the standardized checklists. 8 peer reviewed article hypothesis, vol. 37, no. 1, 2025 • students will identify markers of validity, reliability, transparency, and relevance of research methods. • students will distinguish methodological quality within a scientific study. data extraction assignment data extraction is the first step in the synthesis process of a systematic review. this step requires that the review team adhere strictly to the inclusion and exclusion criteria identified by the review authors and extract data according to those elements. for this assignment, the students may work independently and then come together as a team to discuss their findings. additional information regarding the data extraction process can be found in chapter 5 of the cochrane handbook for systematic reviews6. start by giving students a collection of predetermined key studies on a chosen topic. provide adaptable templates that students will use to extract data elements of the studies such as population, participant demographics, intervention, methodology, and more. collaborating librarians can provide classroom instruction in records management. learning objectives for the data extraction assignment: • students will design a data extraction process for a systematic review. • students will describe the process of extracting specific criteria. • students will differentiate between data collection, data management, and process documentation. this assignment is best paired with the qualitative synthesis assignment. qualitative synthesis assignment qualitative synthesis of the data extracted is a required aspect of a systematic review. this process is completed regardless of whether or not a quantitative meta-analysis is also done. the qualitative synthesis is a narrative thematic analysis of the themes found within the data and usually includes a summarization of the general characteristics that were identified along with any relationships, patterns, heterogeneity, and degrees of consistency that were observed. this narrative will also describe how the data answers the research question, determine the strengths and weaknesses of the data, identify evidence gaps, and compare the findings with known scholarship. additional information regarding the qualitative evidence synthesis process can be found in chapter 21 of the cochrane handbook for systematic reviews6. students will select five to ten key research articles based on their research question. they will then provide a detailed summary of these articles using an adaptable template or form for identifying themes and categories. collaborating librarians can assist students with planning and executing a literature search and finding other qualitative syntheses as model publications. learning objectives for the qualitative synthesis assignment: 9 peer reviewed article hypothesis, vol. 37, no. 1, 2025 • students will develop a research question. • students will differentiate between analyzing and synthesizing qualitative data. • students will apply critical thinking skills. • students will develop advanced writing skills. • students will describe the qualitative synthesis process of a systematic review. this assignment is best paired with the data extraction assignment. peer-review assignment peer review is a process all scholars will experience in their career, whether it is in the form of receiving or providing this feedback. many of the previously mentioned assignments would be appropriate for a peer review assignment. the systematic review protocol assignment and the systematic search and search methods assignment could be reviewed using the prisma-protocol and the prisma-search checklists14,47,48. an initial literature review can also be a source of a peer review assignment. for this assignment, students would benefit from faculty-developed examples of constructive review feedback along with an explanation of how the peer-review process works, and why it is important in health science scholarship. after the students have completed either the narrative review, systematic review protocol, or the systematic review and search methods assignments noted above, they can pair up, exchange assignments, and conduct a peer review statement based on best practices and readings around the peer review process. alternatively, faculty or the librarian(s) can provide students with published research articles from the health science literature and have them conduct a “post-publication” peer review. grade according to a rubric. while librarians do not often play a role in the peer review process, they may be able to create learning materials around scholarly communication and publication ethics3. learning objectives for the peer review assignment: • students will interpret necessary requirements of existing reporting standards through peer review. • students will examine how peer review supports the scholarly process. • students will identify how to optimally document the systematic review process. this assignment is best paired with the narrative review, protocol, or the search methods assignments. systematized review assignment the systematized review is an introductory review process for students52. while a systematized review does not meet the strict criteria for conducting and reporting a systematic review, it can work as an introduction to the process. this type of review can also be 10 peer reviewed article hypothesis, vol. 37, no. 1, 2025 completed by a single researcher or student whereas a systematic review requires a team of experts. as described by grant and booth in 2009, it is an “attempt to include elements of systematic review process while stopping short of systematic review. . . [and]. . . typically conducted as a postgraduate student assignment.52” have individuals or groups of students conduct a thorough, reproducible literature review with key elements, such as the protocol, bias assessment, and formalized data extraction, excluded. once again, collaborating librarians can assist through teaching review typology education, topic development, comprehensive searching, records management, and more. learning objectives for a systematized review assignment: • students will understand the methodology needed for a systematic review. • students will develop an appropriate research question. • students will determine appropriate inclusion and exclusion criteria. • students will determine which databases and resources are suitable for their research question. • students will develop a systematic search strategy. • students will document a transparent and reproducible search strategy. • students will critically appraise scholarly evidence. • students will synthesize evidence. • students will report the bulk of a systematic review according to the prisma guidelines. discussion this paper presents a solution grounded in cognitive load theory and chunking to address the critical need for structured and supportive methods in teaching systematic review methodology. the previously identified deficiencies in faculty knowledge and mentorship, and the resulting cognitive overload among students, necessitates innovative solutions. by applying principles of clt and chunking, the authors propose modified instructional methods to alleviate cognitive burden and enhance student learning outcomes. the division of the systematic review process into manageable chunks not only facilitates skill acquisition by students, but also empowers them to navigate the complexities of evidence synthesis with confidence. each step of the review process can be parsed into its own separate assignment, which provides the students with an opportunity to learn the details of that step and how it relates to the systematic review process as a whole. the presented assignments, developed in alignment with established systematic review standards and guidelines, offer a structured pathway for faculty and students alike, fostering the development of competent and proficient researchers. ultimately, by advocating for structured learning experiences, the authors seek to bridge the gap between theoretical knowledge and practical application in systematic review education. 11 peer reviewed article hypothesis, vol. 37, no. 1, 2025 by adopting chunked systematic review assignments and collaborating with librarians who have systematic review expertise, graduate health science programs can better equip students with the necessary skills to produce high-quality evidence, thus advancing evidence-based practice and future decision-making. takeaways while the authors of this paper are limited in their ability to assign these exercises, they aspire to promote faculty awareness of standardized, best practice systematic review methodology while providing them with alternative assignments that are pedagogically sound. adopting this approach can potentially result in enhanced research skill development for faculty and students while reducing the burden on librarians who have been asked to teach in ways that are inconsistent with accepted and standardized methods. although the authors’ experiences with the suggestions in this manuscript and the application of these modified assignments has been limited at this point, the few attempts to promote these adaptations to health science faculty have shown promise. examples of such promise include witnessing faculty adapt syllabi with individual assignments mimicking the above examples and the revision of previously assigned systematic and scoping reviews for capstone projects. as health science librarians are fully aware, how each individual faculty is approached regarding modifications to their assignments is dependent on the relationship between the faculty member and the librarian. it is the experience of the authors that librarians who wish to make these suggestions should focus on the faculty they already have strong relationships with first, since they are more likely to be receptive. additionally, librarians might consider offering regular faculty-focused workshops on review typologies and evidence synthesis methodologies. when faculty are made aware of the standardized methods regarding these review types and the constraints they present as student assignments, they may be more likely to adapt their approach. future research is planned to quantifiably evaluate faculty-focused webinars which will provide attendees with instruction regarding the proper methodology of a systematic review, and an introduction to the modified assignments described in this manuscript. this research will be followed by an annual evaluation of the modifications noted in the health science curriculum supported by the authors. acknowledgments we would like to thank yolanda l. jones, nih library editing service, for reviewing the manuscript. references 1. spencer aj, eldredge jd. roles for librarians in systematic reviews: a scoping review. j med libr assoc. 2018 jan;106(1):46–56. 2. rethlefsen ml, farrell am, osterhaus trzasko lc, brigham tj. librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews. j clin epidemiol. 2015 jun;68(6):617–26. 12 peer reviewed article hypothesis, vol. 37, no. 1, 2025 3. rethlefsen ml, schroter s, bouter lm, moher d, ayala ap, kirkham jj, et al. improving peer review of systematic reviews by involving librarians and information specialists: protocol for a randomized controlled trial. trials. 2021 nov 11;22(1):791. 4. pawliuk c, cheng s, zheng a, hal siden h. librarian involvement in systematic reviews was associated with higher quality of reported search methods: a cross-sectional survey of authors from vancouver, bc. j clin epidemiol. 2023 dec 8;111237. 5. gopalakrishnan s, ganeshkumar p. systematic reviews and meta-analysis: understanding the best evidence in primary healthcare. j family med prim care. 2013;2(1):9–14. 6. higgins jpt, chandler j, cumpston m, li t, page m, welch v, editors. cochrane handbook for systematic reviews of interventions version 6.4. 6.4. 2023. 7. kolaski k, romeiser logan l, ioannidis jpa. principles for good scholarship in systematic reviews. developmental medicine & child neurology [internet]. 2023 [cited 2023 nov 8];n/a(n/a). available from: https://onlinelibrary.wiley.com/doi/abs/10.1111/dmcn.15719 8. moore a. red for danger in systematic reviews? eur j hosp pharm. 2021 nov 1;28(6):299–300. 9. moore ra, derry s, mcquay hj. fraud or flawed: adverse impact of fabricated or poor quality research. anaesthesia. 2010 apr;65(4):327–30. 10. nguyen py, kanukula r, mckenzie je, alqaidoom z, brennan se, haddaway nr, et al. changing patterns in reporting and sharing of review data in systematic reviews with meta-analysis of the effects of interventions: cross sectional meta-research study. bmj. 2022 nov 22;379:e072428. 11. pussegoda k, turner l, garritty c, mayhew a, skidmore b, stevens a, et al. systematic review adherence to methodological or reporting quality. systematic reviews. 2017 jul 19;6(1):131. 12. rethlefsen ml, brigham tj, price c, moher d, bouter lm, kirkham jj, et al. systematic review search strategies are poorly described and not reproducible: a cross-sectional meta-research study [internet]. medrxiv; 2023 [cited 2023 nov 15]. p. 2023.05.11.23289873. available from: https://www.medrxiv.org/content/10.1101/2023.05.11.23289873v1 13. page mj, mckenzie je, bossuyt pm, boutron i, hoffmann tc, mulrow cd, et al. the prisma 2020 statement: an updated guideline for reporting systematic reviews. bmj. 2021 mar 29;n71. 14. aromataris e, lockwood c, porritt k, pilla b, jordan z, editors. jbi manual for evidence synthesis [internet]. jbi; 2024 [cited 2023 nov 20]. available from: https://synthesismanual.jbi.global 15. fontelo p, liu f. a review of recent publication trends from top publishing countries. syst rev. 2018 sep 27;7(1):147. 13 https://onlinelibrary.wiley.com/doi/abs/10.1111/dmcn.15719 https://www.medrxiv.org/content/10.1101/2023.05.11.23289873v1 https://synthesismanual.jbi.global peer reviewed article hypothesis, vol. 37, no. 1, 2025 16. ioannidis jpa. the mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. milbank q. 2016 sep;94(3):485–514. 17. smith gd, aveyard h, jane n, penny ki. improving the conduct and reporting of meta-analyses. journal of advanced nursing (john wiley & sons, inc). 2023 sep;79(9):3186–8. 18. torres g, ledbetter l, cantrell s, alomo arl, blodgett tj, bongar mv, et al. adherence to prisma 2020 reporting guidelines and scope of systematic reviews published in nursing: a cross-sectional analysis. j nurs scholarsh. 2024 mar 30; 56(4):531-541. 19. uttley l, quintana ds, montgomery p, carroll c, page mj, falzon l, et al. the problems with systematic reviews: a living systematic review. j clin epidemiol. 2023 apr 1;156:30–41. 20. cañón-montañez w, rodríguez-acelas al. contributions of systematic reviews and meta-analyses to nursing education, research, and practice. aquichan [internet]. 2021 dec [cited 2024 feb 2];21(4). available from: http://www.scielo.org.co/scielo.php?script=sci_abstract&pid=s165759972021000402143&lng=en&nrm=iso&tlng=en 21. cleary m. essentials of building a career in nursing research. nurse res. 2016 jul;23(6):8–12. 22. dotto l, t. o. lemes l, o. spazzin a, sousa ytcs, pereira gkr, bacchi a, et al. acceptance of systematic reviews as master/phd theses in brazilian graduate programs in dentistry. j evid based med. 2020;13(2):125–9. 23. einarsen ka, giske t. nursing students’ longitudinal learning outcomes after participation in a research project in a hospital. international practice development journal. 2019 may;9(1):1–10. https://hdl.handle.net/11250/3041511 24. lesley ml. facilitating students’ success with their own research projects. nurse educ. 2011 jun;36(3):107. 25. milner k, zonsius m, alexander c, zellefrow c. doctor of nursing practice project advisement: a roadmap for faculty and student success. j nur educ. 2019 dec;58(12):728–32. 26. orta r, messmer pr, valdes gr, turkel m, fields sd, wei cc. knowledge and competency of nursing faculty regarding evidence-based practice. j contin educ nur. 2016 sep;47(9):409–19. 27. puljak l, sapunar d. acceptance of a systematic review as a thesis: survey of biomedical doctoral programs in europe. syst rev. 2017 dec 12;6(1):253. 28. ten ham-baloyi w, jordan p. systematic review as a research method in post-graduate nursing education. health sa gesondheid. 2016 dec 1;21:120–8. 14 http://www.scielo.org.co/scielo.php?script=sci_abstract&pid=s1657-59972021000402143&lng=en&nrm=iso&tlng=en http://www.scielo.org.co/scielo.php?script=sci_abstract&pid=s1657-59972021000402143&lng=en&nrm=iso&tlng=en https://hdl.handle.net/11250/3041511 peer reviewed article hypothesis, vol. 37, no. 1, 2025 29. wissinger cl. is there a place for undergraduate and graduate students in the systematic review process? j med libr assoc. 2018 apr 5;106(2):248–50. 30. muraraneza c, mtshali n, bvumbwe t. challenges in postgraduate research supervision in nursing education: integrative review. nurse educ today. 2020 jun 1;89:104376. 31. schnall r. national institute of health (nih) funding patterns in schools of nursing: who is funding nursing science research and who is conducting research at schools of nursing? j prof nurs. 2020 jan 1;36(1):34–41. 32. christian r, palokas m. systematic review methodology in graduate nursing education. jbi evid synth. 2018 mar;16(3):587. 33. ferreira r, sousa l, nobre c, nunes ac, fonseca c, ferreira ó, et al. the development of research skills in nursing postgraduate training. educ sci. 2022 feb;12(2):78. 34. menzies jc, emms k, valler t. developing research knowledge and capability in undergraduate nurses: evaluation of targeted placements. j res nurs. 2021 aug;26(5):408–24. 35. borah r, brown aw, capers pl, kaiser ka. analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the prospero registry. bmj open. 2017 feb 1;7(2):e012545. 36. price c. systematic review as class assignments? [internet]. covidence. 2022 [cited 2023 nov 8]. available from: https://www.covidence.org/blog/elementor-2112/ 37. sayers j, lopez v, howard pb, escott p, cleary m. the leadership role of nurse educators in mental health nursing. issues ment health nurs. 2015;36(9):718–24. 38. venkat mv, o’sullivan ps, young jq, sewell jl. using cognitive load theory to improve teaching in the clinical workplace. mededportal. 2020;16:10983. 39. sweller j, van merriënboer jjg, paas f. cognitive architecture and instructional design: 20 years later. educ psychol rev. 2019 jun 1;31(2):261–92. 40. smith ne, barbé t, randolph j. application of the cognitive load theory in prelicensure nursing education: a quantitative measurement focusing on instructional design. int j nurs educ scholarsh. 2022 jan;19(1):1–12. 41. van merriënboer jjg, sweller j. cognitive load theory in health professional education: design principles and strategies. med educ. 2010 jan;44(1):85–93. 42. coffman s, iommi m, morrow k. scaffolding as active learning in nursing education. teach learn nurs. 2023 jan;18(1):232–7. 43. mauldin b. a novel teaching strategy in nursing pharmacology: learning using cognitive load theory. nurs educ perspect. 2021 nov;42(6):e158–60. 15 https://www.covidence.org/blog/elementor-2112/ peer reviewed article hypothesis, vol. 37, no. 1, 2025 44. van nooijen cca, de koning bb, bramer wm, isahakyan a, asoodar m, kok e, et al. a cognitive load theory approach to understanding expert scaffolding of visual problem-solving tasks: a scoping review. educ psychol rev. 2024 jan 26;36(1):12. 45. mcgowan bs, reed jb, yatcilla jk. graduate student confidence following a for-credit systematic review course pilot. j med libr assoc. 109(2):323–9. 46. usher-smith ja, harrison h, dennison r, kelly s, jones r, kuhn i, et al. developing training and opportunities for students in systematic reviews. med educ. 2021;55(11):1333–4. 47. moher d, shamseer l, clarke m, ghersi d, liberati a, petticrew m, et al. preferred reporting items for systematic review and meta-analysis protocols (prisma-p) 2015 statement. syst rev. 2015 jan 1;4(1):1. 48. rethlefsen ml, kirtley s, waffenschmidt s, ayala ap, moher d, page mj, et al. prisma-s: an extension to the prisma statement for reporting literature searches in systematic reviews. syst rev. 2021 dec;10(1):1–19. 49. lefebvre c, glanville j, briscoe s, featherstone r, littlewood a, metzendorf mi, et al. chapter 4: searching for and selecting studies [internet]. cochrane training. 2024 [cited 2024 jan 12]. available from: https://training.cochrane.org/handbook/current/chapter-04 50. burls a. what is critical appraisal? [internet]. 2009 [cited 2023 nov 8]. available from: http://www.bandolier.org.uk/painres/download/whatis/what_is_critical_appraisal.pdf 51. casp checklists critical appraisal skills programme [internet]. casp critical appraisal skills programme. [cited 2023 nov 28]. available from: https://casp-uk.net/casp-tools-checklists/ 52. grant mj, booth a. a typology of reviews: an analysis of 14 review types and associated methodologies. health info libr j. 2009 jun;26(2):91–108. 16 https://training.cochrane.org/handbook/current/chapter-04 http://www.bandolier.org.uk/painres/download/whatis/what_is_critical_appraisal.pdf https://casp-uk.net/casp-tools-checklists/ hypothesis, vol. 36, no. 2, 2024 unpacking decisions rendered by the hypothesis editor margaret a. hoogland, mls, ahipa aassociate professor and clinical medical librarian, university libraries, the university of toledo, toledo, ohio, https://www.orcid.org/0000-0002-9932-3605 , margaret.hoogland@utoledo.edu cite as: hoogland ma. unpacking decisions rendered by the hypothesis editor. hypothesis. 2024;36(2). doi:10.18060/28465. hoogland. all works in hypothesis are licensed under a cc by-nc 4.0 deed attribution-noncommercial 4.0 international. authors own copyright of their articles appearing in hypothesis. readers may copy articles without permission of the copyright owner(s), as long as the author(s) are acknowledged in the copy, and the copy is used for educational, not-for-profit purposes. for any other use of articles, please contact the copyright owner(s). 1 https://www.orcid.org/0000-0002-9932-3605 https://orcid.org/ mailto:margaret.hoogland@utoledo.edu https://creativecommons.org/licenses/by-nc/4.0/ https://creativecommons.org/licenses/by-nc/4.0/ hypothesis, vol. 36, no. 2, 2024 this editorial breaks down the logic used by the hypothesis editor when selecting a verdict for each submission: verdict 1: rejection without peer review (aka desk reject) within 3-7 business days, the editor notifies the author(s), because the submission does not meet the aim or scope of the journal.1 hypothesis “failure” submissions, perhaps because few journals offer this category, attract the most out of scope articles. a well-written manuscript that is poorly organized, utilizes an incorrect study design, or contains a flawed data analysis could also be rejected at this stage.2 hypothesis is a community journal committed to working with first time authors and emerging researchers. unless a manuscript is completely out of scope, the editor will contact the author(s) instead of using the desk reject option. verdict 2: rejection outright rejection occurs rarely with hypothesis, but this editor may render this verdict for the following reasons: 1. out of scope • as written, the manuscript would not appeal to readers of hypothesis and the author(s) choose not to revise the manuscript. 2. not innovative • before making this decision, the editor will make author(s) aware of other studies related to their research topic, encourage them to read the article(s), and revise the manuscript to reflect these findings. if the author(s) remain unwilling to incorporate the new articles or adjust the manuscript, the editor could suggest changing the manuscript to a different submission category and sending it out for a second round of peer review. if none of these options are successful, the editor must reject the manuscript. 3. author decision(s), which is registered as a rejection in the system, usually fall into two categories: • author(s) choose to withdraw the manuscript from the journal. • author(s) view the requested revisions as unacceptable, or feel that they completely change the focus of the manuscript so they decide not to move forward with revising the manuscript. when reviewing author decision requests, this editor will usually suggest scheduling a meeting to discuss the submission, the requested revisions, and determine a workaround so that the author might consider submitting a revised manuscript. if the author(s) is not amenable to these suggestions, the editor will respect their decision and decline the submission. 2 hypothesis, vol. 36, no. 2, 2024 verdict 3: request revisions a request for revisions is a common occurance and should not discourage the author(s) – this editor renders this verdict 90% of the time. instead, they should read peer reviewer, associate editor, and editor suggestions and seize the opportunity to expand upon or clarify parts of your manuscript. before making changes that do not directly respond to comments from peer reviewers, the associate editor, or the editor, author(s) should consult with the editor or associate editor. if peer reviewer comments disagree with journal submission guidelines, check with the editor before putting substantial time into making revisions. the editor’s deadline for revised submissions varies, but the goal is to give the author(s) a minimum of four-to-six weeks to make the requested edits. author(s) who do not complete revisions by the next issue consideration deadline are given one additional chance to submit to the following issue. if, after 12 months, the author(s) do not provide a revised manuscript or communicate an intent to continue, this editor will decline the submission.2,3 if the author(s) have questions or serious disagreements with the review comments, they may choose to not submit a revised manuscript. before making this decision, the editor hopes the author(s) would share their concerns and meet with the editor to see if they can reach an agreement regarding the requested revisions. verdict 4: revisions requested, manuscript will undergo a second round of peer review at times, the editor might read peer reviewer comments, consult with an associate editor, and decide the manuscript is a better fit for a different submission category. in these cases, the manuscript will undergo a second round of peer review. the editor or associate editor might read the comments and recognize that the requested changes will substantially change the manuscript. in such cases, the revised manuscript would benefit from an additional round of peer review. this editor prefers to send the revised manuscript back to peer reviewers of the original manuscript under these circumstances. while this decision will frustrate authors and delay the publication of a manuscript, this editor can attest that undergoing two rounds of peer review does improve the quality of a manuscript.4 verdict 5: accepted with minor revisions a request for minor revisions means that the editor, associate editor, and peer reviewers suggest minimal adjustments to terminology, phrasing, additional citations, or an updated literature review. a request for minor revisions is a common response in scholarly publishing. when the author(s) receive this verdict from hypothesis, they do not need to submit a response to reviewer comments in addition to a revised and unblinded manuscript. verdict 6: accepted without revisions author(s) should be ecstatic to receive such a determination for a peer reviewed article! if the 3 hypothesis, vol. 36, no. 2, 2024 author(s) chooses to make revisions, the editor will review and usually accept the proposed changes before moving the manuscript to copy editing. acknowledgements i thank laura lipke and jessica d. gilbert redman, who read and shared comments with me, which improved the quality of this editorial. references 1. beugelsdijk s, bird a. how to avoid a desk reject: do’s and don’ts. j int bus stud [internet]. 2024 jun 17 [cited 2024 jul 12];s41267-024-00712–8. available from: https://link.springer.com/10.1057/s41267-024-00712-8 2. bordage g. reasons reviewers reject and accept manuscripts: the strengths and weaknesses in medical education reports. acad med j assoc am med coll. 2001 sep;76(9):889–96. 3. boruff j, kraft m, carroll aj. introducing the journal of the medical library association’s manuscript resubmission deadlines: creating accountability structures for our authors. j med libr assoc [internet]. 2024 may 22 [cited 2024 jun 4];112(2):64–6. available from: https://jmla.mlanet.org/ojs/jmla/article/view/1902 4. hoogland ma, natal g, wilmott r, keating cf, caruso d. gauging academic unit perceptions of library services during a transition in university budget models. evid based libr inf pract [internet]. 2024 jun 14 [cited 2024 jul 16];19(2):23–50. available from: https://journals.library.ualberta.ca/eblip/index.php/eblip/article/view/30379 4 https://link.springer.com/10.1057/s41267-024-00712-8 https://jmla.mlanet.org/ojs/jmla/article/view/1902 https://journals.library.ualberta.ca/eblip/index.php/eblip/article/view/30379 editorial reviewed article hypothesis , vol. 36, no. 2, 2024 mla 2024 annual meeting and jmla biannual research caucus research awards lindsay blake, edd, mlis, ahipa aclinical services coordinator, uams library, university of arkansas for medical sciences, little rock, arkansas, https://www.orcid.org/0000-0002-8234-8611 , leblake@uams.edu cite as: blake l. mla 2024 annual meeting and jmla biannual research caucus research awards. hypothesis. 2024;36(2). doi:10.18060/28301. blake. all works in hypothesis are licensed under a cc by-nc 4.0 deed attribution-noncommercial 4.0 international. authors own copyright of their articles appearing in hypothesis. readers may copy articles without permission of the copyright owner(s), as long as the author(s) are acknowledged in the copy, and the copy is used for educational, not-for-profit purposes. for any other use of articles, please contact the copyright owner(s). 1 https://www.orcid.org/0000-0002-8234-8611 https://orcid.org/ mailto:leblake@uams.edu https://creativecommons.org/licenses/by-nc/4.0/ https://creativecommons.org/licenses/by-nc/4.0/ editorial reviewed article hypothesis , vol. 36, no. 2, 2024 congratulations to the award-winning research papers and posters from mla ’24 portland as well as the biannual 2022-2023 jmla research paper winner! the mla research caucus is pleased to announce the winners for best research papers and posters presented at the mla 2024 portland annual meeting. thank you to all the judges who volunteered their expertise to help select these deserving awardees, both in the pre-judging phase and during the conference. to learn more about the awards and selection process, visit the research caucus website: https://www.mlanet.org/communities/all-bb-communities/research/. mla 2024 annual meeting contributed paper awards 1st place – expert-recommended tasks for hospital librarians during a healthcare system merger or acquisition: an e-delphi consensus statement. authors: jaclyn morales, senior librarian, stacy posillico, senior librarian, and saori wendy herman, assistant dean of library services, zucker school of medicine at hofstra northwell. objective: little empirical research is available to guide hospital librarians through the healthcare system merger or acquisition process. in order to address this knowledge gap, a literature review and the e-delphi research method were used to develop expert-driven recommendations to prioritize those tasks that librarians should undertake when consolidating the delivery of library services to a newly merged, geographically distributed healthcare system. methods: a review of existing literature through january 2021 was performed and updated in october 2023 to discover reports, conference presentations, and articles in which hospital librarians shared their experiences with the merger process. twenty-nine hospital librarians and library professionals who had experienced a merger or acquisition in the last ten years at healthcare systems within the seven nnlm regions agreed to participate in this study as experts. using a modified e-delphi process that was conducted according to credes standards, the expert panelists responded to four rounds of questionnaires during april to december 2022. in the first round, they identified tasks that librarians should undertake during the merger process. tasks were then eliminated or prioritized in rounds 2 through 4, based upon the experts’ rating of each task using a seven-point likert scale in which 1 equaled “not recommended” and 7 equaled “essential recommendation.” those tasks rated as either 5, 6, or 7 by ≥75% of the panelists at the conclusion of the round were included in the final statement of recommended tasks. results: a consensus-based statement of 330 recommended tasks for librarians to consider during a healthcare system merger or acquisition was created. the final set of recommended tasks were grouped into four main domains: healthcare organization tasks, library collections & information systems, library administration, and library staff integration & interconnection. tasks related to information technology and services, vendor relations, and library organizational structure were more likely to be prioritized than tasks related to marketing and outreach and physical library space. expert insight in the final round highlighted the importance of understanding context 2 https://www.mlanet.org/communities/all-bb-communities/research/ editorial reviewed article hypothesis , vol. 36, no. 2, 2024 and culture when undertaking any recommended task. conclusions: further research and refinement after utilization will likely be needed, but the priority tasks recommended by the expert panel can be used immediately by hospital librarians to create a plan of action for providing library services as healthcare systems transformatively expand and address library assimilation and consolidation after a merger occurs. 2nd place – this is how we do it: tenure & promotion in academic health sciences libraries. author: erin reardon, public health informationist, emory university. objectives: the traditional three arms of the tenure-track professor’s duties are teaching, service, and research. tenure-track academic librarians, though their numbers are dwindling, are expected to meet the same or similar requirements to achieve tenure. whether academic librarians should have tenure or tenure-like status is a matter of some debate, but what has not been examined are the criteria by which academic librarians are evaluated in order to be awarded tenure. this study aims to examine and compare these criteria across academic health sciences libraries’ promotion and tenure documents. methods: data from the association of academic health sciences libraries (aahsl) descriptive statistics from fiscal year 2021-2022 was used for this project, specifically item d.14a, faculty appointment status. of the 130 aahsl member institutions who responded to the survey, there were 29 institutional health sciences libraries who reported a tenure-track faculty appointment system for their staff librarians. this study does not examine the promotion requirements for libraries that reported other faculty-like appointments with tenure-like systems such as continuous appointment, non-tenure-track faculty appointment, faculty appointment outside the library, or no faculty appointment. through internet and institutional website searching, documentation of the tenure criteria was obtained for 21 libraries. a content analysis was undertaken using maxqda to count the number of documents in which certain words and themes appeared. the resulting data was organized into domains of criteria (such as librarianship, research, and service) and elements of those criteria (such as job performance, publications, and committee work). results: twenty of the 21 institutions’ documents described “service” as a domain; 19 documents each describe “librarianship” and “research and scholarship” as domains. four documents contained the domain of “teaching and learning,” and two documents contained a domain for “diversity, equity, and inclusion” (dei). in the “librarianship” domain, 117 individual words or themes were coded as elements. of these elements, 67 (or 58%) only appear in one document, showing a wide variety in the description of the work of the librarian. in the “research and scholarship” domain, 57 elements were coded, of which 37 (or 57%) only appear in one document. in the “service” domain, 52 elements were coded, of which 30 (or 58%) only appeared in one document. in “teaching and learning,” 14 elements were coded with no crossover between the four documents in which this domain was found. elements were unable to be coded for the dei domain. conclusions: there is a wide variety in the way the work of the librarian is described, as well as what constitutes research or service. these documents describe multiple 3 editorial reviewed article hypothesis , vol. 36, no. 2, 2024 ways to engage with librarianship, research, service, and teaching, but the amount of overlap or lack thereof between the documents suggests that academic health sciences librarians’ profession is poorly defined or difficult to define in these criteria. 2nd place – an exploration of basic/life science information professionals: educational background, liaison roles and carnegie classification. authors: jeremy kupsco, research informationist, emory university, laura lipke, health science librarian, binghamton university, and stephanie schulte, director, health sciences library, ohio state university. objectives: since the early 2000’s and the explosion of electronic information resources, information professionals have been struggling to reconnect to the basic science community. the information profession has expanded its roles beyond collection development to knowledge regarding the data service and research needs of this specialized community. as late as 2022, literature acknowledges that the outreach struggles continue and that this population is still unaware of the “capabilities of a science liaison librarian.” scholarship has supported the ability of information professionals to fulfill the roles of training and instruction. methods: the purpose of this study is to identify differences in collaboration between information professionals with and without degrees or training in basic science who work with the basic science community. the study will also examine whether carnegie classification of the institution impacts information professionals’ interactions. participants for this study were identified via their institutional websites as well as through the email lists for the basic science caucus of the medical library association, the science and technology section of the american library association, the health sciences special interest group of the association of college and research libraries and the stem librarians collaborative on discord. they were recruited via direct email. the study used semi-structured interviews via zoom to gather data. this data will be analyzed thematically. in addition, basic demographic information from the recruiting survey along with publicly available data about their institutions will provide context for addressing the research questions. results: to date, the team has interviewed fifteen basic science librarians from various backgrounds and institutions and transcription of the recorded zoom interviews has begun. a few noted emerging themes such as librarians that attended basic science program events were more successful in building relationships; most liaison skills for these programs were learned on the job; scientific intellectual curiosity is a skill needed to be successful; and that librarians with a science degree were able to connect with these programs because they spoke ”their language.” a detailed thematic analysis of the results will be provided at the presentation of this study. conclusions: based on the initial findings of this study, it appears that liaisons with a basic science degree do have an advantage, when connecting to the basic science community. more importantly, though, is for the liaison to look for opportunities to interact/support grant writing, research and sponsored programs, and to demonstrate curiosity regarding the research of these programs. further thematic analysis and findings regarding the basic science liaison librarians role with this population, and a discussion of how carnegie classifications play a role in successful collaborations will be presented. 4 editorial reviewed article hypothesis , vol. 36, no. 2, 2024 mla 2024 annual meeting contributed poster awards 1st place – using bibliometric analytic techniques to measure the scholarly impact of a health professions education teaching academy. authors: sarah cantrell, associate director, beth blackwood, research & education librarian, deborah engle, diana mcneill, and kristin dickerson, duke university medical center library & archives. 2nd place – stronger together exploring medical students’ experiences with chatgpt. authors: authors: emily hannum, library technical assistant iii and nadine dexter, director, harriet f. ginsburg health sciences library, university of central florida college of medicine. background: large language model ai use has increased dramatically in the past few years, representing a unique method to engage in collaborative research. with this field developing at a rapid pace, it is valuable to understand how ai offers potential for assisting in medical education. this highlights a need to understand how ai tools are currently being implemented in medical education environments, a consideration of the effectiveness of these tools, and reflection on the ethical implications of ai use in medical education. objectives: the goal of this study was to evaluate how frequently and in what ways medical students are using chatgpt for their studies or extracurriculars, and their level of satisfaction in using this resource for their academic-related needs. while many studies have shown chatgpt’s potential applications in medical education, they do not explore how students currently use chatgpt and other ai platforms. we believe understanding current medical students’ use of ai platforms is an essential step towards integrating them into medical education. methods: medical students from the university of central florida’s college of medicine were asked to voluntarily complete a survey including likert scale and open response questions regarding their use of and comfort with chatgpt. the survey consisted of 11 questions, remained available for 14 days, and 64 medical students participated. results: regardless of age, the use and confidence of chat gpt remains similar on average among everyone who took the survey. most medical student respondents have used chatgpt despite skepticism of usefulness and accuracy of information obtained. furthermore, medical students are also either supportive of or indifferent to recommending chatgpt for medical students. the most frequently used sources for verifying chatgpt information included google and clinical apps such as uptodate and epocrates. some of the major concerns students cited for chatgpt use included receiving simplistic answers, falsified studies, reflection of user bias, and ethical implications. conclusion: despite the general skepticism surrounding the benefits of using large language model ai such as chatgpt, medical students appear to already be incorporating these tools into their education, albeit cautiously. this cautious exploration suggests that as ai tools continue to advance so will their application in medical education. medical school faculty and support staff must begin to familiarize themselves with these tools and the potential implications of their use in academia. 5 editorial reviewed article hypothesis , vol. 36, no. 2, 2024 there are currently no definitive means to consistently identify ai-crafted work nor evaluate potential bias within the ai tool itself, emphasizing the need for ai familiarity to keep pace with ai technological advancements. further research on the application of additional ai tools is necessary. 3rd place – an environmental scan of evidence synthesis projects published by yale university and yale new haven hospital authors: preliminary findings of scoping reviews. authors: alyssa grimshaw, clinical research and education librarian and holly grossetta nardini, associate director, harvey cushing/john hay whitney medical library, yale university. objective: to better understand our institutions’ user behavior, we investigated evidence synthesis publications that have been produced at our institution. methods: an exhaustive search was performed across eighteen databases to identify papers associated with yale university or yale new haven hospital using keywords or publication types related to scoping and systematic reviews. a dashboard was developed for monitoring evidence synthesis publications. papers with a yale/ynhh author listed as either the first or last author were incorporated. the analysis centered on extracting information concerning library engagement in the review process and compliance with reporting best practices. results: yale’s first and last authors have contributed to 90 scoping reviews. among these, 59% of the reviews involved librarians, either through co-authorship (38%), mention in the methods section (3%), or acknowledgment (18%). the data indicates that the registration of a protocol associated with a review and reproducible search compliance were more prevalent when librarians were involved. based on the findings, the library intends to offer 15-minute instruction sessions focused on improving search reporting and recommendations for publishing evidence synthesis protocols. conclusion: the findings of this environmental scan could impact the library’s policies and procedures relating to librarian involvement in evidence synthesis projects, future staffing support, and creation of new training modules for patrons that clarify proper reporting of searching and methodology in evidence synthesis projects. jmla biannual research article award kahili-heede, m. k., patil, u., hillgren, k. j., hishinuma, e., & kasuya, r. (2022). library instruction and wikipedia: investigating students’ perceived information literacy, lifelong learning, and social responsibility through wikipedia editing. journal of the medical library association: jmla, 110 (2), 174–184. https://doi.org/10.5195/jmla.2022.1291 objectives: this article presents a multiyear pilot study delineating practical challenges, solutions, and lessons learned from wikipedia editing experiences with first-year medical students at the john a. burns school of medicine at the university of hawai’i at mānoa. the purpose of our project was to determine the feasibility and effectiveness of wikipedia editing to improve information literacy and lifelong learning skills and to investigate aspects of social responsibility in first-year medical students. methods: lessons were provided through a combination of in-person and online 6 https://doi.org/10.5195/jmla.2022.1291 editorial reviewed article hypothesis , vol. 36, no. 2, 2024 instruction via the wikiedu learning management system (lms). students next selected a health-related wikipedia article to edit. after the editing experience, structural completeness data were collected from the wikiedu lms. feedback was collected via an anonymous retrospective pre-post survey to assess the students’ attitudes toward their perceived information literacy skills and the social responsibility of improving wikipedia articles. nonparametric tests were conducted to compare pre versus post outcomes. results: fifty-seven (79%) participants in the 2018 cohort and forty-nine (64%) participants in the 2019 cohort completed the retrospective pre-post survey. in both cohorts, respondents showed statistically significant increases (p<.05) in self-rating of all ten domains of information literacy and social responsibility after completing the program. conclusions: this study showed that medical students are competent editors of wikipedia and that their contributions improve both the quality of the articles and their own perceived information literacy. additionally, editing medicine-related articles provides an opportunity to build students’ social responsibility by improving content on an open platform that reaches millions each day. 7 peer reviewed article hypothesis, vol. 36, no. 2, 2024 librarian and clinical faculty collaborative use of formative assessment in a longitudinal evidence-based medicine curriculum sa’ad laws, ma, mlis*a, mai a. mahmoud, mbbsb, moune jabre, mdc, amal khidir, mbbsd, ziyad mahfoud, phde acorresponding author, health sciences library, weill cornell medicine – qatar, doha, qatar, sal2018@qatar-med.cornell.edu bassociate professor of teaching in medicine/assistant dean for faculty affairs, department of medical education, weill cornell medicine – qatar, doha, qatar cassistant professor of clinical obstetrics and gynecology/assistant dean for academic affairs, department of medical education, weill cornell medicine – qatar, doha, qatar dassociate professor of pediatrics, department of medical education, weill cornell medicine – qatar, doha, qatar eprofessor of research in population health sciences, department of medical education, weill cornell medicine – qatar, doha, qatar, division of epidemiology, department of population health sciences, weill cornell medicine, new york, new york cite as: laws s, mahmoud ma, jabre m, khidir a, and mahfoud z. librarian and clinical faculty collaborative use of formative assessment in a longitudinal evidence-based medicine curriculum. hypothesis. 2024;36(2). doi:10.18060/28048 laws, mahmoud, jabre, khidir, mahfoud. all works in hypothesis are licensed under a cc by-nc 4.0 deed attribution-noncommercial 4.0 international. authors own copyright of their articles appearing in hypothesis. readers may copy articles without permission of the copyright owner(s), as long as the author(s) are acknowledged in the copy, and the copy is used for educational, not-for-profit purposes. for any other use of articles, please contact the copyright owner(s). 1 mailto:sal2018@qatar-med.cornell.edu https://creativecommons.org/licenses/by-nc/4.0/ peer reviewed article hypothesis, vol. 36, no. 2, 2024 background the authors developed a longitudinal curriculum for teaching thirdand fourth-year undergraduate medical students evidence-based medicine (ebm). this curriculum involved substantial librarian involvement, the use of formative assessment as a teaching method, and progressive repetition of skills. methods students in three clinical clerkships (medicine, pediatrics, ob/gyn) completed ebm assignments based on real-world scenarios in this quasi-experimental study. each clerkship required students to submit an ebm plan, from which they received directed feedback from both the clerkship director and a librarian after submitting preliminary ebm plans. this study tracked student pico and search submissions to determine if repeated exposure to ebm feedback resulted in improved summative assessments, both in isolation and longitudinally. results students’ pico and searching performance improved between mid-clerkship formative feedback and end-clerkship summative assessment in all three clerkships using rubrics developed by the authors for each clerkship derived from existing literature. when examining student performance sequentially over three clerkships, there was significant improvement between the first and second clerkships, but this did not carry into the third clerkship. discussion our findings suggest that the significant inclusion of a librarian and feedback appears to have positive effects on student performance. while it may seem obvious that feedback results in improved outcomes, this method doesn’t appear widespread in medical education. repetition, while not having a lasting increase in performance, may still be warranted to increase exposure to authentic cases and evidence types. introduction in medical education, enhancing the ability to make well-informed clinical decisions is crucial to improving patient care. central to this decision-making process is evidence-based medicine (ebm), a methodology that integrates clinical expertise, patient values, and the best available evidence to guide healthcare decisions. ebm is traditionally described as “the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients.”1 the use of ebm requires a holistic understanding of question formation, searching, critical appraisal, and clinical application.2 the acquisition of these skills is a fundamental aspect of healthcare provider education and a lifelong learning skill that has been identified by multiple medical education accreditation organizations at both undergraduate and graduate levels.3,4 while medical education has evolved to include ebm as a fundamental component, there remains a pressing need to ensure that students grasp the theoretical aspects and develop the skills to apply ebm in real-world clinical scenarios. in this context, several studies have examined different methods of teaching ebm to medical trainees. an important area of exploration in these studies is the timing, duration, and intensity of ebm instruction within the undergraduate curriculum. some studies have focused on implementing ebm instruction early during preclinical years,5,6 while others have opted to delay until clinical-based years.7,8 a common rationale in studies that advocated for delay was implementation at a time when students were interacting with patients in the clinical setting to give more context to ebm learning. in another case, faculty have utilized other approaches, 2 peer reviewed article hypothesis, vol. 36, no. 2, 2024 such as the use of peer instructors, a capstone course for fourth-year undergraduate medical students, and dedicated seminars.9-11 although these studies include an array of teaching methods, it is notable that none were conducted in a clinical setting. both clinical and non-clinical-based instruction demonstrated an improvement in skills and attitudes compared to pre-instruction. despite the gains noted in several studies, there does not appear to be a consensus about the best methods for instructing trainees in ebm. ilic and maloney note that most instructional methods show some level of improvement in competencies; there is no clear benefit of one method over others.12 a notable trend in many studies of ebm in both graduate and undergraduate medical education has been the inclusion of librarians in various roles. while it has been noted that librarians are an underutilized resource for ebm instruction,13 several studies have shown that librarians are adept in instructing students on question formulation and the best methods for searching clinical databases for evidence.14,15 in one example of librarian involvement, minuti et al. described a flipped-classroom module that used readings, video, and exercise in support of sessions with first-year students that focused on pico formulation and searching and later with second-year students where additional searching techniques were taught and clinical faculty followed with critical appraisal instruction.16 minuti et al. noted that librarian involvement was significant for assisting in meeting accreditation requirements and the positive impressions that faculty had after the course. another notable example of librarian integration in ebm instruction is from a study of trainees’ skills and attitudes in an emergency medicine course.17 this course incorporated significant levels of librarian involvement and emphasis on searching skills. trainees reported significant improvements in searching-related skills, but they noted that additional instruction is necessary for sustained application and reinforcement. the present study is a further exploration of the benefits of both varied methods in teaching ebm to medical trainees as well as the significant incorporation of librarians within ebm instruction. both of these areas of inquiry for teaching ebm are unresolved in the current literature and vital for the future development of ebm instruction to medical trainees. with this in mind, the objectives of this study are to evaluate the effectiveness of our ebm curriculum within each clerkship, as well as longitudinally between the first and third clerkships. methods study population and characteristics this quasi-experiment focused on ebm instruction within three clinical clerkships: medicine, pediatrics, and obstetrics and gynecology (ob/gyn). the study was conducted at weill cornell medicine – qatar (wmc-q) in doha, qatar. wmc-q follows a four-year, american-based medical curriculum from the main weill cornell medicine campus in new york, usa. these three clerkships were specifically chosen because they are the only three that have explicit ebm curricular elements. data from 32 students who completed all three clerkships were analyzed in this study. approval for this study was granted by the wcm-q institutional review board on february 10, 2022 (21-00026). because this was a retrospective analysis, the study team requested and was granted a waiver of informed consent of subjects. before entering these clerkships, students received preliminary ebm instruction during their first year of preclinical education. in this context, students took a course on clinical 3 peer reviewed article hypothesis, vol. 36, no. 2, 2024 epidemiology and research methods, followed by a 10-hour, face-to-face, course utilizing lectures and labs to emphasize ebm fundamentals, question formulation, searching, and critical appraisal. while the ebm curriculum of the three clerkships in this study followed a similar track, each clerkship employed slight variations to attain their ebm objectives. the distribution and progression of students through the three clerkships is shown in table 1. it is noteworthy that the ebm instructional model introduced in the pediatrics clerkship was implemented during the second clerkship of the academic year. as a result, none of the students included in this study had pediatrics as their initial clerkship. table 1: student progression through clerkship with ebm instructions ob/gyn pediatrics internal medicine first clerkship 13 0 9 second clerkship 12 10 10 third clerkship 7 22 3 internal medicine clerkship students in the internal medicine clerkship began the course by completing an asynchronous elearning module that reviewed ebm fundamentals. the elearning module in the internal medicine clerkship is a gamified experience, mimicking the structure of the tv game show jeopardy, where students select from four levels of questions across five categories that increase in difficulty and point value. the module was internally produced using standard web development tools, such as html and css. the elearning module’s content included 20 total questions on pico formation, searching techniques, evidence appraisal, and ebm question type (i.e., harm, therapy, diagnosis, prognosis). each category included four questions that escalated in the level of difficulty and point value for correct answers (range 2-8 points). each question offered feedback for both correct and incorrect answers. the main objective of the elearning module is to refresh students on ebm fundamentals and highlight potential gaps in ebm comprehension. students received a final score, with a maximum of 100 points possible, and the module concluded with a leaderboard of the top score for every iteration of the course. all elearning module data is available to both the librarian and clerkship director. at the midpoint of the clerkship, students were grouped into teams of three to four students by faculty and each team was required to submit a group ebm plan. the content of the ebm plan must be drawn from the clinical experience of at least one of the students in the team. the plan included the case details, pico, and search strategy. after submission, each ebm plan underwent two reviews. first, the clerkship director reviewed the plan for clinical viability, either approving it or requesting revisions. in parallel, the librarian evaluated each ebm plan’s pico and search strategy. additionally, each team met with the librarian to receive feedback on their pico and search strategy and discussed potential amendments for various aspects of each team’s ebm plan. the summative assessment for ebm in the internal medicine clerkship was a team-based oral presentation. teams had to provide details about their patient’s case, pico, search strategy, appraisal of a selected research article, and how they could apply the evidence from their selected article to their patient. after the presentation, fellow students, the librarian, and the director were invited to ask the presenting team questions about their presentation. the librarian assessed the pico and search strategy, while the director evaluated the clinical and team-based components of each team’s ebm 4 peer reviewed article hypothesis, vol. 36, no. 2, 2024 presentation. all assessments used a rubric with four levels, from inadequate to exceeds expectation, with each level receiving a score of 0 to 5 (see appendices for detailed rubric). pediatrics clerkship students in the pediatrics clerkship started the course with a required short asynchronous elearning module that reviewed key aspects of ebm, such as searching and constructing a pico. students completed four “knowledge check” questions at various points in the tutorials to verify content comprehension. each question offered feedback for both correct and incorrect answers. all students’ responses were available to both the librarian and clerkship director. at the midpoint of the clerkship, each student submitted an individual ebm plan based on a clinical encounter they had experienced during the clerkship. the ebm plan included the details of their case, pico, and preliminary search strategy. each ebm plan was reviewed by the clerkship director for clinical content and relevance. the clerkship director then either approved each plan or asked the student to submit a revised plan before the student could proceed. the librarian reviewed the pico and search strategy. both the clerkship director’s and the librarian’s feedback were sent to the student via email. although no individual meetings were required, students were highly encouraged to set up individual meetings with either the librarian or clerkship director, if they felt this was needed. at the end of the clerkship, students submitted a written ebm project. student projects introduced their patient’s case, pico, search strategy, and appraisal of a selected research article, and explained how they could apply the evidence from their article to their patient. the librarian reviewed each ebm project, provided comments on the pico and search strategy, and then forwarded these to the clerkship director, who completed the grading of clinical and appraisal content. all assessments used a rubric with four levels, from inadequate to exceeds expectation, with each level receiving a score of 0 to 3 (see appendices for detailed rubric). obstetrics and gynecology (ob/gyn) clerkship students started the ob/gyn clerkship by completing an asynchronous elearning module that reviewed ebm fundamentals ranging from question formation, searching, and appraisal, to application. at the start of the elearning module, students were presented with an ob/gyn-based case. students then worked through ten questions from three domains, searching/pico, appraisal, and application, based on the case and reviewed a systematic review relevant to the case. each question offered feedback for both correct and incorrect answers. students received a final score with a maximum of ten points. this elearning module is notable for giving students additional exposure to appraisal and application of systematic reviews, as much of their previous instruction had focused on primary studies, such as randomized controlled trials. at the midpoint of the clerkship, each student submitted an individual ebm plan based on a clinical encounter they had experienced during the clerkship. the ebm plan included the details of their case, pico, and preliminary search strategy. each student’s ebm plan was reviewed by the clerkship director for clinical content and either approved or not approved with revisions required. the librarian reviewed the pico and search strategy. students met with the librarian to receive feedback on their pico and search strategy and discussed any general questions they had about their project. at the end of the course, students submitted a write-up project that included a significant section devoted to ebm. students introduced their patient’s case, pico, search strategy, and appraisal of a selected research article, and explained how they could apply the evidence from their article to their patient. students were 5 peer reviewed article hypothesis, vol. 36, no. 2, 2024 required to incorporate a minimum of five references to support their arguments and all references were suggested to be no older than five years old. all assessments used a rubric with four levels, from not observed to achieved, with each level receiving a score of 0 to 3 (see appendices for detailed rubric). table 2: comparison of ebm instruction between clerkships elearning format individual/group mid-clerkship plan feedback format summative assessment format internal medicine gamification group mandatory in-person oral presentation pediatrics case-based individual mandatory asynchronous text-based written ob/gyn case-based individual mandatory in-person written collaboration between librarians and clinical faculty a notable feature of this ebm curriculum is the significant interaction between the librarian and clerkship directors. in each clerkship, the librarian worked with the clerkship directors to develop elearning cases and materials. materials were selected by the librarian and clerkship directors to address noted deficiencies from students in past course iterations. additionally, the librarian maintained a continuous dialogue with each clerkship director when assessing students’ mid-clerkship ebm plans. for instance, there were several occasions where clinical aspects, such as if the student’s ebm plan utilized appropriate interventions, affected the assessment of the student’s search. during these occasions, the librarian and clerkship director would discuss these and similar aspects of student plans to offer the most useful and appropriate feedback. finally, each clerkship director made an effort to incorporate the librarian into the summative assessment phase of each ebm project. for instance, in the internal medicine clerkship, the librarian attended ebm presentation sessions and offered feedback about pico, searching, and other relevant aspects of the presentations. finally, the librarian and each clerkship director met annually to review the ebm session for that year and discuss revisions, as needed, for the following year’s sessions. project data eligible students for this project were limited to students who completed all three clerkships during the july 2020 to june 2021 clerkship cycle. this time frame was selected because it was the first complete cycle of clerkship rotations for an academic year in which the present curriculum was delivered. since midpoint feedback data was not originally scored using a rubric, the librarian retrospectively reviewed each midpoint plan using the summative rubric of the respective clerkship. the librarian was blinded to the summative score of each student during retrospective scoring. both midpoint and summative data were then compiled using excel and included each student’s score for pico and searching. data was also included to indicate the sequential order in which each student completed the three clerkships. statistics for each clerkship, students’ scores on pico and searching individually, and then a combined pico plus searching score were summarized using means and standard deviations (sd). to check for improvement in such scores within a clerkship, mid-clerkship scores were 6 peer reviewed article hypothesis, vol. 36, no. 2, 2024 compared to the scores at the end of the clerkship using the paired t-test. the effect of the interventions in this study made more sense when using mid-clerkship scores as the pre-test scores so that the actual interventions could be better isolated for analysis. as a sensitivity analysis, each of these comparisons were repeated using the wilcoxon signed rank test. to check if student scores improved over time; that is, from first to second to third clerkship, the friedman’s test was used to compare the scores over the sequence of clerkships. if differences in scores over time existed, then the paired t-tests were used to test if scores were different between first and second, first and third, or second and third clerkships. there were no adjustments for multiple tests. as a sensitivity analysis, all the pairwise comparisons above were repeated using the wilcoxon signed rank test. to check on the effect of clerkship type on student scores, we compared the three clerkships, internal medicine, obgyn, and pediatrics using friedman’s test with wilcoxon signed rank test for multiple comparisons if needed. a p-value of 0.05 or less was considered statistically significant. all analyses were done using ibm-spss (version 26, armonk, ny). results overall, 32 student records with complete data from all three clerkships were used for analysis. for their first clerkship, 19 students had medicine and 13 had obgyn. for their second clerkship, 10 had medicine, 12 had ob/gyn and 10 had pediatrics, and for their third clerkship, 3 students had medicine, 7 had ob/gyn and 22 had pediatrics. the ebm instructional model for the pediatrics clerkship that is described above was introduced during the second clerkship of the academic year used here, thus no students in this study had pediatrics as their first clerkship. comparing searching and pico scores over time in this study, we investigated whether the inclusion of a librarian in ebm instruction could lead to improvements in student scores in ebm over time during their clinical clerkships in internal medicine, ob/gyn, and pediatrics. our analysis focused on three key variables: pico scores, searching scores, and overall scores. we compared these scores from mid-clerkship feedback (mid-clerkship score) to the end of clerkship summative ebm assessment (end clerkship score). we observed significant increases in all three variables (see table 3, column 5 for detailed data). importantly, these improvements were consistent across all clerkships, regardless of whether it was the student’s first, second, or third clerkship. sensitivity analysis using the nonparametric alternative confirmed the significance of these findings (see table 3, column x for p-values). we found significant changes in mid-clerkship scores for search (table 3, paired t-test p=0.001) and overall (paired t-test p=0.018, table 3) over time but not for pico (paired t-test p=0.158, table 3). however, we have not found any significant changes in end clerkship scores over time (column 7 for p-values, table 3). there were significant increases in student scores on formulating the pico, search strategy, and total pico/search, from mid-clerkship feedback to end-of-clerkship summative ebm assessment. these significant increases were observed in all clerkships regardless of whether the clerkship sequence was the student’s first, second, or third (see table 3). sensitivity analysis using the nonparametric alternative resulted in the same significance levels (p-values <0.001). there were significant changes over time in mid-clerkship scores on search and 7 peer reviewed article hypothesis, vol. 36, no. 2, 2024 overall. students scored significantly higher on the second and third clerkships as compared to the first clerkship (see table 3). there were no significant changes in end clerkship scores over time. table 3: comparing scores within and between the sequence of clerkships mid-clerkship score end clerkship score midclerkship score end clerkship score clerkship order variable mean±sd mean±sd p-value p-value over time p-value over time first clerkship pico 1.92±0.38 2.64±0.48 <0.001 0.158 0.383 second clerkship pico 2.11±0.40 2.48±0.47 <0.001 third clerkship pico 2.00±0.40 2.52±0.43 <0.001 first clerkship search 1.31±0.28 2.39±0.52 <0.001 0.001* 0.275 second clerkship search 1.63±0.51 2.28±0.55 <0.001 third clerkship search 1.66±0.41 2.16±0.63 <0.001 first clerkship overall 3.23±0.49 5.03±0.74 <0.001 0.018* 0.142 second clerkship overall 3.73±0.68 4.77±0.75 <0.001 third clerkship overall 3.66±0.53 4.67±0.8 <0.001 *significant at the 5% level comparing searching and pico scores at mid-clerkship and end of clerkship in addition to examining student progression over time through the three clerkships, we wanted to examine if student performance improved within each clerkship. we found that the increase in scores on pico from mid-clerkship feedback to the end of clerkship summative ebm assessment was significantly higher in the first clerkship compared to the second clerkship (paired t-test p=0.005, table 4) but not the third clerkship (paired t-test p=0.167, table 4). additionally, the increase in search score and the overall score was significantly higher in the first clerkship compared to both the second clerkships (search, paired t-test p=0.005, overall, paired t-test p=0.001, table 4) and the third clerkships (search, paired t-test p=0.001, overall, paired t-test p=0.003, table 4). the increase in scores on pico from mid-clerkship feedback to end-of-clerkship summative ebm assessment was significantly higher in the first clerkship as compared to the second one but not the third one. as for the increase in search score and overall score, they were significantly higher in the first clerkship than both the second and the third clerkships. see table 4. 8 peer reviewed article hypothesis, vol. 36, no. 2, 2024 table 4: comparing the increase in scores over the sequence of clerkships first clerkship second clerkship third clerkship mean±sd mean±sd mean±sd p-value 1 vs. 2 p-value 1 vs. 3 p-value 2 vs. 3 increase in pico score 0.72±0.49 0.38±0.54 0.52±0.56 0.005* 0.167 0.354 increase in search score 1.08±0.56 0.66±0.50 0.50±0.67 0.005* <0.001* 0.245 increase in overall score 1.80±0.78 1.03±0.67 1.02±0.89 <0.001* 0.003* 0.931 *significant at the 5% level comparing student performance by clerkship students’ scores on pico in the internal medicine clerkship were significantly higher than those in the other two clerkships (p=0.001). on the other hand, there were no significant differences in student search scores between the three clerkships (p=0.327) (see table 5). table 5: comparing the scores between the different types of clerkships regardless of sequence med ob/gyn peds mean±sd mean±sd mean±sd p-value pico score 2.77±0.34 2.47±0.51 2.41±0.45 0.001* search score 2.28±0.33 2.38±0.71 2.17±0.6 0.327 overall score 5.05±0.41 4.84±0.95 4.58±0.79 0.170 *significant at the 5% level discussion our results in this study align with previous studies that indicate the benefits of including librarians in ebm instruction for medical students.18-21 the librarian in this study provided support for question formulation and database searching in addition to providing feedback to clerkship directors that was used in summative assessment. as many studies have noted, the inclusion of librarians in health science instruction can offer immediate and long-term benefits to students. for example, librarian inclusion allows faculty to design instruction that reflects the team-based modeling that many students will encounter when providing care for patients.22 as other studies have noted, the inclusion of librarians and other similar professionals in the instruction of medical trainees has both affective and cognitive benefits.12,22,23,24 although not the focus of this study, the increased participation of librarians in ebm instruction does provoke two important questions: what level of training is sufficient and needed for this level of inclusion, and what level of burden is required from the librarian due to this level of course support? 9 peer reviewed article hypothesis, vol. 36, no. 2, 2024 another major finding of this study is that implementing a curricular design that utilizes formative assessment of student question formation (pico) and searching had consistent benefits for students between formative and summative assessment. while it may seem obvious to suggest that feedback leads to improvements in summative assessment, it should be noted that this is one of the first reported ebm courses that incorporated formative assessment as a fundamental element of curricular design. the only ebm course that was identified as using feedback was atwa and abdelaziz,25 and information on the methods and nature of the feedback to a take-home assignment in this study is unclear. many of the studies we surveyed here utilized lectures, workshops, or a combination of the two.26 our results suggest that while there were initial gains between mid-clerkship feedback and final assessments between a student’s first and second clerkship, this progression was not sustained into the third clerkship. one reason for this could be that students had reached a learning plateau after their second clerkship and any subsequent instruction was generally not as effective for most students after this point in the sequence. alternatively, this result could indicate that students suffered some level of fatigue from the repletion of ebm instruction. with this said our results suggest that students retained the knowledge and skills needed for ebm over time. our study also supports the idea that providing repetitive, longitudinal instruction in ebm, despite the risk of fatigue, is beneficial for medical trainees. limitations our study revealed that the inclusion of a librarian in ebm instruction positively impacted students’ scores, but also has its limitations. ebm assessments in the internal medicine clerkship were given as a group. because of this, there is the possibility that a student who would have otherwise scored higher or lower in each student’s group was either aided or hindered by their group members. in subsequent studies it would be interesting to see the effect of a control group that could demonstrate how the lack of a librarian’s involvement might impact students’ scores. this study utilized only one scorer (librarian) to assess students. in future iterations of similar studies, it would be beneficial to have multiple scorers trained in advanced search skills, as this would limit potential bias during assessment. this study focused on student searching and pico formation; however, it would be interesting to also examine how the instruction detailed in this study affects student evidence assessment and clinical application scores. a final limitation of this study involves the use of pico as an assessment measure. while pico has been used as a method in the formulation of clinical questions, it does have its limitations and opponents. for instance, studies have cited pico’s limited utility for non-therapy questions27, ability to effectively translate to database searches28, and non-superiority to alternative methods for formulating questions.29 conclusion the results of this study suggest that students benefit from varied teaching methods for ebm instruction. specifically, the use of formative feedback appears to lead to positive outcomes in summative assessments. our results reiterate and expand on previous studies that indicate that repeated instructional efforts to teach ebm to trainees are beneficial and result in increased knowledge and skills. finally, our study advocates for greater inclusion of information professionals in ebm instruction, specifically in the areas of pico/question formation and searching. 10 peer reviewed article hypothesis, vol. 36, no. 2, 2024 corresponding author sa’ad laws, librarian, education & research, sal2018@qatar-med.cornell.edu ethical declarations ethical approvals and consent to participate because the study involved only minimal risk and all study data was deidentified with all links to original identifiers destroyed, a waiver of informed consent was approved by the weill cornell medicine qatar institutional review board on february 10, 2022 (irb number: 21-00026). the declaration of helsinki was followed in conducting the study. all methods were carried out in accordance with relevant guidelines and regulations. consent for publication not applicable. data availability statement the datasets used and/or analyzed during the current study are available from the corresponding author upon reasonable request. competing interest the authors have no competing interests to declare. funding not applicable. authors’ contributions sa’ad laws assisted in conceptualization, data curation, methodology, and project administration, and is the lead author of the manuscript. ziyad mahfoud assisted with conceptualization, study design, formal analysis, and writing the manuscript. mai a. mahmoud, moune jabre, and amal khidir assisted with conceptualization, study design, methodology, and revision of the manuscript. acknowledgements the authors would like to thank qatar foundation for their support of this research. references 1. sackett dl, rosenberg wmc, gray jam, haynes rb, richardson ws. evidence based medicine: what it is and what it isn’t. bmj. 1996 jan 13;312(7023):71–2. doi:10.1136/bmj.312.7023.71 2. djulbegovic b, guyatt gh. progress in evidence-based medicine: a quarter century on. lancet lond engl. 2017 jul 22;390(10092):415–23. doi:10.1016/s0140-6736(16)31592-6 3. liasion committee on medical education. standards, publications, & notification forms [internet]. 2022 [cited 2022 may 16]. available from: ttps://lcme.org/publications/ 4. acgme program requirements for graduate medical education in internal medicine [internet]. 2020. available from: 11 mailto: sal2018@qatar-med.cornell.edu ttps://lcme.org/publications/ peer reviewed article hypothesis, vol. 36, no. 2, 2024 https://www.acgme.org/portals/0/pfassets/programrequirements/140_internalmedicine_2020.pdf 5. holloway r, nesbit k, bordley d, noyes k. teaching and evaluating first and second year medical students’ practice of evidence-based medicine. med educ. 2004 aug;38(8):868–78. doi:10.1111/j.1365-2929.2004.01817.x 6. menard l, blevins ae, trujillo dj, lazarus kh. integrating evidence-based medicine skills into a medical school curriculum: a quantitative outcomes assessment. bmj evid-based med. 2021 oct;26(5):249–50. doi:10.1136/bmjebm-2020-111391 7. ismach rb. teaching evidence-based medicine to medical students. acad emerg med. 2004 dec;11(12):e6-10. doi:10.1197/j.aem.2004.08.037 8. aiyer mk, dorsch jl. the transformation of an ebm curriculum: a 10-year experience. med teach. 2008;30(4):377–83. doi:10.1080/01421590701881632 9. mai dh, taylor-fishwick js, sherred-smith w, pang a, yaworsky j, whitty s, et al. peer-developed modules on basic biostatistics and evidence-based medicine principles for undergraduate medical education. mededportal. 2020 nov 24;16:11026. doi:10.15766/mep_2374-8265.11026 10. anderson cr, haydek j, golub l, leong t, smith dt, liebzeit j, et al. practical evidence-based medicine at the student-to-physician transition: effectiveness of an undergraduate medical education capstone course. med sci educ. 2020 jun;30(2):885–90. doi:10.1007/s40670-020-00970-9 11. weberschock tb, ginn tc, reinhold j, strametz r, krug d, bergold m, et al. change in knowledge and skills of year 3 undergraduates in evidence-based medicine seminars. med educ. 2005 jul;39(7):665–71. doi:10.1111/j.1365-2929.2005.02191.x 12. ilic d, maloney s. methods of teaching medical trainees evidence-based medicine: a systematic review. med educ. 2014;48(2):124–35. doi:10.1111/medu.12288 13. boykan r, jacobson rm. the role of librarians in teaching evidence-based medicine to pediatric residents: a survey of pediatric residency program directors. j med libr assoc. 2017 oct 2;105(4):355–60. doi:10.5195/jmla.2017.178 14. ma ksk, chang hc, krupat e. teaching evidence-based medicine with electronic databases for preclinical education. adv physiol educ. 2021 dec 1;45(4):849–55. doi:10.1152/advan.00057.2021 15. swanberg sm, mi m, engwall k. an integrated, case-based approach to teaching medical students how to locate the best available evidence for clinical care. mededportal. 2017 jan 19;13:10531. doi:10.15766/mep_2374-8265.10531 16. minuti a, sorensen k, schwartz r, king ws, glassman nr, habousha rg. librarians flip for students: teaching searching skills to medical students using a flipped classroom 12 https://www.acgme.org/portals/0/pfassets/programrequirements/140_internalmedicine_2020.pdf https://www.acgme.org/portals/0/pfassets/programrequirements/140_internalmedicine_2020.pdf peer reviewed article hypothesis, vol. 36, no. 2, 2024 approach. med ref serv q. 2018 jun;37(2):119–31. doi:10.1080/02763869.2018.1439184 17. dorsch jl, aiyer mk, meyer le. impact of an evidence-based medicine curriculum on medical students’ attitudes and skills. j med libr assoc. 2004 oct;92(4):397–406 18. haley j, mccall rc, zomorodi m, de saxe zerdan l, moreton b, richardson l. interprofessional collaboration between health sciences librarians and health professions faculty to implement a book club discussion for incoming students. j med libr assoc. 2019 jul;107(3):403–10. doi:10.5195/jmla.2019.563 19. gill kd, parker r. an untapped resource? opportunities for faculty-librarian collaboration to enhance drug information resource utilization in pharmacy education. j med libr assoc. 2022 oct 1;110(4):478–84. doi:10.5195/jmla.2022.1486 20. muellenbach jm, houk km, e thimons d, rodriguez b. integrating information literacy and evidence-based medicine content within a new school of medicine curriculum: process and outcome. med ref serv q. 2018;37(2):198–206. doi:10.1080/02763869.2018.1439225 21. boruff jt, thomas a. integrating evidence-based practice and information literacy skills in teaching physical and occupational therapy students. health inf libr j. 2011 dec;28(4):264–72. doi:10.1111/j.1471-1842.2011.00953.x 22. brian r, orlov n, werner d, martin sk, arora vm, alkureishi m. evaluating the impact of clinical librarians on clinical questions during inpatient rounds. j med libr assoc. 2018 apr;106(2):175–83. doi:10.5195/jmla.2018.254 23. tahmasebi m, adibi p, zare-farashbandi f, papi a, rahimi a. the educational role of clinical informationist on improving clinical education among medical students: based on kirkpatrick model. j educ health promot. 2020;9:28. doi:10.4103/jehp.jehp_439_19 24. krom zr, batten j, bautista c. a unique collaborative nursing evidence-based practice initiative using the iowa model: a clinical nurse specialist, a health science librarian, and a staff nurse’s success story. clin nurse spec cns. 2010 apr;24(2):54–9. doi:10.1097/nur.0b013e3181cf5537 25. atwa h, abdelaziz a. evidence-based medicine (ebm) for undergraduate medical students: a six-step, integrative approach. med teach. 2017 apr;39(sup1):s27–32. doi:10.1080/0142159x.2016.1254750 26. al shahrani as. development and evaluation of an evidence-based medicine module in the undergraduate medical curriculum. bmc med educ. 2020 aug 6;20(1):256. doi:10.1186/s12909-020-02181-7. 27. huang x, lin j, demner-fushman d. evaluation of pico as a knowledge representation for clinical questions. amia annu symp proc. 2006;2006:359-363. 28. eriksen mb, frandsen tf. the impact of patient, intervention, comparison, outcome (pico) as a search strategy tool on literature search quality: a systematic review. j med libr 13 peer reviewed article hypothesis, vol. 36, no. 2, 2024 assoc. 2018;106(4):420-431. doi:10.5195/jmla.2018.345 29. kloda la, boruff jt, cavalcante as. a comparison of patient, intervention, comparison, outcome (pico) to a new, alternative clinical question framework for search skills, search results, and self-efficacy: a randomized controlled trial. j med libr assoc. 2020;108(2):185-194. doi:10.5195/jmla.2020.739 14 editorial reviewed article hypothesis, vol. 36, no. 1, 2024 incentives that drive the editorial peer review process jonathan d. eldredge, phd, ahip, fmla, associate editora aprofessor and evidence-based practice librarian health sciences library and informatics center; school of medicine; and college of population health, university of new mexico, albuquerque, new mexico,https://www.orcid.org/0000-0003-3132-9450 , jeldredge@salud.unm.edu cite as: eldredge jd. incentives that drive the editorial peer review process. hypothesis. 2024;36(1). doi:10.18060/27697 eldredge. all works in hypothesis are licensed under a cc by-nc 4.0 deed attribution-noncommercial 4.0 international. authors own copyright of their articles appearing in hypothesis. readers may copy articles without permission of the copyright owner(s), as long as the author(s) are acknowledged in the copy, and the copy is used for educational, not-for-profit purposes. for any other use of articles, please contact the copyright owner(s). 1 https://www.orcid.org/0000-0003-3132-9450 https://orcid.org/ mailto:jeldredge@salud.unm.edu https://creativecommons.org/licenses/by-nc/4.0/ https://creativecommons.org/licenses/by-nc/4.0/ editorial reviewed article hypothesis, vol. 36, no. 1, 2024 abstract a research project can often take a long, circuitous route from the proposal stage, proceeding on to the institutional review board’s scrutiny, followed by the research project implementation, through to the analysis and writing of the manuscript. securing grant funding often adds an extra 6-12 months to the process. for many busy health information professionals (hips), research projects sometimes feel like an added burden in the face of more pressing demands for their time and energy. when finally submitting a manuscript to a journal for consideration, authors might feel as if they already have been on a prolonged quest simply to produce their manuscripts. submitting a manuscript for publication can be a humbling experience. after all of the effort in reaching the submission stage, the manuscript needs to be evaluated critically by editorial peer reviewers. editors consider these peer reviewers’ recommendations before accepting or rejecting the submitted manuscripts. many times, the editors might require “major revisions” based on the peer reviewers’ evaluations before the editors consider the manuscript further. in some cases, manuscripts go through an additional cycle of review before the editors make their decision. understandably, many authors complain about the extra work required to revise a manuscript. by the manuscript submission stage, authors cannot help but feel emotionally invested in guiding their manuscripts through to publication. peer reviewers or editors still might not be satisfied by the author’s additional revisions. the editors still might reject the revised manuscript or demand even further revisions on a subsequent cycle. this column outlines why authors, peer reviewers, and editors often misunderstand each other’s roles in this process. peer reviewers the vast majority of peer reviewers are volunteers, who often perform this vital service outside of work hours. only recently have some workplaces credited employees with providing this service. so, why do they do it? a 2021 scoping review summarized research journals’ peer reviewers’ motivations for serving in this largely invisible yet vital role for journals. internal motivations were largely idealistic: professional obligations and reciprocity, a desire to learn, “enjoyment/satisfaction”, staying current in the field, assuring scientific rigor in the research enterprise, and preserving the prestige and reputation of a specific journal. external motivations that peer reviewers reported were: professional reputation, recognition as an expert, and fostering good relationships with editors1. as a long-time peer reviewer who has reviewed more than 70 manuscripts, most of the internal motivations reported in this 2021 scoping review resonate. in contrast, the three major external motivations uncovered in this scoping review did not seem to similarly ring true. there might be other internal motivations not captured in this scoping review. as someone steeped in the evidence based practice paradigm, this author wants a research article to accurately represent its findings and limitations to aid readers’ critical appraisal. while in the role of a peer reviewer, this author wants to uphold sound research standards. he has no 2 editorial reviewed article hypothesis, vol. 36, no. 1, 2024 qualms about subjecting even valid studies to an acid test that few, if any, readers will have the time or skill to pursue themselves. while in this peer reviewer role, he views prospective readers as relying on these quality assurance steps when reviewing these studies. this role does not imply that peer reviewers anonymously can offer feedback in an unprofessional manner to their well-meaning author colleagues. anonymous peer reviewer comments should always be constructive and professional, even when rejecting a manuscript2. peer reviewers have often published in the same journal and likely hope to publish in the same journal again, so they have a basic desire to aid the journal’s continued success. the internal altruistic motivations to serve as an editorial peer reviewer largely outweigh the external reasons. serving as an editorial peer reviewer seems to be a largely thankless task if viewed from a solely external perspective. the aforementioned scoping review also touches on the disincentives to serve as an editorial peer reviewer: limited time, critiques not fully respected or followed, and feeling underappreciated. these disincentives have been echoed by others3-5. editors editors are volunteers who perform their services mostly outside of work hours. this author has served as an associate editor for four journals and as a column editor for one journal6. journal editors must have a far-reaching and expansive view of the entire journal rather than any one individual article, column, or issue. they have a strong, unceasing drive to guarantee the utmost quality of the content and production of their journal. editors for health information professionals’ journals are held in high regard, perhaps because this prestige implies that our editors will serve our profession well in every regard. in the larger realm of health professions journal editors, our own editors seem to form a positive personal identity with their own journals over time. editors care about the overall appeal of their journals for their readers and prospective readers. editors want their journals to be read, discussed, and to have a positive influence in the lives of their readers. they seek to gauge their journals’ content to the interests of their readers. editors recognize that their journals fill a niche in the hip publishing ecosystem. over time readers have come to expect the same kind of content within their journals. a more research-oriented journal, for instance, will emphasize greater methodological rigor than a more pragmatically-oriented journal. sometimes editors will sense that they are publishing too much on the same subject. editors might curtail publishing additional manuscripts on the same topic, delay publication of submissions on the topic, or reject submissions outright in some cases. editors should be professional by explicitly notifying prospective authors that their submitted manuscript will not be a good fit for the journal at the outset. editors furthermore can be constructive by suggesting alternative journals where the manuscript might be a better fit. at the same time, if their usage data suggests that readers are drawn to certain subjects, the editors might expand their coverage of these popular topics. in these ways, editors’ interests align tightly with most readers, but not necessarily with authors. 3 editorial reviewed article hypothesis, vol. 36, no. 1, 2024 editors sometimes point to their low acceptance rate for submitted manuscripts as an indicator of quality7. these editors do not seem to realize that they might be discouraging possible authors to submit to eligible, in-scope journals. it seems to this editor that a journal should be as explicit as possible about what it expects in submitted manuscripts rather than to succumb to a simplistic manuscript acceptance rate as a metric of value. authors authors navigating the peer review process tend to view the process differently than the editors or the peer reviewers. authors prioritize the acceptance and publication of their own individual manuscripts over any other article that other authors are vying to publish in the same journal. indeed, authors are more inclined to magnify the importance of their own manuscript compared to peer reviewers or editors. authors seem motivated largely by a desire to reach readers who potentially share an interest in the same subjects. other incentives might complicate authors’ motivations when publishing to fulfill institutional requirements for promotion. in such cases, authors might interact with editors and peer reviewers with a sense of urgency or even desperation. on balance, authors and editors alike generally recognize that readers have limited time and attention spans8, so they should avoid trying to hype the importance of their articles. authors need to realize the following, in light of these incentives by all involved parties in their interactions with peer reviewers and editors: 1. editors and peer reviewers are colleagues who are concerned with the overall quality and readership of their journal; 2. editors and peer reviewers are volunteers for these roles, oftentimes working outside regular work hours; 3. authors should pay special attention to their choices of words when communicating with peer reviewers and editors. these words are all that the editors and peer reviewers have to form impressions, conscious or unconscious, of the author. these impressions might affect the editors’ and peer reviewers’ management of a particular manuscript. personal attacks or disparaging tones should be avoided. authors should remember that editors and peer reviewers are colleagues who are highly sensitive to language. references 1. mahmić-kaknjo m, utrobičić a, marušić a. motivations for performing scholarly prepublication peer review: a scoping review. account res. 2021;28(5):297-329. doi:10.1080/08989621.2020.1822170 2. gerwing tg, allen gerwing am, avery-gomm s, choi cy, clements jc, rash ja. quantifying professionalism in peer review. res integr peer rev. 2020;5:9. doi:10.1186/s41073-020-00096-x 4 https://dx.doi.org/10.1080/08989621.2020.1822170 https://dx.doi.org/10.1186/s41073-020-00096-x editorial reviewed article hypothesis, vol. 36, no. 1, 2024 3. ellwanger jh, chies jab. we need to talk about peer-review-experienced reviewers are not endangered species, but they need motivation. j clin epidemiol. 2020;125:201-205. doi:10.1016/j.jclinepi.2020.02.001 4. bernstein j. free for service: the inadequate incentives for quality peer review. clin orthop relat res. 2013;471(10):3093-3097. doi:10.1007/s11999-013-3216-z 5. goodman s. the scholarly skill almost no one is teaching. chronicle of higher education. 2022 nov 11.[internet] available from: <https://www.chronicle.com/article/the-scholarly-skill-almost-no-one-is-teaching>. 6. eldredge j. jama journal reviews: analysis and master index 1992-1996. medical library association news. 1997 feb; (293):22-4. 7. eldredge j. characteristics of peer reviewed clinical medicine journals. med ref serv q. 1999;18(2):13-26. doi:10.1300/j115v18n02_02 8. west jd, bergstrom ct. misinformation in and about science. proc natl acad sci usa. 2021;118(15):e1912444117. doi:10.1073/pnas.1912444117 5 https://dx.doi.org/10.1016/j.jclinepi.2020.02.001 https://dx.doi.org/10.1007/s11999-013-3216-z <https://www.chronicle.com/article/the-scholarly-skill-almost-no-one-is-teaching> https://dx.doi.org/10.1300/j115v18n02_02 https://dx.doi.org/10.1073/pnas.1912444117 peer reviewed article hypothesis , vol. 34, no. 1, 2022 a pilot project exploring low-tech collaborative board gaming on student attitudes toward interprofessional education rachel keiko stark, ms, ahipa, eugenia opuda, mls, medb, ahealth sciences librarian, university library, california state university sacramento, sacramento, california, https://orcid.org/0000-0001-7946-2213 , stark@csus.edu bassistant professor, health and human services librarian, dimond library, university of new hampshire, durham, new hampshire, https://orcid.org/0000-0003-1410-6546 , eugenia.opuda@unh.edu objectives: exploring the impact of a small, collaborative gaming experience among undergraduate students on student attitudes and beliefs about interprofessional education (ipe). methods: health sciences librarians at two state universities used a low-tech, collaborative board game to facilitate small, in-person group gaming. the authors used an adapted version of the interprofessional attitudes scale (ipas), a validated instrument that measures the four domains of the 2011 ipec core competencies for interprofessional collaborative practice, as a pre-/post-assessment of the impact of the gaming experience. results: while anecdotal evidence indicated that playing a collaborative board game had an impact on the participating students, there was no statistical evidence that collaborative board games increase positive or negative attitudes about ipe and interprofessional collaborative practice. conclusions: more student participation is required for statistical significance. recruiting students and ensuring that students have enough time to complete the game play is key to possible future success. introduction interprofessional education in health sciences curriculum interprofessional education (ipe) is a growing trend among health sciences curriculums that integrates students from multiple disciplines into teams for collaborative problem solving. the world health organization1 reports that ipe builds collaborative skills and knowledge for future practitioners to tackle global health challenges in order to improve health outcomes around the world. an effective ipe curriculum may also impact students by increasing positive attitudes2, and lead to an increase in collaborative behavior and patient satisfaction3. understanding each individual’s professional role and expertise and developing respect and trust through clear communication are critical to effective ipe4. elements that contribute to effective ipe include maintaining open dialogue, shifting 1 https://orcid.org/0000-0001-7946-2213 https://orcid.org/ mailto:stark@csus.edu https://orcid.org/0000-0003-1410-6546 https://orcid.org/ mailto:eugenia.opuda@unh.edu peer reviewed article hypothesis , vol. 34, no. 1, 2022 from an egocentric to a collaborative mind-set, developing mutual trust, and establishing mutual goals, all of which contribute to higher levels of team satisfaction4. however, a systematic review5 identified a list of barriers to implementing ipe including stereotyping among faculty and students involved in the ipe program and varying characteristics of students from different backgrounds. the authors note that hierarchies exist among health sciences professionals, with faculty members and students in medicine typically at the top. these hierarchies enable faculty behaviors to be guided by certain attitudes. students tend to mimic and adopt those faculty attitudes, thus making collaborative learning and establishing mutual respect challenging5. positive attitudes, on the other hand, are valuable for facilitating an effective ipe curriculum5,6 through open-mindedness and enthusiasm for participation. sunguya et al.5 further suggest benefits to preparing ipe students in advance with the acknowledgement that participating students and faculty have different learning needs and approaches to care. additionally, faculty involvement in ipe may also contribute to the success or failure of ipe programs. sunguya et al.5 state that faculty level of preparedness, attitudes and biases, and ability to manage a class impact a student’s experience of the ipe curriculum. giordano, umland, and lyons6 and gary, gosselin, and bentley7 note the importance of positive faculty engagement and support for students to the success of the ipe curriculum. games as a tool for collaborative learning a systematic review investigating the use of escape rooms in education noted that the game strengthened relationships by creating a sense of belonging among the players, improving creative problem-solving skills, fostering collaborative leadership behaviors, and motivation8. similarly, another study9 noted the positive impact of an escape room-type game in increasing ipe behavior and communication among students. however, fotaris and mastoras8 note that creating an escape room activity is time-consuming and may be limited due to lack of time commitment and limited resources. board games, on the other hand, have been used to increase empathy and collaborative attitudes10-12, and enhance learning and knowledge12,13. using a preexisting board game reduces the time commitment needed to design a game from scratch and is relatively inexpensive and accessible in comparison to escape rooms. the tactile aspect of board gaming and other low-tech gaming options can encourage collaborative student engagement and learning in library settings14,15. modern tabletop games also have systems of rules that help students learn about their own abilities with no fear of embarrassment, and students can transfer important skills learned during the gaming period to their academic experience16,17. libraries are offering more opportunities for gaming with their users, and more librarians are offering games as a means of engaging and teaching important skills to their users. research question this research aimed to explore the use of collaborative, low-tech board gaming on student attitudes and beliefs about interprofessional education using an adapted version of the interprofessional attitudes scale (ipas), a validated scale created by norris et al.18. the ipas was selected as a pre/post-measurement for this research study. validation results for the ipas indicate that the scale has good factor structure 2 peer reviewed article hypothesis , vol. 34, no. 1, 2022 and internal consistency18. the ipas was used with permission from norris et al., and modified, with permission, to better represent the degrees and programs available at the universities in this study. the ipas measures the four domains of the 2011 ipec core competencies for interprofessional collaborative practice19 including the values and ethics for interprofessional practice, roles and responsibilities, interprofessional communication, and team and teamwork. what was particularly attractive to the authors about the ipas was that the tool measured, among other things, individual students’ agreement with statements about teamwork roles and responsibilities, which aligned with the authors’ interest in collaboration amongst students. the ipas uses likert scale questions to record students’ agreement with statements regarding teamwork roles and responsibilities, patient-centered care, unprofessional biases, diversity and ethics, and community-centeredness. by utilizing ipas, the authors attempted to determine if playing a collaborative board game, that puts players into various assigned public health professional roles to prevent a worldwide pandemic, increases positive self-reported student attitudes and beliefs about interprofessional education and interprofessional collaborative practice. giordano, umland, and lyons6 and gary, gosselin, and bentley7 recommend an initial assessment of attitudes towards ipe for developing an effective curriculum and call for a reevaluation of those attitudes throughout the program. ipe requires students to work together while respecting the individual roles, skills, and experiences of each student. the authors have experience with ipe in nonclinical settings and intended to leverage that experience to explore if a collaborative board game encourages the practical requirements of ipe, such as working together while respecting individual roles, skills, and the experiences of the participating students. the author at sacramento state university has been part of a faculty learning community focused on ipe and runs an ipe book club with a nonlibrary faculty member focused on aging. the ipe book club offers an ipe certificate for participation through the interprofessional education center for innovative teaching and learning. the ipe program at university of new hampshire is a small, but growing initiative. the author at university of new hampshire works with students and faculty who are involved in the support and participation in the ipe program. with these experiences and with the authors’ experiences playing the board game, it was possible for the game to serve as an option for students who might not have a chance to participate in ipe opportunities otherwise. methods background and settings participants were recruited from two large state universities, one on the west coast and one on the east coast: sacramento state university in california and the university of new hampshire in new hampshire. sacramento state university is located in a large urban area and is a majority commuter campus with approximately 28,000 students at the time this research was conducted. it offers undergraduate and graduate degrees in various allied health disciplines, including but not limited to nursing (undergraduate and graduate degrees), physical therapy (graduate degree), communication sciences disorders (undergraduate and graduate degrees), and gerontology (undergraduate and graduate certificates). sacramento state university is 3 peer reviewed article hypothesis , vol. 34, no. 1, 2022 a hispanic-serving institution and an asian-american, and native american pacific islander-serving institution. the university of new hampshire is a rural campus and the flagship institution of the state of new hampshire. the university of new hampshire campus had approximately 12,000 students at the time this research was conducted, offered undergraduate and graduate degrees in various allied health disciplines, including but not limited to nursing, communication sciences and disorders, occupational therapy, and health management and policy. background the study was active from january 2018 april 2019. the authors initially intended to recruit student participants who were involved in interprofessional education curriculums, but due to very limited enrollment, broadened the study to include students in the college of health and human services, and eventually to any student enrolled at sacramento state university. preand post-assessment ipas surveys were printed out for students for each session and provided before and after game play. a total of ten students participated in the study and completed the preand post-assessments. the authors held a total of six in-person gaming sessions in which participants played a collaborative board game titled, pandemic. at sacramento state university only three sessions had students: one large group of seven students, and two small groups. in the large group, only six students completed the ipas and played the entire time. one small group had three students, but only two completed the ipas and the other small group had two students, but only a single student completed the ipas and was interested in playing the game. at university of new hampshire, only one session had one student who completed the ipas. pandemic was chosen because it is a collaborative board game with a health sciences emphasis. in the game, each participant is assigned a specific role based on real-life health sciences professions and disciplines, and the players must work cooperatively with other players to eradicate and treat four different diseases before any one of the diseases becomes a pandemic. players randomly select a role card, which describes what the player can do in addition to the normal parts of play in the game. each role has a different special function, and some roles are more helpful for different strategies than others. for example, when playing as the dispatcher, you can move players around the board easier and faster than players moving on their own, or, when playing as the researcher, you can give your research (cards that players must collect in a matched set to start eradicating a specific disease) to other players or move research from player to player. while these roles are very simplified, they offer insight into the various roles that people occupy in a real health care team. each session consisted of one to six study participants and lasted approximately 30 to 90 minutes. the authors booked private library spaces to hold the study, with the understanding that the library is space that is shared by all of the various departments and no single program or discipline has a claim on the space used. in some instances, the authors participated in the game play, particularly when there was only one study participant. however, the author at sacramento state university participated with group game play when requested to do so by the students and served as a moderator when requested with the larger group of students. a moderator is someone who doesn’t need to play the game, but can read the rules to the participants, particularly when the game play stalls due to different interpretations of the rules. the authors did not have a formal 4 peer reviewed article hypothesis , vol. 34, no. 1, 2022 measurement that recorded whether students had previously played the game. irb this research study was submitted to and approved by the institutional review board at sacramento state university on january 24, 2018, irb-17-18-105, and a collaborative research agreement was signed between the institutions. marketing to encourage students to participate in the research project, fliers were created using the piktochart web tool and distributed at both campuses. at sacramento state university, fliers and information were sent to faculty members in the health sciences programs, as well as distributed to the interprofessional education student organization and the gaming club for students. similarly, at university of new hampshire, fliers were shared with faculty members in the health sciences programs to email to students, presented at the end of library instruction sessions, and physically displayed in bulletins. students were offered snacks as part of the experience, which were provided by the researcher from sacramento state university, and pizza provided by the researcher at university of new hampshire. results a total of ten students from both institutions completed both the preand post-ipas assessments. most students had science or health sciences backgrounds but did not have previous experience with ipe. student majors included biology, chemistry, engineering, recreation therapy, and gerontology. only two students who completed pre/post-ipas assessments indicated previous experience with ipe, but that previous experience did not seem to have a strong impact on their scores on the ipas. each participant was asked to fill out hard copy preand post-ipas assessments and one author documented the de-identified data into an institutional qualtrics survey software. the 26th version of the spss statistical software was used to run multiple anova and t-tests. findings are presented below in table 1 and table 2. 5 peer reviewed article hypothesis , vol. 34, no. 1, 2022 table 1. paired sample statistics pair mean std. deviation std. error mean teamwork pre 53.40 4.169 1.318 teamwork post 54.20 4.709 1.489 patient-centered pre 33.70 1.829 .578 patient-centered post 34.30 .949 .300 biases pre 13.60 1.838 .581 biases post 15.40 3.406 1.077 diversity ethics pre 27.10 1.287 .407 diversity ethics post 26.90 1.912 .605 community pre 38.30 2.627 .831 community post 39.20 3.425 1.083 additional pre 80.10 6.045 1.912 additional post 84.00 7.196 2.275 all pre 246.20 9.773 3.090 all post 254.00 9.262 2.929 table 2. paired samples test mean std. deviation std. error mean 95% ci lower 95% ci upper t df sig. (2tailed) teamwork -.800 5.750 1.818 -4.914 3.314 -.440 9 .670 patientcenteredness -.600 1.897 .600 -1.957 .757 -1.000 9 .343 biases -1.800 4.662 1.474 -5.135 1.535 -1.221 9 .253 diversity and ethics .200 2.150 .680 -1.338 1.738 .294 9 .775 community -.900 2.998 .948 -3.045 1.245 -.949 9 .367 additional -3.900 7.965 2.519 -9.597 1.797 -1.548 9 .156 all -7.800 10.064 3.183 -15.000 -.600 -2.451 9 .037 results from the preand post-ipas assessments suggest that playing a collaborative board game had a modest impact on student attitudes towards interprofessional education. for all items of the ipas, with the exception of the diversity and ethics item, post-assessments had slightly higher scores in comparison to pre-assessments. it is difficult to interpret what these modest shifts in preand post-assessment scores 6 peer reviewed article hypothesis , vol. 34, no. 1, 2022 might mean without examining each item’s subscale. for example, the teamwork, roles, and responsibilities subscale includes disparate items such as “shared learning will help me think positively about other professionals,” “patients would ultimately benefit if health sciences students worked together to solve patient problems,” as well as “it is not necessary for health sciences students to learn together.” thus, without examining each subscale, a higher post-assessment score would not necessarily suggest an overall more positive attitude toward teamwork. subscale analysis for the data is not included in this work due to the small sample size. for the subscales to be useful in understanding the effects of the intervention, a larger sample size is needed. for interprofessional biases, subscale items include “health professionals/students from other disciplines have prejudices or make assumptions about me because of the discipline i am studying,” “i have prejudices or make assumptions about health professionals/students from other disciplines,” and “prejudices and assumptions about health professionals from other disciplines get in the way of delivery of health care.” similarly, this item is challenging to decipher without examining each subscale item. though study participants scored higher in the post-assessment, some students only interacted with one of the authors during game play rather than with other students, limiting the possible validity of the results presented on the other hand, the patient-centeredness, diversity and ethics, and community-centeredness scales include subscale items that were more analogous and would indicate that higher post-assessment scores equate to improved attitudes towards ipe after game play. that said, the diversity and ethics scale was the only scale that had a modest decrease in post-assessment scores in comparison to the pre-assessment scores. due to the small number of participants, there is no statistical significance to the results of the ipas assessment. however, anecdotal comments from participants indicated that playing the board game with students from other majors was an experience that helped them discover their own biases and was a lesson in teamwork. for example, at sacramento state university, the six students who participated in the largest group of players, often had difficulty agreeing on actions and working together. the biology students with a confirmed health sciences interest commented to the author that they had not worked with students from some of the other majors before and that it was a learning experience about how different people approach problems. another student from a different session than the one previously described, commented that it was sometimes frustrating to play the game when they had to depend on other players who interpreted things differently. while this is anecdotal, these comments indicate that the experience of playing the collaborative board game did cause students to consider key areas of ipe that are measured by the ipas, particularly teamwork roles and responsibilities and interprofessional biases. lessons learned while there is little statistical impact in the current study and not all students completed the ipas assessment or chose to share comments, the comments they did receive lead the authors of this paper to postulate that with more participants there 7 peer reviewed article hypothesis , vol. 34, no. 1, 2022 could be significant data collected. furthermore, for universities where ipe opportunities are limited for undergraduate students, offering a collaborative health sciences-focused board game experience can be a cost-saving opportunity for low-risk ipe. ipe educational opportunities ideally allow students to learn how to work with other students with different backgrounds and experiences with a goal that is reflective of a health care focus1,2. playing a board game with a health care focus, such as working as a team where each member has specific skills and different roles to prevent a pandemic, allows students to have access to an ipe-style experience without having to wait to be enrolled in the higher-level courses that are often where students encounter ipe. the authors note that another significant limitation to this study is the small participant size and possible self-selection. challenges with marketing and student time and interest contributed to the small participant pool size. the few students who participated may have had some familiarity with board gaming and enjoyed collaborative games prior to playing pandemic. it is possible that offering the gaming sessions in a dedicated space specifically for health sciences students to converge, such as a health sciences lounge or at a health sciences library, might assist future research with recruitment. alternatively, offering gaming sessions as part of student orientations for ipe programs could potentially contribute to increased engagement. having to travel to different areas of campus, which might be far from health sciences classrooms, was a deterrent for participation. students were sometimes reluctant to complete the pre-/post-ipas and this did limit the number of results. being able to offer incentives for the completion of the ipas probably would have increased participant completion rates. the ipas would also benefit from collecting more specific demographic data, and in future research utilizing the ipas, it would be very helpful for the ipas to be modified to reflect this need. many of the ipe opportunities at the participating universities are only open to students enrolled in higher-level health sciences programs. the authors hoped that by opening the gaming ipe experience to students in nonhealth sciences focused programs, they could increase participation in the study, and more students would be able to experience ipe for the first time. the authors felt that by being more open and inclusive of students in nonhealth sciences departments, this ipe experience would also be more reflective of the interprofessional practice (ipp) experience. ipp is often a clinical experience and ipe is designed to help prepare students to enter clinical settings where ipp is an expected experience, however in many settings, ipp includes professionals such as social workers, advocates, and religious counsellors, who are not well-represented in ipe experiences in the published literature. the authors hope that future ipe experiences will be more inclusive to the students who will take on these roles and that future research will find a way to record if ipe with students outside health sciences disciplines is helpful for professional development. this research was conducted before the covid-19 pandemic, and the authors recognize that it might be difficult to replicate this research during a pandemic as in-person gathering and playing board games are not recommended at the time of the writing of this paper. there is an application-based mobile version of the game that could be utilized. however, without the in-person interaction, it would be difficult to 8 peer reviewed article hypothesis , vol. 34, no. 1, 2022 determine if the ipe interaction occurred utilizing measurements based on in-person ipe interventions. it is also important to acknowledge that a version of the game that requires the use of mobile devices is an equity issue, particularly for students who customarily utilize campus-provided devices, as not all campuses provide mobile devices to all students, internet access can be an issue, and it might not be permissible for students to download and use the application-based version of the game on a campus-provided device. it is also worth noting that playing a game about preventing a pandemic while simultaneously experiencing a real-life pandemic might not be the most appropriate choice, so other collaborative board games should be explored as an option. the authors recognize that playing a game about preventing a pandemic before experiencing a pandemic in real life might affect participants who have played the game, such as increased understanding of the importance of collaboration from professionals around the world, and hope that future research will explore these and other concepts as appropriate and with all due consideration to the well-being of the participants. conclusion gamification and ipe are two unique services that libraries can offer their institutions. using a popular board game provides an opportunity to engage students in health sciences-related majors who are interested in games but might not otherwise be interested in participating in ipe opportunities. this may be particularly important for students who choose to work in health-focused fields, such as health advocacy, medical engineering, recreational therapy, and health care administration. these student groups are generally overlooked in traditional ipe activities but may benefit from ipe since they will most likely work with traditional health sciences practitioners in their future career paths. it is also important to note that using a relatively low-cost game, which does not require special technology or access to the internet to use, ensures that students and institutions with different levels of access to technology and with different budgets have an opportunity to offer a low-stress ipe experience for students. while there are many examples of successful gamification and of ipe in libraries, we understand that our research does not represent success, but rather an opportunity for future research. acknowledgements: the authors would like to thank and acknowledge all the students who participated in this research and completed the ipas. the authors would like to thank and acknowledge nina exner for her assistance with the spss software and her kindness in assisting the authors with this paper. declaration of interest: none to report 9 peer reviewed article hypothesis , vol. 34, no. 1, 2022 contributions: rachel keiko stark facilitated a request for a collaborative irb. both authors marketed and conducted the research studies at each of their institutions and gathered formal ipas responses. rachel keiko stark entered all preand post-ipas assessment responses into qualtrics while eugenia opuda ran the responses through the spss software and ran paired sample statistics. nina exner was consulted on interpreting the data obtained through spss and the authors thank and appreciate her assistance. rachel keiko stark also gathered informal participant comments. both authors contributed to writing and editing the manuscript. references 1. world health organization. framework for action on interprofessional education and collaborative practice [internet]. geneva, switzerland: who press; 2010. [2010; cited 2021 jun 14]. available from: https://www.who.int/publications/i/item/framework-for-action-on-interprofessionaleducation-collaborative-practice 2. reeves s, perrier l, goldman j, freeth d, zwarenstein m. interprofessional education: effects on professional practice and healthcare outcomes. cochrane database of syst rev [internet]. 2013;(3):cd002213. doi:10.1002/14651858.cd002213.pub3 3. reeves s, palaganas j, zierler b. an updated synthesis of review evidence of interprofessional education. j allied health. 2017 mar;46(1):56–61. 4. lateef f. inter-professional education, inter-professional practice and team science: learning together; working together. eimj. 2018 feb;10(4):81–91. doi:10.21315/eimj2018.10.4.8 5. sunguya bf, hinthong w, jimba m, yasuoka j. interprofessional education for whom? — challenges and lessons learned from its implementation in developed countries and their application to developing countries: a systematic review. plos one. 2014 may;9(5): e96724. doi:10.1371/journal.pone.0096724 6. giordano c, umland e, lyons kj. attitudes of faculty and students in medicine and the health professions toward interprofessional education. j allied health. 2012 mar;41(1):21–5. 7. gary jc, gosselin k, bentley r. health science center faculty attitudes towards interprofessional education and teamwork. j interprof care. 2018 mar;32(2):231–4. doi:10.1080/13561820.2017.1376626 8. fotaris p, mastoras t. escape rooms for learning: a systematic review. in: proceedings of the 12th european conference on game based learning [internet]. acpi; 2019 oct. 30. doi:https://dx.doi.org/10.34190/gbl.19.179 9. friedrich c, teaford h, taubenheim a, boland p, sick b. escaping the professional silo: an escape room implemented in an interprofessional education curriculum. j interprof care. 2019 sep;33(5):573–5. doi:10.1080/13561820.2018.1538941 10. nieh h-p, wu w-c. effects of a collaborative board game on bullying intervention: a group-randomized controlled trial. j school health. 2018 oct;88(10):725–33. doi:10.1111/josh.12675 11. arnab s, clarke s, morini l. co-creativity through play and game design thinking. ejel. 2019 sep;17(3). doi:10.34190/jel.17.3.002 10 https://www.who.int/publications/i/item/framework-for-action-on-interprofessional-education-collaborative-practice https://www.who.int/publications/i/item/framework-for-action-on-interprofessional-education-collaborative-practice https://doi.org/10.1002/14651858.cd002213.pub3 http://dx.doi.org/10.21315/eimj2018.10.4.8 http://dx.doi.org/10.1371/journal.pone.0096724 http://dx.doi.org/10.1080/13561820.2017.1376626 https://dx.doi.org/10.34190/gbl.19.179 https://dx.doi.org/10.1080/13561820.2018.1538941 http://dx.doi.org/10.1111/josh.12675 http://dx.doi.org/10.34190/jel.17.3.002 peer reviewed article hypothesis , vol. 34, no. 1, 2022 12. singh p, niranjan i. using groupwork in the designing and playing of board games in occupational health and safety. groupwork. 2012 jan;22(3):65–82. 13. gauthier a, kato pm, bul kcm, dunwell i, walker-clarke a, lameras p. board games for health: a systematic literature review and meta-analysis. games health j. 2019 apr;8(2):85–100. doi:10.1089/g4h.2018.0017 14. vrbancic ek, byerley sl. high-touch, low-tech: investigating the value of an in-person library orientation game. coll undergrad libr. 2018 jan;25(1):39–51. doi:10.1080/10691316.2017.1318429 15. renner br, cahoon e, allegri f. low-tech scavenger hunt model for student orientation. med ref serv q. 2016 oct;35(4):372–87. doi:10.1080/02763869.2016.1220754 16. alvarez v. engaging students in the library through tabletop gaming. kq. 2017;45(4):40–9. 17. elkins aj, hollister jm. power up: games and gaming in library and information science curricula in the united states. jelis. 2020 apr;61(2):229–52. doi:10.3138/jelis.2019-0038.r1 18. norris j, carpenter jg, eaton j, guo j-w, lassche m, pett ma, blumenthal dk. the development and validation of the interprofessional attitudes scale: assessing the interprofessional attitudes of students in the health professions. acad med. 2015 oct;90(10):1394–400. doi:10.1097/acm.0000000000000764 19. interprofessional education collaborative expert panel. core competencies for interprofessional collaborative practice: report of an expert panel [internet]. washington, dc: interprofessional education collaborative; 2011 [cited 2021 jun 14]. available from:https://ipec.memberclicks.net/assets/2011-original.pdf 11 https://doi-org.unh.idm.oclc.org/10.1089/g4h.2018.0017 http://dx.doi.org/10.1080/10691316.2017.1318429 http://dx.doi.org/10.1080/02763869.2016.1220754 http://dx.doi.org/10.3138/jelis.2019-0038.r1 http://dx.doi.org/10.1097/acm.0000000000000764 https://ipec.memberclicks.net/assets/2011-original.pdf peer reviewed article hypothesis, vol. 36, no. 2, 2024 question formulation skills building among dental hygiene students: a randomized controlled trial jonathan d. eldredge, mls, ma, phd, ahip-d, fmlaa, christine n. nathe, rdh, msb, aprofessor, health sciences library and informatics center, school of medicine, college of population health, university of new mexico albuquerque, new mexico, https://www.orcid.org/0000-0003-3136-9450 , jeldredge@salud.unm.edu bprofessor and director, division of dental hygiene; vice chair, department of dental medicine, school of medicine, university of new mexico, albuquerque, new mexico, cnathe@salud.unm.edu cite as: eldredge jd and nathe cn. question formulation skills building among dental hygiene students: a randomized controlled trial. hypothesis. 2024;36(2). doi: 10.18060/28153. eldredge, nathe. all works in hypothesis are licensed under a cc by-nc 4.0 deed attribution-noncommercial 4.0 international. authors own copyright of their articles appearing in hypothesis. readers may copy articles without permission of the copyright owner(s), as long as the author(s) are acknowledged in the copy, and the copy is used for educational, not-for-profit purposes. for any other use of articles, please contact the copyright owner(s). 1 https://www.orcid.org/0000-0003-3136-9450 https://orcid.org/ mailto:jeldredge@salud.unm.edu mailto:cnathe@salud.unm.edu https://creativecommons.org/licenses/by-nc/4.0/ https://creativecommons.org/licenses/by-nc/4.0/ peer reviewed article hypothesis, vol. 36, no. 2, 2024 purpose: dental hygiene students at our institution learn evidence based practice skills during their first year. we noted over the years that students markedly improved their formulated questions after training and application of their skills with the assistance of a rubric. research question: how much training do students require in order to use the question formulation rubric effectively: a brief 5-minute overview or a 25-minute training that includes a student peer assessment application exercise? methods: randomized controlled trial. all pre-randomized 24 students took the question formulation pre-test on the first day of the course on january 22nd. as expected, there were no statistical differences between the intervention or control group pre-test scores. the instructors administered the post-test on february 19th after the intervention group had received the training and rubric with the control group only having received the rubric with a brief explanation. results: the investigators employed a paired t-test to analyze the preand post-test score differences for each student in the intervention and control groups. surprisingly, the students’ average post-test scores were 41.75 for the control group and 43.67 for the intervention group on a 70-point scale, which were not markedly different. the initial paired t-test of the post-test scores confirmed no statistical difference either. further analysis revealed that six (6) students in the control group scored considerably higher than their classmates. interviews discovered that four (4) of these students collaborated with their fellow students in the intervention group following the instruction so they actually did benefit from peer instruction. in other words, some contamination occurred. the other two (2) students learned and practiced alone using the rubric. conclusions: additional analysis suggests that dental hygiene students benefit by experiencing extra instruction, with the cautionary caveats that this study was limited by some contamination and that it took place during the covid-19 era. introduction the modern dental hygienist must be well-versed in the concepts and skills of evidence based practice (ebp).1 dental hygiene degree programs regularly teach ebp skills to meet this need.2,3,4,5 the ebp process begins with the question formulation step followed by searching for the best available evidence to answer that question. the skill of asking an ebp question that will yield an answer requires training and practice.6 the initial ebp step of question formulation has commonly used a four-part system known by its acronym of pico. this system consists of identifying the patient, the intervention, a comparison, and an outcome. this system works for those ebp questions relating to treatment, but not for the other estimated 50% of questions in ebp.7 most of those other ebp questions relate to diagnosis, prognosis, or epidemiological aspects of patient care.8,9 pico also becomes problematic or at least inconclusively effective in positioning the practitioner for the second ebp step of searching for the best available evidence.10,11,12 at the university of new mexico, we had observed our dental hygiene students struggling 2 peer reviewed article hypothesis, vol. 36, no. 2, 2024 with the pico system in our ebp training sessions for several years, lending practical credence to the aforementioned research that had found deficiencies with pico. one reason for students struggling might be that most dental hygiene questions raised in ebp sessions related to diagnosis, epidemiology, or dental materials rather than the overriding treatment orientation of pico. while a number of ebp question formulation frameworks exist,13 these seemed most relevant to health information practitioners rather than to dental hygienists. an alternative and more recent question formulation and rubric system known as fac (focus, amplify, compose) had demonstrated promising results in a producing effective ebp questions14 so we tried the fac framework and rubric with our dental hygiene students. the center for teaching innovation at cornell university defines a rubric as “a type of scoring guide that assesses and articulates specific components and expectations for an assignment.”15 rubrics are increasingly common in health professions education to articulate criteria and expectations of student performance 16,17 and had been used routinely at our institution for over a decade. the fac framework and rubric seemed to work well in other curricula so we decided to test it with a randomized controlled trial with our dental hygiene students. our hypothesis was that our dental hygiene students, once oriented to the rubric for 5 minutes and trained for 25 minutes on the fac question formulation approach, would score better by a statistically significant margin compared to students only oriented for 5 minutes on the rubric. methods this randomized controlled trial employed a single-center, parallel-group, two-arm pre-test/post-test design to compare the effectiveness of two different training approaches involving undergraduate dental hygiene students using a question formulation rubric. this randomized controlled trial in a dental hygiene course involved 24 undergraduate students who had completed prerequisite basic sciences courses such as anatomy and physiology courses alongside hundreds of other undergraduate students. the dental hygiene course met once per week for three hours. using a random number generator, the authors randomized 12 students into a control group that received a brief explanation of a rubric for question formulation in a class session separate from the intervention group. the 12 students randomized into the intervention group received separate instruction and engaged in hands-on exercises aligned with the rubric. the authors measured student improvement in both the control group and the intervention group using an identical preand post-test. the authors received university of new mexico human research protections office approval # 19-008 on january 10, 2019. the detailed methods description follows. pre-test the authors asked students on the first day of the course on january 22nd to formulate a question prompted by a baseline clinical vignette that served as the pre-test, identifying themselves with unique four-digit numbers known only to them: you are a dental hygienist at a rural clinic in northern new mexico. today you are enjoying your career, even though you miss your dental hygienist student friends back at the university of new mexico. you are caring for chris garcia, age 38, who has multiple sclerosis (ms). chris is wheelchair bound and experiences muscle weakness with occasional episodes of losing coordination in one or the other arm. this is a moderate form of ms according to the mcdonald 3 peer reviewed article hypothesis, vol. 36, no. 2, 2024 diagnostic criteria scale. chris has never smoked tobacco and only drinks socially on an occasional basis. chris’ vital signs are 120/70 mmhg blood pressure, pulse 74 beats per minute, and 15 breaths per minutes for respirations. you are trying to determine the best way to assess and treat chris’ plaque buildup due to a lack of access to any dental services over the past five years. the students were directed to: “formulate a question based on this clinical vignette that, when answered by the dental hygienist or other colleagues, will lead to the best treatment of this patient. take no longer than five (5) minutes. write legibly.” intervention on february 5th the students received their introduction to ebp. in a separate classroom apart from their other classmates, the control group students received a 5-minute overview of the rubric and then trained on ebp searching skills during the remaining class time. the intervention group met separately for their own group’s training. intervention group students first answered on their own the question, “why do you think that formulating answerable questions will be important for your individual professional education and for your career?” five students shared their answers with other members of the intervention group. the intervention group then received the identical search training as the control group. in addition, the intervention group also learned question formulation skills during a separate 25 minutes training session that aligned with the rubric (figure 1) in an active learning18,19,20 environment with hands-on application, student peer assessment, and peer coaching elements. david a. kolb’s experiential learning cycle21 of concrete experience, reflective observation, abstract conceptualization, and active experimentation informed creation of these learning experiences. 4 peer reviewed article hypothesis, vol. 36, no. 2, 2024 figure 1. rubric for evaluating formulated ebd questions post-test two weeks later on february 19th, the instructors administered the post-test to measure the amount of improvement in question formulation skills that had been retained beyond only a few hours or days in the intervention and control groups. all students had ongoing access to the rubric the two-week interim and during the post-test assessment. following the post-test, all students received the question formulation training in class that only the intervention group initially had received to adhere to the ethical principle of equalizing the treatment between groups. students in the intervention group paired up with control group students to practice question formulation skills by writing questions of their own. all students completed a question formulation exercise as part of a graded course assignment once both control and intervention groups had received the same training. the two authors scored the preand post-tests using the rubric depicted in figure 1. on the few occasions where their scores 5 peer reviewed article hypothesis, vol. 36, no. 2, 2024 differed, the authors discussed and resolved their scores. the campus clinical and translational center statistician and the authors analyzed total pre-test and post-test scores using a t-test with the stata statistical packagetm. results all 24 students took the question formulation pre-test on the first day of the course on january 22nd. the two instructors scored the students’ formulated questions with the students’ identities concealed. table 1 describes the average age, the gender, and final course grade for each student allocated aggregated into either control or intervention groups. table 1 indicates that the two groups were demographically the same. as expected, there were no major differences between the intervention or control groups’ pre-test scores (table 2). on a 70-point rubric, control group students scored 14.67 and the intervention group students scored 18.25 (p=0.2859). table 1. demographic characteristics characteristic control intervention males 1 0 females 11 12 average age 24 24 final course grade a a table 2. pre-test results group n mean std dev std error minimum maximum control 12 14.6667 7.84 2.26 5.0 33.0 intervention 12 18.2500 8.20 2.37 5.0 30.0 p=0.2859 the instructors later administered the post-test after only the intervention group had received the 25-minute instruction session with student peer instruction using the rubric while the control group only had received the introduction to the rubric. the instructors scored the intervention and control groups’ post-tests with the students’ identities concealed using the same scoring rubric. all students’ scores on the post-test following the intervention of control treatments demonstrated that students markedly improved their scores between baseline and spotcheck post-test. the control group improved from 14.67 to 41.75 while the intervention group improved from 18.25 to 43.67, representing 27.08and 25.42-point increases, respectively, on the 70-point scale. this resulting improvement for both groups was consistent with a previous quasi-experimental study.22 unexpected result surprisingly, the initial average spot check post-test scores on a 70-point scale between the intervention and control groups were similar. the average scores on table 3 amounted to 41.75 for the control group and 43.67 for the intervention group. an initial analysis using a paired t-test furthermore suggested no statistical difference between the intervention and 6 peer reviewed article hypothesis, vol. 36, no. 2, 2024 control groups. the instructors closely examined the de-identified scores one-by-one and noticed a bifurcation in the results data. six students in the control group scored noticeably higher than their fellow classmates in the control group. table 3. initial post-test results group n mean std dev std error minimum maximum control 12 41.75 29.18 8.42 10.0 70.0 intervention 12 43.67 21.09 6.09 18.0 70.0 p=0.8554 one instructor interviewed these six students to learn about their experiences. two of these six control group students decided to learn how to use the rubric on their own in order to score highly on a graded assignment using the same skills later in the course. the other four of these control group students had sought peer instruction from their classmates in the intervention group. after removing these contaminated four control group students from the analysis of scores, the overall average scores of intervention and control group post-tests became 43.67 and 27.63, respectively as noted in table 4. these differences between the intervention group and control group approached statistical significance using a paired t-test with a p value of 0.1428. still, the contamination muddled any clear result. table 4. de-contaminated post-test results group n mean std dev std error minimum maximum control 8 27.63 25.57 9.04 10.0 70.0 intervention 12 43.67 21.09 6.09 18.0 70.0 p=0.1428 discussion this randomized controlled trial in a larger dental hygiene degree program tested the hypothesis that an extra 25-minute training on ebp question formulation would result in statistically different scores on a rubric compared to a control group. both control and intervention groups benefited from receiving a 5-minute explanation of the scoring rubric. they scored higher than their baseline pre-test scores on their spot-check post-tests. the control and intervention groups’ scores of 41.75 and 43.67 on a 70-point rubric were similar and not statistically significantly different using a paired t-test. most hypothetical students likely would have wanted the higher average scores of 1.92 points more (almost a 3-point difference on a 100-point scale) by belonging to the intervention group even if we had obtained similar results without removing the contaminated students in the analysis. this difference might raise their score by half a grade in some instances. the average scores were not markedly different and not statistically different, however. perhaps this experiment presents an instance where a lack of statistical significance still has a meaning for competitive students conscious of their grades? limitations 7 peer reviewed article hypothesis, vol. 36, no. 2, 2024 this study had three limitations. first, it took place as the us was emerging from the covid-19 pandemic with its psychosocial, clinical, and logistical preoccupations. dental profession education programs with their sustained close physical proximity to patients during clinical training posed challenges due to the continuous threats caused by the covid-19 pandemic and its reoccurrences.23,24,25 second, while our dental hygiene program was on the larger end of the enrollment continuum nationally, our study would have always had power issues with reaching statistical significance. an unexpected limitation, one that we had thought that we had sufficiently guarded against, was the contamination involving four control group students who collaborated with students in the intervention group. failure caused by contamination contamination between intervention and control groups occurs in an educational or behavioral experiment when members of the control group learn about and adhere to the instruction intended only for the intervention group. the contamination that unfolded in the present study of dental hygiene students possibly reduced the differences in results to suggest no different effect to the intervention group. contamination appears to be a far more widespread issue across randomized controlled studies. one systematic review of behavioral interventions in type 2 diabetics revealed that 13 of the 20 studies examined were vulnerable to contamination between groups and that 45% of the studies did not acknowledge possible contamination. 26 one possible solution would be for all students at a single site to receive only the intervention or the control while all students at another site might receive the opposite treatment in a cluster form of randomized controlled trial.27,28 the small size of the grant for this study would have not afforded a multi-site randomized controlled trial, particularly when the researchers at the university of new mexico thought they had managed to avoid contamination with the study design. a delphi study expert consensus confirmed that while “efforts to avoid or minimize contamination can be incorporated into the design, conduct or analysis of trials. . . such strategies may be ineffective. . . ” [page 196].29 their delphi study further found that, “contamination was thought most likely in trials conducted in settings where respondents worked, lived or interacted closely together.”[page 200].29 congruently, this dental hygiene program provided numerous interactive opportunities among its students. lessons learned this randomized controlled study provides a model for others to adapt to their particular educational or behavioral interventions to gauge the relative benefit of the intervention over the benefits of a control. the control need not be a placebo in the form of simply doing nothing; the standard education or behavior treatment could prove to be a worthy control. this study also offers the possibility of using a less complex quasi-experiment with its preand post-tests, provided the researchers recognize the greater possibilities of confounding in a quasi-experiment. this study tended to still validate the use of active learning. beyond these positive lessons, there are several cautionary lessons to be learned from this study: 1. even with a keen awareness of and experiences in mitigating the dangers of contamination, the researchers did not foresee this instance of contamination. 2. the researchers did not recognize the current undergraduate student experience that caused the student cohort in this dental hygiene program to have many existing regular 8 peer reviewed article hypothesis, vol. 36, no. 2, 2024 interactions among the students in their class due to: • enrollment by some dental hygiene students in a first-semester first-year course for potential future health professionals; • pre-dental career club membership by most students; • pre-existing study group dyads and triads within the classmates; • cross-cutting undergraduate organization (hellenic, sports, social, social justice clubs, etc.) affiliations already existing among these dental hygiene students. 3. while some statistical adjustments can be made in isolated instances of contamination, the small total numbers of students (n=24) in each cohort pushed the usable number of non-contaminated students downward, lowering power so statistical significance could no longer be achieved. 4. in an active learning environment that encourages students to engage regularly in peer instruction, the challenges to avoiding contamination multiply. 5. vigilance must be practiced to guard against contamination in randomized controlled trials when the intervention and control groups are in interactive social, behavioral, or educational contexts. conclusion contamination between the intervention and control groups apparently equalized the results between the groups, thereby neutralizing the hypothesized benefit of a longer, more in-depth training for the intervention group. although the authors made reasonable efforts to avoid contamination between the groups, they did not recognize the pre-existing social and academic relationships of students across the intervention and control groups. additional active learning training among students in the intervention group for 25-minutes beyond a brief 5-minute introduction provided to the control group still appears to have yielded higher scores for the intervention group students. given the importance of the question formulation step in the ebp process and the limitations of this study, it should be replicated at other dental hygiene programs. acknowledgements the authors thank their students for their participation in this study. the authors appreciated the pre-submission critical reading of their manuscript by colleagues: ingrid hendrix, mils, ahip; laura j. hall, mfa. mba; heidi miller sims, and robin a. gatlin, rdh, ms. finally, professional statistician eunice choi’s involvement throughout the study period helped the authors adjust to unexpected forms of contamination. funding this work was supported by the university of new mexico school of medicine scholarship in education allocation committee (seac) grant that was funded beginning may 20, 2019. 9 peer reviewed article hypothesis, vol. 36, no. 2, 2024 credit jon eldredge conceptualized the project, secured funding and human subjects research board approval as principal investigator, conducted and administered the study, validated the rubric, analyzed the results, curated the data, and wrote the original draft of the manuscript. christine nathe, provided the resources in terms of allowing students she supervised in her course to participate in the study, analyzed the results, and edited the final manuscript. references 1. forrest jl, miller sa, overman pr, newman mg. evidence-based decision making: a translational guide for dental professionals. 1st ed. philadelphia: lippincott williams & wilkins; 2009. 2. stanley jl, hanson cl, van ness cj, holt l. assessing evidence-based practice knowledge, attitudes, access and confidence among dental hygiene educators. j dent hyg. 2016 jun;90 suppl 1:42. pmid: 27458317. 3. santiago santiago v, cardenas m, charles al, hernandez e, oyoyo u, kwon sr. evidence-based practice knowledge, attitude, access and confidence: a comparison of dental hygiene and dental students. j dent hyg. 2018 apr;92(2):31-37. pmid: 29739845. 4. laurence b, smith d. evidence-based dental education: suggested course outlines for firstand second-year dental hygiene students. int j dent hyg. 2014 aug;12(3):234. doi: 10.1111/idh.12087. epub 2014 may 15. pmid: 24826992. 5. forrest jl, overman p. keeping current: a commitment to patient care excellence through evidence based practice. j dent hyg. 2013 jan;87 suppl 1:33-40. pmid: 24046340. 6. frantsve-hawley j, clarkson je, slot de. using the best evidence to enhance dental hygiene decision making. j dent hyg. 2015 feb;89 suppl 1:39-42. pmid: 25691026. 7. huang x, lin j, demner-fushman d. evaluation of pico as a knowledge representation for clinical questions. amia annu symp proc. 2006;2006:359-63. pmid: 17238363; pmcid: pmc1839740. 8. ely jw, osheroff ja, ebell mh, bergus gr, levy bt, chambliss ml, evans er. analysis of questions asked by family physicians regarding patient care. west j med. 2000 may;172(5):315-9. doi: 10.1136/ewjm.172.5.315. pmid: 18751285; pmcid: pmc1070879. 9. bjerre lm, paterson nr, mcgowan j, hogg w, campbell cm, viner g, archibald d. what do primary care practitioners want to know? a content analysis of questions asked at the point of care. j contin educ health prof. 2013 fall;33(4):224-34. doi: 10.1002/chp.21191. pmid: 24347101. 10. schardt c, adams mb, owens t, keitz s, fontelo p. utilization of the pico framework to improve searching pubmed for clinical questions. bmc med inform decis mak. 2007 jun 10 peer reviewed article hypothesis, vol. 36, no. 2, 2024 15;7:16. doi: 10.1186/1472-6947-7-16. pmid: 17573961; pmcid: pmc1904193. 11. hoogendam a, de vries robbé pf, overbeke aj. comparing patient characteristics, type of intervention, control, and outcome (pico) queries with unguided searching: a randomized controlled crossover trial. j med libr assoc. 2012 apr;100(2):121-6. doi: 10.3163/1536-5050.100.2.010. pmid: 22514508; pmcid: pmc3324808. 12. eriksen mb, frandsen tf. the impact of patient, intervention, comparison, outcome (pico) as a search strategy tool on literature search quality: a systematic review. j med libr assoc. 2018 oct;106(4):420-431. doi: 10.5195/jmla.2018.345. epub 2018 oct 1. pmid: 30271283; pmcid: pmc6148624. 13. davies ks. davies ks. formulating the evidence based practice question: a review of the frameworks. eblip [internet]. 2011jun.24 [cited 2024 jul.15];6(2):75-0. available from: https://journals.library.ualberta.ca/eblip/index.php/eblip/article/view/9741 14. eldredge j, schiffma, langsjoen jo, jerabek rn. question formulation skills training using a novel rubric with first-year medical students. j med libr assoc. 2021 jan 1;109(1):68-74. doi: 10.5195/jmla.2021.935. pmid: 33424466; pmcid: pmc7772986. 15. anonymous. using rubrics. center for teaching innovation. cornell university. [cited 2024 july 15]. available from: https://teaching.cornell.edu/teaching-resources/assessment-evaluation/using-rubrics 16. allen d, tanner k. rubrics: tools for making learning goals and evaluation criteria explicit for both teachers and learners. cbe life sci educ. 2006;5(3):197-203. doi:10.1187/cbe.06-06-0168 17. cockett a, jackson c. the use of assessment rubrics to enhance feedback in higher education: an integrative literature review. nurse educ today. 2018;69:8-13. doi:10.1016/j.nedt.2018.06.022 18. eldredge jd, hannigan gg. emerging trends in health sciences librarianship. in: wood ms, ed. health sciences librarianship. rowman & littlefield; 2014: 57-83. 19. mclaughlin je, roth mt, glatt dm, et al. the flipped classroom: a course redesign to foster learning and engagement in a health professions school. acad med. 2014;89(2):236-243. doi:10.1097/acm.0000000000000086 20. prince m. does active learning work? a review of the research. journal of engineering education. 2004;93(3):223-231. doi:10.1002/j.2168-9830.2004.tb00809.x 21. kolb da. experiential learning: experience as the source of learning and development. 2nd ed. upper saddle river (nj): pearson education, 2014. 390p. 22. eldredge j, schiffma, langsjoen jo, jerabek rn. question formulation skills training using a novel rubric with first-year medical students. j med libr assoc. 2021;109(1):68-74. doi: 10.5195/jmla.2021.935. pmid: 33424466; pmcid: pmc7772986. 11 https://journals.library.ualberta.ca/eblip/index.php/eblip/article/view/9741 https://teaching.cornell.edu/teaching-resources/assessment-evaluation/using-rubrics peer reviewed article hypothesis, vol. 36, no. 2, 2024 23. bashary nz, levine mh. teaching strategy adaptations in undergraduate dental education during the covid-19 pandemic. j dent educ. 2024;88(6):865-871. doi:10.1002/jdd.13493 24. mather c, colgan l, binnie v, donn j, mckerlie r, bell a. covid-19 adaptations for biomedical teaching and assessment within the undergraduate dental curriculum. adv exp med biol. 2023;1397:43-54. doi:10.1007/978-3-031-17135-2_3 25. santos gnm, da silva hec, leite af, et al. the scope of dental education during covid-19 pandemic: a systematic review. j dent educ. 2021;85(7):1287-1300. doi:10.1002/jdd.12587 26. droske ca, pearson tn, velkovich sj, et al. variations in the design and use of attention control groups in type 2 diabetes randomized controlled trials: a systematic review. curr diab rep. 2023;23(9):217-229. doi:10.1007/s11892-023-01514-2 27. armijo-olivo s, mohamad n, sobral de oliveira-souza ai, de castro-carletti em, ballenberger n, fuentes j. performance, detection, contamination, compliance, and cointervention biases in rehabilitation research: what are they and how can they affect the results of randomized controlled trials? basic information for junior researchers and clinicians. am j phys med rehabil. 2022;101(9):864-878. doi:10.1097/phm.0000000000001893 28. giraudeau b, weijer c, eldridge sm, hemming k, taljaard m. why and when should we cluster randomize?. j epidemiol popul health. 2024;72(1):202197. doi:10.1016/j.jeph.2024.202197 29. howe a, keogh-brown m, miles s, bachmann m. expert consensus on contamination in educational trials elicited by a delphi exercise. med educ. 2007;41(2):196-204. doi:10.1111/j.1365-2929.2006.02674.x 12 hypothesis, vol. 37, no. 1, 2025 reply to bakker et al.: assessing the accuracy of the scite citation classification system requires the same definitions to be used for training as for testing sean c. rife, phda, joshua m. nicholson, phdb, ashish uppala, mbac, domenic rosati, msd ahead of academic relations at research solutions, co-founder of scite, and associate professor of psychology, murray state university, murray, ky, https://www.orcid.org/0000-0002-6748-0841 , srife@researchsolutions.com bchief strategy officer at research solutions and co-founder of scite, brooklyn, ny, https://www.orcid.org/0000-0002-1111-1828 , jnicholson@researchsolutions.com chead of engineering and product, portal innovations, llc, chicago, il, https://www.orcid.org/0000-0001-8748-1465 , ashish.uppala.93@gmail.com dhead of ai, research solutions and doctoral candidate, dalhousie university, brooklyn, ny, https://orcid.org/0000-0003-2666-7615 , drosati@researchsolutions.com, social media cite as: rife sc, nicholson jm, uppala a, rosati d. reply to bakker et al.: assessing the accuracy of the scite citation classification system requires the same definitions to be used for training as for testing. hypothesis. 2025;37(1). doi:10.18060/28018 rife, nicholson, uppala, rosati. all works in hypothesis are licensed under a cc by-nc 4.0 deed attribution-noncommercial 4.0 international. authors own copyright of their articles appearing in hypothesis. readers may copy articles without permission of the copyright owner(s), as long as the author(s) are acknowledged in the copy, and the copy is used for educational, not-for-profit purposes. for any other use of articles, please contact the copyright owner(s). 1 https://www.orcid.org/0000-0002-6748-0841 https://orcid.org/ mailto:srife@researchsolutions.com https://www.orcid.org/0000-0002-1111-1828 https://orcid.org/ mailto:jnicholson@researchsolutions.com https://www.orcid.org/0000-0001-8748-1465 https://orcid.org/ mailto:ashish.uppala.93@gmail.com https://orcid.org/0000-0003-2666-7615 https://orcid.org/ mailto:drosati@researchsolutions.com https://creativecommons.org/licenses/by-nc/4.0/ hypothesis, vol. 37, no. 1, 2025 abstract bakker, theis-mahon, and brown1 recently presented an analysis of citations in scite.ai, a citation analysis platform. they concluded that the scite algorithm inaccurately classified citations, particularly those they regarded as supporting previous work. while we strongly believe that independent assessments of scite’s classifications are valuable, we argue that bakker et al.’s assessment is incorrect, and that their conclusions are due to their definition of what constitutes supporting or contrasting citations. additionally, bakker, et al. restricted their analyses to citations of retracted works in systematic literature reviews, which artificially limits the types of statements that could be considered supporting or contrasting a specific claim. in our reply, we document the rationale for scite’s classification scheme. we also provide examples of how scite classifies different types of citations, as well as how these classifications differ from those presented in bakker et al. in a recent issue of hypothesis, bakker, theis-mahon, and brown1 present an analysis of citation classifications by scite.ai, a platform that facilitates citation analyses using machine learning. they conclude that the accuracy of scite’s classifications of citations is low, based on their own coding of citations. while there is value in this type of analysis, we believe there is a fundamental difference in how citation classifications are being coded and assessed; hence, the discrepancy from their analysis and our own. first, we wish to highlight some positive aspects of bakker et al.’s1 work. as co-founders of scite and practicing researchers, we appreciate any attempt to assess the accuracy of our citation classifications and the overall utility of the product suite scite provides. we have published our own assessment and in-depth treatment of how scite works elsewhere2, yet outside investigations are free from the biased attachment that comes with developing a new tool and are useful in the continual development and improvement of scite. additionally, we think the study of retracted publications is an interesting (and understudied) bibliometric topic; in fact, we have done our own investigation of citations of retracted publications3 and uncovered a number of concerning phenomena (e.g., authors citing retracted works while apparently unaware that they have been retracted). this is not mere dicta on our part we genuinely believe efforts like those of bakker et al.1 are valuable, and wish to see more of them in the future. however, bakker et al.’s1 work has a critical flaw, in that it does not appear to assess what it claims it is assessing. bakker et al. report that a citation was classified as supporting if it was included in a review “without indication of its retracted status or significant critique of the research quality,” and classified as contrasting if it was “described as retracted, or if concerns about the publication or underlying research were discussed” (p. 3). while these are perfectly valid attributes one may attach to a dataset of citation statements, they differ markedly from the annotations in scite’s training dataset. in contrast, scite’s classification schema is very different. as we note2: scite focuses on the authors’ reasons for citing a paper. (...) we consider that for capturing the reliability of a claim, a classification decision into supporting or contrasting must be backed by scientific arguments. the evidence involved in our assessment of citation intent is directed to the factual information presented in the 2 hypothesis, vol. 37, no. 1, 2025 citation context, usually statements about experimental facts and reproducibility results or presentation of a theoretical argument against or agreeing with the cited paper. (p. 888) this approach, which was part of the training given to annotators who created the labels in scite’s training data differs markedly from that of bakker et al.1 in two critical ways: first, noting or failing to note that a cited paper has been retracted does not indicate that the citing paper is presenting supporting or contrasting evidence. it simply looks at whether the authors are aware of the retraction or not, or whether or not they meaningfully engage with the cited work. indeed, most debates or direct refutations in science have little to do with retractions; for example, there are over 64,000 retractions in the scholarly literature at the time of this writing3, while there are over 6.4 million contrasting citations in the scite database. second, while important, retractions are distinct from disagreements and distinct from looking at supporting or contrasting evidence. for example, in another independent study looking at disagreements in the literature, there was no mention of retractions or use of retractions in the definition of what is a scientific disagreement4. to provide concrete examples, table 1 (reprinted from nicholson et al.2) shows a sample of citations and how they are classified, as well as an explanation of each classification. we should note that scite’s approach to citation classification is more precise than sentiment analysis (the automatic classification of statements as generally positive or generally negative), in that it focuses specifically on whether or not new evidence supports or contrasts a claim. as such, statements that refer to another paper with a positive or negative valence are deliberately classified by scite as mentioning (see, for example, the fifth example in table 1). 3 hypothesis, vol. 37, no. 1, 2025 table 1. examples of citation classifications citation statement classification explanation “in agreement with previous work (nicholson et al., 2015), the trisomic clones showed similar aberrations, albeit to a lesser extent (supplemental figure s2b).” supporting “in agreement with previous work” indicates support, while “the trisomic clones showed similar aberrations, albeit to a lesser degree (supplemental figure s2b)” provides evidence for this supporting statement. “in contrast to several studies in anxious adults that examined amygdala activation to angry faces when awareness was not restricted (phan, fitzgerald, nathan, & tancer, 2006; stein, goldin, sareen, zorrilla, & brown, 2002; stein, simmons, feinstein, & paulus, 2007), we found no group differences in amygdala activation.” contrasting “in contrast to several studies” indicates a contrast between the study and studies cited, while “we found no group differences in amygdala activation” indicates a difference in findings. “the amygdala is a key structure within a complex circuit devoted to emotional interpretation, evaluation and response (stein et al., 2002; phan et al., 2006).” mentioning this citation statement refers to phan et al. (2006) without providing evidence that supports or contrasts the claims made in the cited study. “in social cognition, the amygdala plays a central role in social reward anticipation and processing of ambiguity [87]. consistent with these findings, amygdala involvement has been outlined as central in the pathophysiology of social anxiety disorders [27], [88].” mentioning here, the statement “consistent with these findings” sounds supportive, but, in fact, cites two previous studies: [87] and [27] without providing evidence for either. such cites can be valuable, as they establish connections between observations made by others, but they do not provide primary evidence to support or contrast the cited studies. hence, this citation statement is classified as mentioning. “for example, a now-discredited article purporting a link between vaccination and autism (wakefield et al., 1998) helped to dissuade many parents from obtaining vaccination for their children.” mentioning this citation statement describes the cited paper critically and with negative sentiment but there is no indication that it presents primary contrasting evidence, thus this statement is classified as mentioning. 4 hypothesis, vol. 37, no. 1, 2025 in contrast, bakker et al.’s methodology classifies citations based on methodological discussion and acknowledgement of a target article’s retraction. this results in many citations that scite (justifiably, we believe, given the criteria outlined in nicholson et al.) classifies as mentioning being classified as supporting. (a previous study5, which describes the data used in bakker et al. in greater detail, does not provide access to the specific examples used in bakker et al. as such, we present randomly-selected examples of citations to retracted works in systematic reviews.) consider the following citation6 in gatto (2020; emphasis added)7, which scite classifies as mentioning: “interestingly, recent studies have demonstrated that magnesium treatment protects cognitive function and synaptic plasticity by inhibiting gsk-3β in sporadic alzheimer’s disease (ad) model rats (xu et al, 2014). as such, mg 2+ ions are a critical factor in controlling synapse density and plasticity, showing a reduction in aβ-plaques and cognitive deficits in appswe/ps1de9 mice, a transgenic mouse model of ad (li et al, 2014). however, the effects of dietary mg 2+ deficiency on learning and memory are not followed by changes in the spine density and morphology of hippocampal neurons of rodent models (serita et al, 2019), and further examination of this topic is undoubtedly needed. (p. 573)” since this citation does not mention that li et al. is retracted, nor does it contain a significant methodological critique, bakker et al. would classify it as supporting. another example can be found in a review by zhang et al. (20218; emphasis added), citing kaur (2014)9, which scite also classifies as mentioning: “some cross-sectional studies have shown a close association between thyroid disease and metabolic disorders (i.e., metabolic syndrome [mets] and its components) (11–13). mets is characterized by a cluster of abnormal metabolic parameters consisting of insulin resistance, central obesity, type 2 diabetes, impaired glucose tolerance, hyperinsulinemia, and dyslipidemia (14). the global prevalence of mets is between 11.6% and 62.5% (15). (p. 2)” again, because kaur does not mention the fact that zhang et al. was retracted, bakker et al. would classify this citation as supporting. there are also instances where bakker et al. would presumably classify a citation as contrasting while scite would classify it as mentioning. for example, one paper10 cites tsukumo et al. (2007; emphasis added)11, stating: “a total of 37% of the retrieved articles had a positive-citation pattern; meanwhile, 63% had a negative-citation pattern. the most cited article with a negative-citation-pattern was published in 2007 and was retracted in 2016 [24]. thus far, it has received a total of 490 citations and of these, 58 were from after the retraction of the article. (p. 9)” because the citing paper does not present findings contrary to tsukumo et al., scite classifies this citation as mentioning. however, because it acknowledges the retraction of the target article, bakker et al. would classify this citation as contrasting. in summary, we think that while bakker et al.’s1 efforts are laudable, their coding system bears no resemblance to that of the scite training dataset. we therefore strongly disagree with 5 hypothesis, vol. 37, no. 1, 2025 bakker et al.’s conclusion that the accuracy of scite’s classifications is low. while we would all benefit from outside assessments of scite’s accuracy, scite included, this will have to come from future works that rely on a classification scheme that is the same, or at least similar, to that of scite. until such a paper is published, we recommend that readers consult our previous work2 for information on the accuracy of scite’s classifications. disclosure of conflicts all authors are employees/contractors of and shareholders in research solutions, inc., the company that owns scite. references 1. bakker c, theis-mahon n, brown sj. evaluating the accuracy of scite, a smart citation index. hypothesis: research journal for health information professionals. 2023 sep 13;35(2). doi:10.18060/26528 2. nicholson jm, mordaunt m, lopez p, uppala a, rosati d, rodrigues np, grabitz p, rife sc. scite: a smart citation index that displays the context of citations and classifies their intent using deep learning. quant sci stud. 2021 nov 5;2(3):882-98. doi:10.1162/qss_a_00146 3. scite.ai: scite search; [reviewed 2024 may 21; cited 024 may 21]. available from: https://scite.ai/search?hasretraction=true&mode=all 4. lamers ws, boyack k, larivière v, sugimoto cr, van eck nj, waltman l, murray d. meta-research: investigating disagreement in the scientific literature. elife. 2021 dec 24;10:e72737. doi:10.7554/elife.72737 5. brown sj, bakker cj, theis-mahon nr. retracted publications in pharmacy systematic reviews. j med libr assoc. 2022 jan 1;110(1):47. doi:10.5195/jmla.2022.1280 6. li w, yu j, liu y, huang x, abumaria n, zhu y, huang x, xiong w, ren c, liu xg, chui d. elevation of brain magnesium prevents synaptic loss and reverses cognitive deficits in alzheimer’s disease mouse model. mol brain. 2014 dec;7(1):1-20. doi:10.1186/s13041-014-0065-y 7. gatto rg. molecular and microstructural biomarkers of neuroplasticity in neurodegenerative disorders through preclinical and diffusion magnetic resonance imaging studies. j integr neurosci. 2020 sep 30;19(3):571-92. doi:10.31083/j.jin.2020.03.165 8. zhang c, gao x, han y, teng w, shan z. correlation between thyroid nodules and metabolic syndrome: a systematic review and meta-analysis. front endocrinol. 2021 sep 16;12:730279. doi:10.3389/fendo.2021.730279 9. kaur j. a comprehensive review on metabolic syndrome. cardiol res pract. 2014 oct;2014. doi:10.1155/2014/943162 6 https://scite.ai/ https://scite.ai/search?hasretraction=true&mode=all hypothesis, vol. 37, no. 1, 2025 10. stavale r, ferreira gi, galvão ja, zicker f, novaes mr, oliveira cm, guilhem d. research misconduct in health and life sciences research: a systematic review of retracted literature from brazilian institutions. plos one. 2019 apr 15;14(4):e0214272. doi: 10.1371/journal.pone.0214272 11. tsukumo dm, carvalho-filho ma, carvalheira jb, prada po, hirabara sm, schenka aa, araujo ep, vassallo j, curi r, velloso la, saad mj. loss-of-function mutation in toll-like receptor 4 prevents diet-induced obesity and insulin resistance. diabetes. 2007 aug 1;56(8):1986-98. doi:10.2337/db06-1595 7 hypothesis, vol. 37, no. 2, 2025 mid-career researcher ideas for navigating challenges and opportunities margaret a. hoogland, mls, ahipa aassociate professor and clinical medical librarian, university libraries, the university of toledo, toledo, ohio, https://www.orcid.org/0000-0002-9932-3605 , margaret.hoogland@utoledo.edu cite as: hoogland ma. mid-career researcher ideas for navigating challenges and opportunities. hypothesis. 2025;37(2). doi:10.18060/29203 hoogland. all works in hypothesis are licensed under a cc by-nc 4.0 deed attribution-noncommercial 4.0 international. authors own copyright of their articles appearing in hypothesis. readers may copy articles without permission of the copyright owner(s), as long as the author(s) are acknowledged in the copy, and the copy is used for educational, not-for-profit purposes. for any other use of articles, please contact the copyright owner(s). 1 https://www.orcid.org/0000-0002-9932-3605 https://orcid.org/ mailto:margaret.hoogland@utoledo.edu https://creativecommons.org/licenses/by-nc/4.0/ https://creativecommons.org/licenses/by-nc/4.0/ hypothesis, vol. 37, no. 2, 2025 in september 2024, we formed our inaugural editorial board. each member contributed ideas, candid thoughts, and served as a sounding board and think tank for what proved to be a wild year. i cannot thank them enough for serving in this capacity. recently, we welcomed jaclyn morales to our marketing team and carol ng-he as our newest associate editor. jacyln brings a wealth of experience, enthusiasm, and energy to our group. carol will lead our efforts in creating and introducing a new submission category for hypothesis. we encourage you to register on our website, follow us on social media, or connect with us on linkedin so that you can keep up with the latest news. as a mid-career researcher, i am sharing approaches that i take for handling my current and future research projects: 1. at most, i will join two research projects a year – depending on my other commitments, this might be one project. this will allow me to fulfill my obligations and remain a faculty member in good standing1. when i need to decline an opportunity, i will happily share the opportunity with others. 2. capitalize on your strengths. • idea generator i come up with ideas very quickly – but most of the ideas will never leave the imaginary phase. if you struggle to come up with an idea, topic, or question – i love to brainstorm ideas and no idea is bad. i am always thankful for those individuals, who are willing to assist me in culling through my ideas to identify the good ones! • energy and enthusiasm. i get excited very easily and i happen to have a lot of energy. how much or little energy i throw into a situation varies by how much i think the project or group needs it. when i run low on both or either of these, it is time to have a “check-in” conversation so that i can identify the issue(s) and then to find a way to get off a project, rework it, or accept that i just need to keep at it. • browse the journal issue table of contents. we are all “looking” for the gap in the literature or a way to reframe a question so that it might become feasible. one easy way to follow trends is to subscribe to the journal table of contents. if you are not able to attend a conference or if you are not sure research and/or scholarly activities are for you, this is a fantastic way to keep up with the trends in our field. • avid reader. this practice of browsing the issues means that my to-read folder and have-read folders are equally large and they are continuously growing. i have added subfolders within the have-read folder so that i can more easily find articles. most weeks, i read anywhere from three-to-five articles. if you are ever in need of a citation on a topic, let me know – i likely have read something about it. 3. let your ideas hibernate and mature! i have time and so i can now justify messing up, trying new things, declining projects that do not “immediately” grab my attention, and refining a question or idea until it becomes feasible. take advantage of this! when a team of enthusiastic new or emerging researchers asks me for 2 https://orcid.org/0000-0003-0228-7546 https://orcid.org/0009-0007-4755-5483 https://journals.indianapolis.iu.edu/index.php/hypothesis/user/register https://x.com/mlahypothesis?s=20 https://www.linkedin.com/posts/mla-hypothesis_hypothesis-linkedin-activity-7270485061706092545-tm2y hypothesis, vol. 37, no. 2, 2025 advice or invites me to join a project, i try to be extra thoughtful when i decline an opportunity, because i have experienced the pressure and stress they are facing. 4. trust yourself – your instincts are rarely wrong. programs, projects, and campaigning for something new or reworking existing services go better when i follow my instinct. to minimize bias, i try to back up the instinct with data and i rely on feedback from individuals, who can provide insight and hold me accountable. at times, they propel me out of the phase, where self-doubt instead of the known facts are leading the situation. 5. “the more i learn, the less i know.2” this quote highlights how experience, knowledge, interactions, and more influence our perspectives and outlooks. as evidenced by the chart, however, these qualities can lead to self-doubt and delays: initial statement reframed statement the more i learn... -the more experience i accumulate. -the greater knowledge i obtain. -the better i can anticipate the time needed to complete a project. the less i know because... -i am more aware of bias3. -projects take longer to accomplish, because i spend extra time debating approaches. -i realize the importance of reworking existing study designs4. -i pilot test and adjust my data collection instrument multiple times. -i can reanalyze collected data and do more with projects5,6. when looking at this chart, the “less i know” entries are, naturally, greater than “the more i learn entries,” because i am always learning. the “less i know” entries should not stop you from doing research or projects. instead, recognize that every situation is different. move out of your comfort zone and chat with others about your research problems or ideas. the process of preparing and then explaining the problem or situation to someone could result in a new idea or approach. 6. get comfortable with your quirks. unconventional is the best way to describe my life and approach. if you met me on the street and did not immediately identify me as a librarian, i would consider this a successful encounter. i resist long term commitments for a project, services, course, etc. unless i can finagle the option to review and to adjust things on a regular basis. post-covid and to the present, i adopted the mentality of planning with great detail for certain 3 hypothesis, vol. 37, no. 2, 2025 things, but usually i deal with the big “must dos” every day, week, month, and year. progress and especially the word “done or finished” excite me. at times, especially since taking on the role of editor, i have gotten discouraged and frustrated. the counterbalance is the satisfaction i get from changing and improving how i engage my faculty, staff, learners, and community. i thank you for your interest in and support of hypothesis. the editorial team and i look forward to working and interacting with you! references 1. paschke-wood j, gallaspy af. great expectations: adapting a research agenda to changing scholarship requirements. j acad librariansh [internet]. 2022 jul [cited 2024 feb 23];48(4):102527. https://doi.org/10.1016/j.acalib.2022.102527 2. hoogland jl. quote. 2025. 3. zeng l, brignardello-petersen r, hultcrantz m, siemieniuk rac, santesso n, traversy g, et al. grade guidelines 32: grade offers guidance on choosing targets of grade certainty of evidence ratings. j clin epidemiol. 2021 sep;137:163–75. https://doi.org/10.1016/j.jclinepi.2021.03.026 4. eldredge jd. inventory of research methods for librarianship and informatics. j med libr assoc jmla. 2004 jan;92(1):83–90. 5. boscardin ck, sewell jl, tolsgaard mg, pusic mv. how to use and report on p-values. perspect med educ [internet]. 2024 apr 26 [cited 2025 jan 9];13(1):250–4. https://doi.org/10.5334/pme.1324 6. miettunen j, nieminen p, isohanni m. statistical methodology in general psychiatric journals. nord j psychiatry. 2002;56(3):223–8. https://doi.org/10.1080/080394802317607219 4 https://doi.org/10.1016/j.acalib.2022.102527 https://doi.org/10.1016/j.jclinepi.2021.03.026 https://doi.org/10.5334/pme.1324 https://doi.org/10.1080/080394802317607219 peer reviewed article hypothesis, vol. 37, no. 2, 2025 the impact of mental health first aid (mhfa) training in libraries: a pilot study mary margaret thomas, mlis (she/her/hers)a aclinical education librarian, hardin library for the health sciences, university of iowa, iowa city, ia,https://www.orcid.org/0000-0003-2197-5440 , mary-m-thomas@uiowa.edu cite as: thomas mm. the impact of mental health first aid (mhfa) training in libraries: a pilot study. hypothesis. 2025;37(2). doi:10.18060/29133 thomas. all works in hypothesis are licensed under a cc by-nc 4.0 deed attribution-noncommercial 4.0 international. authors own copyright of their articles appearing in hypothesis. readers may copy articles without permission of the copyright owner(s), as long as the author(s) are acknowledged in the copy, and the copy is used for educational, not-for-profit purposes. for any other use of articles, please contact the copyright owner(s). 1 https://www.orcid.org/0000-0003-2197-5440 https://orcid.org/ mailto:mary-m-thomas@uiowa.edu https://creativecommons.org/licenses/by-nc/4.0/ https://creativecommons.org/licenses/by-nc/4.0/ peer reviewed article hypothesis, vol. 37, no. 2, 2025 abstract objective: this study seeks to explore how mental health first aid (mhfa) training impacts librarians working in both academic health sciences and public libraries. specifically, it aims to investigate how mhfa training affects their mental health literacy, professional confidence, and their ability to provide mental health information and resources. methods: this qualitative research study explores the perspectives and experiences of seven librarians from two different types of libraries, an academic health sciences library and a public library system. semi-structured interviews were conducted to gather participants’ experiences and insights regarding mhfa training. this project received irb approval. results: mental health first aid training enhances participants’ awareness of mental health challenges, boosts their professional confidence, and improves mental health literacy. in addition to being a skills-based training that teaches de-escalation techniques, it also helps reduce stigma surrounding mental health conditions. after completing mhfa training, participants report increased confidence in supporting individuals experiencing mental health challenges and assisting them in obtaining appropriate help. conclusion: mental health first aid is a valuable workplace training for both academic and public libraries, benefiting all types of library staff. it enhances awareness of mental health conditions and reduces stigma while simultaneously increasing mental health literacy and professional confidence. additionally, it equips participants with the skills and confidence to appropriately assist individuals, both library staff and users, experiencing mental health challenges. furthermore, mental health first aid training cultivates greater empathy and compassion, contributing to a more supportive workplace culture that is welcoming and inclusive. content warning: this paper discusses mental illness and suicide. introduction mental illness is very common. according to the national institute of mental health, “more than one in five u.s. adults live with a mental illness (59.3 million in 2022; 23.1% of the u.s. adult population)”1(p1). however, even though it is very prevalent, henderson et al., report that "more than 70% of people with mental illness receive no treatment due to a combination of discrimination, stigma, and access to health care"2(p777). furthermore, the covid-19 pandemic led to an increase in mental health conditions and exacerbated existing ones. the world health organization reported that the covid-19 pandemic triggered a 25% increase in prevalence of anxiety and depression worldwide3. according to corrigan and rao, who study mental illness stigma, "people with mental illness have long experienced prejudice and discrimination"4(p1). they go on to explain stigma as, "negative stereotypes, such as notions of dangerousness or incompetence, often associated with mental illness, can be harmful to people living with mental illnesses"4(p1-2). concealing a mental illness, especially long-term, can have negative impacts. this statement clarifies the challenges of stigma and thus the importance of feeling comfortable disclosing as well as 2 peer reviewed article hypothesis, vol. 37, no. 2, 2025 discussing mental health conditions. mental health training programs, of which there are several kinds, are an effective way to reduce stigma5,6. mental health first aid (mhfa) is generally considered the gold standard of mental health training programs because it is an evidence-based, early-intervention course in addition to one of the most comprehensive mental health training programs available7. mhfa is a mental health training program for laypeople. it is a standardized educational program developed to address mental health problems and prevent suicide by educating the public to destigmatize mental illness as well as providing de-escalation techniques. mental health first aid is often likened to cpr as both are skills-based training programs; however, mhfa focuses specifically on mental health7,8. it is a required eight-hour minimum educational training program taught by certified instructors. mental health first aid was originally developed and implemented in australia but has since spread worldwide8. there are different versions and add-ons to mhfa. for example, higher ed is one version which focuses on supporting college students, the youth version is specific to adolescents, and a popular add-on to mhfa is narcan training. mhfa includes many interactive learning opportunities that make applying the skills learned in a real-life situation easier such as role-playing and scenarios. mhfa participants, called mental health first aiders, learn the five-step algee action plan which is used during a crisis. the algee action plan is detailed below in figure 1. figure 1. five-step algee action plan mental health first aid addresses many misconceptions, such as the belief that asking someone if they are thinking about killing themself will cause them to act on it9,10. coupled with dispelling common myths, it also teaches mental health first aiders to ask difficult questions such as, "are you thinking of hurting yourself?" to people in distress9. while algee, the five-step action plan, was designed to be used in crisis situations, it can be used in non-emergency situations as well. typically, instructors bring algee to the training, a stuffed koala wearing a mhfa shirt. see figure 2. he is the mascot but also helps mental health first aiders remember the mnemonic for the action plan. 3 peer reviewed article hypothesis, vol. 37, no. 2, 2025 figure 2. algee koala the algee action plan was developed to guide mental health first aiders, typically laypeople, in providing effective help and support to individuals experiencing a mental health crisis. the first step of algee, "assess for risk of suicide or harm," helps determine the urgency of the situation and if immediate action is needed. this step is crucial but can be difficult especially when asking someone if they are thinking of killing themself. however, this step gets easier with practice. the next four steps are more natural for library professionals, in the author’s opinion, as they are already a part of their professional skill set. "listen nonjudgmentally" is important to make the other person feel comfortable and allow them to say as much as possible. oftentimes, if a person feels judgment or senses stigma, they may choose not to disclose their true feelings or share their experiences. the "giving reassurance and information" step is important to offer support and validate what they are experiencing. "encouraging appropriate help" is vital since so many people with mental illness do not seek treatment but to also assist them in finding appropriate treatment. these two steps correspond with providing consumer health information, a skill possessed by many librarians. the final step is to "encourage self-help or self-care"5. it is essential that mental health first aiders prioritize their own mental health when supporting an individual experiencing any type of mental health crisis, not only serious crises such as individuals considering suicide or self-harm. providing support can be exhausting. even less stressful mental health challenges necessitate self-care, such as supporting a friend going through a divorce. 4 peer reviewed article hypothesis, vol. 37, no. 2, 2025 efficacy of mhfa training numerous studies, including systematic reviews, have been conducted to evaluate the effectiveness of this training as an intervention. currently, the majority of these studies focus on populations such as healthcare professionals (e.g., pharmacists and nurses) or specific settings such as the mhfa youth training for adolescents. advincula et al., found in their systematic review that, "pharmacists who completed mhfa training were found to engage in conversations about mental health more frequently, offer compassionate support, and demonstrate a willingness to assist individuals struggling with mental health crises"11(p2). the most frequently reported outcomes following mhfa training are increased confidence regarding mental health issues and reduced stigma11. connection to libraries in libraries, mental health challenges come in a variety of situations and are not solely an issue for public-facing staff12-14. there is a widespread belief that mental health challenges in libraries are limited to public-facing roles, such as dealing with a disruptive patron at the service desk or supporting a student in psychological distress15-17. while these examples are valid, it is equally important to recognize that mental health challenges arise in a variety of other situations within library settings. potential scenarios for applying mhfa skills in non-public-facing situations include listening to a coworker who is struggling and helping them identify appropriate help, or a supervisor encouraging their employee to utilize the employee assistance program (eap). it is common for workplaces, including libraries, to offer a variety of training sessions such as cpr and stop the bleed to better equip staff for emergency situations. mental health challenges occur regularly in all types of libraries, both for staff as well as users. navigating these situations can be intimidating, and library staff may not feel comfortable assisting an individual with appropriate mental health help or treatment in the same way they would when assisting with a physical health condition. mhfa training can help with this, since it increases participants’ knowledge of mental health conditions, professional confidence, and mental health literacy. it also reduces mental illness stigma. after completing mhfa training, participants feel more confident helping someone experiencing a mental health challenge11. mhfa teaches participants that a person experiencing a panic attack should be treated similarly to a person who broke their arm since they are both health conditions and require treatment. in a randomized control trial, kitchner and jorm found “improvements in participants’ mental health literacy, including knowledge, stigmatizing attitudes, confidence and help provided to others"18(p1) post mhfa training. many people do not feel comfortable disclosing or discussing their mental illness with a supervisor or coworker due to stigmatization and fear19,20. in their article academic librarians’ experiences and perceptions on mental illness stigma and the workplace, burns and green discuss the results of their study on academic librarians living with mental illness in their workplaces (academic libraries). they found that due to stigmatization and shame, "many people felt that they could not share their mental health diagnosis with their coworkers or supervisors"17(p649). unfortunately burns and green’s most obvious finding was, "there is a fear of disclosing one’s own mental illness in the academic library environment"17(p653). burns 5 peer reviewed article hypothesis, vol. 37, no. 2, 2025 and green’s findings underscore the importance of training programs like mhfa that provide education on mental health conditions and teach non-stigmatizing language. supervisors and administrators have a huge impact on their employees’ lives and by appropriately handling mental health challenges, they create positive work environments and foster overall well-being in their libraries. many types of libraries support students, particularly academic libraries serving large undergraduate populations, and can help students navigate and access healthcare. in her article libraries as a mental health resource, lozano makes the connection as to how uniquely situated academic libraries are in supporting students’ mental health. she explains that libraries are often thought of as information hubs and are already established as free sources of academic support. since the library is already used for research and study, information obtained at the library has more gravitas and is more likely to be trusted14(p67). a considerable number of colleges and universities have student counseling centers where students can access behavioral health treatments. lozano suggests that libraries create resources such as flyers with 988, the suicide and crisis lifeline, along with other resources and services for students14. this highlights the importance of libraries utilizing their information expertise to provide resources and better support the communities they serve. public libraries are at the forefront in implementing mental health first aid training. this is unsurprising given that many public libraries act as safety nets for their communities and that their staff interact regularly with patrons living with serious and persistent mental illness21-23. throgmorton, a public librarian, recommends mental health first aid as a workplace training for libraries, noting that "while mental health first aid may not work miracles, it can deescalate library disruptions"24(p23). study purpose to date few studies exist examining the impact of mental health first aid training on librarians. the purpose of this study is to investigate the benefits and remaining gaps of mhfa training in libraries, whether and how librarians are utilizing the training in their jobs, and whether the training is sufficient for librarians to provide mental health resources to their communities. by understanding these effects, this study hopes to inform future implementation of mhfa training programs in library settings to better support both library staff and patrons. theoretical framework the theoretical framework that most informs this research study is mental health literacy. mental health literacy is a subcategory of health literacy and is a major component of the mental health first aid program. health literacy focuses on improving people’s ability to understand medical information and make effective medical decisions, primarily concerning physical health conditions. jorm et al., defines mental health literacy as "knowledge and beliefs about mental disorders which aid their recognition, management or prevention"11(p182). jorm et al., later built on the previous definition by adding the following components: 6 peer reviewed article hypothesis, vol. 37, no. 2, 2025 knowledge of how to prevent mental disorders; recognition of when a disorder is developing; knowledge of help-seeking options and treatments available; knowledge of effective self-help strategies for milder problems; and first aid skills to support others who are developing a mental disorder or are in a mental health crisis25(p231). consequently, improved mental health literacy strengthens help-seeking behaviors, resulting in more people getting help. methods this study included seven participants from two contrasting libraries within the state of colorado: four participants were from an academic health sciences library and three participants were from a public library system. the strauss health sciences library is located on the university of colorado’s anschutz medical campus and serves a large public research university situated in an urban area. douglas county libraries is a public library system with eight libraries located in suburban areas within douglas county. participants were invited to take part in this study via emails sent to their respective libraries, as the author was aware those libraries had previously held mental health first aid training programs. requirements for participants were as follows: hold a master’s degree in library and information science, must be working as a library professional, and have taken mhfa training within the past five years. this study employed convenience sampling to select participants and included all individuals regardless of whether they held public-facing or behind-the-scenes positions. see figures 3 and 4 for participant demographics. figure 3. participants’ library type 7 peer reviewed article hypothesis, vol. 37, no. 2, 2025 figure 4. participants’ position type semi-structured interviews lasting 30-45 minutes were conducted with each participant during the spring of 2022 via zoom. the interviews were designed in sections, beginning with simpler questions to ease participants into the interview, with questions getting more difficult as time went on. the first section began with participants sharing factual details, such as when they completed mhfa, the format of the training, and an example of something they learned. in the next section, participants were asked to describe their job, how mhfa has influenced their work as librarians, and whether they had developed any digital or physical mental health resources, such as libguides or bookmarks. participants were also asked whether their library offered physical health resources, in order to compare with mental health resources. then they were asked how mhfa has shifted their professional confidence, to rate their professional confidence after mhfa, and to explain what confidence means to them. the last section covered the gaps of mhfa training and whether interviewees would recommend mhfa to other librarians, along with anything else they wished to share. see figure 5 below for a complete list of interview questions. this is a qualitative study. therefore, the author performed a content analysis of the qualitative data and a descriptive analysis of the quantitative data. excel and word were used for the descriptive analysis visualizations. the institutional review board at emporia state university granted permission to conduct this research study (protocol id: 22044). 8 peer reviewed article hypothesis, vol. 37, no. 2, 2025 figure 5. participants’ interview questions results unanimously, participants agreed that mental health first aid training was beneficial. they were also grateful that the training came to their library, making it more convenient for them to attend. additionally, every single participant would recommend it to others without reservations. 9 peer reviewed article hypothesis, vol. 37, no. 2, 2025 themes during the content analysis of the interview transcripts five themes emerged. the first theme identified in this study is that mental health first aid is a skills-based training. participants learn the five-step action plan, algee, and de-escalation techniques for crisis situations. in addition to the algee action plan, participants repeatedly mentioned the mhfa manual as a most liked item. they found it to be a useful resource and appreciated being able to keep it. the second theme that emerged is that mhfa supports building better relationships. it improves listening skills which enhances awareness of users’ needs. it also elevates non-judgmental behavior which is important for gaining trust and destigmatization. the third and fourth themes are interconnected. the third theme that stood out is that mhfa plays a key role in raising awareness about mental illness and mental health challenges. the fourth theme revealed increased mental health literacy post mhfa. these themes are interwoven because increased mental health awareness typically leads to increased mental health literacy. furthermore, enhanced mental health literacy fosters more effective professional dialogues among supervisors, employees, and coworkers, as well as enriching personal relationships outside the workplace. the fifth theme revealed that the majority of participants applied the knowledge and skills gained from mhfa training in their personal lives while supporting friends and family. participants shared that the most frequently used skills and tools included drawing on their background knowledge of mental illnesses, using appropriate and educated language, recognizing signs of individuals in distress, navigating mental health information and treatment options, and providing support to individuals experiencing mental health conditions. professional confidence mental health first aid elevates professional confidence regarding mental health. six out of seven participants saw an increase in their professional confidence regarding mental health treatments and resources post mhfa training. the seventh participant already had high professional confidence prior to mhfa training, thus there was no change. unfortunately, the five-point likert scale was too limiting, and a seven-point likert scale would have been preferable, since two participants responded "confident" which was not available on the five-point scale. accordingly, only the participants’ textual labels of confidence (e.g., "more confident") were included. see figure 6. figure 6. connection between participants’ professional confidence and position type 10 peer reviewed article hypothesis, vol. 37, no. 2, 2025 among the six librarians who reported an increase in their overall professional confidence after mhfa training, scores were related to their position type. all librarians with public-facing roles responded, "more confident" while the behind-the-scenes librarians responded "confident." both participants who responded "confident" clarified what they meant with hypothetical scenarios involving assisting patrons at the service desk. one participant who responded "confident" explained what professional confidence meant to them as: yeah, so that i was thinking, it would mean that i would know like right off the top of my head, like some resources, probably web-based resources to go to, and that i just know them off the top of my head or maybe a specific like if there was like a database that focused on mental health. so right now, i’d be using my librarian skills like from a known association or government entity, not being able to rattle off a list of them that i learned. it is noteworthy that professional confidence levels after mhfa training vary depending on the type of position held. the author speculates that librarians in public-facing roles reported higher confidence levels because they possess a stronger skill set as practicing information professionals regarding assisting patrons with mental health information, resources, and treatments. however, more research is needed to determine the true cause or if this is a coincidence. additionally, the reduced levels of professional confidence reported by participants who selected "confident" may be attributed to the researcher’s lack of clarity regarding the scope and definition of professional confidence, since both participants who responded "confident" elaborated on their responses by referencing scenarios in which they assisted patrons at the service desk. accordingly, the author recommends that future researchers explicitly acknowledge that mental health challenges can arise in various library settings, including behind-the-scenes environments, and are not confined solely to public-facing roles. shortcomings this study examined whether participants used mhfa to create mental health resources, digital (e.g., libguides) or physical (e.g., bookmarks), in their current positions. unfortunately, due to several factors, none of the librarians interviewed created mental health resources. although some librarians were constrained by their current roles and unable to develop mental health resources themselves, these resources were being created elsewhere within their library system by other staff members. others felt it was unnecessary for their patrons, given the availability of nearby mental health resources such as a crisis center. additionally, some librarians noted that their health sciences library already had appropriate resources within the collection. finally, those who were not public-facing did not consider it part of their position. moreover, the use of convenience sampling may have introduced bias, as participants could have self-selected based on favorable perceptions of mhfa. despite these shortcomings, the author still recommends mhfa as a worthwhile training program for its effectiveness in reducing stigma. lessons learned another finding from this study is that the mhfa training format matters. one participant reported feeling "emotionally burned out" after the training and requested trigger warnings 11 peer reviewed article hypothesis, vol. 37, no. 2, 2025 before tough subjects. they explain below why an all-day training session was too much: it was underestimated how emotionally difficult it (mhfa) is. i know it took me probably two to three days to get out of the emotional burnout or fog. the videos are so well done, the exercises are so well done that you really get involved in this. and having an all-day thing you deal with personal memories and maybe personal experiences. this training deals with serious emotional content, includes many realistic scenario videos, and may trigger and cause difficult memories to resurface for some participants. therefore, based on participant feedback, the author recommends changing the format of the training by breaking it up into multiple sessions on different days, offering a quiet room for breaks, and using trigger warnings. furthermore, the quality of the instructor matters. participants reported an overall more positive experience and held a more favorable perception of mhfa training when the instructor was enthusiastic and engaged, compared to those who were indifferent. moreover, given that all participants were librarians, they particularly valued instructors who recognized and addressed the unique challenges of libraries, an insight that most instructors lacked. additionally, participants were questioned about ways to improve the training. a few participants felt there was not enough role play practice in mhfa. they wanted more practice including scripted language to use in real-life situations or practice focusing on a couple of specific situations. ryan dowd’s homelessness training, a series of online training courses aimed at improving library professionals’ interactions with individuals experiencing homelessness, were given as an example since this training gives scripted responses for certain situations26. figure 7. participants’ perspectives on mhfa training conclusion given the small scale and qualitative nature of this study, further research is recommended to explore the impact of mental health first aid in library settings. there is a need for more awareness of mental health first aid and other mental health training programs in addition to implementing these training programs in libraries, especially academic libraries. currently, 12 peer reviewed article hypothesis, vol. 37, no. 2, 2025 most libraries implementing mhfa are public libraries; however, this training program teaches beneficial skills that are applicable in other types of libraries as well. even though participants in this study did not use their mhfa skills to create mental health resources, the program remains valuable and with broad applications. this is especially important given how pervasive stigma is, as mhfa serves as an effective tool in combating it. since mental health challenges occur outside of work, knowing how to handle these situations anywhere is beneficial. as previously stated, many mental health first aiders, including participants in this study, use their mhfa skills in their personal lives, listening, encouraging, and assisting friends and family with getting help. consequently, the widespread use of mental health first aid skills by participants has the potential to generate a positive ripple effect, enhancing overall well-being across the broader community. overall, this study establishes that mental health first aid is beneficial and relevant training for academic and public libraries. mhfa training helps libraries educate staff, destigmatize mental illness, and provide de-escalation techniques to better meet their community’s needs. the covid-19 pandemic underscored the need for greater advocacy and interventions regarding mental health. mental health first aid increases mental health awareness and literacy, fostering greater empathy and compassion for individuals experiencing mental health challenges. additionally, it enhances librarians’ professional confidence regarding mental health, contributing to a more supportive and inclusive workplace culture. mhfa supports an integrative approach to well-being by addressing mental health conditions and improves parity between mental and physical health. library professionals routinely interact with individuals, both library staff and users, experiencing mental health challenges. the goal is to ensure these individuals receive the same level of empathy, support, information, and resources as those experiencing physical health conditions. moreover, mhfa is a natural extension of librarians’ skill sets since librarians are information professionals who possess existing expertise in navigating accurate and reliable information. libraries strive to be welcoming and inclusive. mental health first aid helps create both a psychologically welcoming and physically safe environment for library users and staff. acknowledgments the author gratefully acknowledges janna lawrence, jennifer deberg, and sarah andrews for their valuable assistance with this research paper. special thanks are also extended to susan lessick, emily vardell, shanda hunt, and kevin thomas for their generous support throughout the research project. this research was developed through the training and support provided by the mla research training institute (rti) and funded in part by the institute of museum and library services (imls). references 1. national institute of mental health. mental illness statistics. [cited 2025 may 6]. available from: https://www.nimh.nih.gov/health/statistics/mental-illness. 2. henderson c, evans-lacko s, thornicroft g. mental illness stigma, help seeking, and 13 https://www.nimh.nih.gov/health/statistics/mental-illness peer reviewed article hypothesis, vol. 37, no. 2, 2025 public health programs. am j public health. 2013;103(5):777-80. doi:10.2105/ajph.2012.301056 3. world health organization. covid-19 pandemic triggers 25% increase in prevalence of anxiety and depression worldwide. [cited 2025 june 6]. available from: https://www.who.int/news/item/02-03-2022-covid-19-pandemic-triggers-25-increase-inprevalence-of-anxiety-and-depression-worldwide. 4. corrigan pw, rao d. on the self-stigma of mental illness: stages, disclosure, and strategies for change. can j psychiatry. 2012;57(8):464-9. doi:10.1177/070674371205700804 5. kitchener ba, jorm af. mental health first aid training for the public: evaluation of effects on knowledge, attitudes and helping behavior. bmc psychiatry. 2002;2:10. doi:10.1186/1471-244x-2-10 6. lamontagne ad, martin a, page km, reavley nj, noblet aj, milner aj, et al. workplace mental health: developing an integrated intervention approach. bmc psychiatry. 2014;14:131. doi:10.1186/1471-244x-14-131 7. national council for mental wellbeing. mental health first aid. [cited 2025 may 6]. available from: https://www.mentalhealthfirstaid.org/. 8. national council for mental wellbeing. about mhfa: what is mental health first aid? [cited 2025 may 6]. available from: https://www.mentalhealthfirstaid.org/about/. 9. monette m. the boundaries of mental health first aid. cmaj. 2012;184(17):e891-2. doi:10.1503/cmaj.109-4298 10. hadlaczky g, hökby s, mkrtchian a, carli v, wasserman d. mental health first aid is an effective public health intervention for improving knowledge, attitudes, and behaviour: a meta-analysis. int rev psychiatry. 2014;26(4):467-75. doi:10.3109/09540261.2014.924910 11. advincula tj, abubakar m, carandang rr. effects of mental health first aid training in hospital and community pharmacists: a systematic review. j am pharm assoc. 2025; 65(3):1-10. doi:10.1016/j.japh.2025.102344 12. gibbons p. librarians as second responders: a guide to mental health resources for librarians. inf today. 2023;40(5):16-9. 13. grimes nd. navigating mental health and wellness in communities: a review of the literature and implications for libraries, librarians, and library workers. ref libr. 2024;65(3/4):81-104. doi:10.1080/02763877.2024.2394070 14. lozano h. libraries as a mental health resource. public serv q. 2022;18(1):65-7. doi:10.1080/15228959.2021.2015051 15. harrington s. the perils of public service: emotional labor and mental illness in library employees. in: dube m, wade c, editors. lis interrupted: intersections of mental illness and 14 https://doi.org/10.2105/ajph.2012.301056 https://www.who.int/news/item/02-03-2022-covid-19-pandemic-triggers-25-increase-in-prevalence-of-anxiety-and-depression-worldwide https://www.who.int/news/item/02-03-2022-covid-19-pandemic-triggers-25-increase-in-prevalence-of-anxiety-and-depression-worldwide https://doi.org/10.1177/070674371205700804 https://doi.org/10.1186/1471-244x-2-10 https://doi.org/10.1186/1471-244x-14-131 https://www.mentalhealthfirstaid.org/ https://www.mentalhealthfirstaid.org/about/ https://doi.org/10.1503/cmaj.109-4298 https://doi.org/10.3109/09540261.2014.924910 https://doi.org/10.1016/j.japh.2025.102344 https://doi.org/10.1080/02763877.2024.2394070 https://doi.org/10.1080/15228959.2021.2015051 peer reviewed article hypothesis, vol. 37, no. 2, 2025 library work. sacramento (ca): library juice press; 2021:153-165. 16. rondinelli m. what’s missing in conversations about libraries and mental illness. in the library with the lead pipe. 2024 june 19. https://www.inthelibrarywiththeleadpipe.org/2024/conversations-about-libraries/ 17. burns e, green kec. academic librarians’ experiences and perceptions on mental illness stigma and the workplace. c&rl. 2019;80(5):638-57.doi:10.5860/crl.80.5.638 18. kitchener ba, jorm af. mental health first aid training in a workplace setting: a randomized controlled trial [isrctn13249129]. bmc psychiatry. 2004;4:23. doi:10.5860/crl.80.5.638 19. bennett a. full disclosure? issues around disclosing mental illness in an academic library workplace. in: wade c, dub m, editors. lis interrupted: intersections of mental illness and library work. sacramento (ca): library juice press; 2021:87-104. 20. corrigan p, matthews a. stigma and disclosure: implications for coming out of the closet. j ment health. 2003;12(3):235-48. doi:10.1080/0963823031000118221 21. pressley t. public libraries, serious mental illness, and homelessness: a survey of public librarians. public libr q. 2017;36(1):61-76. doi:10.1080/01616846.2017.1275772 22. peet l. mental wellness: libraries across the country are finding new ways to support patrons experiencing mental health challenges—as well as the staff who serve them. libr j. 2019 june 11. https://www.libraryjournal.com/story/mental-wellness 23. stringer h. libraries as mental health hubs. monit psychol. 2020 april 1;51(3). https://www.apa.org/monitor/2020/04/libraries-health-hubs 24. throgmorton k. mental health first aid: training librarians to help patrons in crisis. am libr. 2017 march 1. https://americanlibrariesmagazine.org/2017/03/01/mental-health-first-aid/ 25. jorm af. mental health literacy: empowering the community to take action for better mental health. am psychol. 2012;67(3):231-43. doi:10.1037/a0025957 26. dowd r. homeless training. [cited 2025 may 6]. available from: https://www.homelesstraining.com/. 15 https://www.inthelibrarywiththeleadpipe.org/2024/conversations-about-libraries/ https://doi.org/10.5860/crl.80.5.638 https://doi.org/10.5860/crl.80.5.638 https://doi.org/10.1080/0963823031000118221 https://doi.org/10.1080/01616846.2017.1275772 https://www.libraryjournal.com/story/mental-wellness https://www.apa.org/monitor/2020/04/libraries-health-hubs https://americanlibrariesmagazine.org/2017/03/01/mental-health-first-aid/ https://doi.org/10.1037/a0025957 https://www.homelesstraining.com/ peer reviewed article hypothesis, vol. 37, no. 2, 2025 update and expansion of an interprofessional education search strategy validation study for nlm pubmed, ovid medline, and elsevier embase luke barron, mslsa, sophie nachman, msis, mphb, rebecca carlson, mlsc aclinical librarian, health sciences library, university of north carolina at chapel hill, chapel hill, north carolina, https://www.orcid.org/0009-0006-5288-193x , luke_barron@unc.edu bgraduate assistant, health sciences library, university of north carolina at chapel hill, chapel hill, north carolina,https://www.orcid.org/0000-0002-9682-7357 , asnachman@outlook.com cpharmacy librarian, health sciences library, university of north carolina at chapel hill, chapel hill, north carolina, https://www.orcid.org/0000-0002-4380-8435 , rcarlson@unc.edu cite as: barron l, nachman s, and carlson r. update and expansion of an interprofessional education search strategy validation study for nlm pubmed, ovid medline, and elsevier embase. hypothesis. 2025;37(2). doi:10.18060/28745 barron, nachman, carlson. all works in hypothesis are licensed under a cc by-nc 4.0 deed attribution-noncommercial 4.0 international. authors own copyright of their articles appearing in hypothesis. readers may copy articles without permission of the copyright owner(s), as long as the author(s) are acknowledged in the copy, and the copy is used for educational, not-for-profit purposes. for any other use of articles, please contact the copyright owner(s). 1 https://www.orcid.org/0009-0006-5288-193x https://orcid.org/ mailto:luke_barron@unc.edu https://www.orcid.org/0000-0002-9682-7357 https://orcid.org/ mailto:asnachman@outlook.com https://www.orcid.org/0000-0002-4380-8435 https://orcid.org/ mailto:rcarlson@unc.edu https://creativecommons.org/licenses/by-nc/4.0/ peer reviewed article hypothesis, vol. 37, no. 2, 2025 introduction interprofessional education (ipe) has grown exponentially since the seminal 2010 world health organization framework1,2. to meet the ongoing needs of ipe scholars, an interprofessional research team created and validated pubmed ipe search hedges3 using relative recall methods4. the authors developed a gold standard set from ipe systematic reviews, then created candidate search hedges for pubmed. these were tested for total result count and relative recall, or the percentage of relevant articles retrieved from the gold standard. three hedges were ultimately retained as recommended search options: a narrow phrase search, a title search, and a broad keyword search; they achieved 75-95% relative recall. these offer search options for librarians and researchers, from a narrow, focused search for quick retrieval of useful publications to a broad, high-recall search for literature reviews3. however, they are limited to pubmed searching. methods this study extended the previous study by translating these pubmed searches for use in ovid medline and elsevier embase and repeating the validation process. the pubmed gold standard set, composed of pubmed identification numbers (pmids), was usable without modification in ovid medline, but required adaptation for embase via conversion of pmids to embase ids followed by removal of duplicate entries. the previous search syntax was manually translated from nlm pubmed to ovid and embase formats. the team measured the translated search strategies against the gold standard set to record their relative recall. results the ovid medline and elsevier embase translations of the search hedges were successfully validated and all search strategies are presented in table 1. 2 peer reviewed article hypothesis, vol. 37, no. 2, 2025 table 1. validated search strategies for each database platform. narrow phrase search (pubmed) (ipe[tiab] or "interprofessional education"[mesh] or "interprofessional education"[tw] or "inter-professional education"[tw] or "interdisciplinary education"[tw] or "inter-disciplinary education"[tw] or "multiprofessional education"[tw] or "multidisciplinary education"[tw] or "multi-professional education"[tw] or "multi-disciplinary education"[tw]) narrow title search (pubmed) (interprofessional*[ti] or inter-professional*[ti] or multiprofessional*[ti] or multi-professional*[ti] or interdisciplinary[ti] or inter-disciplinary[ti] or multidisciplinary[ti] or multi-disciplinary[ti] or multioccupational[ti] or interoccupational[ti] or inter-occupational[ti]) and (student*[ti] or educat*[ti] or learn*[ti] or train*[ti] or teach*[ti] or curricul*[ti] or simulat*[ti] or school*[ti] or course*[ti]) broader keyword search (pubmed) (ipe[tiab] or "interprofessional education"[mesh]) or ((interprofessional*[tiab] or inter-professional*[tiab] or interdisciplinary[tiab] or inter-disciplinary[tiab] or multidisciplinary[tiab] or "interprofessional relations" [mesh]) and (student[tiab] or students[tiab] or educate[tiab] or educating[tiab] or educator[tiab] or educators[tiab] or education[tiab] or instructor[tiab] or instructors[tiab] or instruction[tiab] or teaching[tiab] or training[tiab] or trainee[tiab] or trainees[tiab] or curriculum[tiab] or curricula[tiab] or simulation[tiab] or simulations[tiab] or shadowing[tiab] or "clinical practicum*"[tiab] or "clinical placement*"[tiab] or "experiential learning"[tiab] or teamwork[tiab] or "education, professional"[mesh] or "clinical competence" [mesh])) narrow phrase search (ovid medline) "interprofessional education".mp. or "inter-professional education".mp. or "interdisciplinary education".mp. or "inter-disciplinary education".mp. or "multiprofessional education".mp. or "multidisciplinary education".mp. or "multi-professional education".mp. or "multi-disciplinary education".mp.) 3 peer reviewed article hypothesis, vol. 37, no. 2, 2025 narrow title search (ovid medline) (interprofessional*.ti. or inter-professional*.ti. or multiprofessional*.ti. or multi-professional*.ti. or interdisciplinary.ti. or inter-disciplinary.ti. or multidisciplinary.ti. or multi-disciplinary.ti. or multioccupational.ti. or interoccupational.ti. or inter-occupational.ti.) and (student*.ti. or educat*.ti. or learn*.ti. or train*.ti. or teach*.ti. or curricul*.ti. or simulat*.ti. or school*.ti. or course*.ti.) broader keyword search (ovid medline) (ipe.tw. or exp "interprofessional education"/) or ((interprofessional*.tw. or inter-professional*.tw. or interdisciplinary.tw. or inter-disciplinary.tw. or multidisciplinary.tw. or exp "interprofessional relations"/) and (student.tw. or students.tw. or educate.tw. or educating.tw. or educator.tw. or educators.tw. or education.tw. or instructor.tw. or instructors.tw. or instruction.tw. or teaching.tw. or training.tw. or trainee.tw. or trainees.tw. or curriculum.tw. or curricula.tw. or simulation.tw. or simulations.tw. or shadowing.tw. or "clinical practicum*".tw. or "clinical placement*".tw. or "experiential learning".tw. or teamwork.tw. or exp "education, professional"/ or exp "clinical competence"/)) narrow phrase search (elsevier embase) (ipe:ti,ab or ’interprofessional education’/exp or ’interprofessional education’:ti,ab,kw or ’inter-professional education’:ti,ab,kw or ’interdisciplinary education’:ti,ab,kw or ’inter-disciplinary education’:ti,ab,kw or ’multiprofessional education’:ti,ab,kw or ’multidisciplinary education’:ti,ab,kw or ’multi-professional education’:ti,ab,kw or ’multi-disciplinary education’:ti,ab,kw) narrow title search (elsevier embase) (interprofessional*:ti or inter-professional*:ti or multiprofessional*:ti or multi-professional*:ti or interdisciplinary:ti or inter-disciplinary:ti or multidisciplinary:ti or multi-disciplinary:ti or multioccupational:ti or interoccupational:ti or inter-occupational:ti) and (student*:ti or educat*:ti or learn*:ti or train*:ti or teach*:ti or curricul*:ti or simulat*:ti or school*:ti or course*:ti) 4 peer reviewed article hypothesis, vol. 37, no. 2, 2025 broader keyword search (elsevier embase) (ipe:ti,ab or ’interprofessional education’/exp) or ((interprofessional*:ti,ab or inter-professional*:ti,ab or interdisciplinary:ti,ab or inter-disciplinary:ti,ab or multidisciplinary:ti,ab) and (student:ti,ab or students:ti,ab or educate:ti,ab or educating:ti,ab or educator:ti,ab or educators:ti,ab or education:ti,ab or instructor:ti,ab or instructors:ti,ab or instruction:ti,ab or teaching:ti,ab or training:ti,ab or trainee:ti,ab or trainees:ti,ab or curriculum:ti,ab or curricula:ti,ab or simulation:ti,ab or simulations:ti,ab or shadowing:ti,ab or ‘clinical practicum*’:ti,ab or ‘clinical placement*’:ti,ab or ‘experiential learning’:ti,ab or teamwork:ti,ab or ’vocational education’/exp or ’clinical competence’/exp)) the searches’ relative recall was almost identical across nlm pubmed, ovid medline, and elsevier embase (table 2). the narrow phrase search achieved 73-74% relative recall, the title search 85%, and the broad keyword search achieved 95-96% across all three databases. the number of search results was also similar across databases; however, embase had slightly more results for each search. since the results are so similar across platforms, the usage recommendations from the original paper for the pubmed search hedges3 extend to the ovid and embase searches also. specifically, the narrow phrase search can be used for quick retrieval of relevant articles. the narrow title search may be used for slightly more rigorous searching and the broad keyword search achieves the recommended recall for comprehensive or systematic literature reviews. table 2. relative recall for each of the search strategies by database platform. pubmed ovid medline elsevier embase narrow phrase search 73.43% 73.43% 73.86% narrow title search 83.98% 83.98% 84.47% broader keyword search 96.48% 96.48% 95.45% implications these search strategies provide ipe scholars with more data-driven options for finding literature. using these searches can help librarians and researchers feel more confident in the comprehensiveness of their literature search strategies, with similar relative recall for each available platform. this work also provides useful information for future search validations; since translation across database platforms requires little extra investment, researchers can easily develop hedges for multiple databases. these validated search hedges will help scholars more easily and efficiently find ipe scholarship. references 1. world health organization. framework for action on interprofessional education & collaborative practice [internet]. [cited 2025 sep 12]. available from: 5 peer reviewed article hypothesis, vol. 37, no. 2, 2025 https://www.who.int/publications/i/item/framework-for-action-on-interprofessional-educationcollaborative-practice 2. cadet t, cusimano j, mckearney s, honaker j, o’neal c, taheri r, et al. describing the evidence linking interprofessional education interventions to improving the delivery of safe and effective patient care: a scoping review. journal of interprofessional care [internet]. 2024 may 3 [cited 2025 sep 12];38(3):476–85. available from: https://www.tandfonline.com/doi/full/10.1080/13561820.2023.2283119 3. carlson r, nachman s, zerden l, mani n. validation of an interprofessional education search strategy in pubmed to optimize ipe literature searching. journal of the medical library association [internet]. 2024 jan 11 [cited 2025 sep 12];112(1):33–41. available from: http://jmla.pitt.edu/ojs/jmla/article/view/1742 4. sampson m, zhang l, morrison a, barrowman nj, clifford tj, platt rw, et al. an alternative to the hand searching gold standard: validating methodological search filters using relative recall. bmc med res methodol [internet]. 2006 dec [cited 2025 sep 12];6(1):33. available from: https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/1471-2288-6-33 6 https://www.who.int/publications/i/item/framework-for-action-on-interprofessional-education-collaborative-practice https://www.who.int/publications/i/item/framework-for-action-on-interprofessional-education-collaborative-practice https://www.tandfonline.com/doi/full/10.1080/13561820.2023.2283119 http://jmla.pitt.edu/ojs/jmla/article/view/1742 https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/1471-2288-6-33 peer reviewed article hypothesis, vol. 36, no. 2, 2024 combining web analytics and a web-based user survey for libguides assessment carmen howard, msls*a, jung mi scoulas, phdb, allan berry, mslis, mac, deborah l. lauseng, amlsd asenior instructor and regional health sciences librarian, library of the health sciences – peoria, university of illinois chicago, peoria, illinois, https://www.orcid.org/0000-0002-3460-5417 , choward4@uic.edu bassociate professor and assessment coordinator, university library, university of illinois chicago, chicago, illinois, https://www.orcid.org/0000-0002-7536-295x , jscoul2@uic.edu cclinical assistant professor and web services librarian, university library, university of illinois chicago, chicago, illinois, https://www.orcid.org/0000-0002-1012-8601 , aberry3@uic.edu dassociate professor and regional head librarian, library of the health sciences – peoria, university of illinois chicago, peoria, illinois, https://www.orcid.org/0000-0001-9067-2932 , dlauseng@uic.edu cite as: howard c, scoulas jm, berry a, and lauseng dl. combining web analytics and a web-based user survey for libguides assessment. hypothesis. 2024;36(2). doi:10.18060/27669 howard, scoulas, berry, lauseng. all works in hypothesis are licensed under a cc by-nc 4.0 deed attribution-noncommercial 4.0 international. authors own copyright of their articles appearing in hypothesis. readers may copy articles without permission of the copyright owner(s), as long as the author(s) are acknowledged in the copy, and the copy is used for educational, not-for-profit purposes. for any other use of articles, please contact the copyright owner(s). 1 https://www.orcid.org/0000-0002-3460-5417 https://orcid.org/ mailto:choward4@uic.edu https://www.orcid.org/0000-0002-7536-295x https://orcid.org/ mailto:jscoul2@uic.edu https://www.orcid.org/0000-0002-1012-8601 https://orcid.org/ mailto:aberry3@uic.edu https://www.orcid.org/0000-0001-9067-2932 https://orcid.org/ mailto:dlauseng@uic.edu https://creativecommons.org/licenses/by-nc/4.0/ peer reviewed article hypothesis, vol. 36, no. 2, 2024 abstract objective: the aim of this method is to assess an online library guide, using both qualitative and quantitative methods, with the specific objective of discovering who is using the guide and why, as well as understanding the impact of the guide on user learning outcomes. approach: this method combines the use of web/data analytics (quantitative) and a web-based survey (quantitative and qualitative) with an existing model for measuring learning outcomes. by combining these methods, a more robust assessment of an online guide can be achieved. data type used: the discussed online guide assessment used data from google analytics and from a web-based survey, created in qualtrics, with learning outcome questions developed using the project outcome for academic libraries toolkit. strengths: the combined analysis of the web analytics and survey data resulted in an expanded assessment of the library guide and a richer picture of guide users, their behavior, and their learning outcomes. limitations: individually, each method used in this research has limitations. web analytics are unable to determine users’ motivation (why) for using the guide and whether they have learned (impact) from their behavior/interaction while using the guide. surveys collect self-reported data and may be subject to recall bias. however, when the methods are combined, each helps to counteract the individual limitations. overview library guides, a type of library-created online resource that is frequently built using springshare’s libguides, are labor-intensive to maintain over time. for this reason, many libraries seek to assess their guides’ use and effectiveness. most of these assessments focus on quantitative data, whereas a few other studies have used qualitative methods, and very few have used a combination of methods.1,2 while each method has its own strengths, griffin and taylor highlighted the need to incorporate qualitative data with quantitative usage data and advocated for “using multiple data sources and triangulating findings between data points”.1 analytics data provides a picture of actual user behavior through tracking and reporting traffic patterns with detailed aggregate statistics. because it tracks every interaction, analytics data eliminates recall bias. however, interaction tracking is not always as complete and informative as might be expected. although web analytics are objective in how data is gathered, the choice of what data to gather is not. tracking movement on a website also cannot reveal user motivations, if they found what they were looking for, or whether they learned anything from the interaction. surveys provide a structured format for collecting information on various topics of interest, including demographics, behaviors, and attitudes. they have the potential to reach a range of patrons and ask them questions to better understand their motivations, perceptions, and experiences. although they have many advantages, surveys also come with limitations. for 2 peer reviewed article hypothesis, vol. 36, no. 2, 2024 example, they rely on self-reported data provided by individuals who voluntarily participate, which may introduce biases.3 survey fatigue can occur due to the large volume of survey requests, potentially impacting response rates. in addition, the extensive nature of some surveys with multiple questions can discourage participants from dedicating the time necessary to complete them. our research team sought to assess the university of illinois chicago (uic) library’s evidence based medicine (ebm) libguides pages (our guide) and plan for the future of our guide using data (irb protocol #2019-0449). given higher-than-anticipated usage statistics and multiple external requests to reuse content, our research team speculated that our guide was being used more broadly than initially intended. we aimed to understand user behavior, both what they did and why they did it, and to measure the impact of our guide on user learning experience and outcomes.4 additionally, we sought to provide evidence for transitioning our guide to a fully open educational resource (oer).4 therefore, we developed an assessment approach employing two complementary methods used concurrently: quantitative usage data and a web-based user survey. our research team initially reviewed the usage statistics integrated into libguides; however, we found the data limited to high-level analysis, such as page views, monthly trends of use, etc. given our interest in a deeper understanding of users and their use patterns, we determined that google analytics (ga) would provide a more robust data set. we also realized that our questions could not be fully answered by analytics and that we would need to obtain users’ input through a survey. our interest in measuring educational outcomes led us to the project outcome for academic libraries toolkit, hereafter referred to as the project outcome model.5 the project outcome model is a free online toolkit developed to measure the impact of library programs and services. we adapted questions from the project outcome model into our survey to gain insights into user knowledge and confidence. the combination of two methods, web/data analytics and a web-based survey, provides a more robust approach in the assessment of online guides. this combination gave our research team a richer understanding of our users, their behavior, and their perceptions of learning outcomes. for librarians interested in expanding their evaluation of library guides or other webpages, this article focuses on our process and the strengths and limitations of each method and tool we used, as well as their combination. a summary of those strengths and limitations can be found in table 1 following the discussion. for a brief review of the literature on evaluating library guides and a discussion of our research findings, please see our team’s previously published work.4 example with discussion uic’s ebm (evidence based medicine) guide (https://researchguides.uic.edu/ebm) is a subject guide that introduces the basic concepts in ebm with pages such as “levels of evidence” and “asking clinical questions.” additionally, our guide covers higher level concepts like “clinical filters” and “appraisal.” our guide was initially created for students in the uic college of medicine and for non-uic students in an online course for librarians. however, integrated tracking statistics indicated that it was being used by far more people than this limited pool of intended students. our research team needed to analyze user behavior to determine who was using our guide and why, so we looked at what methods 3 https://researchguides.uic.edu/ebm peer reviewed article hypothesis, vol. 36, no. 2, 2024 would assist us in this specific need. we also found that some pages were used at significantly higher rates, which led us to develop survey questions about individual page use and to use analytics data to look specifically at page traffic. at the beginning of our research project, we considered a few different methods and tools. as the project progressed, we discovered several strengths and limitations of the methods and tools that we chose. see table 1 for a summary. quantitative method web/data analytics our research team chose to look at web analytics data because we believed it could answer questions about how our users interacted with our guide the pages they visited, how long they stayed, and how they were interacting with the pages. we also knew that it could inform us about the devices used, how users found our guide, and their general geographic location. analytics data can be obtained from various tools. our research team considered and/or used the following tools described below. springshare’s integrated "guide tracking" our research team initially used the integrated statistics from springshare’s libguides primarily because this is the software used to develop our guide. while this was helpful in seeing counts of views per page and monthly trends in overall use, the data is only available at a high level. although springshare does have additional tracking features, we needed a more comprehensive, granular, look at how the site was used. google analytics (ga) our research team decided to move to ga because ga was currently being used by the library and is widely regarded as an industry standard.6,7 it is a clear market leader8,9 and considered a required skill for entry-level marketing analytics professionals.10 ga met our need to see how users interacted with our guide. during our research, we used the latest version available of ga called “universal analytics” (ua). google has subsequently released a major update to the ga system, named “google analytics 4” (ga4). ga provides rich data for user behavior analysis. we were specifically interested in the following quantitative information from ga: • how users were finding our guide • how they moved between guide pages • how they used specific pages (most-visited pages and average time on a page) • their general geographic location, such as country and city (we did not collect specific location data, such as street or ip addresses) • the type of device they were using to access the page. 4 peer reviewed article hypothesis, vol. 36, no. 2, 2024 the first issue we encountered involved urls and ga reporting. we configured ga to only return data about our guide itself. our guide consists of 10 total urls: three unique urls for the home page and one unique url for each of seven additional pages. we elected to remove urls with third party parameters for simplicity; for example, facebook or google translate, would sometimes add proprietary url parameters like “translate.googleusercontent. com/translate_c?depth=1&hl=ar&prev=search&rurl=translate.google.com&sl=en&sp=nmt4&u=” to the beginning of our urls and for their own purposes of user tracking. accounting for these additional urls significantly increased the complexity of our report data. most of these modified urls, while presumably representing legitimate user requests, were each only accessed during a single session. therefore, we explicitly excluded these proprietary urls by filtering out all traffic except to our expected 10 urls. these 10 urls covered 99.6% of all unique users from the full ga pool during the research time-period. no excluded url represented more than 0.05% of our guide’s unique users. the added simplicity seemed worth the small cost in comprehensiveness. additionally, our research team learned the importance of understanding the way that ga data points are calculated. initially, we anticipated using “bounce rate” and “session duration” to inform our understanding of user behaviors. however, upon further investigation into our guide’s specific data, all uic guides data, and a deeper review of ga processing, we determined that these metrics did not reveal much about the use of educational subject guides. when a user visits a single page without further interaction, ga considers this a "bounce."11 our guide had a large proportion of "bounces" (86% of sessions). for example, consider a student completing a course assignment. they might spend 10 minutes using a guide page, afterward closing their browser without further navigation. most librarians would call this a success. however, since “session duration” is calculated by time-stamping each action that a user takes (for example, clicking a link), ga counts this as a "bounce", with a “session duration” of zero seconds.11, 12 sessions can also end automatically after an arbitrary time expires without any actions.12 google documentation acknowledges that a high bounce rate is only problematic if you are trying to direct people to another page,11 such as when your home page is only a waypoint on the way to a product page. thus, while "bounces" were considered an important metric in ua, they were later de-emphasized in ga4 in favor of other engagement metrics. in the case of our guide, even the home page is an educational page, not meant to direct users to subsequent pages, so high “bounce rates” were not a concern for us. for these reasons, our research team found “average time on page” to be more informative in evaluating the success of our guide than either “session duration” or “bounce rate.” as we learned more about ga, we belatedly realized additional features could have been activated prior to our research study, which may have provided even deeper insights. one example is “scroll depth,” or how far a given user scrolls down a page, which can be a proxy for user engagement. however, tracking data for “scroll depth” in ua required activation before data collection; at the outset of our project, we did not consider it relevant. later, it seemed problematic to alter our data collection midstream. ga4 provides this feature automatically, which should provide additional rich data for future research. 5 peer reviewed article hypothesis, vol. 36, no. 2, 2024 quantitative & qualitative method web survey while ga offers insights into general user patterns, it does not provide information on user motivation and reasoning or assess the impact of their usage on learning outcomes. so, in addition to the ga data, our research team developed a survey to gather information from users regarding their usage of our guide and the extent to which they achieved their learning objectives. in the survey, we collected various information, including demographics, user behavior, future referral intentions, user perceptions, satisfaction levels, and user learning outcomes in terms of knowledge and confidence. multiple survey tools and models are available. our research team adapted the project outcome model to assess learning outcomes by using a survey administered through qualtrics software. project outcome model to measure users’ learning outcomes, we utilized two questions from the project outcome model.5 this commonly used model is based on a series of self-assessment learning statements and uses a likert scale. we asked questions related to the user’s change in knowledge and confidence. for example, we adapted this question: “after using the ebm guide, i feel more confident about my ability in each of the following areas.”4,5 the project outcome model allowed us to gain insights into users’ learning without asking them to take a preand post-test of their knowledge. however, we acknowledge that survey responses were based on self-perception and that the question formatting used in the project outcome model has been noted4 to have the potential of acquiescence bias, or the “tendency for the interviewee to agree with the questioner.”13 qualtrics qualtrics is a web survey platform used to create and distribute surveys. additional web survey tools are noted in table 1. our research team chose qualtrics as our survey tool due to its many useful features and its availability at our institution. it enables researchers to gather, analyze, and report data effectively, with features to enhance data security.14 however, it is important to note that there are financial costs associated with using advanced features and conducting surveys with larger sample sizes. this may pose accessibility and affordability challenges for survey designers operating on limited budgets. additionally, certain features of qualtrics may have a learning curve, particularly when it comes to coding scales and setting up surveys as originally intended. this learning process is crucial for ensuring accurate data analysis and utilizing the platform to its fullest potential. 6 peer reviewed article hypothesis, vol. 36, no. 2, 2024 table 1. strengths and limitations of methods and tools analytics survey product (platform) springshare libguides analytics google analytics project outcome model qualtrics (platform) research questions this data did not answer our research questions but provided an overview of guide usage. 1. who are the users? (demographics such as geographic location and device used.) 2. how are users interacting with our guide? 1. what learning outcomes are achieved from the users’ perspectives? 1. who are the users? (detailed demographics such as educational level and profession.) 2. what are the users’ motivations? method: strengths 1. comprehensive data 2. actual use patterns/behaviors 3. automated data after set-up 4. eliminates recall bias 1. qualitative data 2. motivational data 3. question structure flexibility allowing integration of existing questions from the project outcome model or other models5. tool: strengths 1. easy to use 2. native to the webtool/publishing platform 3. ability to compare with our other guides 1. cost-effective 2. detailed statistics regarding traffic patterns and use 3. industry standard 1. provides existing questions related to learning outcomes 1. comprehensive options 2. analysis and reporting features 3. enhanced data security features method: limitations 1. unable to understand users’ motivation (why) and whether they have learned (impact) 2. unable to collect certain demographic info (i.e. educational level) 3. possible user privacy concerns 1. self-reported data which may be skewed due to self-selected participation 2. may be subject to recall bias 3. survey fatigue tool: limitations 1. limited data points 1. data analysis can be complicated with proprietary urls 2. data point calculations (for example, “bounce rate”) may not be intuitive 1. may be subject to acquiescence bias 1. financial costs to access advanced features 2. may have learning curve for sophisticated features methods: potential alternatives 1. quantitative survey 1. focus groups 2. semi-structured interviews 3. usability testing 7 peer reviewed article hypothesis, vol. 36, no. 2, 2024 tools: potential alternatives 1. adobe analytics 2. amplitude 3. cloudflare web analytics 4. matomo analytics 5. mixpanel 6. other options are available 1. several models are available depending on your research topic 1. email distribution 2. google forms 3. libwizard 4. polling 5. survey monkey 6. other options are available conclusions our research team found that the use of both quantitative and qualitative data provided a richer picture of our users, their behavior, and their learning outcomes. we suggest that ga data and survey data each have their own strengths. when combined, these methods are complementary; each helps to counteract the limitations inherent in the other. by combining methods to include both web/data analytics and a web-based survey that incorporates the project outcome model, we were able to assess libguides pages more robustly and develop an expanded methodological assessment approach. the strength of the combined method can be seen in a variety of ways. first, ga kept the survey shorter. because ga could easily answer some of our questions, we did not have to ask users about their navigation of the site or about the devices they used. keeping surveys shorter helps to address survey fatigue. second, an anonymous web-based survey can provide insights to many complex research questions like understanding user motivation and outcomes. for example, although ga could tell us if a user was coming to our site from an educational site, like a course management system, ga cannot provide further insight into educational uses or outcomes. using the project outcome model to measure changes in user confidence and their perception of increased knowledge has provided a clearer understanding of user learning. the learning outcome data, which showed an increase in self-reported knowledge and confidence, was used to justify expanded work on our guide. for detailed results and a discussion of our findings, please see our previously published article.4 finally, the combination of the two methods removed the uncertainty of some ga data points and increased our confidence overall that the survey participants were representative of all users. our research team included questions in the survey to look for data agreement or lack of agreement with data gathered by ga. for example, in terms of geographic distribution, a brief examination of ga data in preparation for our research revealed that most users were not coming from the chicago area. this could be an indication that they might not be uic affiliates, although many uic students and faculty do attend and teach courses remotely. this data point led us to include a question on the survey about uic affiliation. when we ran the survey and looked at concurrent ga data, we confirmed that most users were not associated with uic. another example, where we included a question to look for data agreement but did not find agreement, was in comparing how users came to our guide. ga reported much higher 8 peer reviewed article hypothesis, vol. 36, no. 2, 2024 levels of discovery by an organic search than users self-reported in the survey. although it was extra work and our results were somewhat mixed, we felt this effort to look for data agreement was worth the investment for our overall increased confidence in the results.4 our assessment was enhanced by carefully examining the strengths and limitations of each method and tool used to address our research questions. further, our assessment was strengthened through a combination of methods and tools. the most important characteristic in this type of assessment is the combination of quantitative data with a qualitative component. our research team proposes that this combined method could be used by others to evaluate not just libguides pages but other types of web resources. alternative tools can be used in implementing this proposed method. for example, if your institution/organization uses primarily google forms to gather feedback from your users, this tool could be used instead of qualtrics. if you want to explore challenges or behaviors for non-users or frequent users in depth, conducting focus groups could be more appropriate. librarians and researchers simply need to examine the purpose of their project and the most appropriate quantitative and qualitative methods and tools available to address their questions. resources web content • project outcome for academic librarians toolkit: https://acrl.projectoutcome.org/about • google. analytics help. glossary (ga4): https://support.google.com/analytics/topic/9355633?hl=en&ref_topic=9143232&sjid=95027 3908224713767-na • the ultimate google analytics glossary (2023 edition): https://www.lovesdata.com/blog/google-analytics-glossary further reading • connaway ls, radford ml. survey research and the questionnaire. in: research methods in library and information science. 7th ed. santa barbara, calif.: libraries unlimited; 2021. p. 129-168. • farney t. getting the best google analytics data for your library. libr technol rep. [internet] 2016;52(7):5-8. available from: https://doi.org/10.5860/ltr.52n7 • watson ca. can an understanding of data use in academic libraries assist in public service efforts? public serv q. [internet] 2021;17(1):59-64. available from: https://doi.org/10.1080/15228959.2020.1860861 9 https://acrl.projectoutcome.org/about https://support.google.com/analytics/topic/9355633?hl=en&ref_topic=9143232&sjid=950273908224713767-na https://support.google.com/analytics/topic/9355633?hl=en&ref_topic=9143232&sjid=950273908224713767-na https://www.lovesdata.com/blog/google-analytics-glossary https://doi.org/10.5860/ltr.52n7 https://doi.org/10.1080/15228959.2020.1860861 peer reviewed article hypothesis, vol. 36, no. 2, 2024 credit carmen howard: conceptualization, methodology, investigation, resources, writing original draft, writing review & editing jung mi scoulas: conceptualization, methodology, formal analysis, investigation, resources, data curation, writing original draft, writing review & editing allan berry: conceptualization, methodology, investigation, resources, data curation, writing original draft, writing review & editing deborah l. lauseng: conceptualization, methodology, investigation, resources, data curation, writing original draft, writing review & editing references 1. griffin m, taylor ti. employing analytics to guide a data-driven review of libguides. j web librariansh. [internet] 2018 [cited 2023 jul 26] 12(3):147-159. available from: https://doi.org/10.1080/19322909.2018.1487191 2. jackson r, stacy-bates kk. the enduring landscape of online subject research guides. ref & user serv q. [internet] 2016 spring [cited 2024 jan 23] 55(3): 219-225. available from: https://doi.org/10.5860/rusq.55n3.219 3.mcnabb de. response error. in: nonsampling error in social surveys. sage publications, inc.; 2014:114-134. available from: https://doi.org/10.4135/9781483352923 4. lauseng dl, howard c, scoulas jm, berry a. assessing online library guide use and open educational resource (oer) potential: an evidence-based decision-making approach. j web librariansh. [internet] 2021 [cited 2023 jul 26] 15(3):128-153. available from: https://doi.org/10.1080/19322909.2021.1935396 5. performance measurement task force/public library association (pla), project outcome for academic libraries task force/association of college & research libraries (acrl). project outcome for academic libraries toolkit [internet]. chicago; american library association; 2019 [cited 2023 jul 26]. available from: https://acrl.projectoutcome.org/about 6. packer j. google analytics alternatives: a guide to navigating the world of options beyond google [internet]. quantable, llc; 2023 feb 13 [cited 2023 jul 26]. available from: http://arks.princeton.edu/ark:/88435/dsp013n204232m 7. jansen bj, jung s, salminen j. measuring user interactions with websites: a comparison of two industry standard analytics approaches using data of 86 websites. plos one. [internet]. 2022 [cited 2024 jan 25] 17(5): 3. available from: https://doi.org/10.1371/journal.pone.0268212 8. datanyze. market share of leading web analytics technologies worldwide in 2023 [internet]. in: statista. 2023 nov 27 [cited 2024 jan 26]. available from: https://www.statista.com/statistics/1258557/web-analytics-market-share-technologyworldwide/ 9. 6sense. google analytics market share [internet]. 2024 [cited 2024 jan 24]. available 10 https://doi.org/10.1080/19322909.2018.1487191 https://doi.org/10.5860/rusq.55n3.219 https://doi.org/10.4135/9781483352923 https://doi.org/10.1080/19322909.2021.1935396 https://acrl.projectoutcome.org/about http://arks.princeton.edu/ark:/88435/dsp013n204232m https://doi.org/10.1371/journal.pone.0268212 https://www.statista.com/statistics/1258557/web-analytics-market-share-technology-worldwide/ https://www.statista.com/statistics/1258557/web-analytics-market-share-technology-worldwide/ peer reviewed article hypothesis, vol. 36, no. 2, 2024 from: https://6sense.com/tech/analytics/google-analytics-market-share 10. mcguirk m. performing web analytics with google analytics 4: a platform review. j mark anal, (2023). 11(4), 854-868. available from: https://doi.org/10.1057/s41270-023-00244-4 11. google. bounce rate [internet]. mountain view, ca: google, llc, 2023 [cited 2023 jul 26]. available from: https://web.archive.org/web/20230705160536/https://support.google.com/analytics/answer/1009409?hl=en 12. google. how a web session is defined in universal analytics [internet]. mountain view, ca: google, llc, 2023 [cited 2024 jan 23]. available from: https://web.archive.org/web/20231226204029/https://support.google.com/analytics/answer/2731565?hl=en 13. upton g, cook i. acquiescence bias. in: a dictionary of statistics. oxford university press; 2014 [cited 2024 jan 23]. available from: https://www.oxfordreference.com/view/10.1093/acref/9780199679188.001.0001/acref9780199679188-e-1783 14. qualtrics. qualtrics security and compliance [internet]. provo, ut: qualtrics; 2023 [cited 2023 jul 26]. available from: https://www.qualtrics.com/platform/security/ 11 https://6sense.com/tech/analytics/google-analytics-market-share https://doi.org/10.1057/s41270-023-00244-4 https://web.archive.org/web/20230705160536/https://support.google.com/analytics/answer/1009409?hl=en https://web.archive.org/web/20230705160536/https://support.google.com/analytics/answer/1009409?hl=en https://web.archive.org/web/20231226204029/https://support.google.com/analytics/answer/2731565?hl=en https://web.archive.org/web/20231226204029/https://support.google.com/analytics/answer/2731565?hl=en https://www.oxfordreference.com/view/10.1093/acref/9780199679188.001.0001/acref-9780199679188-e-1783 https://www.oxfordreference.com/view/10.1093/acref/9780199679188.001.0001/acref-9780199679188-e-1783 https://www.qualtrics.com/platform/security/ peer reviewed article hypothesis, vol. 37, no. 2, 2025 the relevance of health equity to the evidence synthesis process: results from a quantitative survey of health information professionals stacy torian, mlis, maa, stephen maher, msis, lmswb aassistant curator: librarian for health sciences, bobst library, new york university, new york, new york,https://www.orcid.org/0009-0002-9859-5075 , stacy.torian@nyu.edu bassistant curator: librarian for social work & psychology, bobst library, new york university, new york, new york,http://orcid.org/0000-0002-6628-6756 , stephen.maher@nyu.edu cite as: torian s and maher s. the relevance of health equity to the evidence synthesis process: results from a quantitative survey of health information professionals. hypothesis. 2025;37(2). doi:10.18060/28655 torian, maher. all works in hypothesis are licensed under a cc by-nc 4.0 deed attribution-noncommercial 4.0 international. authors own copyright of their articles appearing in hypothesis. readers may copy articles without permission of the copyright owner(s), as long as the author(s) are acknowledged in the copy, and the copy is used for educational, not-for-profit purposes. for any other use of articles, please contact the copyright owner(s). 1 https://www.orcid.org/0009-0002-9859-5075 https://orcid.org/ mailto:stacy.torian@nyu.edu https://www.orcid.org/0000-0002-6628-6756 https://orcid.org/ mailto:stephen.maher@nyu.edu https://creativecommons.org/licenses/by-nc/4.0/ https://creativecommons.org/licenses/by-nc/4.0/ peer reviewed article hypothesis, vol. 37, no. 2, 2025 abstract background: in their work as research collaborators, health information professionals often contribute both extensive knowledge of evidence synthesis methods and an engaged awareness of the sociopolitical context in which evidence synthesis takes place. within this context, researchers are increasingly being encouraged to demonstrate how their work will advance health equity. our study sought to determine how confident health information professionals feel discussing health equity in an evidence synthesis context and how relevant they think it is to the process. methods: using the email lists of u.s.-based and international library organizations, the authors conducted a quantitative survey of health information professionals with experience working on evidence synthesis projects (e.g., systematic or scoping reviews) within the past 10 years. the authors used the survey software qualtrics to calculate descriptive statistics for each of the survey questions and analyze correlations between responses to different questions. results: while a majority of health information professionals surveyed exhibited both familiarity with health equity as a concept and a belief in its relevance to evidence synthesis, far fewer were highly comfortable discussing health equity with researchers. only 6% of participants had used the prisma-equity extension (a health equity-centered evidence synthesis tool) in a systematic review. discussion: the study results suggest that a number of health information professionals would benefit from training on health equity-centered evidence synthesis tools and more experience having health equity-related research discussions. introduction “evidence synthesis” is a term that describes different methods of identifying, evaluating, and analyzing research literature on a topic1. one of the most common types of evidence synthesis is the systematic review. recognized as the “best known type of review”2, the systematic review presents a synthesis and evaluation of all the available literature on a research topic, along with a suggested course of action in the realm of policy or practice3,4. systematic reviews require a well-defined research question or topic, a research protocol, a comprehensive literature search strategy, explicit criteria regarding what literature will or will not be included, careful literature appraisal and synthesis, and a detailed reporting of the reviewers’ methods and findings5. systematic reviews are a critical component of evidence-based practice (ebp), an approach to healthcare that emphasizes “the integration of best available evidence, clinical expertise, and patient values and circumstances”6 in the context of clinical practice. for over a decade, the national academy of medicine (formerly the institute of medicine), the cochrane handbook for systematic reviews of interventions, and the agency for healthcare research and quality have recommended that research teams conducting systematic reviews seek the support of librarians, information specialists, or other expert searchers when crafting literature searches5,7,8. while researchers often consult health information professionals for assistance with literature searching, health information 2 peer reviewed article hypothesis, vol. 37, no. 2, 2025 professionals contribute more than search strategies to the evidence synthesis process. those additional contributions include educating research teams on how evidence syntheses are conducted and facilitating the planning of the study methodology9. although systematic reviews are recognized as an important source of synthesized research, researchers and policymakers have expressed concerns about the absence of an equity perspective in the reporting of systematic review results and recommendations10-13. the world health organization (who) has defined equity as “the absence of unfair, avoidable or remediable differences among groups of people, whether those groups are defined socially, economically, demographically, or geographically or by other dimensions of inequality (e.g. sex, gender, ethnicity, disability, or sexual orientation).14” an equity perspective examines how the interventions recommended in the review could address unfair inequalities in health outcomes15. such a perspective also requires researchers to think critically about the review topics they choose, the types of literature they review, the details they report in their findings, and the environmental context within which any interventions they recommend might take place16-18. the study that follows is an attempt to determine what views health information professionals hold on the subject of health equity in evidence synthesis, to what extent they engage in discussions about this subject with researchers, and their level of comfort with these types of discussions. it begins with a literature review on the role of health information professionals and health equity considerations in evidence synthesis, followed by the presentation of the study results, discussion of the study findings, and suggestions for further research and reflection. literature review the role of health information professionals in evidence synthesis some of the earliest examples of librarians’ involvement in evidence syntheses occurred in the 1990s, with an increase in the use of systematic reviews in medicine to combine the results of clinical trials19. these instances correlated with the beginning of electronic publishing, and the emergence of the internet as a medium for disseminating information20,21. specifically with respect to biomedical literature, the launch of pubmed in 1997 provided researchers, via the internet, the ability to leverage the national library of medicine’s online database, medline, to search its comprehensive bibliographic index of medical articles (index medicus)22. librarians were the early adopters of these online databases, and clinicians and researchers recognized their expertise with indexing practices and familiarity with controlled vocabularies. they valued the librarian’s skills for facilitating their approaches to meta-analyses and other evidence synthesis projects20. as the electronic publishing of articles proliferated, so too did the coverage of online databases to index and aid researchers in the searching and retrieval of this material. this would contribute to an increase in the number of systematic reviews23. with this increase came the need for guidelines for researchers to follow in conducting a systematic review and other evidence syntheses. these guidelines, including those from the cochrane collaboration, the national academy of medicine, and the joanna briggs institute, identified librarians as collaborators either for assisting in the development of search strategies for retrieving relevant materials or in managing the methods process5,24,25. arksey and o’malley the authors of the 3 peer reviewed article hypothesis, vol. 37, no. 2, 2025 original methodological framework for scoping reviews made a similar recommendation, suggesting that non-librarians may not be able to design a sufficient search strategy for evidence syntheses26. in the mid-2010s, researchers started to test the merits of these recommendations within their respective disciplines. in general internal medicine, librarian co-authorship in systematic reviews correlated with higher quality search strategies27. in pediatrics, researchers considered search strategies conducted by librarians to be more reproducible28,29. there have also been studies advocating for the involvement of librarians in evidence syntheses in education30 and dentistry31. as their reputations for being “natural advisors and partners”32 on evidence syntheses have spread, librarians have investigated their roles and the challenges that come from these collaborations. one of the challenges librarians and information professionals have identified is that as they accumulate experience working on evidence syntheses, they find themselves educating researchers on these methodologies in addition to collaborating with them on their search strategies33. another challenge centers on authorship and acknowledgement of the contribution librarians and information professionals provide in evidence syntheses. despite the recommendations and evidence of the benefit of their contributions, librarians have found that their efforts in the search process can be perceived as invisible labor34 and can go unrecognized35,36. as a result of these challenges, many librarians have developed service models for managing demand and researcher expectations37–42. health equity and evidence synthesis over the years, scholarship on health equity and evidence synthesis has evolved in parallel with the broader discourse on health equity in the field of public health. in their 2007 paper, cochrane’s elizabeth waters and rebecca armstrong announced the beginning of a cochrane review group for public health and health promotion, asserting that the group would “help ensure that reviews will be oriented towards building evidence for equity and reducing inequalities and best meet the needs of decision-makers, practitioners and consumers.43” during roughly the same time period, in 2003, public health researchers timothy evans and hilary brown introduced progress, an acronym that stands for multiple “socially stratifying forces” that have consequences for health equity, namely “place of residence, religion, occupation, gender, race/ethnicity, education, socioeconomic status, and social networks and capital.44” progress can assist researchers in highlighting demographic characteristics and socioeconomic circumstances that can impact the effectiveness of interventions45. recognizing both the potential and the limitations of this tool, a group of researchers mapping studies on the health of young people developed an enhancement of progress in 2008 called progress plus to encompass personal factors that can lead to inequitable health outcomes46. though not used consistently in systematic reviews, both progress and progress plus continue to be recognized as helpful tools for presenting and assessing systematic review results from a health equity standpoint45,47–49. another public health development that greatly influenced scholars promoting health equity in evidence synthesis was a 2008 report by the world health organization’s commission on 4 peer reviewed article hypothesis, vol. 37, no. 2, 2025 social determinants of health (csdh) urging governmental stakeholders and international organizations to take substantive steps to ensure that under-resourced communities have equitable access to the resources that will help them maintain good health50. in a 2010 paper acknowledging the csdh’s call, tugwell et al. provided guidance to public health researchers seeking to incorporate a health equity perspective into their systematic reviews. their recommendations included creating a logic model to envision how particular interventions might impact specific groups, taking into account the environmental context within which the interventions might be implemented, using progress plus to decide which outcomes are most important to assess for people experiencing health inequities, and considering the inclusion of study types beyond the randomized control trial51. in the decade following the formation of the csdh and the publication of the 2010 paper, guidance on how to incorporate health equity considerations into the evidence synthesis process became more detailed and formalized. in 2012 (a year after the acceptance of the rio political declaration on social determinants of health at the world conference on social determinants of health)52, welch et al. published the prisma-equity extension (prisma-e), a protocol to guide researchers doing equity focused-systematic reviews on how to report their evidence synthesis results in a way that centers health equity concerns53,54. prisma-e is based on the preferred reporting items for systematic reviews and meta-analysis (prisma) checklist, one of many protocols and tools researchers use to structure and transparently report the methods and results of their systematic reviews. in subsequent years, the 2013 paper "health equity: evidence synthesis and knowledge translation methods”16 and the cochrane handbook’s chapter on health equity published in 202047 provided further guidance. the cochrane chapter is referenced in the joanna briggs institute (jbi) manual for evidence synthesis, alongside a statement urging authors to “consider equity, diversity and inclusion in the planning and conduct of systematic and scoping reviews.55” despite this guidance, some researchers still struggle to bring an equity perspective to their systematic reviews. analyses conducted within the past several years have revealed inconsistent reporting of intervention effects by demographic or socio-economic factors in groups of systematic reviews on cataract13, covid-1949, and loneliness and social isolation48. one issue is that the individual studies included in reviews do not always include population data related to most of the progress or progress plus factors13. secondly, even when such data are included, systematic reviewers do not always consider these factors in their analyses48. as public health researchers have noted, not acknowledging how an intervention’s impact might differ across demographic and socio-economic groups could lead systematic reviewers to recommend public health interventions that actually amplify existing inequities without benefiting the populations who are already experiencing disproportionately negative health outcomes49,56. in recognition of the importance of health equity in evidence synthesis, the cochrane equity methods group authored a 2023 position statement stating, “we are committed to ensuring that all cochrane authors consider health equity in their reviews.17” a november 2024 cochrane webinar titled “equity in all cochrane reviews,” further emphasized this objective. there the presenters recommended that researchers consider an equity perspective not only for “equity-focused reviews” but also for reviews “aimed at the general population.57” 5 peer reviewed article hypothesis, vol. 37, no. 2, 2025 as health information professionals are more sought after in the production of evidence syntheses, they are well-positioned to introduce or encourage researchers to bring an equity perspective to their systematic reviews, particularly where it concerns research question development, literature searching, and the use of tools such as progress plus or prisma-e. the purpose of this study was to determine how familiar librarians are with health equity in evidence synthesis and how confident they feel introducing or discussing an equity perspective during meetings with researchers. our primary goal is to promote dialogue about health equity in evidence synthesis and related methodological considerations among health information professionals in the health sciences. methods the authors arranged the survey into three sections. the first section of the survey, which included a statement from the authors’ institutional review board (irb), sought to identify how experienced the participant was in collaborating in evidence synthesis projects. if they had been involved with one within the past 10 years, they advanced to the subsequent sections. in the second section, the authors asked participants to share which supporting documents they used or knew of in these projects. in the third section the authors inquired about the participants’ familiarity with the concept of health equity. the authors provided the definition of health equity offered by burford et al. in 2013: “the absence of avoidable and unfair inequalities in health, and social determinants of health.15” this definition cites contributions from whitehead (1992)58, bosch-capblanch et al. (2012)59, and the world health organization (2011)52. it was in this third section that the authors assessed the participants’ familiarity with the prisma-equity extension. the authors developed their survey questions (see supplemental materials) in qualtrics60 and opened the survey for completion from february to march 2024. after the completion period, the authors exported the data from qualtrics to google sheets for initial tabulation and calculations and, finally, the construction of the tables and graphs in this study. the authors recruited participants anonymously through various u.s.-based professional associations, including the medical library association (via the mla connect newsletter and the systematic review caucus, nursing and allied health caucus, new members caucus, latinx caucus, african american medical librarians alliance caucus, and social justice caucus email lists) and the association of college and research libraries (acrl). the authors also included international groups like the international federation of library associations and institutions (ifla) and the non-profit organizations the cochrane collaboration and the campbell collaboration, whose missions focus on the promotion and dissemination of evidence-based research. health information professionals who subscribe to the newsletters and email lists of these organizations received the authors’ solicitation to participate in the survey. three hundred twenty participants completed the survey. out of these, 97% (310) of the participants acknowledged having been involved in an evidence synthesis (systematic review, scoping review, meta-analysis) within the past 10 years, an indication that the authors recruited participants through appropriate professional organizations. 6 peer reviewed article hypothesis, vol. 37, no. 2, 2025 results the authors used qualtrics to calculate descriptive statistics for the answers from the 310 survey participants. the statistics include four cross tabulations, each comparing survey participants’ answers to one question with their answers to another question. because of rounding, there are situations where the total percentage breakdown does not equal 100% (e.g., figure 2). the data is presented in summary form below and through the graphs and tables that follow the summary. the authors, informed by their own experience as health sciences librarians, approached this study with the belief that a research consultation for an evidence synthesis project is often the best opportunity a health information professional has to offer their methodological expertise and to talk with researchers about the role of health equity in the evidence synthesis process. to gain perspective on the survey participants’ methodological expertise, the authors provided a list of well-known evidence synthesis methodology documents to measure their familiarity with said documents. as noted in figure 1, a significant portion of survey participants have used the evidence synthesis supporting documents listed in the survey (i.e., campbell, cochrane, joanna briggs institute, and prisma). forty-seven participants mentioned 49 additional resources (table 1; see supplemental materials ). as noted in figure 2, roughly 84% of the 310 participants indicated that they had taken the initiative to suggest these supporting documents or that both they and the research team had suggested them, as opposed to 12% who indicated that the research team alone had suggested the documents (the remaining 5% did not answer the question). figure 1. which supporting documents did you refer to for these evidence syntheses? 7 peer reviewed article hypothesis, vol. 37, no. 2, 2025 figure 2. how were the supporting documents introduced and or suggested? considered in aggregate, these responses indicate a wide range of evidence synthesis methodological knowledge among the survey participants. notably, three of the four supporting documents mentioned in the survey contain chapters on equity (cochrane, jbi)47,55 or offer supplemental documentation related to equity (prisma)53. in order to take advantage of the opportunity that an evidence synthesis research consultation provides for discussing health equity with researchers, health information professionals must be conversant with the concept of health equity. after being provided with a definition of health equity, 60% of the survey participants indicated that they were moderately familiar or very familiar with the concept of health equity, while 26% were slightly familiar, 7% were not familiar at all, and 7% did not answer (figure 3). additionally, 42% of participants have discussed health equity with researchers during evidence synthesis collaborations (figure 4). in most cases (75%), the research team initiated the discussions, but in 21% of cases the health information professionals initiated the discussions (figure 5). as noted in correlation chart 1, health information professionals who were very or moderately familiar with the concept of health equity were more likely to have had discussions about the topic during research collaborations on evidence synthesis than were those who were only slightly familiar or unfamiliar with the concept. 8 peer reviewed article hypothesis, vol. 37, no. 2, 2025 figure 3. how familiar are you with the concept of health equity? 9 peer reviewed article hypothesis, vol. 37, no. 2, 2025 figure 4. in your work collaborating with researchers on evidence syntheses, have you ever had discussions about health equity? correlation chart 1. 10 peer reviewed article hypothesis, vol. 37, no. 2, 2025 figure 5. who initiated the discussion about health equity? regarding the relevance of health equity to the evidence synthesis process, nearly three-quarters (64%) of the survey participants believe that health equity discussions are moderately or very relevant to the evidence synthesis process, while 24% believe they are only slightly relevant or not relevant at all (12% did not respond to the question) (figure 6). as shown in correlation chart 2, participants who were moderately or very familiar with the concept of health equity were more likely to indicate that discussions about health equity were very relevant to the evidence synthesis process. additionally, participants who thought health equity was very relevant to the evidence synthesis process were more likely to have had discussions about health equity with researchers (correlation chart 3). 11 peer reviewed article hypothesis, vol. 37, no. 2, 2025 figure 6. how relevant do you think discussions about health equity are in the evidence synthesis process? correlation chart 2. 12 peer reviewed article hypothesis, vol. 37, no. 2, 2025 correlation chart 3. despite the relatively high percentage of participants indicating familiarity with the concept of health equity and affirming the relevancy of health equity to the evidence synthesis process, only about half of the participants surveyed (46%) feel very or somewhat comfortable discussing health equity with researchers (figure 7). about 28% are neither comfortable or uncomfortable, while 14% are somewhat or very uncomfortable. furthermore, even though 79% of the participants have used the prisma reporting guideline, 52% were not at all familiar with the prisma-equity extension and only 6% had used it in a systematic review (figure 8). as expected, those who were moderately or very familiar with the concept of health equity were more likely to be somewhat comfortable or very comfortable discussing the concept with researchers (correlation chart 4). figure 7. how comfortable are you discussing health equity with researchers? 13 peer reviewed article hypothesis, vol. 37, no. 2, 2025 figure 8. how familiar are you with the prisma-equity extension? correlation chart 4. discussion the survey participants who were involved in an evidence synthesis project within the past 10 years (310) exemplified the knowledge and experience reflected in the aforementioned guidelines that identify librarians as collaborators. less than 3% of the participants reported 14 peer reviewed article hypothesis, vol. 37, no. 2, 2025 not using any of the examples of supporting documents provided in the survey. seventy-nine percent of participants referred to prisma in their collaboration on evidence syntheses, the most frequently used supporting document. in addition to demonstrating their familiarity with the supporting documents provided in the survey, the participants provided a substantive list of “other” examples (see supplemental materials, table 1). these “other” examples, created or authored by organizations or individuals in north america and europe, underscore the global nature of evidence synthesis work. prior studies have shown how the role of librarians and information professionals has evolved from designing search strategies to educating researchers in evidence synthesis methodologies33. the participants verified this observation, indicating that in 55% of their experiences, they alone introduced or suggested the supporting documents to the research team (as shown in figure 2). this is consistent with the findings of logan (2023) who found that, among a group of 42 researchers who had co-authored reviews with librarians, 64.3% were motivated by librarians’ methodological expertise32. the participants’ responses to these questions confirmed the authors’ hypothesis, that because health information professionals are more sought after in the production of evidence syntheses, they could be well-positioned to introduce or encourage researchers to bring an equity perspective to this work. when asked about health equity, 86% of the participants had some familiarity with the concept (as shown in figure 3) and when asked whether they had had discussions about health equity with researchers, 42% said yes (as shown in figure 4). in those discussions about health equity, the participants said the researchers mostly initiated the topic. while this finding could reflect the fact that health information professionals are often not the lead researchers on evidence synthesis projects, it could also suggest that there is more to be done to increase their understanding of the concept of health equity and its role in evidence synthesis. it is notable that 42% of participants provided a neutral response to the comfortability question (28%) or expressed some level of discomfort with discussing health equity with researchers (14%) (as shown in figure 7). the level of comfort with discussing the concept could play a role in the willingness to initiate health equity discussions with researchers. while the level of comfort could be related to knowledge or understanding, it could also be due to factors indiscernible from the survey questions. overall the results suggest possible connections between a health information professional’s thinking about the relevance of health equity to evidence synthesis and their experience discussing health equity with researchers. the participants who thought health equity was very relevant to evidence synthesis were more likely to have had discussions about health equity with researchers (correlation chart 3). similarly, participants unfamiliar with the meaning of health equity were more likely to feel uncomfortable discussing health equity with researchers (correlation chart 4). the fact that 52% of participants described themselves as “not at all familiar” with the prisma-equity extension presents an opportunity for health information professionals. given their role in methodology discussions, it is possible that their mentioning prisma-e during these discussions could have a meaningful impact on incorporating health equity into this research. this is not to suggest that prisma-e is the only supporting document that 15 peer reviewed article hypothesis, vol. 37, no. 2, 2025 considers health equity. other tools, like progress plus, could be considered. health information professionals may also take the initiative, in collaboration with their research teams, to develop their own supporting documents. limitations although the authors solicited participants from international professional organizations, the survey was not designed to identify the geographic location of the participants. the quantitative nature of the survey also restricted any possibility for nuance, particularly around the questions about discussions with research teams. while asking who initiated the health equity discussion is important, missing from this survey are details about the quality and dynamics of these discussions with research teams. a possible direction for future research would be to conduct qualitative interviews with health information professionals about experiences collaborating on evidence synthesis and the subject of health equity. this approach may address the limitation of human memory our participants may have had in recalling their experiences collaborating in evidence syntheses. conclusion health information professionals have a history of making meaningful contributions to evidence synthesis, and this study illustrates that they understand the potential of bringing a health equity perspective to this work. their experiences as methodological advisors provide a solid foundation for developing or deepening expertise in equity-centered evidence synthesis methods and tools such as prisma-e. while 36% of the participants in this study had some level of familiarity with the health equity-focused evidence synthesis tool prisma-e, slightly over half (52%) did not, which suggests that librarian training related to this tool would be beneficial. a critical action moving forward will be staying up to date on discussions about tools like prisma-e, as well as other tools for incorporating health equity considerations into the evidence synthesis process, such as progress and progress plus. staying up to date could involve active participation in meetings where equity-centered evidence synthesis tools and methods are being discussed and debated, such as public webinars sponsored by medical research organizations or library organizations. the finding that 42% of the study participants have had health equity discussions with research teams suggests that a substantial number of researchers would be open to health equity-related evidence synthesis guidance conveyed by health information professionals. health information professionals could offer this guidance through research consultations and during library instruction sessions. with their knowledge of information sources, health information professionals are particularly well positioned to convey the guidance that pertains to literature searching (such as encouraging researchers to take an expansive view of what constitutes legitimate research evidence). as health information professionals continue to collaborate on evidence synthesis projects, it is important that they be comfortable communicating about health equity with evidence synthesis researchers. the finding that 72% of study participants are less than “very comfortable” with discussing health equity with researchers is concerning and suggests that health information professionals might benefit from training on how to approach the topic of health equity during research discussions. 16 peer reviewed article hypothesis, vol. 37, no. 2, 2025 the literature supports incorporating health equity considerations and values into every stage of the evidence synthesis process. by obtaining training on equity-centered evidence synthesis methods and bringing a health equity perspective to their collaborations with researchers, health information professionals can strengthen their knowledge base and support the public good. corresponding author stacy torian, librarian for health sciences (allied health), new york university libraries, stacy.torian@nyu.edu ethical declarations because it involved minimal risk of harm to the survey participants, our study received an exemption from u.s. federal policy on human subjects research via our institutional review board. to ensure the privacy of participants, the authors did not collect personal identifiers or geographic location data. the institutional review board of new york university approved the study on december 18, 2023 (study irb-fy2024-7965). data availability statement data are summarized in the tables and figures. credit statement stacy torian: conceptualization, methodology, analysis, data collection, writing, manuscript revision stephen maher: conceptualization, methodology, analysis, data visualization, writing, manuscript revision both torian and maher are lead authors of the manuscript. references 1. cochrane [internet]. london: cochrane; 2000-2024. evidence synthesis what is it and why do we need it?; 2023 may 4 [cited 2024 nov 6]. available from: https://www.cochrane.org/news/evidence-synthesis-what-it-and-why-do-we-need-it 2. grant mj, booth a. a typology of reviews: an analysis of 14 review types and associated methodologies. health information & libraries journal. 2009 jun;26(2):91-108. 3. cochrane library [internet]. london: cochrane; c2000-2024. about cochrane reviews: what is a systematic review?; [cited 2024 nov 7]. available from: https://www.cochranelibrary.com/about/about-cochrane-reviews 4. joanna briggs institute (jbi) [internet]. adelaide: jbi. jbi critical appraisal tools: checklist for systematic reviews and research syntheses; 2017 [cited 2024 nov 8]; p. 2. 17 mailto:stacy.torian@nyu.edu https://www.cochrane.org/news/evidence-synthesis-what-it-and-why-do-we-need-it https://www.cochranelibrary.com/about/about-cochrane-reviews peer reviewed article hypothesis, vol. 37, no. 2, 2025 available from: https://jbi.global/sites/default/files/2019-05/jbi_critical_appraisalchecklist_for_systematic_reviews2017_0.pdf 5. eden j, levit l, berg a, and morton s, editors. finding what works in health care: standards for systematic reviews [internet]. washington, dc: the national academies press; c2011 [cited 2024 nov 6]. available from: https://nap.nationalacademies.org/read/13059/chapter/1 6. american physical therapy association [internet]. alexandria: american physical therapy association. components of evidence-based practice; 2020 mar 23 [cited 2024 nov 6]; available from: https://www.apta.org/patient-care/evidence-based-practiceresources/components-of-evidence-based-practice 7. cumpston m, flemyng e. chapter ii: planning a cochrane review [updated 2023 aug]. in: higgins jpt, thomas j, chandler j, cumpston m, li t, page mj, welch va, editors. cochrane handbook for systematic reviews of interventions [internet]. version 6.5. london (uk): cochrane; c2024 [cited 2024 nov 7]. [about 5 screens]. available from: https://training.cochrane.org/handbook/current/chapter-ii#section-ii-3 8. relevo r, balshem, h. finding evidence for comparing medical interventions. in: methods guide for effectiveness and comparative effectiveness reviews [internet]. rockville (md): agency for healthcare research and quality (us); c2008[cited 2024 nov 7]. available from: https://www.ncbi.nlm.nih.gov/books/nbk53479/ 9. spencer aj, eldredge jd. roles for librarians in systematic reviews: a scoping review. j med libr assoc. 2018 jan 12;106(1). https://doi.org/10.5195/jmla.2018.82 10. petticrew m, whitehead m, macintyre sj, graham h, egan m. evidence for public health policy on inequalities: 1: the reality according to policymakers. j epidemiol community health. 2004 oct;58(10):811-6. https://doi.org/10.1136/jech.2003.015289 11. welch, v., tugwell, p., petticrew, m., de montigny, j., ueffing, e., kristjansson, b., mcgowan, j., benkhalti jandu, m., wells, g., brand, k., & et al. (2010). how effects on health equity are assessed in systematic reviews of interventions. cochrane database of systematic reviews, 12. https://doi.org/10.1002/14651858.mr000028.pub2 12. aves t, kredo t, welch v, mursleen s, ross s, zani b, et al. equity issues were not fully addressed in cochrane human immunodeficiency virus systematic reviews. j clin epidemiol. 2017 jan 1;81:96-100. https://doi.org/10.1016/j.jclinepi.2016.09.008 13. evans j, mwangi n, burn h, ramke j. equity was rarely considered in cochrane eyes and vision systematic reviews and primary studies on cataract. j clin epidemiol. 2020 sep 1;125:57-63. https://doi.org/10.1016/j.jclinepi.2020.04.024 14. world health organization [internet]. geneva: world health organization; 2024. health equity; [cited 2024 nov 12]. available from: https://www.who.int/health-topics/health-equity#tab=tab_1 18 https://jbi.global/sites/default/files/2019-05/jbi_critical_appraisal-checklist_for_systematic_reviews2017_0.pdf https://jbi.global/sites/default/files/2019-05/jbi_critical_appraisal-checklist_for_systematic_reviews2017_0.pdf https://nap.nationalacademies.org/read/13059/chapter/1 https://www.apta.org/patient-care/evidence-based-practice-resources/components-of-evidence-based-practice https://www.apta.org/patient-care/evidence-based-practice-resources/components-of-evidence-based-practice https://training.cochrane.org/handbook/current/chapter-ii#section-ii-3 https://www.ncbi.nlm.nih.gov/books/nbk53479/ https://doi.org/10.5195/jmla.2018.82 https://doi.org/10.1136/jech.2003.015289 https://doi.org/10.1002/14651858.mr000028.pub2 https://doi.org/10.1016/j.jclinepi.2016.09.008 https://doi.org/10.1016/j.jclinepi.2020.04.024 https://www.who.int/health-topics/health-equity#tab=tab_1 peer reviewed article hypothesis, vol. 37, no. 2, 2025 15. burford bj, welch v, waters e, tugwell p, moher d, o’neill j, koehlmoos t, petticrew m. testing the prisma-equity 2012 reporting guideline: the perspectives of systematic review authors. plos one. 2013 oct 10;8(10):e75122. 16. welch va, petticrew m, o’neill j, waters e, armstrong r, bhutta za, et al. health equity: evidence synthesis and knowledge translation methods. syst rev. 2013 dec;2(1):43. https://doi.org/10.1186/2046-4053-2-43 17. parker r, petkovic j, pardo pardo j, darzi a, dewidar o, khabsa j, et al. the equity group: supporting cochrane’s social responsibility of improving health equity. cochrane evid synth meth. 2023 may;1(3):e12012. https://doi.org/10.1002/cesm.12012 18. petticrew, m, glover re, volmink j, blanchard l, cott é, knai c, et al. the commercial determinants of health and evidence synthesis (codes): methodological guidance for systematic reviews and other evidence syntheses. syst rev. 2023 sep 14; 12(1):165. https://doi.org/10.1186/s13643-023-02323-0 19. schell cl, rathe rj. meta-analysis: a tool for medical and scientific discoveries. bull med libr assoc. 1992 jul;80(3):219. 20. mead tl, richards dt. librarian participation in meta-analysis projects. bull med libr assoc. 1995 oct;83(4):461. 21. smith jr. jt. meta-analysis: the librarian as a member of an interdisciplinary research team. libr trends. 1996;45(2):265–79. 22. medline history [internet]. u.s. national library of medicine; [cited 2024 nov 1]. available from: https://www.nlm.nih.gov/medline/medline_history.html 23. bastian h, glasziou p, chalmers i. seventy-five trials and eleven systematic reviews a day: how will we ever keep up? plos med. 2010 sep 21;7(9):e1000326. https://doi.org/10.1371/journal.pmed.1000326 24. higgins jpt, thomas j, chandler j, cumpston m, li t, page mj, et al., editors. cochrane handbook for systematic reviews of interventions [internet]. vol. version 6.5 (updated august 2024). cochrane; 2024 [cited 2024 nov 5]. available from: https://training.cochrane.org/handbook 25. aromataris e, lockwood c, porritt k, pilla b, jordan z, editors. jbi manual for evidence synthesis [internet]. jbi; 2024 [cited 2024 nov 5]. available from: https://jbi-global-wiki.refined.site/space/manual 26. arksey h, o’malley l. scoping studies: towards a methodological framework. int j soc res methodol. 2005 feb 1;8(1):19–32. https://doi.org/10.1080/1364557032000119616 27. rethlefsen ml, farrell am, osterhaus trzasko lc, brigham tj. librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews. j clin epidemiol. 2015 jun 1;68(6):617–26. 19 https://doi.org/10.1186/2046-4053-2-43 https://doi.org/10.1002/cesm.12012 https://doi.org/10.1186/s13643-023-02323-0 https://www.nlm.nih.gov/medline/medline_history.html https://doi.org/10.1371/journal.pmed.1000326 https://training.cochrane.org/handbook https://jbi-global-wiki.refined.site/space/manual https://doi.org/10.1080/1364557032000119616 peer reviewed article hypothesis, vol. 37, no. 2, 2025 https://doi.org/10.1016/j.jclinepi.2014.11.025 28. koffel jb, rethlefsen ml. reproducibility of search strategies is poor in systematic reviews published in high-impact pediatrics, cardiology and surgery journals: a cross-sectional study. plos one. 2016 sep 26;11(9):e0163309. https://doi.org/10.1371/journal.pone.0163309 29. meert d, torabi n, costella j. impact of librarians on reporting of the literature searching component of pediatric systematic reviews. j med libr assoc. 2016;104(4):267–77. https://doi.org/10.3163/1536-5050.104.4.004 30. ramirez d, foster mj, kogut a, xiao d. adherence to systematic review standards: impact of librarian involvement in campbell collaboration’s education reviews. j acad librariansh. 2022 sep 1;48(5):102567. https://doi.org/10.1016/j.acalib.2022.102567 31. schellinger j, sewell k, bloss j, ebron t, forbes c. the effect of librarian involvement on the quality of systematic reviews in dental medicine. plos one. 2021 sep 1;16(9). https://doi.org/10.1371/journal.pone.0256833 32. logan j. why do researchers co-author evidence syntheses with librarians? a mixed-methods study. res synth methods [internet]. 2023 feb 20 [cited 2023 mar 23]. https://doi.org/10.1002/jrsm.1629 33. nicholson j, mccrillis a, williams jd. collaboration challenges in systematic reviews: a survey of health sciences librarians. j med libr assoc. 2017 oct 2;105(4):385–93. https://doi.org/10.5195/jmla.2017.176 34. ross-white a. search is a verb: systematic review searching as invisible labor. j med libr assoc. 2021 oct 5;109(3):505–9. https://doi.org/10.5195/jmla.2021.1226 35. bloss j, sewell k, schellinger j, haberstroh a. health sciences and medical librarians conducting research and their experiences asking for co-authorship. j med libr assoc. 2022 oct;110(4):449–62. https://doi.org/10.5195/jmla.2022.1485 36. brunskill a, hanneke r. the case of the disappearing librarians: analyzing documentation of librarians’ contributions to systematic reviews. j med libr assoc. 2022 oct;110(4):409–18. https://doi.org/10.5195/jmla.2022.1505 37. demetres m, wright d, hickner a, jedlicka c, delgado d. a decade of systematic reviews: an assessment of weill cornell medicine’s systematic review service. j med libr assoc. 2023 jul;111(3):728–32.12. https://doi.org/10.5195/jmla.2023.1628 38. kallaher a, eldermire e, fournier c, ghezzi-kopel k, johnson k, morris-knower j, et al. library systematic review service supports evidence-based practice outside of medicine. j acad librariansh. 2020 nov;46(6). https://doi.org/10.1016/j.acalib.2020.102222 39. luca e, ulyannikova y. towards a user-centred systematic review service: the transformative power of service design thinking. j aust libr inf assoc. 2020;69(3):357–74. 20 https://doi.org/10.1016/j.jclinepi.2014.11.025 https://doi.org/10.1371/journal.pone.0163309 https://doi.org/10.3163/1536-5050.104.4.004 https://doi.org/10.1016/j.acalib.2022.102567 https://doi.org/10.1371/journal.pone.0256833 https://doi.org/10.1002/jrsm.1629 https://doi.org/10.5195/jmla.2017.176 https://doi.org/10.5195/jmla.2021.1226 https://doi.org/10.5195/jmla.2022.1485 https://doi.org/10.5195/jmla.2022.1505 https://doi.org/10.5195/jmla.2023.1628 https://doi.org/10.1016/j.acalib.2020.102222 peer reviewed article hypothesis, vol. 37, no. 2, 2025 https://doi.org/10.1080/24750158.2020.1760506 40. mckeown s, ross-white a. building capacity for librarian support and addressing collaboration challenges by formalizing library systematic review services. j med libr assoc. 2019 jul;107(3):411–9. https://doi.org/10.5195/jmla.2019.443 41. roth s. transforming the systematic review service: a team-based model to support the educational needs of researchers. j med libr assoc. 2018 oct;106(4):514–20. https://doi.org/10.5195/jmla.2018.430 42. yang l, orchanian-cheff a, anderson m, farrell a, tripp t. implementing a three-tiered service model for knowledge syntheses at an academic teaching hospital. j can health libr assoc. 2020 aug;41(2):46–66. https://doi.org/10.29173/jchla29436 43. waters e, armstrong r. knowledge synthesis, translation and exchange in public health and promotion: the role of a cochrane review group. prompt educ. 2007; 14(1): 34-35. 44. evans t, brown h. road traffic crashes: operationalizing equity in the context of health sector reform. inj control saf promot. 2003 mar-jun;10(1-2):11-2. https://doi.org/10.1076/icsp.10.1.11.14117 45. karran el, cashin ag, barker t, boyd ma, chiarotto a, dewidar o, et al. using progress-plus to identify current approaches to the collection and reporting of equity-relevant data: a scoping review. j clin epidemiol. 2023 nov;163:70-78. https://doi.org/10.1016/j.jclinepi.2023.09.017 46. kavanagh j, oliver s, and lorenc t. reflections on developing and using progress-plus. equity update [internet]. 2008 mar 3 [cited 2014 nov 7]; 2(1): 1,3. available from: http://methods.cochrane.org/sites/methods.cochrane.org.equity/files/uploads/equity _update_vol2_issue1.pdf 47. welch va, petkovic j, jull j, hartling l, klassen t, kristjansson e, et al. chapter 16: equity and specific populations [updated october 2019]. in: higgins jpt, thomas j, chandler j, cumpston m, li t, page mj, welch va, editors. cochrane handbook for systematic reviews of interventions [internet]. version 6.5. london (uk): cochrane; c2024 [cited 2024 nov 7]. available from: https://training.cochrane.org/handbook/current/chapter-16 48. madani mt, madani l, ghogomu et, dahrouge s, hébert pc, juando-prats c, mulligan k, welch v. is equity considered in systematic reviews of interventions for mitigating social isolation and loneliness in older adults? bmc public health. 2022 dec 1;22(1):2241. https://doi.org/10.1186/s12889-022-14667-8 49. hartwell m, lin v, gatewood a, sajjadi nb, garrett m, reddy ak, et al. health disparities, covid-19, and maternal and childbirth outcomes: a meta-epidemiological study of equity reporting in systematic reviews. j matern fetal neonatal med. 2022 dec 12;35(25):9622-30. https://doi.org/10.1080/14767058.2022.2049750 21 https://doi.org/10.1080/24750158.2020.1760506 https://doi.org/10.5195/jmla.2019.443 https://doi.org/10.5195/jmla.2018.430 https://doi.org/10.29173/jchla29436 https://doi.org/10.1076/icsp.10.1.11.14117 https://doi.org/10.1016/j.jclinepi.2023.09.017 http://methods.cochrane.org/sites/methods.cochrane.org.equity/files/uploads/equity_update_vol2_issue1.pdf http://methods.cochrane.org/sites/methods.cochrane.org.equity/files/uploads/equity_update_vol2_issue1.pdf https://training.cochrane.org/handbook/current/chapter-16 https://doi.org/10.1186/s12889-022-14667-8 https://doi.org/10.1080/14767058.2022.2049750 peer reviewed article hypothesis, vol. 37, no. 2, 2025 50. commission on social determinants of health. closing the gap in a generation: health equity through action on the social determinants of health. final report of the commission on social determinants of health. geneva: world health organization; 2008. 247 p. 51. tugwell p, petticrew m, kristjansson e, welch v, ueffing e, waters e, et al. assessing equity in systematic reviews: realising the recommendations of the commission on social determinants of health. bmj. 2010 sep 13;341:c4739. https://doi.org/10.1136/bmj.c4739 52. rio political declaration on social determinants of health [internet]. world health organization. geneva: world health organization; 2011 oct 19-21 [cited 2024 nov 13]. 7 p. available from: https://www.who.int/publications/m/item/rio-political-declaration-on-socialdeterminants-of-health 53. welch v, petticrew m, tugwell p, moher d, o’neill j, waters e, et al. prisma-equity 2012 extension: reporting guidelines for systematic reviews with a focus on health equity. plos med. 2012 oct 30; 9(10): e1001333. https://doi.org/10.1371/journal.pmed.1001333 54. welch v, petticrew m, petkovic j, moher d, waters e, white h, et al. extending the prisma statement to equity-focused systematic reviews (prisma-e 2012): explanation and elaboration. int j equity health. 2015 oct 8;14:92. https://doi.org/10.1186/s12939-015-0219-2 55. aromataris e, lockwood c, porritt k, pilla b, jordan z, editors. jbi manual for evidence synthesis [internet]. adelaide: jbi; c2024. chapter 2.1, equity, diversity, and inclusion; [cited 2024 nov 8]. available from: https://jbi-globalwiki.refined.site/space/manual/355827799/2.1+equity,+diversity,+and+inclusion 56. white m, adams j, heywood p. how and why do interventions that increase health overall widen inequalities within populations?. in social inequality and public health 2009 apr 22 (pp. 65-82). bristol, uk: policy press; c2009: p. 65-82. https://doi.org/10.51952/9781847423221.ch005 57. cochrane training [internet]. [place unknown]: cochrane and health equity thematic group; 2024 november. [video], equity in all cochrane reviews; [cited 2025 january 22]; [32 min., 52 sec]. available from: https://training.cochrane.org/resource/equity-in-all-cochrane-reviews 58. whitehead m. the concepts and principles of equity and health. int j health serv. 1992;22(3):429-45. https://doi.org/10.2190/986l-lhq6-2vte-yrrn 59. bosch-capblanch x, lavis jn, lewin s, atun r, røttingen ja, dröschel d, beck l, abalos e, el-jardali f, gilson l, oliver s, wyss k, tugwell p, kulier r, pang t, haines a. guidance for evidence-informed policies about health systems: rationale for and challenges of guidance development. plos med. 2012;9(3):e1001185. https://doi.org/10.1371/journal.pmed.1001185 60. qualtrics. qualtrics xm. provo (ut): qualtrics. c2005 [updated 2024 march; cited 2024 nov 14]. available from: https://www.qualtrics.com 22 https://doi.org/10.1136/bmj.c4739 https://www.who.int/publications/m/item/rio-political-declaration-on-social-determinants-of-health https://www.who.int/publications/m/item/rio-political-declaration-on-social-determinants-of-health https://doi.org/10.1371/journal.pmed.1001333 https://doi.org/10.1186/s12939-015-0219-2 https://jbi-global-wiki.refined.site/space/manual/355827799/2.1+equity,+diversity,+and+inclusion https://jbi-global-wiki.refined.site/space/manual/355827799/2.1+equity,+diversity,+and+inclusion https://doi.org/10.51952/9781847423221.ch005 https://training.cochrane.org/resource/equity-in-all-cochrane-reviews https://doi.org/10.2190/986l-lhq6-2vte-yrrn https://doi.org/10.1371/journal.pmed.1001185 https://www.qualtrics.com peer reviewed article hypothesis , vol. 35, no. 2, 2023 gaining ground: oer at three health sciences institutions jessica kirschner, mlisa, jenn monnin, mlisb, christine andresen, mls, msitc adigital publishing coordinator, viva (virginia’s academic library consortium), richmond, virginia, https://www.orcid.org/0000-0002-7277-9530 , jkirsch4@gmu.edu bscholarly engagement librarian, health sciences library, west virginia university, morgantown, west virginia, https://www.orcid.org/0000-0002-4842-2814 , jennifer.monnin@mail.wvu.edu cresearch and education informationist, musc libraries, medical university of south carolina, charleston, sc, https://www.orcid.org/0000-0001-5551-1770 , andresen@musc.edu background: open educational resources (oer) are rising in popularity at higher education institutions to help combat high textbook costs. to date, little research has been published discussing the impact of oer in the health sciences context. this comparative case study seeks to fill this gap. experience: the article shares oer programs and advocacy efforts at three institutions, including a brief history of oer work, barriers and failures, current successes, and future directions at each institution. discussion: all three institutions are making efforts in oer outreach and advocacy, informed and impacted by their institutional community. two institutions have an oer grant with increasing submissions from health sciences faculty. oer work is completed by one librarian at two of the institutions, while the third completes the work through a committee of stakeholders from across the institution. all cases include oer advocacy and outreach through library workshops and working to establish faculty partnerships. barriers at all three institutions include a lack of time and funds to dedicate to finding or developing oer. unique barriers include a desire to work with for-profit companies, concerns over disadvantaging students on their qualifying exams, and the sustainability of institutional efforts in oer. takeaways: libraries are uniquely situated to support faculty and staff as they incorporate oer into their practices. while health sciences faculty are typically slower to adopt oer, once they do take this step, they become strong advocates for open practices, the librarian, and the library. background textbooks and ancillary materials are a key component of higher education, providing important content and helping to guide student learning. yet textbook costs have risen exponentially, often at a rate faster than other industries, including child care, housing, and food.1 these disproportionately high costs impact students’ academic careers, including which courses students take, how well they do in those courses, and which majors they select.2-4 open educational resources (oer) have emerged as one solution to address the barriers that high-cost textbooks create and enhance student success. 1 https://www.orcid.org/0000-0002-7277-9530 https://orcid.org/ mailto:jkirsch4@gmu.edu https://www.orcid.org/0000-0002-4842-2814 https://orcid.org/ mailto:jennifer.monnin@mail.wvu.edu https://www.orcid.org/0000-0001-5551-1770 https://orcid.org/ mailto:andresen@musc.edu peer reviewed article hypothesis , vol. 35, no. 2, 2023 unesco defines oer as learning, teaching and research materials in any format and medium that reside in the public domain or are under copyright that have been released under an open license, that permit no-cost access, re-use, re-purpose, adaptation and redistribution by others.5 an oer can be any type of instructional material that has been openly licensed, including textbooks, videos, images, slide decks, and question banks. although a few oer publishers have emerged (e.g. openstax6), most oer are created by faculty as a perfect resource for a class they teach. sometimes, students become a central part in the creation of oer, a practice known as open pedagogy or oer-enabled pedagogy. examples of open pedagogy include collaboratively authoring a textbook as a part of a class, editing wikipedia articles, or generating study guide questions.7,8 studies show that oer can alleviate student financial burdens, reduce barriers to access, and increase learner engagement, with similar or improved learning outcomes compared to their commercial counterparts.9,10 as such, these resources have gained popularity, especially in high enrollment and general education courses. to support and encourage this growth, many academic libraries have developed oer initiatives. libraries are a natural fit to lead textbook affordability initiatives, which are focused on increasing access and student success. this focus is an extension of the library values of universal access and supporting the students, teachers, and researchers of their home institutions. the work is similarly an extension of both long-existing library services, such as instruction support, and emerging services, such as publishing support.11-13 often in partnership with other university units, library oer initiatives educate about, advocate for, and support the adoption, adaptation, and creation of oer. instructors typically have a low awareness and understanding of oer and their underlying copyright licenses.14,15 in order to encourage oer usage, library resources and staff work to overcome this barrier, as well as other commonly articulated barriers to adoption, such as difficulty in locating appropriate resources or concerns about quality.14,16,17libraries also often administer grant programs and/or provide access to platforms to enable the creation of resources.11 research shows that these initiatives result in a higher adoption of oer resources.18 oer in health sciences many case studies provide an overview of oer initiatives or analyze the impact of oer in academic courses (e.g., see the open education group bibliography).19 however, few articles study oer specifically in the health sciences context. those that do venture into this area may review health science specific resources, like free open access medical education (foam), a website that shares oer for medical education.20-22 additionally, a few case studies explore library oer support in the health sciences, such as bridgeman’s review of the open and affordable textbooks 2 peer reviewed article hypothesis , vol. 35, no. 2, 2023 program at rutgers biomedical and health sciences, which shares successes such as a student-developed course pack for a psychiatric clerkship program and an open radiology textbook developed in collaboration with students.23 it is unclear whether this limited existing literature results from a lack of write-ups, the focus of initiatives on undergraduate courses, or a true lack of oer use in the field. like their academic peers, students in health sciences feel the burden of the cost of higher education. according to the national center for education statistics, the average loan balances for those who complete medical and health sciences doctoral programs doubled between 1999/2000 and 2015/2016. by that time, nearly 75% of health sciences doctoral students and 81% of medical students were burdened with student loans.24 while textbooks may seem like a small portion of this debt, the cost of textbooks can have a disproportionate impact on students’ higher education experiences. according to a survey of over 700 institutions, most emergency aid for students (e.g. campus vouchers or emergency loans) is between $100-$500.25 this amount can equal the cost of a student’s textbooks. since faculty can choose which texts to assign, they have the power to remove this stressor and impediment to success, completion, and retention by using oer. this article aims to fill a gap in the literature by providing three case studies of library support for oer in the health sciences. these experiences will highlight both the library programs and the concerns and interests of the health sciences faculty served. by sharing our stories, the authors hope to provide an understanding of the unique barriers health sciences faculty may face in adopting oer and outline a foundation for other librarians looking to build support for oer in the health sciences. experiences in this article, the authors share their experiences supporting oer in the health sciences at three institutions: virginia commonwealth university (vcu), west virginia university (wvu), and the medical university of south carolina (musc). appendix a26-30 provides an overview of these institutions. in each section, the author’s initials are used in place of other pronouns: jk in vcu, jm in wvu, and ca in musc. vcu background: oer at vcu vcu’s diverse student body includes a large percentage with backgrounds especially vulnerable to textbook costs, such as underrepresented minorities, pell grant recipients, and first-generation students (all approximately 1/3 of incoming freshman classes).27,31 because of this student body composition, textbook affordability has been a priority for vcu libraries (vcul) since 2016,32 beginning with a statewide membership with the open education network (oen) through viva, virginia’s academic library consortium, and corresponding open textbook workshops. in 2017, vcul launched a grant program, the affordable course content awards,33 to 3 peer reviewed article hypothesis , vol. 35, no. 2, 2023 provide financial and project support to those looking to adopt, customize, or create free resources. this program is offered annually, funding 32 projects over six cohorts. viva also awards grants for oer adoption and creation, which are open to all faculty in virginia. vcu faculty typically apply each viva grant cycle, with 11 successful applications over 5 cycles. to support the initiative, vcul hired a dedicated oer librarian in 2019. while based on the monroe park campus, the oer librarian supports the whole vcu community. since then, vcu’s open and affordable course content initiative has slowly grown. starting in early 2022, the initiative began hiring student workers to support faculty oer publishing, primarily via a locally hosted pressbooks instance.34 the current initiative focuses on support of textbook affordability through consultations, outreach and education (e.g. workshops), a grant program, and oer creation support. some hurdles emerged from supporting one initiative across two campuses, such as where to schedule workshops to ensure campus parity. these barriers will not be explored here, as the article focuses specifically on support on the health sciences campus. barriers and failures oer support is equally available to both vcu campuses. however, uptake seems to be slower on the mcv campus. while this difference could be partly due to mcv’s smaller size (5 schools vs monroe park’s 10), it may be more reflective of the priorities and availability of those on the health sciences campus. many mcv faculty have clinical responsibilities alongside teaching and research requirements, leaving them less time to focus on other projects. confusion about oer and the grant program often emerges when discussing oer with mcv faculty. while confusion about oer seems omnipresent in oer advocacy at vcu, the latter seems to be unique to the health sciences campus. for example, some expressions of interest for vcul’s grant program focused on funding the development of new courses (e.g. a complete canvas course shell), rather than new course materials (e.g. a textbook or assignments). while this confusion may result from the language used in marketing, similar questions are not encountered on the academic campus. further, these projects often focus on non-credit-bearing classes, such as preparation for global residencies, making the projects misaligned with the grant programs. jk has also encountered faculty interest in working with or publishing materials via commercial options, such as partnering with a commercial company to produce an app for diagnostics. while this preference for commercial options is not unique to the health sciences, this hesitation to move to free and open seems more pronounced in the consultations with mcv faculty. even when there is interest in oer, finding appropriate resources is often more difficult due to fewer available oer in the health sciences. to overcome this dearth, jk recommends faculty search granularly by specific topics to find short, focused resources, which are easier to create and thus more likely to exist. these resources can then be remixed into a larger resource aligned with class syllabi. jk also gives this recommendation on the academic campus for more niche topics. 4 peer reviewed article hypothesis , vol. 35, no. 2, 2023 successes even with these hurdles, jk has encountered oer successes on the mcv campus. faculty requests for assistance in locating affordable textbooks are consistent, if few in number. one faculty member even expressed interest in developing a structured open pedagogy project, which was the first such consultation jk received across both campuses. although few health sciences faculty attended open education week events or open textbook workshops in person, when the latter moved to online and asynchronous in summer 2022, 7 out of 28 participants (25%) were from the mcv campus, including 4/20 who completed the course (20%). four of the 7 enrolled mcv faculty had not previously interacted with jk through workshops or consultations, which implies they enrolled out of an interest in textbook affordability. additionally, vcul has seen a number of health sciences grant applications for both the institutional and state grant programs. out of these, four projects were successfully awarded local grants and one was awarded state funding. these funded projects are some of the most innovative and diverse that vcul has supported. examples include an interactive histology atlas,35 which is one of the few vcu oer to break from replicating the print book to take advantage of the digital environment, and student-created modules reviewing dermatological manifestations of rheumatological diseases with a focus on providing examples from persons of color,36-38 filling a gap in existing medical education. these faculty–and students–made creating free and pedagogically advantageous resources a priority. they also became oer advocates, many discussing both oer and the grant program with their colleagues. faculty development staff at health sciences schools also emerged as key oer advocates, a relationship not replicated on the academic campus. these staff are always willing to promote vcul programs, which is often more impactful than emails from the library. they also invited jk to give school-specific workshops, which has rarely happened otherwise. jk also delivered department-level presentations, again, a rare occurrence elsewhere. for those who cannot find an oer and do not wish to create something new, vcul offers support for unlimited license ebooks. although not open, these materials offer professors a wider choice of available resources, oftentimes the same that they are already assigning, while still enabling free access for students. health sciences course materials are often more standardized than those on the academic campus, so ebooks can have a long and meaningful impact when open is not available. future directions vcul will continue to ensure that the health sciences campus is included in decisions, from scheduling and logistics to communications. vcul may explore departmental-specific outreach, which has already seen high interest and impact on the mcv campus. 5 peer reviewed article hypothesis , vol. 35, no. 2, 2023 wvu background: oer at wvu the wvu libraries (wvul) oer committee was established in 2016. over time, the oer committee became large enough that two subcommittees were formed: the oer grant subcommittee and the oer advocacy and faculty development subcommittee. most of the committee work is completed through these subcommittees, with both groups occasionally convening to discuss oer at the institution more broadly. most wvul locations, the wvu teaching and learning commons (tlc), and the provost’s office have representation on the committee, with the potential for a representative from the student government association as well. library administration is highly supportive of incorporating open practices into all library work. in fact, values of open access were incorporated directly into the 2023 strategic roadmap. goal 2 of the roadmap is to “be a leader in the creation and dissemination of knowledge,”39 with an initiative to “pursue alternative research and publishing models to improve access to materials in anticipation of rising collections costs and budgetary restrictions.”39 successes wvul’s oer initiatives and support have grown over the years. currently, the library has an oer grant, an open textbook workshop initiative, the shining minds textbook program, and an educational workshop series. because oer work is completed through the oer committee, it is hard to parse out health sciences specific participation. that said, all opportunities discussed are equally available to the health science center (hsc), and three health sciences librarians serve on the oer committee. the oer grant was born out of a partnership between wvul and the tlc. the purpose is to “encourage development of alternatives to high cost textbooks, lower the cost of college for students, and support faculty who wish to implement new pedagogical models for classroom instruction.”40 the grant has three available categories: incorporate, for instructors seeking to replace an existing commercial textbook with a no or low-cost (less than $50) alternative; innovate, for instructors with plans to remix new or existing openly licensed content for their course; and create, for instructors with plans to create a new oer to meet the needs of their course. library support is available for awardees in all three categories, including creative commons license education and subject liaison librarian consultations. submissions for this grant run annually and have grown over time, including some successful submissions from health sciences instructors. the open textbook workshop initiative is a combination training program and open textbook review. attendees participate in a two-hour session on discovering open textbooks in their field, then write a short review of an open textbook. after attending both the session and writing their review, instructors are eligible to receive a $200 stipend for their efforts.41 6 peer reviewed article hypothesis , vol. 35, no. 2, 2023 the shining minds textbook program was established in 2021. when students own textbooks published within the last 10 years that they no longer need, they can donate them to the library. these textbooks are then cataloged and shelved together within the library, and students can check out a book for the semester.42 while not oer, this program increases access to some textbooks for students, with health science library’s (hsl) shining minds textbook collection continuing to grow. in addition to these programs, members of the wvul oer committee hosted a workshop series titled exploring open education: concepts and applications during open education week 2022. faculty, staff, and graduate students attended sessions on oer, open pedagogy, and creative commons; how to find oer; popular platforms for publishing oer; and an information session about applying for the wvu oer grant.43 the committee received positive feedback. attendance was higher than anticipated, with a core group of people attending multiple sessions throughout the week. barriers and failures at wvu, faculty in the health sciences are hesitant to adopt oer. a common barrier is the perceived time investment given competing responsibilities and priorities. some faculty in the hsc have teaching responsibilities, but other faculty spend most of their time in clinics and/or teaching medical residents at the point of care. wvul has struggled to make inroads with this latter group. similarly, given wvu’s r1 status, the institution places emphasis on research activities, leaving faculty little time to explore ways to incorporate oer into their teaching responsibilities. anecdotally, another reason health sciences faculty hesitate to incorporate oer is the fear of inadvertently hindering students’ success on their qualifying exams. jm has suggested to faculty that they could incorporate open pedagogy into program design, specifically through creating study material for board exams as students progress through the program. while this idea works in theory, it remains untested at the time of this writing. future directions there are several projects or initiatives on the horizon. several stem from wv house bill 2853, which mandates the expansion of oer for public colleges and universities in west virginia.44 this mandate will require a course marking system as well as annual reporting to the state on the number of oer courses offered, the amount of students taking these courses, costs savings per student, and aggregated savings for all students. tracking this data will also be useful for wvul in measuring the culture of oer at the institution over time. the oer advocacy and faculty development subcommittee is piloting an affordable syllabus review service. this on-demand service is available to wvu faculty and graduate student instructors looking to reduce the cost of course materials. instructors will work with a librarian to review their syllabus for affordable, free, and/or openly licensed alternatives to current commercial course content. the service is also able to 7 peer reviewed article hypothesis , vol. 35, no. 2, 2023 help instructors identify course materials that represent perspectives and content from/about historically marginalized populations and scholars, introducing representative and recognitive justice45 principles into consultations. at this time, the pilot has stalled due to staffing turnover. wvul also aims to create a health sciences specific oer guide, with strategies for finding, implementing, and creating open resources in the health sciences, and revise the oer grant application. many new open services and programs are likely to follow given the immense support of library administration for incorporating open practices into library work. musc background: oer at musc musc libraries (muscl) is in the preliminary stages of developing an oer program and does not currently have formal affordable learning initiatives in place. early efforts focused on promoting library-subscribed resources and investigating institutional purchasing options for all requested materials to alleviate student costs. recognizing muscl’s role in leading affordable learning initiatives at our institution, library administration encouraged ca to earn a certificate in oer librarianship from the oen in 2022, to lead the investigation to identify best practices for oer initiatives in health sciences education and implement them at musc.46 with support from the provost’s office and the office of institutional effectiveness, reducing student costs has become a priority across campus, opening the door for oer efforts to begin. successes as an institution in the early stages of launching an oer initiative, musc is still in the initial phase of outreach and communication but is well-positioned to make great strides during the 23-24 academic year. ca was selected as one of musc’s inaugural education innovation advocates (eia), a pilot program encouraging faculty to “think big” and reimagine the teaching and learning experience. this created an opportunity to bring oer to the attention of senior leadership and garner their invaluable support,47 and provided a contact with a direct line of communication to each college. this role also provides the library with unique opportunities to connect with faculty, including an oer poster presentation at an institutional teaching and learning conference and a corresponding video story to raise awareness about oer on campus. additionally, ca is a designated member of musc’s education committee, whose membership includes key stakeholders in the instructional/educational technology departments across campus, again providing the library with a valuable opportunity to communicate oer efforts directly to individuals who support teaching faculty. building relationships and opening lines of communication has been our biggest success to date. barriers and failures barriers to oer adoption at an academic health sciences center are not significantly 8 peer reviewed article hypothesis , vol. 35, no. 2, 2023 different than at a multidisciplinary university. barriers include a lack of awareness about the existence of oer and its benefits and where to find high-quality oer. musc’s health sciences faculty consider content, design, usability, engagement, and readability when assessing quality for oer adoption. competing priorities are also a challenge for musc faculty who do not have the time or energy to find and appraise oer to redesign their courses around. conversations with musc faculty identified additional barriers such as the lack of funding or awards to encourage oer adoption/creation and uncertainty about the long-term sustainability of institutional oer initiatives. future directions the future landscape of oer initiatives is uncertain as musc explores uncharted territory, but first steps include expanding outreach and education efforts with an oer libguide and workshop offerings. the partnership among south carolina academic libraries (pascal) launched south carolina affordable learning (scale) in 2020, an initiative seeking to reduce the cost of higher education by promoting the use of high-quality, low-cost/no-cost learning materials. muscl plans to apply for the next cycle of the scale affordable learning grants, which aim to support member libraries in increasing institutional awareness of affordable learning initiatives and create programs that encourage the adoption, revision, or development of oer.48 with support from musc leadership, ca hopes to form an interdisciplinary oer committee. this group of key stakeholders would be charged with planning major oer and oa initiatives, including grant funding, an author fund, publishing support, faculty awards, bootcamps, and microcredentials for faculty who demonstrate mastery with affordable learning educational innovations. discussion each of the presented oer programs varies in its development, growth, successes, and failures. see appendix b for a side by side comparison of the institutions. while each institution’s story is reflective of its individual context and populations, reviewing them in tandem can begin to shed light on the state of oer at health sciences supporting institutions. oer initiatives these three institutions follow the common trend of strong library support and advocacy for developing, growing, and sustaining their campus’s oer initiatives. despite this common support, the three institutions vary in the current state of their initiatives. while both vcu’s and wvu’s efforts began around 2016, musc is in the early stages of initiative development. this gap may be reflective of the differing focuses of the institutions. vcu and wvu both include academic departments, which align with the original focus of many oer efforts, introductory and general education courses, as exemplified by openstax’s core textbook offerings.6 meanwhile, musc is solely focused on health sciences, disciplines that have traditionally been slower to turn to open education. the health sciences focus may also be tied to musc’s worry about 9 peer reviewed article hypothesis , vol. 35, no. 2, 2023 the long-term sustainability of their institutional oer efforts. while the other institutions may have questions about the future directions of their initiatives, they do not share musc’s uncertainty about initiative sustainability. this worry could alternately be tied to its early stage, while the other two initiatives studied here have a proven record. in another difference, wvu is the sole case where a majority of oer work is completed at the committee level. the inclusion of important stakeholders from across the institution helps demonstrate oer as an institutional priority and keeps the library informed of other oer-related initiatives and ideas across the organization. although not yet successful, both vcu and musc have expressed interest in replicating this model for exactly the reasons wvu has chosen to adopt it. even though musc is just starting oer work, their efforts already seem to be gaining interest on the institutional level. the inclusion of the librarian in the eia program and on the education committee will be excellent opportunities for oer outreach and advocacy efforts, as well as an excellent place to find future collaborators. in our discussions, the authors also learned that both vcu and wvu have numerous initiatives in the realm of open access more broadly, and musc also has interest in expanding into open access more generally. many of these open access efforts have seen wide success in the health sciences, likely due to those disciplines’ focus on research and health sciences funding mandates for open access publications.49,50 further discussion of other open efforts is outside the scope of this article. faculty engagement despite wvu’s and vcu’s successful oer initiatives, both case studies reflect that health sciences faculty are slower to engage with oer than those in other disciplines. however, this slow engagement does not correspond to a lack of interest. musc’s celebration of their nascent exploration into oer shows how the concept resonates with health sciences faculty and administration. vcu’s and wvu’s initiatives continually see interest from health sciences faculty in workshops and grants, emphasizing that these faculty often share the same dedication to teaching evidenced by those in other disciplines. while there is no clear reason for the lack of high health sciences faculty engagement with oer, a couple of possibilities emerge from the case studies. first, all three narratives emphasize that health sciences faculty face many competing responsibilities. in addition to teaching and a focus on research, many health sciences faculty have clinical responsibilities, which often take priority. this addition to their workload makes it challenging to find time to find, implement, or create oer. additionally, the challenge of finding high-quality, relevant oer presents a strong barrier to adoption. this issue has been encountered by all three institutions, although not explicitly discussed in wvu’s case study. while difficulty finding appropriate resources is a general barrier to oer adoption,14,17 it is a lower hurdle for those in general education due to the emergence of resources like openstax.6 conversely, the authors found that health sciences instructors must often search oer databases and 10 peer reviewed article hypothesis , vol. 35, no. 2, 2023 the web for appropriate and quality resources that are aligned with their courses, a further strain on their busy schedules. few to no existing resources on a topic means that faculty are not able to remix oer into a new, customized resource for their class. instead, they must create new content, an even larger ask amidst their competing priorities. both wvu and vcu sought to work around the issue by focusing on providing free access to current textbooks to students, whether through library-licensed ebooks or a dedicated collection of print textbooks. with this lack of bandwidth and available resources, it is perhaps no surprise that the case studies also mention a general lack of awareness and understanding of the concept of oer and its benefits. without clear examples to point to or colleagues to learn from, the concept may seem less relevant to health sciences faculty. conversely, vcu mentions beginning to gain traction through faculty advocates who previously worked with their initiative, which implies that once inroads have been made they quickly prove fruitful. unique health sciences concerns through these case studies, some health science specific oer concerns come to light. for example, vcu encountered a misalignment with grant programs when faculty wished to create courses rather than materials. this issue may be linked to health sciences schools’ pre-professional focus and connection to hospitals, which can include post-doctoral instruction for residents and fellows. in other words, their teaching expands beyond credit-bearing courses to continuing education for practitioners. while advocates of open education promote its applicability to lifelong learning, this application is often a positive side effect rather than a focus of many institutional oer programs. additionally, many health sciences curriculums focus on preparing students for their qualifying exams (e.g. nclex for nursing, pance for physicians assistants, etc.), which a potential practitioner must pass before they can legally practice. similar requirements also exist in a few other disciplines, such as teacher licensure exams. wvu’s case study shared the common perception by their faculty that incorporating oer into the curriculum will disadvantage students on their qualifying exams, and this concern is echoed across the authors’ institutions. the exams and their preparatory materials are often copyrighted, making it difficult to generate oer to prepare students for this proprietary exam. to our knowledge, this concern has not been discussed, and the authors suggest it as a good area for further research. takeaways the uptake of open education by health sciences faculty is often slower due to competing demands for their time and limited relevant oer. although these barriers are not unique, they seem to be more entrenched in health sciences than in other disciplines. despite these hurdles, an increasing number of health sciences faculty are interested in incorporating oer into their instruction. based on our case studies, the authors 11 peer reviewed article hypothesis , vol. 35, no. 2, 2023 recommend supporting oer in the health sciences by: • growing library support. already a trusted support system, librarians can build on existing connections and expertise to educate about and advocate for faculty adoption of open educational practices. • identifying avenues for institutional support to strengthen the impact of oer initiatives. • establishing relationships with health sciences faculty. they can become strong advocates for oer, the library, and the librarian as an essential component of student education, and peer-to-peer advocacy can often have a stronger impact than librarian advocacy alone. by continued outreach, advocacy, and support for oer, librarians and other oer advocates can make inroads for textbook affordability on health sciences campuses. additional research into oer in the health sciences is needed, including exploring ways to better support opening up proprietary yet required qualifying exams. the authors hope our experiences provide a foundation for how librarians can serve as advocates for open practices and grow support for oer in the health sciences. credit statement jessica kirschner: conceptualization, methodology, writing original draft, writing review & editing, visualization jenn monnin: conceptualization, methodology, writing original draft, writing review & editing christine andresen: writing original draft, writing review & editing. references 1. perry, m. chart of the day. . . or century? [internet]. american enterprise institute. [cited 2023 may 24]. available from https://www.aei.org/carpe-diem/chart-of-the-day-or-century-6/ 2. nagle c, vitez k. fixing the broken textbook market, third edition [internet]. student pirgs; 2021 feb [cited 2023 may 25]. available from: https://studentpirgs.org/2021/02/24/fixing-the-broken-textbook-market-third-edition/ 3. viva. virginia course materials survey [internet]. [cited 2023 may 25]. available from: https://vivalib.org/va/open/survey 4. florida dlss. student textbook and course materials survey and information [internet]. [cited 2023 may 25]. available from: https://dlss.flvc.org/colleges-and-universities/research/textbooks 5. unesco. open educational resources [internet]. [cited 2023 may 24]. available from: https://www.unesco.org/en/open-educational-resources 6. openstax — free textbooks online with no catch [internet]. [cited 2023 may 25]. available from: https://openstax.org/ 7. mays ee. a guide to making open textbooks with students [internet]. the rebus community for open textbook creation; 2017 [cited 2023 jul 25]. available from: https://press.rebus.community/makingopentextbookswithstudents/ 8. mendes tb, dawson j, evenstein sigalov s, kleiman n, hird k, terenius o, et al. wikipedia in health professional schools: from an opponent to an ally. med sci 12 https://www.aei.org/carpe-diem/chart-of-the-day-or-century-6/ https://studentpirgs.org/2021/02/24/fixing-the-broken-textbook-market-third-edition/ https://vivalib.org/va/open/survey https://dlss.flvc.org/colleges-and-universities/research/textbooks https://www.unesco.org/en/open-educational-resources https://openstax.org/ https://press.rebus.community/makingopentextbookswithstudents/ peer reviewed article hypothesis , vol. 35, no. 2, 2023 educ. 2021 sep 29;31(6):2209–16. available from: https://doi.org/10.1007/s40670-021-01408-6 9. hilton j. open educational resources and college textbook choices: a review of research on efficacy and perceptions. educ technol res dev. 2016 aug;64(4):573–90. available from: https://doi.org/10.1007/s11423-016-9434-9 10. hilton j. open educational resources, student efficacy, and user perceptions: a synthesis of research published between 2015 and 2018. educ technol res dev. 2020 jun;68(3):853–76. available from: https://doi.org/10.1007/s11423-019-09700-4 11. walz a, jensen k, salem, jr. j. spec kit 351: affordable course content and open educational resources (july 2016) [internet]. association of research libraries; 2016 [cited 2023 may 24]. (association of research libraries, george la, editors. spec kit). available from: http://publications.arl.org/affordable-course-contentopen-educational-resources-spec-kit-351/ 12. reed jb, jahre b. reviewing the current state of library support for open educational resources. collect manag. 2019 jul 3;44(2–4):232–43. available from: https://doi.org/10.1080/01462679.2019.1588181 13. todorinova l, wilkinson zt. incentivizing faculty for open educational resources (oer) adoption and open textbook authoring. j acad librariansh. 2020 nov;46(6):102220. available from: https://doi.org/10.1016/j.acalib.2020.102220 14. seaman j, seaman j. opening the textbook: educational resources in u.s. higher education, 2017 [internet]. babson survey research group; 2017 [cited 2023 may 24]. available from: https://www.bayviewanalytics.com/reports/openingthetextbook2017.pdf 15. seaman j, seaman j. coming back together: educational resources in u.s. k-12 education, 2022 [internet]. bay view analytics; 2022 [cited 2023 may 24]. available from: https://www.bayviewanalytics.com/reports/k-12 oer comingbacktogether.pdf 16. kimball r, halling d, neville b, herbert b. motivations and barriers in the adoption of oers: the role of subject librarians. j acad librariansh. 2022 jul;48(4):102542. available from: https://doi.org/10.1016/j.acalib.2022.102542 17. belikov om, bodily r. incentives and barriers to oer adoption: a qualitative analysis of faculty perceptions. open prax. 2016 jul 1;8(3):235. available from: https://doi.org/10.5944/openpraxis.8.3.308 18. spilovoy t, seaman j, ralph n. the impact of oer initiatives on faculty selection of classroom materials [internet]. bay view analytics; 2020 [cited 2023 may 24]. available from: https://www.bayviewanalytics.com/reports/impactofoerinitiatives.pdf 19. open education group [internet]. [cited 2023 may 24]. available from: https://openedgroup.org/ 20. free open access medical education for students [internet]. 2016 [cited 2023 may 24]. available from: https://foamedstudent.com/ 21. burkholder t, bellows j, king r. free open access medical education (foam) in emergency medicine: the global distribution of users in 2016. west j emerg med. 2018 apr 30;19(3):600–5. available from: https://doi.org/10.5811/westjem.2018.3.36825 22. ting dk, boreskie p, luckett-gatopoulos s, gysel l, lanktree mb, chan tm. quality appraisal and assurance techniques for free open access medical education (foam) resources: a rapid review. in: seminars in nephrology. elsevier; 2020. p. 309–19. available from: https://doi.org/10.1016/j.semnephrol.2020.04.011 13 https://doi.org/10.1007/s40670-021-01408-6 https://doi.org/10.1007/s11423-016-9434-9 https://doi.org/10.1007/s11423-019-09700-4 http://publications.arl.org/affordable-course-content-open-educational-resources-spec-kit-351/ http://publications.arl.org/affordable-course-content-open-educational-resources-spec-kit-351/ https://doi.org/10.1080/01462679.2019.1588181 https://doi.org/10.1016/j.acalib.2020.102220 https://www.bayviewanalytics.com/reports/openingthetextbook2017.pdf https://www.bayviewanalytics.com/reports/k-12_oer_comingbacktogether.pdf https://doi.org/10.1016/j.acalib.2022.102542 https://doi.org/10.5944/openpraxis.8.3.308 https://www.bayviewanalytics.com/reports/impactofoerinitiatives.pdf https://openedgroup.org/ https://foamedstudent.com/ https://doi.org/10.5811/westjem.2018.3.36825 https://doi.org/10.1016/j.semnephrol.2020.04.011 peer reviewed article hypothesis , vol. 35, no. 2, 2023 23. bridgeman m. the rutgers university libraries open and affordable textbook (oat) program. med ref serv q. 2021 jul 3;40(3):292–302. available from: https://doi.org/10.1080/02763869.2021.1945864 24. national center for education statistics. trends in student loan debt for graduate school completers [internet]. 2018 [cited 2023 may 25]. available from: https://nces.ed.gov/programs/coe/indicator/tub/graduate-student-loan-debt 25. kruger k, parnell a, wesaw a. landscape analysis of emergency aid programs [internet]. national association of student personnel administrators; 2016 [cited 2023 jul 24]. available from: https://www.naspa.org/images/uploads/main/emergency aid report.pdf 26. vcu irds. fall 2021 facts & figures highlights [internet]. [cited 2023 may 24]. available from: https://irds.vcu.edu/facts-and-figures/ 27. vcu irds. vcu data digest [internet]. vcu data portal. [cited 2023 may 24]. available from: https://data.vcu.edu/datadigest/ 28. wvu facts [internet]. west virginia university. [cited 2022 oct 14]. available from: https://www.wvu.edu/about-wvu/wvu-facts 29. morgantown student body diversity [internet]. consumer information — wvu. [cited 2023 may 24]. available from: https://consumerinformation.wvu.edu/wvu-morgantown-student-body-diversity 30. 2022 musc fact card [internet]. [cited 2023 may 24]. available from: https://education.musc.edu/students/enrollment/institutional-reports/fact-card/2022 31. schev financial aid reports [internet]. state council of higher education for virginia. [cited 2023 may 24]. available from: https://research.schev.edu/info/reports.guide-to-the-financial-aid-reports 32. vcu libraries. open and affordable course content initiative [internet]. [cited 2023 may 24]. available from: https://www.library.vcu.edu/research-teaching/open-and-affordable-course-content/ 33. vcu libraries. affordable course content awards [internet]. [cited 2023 may 25]. available from: https://guides.library.vcu.edu/acca 34. vcu open and affordable course content initiative pressbooks catalog [internet]. [cited 2023 jul 25]. available from: https://pressbooks.rampages.us/catalog/vculoacc 35. bigbee j, pakurar a. digital histology [internet]. [cited 2023 may 24]. available from: https://digitalhistology.org/ 36. shapiro s, koppu s, masood m, rubinstein b, syed h. dermatologic manifestations of rheumatic disease: cutaneous manifestations of sarcoidosis [internet]. 2022. available from: https://scholarscompass.vcu.edu/oacc/12 37. kang j, nunley j, ho t, masood m, nutan f, rubinstein b. dermatologic manifestations of rheumatic disease: cutaneous manifestations of vasculitides [internet]. 2023. available from: https://scholarscompass.vcu.edu/oacc/16 38. shan d, kang j, cinats a, nunley j, rubinstein b. dermatologic manifestations of rheumatic disease: malignant cutaneous tumors [internet]. 2023. available from: https://scholarscompass.vcu.edu/oacc/18 39. libraries — goal 2 [internet]. [cited 2022 oct 14]. available from: https://library.wvu.edu/about/strategic-roadmap/goal-2 40. libraries — oer grant application [internet]. [cited 2022 oct 14]. available from: https://library.wvu.edu/initiatives/oer/grant 41. libraries — 2022 open textbook workshop invitation [internet]. [cited 2022 oct 14]. available from: https://library.wvu.edu/initiatives/initiatives/oer/2022-open14 https://doi.org/10.1080/02763869.2021.1945864 https://nces.ed.gov/programs/coe/indicator/tub/graduate-student-loan-debt https://www.naspa.org/images/uploads/main/emergency_aid_report.pdf https://irds.vcu.edu/facts-and-figures/ https://data.vcu.edu/datadigest/ https://www.wvu.edu/about-wvu/wvu-facts https://consumerinformation.wvu.edu/wvu-morgantown-student-body-diversity https://education.musc.edu/students/enrollment/institutional-reports/fact-card/2022 https://research.schev.edu/info/reports.guide-to-the-financial-aid-reports https://www.library.vcu.edu/research-teaching/open-and-affordable-course-content/ https://guides.library.vcu.edu/acca https://pressbooks.rampages.us/catalog/vculoacc https://digitalhistology.org/ https://scholarscompass.vcu.edu/oacc/12 https://scholarscompass.vcu.edu/oacc/16 https://scholarscompass.vcu.edu/oacc/18 https://library.wvu.edu/about/strategic-roadmap/goal-2 https://library.wvu.edu/initiatives/oer/grant https://library.wvu.edu/initiatives/initiatives/oer/2022-open-textbook-workshop-invitation https://library.wvu.edu/initiatives/initiatives/oer/2022-open-textbook-workshop-invitation peer reviewed article hypothesis , vol. 35, no. 2, 2023 textbook-workshop-invitation 42. libraries — shining minds textbook program [internet]. [cited 2022 oct 14]. available from: https://library.wvu.edu/initiatives/oer/initiatives/shining-minds-textbook-program 43. exploring open education workshop collection [internet]. the research repository @ wvu. [cited 2023 jul 11]. available from: https://researchrepository.wvu.edu/exploring-open-ed/ 44. hb 2853 text [internet]. [cited 2022 oct 14]. available from: https://www.wvlegislature.gov/bill status/bills text.cfm?billdoc=hb2853%20intr.htm &yr=2019&sesstype=rs&i=2853 45. lambert sr. changing our (dis) course: a distinctive social justice aligned definition of open education. j learn dev. 2018;5(3):225–44. available from: https://doi.org/10.56059/jl4d.v5i3.290 46. open education network. certificate in oer librarianship [internet]. [cited 2023 may 24]. available from: https://open.umn.edu/oen/oercert 47. medical university of south carolina. education innovation advocate [internet]. [cited 2023 may 24]. available from: https://web.musc.edu/innovation/get-involved/education-innovation-advocate 48. pascal. scale affordable learning grants [internet]. [cited 2023 may 24]. available from: https://pascalsc.libguides.com/c.php?g=1004681&p=7277900 49. 2023 nih data management and sharing policy — nih office of intramural research [internet]. [cited 2023 may 31]. available from: https://oir.nih.gov/sourcebook/intramural-program-oversight/intramural-datasharing/2023-nih-data-management-sharing-policy 50. white house office of science and technology policy (ostp). desirable characteristics of data repositories for federally funded research [internet]. executive office of the president of the united states; 2022 may [cited 2023 may 31]. available from: https://repository.si.edu/handle/10088/113528 15 https://library.wvu.edu/initiatives/initiatives/oer/2022-open-textbook-workshop-invitation https://library.wvu.edu/initiatives/initiatives/oer/2022-open-textbook-workshop-invitation https://library.wvu.edu/initiatives/initiatives/oer/2022-open-textbook-workshop-invitation https://library.wvu.edu/initiatives/oer/initiatives/shining-minds-textbook-program https://researchrepository.wvu.edu/exploring-open-ed/ https://www.wvlegislature.gov/bill_status/bills_text.cfm?billdoc=hb2853%20intr.htm&yr=2019&sesstype=rs&i=2853 https://www.wvlegislature.gov/bill_status/bills_text.cfm?billdoc=hb2853%20intr.htm&yr=2019&sesstype=rs&i=2853 https://doi.org/10.56059/jl4d.v5i3.290 https://open.umn.edu/oen/oercert https://web.musc.edu/innovation/get-involved/education-innovation-advocate https://pascalsc.libguides.com/c.php?g=1004681&p=7277900 https://oir.nih.gov/sourcebook/intramural-program-oversight/intramural-data-sharing/2023-nih-data-management-sharing-policy https://oir.nih.gov/sourcebook/intramural-program-oversight/intramural-data-sharing/2023-nih-data-management-sharing-policy https://repository.si.edu/handle/10088/113528 peer reviewed article hypothesis, vol. 37, no. 2, 2025 an examination of health information professionals’ discourse surrounding knowledge synthesis: a content analysis christine j. neilson, mlisa ahealth sciences librarian, neil john maclean health sciences library, university of manitoba, winnipeg, mb canada, https://www.orcid.org/0000-0002-0525-737x , mailto:christine.neilson@umanitoba.ca cite as: neilson cj. an examination of health information professionals’ discourse surrounding knowledge synthesis: a content analysis. hypothesis. 2025;37(2). doi:10.18060/28782 neilson. all works in hypothesis are licensed under a cc by-nc 4.0 deed attribution-noncommercial 4.0 international. authors own copyright of their articles appearing in hypothesis. readers may copy articles without permission of the copyright owner(s), as long as the author(s) are acknowledged in the copy, and the copy is used for educational, not-for-profit purposes. for any other use of articles, please contact the copyright owner(s). 1 https://www.orcid.org/0000-0002-0525-737x https://orcid.org/ mailto:mailto:christine.neilson@umanitoba.ca https://creativecommons.org/licenses/by-nc/4.0/ https://creativecommons.org/licenses/by-nc/4.0/ peer reviewed article hypothesis, vol. 37, no. 2, 2025 abstract introduction: knowledge synthesis (ks) is an umbrella term that encompasses a family of research methods that aim to draw insights from existing bodies of research literature through established processes of review and analysis. providing support services for ks is part of many health libraries’ day-to-day business and has been for several years. methods: this article presents the results of a content analysis conducted to gain insight into our collective relationship with ks using journal articles and conference abstracts associated with the canadian health libraries association and the medical library association. the study is framed in terms of three broad questions: what do health information professionals talk about when they discuss knowledge synthesis? how do they talk about it? and who is doing the talking? descriptive codes and attribute codes were applied to the texts. descriptive codes were grouped and regrouped to create sub-themes and overall themes in a bottom-up fashion. results: three broad themes are evident in the texts: the case for ks work, everyday realities of ks, and pushing back. ks discourse in the venues examined is currently dominated by the voices of health information professionals working in academic libraries, though this was not always the case. commentary: i suggest that our collective relationship with ks work has been changing, and health information professionals are starting to become more comfortable with setting boundaries around ks work.i also suggest that an apparent shift in voice towards academic health information professionals and a general increase in the ks content included in these venues over time could be attributed to 1) the widespread adoption of evidence-based practice among our clientele, and 2) increased emphasis on research impact assessment. introduction this article reports on an examination of texts (journal articles and conference abstracts) from the canadian health libraries association (chla) and the medical library association (mla) to explore the discourse surrounding knowledge synthesis among health information professionals. for the purposes of this study, “health information professionals” (hips) include various professional titles and roles such as librarians, information specialists, informationists, and library technicians. “knowledge synthesis” (ks) is an umbrella term that encompasses a family of research methods that aim to draw insights from existing bodies of research literature through established processes of review and analysis. it is also sometimes referred to as research synthesis or evidence synthesis. among hips, the best known examples of ks are arguably narrative reviews, systematic reviews, and scoping reviews, but the list of review types is extensive and keeps growing as new review types are proposed, developed, and catalogued1. interest in ks and related library services for science and humanities disciplines has increased among information professionals relatively recently2-5, but hips have been aware of ks in the health disciplines for decades and many hips’ daily work is impacted by ks to some degree. many studies have examined library conference programs and journals to provide an overarching interpretation of their content. according to myers, “knowing what librarians 2 peer reviewed article hypothesis, vol. 37, no. 2, 2025 talk about [. . . ] is important because what we talk about is what we are doing, and what we are doing is creating the future work environments, services, and relationships that will shape and define libraries”6. but these studies can also highlight the characteristics of authors, the research methods they employed, the breadth of topics addressed through specific venues, the treatment of a particular topic in specific venues, or themes that provide insight into the experience of a particular population7-12. such studies help us to take a big-picture view of our profession and what is happening within it. once we take a step back, we are prompted to consider the assumptions and beliefs that make up our mental models of day-to-day practice and our profession, and how those models compare to the picture that is presented. the objective of this study was to gain insight into our collective relationship with ks by asking: 1. what do health information professionals talk about when they discuss knowledge synthesis? 2. how do they talk about it? and 3. who is doing the talking? this is a text-based study that examines publicly available journal articles and grey literature; as such, institutional research ethics board approval was not required. researcher position and background i am a canadian health librarian who, at the time of writing this manuscript, has been a practicing librarian for approximately 20 years and involved in ks work for approximately half that time. i identify most with a pragmatic/constructivist research paradigm13,14. even though i currently work in an academic library context, i have previously worked in a hospital library and special health library environments. as part of the broad health library community, and the ks community specifically, i came to this project with an insider perspective, and i have made an effort to be conscious of my own biases throughout this study. methods this study was guided by the methods for thematic coding and analysis described by saldaña15 and braun and clark16. thematic analysis is an iterative process of theming the literature through a bottom-up manner of identification and refinement. it involves becoming familiar with the dataset; coding and recoding individual texts; identifying potential themes; and reviewing, revising, and naming the identified themes and sub themes. the constructivist paradigm is compatible with thematic analysis16,17 because theming goes beyond merely labeling data, as researchers “reframe, reinterpret, and/or connect elements of the data” to construct meaning17. while a lengthy process, this method offers both structure and flexibility for exploring and interpreting the collected texts. as a truly exploratory study, no theoretical framework was used to steer the analysis in a particular way. study procedures were tested using articles gathered through handsearching journal of the canadian health libraries association and coding the collected texts in nvivo 14 software18. emic coding–where the initial generation of descriptive codes is based on the 3 peer reviewed article hypothesis, vol. 37, no. 2, 2025 content of the texts rather than a predefined set of codes based on what the researcher expects to find15–was used for the test. procedures were modified as needed, and the test sample was later re-coded as part of the larger dataset using the methods described below. individual pieces of text were assigned multiple codes when appropriate. the unit of analysis in this study is the whole text; the frequency of a code within individual texts is not considered because of the substantial difference in word counts between conference abstracts and full articles. becoming familiar with the data through screening individual texts (i.e. articles and conference abstracts) were identified through handsearching i) tables of contents for the journal of the canadian health libraries association (jchla) and the journal of the medical library association (jmla), and ii) conference programs/lists of abstracts for chla and mla annual conferences that are posted on the respective association websites. handsearching started with material from 2023 and moved on to earlier years, stopping when either 2 full years of texts were published in that venue that did not meet inclusion criteria, or when all of the material available on the source websites was examined. chla is a national organisation that accepts conference submissions and journal articles in either english or french; conference abstracts available only in french were translated using the free deepl online translator19. titles and abstracts were read in their entirety; for potentially relevant articles, the full text of the article was read as well. eligibility for inclusion in analysis was based on the following criteria: inclusion: • the text is explicitly framed in terms of ks. this includes texts that discuss multiple topics, such as ks services in addition to other research support services. it also includes texts that mention how the topic the authors focus on relates to ks. for example, articles reporting on the development of a search filter that mention the potential usefulness of the filter for ks work would be included, while articles that report on the development of a search filter without mentioning ks would be excluded. • where author supplied keywords were available for conference texts, the appearance of relevant keywords made them eligible. • where specific research products were discussed, review types listed in sutton et al.’s 2019 article “meeting the review family: exploring review types and associated information retrieval requirements20” were considered to be ks. exclusion: • texts that are not framed in terms of ks. • texts that report on a ks project, rather than discuss ks as a topic. • texts that discuss how or where to locate published ks articles (for example, search filters for identifying systematic reviews). • conference texts without an abstract (i.e.: only a title and list of authors is supplied). • conference keynotes. 4 peer reviewed article hypothesis, vol. 37, no. 2, 2025 • journal texts that fall into the following categories: book reviews, product reviews, messages from the journal editor, journal announcements, notices in memoriam. • chla conference abstracts that were published in jchla. – the conference programs posted on the chla website21 were considered the authoritative source for chla conference material. however published abstracts from the chla 2022 conference were used to supplement incomplete program information that year, after eligible texts were identified in the program. the dataset does not include chla conference abstracts from 2020 or 2021. sufficient program information was not available for 2021, and the 2020 chla conference was cancelled due to covid-19 pandemic lockdowns. initial coding texts were reread and assigned two types of codes using nvivo 14 software: descriptive codes that describe the texts’ content15, and attribute codes that capture texts’ characteristics. a provisional descriptive code book with codes addressing the first research question “what do health information professionals talk about when they discuss knowledge synthesis?” was developed when testing study procedures, as described above; this codebook was used and adapted as needed to code the complete data set. provisional codes were grouped and regrouped, folding related codes into broader codes. for example, the codes bibliographic databases, software tools, search filters, and other tools were collapsed into the single code tools. coding was reviewed before moving on to the theming stage. the complete list of finalized codes for the first research question, their definitions, the number of texts where the codes were applied, and examples of passages where the codes were applied is available as a supplemental file (see data availability statement). a separate set of emic codes and annotations were used in nvivo to note the occurrence of terms and observations relating to the second research question “how do they talk about it?” a list of the emic codes used is also available. attribute coding was used to gather information about the characteristics of the individual texts, as detailed in table 1. the two codes that address the research question “who is doing the talking?” were author workplace affiliations present and geographic areas represented by author affiliations. data collection for authors’ listed affiliations was based on the presence or absence of a given category rather than counts of affiliations for all authors of a work. more than one category was applied when the list of authors represented multiple affiliation types. when the type of organisation listed was unclear from the text, i consulted the organisation’s website to determine the affiliation category. libraries serving organisations that primarily provide patient care were classified as hospital libraries. libraries that serve organisations that primarily provide health education were classified as academic libraries. affiliations to individual university departments, specific research units, or other traditional special library environments were classified as special libraries. where job titles and affiliations listed in a text did not indicate the educational background of an individual – for example, “research associate” or “director” – the author was assumed to be a hip because hips are the primary community associated with the communication venues examined. when the listed author titles or credentials indicated such status as students who are not also library staff, library 5 peer reviewed article hypothesis, vol. 37, no. 2, 2025 school faculty, non-library faculty (ex: nursing faculty), clinician, or software engineer, the author was designated a “non-library worker”. geographic areas represented by author affiliations were broken into three categories: canada, united states, and another country. there are no geographic restrictions on who may participate in the venues examined here, but since chla and mla are national associations, canada and the united states were considered separately from other countries. table 1. attribute coding categories and associated codes category codes data entry mechanism text source chla conference, mla conference, chla-mla joint conference, jchla or jmla dropdown selection year year of publication or year the conference was held free text type of text research, program/case description, commentary, other dropdown selection text format article, oral presentation, poster presentation, lighting talk dropdown selection number of authors number of authors listed free text author workplace affiliations present hospital library, academic library, special library, public or school library, non-library worker author, affiliation not included for 1 or more authors dropdown selection geographic areas represented by author affiliations canada, united states (usa), another country dropdown selection theming the data codes, much like subject headings, are tags that note the presence of specific topics or ideas. themes represent broader constructs. the process of theming is a sense-making exercise that involves sorting and resorting the final set of codes to explore the relationships between them. these relationships were interpreted and integrated into a set of broad themes that relate to the research question, “what do health information professionals talk about when they discuss knowledge synthesis?” quantitative data collected through attribute coding and tallying codes and annotations related to research question two, “how do they talk about it?”, as mentioned above, inform the overall themes as appropriate. limitations the reader should consider three study limitations when reading the findings that follow. ambiguity in authors’ writing some conference abstracts in the study would have been excluded if not for the inclusion of relevant author-supplied keywords (usually “systematic reviews”). these keywords indicate that the authors’ own understanding of their work 6 peer reviewed article hypothesis, vol. 37, no. 2, 2025 frames it in terms of ks, despite not making that connection for the reader in their title or abstract. in years where conference abstracts were not accompanied by keywords, abstracts that fall into this category may have been excluded from this study. likewise, the terminology that authors use can be unpredictable and different authors used the same terms in different ways. an example is the term literature review; many authors seemed to use the term to refer to narrative reviews, some authors clearly used it to refer to systematic reviews22, and others indicated that they were using it as an umbrella term for all ks reviews23. the terminology hips use has also changed over time. this is beautifully illustrated by two conference abstracts that appear to describe the same service. fang’s 2006 abstract refers to “evidence based reports” that are produced using a process that sounds like ks procedures24. later in 2013 fang and colleagues did not detail the process used, but name the end product as “systematic reviews” instead of evidence based reports25. during handsearching, some relevant texts may have been excluded because the language used by individual authors did not obviously suggest that they were referring to ks, but since this project is not a systematic review, capturing all texts relating to ks is not essential. incomplete information in conference abstracts the level of detail provided in conference abstracts varied from abstract to abstract, year to year, and conference to conference. some information shared verbally during a presentation may not have been included in the associated abstract because of word count restrictions. similarly, there were 27 conference texts included in the study that described partial study results in the abstract, and 76 that provided no results at all. some authors indicated that results were not available at the time the abstract was submitted, some indicated that results would be discussed during the presentation, and in some cases results were omitted without explanation. as a result, coding may not have completely captured presentation content. in terms of author information, some texts did not list affiliations for all of the authors of a given abstract. the ways that author affiliations are presented in the texts could also have had an impact on attribute coding, since some authors may have multiple affiliations to choose from and the level of affiliation reported (for example affiliation with a specific university library vs. the university itself) could have varied between authors. quality of translation the quality of automated translation for french material is not as good as professional translation, but as an unfunded study it was not feasible to obtain professional translation for the french texts that were screened. there were some structural issues with translated texts but the message came across well enough for the purposes of this project, and working with an imperfect text was preferable to discounting french items simply because of language. 7 peer reviewed article hypothesis, vol. 37, no. 2, 2025 findings sample characteristics a total of 537 texts were included in the analysis. 274 (51%) were classified as “research”, 203 (38%) were classified as “program descriptions”, 21 (4%) were “commentary”, and 39 (7%) were classified as “other”. most of the included texts (n=284, 53%) came from the mla annual conference. when comparing the american and canadian venues, both the mla conference and jmla were the source of more texts about ks than their chla counterparts, but the chla venues had more ks texts as a percentage of the overall number of texts in the individual conferences/journal issues than the mla venues (see figure 1 for the distribution of texts over time). figure 1. sources of sample texts over time. note that chla and mla held a joint conference in 2016. the 2021 chla conference was excluded because of insufficient information in the posted program. the 2020 chla conference was cancelled due to the covid-19 pandemic lockdowns. a mix of author affiliation types–hospital library, special library, academic library, and non-library workers–were present throughout the time period examined, but figure 2 suggests that around 2013, or shortly after, the voice of academic librarians became most prominent in discussions of ks. 8 peer reviewed article hypothesis, vol. 37, no. 2, 2025 figure 2. counts of author affiliation types present in the sample by year. the venn diagrams in figures 3 and 4 show that in some cases authors from different affiliation types or countries worked together to produce texts, but for the most part authors from different affiliation types or different countries did not mix. the number of texts that only listed authors with academic library affiliations was more than double the number of texts where authors with academic library affiliations worked with other affiliations (242 compared to a combined 104 texts). in contrast, 50 texts only listed authors with a hospital library affiliation compared to 56 texts where authors with a hospital library affiliation worked with individuals with another affiliation, and 79 texts only listed authors with special library affiliations compared to 69 texts where authors with a special library affiliation worked with individuals with another type of affiliation. 9 peer reviewed article hypothesis, vol. 37, no. 2, 2025 figure 3. author geographic affiliation types present in sample texts. an additional five texts did not include geographic information for any authors (not represented here). two notable language-related features of the texts emerged during analysis: the terms used for ks (either generally or relating to specific forms of ks) and the use of language that implies a power relationship when discussing hips’ ks work. 10 peer reviewed article hypothesis, vol. 37, no. 2, 2025 figure 4. author affiliation types present in the sample. note: due to the small number of texts with authors with school and public library affiliations, these texts have been folded into the categories based on the other author affiliation types present. these categories have been marked with an asterix (*). an additional six texts did not include affiliations for any authors (not represented here). there were 102 terms used to refer to ks within the sample. 460 texts (86%) used the specific term “systematic review” (sr), and 17 texts (3%) used other terms that incorporated the word “systematic”, such as “systematic literature review”, “systematic overview”, “systematic scoping review”, and “systematic-like reviews”. ten texts made brief mention of other types of reviews but defaulted to focusing on srs, compared to three texts that made brief mention of srs but went on to focus on other types of reviews. srs were also given prominence when ks was generally discussed, with phrasing like “evidence syntheses such as systematic reviews” (n=3 texts) and “systematic and other reviews” (n=34 texts). three texts explicitly stated that the term “systematic review” was being used to refer to all varieties of ks. other specific methodologies that were mentioned relatively frequently were “meta-analyses” (n=71 texts) and “scoping reviews” (n=71 texts), with an additional three terms (n=4 texts) that incorporated the word “scoping”. in cases where authors referred to ks broadly, the two most common terms used were “evidence synthesis” (n=58 texts) and “knowledge synthesis” (n=42 texts). a strong link between ks and evidence is apparent in the ks nomenclature used in the texts, as there were 18 terms used for ks in addition to “evidence syntheses” that incorporated the word “evidence”. examples include “evidence based syntheses”, “evidence based products” and “evidence based reviews.” 11 peer reviewed article hypothesis, vol. 37, no. 2, 2025 the language that the authors used to describe their ks work can imply power relationships between hips and clients, whether used consciously or unconsciously. at least one instance of power-related language was observed in 238 (44%) of the texts. three types of language were used in the sample: power-neutral language, language that places power with the client, and language that implies power is shared by hips and clients (see figure 5). some authors used language that falls into different power categories interchangeably within the same text. vague but power-neutral language was often used to describe hips’ ks work, such as “participating” in ks, “contributing” to ks, or being “involved” in ks. a similar number of texts used language that imply a secondary role for hips, where clients hold the power. examples of this kind of language are: clients “using” hips or “including” hips in ks; references to “serving” clients outside of descriptions of formal library services, such as “health sciences librarians now serve on systematic review teams”; and “supporting” clients that are undertaking ks projects, which was the most common language used in this category. finally, language that implies that hips and clients share power included “partnering” or “collaborating” with clients, and referring to the hip-client relationship as collegial. as figure 5 illustrates, all three categories of language were noted over time, with instances of power-neutral language and language that situates power with the client consistently occurring more often than instances of power-sharing language in texts from the latter half of the timeframe covered by the dataset. figure 5. counts of texts with power-related language by year of publication. themes three broad themes and four subthemes were identified in the sample. note that individual texts often contain content that is related to multiple codes and themes. individual codes can also be inter-related across themes and subthemes. for example, the code “hip workload” 12 peer reviewed article hypothesis, vol. 37, no. 2, 2025 affects and is affected by the code “library policies and processes for ks” and theme 3, pushing back. likewise, the code “burnout” is listed under matters that concerns individual hips; the occurrence of burnout in a given context may or may not lead to elements of pushing back, like “scaling back ks service”. theme 1: the case for ks work the case for ks work is persistent throughout the dataset, starting with the first occurrence of associated codes in 2001 (see figure 6 and table 2). this theme relays several messages to the reader. according to this theme ks is an important part of both the overall research landscape and evidence-informed practice for health professionals, as the highest level of evidence in the evidence pyramid. the importance of ks, and demand for ks support from hips’ clients, has been growing and only continues to grow. ks presents opportunities for hips to make a significant contribution to research, improve their credibility with clients, prove their worth to employers, and maybe even bring in extra funding to the library if researchers are charged for ks services. moreover, we are told that hips are important for ks work. authors frequently link hips to better quality end products, citing recommendations to work with a librarian from recognized ks methods guidance and studies indicating that ks projects with hip contributors are of better quality than those ks projects without a hip. language such as “important”, “valuable”, “significant”, “key”, “central”, “crucial”, “essential”, “vital”, and “pivotal” were used to describe hips’ role and expertise. some authors made more elaborate statements on the importance of hips to ks such as “the full scope of systematic reviews and related methodologies comprises a tapestry of interconnected skills through which librarians form the weft, being connected in various ways at different levels throughout the entire process, with new roles and tasks emerging for the profession to explore”26. this “full scope” refers to hips taking on a wide-ranging set of activities from screening to project management, as some authors see hips as a driving force for ks projects: “the involvement of an informationist can keep the project moving forward even when team members rotate off because of other duties”27. finally, this theme tells us that hips have an obligation to take on various facets of ks work. this includes: • educating others: “in our role as information professionals, it is our responsibility to actively clarify the indications for and differences between systematic reviews, “systematic-like” reviews, comprehensive literature reviews, and patient-specific precision inquiries as we proactively engage in training our users and colleagues alike”28. (emphasis added) • improving the quality of research publications: “librarians should use their unique knowledge and experience to ensure that reporting of search strategies in meta-analyses is improved”29. (emphasis added) • advancing ks methodologies: “due to the developing nature of scoping reviews, we also suggest how librarians could and should involve themselves in this emerging area of the knowledge synthesis landscape”30. (emphasis added) 13 peer reviewed article hypothesis, vol. 37, no. 2, 2025 • and simply furthering hips’ own best interests: “to be successful, librarians must embrace this increased involvement in supporting the production of higher levels of evidence”31 (emphasis added). figure 6. number of texts coded to themes over time table 2: codes mapped to theme 1: the case for ks work theme 1: the case for ks work -hips are important for ks -hips are uniquely positioned -hips [could] do more than search -ks on the rise -ks benefits hips -link between ks and ebp -link between ks and research funding theme 2: everyday realities of ks most of the descriptive codes used in this study fell into the theme everyday realities of ks, and this theme is persistent throughout the texts, starting in the year 2000 (see figure 6). many of the codes directly relate to everyday practice, and they were discussed in a combination of positive, neutral, and negative ways. this theme tells us about the challenges of ks work, how hips attempt to mitigate these challenges, and their successes through four subthemes: the world of ks, system-level matters, matters of daily practice, and matters concerning individual hips (see table 3: codes mapped to theme 2: everyday realities of ks). subtheme: the world of ks the world of ks focuses on aspects of ks methods, ks products, and the researchers doing ks work. this sub-theme tells us that while ks research may have been the domain of elite researchers in 200532, that did not remain the case. in some instances hips have actively 14 peer reviewed article hypothesis, vol. 37, no. 2, 2025 worked with faculty to promote ks and increase the number of students undertaking ks projects33. other times, ks has been embraced by students–or at least by their faculty mentors–without hip intervention. in 2018, wissinger noted that “in the last year, i have noticed a substantial increase in the number of graduate students who have been sent to me because their faculty advisor told them they need to do a systematic review. this was more of an oddity than a concern until undergraduate students started coming to me because they had received the same advice”34. in 2022 foster, sullton, and crews reported that 50% of 225 respondents to their faculty survey indicated that they expect graduate students to be able to conduct ks reviews35. the quality of ks publications and adherence to methods and reporting guidance appears to be of great interest to hips, and this theme tells us that many researchers – including academic faculty, health practitioners, and students alike do not truly understand what is required to do ks properly or have the skills to do it. some clients opt not to work with hips. 15 peer reviewed article hypothesis, vol. 37, no. 2, 2025 table 3: codes mapped to theme 2: everyday realities of ks theme 2: everyday realities of ks 2.1 the world of ks -ks 101 -ks reporting -ks standards & guidelines -quality of ks work/products -researcher resistance -researchers are ill-equipped for ks -researchers are not working with hips -student ks projects 2.2 system-level matters -decreased demand for ks service -external optics -fee-based service -ks is not a priority -ks services -library administration -library policies & processes for ks -no demand for ks support -resource limitations -service expansion/scalability -standardisation -sustainability of ks service -teamwork in providing -ks service -tiered services 2.3 matters of daily practice -communication with[in] the research team -consumer health -data management -hip involvement in journal peer review -journal editors/publishers -ks instruction for clients -languages other than english -negotiating with clients -predatory publications -retractions/corrections for publications -searching -tools 2.4 matters concerning individual hips -ability to support ks -feeling disrespected/undervalued -getting credit for ks work -interpersonal challenges -hip agency -hip skill development -hips’ return on investment -hips’ workload -restrictions on hip scope of practice -stress / burnout some push back against hips’ guidance, as noted by yang and colleagues: “the iss [information specialists] stated that clients would push to take shortcuts that would sacrifice the quality of a systematic review and at times go against review standards”36. in other cases, hips give researchers what they need to successfully engage in ks, as roth notes: “i have seen changes in the level of work performed by our graduate students, who have said that without this service, they would have never been able to do this type of research”37. subtheme: system-level matters the subtheme system-level matters includes matters that are larger than the individual hip, though individuals may contribute to, or be affected by, them. descriptions of formalized ks 16 peer reviewed article hypothesis, vol. 37, no. 2, 2025 services and library policies and procedures for ks work feature heavily in this theme, as authors share their attempts to scope ks work and organize services to manage the associated challenges. two codes that stand out from the others in this subtheme are “external optics” and “library administration”. the occurrence of “external optics” suggests that we are concerned about outsider views on hips, ks services, and the profile of hips and their libraries. the code “library administration” is somewhat related to external optics, as the texts in the dataset were often written by rank-and-file hips, rather than library leaders. library administration may have different views than the hips doing ks work38,39, and while they may not be directly involved in ks work, administrators’ views and decisions affect ks services. hips want administrators to be on their side; in addition to generic calls for “support”, hips see a role for administration in ensuring appropriate training38,40,41, finding ways to work ks into existing roles without burning hips out38,42,43, and giving hips permission to decline taking on ks projects when appropriate44,45. subtheme: matters of daily practice matters of daily practice are those that are most directly tied to the technical aspects of daily ks work. frequently used codes in this theme are “searching”, “tools”, and “instruction for clients”. subtheme: matters concerning individual hips this theme looks inwards, dealing with issues that impact individual hips involved in ks work, from practical matters like skill development to those related to job satisfaction and burnout. this subtheme tells us that those who undertake ks work can face significant challenges. the code “hip workload”–used for mentions of the volume of ks work, the amount of time it takes, references to work efficiency, or balancing ks work with other responsibilities–has been consistently present over the years. this code is distinct from, but ties to, the theme 1 code “ks on the rise”. for example, saragossi’s 2022 conference abstract “bibliometric analysis as a method to justify library personnel” speaks to the increasing volume of ks work overtime, and we can infer the impact of this increased output on hip workloads: “there has been a consistent increase in the output of evidence syntheses at stony brook university over time. the number has doubled since 2014. based on first quarter publications for 2022 it can be inferred that we will double last year’s output. additionally, from 2015-2021 the output has increased by 75% while library personnel has remained consistent at 5 full-time health sciences librarians46”. hips have also been discussing ks work in terms of return on investment, often characterizing a loss in terms of projects abandoned by the research team before completion, and the return as authorship or otherwise getting credit for their work in ks publications. seeing ks work as an investment, and recent mentions of hip personal agency–either in terms of hips’ desire for agency, or hips possessing the ability to choose what ks work they undertake and having control over both how the work is completed and the products of their work–may signal a shift in how hips view their role in ks. theme 3: pushing back 17 peer reviewed article hypothesis, vol. 37, no. 2, 2025 the codes belonging to theme 3: pushing back started appearing in 2013 and while the frequency of these codes is comparatively low, they are a stark contrast to theme 1 (see figure 6 and table 4). this theme suggests that hips’ relationship to ks and ks researchers is changing. hips perceive themselves as–or want to be–“[. . . ] coauthors and equal collaborators with faculty, not just as ‘free labor’ or at-will service professionals”47. rather than struggle to meet ks demand, some hips are being “more discerning about whom [they] provide the service to”48 or are otherwise scaling back on the services they offer. some hips have started imposing conditions on service to manage demand, fund positions, facilitate better quality end products, ensure hips receive authorship on publications, and give hips a mechanism to walk away from projects that turn out to be methodologically poor. examples of such conditions include: requiring completed protocols in advance; only working with teams that have a minimum number of team members; requiring researchers sign memoranda of understanding outlining terms of service; or charging fees for ks service. sometimes the results of introducing these conditions are unexpected. for example, lackey and colleagues found that demand for service went up, not down, after moving to a fee-based “core facility” system49. while authors discussed ways to deal with high levels of demand for ks service throughout the years examined, an explicit statement that hips’ needs take priority only emerged in deaver and colleagues’ 2022 mla conference abstract. they state, “rather than focus on what systematic review service users want, we chose to first consider what the librarians running the service need”50. the occurrence of evocative, sometimes almost antagonistic, language used in the texts also suggests that an undercurrent of dissatisfaction with elements of the status quo may be building. the conference abstracts entitled “sharpening the other side of the dual edged systematic review sword: expecting more from our users”51, “required, not suggested: creating a rigorous protocol template for improved uptake and efficacy”52, and campbell and dorgan’s account of ks instruction at their library, below, suggest that not everything is okay: these [classes] frequently fill up within a day or two of being posted, even though there is no formal advertising. this popularity is a double-edged sword. we know that offering the class has actually increased the demand for consultations, because participants inform us that they have made an appointment after having taken the class. one of our librarian colleagues described the courses as effectively “painting targets on our backs53” (emphasis added). table 4: codes mapped to theme 3: pushing back theme 3: pushing back -conditions on service -hips not interested in ks -putting hip needs first -scaling back ks service -there’s more to research support than srs -there’s more to searching than ks commentary this exploratory study was guided by three broad questions: what do health information professionals talk about when they discuss knowledge synthesis? how do they talk about it? 18 peer reviewed article hypothesis, vol. 37, no. 2, 2025 and who is doing the talking? the data gathered through these individual questions are interconnected, and most useful when considered together. health information professionals communicate their practice-driven innovations and research so that the entire hip community can learn and grow. therefore, it is unsurprising that most of the codes identified in this study fell into the theme everyday realities of ks. this is consistent with spencer and eldrege’s scoping review on the roles hips play in ks54. however, a key difference between spencer and eldrege’s results and those presented here is that their results are largely framed in an outwards-looking fashion, focusing on what hips do and their outputs. this study adds to the literature by looking inwards as well: noting matters that affect individual hips, asking whose voices are most prominent, and being mindful that these texts represent discussions that we have amongst ourselves, not with our clients. these additional elements that are less closely tied to everyday practice can prompt us to think more deeply about our collective relationship to ks. perhaps the most thought-provoking things to come from the results presented here are 1) the prominence of the case for ks work and the tensions between that theme and pushing back, 2) the shift in author voice that took place around 2013, and 3) how the results of this study relate to our professional identity. the case for ks work vs. pushing back after being immersed in the texts, the most striking thing to emerge was the prominence of the case for ks work. it is reasonable to expect promotion of the hip’s role in earlier texts, such as mcgowan and sampson’s 2005 article “systematic reviews need systematic searchers”32. but twenty years later we are constantly reminding each other – even if it is only by a line or two in a background section how important ks is to our clients and our everyday business, and how important hips are to ks. surprisingly, at times we are also told that we have an obligation to take on this work. the audience for these texts is fellow hips, not health researchers or other actors in the research world who may not be aware of our expertise, so why is this theme so prevalent? some hips expressed that they felt undervalued, prompting the code feeling disrespected / undervalued; does stressing the importance of ks work and hips’ role in ks stem from a widespread feeling of being underappreciated? or is it something else? theme 1 brings to mind certain aspects of ettarh’s 2018 essay on vocational awe and librarianship; namely that our work is important and it must be done for the good of our clientele and society more broadly, even though it might contribute to job creep and burnout55.the following quote from 2022 hints at an underlying mindset implied by theme 1: the aim of our research is to inspire medical librarians to become guardians of their organization’s systematic review output either by offering peer review of researchers developed searches, or by offering librarian-mediated searches. that [way] each sr project should be based on a high quality search from the start of the project56. (emphasis added) one way or another, some hips appear to have become the self-appointed guardians of ks integrity. this notion is reinforced by hannake’s commentary on the benefits of student ks projects. they point out that many hips have difficulty accepting ks as learning exercises, and feel obliged to talk students out of doing ks projects altogether57. ks methods are research methods like any other; would a hip be determined to talk a student out of doing an 19 peer reviewed article hypothesis, vol. 37, no. 2, 2025 interview study or a survey? perhaps the literature-based methods of ks make hips more invested in these works. but making ourselves responsible for the quality of all our clients’ research output is irrational and self-defeating when hips do not have direct control over the work itself. the benefits outlined by theme 1 are appealing: hips get an improved reputation; we can build relationships with clients and add publications to our cvs while making a worthwhile contribution to research; and in some cases, fees for ks work can generate revenue for the library. but the potential challenges discussed in theme 2, everyday realities of ks – such as workload issues, resource limitations, and problems working with researchers have led to theme 3 pushing back. with very few exceptions, the authors commenting on demand for ks service in the texts examined here indicate that the demand is significant. the addition of new roles and responsibilities with high client uptake cannot be sustained without adjusting existing workload and responsibilities. demetres and colleagues showed that hips working in positions where ks is the focus of their job had significantly lower burnout scores than hips who had ks tacked on to other duties43. this may be due to a more balanced workload than that experienced by hips filling multiple roles. high volumes of work may also require a superficial, transactional relationship with clients where hips do not feel that their work is valued, rather than meaningful collaborations. we do know that dedicated ks positions can come with hip-client relationships where hips share more equal status with researchers than they might otherwise enjoy49. a dedicated ks role could also signal that the work these hips are doing is highly valued by their institution. this could also impact job satisfaction. since hips have expressed a desire to be regarded as equals and have agency over their ks work, it is worth asking whether it could be a misstep to formalize ks services in situations where dedicated ks positions are not a possibility. some libraries have already moved to limiting the support they offer for ks, and others may wish to follow suit. a modest but noticeable jump in theme 3 codes in 2023 may indicate that a shift in how hips view ks, and their role in it, is just beginning. only time will tell if and how that shift plays out. the shift towards the academic voice up until 2012, the number of hospital library affiliations, special library affiliations, and academic library affiliations within the texts was not dramatically different. in contrast, myers’ examination of mla conferences from 2001-2019 found that 78% of total conference content was authored or co-authored by hips from academic libraries, and the amount authored or co-authored by hips from hospitals (19%) was roughly proportionate to the number of hospital librarians in the mla membership at the time6. since it is generally accepted that when it comes to conference participation and publication academic hips face fewer barriers, and more supports and incentives for participation, the relatively even representation among sectors from 2000 to 2012 suggests that most academic hips had not embraced ks to the same extent as special or hospital hips in the early 2000s. the year 2013seems to mark a shift in the author affiliation types represented in the collected texts, with academic voices starting to outnumber the voices of hips working in hospitals and special library environments. what could have prompted this shift? this study was not designed to uncover the underlying causes for what was observed. however, there are two potential factors whose influence in the years leading up to 2013 may have eventually led to an increase in both the number of texts framed in terms of ks in general, and academic hips’ voices becoming more prominent. 20 peer reviewed article hypothesis, vol. 37, no. 2, 2025 1. evidence based practice the first thing to consider is that while evidence-based practice (ebp) is currently considered a foundation of western health services and health education, it only started to gain momentum and be widely adopted in medicine and nursing in the 1990s and early 2000s58,59. ebp caught the attention of hips as well – likely in no small part because centering evidence in practice also elevates the importance of the source of that evidence, the library. funk conducted an analysis of word usage in the bulletin of the medical library association and journal of the medical library association in 2013. appendix b of their manuscript shows that use of the word “evidence” in the publications started to take off around 200060. today, academic hips are often involved in teaching aspects of ebp to health students. the hierarchy of evidence for clinical intervention questions places ks–specifically, systematic reviews, meta-analyses, and reviews of reviews–at the top of the evidence pyramid, as the “best” form of evidence to inform clinical decision making. with the increasing pervasiveness of the evidence hierarchy and ebp, it is easy to suppose that students and health practitioners might come to consider systematic reviews and meta-analyses the best form of research evidence, full stop. the texts examined here showed that, as a group, hips have a general tendency to talk about systematic reviews rather than use broad terms, like ks, or discuss other types of reviews. if we combine these two factors, we might begin to understand why some researchers do not understand the full range of ks options and want to do a systematic review for every type of research question, as has been noted by some hips. 2. research impact metrics and research funding decisions we must also consider the potential influence of research metrics. the early 2000’s was a busy time for those interested in research metrics, and bibliometrics specifically. the academic ranking of world universities (now the shanghai ranking) first came out in 200361 followed by the times higher education (the) world university rankings in 200462. the h-index was proposed as a new way of measuring authors’ scholarly impact in 200563, closely followed by several variant metrics64. the database scopus launched in 2004, and underwent various enhancements over the following years: they released author identifiers in 2006; affiliation identifiers that “[allow] users to see a holistic view of an institution’s research output” were released in 200765; the journal performance metrics source-normalized impact per paper (snip) and scimago journal rank were incorporated in 2010; and scopus integrated with orcid persistent identifiers in 201266. 2012 was also the year that the san francisco declaration on research assessment–now simply the declaration on research assessment, or dora–came out as a response to the explosion of readily available research metrics. it featured “recommendations for improving the way in which the quality of research output is evaluated”67. research metrics and performance-based research funding was even discussed in the bmj in 200868. research metrics and institutional rankings can offer universities, and arguably research hospitals, prestige that attracts highly skilled personnel, students, funding69 and philanthropic contributions. for individual researchers, demonstrating high levels of research impact demonstrates career success to those both inside and outside the organisation, which can open doors to the researcher. demonstrating research impact is most directly connected to ks by the citation advantage enjoyed by published ks articles. research published in 2003, 2004, and 2005 demonstrated that in the medical field, rigorous systematic reviews and 21 peer reviewed article hypothesis, vol. 37, no. 2, 2025 meta-analyses received significantly more citations than publications of other research designs70-72. if a researcher wanted to maximize the number of citations they receive to reap the benefits of increased notoriety and demonstrate effectiveness as a researcher in their grant applications, conducting ks projects would help boost their numbers. what’s more, in 2005 patsopoulos and colleagues noted that the superiority in citation impact of meta-analyses and secondarily rcts is consistent with the prominent role given to these designs by evidence-based medicine,1-4 despite the criticisms leveled against both designs.15,20 the further dissemination of hierarchies of evidence may further increase the citations for meta-analyses and rcts. if the proposal that each study should start and end when a meta-analysis is adopted,21 meta-analyses may become even more highly cited74. the latter part of this statement refers to a 2002 article in jama that criticized a general lack of published randomized trials that frame their results within the context of the evidence generated by other trials. at one point the authors of the jama article state “the public would be served better if systematic reviews were always available before new clinical studies began, which would reduce unwanted duplication of research and help ensure that new research had been designed to build on lessons from earlier research”73. this argument expands the relationship between ks and research funding from providing indications of researchers’ success through bibliometrics, to demonstrating to funders that the study being considered for a grant is needed and its funding justified. if no ks work has been done on a specific research question, this may prompt some researchers to take on a ks project as a preliminary step before conducting primary research. trainees who are mentored in such a way could in turn pass the practice on to those they later mentor, contributing to the ever-increasing volume of ks in general and student ks projects noted in the texts in this study. as researchers embraced ks, it was only a matter of time before academic hips began to publish and present on ks more frequently. our collective relationship with ks and professional identity hicks’ 2014 work on librarians’ professional identity found that the idea of service is fundamental to librarians’ professional identity, but there can be a tension between service vs. professional expertise being the defining factor that makes librarians relevant. if language that is power-neutral or that situates power with the client are linked to the idea of service, and language where power is shared is connected to the idea that knowledge and skills are what define hips’ professional identity, this could help explain both why individual texts included a mix of power-language categories and the relatively low use of language implying that power is shared. hicks also notes “not only was the service repertoire used to highlight how much clients needed the services librarians offered, but it was also used to highlight the perceived ignorance of clients regarding these invaluable and indispensable services”74. this statement can be linked to the occurrence of the codes hips are important for ks and researchers are not working with hips. finally, hicks proposes that “recognition was particularly important as librarians needed it to maintain their sense of professional value”74. this ties to hips’ expressed desire within the texts to be considered equals with clients, and mentions of receiving credit for their ks work. 22 peer reviewed article hypothesis, vol. 37, no. 2, 2025 areas for future research there are a variety of areas discussed here that could lend themselves to additional questions and further study. for example, do indications of pushing back continue in these communication venues–or others–over time? how do hips navigate the tension between wanting to contribute their expertise to ks and meet client demand for service, and the need to set boundaries around providing that service? how do service and expertise interact when it comes to hips professional identity? is collaboration between hips from different types of organisations and geographic areas really as rare as the results suggest, and if so, what factors promote these collaborations? readers may identify additional questions as well. conclusion as an exploratory study, this work prompts the reader to reflect on their own relationship with ks, and the relationship between information professionals and ks more broadly. the case for ks work paints an attractive picture. everyone wants to do meaningful work and if individual hips and their libraries can benefit in the process, through enhanced reputation or otherwise, all for the better. but as with anything else, ks work comes with challenges, and some hips may find that everyday ks work may not live up to their expectations or consider it worth the amount of effort involved. while individuals’ personal experience with ks may vary, this study’s findings suggest that our collective relationship with ks work has been gradually changing, and hips are starting to become more comfortable with setting boundaries around ks work for their own benefit. acknowledgments thank you to andrea szwajcer (university of manitoba) for engaging with me in discussions on research in general and this study in particular. thanks also go to the people who gave me feedback on various iterations of this manuscript: andrea szwajcer (university of manitoba), carol cooke (university of manitoba). conflict of interest the author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article. funding the author received no financial support for the research, authorship, and/or publication of this article. data availability statement the following material is available under a cc by-nc-sa 4.0 license, in the university of manitoba dataverse repository(75) at <https://doi.org/10.5683/sp3/tjucsg>. • pdf file complete reference list of texts examined • excel file – cases with attribute coding 23 https://doi.org/10.5683/sp3/tjucsg peer reviewed article hypothesis, vol. 37, no. 2, 2025 • excel file – cross tabs, descriptive codes related to the research question “what do we talk about?” • pdf file code definitions references 1. better together in evidence synthesis: the evidence synthesis taxonomy initiative | jbi [internet]. [cited 2024 nov 4]. available from: https://evidencesynthesistaxonomy.com/better-together-in-evidence-synthesis-the-evidencesynthesis-taxonomy-initiative/. 2. slebodnik m, pardon k, hermer j. who’s publishing systematic reviews? an examination beyond the health sciences. issues in science and technology librarianship [internet]. 2022 [cited 2024 nov 22]; doi:10.29173/istl2671 3. kallaher a, eldermire erb, fournier ct, ghezzi-kopel k, johnson ka, morris-knower j, scinto-madonich s, young s. library systematic review service supports evidence-based practice outside of medicine. the journal of academic librarianship. 2020;46:102222. doi:10.1016/j.acalib.2020.102222 4. ghezzi-kopel k, ault j, chimwaza g, diekmann f, eldermire e, gathoni n, kelly j, kinengyere aa, kocher m, lwoga et, et al. making the case for librarian expertise to support evidence synthesis for the sustainable development goals. research synthesis methods. 2022;13:77–87. doi:10.1002/jrsm.1528 5. premji z, splenda r, young s. an exploration of business librarian participation in knowledge synthesis reviews. college & research libraries. 2022;83:314. doi:10.5860/crl.83.2.314 6. myers b. what we talk about when we talk about medical librarianship: an analysis of medical library association annual meeting abstracts, 2001–2019. j med libr assoc. 2020;108:364–377. doi:10.5195/jmla.2020.836 7. wilson v. something for everyone? a content analysis of provincial library association conference sessions. partnership: the canadian journal of library & information practice & research. 2010;5:1–13. doi:10.21083/partnership.v5i1.1097 8. rudd s. trends in health librarianship topics presented at four uk-based conferences between 2017 and 2022: a thematic analysis. health information & libraries journal. 2024;1–8. doi:10.1111/hir.12561 9. julien h, fena c. thirty-one years of the canadian journal of information and library science: a content analysis. canadian journal of information & library sciences. 2018;42:1–17. 10. chang y-w. characteristics of articles coauthored by researchers and practitioners in library and information science journals. the journal of academic librarianship. 24 https://evidencesynthesistaxonomy.com/better-together-in-evidence-synthesis-the-evidence-synthesis-taxonomy-initiative/ https://evidencesynthesistaxonomy.com/better-together-in-evidence-synthesis-the-evidence-synthesis-taxonomy-initiative/ https://dx.doi.org/10.29173/istl2671 https://dx.doi.org/10.1016/j.acalib.2020.102222 https://dx.doi.org/10.1002/jrsm.1528 https://dx.doi.org/10.5860/crl.83.2.314 https://dx.doi.org/10.5195/jmla.2020.836 https://dx.doi.org/10.21083/partnership.v5i1.1097 https://dx.doi.org/10.1111/hir.12561 peer reviewed article hypothesis, vol. 37, no. 2, 2025 2016;42:535–541. doi:10.1016/j.acalib.2016.06.021 11. sung h-y, parboteeah p. diversity-related research reported in high-impact library and information science journal literature: a content analysis. library & information science research. 2017;39:77–84. doi:10.1016/j.lisr.2017.03.002 12. hashemian m, adibi p, yamani n, rahimi a, zare-farashbandi f. clinical informationist services challenges: a qualitative content analysis of the literature. journal of hospital librarianship. 2021;21:319–327. doi:10.1080/15323269.2021.1982257 13. given lm. constructivism [internet]. the sage encyclopedia of qualitative research methods. sage publications, inc.; 2008 [cited 2024 nov 5]. 14. hewson c. mixed methods research [internet]. the sage dictionary of social research methods. sage publications, ltd; 2006 [cited 2024 nov 5]. 15. saldaña j. the coding manual for qualitative researchers. 2nd ed. los angeles; sage; 2013. 16. braun v, clarke v. using thematic analysis in psychology. qualitative research in psychology. 2006;3:77–101. doi:10.1191/1478088706qp063oa 17. kiger me, varpio l. thematic analysis of qualitative data: amee guide no. 131. medical teacher. 2020;42:846–854. doi:10.1080/0142159x.2020.1755030 18. nvivo 14. lumivero; 2023. 19. deepl translate: the world’s most accurate translator [internet]. cologne germany: deepl se; [cited 2024 nov 5]. available from: https://www.deepl.com/translator. 20. sutton a, clowes m, preston l, booth a. meeting the review family: exploring review types and associated information retrieval requirements. health information & libraries journal. 2019;36:202–222. doi:10.1111/hir.12276 21. conference programmes / programmes de congres canadian health libraries association / association des bibliothèques de la santé du canada [internet]. [cited 2024 nov 6]. available from: https://www.chla-absc.ca/chla_fa_list_3815985.php. 22. white pj. evidence-based medicine for consumers: a role for the cochrane collaboration. j med libr assoc. 2002;90:218–222. 23. lohr am, van gorden n, mcclelland dj, dubinsky e, gerald lb, wilkinson-lee a, carvajal sc. updating search strategies for literature reviews with our2d2: an open-source computer application. j med libr assoc. 2021;109:317–322. doi:10.5195/jmla.2021.1105 24. fang mle, persily g, reavie k, owen d, coppernoll-blach p. expert searchers’ contributions to evidence-based health policy making [internet]. poster presented at: medical library association annnual conference; 2006; pheonix. available from: 25 https://dx.doi.org/10.1016/j.acalib.2016.06.021 https://dx.doi.org/10.1016/j.lisr.2017.03.002 https://dx.doi.org/10.1080/15323269.2021.1982257 https://dx.doi.org/10.1191/1478088706qp063oa https://dx.doi.org/10.1080/0142159x.2020.1755030 https://www.deepl.com/translator https://dx.doi.org/10.1111/hir.12276 https://www.chla-absc.ca/chla_fa_list_3815985.php https://dx.doi.org/10.5195/jmla.2021.1105 peer reviewed article hypothesis, vol. 37, no. 2, 2025 https://www.mlanet.org/d/do/1886 25. fang ml. medical librarian’s contributions to evidence-based health policy decision making in california [internet]. poster presented at: medical library association annual conference; 2013; boston, ma. 26. ginier ec, anderson pf. librarians in systematic review teams: extracting patterns of roles and tasks from the published literature [internet]. oral presentation presented at: medical library association annual conference; 2018; atlanta, ga. 27. price c, riese v, lobner k, friel blanck j, anton b. beyond the search: librarian involvement on the systematic review team [internet]. oral presentation presented at: medical library association annual conference; 2015; austin, tx. 28. fox ze, williams am, blasingame mn, koonce ty, kusnoor sv, su j, lee p, epelbaum mi, naylor hm, desautels sj, et al. why equating all evidence searches to systematic reviews defies their role in information seeking. j med libr assoc. 2019;107:613–617. doi:10.5195/jmla.2019.707 29. clifton s, tonn b, coffman m, ditzler w, hopkins m, richards j. connecting the dots: an analysis of search strategy reporting in meta-analyses [internet]. poster presented at: medical library association annual conference; 2008; chicago, il. 30. morris m, boruff jt, gore gc. scoping reviews: establishing the role of the librarian. j med libr assoc. 2016;104:346–354. doi:10.3163/1536-5050.104.4.020 31. cobus-kuo l, gore g, kloda l. research syntheses in graduate research: a scoping review [internet]. oral presentation presented at: chla/absc annual conference; 2014 jun; montreal, qc. available from: https://www.chla-absc.ca/docs/conferenceprogram2014.pdf. 32. mcgowan j, sampson m. systematic reviews need systematic searchers. j med libr assoc. 2005;93:74–80. 33. nolfi da, sasso md. situating systematic reviews and librarians at the nexus of teaching and research [internet]. oral presentation presented at: medical library association; 2019; chicago, il. 34. wissinger cl. is there a place for undergraduate and graduate students in the systematic review process? j med libr assoc. 2018;106:248–250. doi:10.5195/jmla.2018.387 35. foster mj, fulton s, crews me, halling td, herbert be, sewell r. reflection of needs: faculty survey on systematic review experiences and expectations [internet]. lightning talk presented at: medical library association annual conference; 2022; new orleans, la. 36. yang l, orchanian-cheff a, anderson m, farrell a, tripp t. implementing a three-tiered service model for knowledge syntheses at an academic teaching hospital. journal of the canadian health libraries association / journal de l’association des bibliothèques de la santé 26 https://www.mlanet.org/d/do/1886 https://dx.doi.org/10.5195/jmla.2019.707 https://dx.doi.org/10.3163/1536-5050.104.4.020 https://www.chla-absc.ca/docs/conferenceprogram2014.pdf https://dx.doi.org/10.5195/jmla.2018.387 peer reviewed article hypothesis, vol. 37, no. 2, 2025 du canada. 2020;41:46–66. doi:10.29173/jchla29436 37. roth sc. transforming the systematic review service: a team-based model to support the educational needs of researchers. j med libr assoc. 2018;106:514–520. doi:10.5195/jmla.2018.430 38. desmeules r, campbell s, dorgan m. acknowledging librarians’ contributions to systematic review searching. journal of the canadian health libraries association / journal de l’association des bibliothèques de la santé du canada [internet]. 2016 [cited 2024 jan 8];37. doi:10.5596/c16-014 39. ross-white a. an environmental scan of librarian involvement in systematic reviews at queen’s university: 2020 update. journal of the canadian health libraries association / journal de l’association des bibliothèques de la santé du canada [internet]. 2021 [cited 2024 jan 8];42. doi:10.29173/jchla29517 40. townsend wa, anderson pf, ginier ec, maceachern mp, saylor km, shipman bl, smith je. a competency framework for librarians involved in systematic reviews. j med libr assoc. 2017;105:268–275. doi:10.5195/jmla.2017.189 41. tietgen jj, vardell e. paths to systematic review librarianship: an exploratory study [internet]. oral presentation presented at: medical library association annual conference; 2021; virtual 42. justice em, belleh e, easterby-gannett s, moran d, evans j, riesenberg la. finding our way in the world of systematic reviews: hospital librarians contribute to the creation of systematic reviews [internet]. poster presented at: medical library association annual conference; 2013; boston, ma. 43. demetres mr, wright dn, derosa ap. burnout among medical and health sciences information professionals who support systematic reviews: an exploratory study. j med libr assoc. 2020;108:89–97. doi:10.5195/jmla.2020.665 44. howard am, butcher n. daring to realize the dream of published systematic reviews [internet]. oral presentation presented at: medical library association annual conference; 2017; seattle, wa. 45. nicholson j, mccrillis a, williams jd. collaboration challenges in systematic reviews: a survey of health sciences librarians. j med libr assoc. 2017;105:385–393. doi:10.5195/jmla.2017.176 46. saragossi j. bibliometric analysis as a method to justify library personnel [internet]. lightning talk presented at: medical library association annual conference; 2022; new orleans, la. 47. saimbert mk, pierce j (jenny), hargwood p, oliver jt. librarians collaborating with faculty for scholarly publication [internet]. poster presented at: medical library association annual conference; 2011; minneapolis, mn. 27 https://dx.doi.org/10.29173/jchla29436 https://dx.doi.org/10.29173/jchla29436 https://dx.doi.org/10.5596/c16-014 https://dx.doi.org/10.29173/jchla29517 https://dx.doi.org/10.5195/jmla.2017.189 https://dx.doi.org/10.5195/jmla.2020.665 https://dx.doi.org/10.5195/jmla.2017.176 peer reviewed article hypothesis, vol. 37, no. 2, 2025 48. krause k, varman bg, galati m, huynh n. systematic reviews: the evolution of a new library service [internet]. oral presentation presented at: chla/absc mla joint conference; 2016 may; toronto, on. 49. lackey mj, greenberg h, rethlefsen ml. building the systematic review core in an academic health sciences library. j med libr assoc. 2019;107:588–594. doi:10.5195/jmla.2019.711 50. deaver j, o’hagan ec, billings r. reflecting on librarians’ needs: renewing a systematic review service [internet]. oral presentation presented at: medical library association annual conference; 2022; new orleans, la. 51. sikora l, bass mb. sharpening the other side of the dual edged systematic review sword: expecting more from our users [internet]. oral presentation presented at: chla/absc annual conference; 2018 jun; st. john’s, nl. available from: https://www.chla-absc.ca/docs/chlaabsc18program.pdf. 52. calleja s, zeigler c. required, not suggested: creating a rigorous protocol template for improved uptake and efficacy [internet]. poster presented at: chla/absc annual conference; 2023 jun; halifax, ns. available from: https://www.chla-absc.ca/docs/chlaabsc2023_fullprogram_20230525.pdf. 53. campbell s, dorgan m. what to do when everyone wants you to collaborate: managing the demand for library support in systematic review searching. j can health libr assoc. 2015;36:11–19. doi:10.29173/jchla/jabsc.v36i1.24353 54. spencer aj, eldredge jd. roles for librarians in systematic reviews: a scoping review. j med libr assoc. 2018;106:46–56. doi:10.5195/jmla.2018.82 55. ettarh f. vocational awe and librarianship: the lies we tell ourselves. in the library with the lead pipe [internet]. 2018 [cited 2024 july 19]; available from: https://www.inthelibrarywiththeleadpipe.org/2018/vocational-awe/. 56. bramer wm. information specialists: guardians of scientific output of their institute [internet]. oral presentation presented at: medical library association annual conference; 2022; new orleans, la. 57. hanneke r. the hidden benefits of helping students with systematic reviews. j med libr assoc. 2018;106:244–247. doi:10.5195/jmla.2018.420 58. sackett dl, rosenberg wmc, gray jam, haynes rb, richardson ws. evidence based medicine: what it is and what it isn’t. bmj. 1996;312:71–72. doi:10.1136/bmj.312.7023.71 59. mcmenamin a, sun c, prufeta p, raso r. the evolution of evidence-based practice. nursing management. 2019;50:14. doi:10.1097/01.numa.0000579000.09987.b0 60. funk me. our words, our story: a textual analysis of articles published in the bulletin of the medical library association/journal of the medical library association from 1961 to 28 https://dx.doi.org/10.5195/jmla.2019.711 https://www.chla-absc.ca/docs/chlaabsc18program.pdf https://www.chla-absc.ca/docs/chlaabsc2023_fullprogram_20230525.pdf https://doi.org/10.29173/jchla/jabsc.v36i1.24353 https://doi.org/10.5195/jmla.2018.82 https://www.inthelibrarywiththeleadpipe.org/2018/vocational-awe/ https://dx.doi.org/10.5195/jmla.2018.420 https://dx.doi.org/10.1136/bmj.312.7023.71 https://dx.doi.org/10.1097/01.numa.0000579000.09987.b0 peer reviewed article hypothesis, vol. 37, no. 2, 2025 2010. j med libr assoc. 2013;101:12–20. doi:10.3163/1536-5050.101.1.003 61. about us [internet]. shanghairanking. [cited 2024 nov 18]. available from: https://www.shanghairanking.com/about-arwu. 62. world university rankings [internet]. times higher education (the). 2018 [cited 2024 nov 18]. available from: https://www.timeshighereducation.com/world-university-rankings. 63. hirsch je. an index to quantify an individual’s scientific research output. proceedings of the national academy of sciences of the united states of america. 2005;102:16569–16572. 64. bornmann l, mutz r, daniel h-d. are there better indices for evaluation purposes than the index? a comparison of nine different variants of the index using data from biomedicine. journal of the american society for information science and technology. 2008;59:830–837. doi:10.1002/asi.20806 65. reflecting on 20 years of knowledge discovery: 20 years of supporting research and innovation [internet]. www.elsevier.com. [cited 2024 nov 15]. available from: https://www.elsevier.com/products/scopus/20-years-of-discovery. 66. scopus 20: 20 years of supporting research and innovation [infographic] [internet]. [cited 2024 nov 15]. available from: https://images.ctfassets.net/o78em1y1w4i4/7gq1lpfqnphikyosu0g8wi/9f15d41b3534c0ed9c 7ff5ecfc19a3d8/scopus_20_infographic_v2.jpg. 67. read the declaration [internet]. dora. [cited 2024 nov 13]. available from: https://sfdora.org/read/. 68. watts g. research funding goes metric. bmj. 2008;337:a1805. doi:10.1136/bmj.a1805 69. hicks d. performance-based university research funding systems. research policy. 2012;41:251–261. doi:10.1016/j.respol.2011.09.007 70. montori vm, wilczynski nl, morgan d, haynes rb. systematic reviews: a cross-sectional study of location and citation counts. bmc med. 2003;1:1–7. doi:10.1186/1741-7015-1-2 71. bhandari m, montori vm, devereaux pj, wilczynski nl, morgan d, haynes rb. doubling the impact: publication of systematic review articles in orthopaedic journals. jbjs. 2004;86:1012. 72. patsopoulos na, analatos aa, ioannidis jpa. relative citation impact of various study designs in the health sciences. jama. 2005;293:2362–2366. doi:10.1001/jama.293.19.2362 73. clarke m, alderson p, chalmers i. discussion sections in reports of controlled trials published in general medical journals. jama. 2002;287:2799–2802. doi:10.1001/jama.287.21.2799 29 https://dx.doi.org/10.3163/1536-5050.101.1.003 https://www.shanghairanking.com/about-arwu https://www.timeshighereducation.com/world-university-rankings https://dx.doi.org/10.1002/asi.20806 https://www.elsevier.com/products/scopus/20-years-of-discovery https://images.ctfassets.net/o78em1y1w4i4/7gq1lpfqnphikyosu0g8wi/9f15d41b3534c0ed9c7ff5ecfc19a3d8/scopus_20_infographic_v2.jpg https://images.ctfassets.net/o78em1y1w4i4/7gq1lpfqnphikyosu0g8wi/9f15d41b3534c0ed9c7ff5ecfc19a3d8/scopus_20_infographic_v2.jpg https://sfdora.org/read/ https://dx.doi.org/10.1136/bmj.a1805 https://dx.doi.org/10.1016/j.respol.2011.09.007 https://dx.doi.org/10.1186/1741-7015-1-2 https://dx.doi.rg/10.1001/jama.293.19.2362 https://dx.doi.org/10.1001/jama.287.21.2799 peer reviewed article hypothesis, vol. 37, no. 2, 2025 74. hicks d. the construction of librarians’ professional identities: a discourse analysis / la construction de l’identité professionnelle du bibliothécaire: une analyse de discours. canadian journal of information and library science. 2014;38:251–270. 75. neilson c. data files for: an examination of health information professionals’ discourse surrounding knowledge synthesis: a content analysis [internet]. borealis; 2025 [cited 2025 aug 22]. available from: https://borealisdata.ca/dataset.xhtml?persistentid=doi:10.5683/sp3/tjucsg. 30 https://borealisdata.ca/dataset.xhtml?persistentid=doi:10.5683/sp3/tjucsg editorial reviewed article hypothesis , vol. 37, no. 2, 2025 mla 2025 annual meeting research awards lindsay e. blake, edd, mlis, ahipa aclinical services coordinator, uams library, university of arkansas for medical sciences, little rock, arkansas, https://www.orcid.org/0000-0002-8234-8611 , leblake@uams.edu cite as: blake le. mla 2025 annual meeting research awards. hypothesis. 2025;37(2). doi:10.18060/28904. blake. all works in hypothesis are licensed under a cc by-nc 4.0 deed attribution-noncommercial 4.0 international. authors own copyright of their articles appearing in hypothesis. readers may copy articles without permission of the copyright owner(s), as long as the author(s) are acknowledged in the copy, and the copy is used for educational, not-for-profit purposes. for any other use of articles, please contact the copyright owner(s). 1 https://www.orcid.org/0000-0002-8234-8611 https://orcid.org/ mailto:leblake@uams.edu https://creativecommons.org/licenses/by-nc/4.0/ https://creativecommons.org/licenses/by-nc/4.0/ editorial reviewed article hypothesis , vol. 37, no. 2, 2025 congratulations to the award-winning research papers and posters from mla ’25 pittsburgh! the mla research caucus is pleased to announce the winners for best research papers and posters presented at the mla 2025 pittsburgh annual meeting. thank you to all the judges who volunteered their expertise to help select these deserving awardees both in the pre-judging phase and during the conference. to learn more about the awards and selection process, visit the research caucus on mlanet.org. mla 2025 annual meeting contributed paper awards 1st place – readability and understandability of online patient education in semaglutide: recommendations for consumer health librarians. authors: alexandria q. wilson, senior medical librarian, r. eric heidel, shauntá chamberlin, and kelsey l. grabeel, university of tennessee graduate school of medicine. objectives: media attention and public interest in semaglutide medications like ozempic have led to increased numbers of people searching for information about this drug online. given that national estimates suggest that many people struggle with health literacy, we hypothesize that most online semaglutide information will be written at or above the recommended level of 8th grade for most people. the primary objective is to assess the readability of online information about semaglutide. the secondary objectives will be to assess the understandability and quality. methods: web links were collected by searching three terms (‘ozempic,’ ‘wegovy,’ and ‘semaglutide’) on google in private incognito browsers using three different public computers. the non-sponsored results on the first five pages for each search were manually imported into excel and screenshots of each result page were saved. duplicate links were removed and screened against inclusion/exclusion criteria by two researchers independently. web links were excluded if they met the following criteria: (1) main purpose is advertisement, (2) patient testimonials, (3) content for healthcare professionals, (4) non-text material, and (5) research articles. inconsistencies were solved by discussion. the text from included web links will be extracted and evaluated by two researchers independently. included web links were evaluated for readability using the smog readability formula and flesch reading ease, for understandability using the patient education materials assessment tool (pemat) and for quality using discern. a statistician ran reports for interrater-reliability (cronbach’s alpha), medians and interquartile ranges, and frequency statistics of the evaluations. results: three searches resulted in n=143 links. after duplicate removal, n=125 links were screened for inclusion and n=61 met evaluation criteria. inter-rater reliability coefficients ranged poor (a = 0.37) to perfect (a = 1.00) between reviewers. median scores for smog and fre were 13th grade level and college, respectively. fewer than 10% of the documents were found to be at or below the recommended reading grade level. the documents at or below were from hospital or international government websites. for understandability, the median score of pemat was 62%. for quality, the median overall score of discern was 4 out of 5. conclusions: while the average quality was good, the majority of patient education 2 www.mlanet.org editorial reviewed article hypothesis , vol. 37, no. 2, 2025 available online about semaglutide medications is not written at the appropriate reading level for the average consumer. patient education on semaglutide needs to be rewritten to be at the recommended 8th grade reading level. additionally, with the median understandability at 62%, more needs to be done to improve the education to make it easier for the reader to understand. these findings are likely not unique to semaglutide medications, but the popularity of these medications suggests an increased need for patient education that is more understandable and readable. librarians who distribute online patient education to the public should be aware of the potential high reading grade level for this topic. 2nd place – “social media is imperfect, but it’s the best we have”: understanding and addressing sexual health information seeking challenges among young adults. author: nick vera, user experience and educational strategist, medical university of south carolina. summary: this session explores the critical role of health sciences librarians in addressing sexual health information disparities among young adults in south carolina. by investigating how young adults use social media platforms as primary sources for sexual health information, this study identifies key barriers such as misinformation, algorithmic biases, and challenges in evaluating content credibility. through a dual-method qualitative approach, semi-structured interviews, and digital walkthroughs. this session provides insights into young adults’ strategies and experiences when navigating social media for sexual health information. it highlights opportunities for health sciences librarians to develop targeted digital health literacy resources, including workshops, fact-checking initiatives, and resource guides designed to equip young adults with the skills needed to critically assess and use online sexual health information. attendees will gain practical strategies for implementing digital literacy programs that enhance health information accessibility and empower young adults to make informed decisions about their sexual health. objectives: young adults in south carolina face increasing challenges when seeking accurate sexual health information due to limitations in traditional and authoritative sexual health education and healthcare access. this paper investigates the future of libraries and health science librarians by exploring how young adults use social media platforms to seek sexual health information. it identifies the primary barriers young adults face and investigates opportunities for developing innovative digital health literacy resources. methods: this study employed a dual-method qualitative approach, utilizing 90-minute semi-structured interviews and digital walkthroughs with 20 young adults aged 18-25 from across south carolina. the interviews explored participants’ experiences and strategies when using social media to seek sexual health information, while the digital walkthroughs provided real-time observations of their navigation patterns, platform choices, and interactions. this dual-method approach allowed for an in-depth analysis of the role that social media functionalities, algorithms, and community interactions play in shaping the information-seeking practices of young adults. the data included for analysis consisted of transcribed participant narratives and screen recordings of participants’ digital walkthroughs. all data were analyzed thematically using nvivo to identify salient patterns, challenges, and opportunities for 3 editorial reviewed article hypothesis , vol. 37, no. 2, 2025 interventions such as digital health literacy workshops, collaborative fact-checking initiatives, and the development of educational resources tailored to guide young adults in critically assessing and navigating social media for accurate sexual health information. results: preliminary findings show that while social media offers accessible sexual health resources, young adults struggle to find credible and relevant information. barriers include misinformation, algorithmic biases that obscure accurate content, and challenges in assessing creators’ credibility. there is also mistrust of medical professionals due to perceived cultural incompetence in addressing young adults’ needs, resulting in alternative strategies to navigate these obstacles. these findings emphasize the need for targeted interventions, such as resource guides and digital literacy programs, to improve digital health literacy and provide young adults with the skills to evaluate and effectively use online sexual health information critically. conclusions: the study emphasizes the crucial role of health sciences librarians in connecting young adults with credible sexual health information on social media. instead of focusing solely on users’ information gaps, health librarians can shift their approach by developing resources that align with young adults’ existing social media habits. this involves creating resource guides and digital literacy programs that teach fact-checking techniques, identify trustworthy content creators, and explain platform algorithms. by offering these targeted services, health sciences librarians enhance the accessibility and quality of health information while positioning themselves as central figures in addressing sexual health disparities. this proactive approach meets the evolving information needs of users and highlights the future role of health sciences librarianship in advancing digital health literacy. 3rd place – identifying strengths and gaps in health equity research: a portfolio analysis of federally funded grants utilizing community engaged research methods. authors: poppy krump, information scientist, taneya koonce, consuelo wilkins, spencer desautels, mallory blasingame, alicia cavanaugh, laura logie, tiffany vassell, braveheart gillani, jasmine bell, and nunzia guise, vanderbilt university medical center. summary: to advance the science of community-engaged research, gaining an understanding of previously funded research is required. analysis of federal grants can provide insights into community engagement trends. this analysis was able to identify the trends and gaps in community engaged research, presenting opportunities to improve the way research could be conducted and funded in the future. information scientists, given their expertise in information retrieval and data organization, can play key roles in interpreting, structuring, and implementing grant portfolio analyses. mla 2025 annual meeting contributed poster awards 1st place – what does the body know? nursing students’ epistemic beliefs about embodied health misinformation. author: brynne campbell rice, librarian for health sciences: nursing, new york university. 4 editorial reviewed article hypothesis , vol. 37, no. 2, 2025 summary: academic health librarians are deeply invested in helping future healthcare professionals develop the information literacy skills necessary to navigate an increasingly fraught information landscape. in that vein, this poster presents initial results from a qualitative study exploring how nursing students contend with a particularly complex form of health misinformation. specifically, this study investigates nursing students’ perceptions and epistemic beliefs related to ”embodied health misinformation” the misinformation that arises in the messy intersection where bodily experiences conflict with biomedical evidence. in investigating how nursing students respond to this type of misinformation, the study aims to reveal how they engage with information outside of traditional scholarly evidence, challenging simplistic binaries in the study of misinformation. in doing so, this study contributes to the scholarly conversation around how health professionals’ information literacy practices can balance epistemic justice with a commitment to evidence-based health care. objectives: the objective of this research is to explore nursing students’ perceptions and evaluation of ”embodied health misinformation” misinformation that is woven into bodily experience, where an individuals’ intimate knowledge of their own body is positioned as more credible than biomedical evidence. healthcare professionals need well-developed personal epistemologies to navigate these complexities, yet there’s limited research into how they perceive this type of information. to fill that gap, this study asks: (1) what are nursing students’ perspectives on the body as a source of health information and misinformation? and (2) what are nursing students’ epistemic beliefs related to embodied health misinformation? methods: this study follows a qualitative methodology, employing a series of in-depth, semi-structured interviews with nursing students. in order to elucidate their thoughts on how to contend with bodily information when it appears to convey misinformation, participants are asked to respond to two conflicting information sources on the same topic: a personal health narrative that incorporates subjective, affective health experiences, and an evidence-based information source. the audio of the recorded interviews is transcribed and transcripts are coded inductively for emergent themes in the nursing students’ perspectives on bodily information, and deductively for any specific epistemic beliefs they reveal in their responses. results: preliminary results following the analysis of 6 interviews with undergraduate nursing students reveal key initial themes and topics that include: triangulation of bodily information with external sources of information; the practice of bodily listening; issues of individual vs. generalized health claims; and considerations of time and risk in evaluating health information. conclusions: this project is ongoing, but it is anticipated that the results will contribute to broader conversations about the development of sensitive information evaluation practices among students who are entering health professions. specifically, understanding nursing students’ perceptions and beliefs around embodied information can inform how librarians help prepare health professionals to contend with the full complexity of the health misinformation that they will encounter, as both information consumers and eventually, trusted sources of health information themselves. 2nd place – automatic indexing errors in pubmed: sometimes the apple does fall far from the tree. author: paije wilson, health sciences librarian, university of wisconsin-madison. 5 editorial reviewed article hypothesis , vol. 37, no. 2, 2025 summary: the objective of this study was to explore the prevalence and characteristics of precision errors in a convenient sample of automatically indexed medline records in pubmed. specifically, the sample was comprised of records automatically indexed with the mesh term, malus (the genus name for apple tree). objectives: since 2022, fully automatic indexing has been used to assign medical subject heading (mesh) to medline records in pubmed. though automatic indexing has improved efficiency, there have been concerns relating to its precision. the objective of this study was to explore the prevalence and characteristics of precision errors in a convenient sample of automatically indexed medline records in pubmed. specifically, the sample was comprised of records automatically indexed with the mesh term, malus (the genus name for apple tree). methods: the following search string was used in pubmed to capture records automatically indexed with the mesh term, malus: ”malus[mesh] and indexingmethod automated”. the search retrieved 1,705 records, which were exported from pubmed and then imported into endnote 21. title/abstract screening was conducted in endnote 21, wherein records were categorized as being correctly indexed (i.e., the record was about malus, meaning they referred to the literal tree or fruit), incorrectly indexed, or uncertain (i.e., there was not enough information in the title/abstract to verify if they were correctly indexed with the mesh term, malus). records identified as being incorrectly indexed or uncertain were exported into an excel sheet, wherein they underwent full text screening. during full text screening, records were labeled as being correctly indexed, incorrectly indexed, or ”could not determine” (i.e., the full text couldn’t be accessed for the record). data extraction was performed for records labeled as being incorrectly indexed during full text screening, which consisted of copying quotations where variations of the term, malus, (e.g., apple, apple tree, etc.) were used in the record, and pasting them into an excel sheet. these quotations were then labeled with the type of indexing error (e.g., figurative language, names, acronyms, etc.). results: 1,705 records were automatically indexed with the mesh term, malus. of these records, 1,562 (91.6%) were correctly indexed, 135 (7.9%) were incorrectly indexed, and 8 (0.5%) could not be determined. of the 135 records incorrectly indexed, 80 (59%) were due to the use of figurative language (e.g., ”comparing apples to oranges”, ”apple shaped”, etc.). 50 (37%) were due to names (e.g., apple, inc., miyake-apple technique, apple snails, etc.). 4 (3%) were due to acronyms (e.g., ”access to post partum larc in edinburgh south (apples)”). 1 (1%) was due to a passing reference to sir isaac newton. conclusions: with the recent implementation of the national library of medicine’s newest automatic indexing algorithm, mtix, to pubmed in 2024, studies documenting automatic indexing errors in pubmed are especially vital. in addition to bringing attention to automatic indexing errors, such studies may inform future refinements to the mtix algorithm to reduce erroneous indexing. though limited, this study gives some insight into the prevalence and characteristics of automatic indexing errors in pubmed. as demonstrated by this study, the automatic indexing algorithm can experience errors when encountering words used in non-literal contexts. until automatic indexing errors can be resolved, librarians should be mindful of their existence, and instruct researchers on how to potentially avoid future indexing errors in their own manuscripts. 6 editorial reviewed article hypothesis , vol. 37, no. 2, 2025 3rd place – a multi-modal evaluation: harnessing generative ai to understand the state-of-the-art of literature search automation. authors: taneya koonce, deputy director, annette williams, dario guise, jing su, mallory blasingame, poppy krump, and nunzia giuse, vanderbilt university medical center. summary: in collaboration with informatics colleagues at our institution, this investigation explores the feasibility of delegating tasks currently undertaken by our library team to generative artificial intelligence (ai) tools. through a meticulously designed research endeavor, we sought to comprehensively assess the contemporary capabilities of generative ai in the domain of pubmed search compilation. objectives: in collaboration with informatics colleagues at our institution, this investigation explores the feasibility of delegating tasks currently undertaken by our library team to generative artificial intelligence (ai) tools. specifically, we explored the efficacy of ai tools in formulating reliable search strategies; we will be reporting on the current state-of-the-art of ai tools while engaged in the capacity of “searcher.” through a meticulously designed research endeavor, we sought to comprehensively assess the contemporary capabilities of generative ai in the domain of pubmed search compilation. methods: the project leverages an internally-developed “quick clinical view” feature of our institutional medical record that aims to provide clinicians with rapid insights into patient conditions at the population level by highlighting concepts with high strengths of associations. it uses the database of population-level associations created by an institutional natural language processing (nlp) system, which maps free-text terms from all non-image clinical documents at our medical center to the unified medical language system (umls) terminology. our team is exploring whether generative ai has advanced enough to automatically integrate literature searches into this system—a potentially transformative feature that could rapidly identify unfamiliar/unexpected clinical associations. from the “quick clinical view” of ten health conditions (e.g., small cell lung carcinoma, orthostatic hypotension), we selected two concepts each (i.e., 20 condition/concept pairs) that occur rarely in the general population. to assess the feasibility of automating the literature search integration, the team created generative ai prompts designed to produce pubmed search strategies in the pico format. our team evaluated search strategy results for the 20 selected condition/concept pairs across three different large language models (llms): chatgpt, google gemini, and microsoft copilot by comparing them to the team’s gold-standard search output. results: the llm tools generated collectively 150 terms/phrases tagged as mesh, of which, 58 (39%) were accurate. chatgpt generated accurately labeled mesh terms 45% (41/92) of the time and copilot 31% (17/56). gemini labeled two terms as mesh; neither were accurate. the 64 relevant citations identified by our team through their gold-standard searches were furthermore compared to the search results of each llm. the llm tools’ search results included 47% (30/64) of these citations overall; chatgpt included 17% (11/64), gemini included 22% (14/64), and copilot included 8% (5/64), with instances yielding zero search results. conclusions: the generative ai-created search strategies often failed to produce pubmed results, were frequently inaccurate in labeling search terms/phrases as mesh, and for the searches that did yield results, the gold-standard selected citations were 7 editorial reviewed article hypothesis , vol. 37, no. 2, 2025 present less than half of the time. moving forward, it will be important to evaluate the entire set of citations retrieved by the llm tools as they may offer value-added to the citations identified by the librarians. this study, once more, underlines the critical importance of having knowledge workers with the necessary domain knowledge involved in assessing and evaluating the performance of generative ai tools. 8 peer reviewed article hypothesis, vol. 37, no. 2, 2025 making time for research (and you): using an “intentionality toolkit” to achieve your goals and mitigate feelings of overwhelm thea atwooda apublic and environmental health sciences librarian, indiana university bloomington, bloomington, indiana, https://orcid.org/0000-0001-8348-1097 , tpatwood@iu.edu cite as: atwood t. making time for research (and you): using an “intentionality toolkit” to achieve your goals and mitigate feelings of overwhelm. hypothesis. 2025;37(2). doi:10.18060/28774 atwood. all works in hypothesis are licensed under a cc by-nc 4.0 deed attribution-noncommercial 4.0 international. authors own copyright of their articles appearing in hypothesis. readers may copy articles without permission of the copyright owner(s), as long as the author(s) are acknowledged in the copy, and the copy is used for educational, not-for-profit purposes. for any other use of articles, please contact the copyright owner(s). 1 https://www.orcid.org/0000-0001-8348-1097 https://orcid.org/ mailto:tpatwood@iu.edu https://creativecommons.org/licenses/by-nc/4.0/ https://creativecommons.org/licenses/by-nc/4.0/ peer reviewed article hypothesis, vol. 37, no. 2, 2025 abstract introduction: academic, professional, and personal life are a lot to balance, and there is always more work to do than there are hours in the day. the bullet journal method, the weekly meeting, and the semester plan, which come together as an “intentionality toolkit,” are three practices to help prioritize competing interests, mitigate feelings of overwhelm, and practice saying “no.” experience: each method hosts its own distinct benefits: the bullet journal method offers a strategy for managing your day-to-day tasks, the weekly meeting provides short-term planning and the opportunity to practice saying “no” to requests, and the semester plan helps break down long-term goals into smaller, achievable tasks. discussion: an intentionality toolkit can help in numerous ways. this includes reducing cognitive load; improving prioritization practices; improving openness and flexibility; mitigating perfectionism; owning your time; and incorporating reflective practices into your workweek. this isn’t a perfect toolkit. it is a set of options that might be beneficial to librarians. takeaways: these three techniques create a toolkit to provide structure and personal accountability, specifically to achieve goals that traditionally have less oversight and accountability, like writing and research. introduction we are asked daily to make choices about a variety of competing priorities: emails, consultations, childand elder care decisions, teaching, professional development, sick time (for yourself or someone that depends on your care), research interests, vacation time, service commitments, doctor appointments, travel, and so much more. requests from our supervisors compete with our own priorities or goals, and creating a thoughtful reply can feel daunting – maybe even career-ending if you feel you need to say “no” to a request. our lives are made up of uniquely complex dynamics, which, unaddressed, can feel overwhelming and crushing. this paper highlights three techniques that comprise an “intentionality toolkit.” these include the bullet journal method, the weekly planning meeting, and the semester plan. while not the cure for burnout or the stresses of life, the toolkit is a starting point. it is of use to librarians and professionals who wish to improve control over their day-to-day activities, achieve long-term aspirations, and practice saying “no” to requests that are not aligned with their goals. the bullet journal method ryder carroll published the bullet journal method in 20181. the book formalized his organizational method, publicly available since 20132. carroll created the bullet journal method to help him manage his own attention-deficit/hyperactivity disorder (adhd) and describes the method as “a cross between a planner, diary, notebook, to-do list, and sketchbook”1(p4). the method has a low start-up cost (requiring only a notebook of your choice and a writing instrument) and is highly flexible. 2 peer reviewed article hypothesis, vol. 37, no. 2, 2025 carroll designed the bullet journal method to be one place for all things – “you no longer have to wonder where your thoughts live”1(p18). one notebook with everything in it is a lot easier to keep track of than a smattering of sticky notes, to-do lists written on scraps of paper, meeting notes saved on your pc, project notes in one notebook, webinar notes in a different folder – and on and on. one place for everything (or most everything) lightens cognitive load – or the amount of working memory being used to keep track of the important information in the world around you3. see appendix 1 for more information, images, and examples of the method. in both physical and digital formats, task management and tracking improves productivity, time management, and organization4 as well as task completion and efficiency5,6. journaling is linked to a variety of benefits, including better sleep, a stronger immune system, and higher self-confidence7, and is widely studied in dissertations and theses8-10, knowledge syntheses11-13, and websites and blog posts14-16. a handful of studies explore the benefits of bullet journaling specifically, with results that focus on mindfulness17, the exploration of self18, and improvements to mental health19. intrinsic and extrinsic barriers do exist to journaling, however, and understanding these barriers plays an important role in the adoption of any new practice20. the weekly planning meeting the weekly planning meeting (wpm, or sometimes the “sunday meeting”) was introduced by kerry ann rockquemore, the founder of the national center for faculty development and diversity (ncfdd), in 201021. the wpm takes about 30 minutes, and is composed of three parts: skeleton, brain dump, and task assignment. this meeting can happen at either the beginning or end of a work week – but it is more important that the planning happens before the new week begins. see appendix 1 for more information on the weekly planning meeting in practice. the first step, the skeleton, helps create the foundation for the coming week. this step elucidates the time available to focus on projects and goals. the skeleton for the week is created by blocking off time in your calendar for all those items that are important or critical to work and life. items in this step include appointments, picking up the kids from daycare, classes, team meetings – anything where your absence would be noticed. next, the brain dump allows for the free-flow of information from brain-to-paper and helps make all the tasks for the coming week – and beyond – evident. this list might be long, especially in the beginning. finally, items from the brain dump are prioritized, then assigned to a chunk of time in your calendar. the semester plan the third skill in the intentionality toolkit is the semester plan, a part of the ncfdd’s core curriculum22. while the ncfdd is geared towards teaching faculty, who typically have 10-month academic contracts, a lot of their advice can be modified to a 12-month contract. 3 peer reviewed article hypothesis, vol. 37, no. 2, 2025 a semester plan taps into long-term goal setting while providing the structure to break goals down into achievable pieces. you first brainstorm the goals you would like to achieve and then select two to three goals to prioritize for the semester (or a six-month period – whatever works for you). next, you brainstorm all the steps it could potentially take to accomplish each goal. this helps clarify how long it might take to complete your goal – by breaking the goal down into discrete tasks, you make the implicit more explicit. one way to accomplish this is to frame your tasks using the smart (specific, measurable, achievable, relevant, and time-bound) goals framework, first proposed in 1981 by doran23. this method will help you craft a specific task to accomplish, versus a vague idea to pursue. for example, “revise manuscript” is vague – it is difficult to measure and see your progress. but “spend 25 minutes working on revisions for our manuscript for three days this week” is something that is specific (write for 25 minutes); measurable (three days); achievable (is doable in the time period allotted); relevant (need to revise in order to publish); and time-bound (completed this week). finally, as with the weekly planning meeting, you assign your tasks to a particular week in the semester. experience with the bullet journal method, weekly planning meetings, and semester plans these three tools are the parts of an “intentionality toolkit.” the toolkit helps me mitigate feelings of overwhelm, plan for the future, and say “no” to projects that do not align with my own goals. i feel more intentional about my days, and feel more empowered to navigate tremendous life changes, including having children, searching for and starting new jobs, planning big moves, and experiencing major health events or family crises. i’ve also published several articles, established new services for data management and for evidence synthesis, and navigated a contentious and emotionally exhausting faculty leadership role. my practice – and to be candid, therapy – has helped me keep afloat. day-to-day task management and the bullet journal method when i started my first librarian position in 2012, i wondered how to accomplish all i needed to do. i felt overwhelmed and became a regular “work-crastinator” – where i procrastinated productively by doing small, fun projects, instead of devoting time to bigger, more complex projects. i felt like i didn’t have any long-term goals or aspirations, though i knew i wanted to publish. when i took my first tenure-track job in 2014, publishing became even more pressing, but i couldn’t figure out how to make the time. i didn’t feel as though my degree prepared me to develop a research agenda, much less how to write and publish. i did not recognize that i needed help with the nuances of scaffolding my work and my career, from the granular day-to-day and week-to-week support to the large-scale semester planning to achieve my goals. i felt pulled in many directions, from moving between meetings, to the classroom, to a faculty request – sometimes with very little notice. additionally, our tenure process and criteria were opaque. i felt anxious about doing “enough” to get tenure but was never clear on what “enough” meant, nor how to get there. i took up bullet journaling in august of 2016 to start organizing my day-to-day tasks. i abandoned the practice after less than a month – in part due to an extensive and complicated 4 peer reviewed article hypothesis, vol. 37, no. 2, 2025 family health emergency that required most of my attention. when my family and i returned to our new normal around january 2017, i tried again. since then, i’ve used my bullet journal nearly daily. my first bullet journals started out as very pretty – perhaps closer to what you might imagine if you’ve come across bullet journals on the internet. i liked to include hand-drawn images to separate the months. i used a lot of color to make each week stand out. i outlined boxes to designate as spaces for reflection, doodles, tracking the weather, to-do lists, and more. as i found myself with less free time to spend with my bullet journal, i dropped the practices that were time-consuming. i spent less time optimizing and perfecting my designs and focused instead on writing down what i wanted to track. my bullet journal is my safe space for my work. i use my bullet journal to map out complex projects, brainstorm new ideas, keep track of my tasks, see my progress, and see trends in where i am spending my time (see appendix 1 for more information about my method). in the beginning, i dropped the ball on some projects, but i also learned i did not allow enough time for my tasks, and i was not prioritizing larger projects. as my practice with bullet journaling and time management has progressed, my ability to complete self-started projects has improved. one thing i deeply appreciate about the bullet journal method is its flexibility. there are no set pages – every new page is a new possibility. for example, when i have the sensation of my brain being full, i know i need to offload some information into my bullet journal. i need to pause and take some time to figure out what is causing my cognitive load to get filled up, and why it’s happening. this tends to look like a brain dump plus sorting task. i open my journal to the next new page and write down everything i think i need to accomplish. then i take a quick break. when i revisit my list, i sort each item into one of three categories – must do, could do, and want to do. each category has its own weight, and it helps me see what i “must” do in the next day or so, what i “could” do in the next week, and what i “want” to do – but can be put off to the future. this flexibility also allows me to change my practice in response to my needs for the current day, week, month, or semester. for example, because i only plan my bullet journal a day at a time, the following page is a blank canvas. this makes it a straightforward task to take notes on a meeting or a webinar on the following page. i am not constrained by a pre-printed format – i am free to use any page as i see fit. or perhaps one week i want to track my progress on overhauling a libguide: on a blank page, i can write out each step of the process and when i accomplished the task, as well as notes to myself. self-directed learning with individual courses can have a small table where i track my progress. and long-term projects, like supporting a team on an evidence synthesis project, can have two or more pages devoted to tracking our progress. this isn’t to say this information can’t live elsewhere; but it’s helpful to me to know i have it in a space that is just for me. the bullet journal method’s flexibility extends to layouts and approaches. i have experimented with design – using journals with thick paper that support watercolor and markers, as well as thin paper with minimal amounts of color. i’ve iterated continuously how i track information, especially around what information i track for my annual reports. i have 5 peer reviewed article hypothesis, vol. 37, no. 2, 2025 found it much easier to write a yearly report when i have kept track of my progress on projects in a space allotted for that purpose. the bullet journal works for me – i am not constrained by any one format. the freedom to iterate bleeds into other areas of my life and work: nothing needs to be perfect, because everything can change. this has helped reinforce a “growth mindset” – a way of thinking that embraces change, failure, and development. finally, i want to highlight that when you go looking for ideas about bullet journals, you may be overwhelmed by the artistry that goes into them. or, you may have an idea of what a bullet journal “looks” like: instagram, pinterest, and internet searches will show you highly curated bullet journals. they may be beautifully artistic or look like a scrapbook, full of well-laid out pages that take ample time (and resources) to create. this is not what i am advocating. at its core, the bullet journal method is a method of being intentional about your time. your bullet journal is for you, and you alone. there is no “right” way to keep a bullet journal. if you want to explore drawings and doodles and scrapbooking – of course you can. but you don’t have to. all you need to begin is a notebook, a writing instrument, and a willingness to try. short-term planning, saying “no” with your calendar, and weekly planning meetings in late 2017 and early 2018, i had two pivotal changes: i accepted a newly created position tasked with implementing a brand-new service at my library, which was quickly followed by the birth of my first child. while my bullet journal helped me plan one day at a time and tasks to accomplish in a particular month, i found i needed an intermediate step between thinking about my day and thinking about my month. i had a lot of uncharted territory to navigate. i needed more training. i first came across library juice academy’s working faster, working smarter, productivity strategies for librarians in july of 201824. it was one of my first introductions to breaking out of the busy-ness trap, and planted the seed for intentional, long-term planning. the course was the foundation for practices i refined through another professional development opportunity: the national center for faculty diversity and development’s (ncfdd) core curriculum25. one of the major practices i took away from the core curriculum training, and the second piece of my intentionality toolkit, is the importance of giving myself 30 minutes to plan for the coming week. this was transformative: planning my week puts me in control of my calendar and my time. the weekly planning meeting helps me understand what time i have available to do my work and helps me chip away at my goals. it also reduces the time i spend deciding on what i should do next. the result of the weekly planning meeting is a fully-booked calendar, accomplished one week at a time. this means no last-minute bookings, and i get to decide whether to accept a last-minute meeting request. this works with faculty requests, too – my time is already spoken for, so i may need to decline some requests. “no” is a muscle that i give myself ample opportunities to practice. this transforms my calendar from a passive tool to help remind me to go to meetings, to an active partner in helping me say “no” to last-minute requests. additionally, having my week fully booked means that i am more likely to pause and figure out what i say “yes” to. my weeks are already full of the work i need to accomplish as part of 6 peer reviewed article hypothesis, vol. 37, no. 2, 2025 my job and the goals i have set for myself. looking at a fully booked calendar helps remind me that, to get my work done, i need time. time itself is finite and fickle, and i can’t make more time magically appear. the weekly planning meeting has some potential pitfalls: you must be able to honor your own time, because you are the only one holding yourself accountable to this work, and you must be able to prioritize knowing we can’t do it all in the time we have available to us. humans are notoriously bad at predicting how long a task will take (known as the planning fallacy, coined in 197726,27), with humans often woefully under-estimating how long a task will take. a good rule of thumb is to multiply an initial estimate by three – so if i initially think a task will take me one hour to finish, i give myself three. i also find that i am not afraid to block out time even further in advance. a complicated evidence synthesis project i am undertaking with several librarian colleagues deserved some planning ahead of our kickoff meeting – so i set aside a couple of hours of time for four weeks ahead of the project kick off. if i missed a little bit of this time because of a more urgent matter – say i need to prep for a class or schedule meetings for a hiring committee – then i can adjust, knowing i have set aside time to set myself up for success. one way i have found helpful to honor my time commitments is to frame this blocked off time as meetings with myself. i wouldn’t cancel a meeting with a colleague just to respond to an email. i am just as important as my colleague, and a meeting with myself is just as important as a meeting with a colleague, a student, or a faculty member. prioritizing takes practice, and this practice may elicit some feelings – around worth, productivity, and more. a saying i like to keep in mind is: “if everything is a priority, nothing is a priority”28. there are many techniques on how to prioritize, like completing a structured brain dump29 or filling out an eisenhower matrix30. these techniques help you achieve clarity on where your limited time and energy should go. you may find you want some external accountability. an accountability buddy is one way to set up a supportive network, and programs like cojourn31,32 offer a structure to that support. long-term planning and the semester plan the final piece of my intentionality toolkit is the semester plan. this step, also part of the ncfdd’s core curriculum25, takes some practice. the semester plan relies on our ability to break down our big ideas into small, accomplishable tasks. all my semester planning happens in my bullet journal. i dedicate several pages to this work, because it is highly generative. see appendix 2 for an example of a semester plan. i need space to brainstorm my goals and write out the steps i will take to accomplish my goals. i balance this with a structured chart that lays out what i will accomplish each week. there are many successful ways to approach a semester plan – this might be a spreadsheet or on your whiteboard. because my bullet journal is the home for all my work, my semester plan “lives” in my bullet journal. when i noticed that i kept forgetting to devote time to my semester plan dictated tasks when setting up my week during my weekly planning meeting, i modified my practice. once i complete my semester plan, i add my tasks associated with each week of the semester into 7 peer reviewed article hypothesis, vol. 37, no. 2, 2025 my work calendar. this small modification has helped me immensely, because i’m already in my calendar with my weekly planning meeting, i can easily see what my focus for the week is. my bullet journal is my generative space; my calendar is where i hold myself accountable for my goals. another challenge is figuring out the “right size” for a task that is assigned to a week. i’ve found that it really depends on the difficulty of the task – according to what i perceive as being difficult. for example, i found a cold-call email introducing myself to an advising director really challenging to do, so to make progress on my goal of connecting with students, my goal became “email [a colleague] for feedback on email to advising director.” don’t be afraid to get granular if a task is holding you up. finally – importantly – your semester plan should be considered a guide, not a playbook. the plan won’t be perfect, but it’ll get you closer to a goal than if you didn’t have a plan at all. for example, i’ve been able to complete lengthy, self-paced trainings because of my semester plan. when a project on curriculum mapping took an unexpected pivot, i didn’t abandon my plan or feel like a failure. my plan pivoted, and i adapted my goals as needed to work towards a greater understanding of the areas i work with. discussion: benefits of using an intentionality toolkit the bullet journal method is about cracking the code of your own intentionality. note it’s not productivity – we aren’t machines and there is no way to do more with less, a way of thinking that leads to decreased motivation, stress, lower morale, poor results, and burnout33, especially among women34. you can use the time you have in your working day with intention. and because you are being intentional and documenting, you can see your progress on your projects. an intentionality toolkit can help you in numerous ways. this could include: reducing cognitive load; improving prioritization practices; improving openness and flexibility; mitigating perfectionism; owning your time; and incorporating reflective practices into your workweek. it’s also not a perfect solution and has some limitations, which are briefly discussed below. reducing cognitive load. perhaps one of the biggest benefits of the task-management aspects of bullet journaling is the reduction in cognitive load. the simple act of writing down what is on your mind frees up the space in your brain: that item no longer takes up space in your working memory35. your journal is a safe, consistent, and structured space. it is a reliable place for your brain to offload the tasks it has been carrying around. no more worrying about where that sticky note went, which file you saved that to-do list in, or if that to-do list lives in your email, on your pc, in your phone, or on the fridge. all these items are in the same place, lessening your cognitive load. improving prioritization practices. you will discover, in completing a weekly planning meeting, and likely in your semester plan, that you have more you wish to accomplish than you have time for. this is a painful discovery, and a pitfall of academia and academic work: there is always too much to do, something to perfect, another email to write. none of us can accomplish all our tasks in the working hours we have allotted each week. it’s just not 8 peer reviewed article hypothesis, vol. 37, no. 2, 2025 possible. the weekly planning meeting is regular practice in prioritization. as you continue to practice prioritizing, you’ll improve your confidence in knowing what aligns with your goals, and what might be nice to do, but isn’t possible for now. this can also help with work-life balance and offer a balm in burnout culture, though prioritization is not a solution to burnout. improving openness and flexibility (or changing course when things aren’t working). the combination of the long-term view with a semester plan and the day-to-day advancement on tasks with your bullet journal has you constantly in touch with your progress. you begin to notice trends and make adjustments as necessary. practices are modified, tweaked, or even dropped. dropping a practice is not giving up or failing. walking away from something is hard to do. the narrative that you did nothing or will lose everything by dropping a practice does a huge disservice to your time and your ability to adapt. these tiny practices in adjusting your work help you see that modifications and flexibility happen all the time and are a normal part of work and life. that doesn’t make change any less difficult – but perhaps it makes it more evident. being ok with imperfection / mitigate perfectionism. in bullet journaling, it can be tempting to try to make lines perfectly straight or figure out the “perfect” colors for this month. you may try to develop the “best” semester plan that requires no modifications. you will also be in your bullet journal nearly every day, writing down tasks, checking them off, trying again tomorrow, or crossing them off completely. these are all tiny moments working in imperfection. each piece of the toolkit is for your benefit, and none of them are set in stone or immovable. our lives can change quickly – the flexibility of the bullet journal, and a mindset that our weekly planning meetings and semester plans are guides are the foundations to flexibility. owning your time. time is a finite resource – you cannot make more time appear, and it progresses without ceasing. adopting the weekly planning meeting to block out your calendar a week in advance means you dedicate time to your own goals – you are not at the whims of others. this demonstrates to students, colleagues, faculty, and collaborators that your time is important and has value. you aren’t hoping for a block of time to magically be free – you’ve made it a priority. additionally, this helps you practice saying “no” – consider the time you block off in your calendar as meetings with yourself. you cannot be in two places at once. your time is important. your goals are worth achieving. adhering to your calendar will help you immensely. inevitably, things with a short timeline will turn up, or a collaborator reaches out unexpectedly asking for a quick turnaround. with this practice, you can thoughtfully reply with an emphatic “yes,” a “no,” or an adjustment to your calendar, knowing that you own that reply. 9 peer reviewed article hypothesis, vol. 37, no. 2, 2025 incorporating reflective practices into your work week. one component of the bullet journal method is the practice of deciding what to do with unaccomplished tasks from the previous week, also known as migration. while not formally touched on above, it’s an important piece: looking at what you accomplished, and noting what you didn’t accomplish, helps you see trends. this practice helps you get curious with yourself – was that item just not important this week? why or why not? what is stopping me from accomplishing that task? is this practice helping me or is it causing anxiety or frustration? what do you need your bullet journal to do for you? a few minutes reflecting during your weekly planning meeting can help clarify what is holding you back. it’s not perfect and limitations of the toolkit. brain dumps, calendar blocking, and semester plans can all be profoundly helpful – but they aren’t perfect. you may exist in an organization with faculty that request quick turnaround times; you may struggle to adopt a new habit; some work may come to you outside of normal business hours; you may still feel overwhelmed despite adopting these tools. you may balk at the inflexibility of the toolkit: perhaps telling a faculty member you need several weeks before you can begin working on a project feels like a reprimand waiting to happen. you may feel pressures from your supervisor, time sensitive tasks, and meetings on a short timeframe that are complicated to coordinate. but this may also beg for a change in mindset: if a project is truly a priority, say, partnering on an important grant, you can say yes knowing how to shift your other priorities. as professionals, our time dedicated to our work is important. the point here is not that everyone’s priorities are the same: the point is that your priorities are under your control and guidance. these tools won’t change your working environment, nor can you control what your colleagues do. but – they will help you establish boundaries, practice saying “no,” and they will help you think about long-term opportunities and how to achieve them. your practice will make it easier to assert your values and priorities. takeaways academia is full of possibility, so much so that we could spend all our days on projects, teaching, improving methods and workflows, and more. while much of our work requires us to adhere to deadlines or be in a specific place at a specific time – on the reference desk, in a classroom, in our own workspace – a portion of our work has no deadlines, no outside accountability, and no timeline. for this work we need to rely on other tools to give ourselves structure and accountability. the bullet journal method, the weekly planning meeting, and the semester plan are three such tools to help us look at our work through the lens of the day-to-day and the year-to-year, and to see how our days align with our priorities. changes in life can require adjustments in thinking about how time is spent. that might be major family changes, moving, personal health and wellbeing, your needs as a worker or a caregiver, or any number of big events. at some point, you may reach a point where your mind overflows with tasks, ideas, and to-do lists – and you wish for a way to organize all the stuff in your head. i hope you will experiment with one or more of the methods in the intentionality toolkit i’ve described. if you find a practice that makes your life and work better, i hope you will share it. 10 peer reviewed article hypothesis, vol. 37, no. 2, 2025 acknowledgments the author would like to thank sarah k. oelker for their assistance with developmental edits and reviewing the manuscript, suenita berube for reviewing the manuscript, and erin jerome for reviewing the manuscript. the author would also like to thank all the reviewers’ enthusiasm and encouragement for the manuscript – it wouldn’t be the same without your help. references 1. carroll r. the bullet journal method: track the past, order the present, design the future. new york, new york: portfolio/penguin; 2018. 310 p. 2. mejia z. how the creator behind the viral bullet journal turned his own life hack into a full-time business [internet]. cnbc. 2017 [cited 2024 sep 13]. available from: https://www.cnbc.com/2017/08/02/how-the-creator-behind-the-viral-bullet-journal-turned-hisown-life-hack-into-a-full-time-business.html 3. sweller j. cognitive load during problem solving: effects on learning. cognitive science [internet]. 1988 apr 1 [cited 2025 apr 2];12(2):257–85. available from: https://www.sciencedirect.com/science/article/pii/0364021388900237 4. qi c. a double-edged sword? exploring the impact of students’ academic usage of mobile devices on technostress and academic performance. behaviour & information technology [internet]. 2019 dec 2 [cited 2024 dec 4];38(12):1337–54. available from: https://doi.org/10.1080/0144929x.2019.1585476 5. stawarz k, cox al, blandford a. beyond self-tracking and reminders: designing smartphone apps that support habit formation. in: proceedings of the 33rd annual acm conference on human factors in computing systems [internet]. new york, ny, usa: association for computing machinery; 2015 [cited 2024 dec 4]. p. 2653–62. (chi ’15). available from: https://doi.org/10.1145/2702123.2702230 6. ok mw. use of ipads as assistive technology for students with disabilities. techtrends [internet]. 2018 jan 1 [cited 2024 dec 4];62(1):95–102. available from: https://doi.org/10.1007/s11528-017-0199-8 7. phelan h. what’s all this about journaling? the new york times [internet]. 2018 oct 25 [cited 2024 nov 4]; available from: https://www.nytimes.com/2018/10/25/style/journaling-benefits.html 8. swift bl. we write as dream: journaling to live, imagine, and write just worlds. proquest [internet]. 2024 [cited 2024 nov 1]. available from: https://www.proquest.com/docview/3097386512?pqorigsite=gscholar&fromopenview=true&sourcetype=dissertations%20&%20theses 9. dempsey k. elementary teachers’ use of reflection as a professional development practice: a qualitative study [internet] [ed.d.]. [united states – indiana]: american college of 11 https://www.cnbc.com/2017/08/02/how-the-creator-behind-the-viral-bullet-journal-turned-his-own-life-hack-into-a-full-time-business.html https://www.cnbc.com/2017/08/02/how-the-creator-behind-the-viral-bullet-journal-turned-his-own-life-hack-into-a-full-time-business.html https://www.sciencedirect.com/science/article/pii/0364021388900237 https://doi.org/10.1080/0144929x.2019.1585476 https://doi.org/10.1145/2702123.2702230 https://doi.org/10.1007/s11528-017-0199-8 https://www.nytimes.com/2018/10/25/style/journaling-benefits.html https://www.proquest.com/docview/3097386512?pq-origsite=gscholar&fromopenview=true&sourcetype=dissertations%20&%20theses https://www.proquest.com/docview/3097386512?pq-origsite=gscholar&fromopenview=true&sourcetype=dissertations%20&%20theses peer reviewed article hypothesis, vol. 37, no. 2, 2025 education; 2024 [cited 2024 nov 14]. available from: https://www.proquest.com/pqdtglobal/docview/2973890841/abstract/a9eb1aed4801430cpq/4 10. hodges bellinger k. the write to well-being: a mixed methods case study integrating sel into academic content through journaling to promote adolescent well-being [internet] [ph.d.]. [united states – kansas]: kansas state university; 2024 [cited 2024 nov 14]. available from: https://www.proquest.com/pqdtglobal/docview/3097678256/abstract/a9eb1aed4801430cpq/7 11. sudirman a, gemilang av, kristanto tma. harnessing the power of reflective journal writing in global contexts: a systematic literature review. international journal of learning, teaching and educational research [internet]. 2021 dec 31 [cited 2024 nov 14];20(12):174–94. available from: https://ijlter.net/index.php/ijlter/article/view/754 12. sohal m, singh p, dhillon bs, gill hs. efficacy of journaling in the management of mental illness: a systematic review and meta-analysis. family medicine and community health [internet]. 2022 mar 18 [cited 2024 nov 14];10(1):e001154. available from: https://pmc.ncbi.nlm.nih.gov/articles/pmc8935176/ 13. artioli g, deiana l, de vincenzo f, raucci m, amaducci g, bassi mc, et al. health professionals and students’ experiences of reflective writing in learning: a qualitative meta-synthesis. bmc med educ [internet]. 2021 jul 22 [cited 2024 nov 14];21(1):394. available from: https://doi.org/10.1186/s12909-021-02831-4 14. sutton j. 5 benefits of journaling for mental health [internet]. positivepsychology.com. 2018 [cited 2024 nov 1]. available from: https://positivepsychology.com/benefits-of-journaling/ 15. tabac m. the neuroscience of journaling and its benefits [internet]. clear yo mind. 2022 [cited 2024 nov 14]. available from: https://medium.com/clear-yo-mind/the-neuroscience-ofjournaling-and-its-benefits-a91218773159 16. fagain a. 10 good reasons to keep a journal [internet]. psychology today. 2023 [cited 2024 nov 14]. available from: https://www.psychologytoday.com/us/blog/modernminds/202301/10-good-reasons-to-keep-a-journal 17. ayobi a, sonne t, marshall p, cox al. flexible and mindful self-tracking: design implications from paper bullet journals. in: proceedings of the 2018 chi conference on human factors in computing systems [internet]. new york, ny, usa: association for computing machinery; 2018 [cited 2024 nov 1]. p. 1–14. (chi ’18). available from: https://dl.acm.org/doi/10.1145/3173574.3173602 18. tholander j, normark m. crafting personal information resistance, imperfection, and self-creation in bullet journaling. in: proceedings of the 2020 chi conference on human factors in computing systems [internet]. new york, ny, usa: association for computing machinery; 2020 [cited 2024 nov 1]. p. 1–13. (chi ’20). available from: https://dl.acm.org/doi/10.1145/3313831.3376410 12 https://www.proquest.com/pqdtglobal/docview/2973890841/abstract/a9eb1aed4801430cpq/4 https://www.proquest.com/pqdtglobal/docview/3097678256/abstract/a9eb1aed4801430cpq/7 https://ijlter.net/index.php/ijlter/article/view/754 https://pmc.ncbi.nlm.nih.gov/articles/pmc8935176/ https://doi.org/10.1186/s12909-021-02831-4 https://positivepsychology.com/benefits-of-journaling/ https://medium.com/clear-yo-mind/the-neuroscience-of-journaling-and-its-benefits-a91218773159 https://medium.com/clear-yo-mind/the-neuroscience-of-journaling-and-its-benefits-a91218773159 https://www.psychologytoday.com/us/blog/modern-minds/202301/10-good-reasons-to-keep-a-journal https://www.psychologytoday.com/us/blog/modern-minds/202301/10-good-reasons-to-keep-a-journal https://dl.acm.org/doi/10.1145/3173574.3173602 https://dl.acm.org/doi/10.1145/3313831.3376410 peer reviewed article hypothesis, vol. 37, no. 2, 2025 19. peach e. creative mental health literacy practices: a qualitative study exploring how students use literacy to promote wellbeing and manage mental health conditions while at university. international journal of environmental research and public health [internet]. 2023 [cited 2024 nov 1];20(15):6475. available from: https://www.mdpi.com/1660-4601/20/15/6475 20. abrouq n. factors inhibiting the development process of reflective thinking through journal writing: a literature review. journal of world englishes and educational practices [internet]. 2022 jun 22 [cited 2024 nov 14];4(2):70–9. available from: https://al-kindipublisher.com/index.php/jweep/article/view/3619 21. rockquemore ka. the sunday meeting [internet]. inside higher ed. 2010 [cited 2024 oct 1]. available from: https://www.insidehighered.com/advice/2010/09/20/sunday-meeting 22. rockquemore ka. career advice: a semester needs a plan [internet]. inside higher ed. 2010 [cited 2024 oct 1]. available from: https://web.archive.org/web/20100121081334/http://www.insidehighered.com/advice/winning/winning1 23. doran g. there’s a smart way to write management’s goals and objectives. 1981 [cited 2024 nov 13];70(11). available from: https://community.mis.temple.edu/mis0855002fall2015/files/2015/10/s.m.a.r.t-waymanagement-review.pdf 24. simpkin s. working faster, working smarter: productivity strategies for librarians. [internet]. library juice academy. 2018 [cited 2025 apr 29]. available from: https://web.archive.org/web/20181225015418/http://libraryjuiceacademy.com:80/004productivity.php 25. ncfdd. core curriculum [internet]. [cited 2024 oct 28]. available from: https://www.ncfdd.org/curriculum 26. kahneman d, tversky a. intuitive prediction: biases and corrective procedures [internet]. 1977 [cited 2024 nov 1]. available from: https://apps.dtic.mil/sti/citations/ada047747 27. buehler r, griffin d, ross m. exploring the “planning fallacy”: why people underestimate their task completion times. journal of personality and social psychology. 1994;67(3):366–81. 28. martin k. the outstanding organization: generate business results by eliminating chaos and building the foundation for everyday excellence. new york: mcgraw-hill; 2012. 217 p. 29. vardy m. 4 simple steps to brain dump for a smarter brain [internet]. lifehack. 2012 [cited 2024 dec 4]. available from: https://www.lifehack.org/articles/productivity/how-to-do-the-ultimate-brain-dump.html 30. the eisenhower matrix: introduction & 3-minute video tutorial [internet]. [cited 2024 nov 1]. available from: https://www.eisenhower.me/eisenhower-matrix/ 13 https://www.mdpi.com/1660-4601/20/15/6475 https://www.insidehighered.com/advice/2010/09/20/sunday-meeting https://web.archive.org/web/20100121081334/http://www.insidehighered.com/advice/winning/winning1 https://community.mis.temple.edu/mis0855002fall2015/files/2015/10/s.m.a.r.t-way-management-review.pdf https://community.mis.temple.edu/mis0855002fall2015/files/2015/10/s.m.a.r.t-way-management-review.pdf https://web.archive.org/web/20181225015418/http://libraryjuiceacademy.com:80/004-productivity.php https://web.archive.org/web/20181225015418/http://libraryjuiceacademy.com:80/004-productivity.php https://www.ncfdd.org/curriculum https://apps.dtic.mil/sti/citations/ada047747 https://www.lifehack.org/articles/productivity/how-to-do-the-ultimate-brain-dump.html https://www.eisenhower.me/eisenhower-matrix/ peer reviewed article hypothesis, vol. 37, no. 2, 2025 31. cojourn [internet]. cojourn. [cited 2024 dec 4]. available from: https://www.cojourn.org 32. keehn m. cojourn: harnessing the power of connection to tune into your wisdom, achieve your goals, and create the life you want. s.l.: s.n.; 2019. 380 p. 33. aspect group. overcoming the corporate struggle of “doing more with less” [internet]. aspect group. [cited 2024 nov 1]. available from: https://aspectgroup.com.au/whatsnew/overcoming-the-corporate-struggle-of-doing-more-with-less/ 34. womensmedia. since burnout is rising—who decided doing more with less was a good idea? [internet]. forbes. [cited 2024 nov 1]. available from: https://www.forbes.com/sites/womensmedia/2022/10/10/since-burnout-is-rising-who-decideddoing-more-with-less-was-a-good-idea/ 35. alós-ferrer c. lighten your (cognitive) load. [internet]. psychology today. 2024. [cited 2024 nov 1]. available from: https://www.psychologytoday.com/us/blog/decisions-and-thebrain/202406/lighten-your-cognitive-load 14 https://www.cojourn.org https://aspectgroup.com.au/whats-new/overcoming-the-corporate-struggle-of-doing-more-with-less/ https://aspectgroup.com.au/whats-new/overcoming-the-corporate-struggle-of-doing-more-with-less/ https://www.forbes.com/sites/womensmedia/2022/10/10/since-burnout-is-rising-who-decided-doing-more-with-less-was-a-good-idea/ https://www.forbes.com/sites/womensmedia/2022/10/10/since-burnout-is-rising-who-decided-doing-more-with-less-was-a-good-idea/ https://www.psychologytoday.com/us/blog/decisions-and-the-brain/202406/lighten-your-cognitive-load https://www.psychologytoday.com/us/blog/decisions-and-the-brain/202406/lighten-your-cognitive-load