Research Section Spotlight Hypothesis, vol. 29, no. 2, Fall/Winter 2017 32 “Librarian’s Role in Reproducibility of Research” 2017 Symposium: Research Section Stipend Winner Reflection Kimberly R. Powell, MLIS Life Sciences Informationist Woodruff Health Sciences Center Library Emory University Atanta, GA krpowel@emory.edu Introduction At the 117th Annual Meeting and Exhibition of the Medical Library Association (MLA) in 2017, eight sections of the association jointly sponsored a symposium entitled Librarian’s Role in Reproducibility of Research. The four hour symposium was held Saturday, May 27th as part of the MLA pre-conference activities. Shona Kirtley, Knowledge and Information Manager for the EQUATOR Network at the Centre for Statistics in Medicine at the University of Oxford, served as the keynote speaker. Invited panelists discussed their roles in initiatives aimed to reduce the research waste caused by irreproducible reporting of scientific efforts within the published literature. The four-hour symposium concluded with a hands-on brainstorming activity that asked each of the attendees to propose and reflect on increasing the reproducibility of science. The following is a summary of the information shared and a reflection on the brainstorm suggestions made at the Librarian's Role in Reproducibility of Research Symposium. A LibGuide for the event, including agenda and speaker slides are located at http://mlasymposium.libguides.com/c.php?g=584462&p=4036194. Defining the Reproducibility Crisis In March 2012, a commentary on the reproducibility of preclinical cancer studies was published in Nature. This report by the company Amgen highlighted the disappointing success rate of translating basic science findings into clinical therapeutics, especially with regards to cancer studies. Amgen researchers conducted a review of 53 published studies finding only six (11%) of the results could be replicated [1]. This endorsed an earlier report from the pharmaceutical http://mlasymposium.libguides.com/c.php?g=584462&p=4036194 Research Section Spotlight Hypothesis, vol. 29, no. 2, Fall/Winter 2017 33 company Bayer that from a sample set of 67 published drug reports, only 20-25% were reproducible [2]. In response to these concerning observations, The Lancet launched a series of papers that discuss increasing the value and reducing research waste in the published literature, ultimately becoming The Lancet’s Reduce Research Waste and Reward Diligence (REWARD) Campaign in 2015. In 2016, Nature published survey results from 1,576 researchers, 90% of whom affirmed a reproducibility crisis within the published literature [4]. In 2015, the U.S. National Science Foundation (NSF) released recommendations authored by the Subcommittee on Replicability in Science. Among other findings, this report noted that many terms used to discuss the ability to replicate or generalize a study were applied inconsistently. The report thereby offered the following definition: “Reproducibility refers to the ability of a researcher to duplicate the results of a prior study using the same materials and procedures as were used by the original investigator (…) Reproducibility is a minimum necessary condition for a finding to be believable and informative.” [3] This differs slightly from replicability, defined by the report as “the ability of a researcher to duplicate the results of a prior study if the same procedures are followed but new data are collected” [3]. In contrast, the Nature survey of researchers asked if respondents were able to reproduce results in a “similar experimental system” which “may include slight variations in methods or materials” [4]. Moving forward, as funders, publishers, researchers, librarians, and other stakeholders work to formulate strategies to address the concern, it will be important to work from a standard definition of the problem. Many approaches would conceivably impact both reproducibility and replicability; however, when discussing the scale of the crisis, and proposing focused solutions, it will be important to note the difference. Policies: What are funders and publishers doing to support reproducible research? Proposed solutions for increasing the reproducibility (and replicability) of the published literature can typically be broken into two main approaches: those that target data reporting and those that target process reports, or methodologies. Selective data sharing was cited as one of the top factors contributing to the crises by respondents of the Nature survey. Selective reporting may occur when authors publish a clean story, leaving out factors such as replicates that did not meet expectations, outliers, or statistical tests that did not show desired Research Section Spotlight Hypothesis, vol. 29, no. 2, Fall/Winter 2017 34 results. Data sharing initiatives help address these selective pressures by requiring all underlying data of published summaries and visualizations be made available. This creates options for timely, independent verification and may limit exaggerated reporting. However to address the reproducibility crises rather than just re-use initiatives, more rigorous sharing of research design and data collection methods are also required. Funders of research have put together recommendations and guidelines in both these areas as seen by the NSF data sharing policy (https://www.nsf.gov/bfa/dias/policy/dmp.jsp) and the NIH Rigor and Reproducibility web portal (https://www.nih.gov/research-training/rigor- reproducibility). Additionally, organizations such as the Center for Open Science (COS) (https://cos.io/) in the U.S. and the Enhancing the QUAlity and Transparency Of health Research (EQUATOR) Network (http://www.equator-network.org) in the U.K. also provide resources to address the process of sharing scientific findings. These organizations provide frameworks and guidelines for conducting and reporting reproducible scientific efforts. Guidelines and toolkits for reporting common study types are linked on the EQUATOR Network, while COS provides a framework for project management support, training, and updates on ongoing projects looking at field specific reproducibility issues. Journal publishers have also responded to the call for increased reproducibility, though most have focused on data sharing strategies. Science, Nature, Public Library of Science (PLoS), and Proceedings of the National Academy of Sciences (PNAS) have all established data sharing or data accessibility policies for their authors. Authors in these journals are expected to share the data and statistics underlying their findings, preferably in a public domain repository. The sharing of such data allows for both reproducibility and confirmatory actives as well as the reuse of data for new discoveries. Journal publishers focusing on the increased rigor of published methodologies have done so primarily by exploring new publication types. The Journal of Visualized Experiments (JoVE) has been a leader in this area, developing a publication module of instructional videos aimed to better communicate experimental processes. As a panelist at the 2017 MLA Reproducibility Symposium, JoVE co-founder and CEO, Moshe Pritsker, noted that the classic structure of the journal article has remained largely unchanged since the first publication in 1665 and called emphatically for detailed methods as a stand-alone publication model [5]. Reinforcing his call, another panelist shared an example of the importance of robust materials reporting when a study found that variations in software version, workstation type (Mac or PC), and Mac https://www.nsf.gov/bfa/dias/policy/dmp.jsp https://www.nih.gov/research-training/rigor-reproducibility https://www.nih.gov/research-training/rigor-reproducibility https://cos.io/) http://www.equator-network.org/ Research Section Spotlight Hypothesis, vol. 29, no. 2, Fall/Winter 2017 35 operating system all had a significant impact on the analytic readings of 30 MRI scans [6]. When variables such as computer type and operating system are shown to play a role in reproducible data analysis, there is a clear need for more robust reporting than just product names. Registered reports offer another emerging publication type in support of reproducibility. These are detailed study designs which undergo the peer-review process separate from the final study analysis. Registered reports describe key methodologies such as experimental model, data collection instruments, and statistical methods used in the proposed analysis. These are then reviewed for rigor and reproducibility before data are collected. Currently, COS lists 52 journals which have adopted the publication of registered reports in some capacity. Journals such as Royal Society Open Science and BMC Biology will then provisionally accept results for publication, contingent upon adherence to registered study design, regardless of study outcomes. This slow shift in accepted publication types represents a positive cultural shift in the scientific community by placing emphasis on rigorous scientific processes rather than focusing solely on novel, positive results. Library Services: How can librarians support a culture of reproducible research? As Kirtley emphasizes in her 2016 The Lancet commentary, librarians are well posed to be part of the answer to managing the current reproducibility crisis [7]. Academic and medical librarians are familiar partners in the research lifecycle, from developing robust and comprehensive search strategies, to selecting a journal and assisting with data sharing and management plans. Librarians who work on systematic review projects are intimately familiar with the challenges of adapting complex methods to new database environments. By attuning specific knowledge and services to the language and needs of addressing reproducibility, librarians are equipped to serve as advocates and partners. Even traditional roles such as collection development and access training can be easily adapted to addressing reproducibility as it relates to growing the awareness, availability, and utilization of new publication types. Panelists at the 2017 MLA Reproducibility Symposium shared unique ways librarians at their institutions have contributed to creating reproducible research. Cynthia Hudson-Vitale, at the University of Washington in St. Louis, works with the Institute for Clinical and Translational Sciences to establish a framework for reproducible methods when working with electronic health records (EHRs). Librarians collaborated with researchers to identify 103 variables needed to ensure the reproducibility of EHR analysis: beginning with stating a clear, focused hypothesis Research Section Spotlight Hypothesis, vol. 29, no. 2, Fall/Winter 2017 36 through tracking query language with version notes and access dates, identifying statistical tests and packages, and reporting such specifics with standard documentation. All 103 variables would be needed in any resulting publications or data codebooks or the analyses could not be reliably replicated, regardless if the raw de-identified data is available and accessible. Bart Ragon from the University of Virginia and Kristi Holmes from Northwestern University, echoed the vital importance for librarians to offer collaborative support for open science and data management. To further explore actionable roles and services, participants at the Symposium completed a hands-on exercise in which each person was asked to propose a specific idea for librarian involvement and collaboration. Ideas were then anonymously scored for how well the proposal resonated for an individual’s library and institution. Collected from over the 30 responses, the highest scoring suggestions are listed below. While specifics of each suggestion were not discussed at the Symposium, one possible interpretation of the proposal follows each participant idea. • Host “Reproduce-My-Research” Events Such outreach or training events may take many different approaches. The idea seems to suggest giving researchers and students a formalized setting to reflect on and engage with each other specifically around how to improve the scientific reporting of their manuscripts. As this was the high scoring suggestion, it clearly resonated with many librarians who saw inspiration in this event title within their institutional outreach even without further details. • Offer training to students and early career researchers Many libraries already provide training opportunities to gain a better understanding and engage with resources, publishers, and services. Refocusing or adopting specific language to target reproducibility concerns may be an easy adaptation for services already in place. • Incorporate into the researcher workflow Again, many libraries already have collaborations or specific services targeting various aspects of the research lifecycle. Emphasizing these services as essential for addressing the reproducibility crisis may further campus collaborations and refine librarian roles. • Establish data management best practices Research Section Spotlight Hypothesis, vol. 29, no. 2, Fall/Winter 2017 37 When working with data management strategies, many initial approaches supported by libraries may have been reactive to specific funder mandates. By proactively establishing best practices in line with robust and reproducible science, librarians are natural champions of more accessible research and open data. • Educate Librarians By seeking out learning and professional networking opportunities such as the 2017 Reproducibility Symposium, librarians are educating themselves about the problem, understanding researcher frustrations and emerging policies, and collaborating as a profession to explore targeted services. Opportunities to brainstorm and share challenges and success stories are invaluable for the development of robust and innovative services. • Collaborate with institutional offices Collaboration is implied in many of the top-scoring suggestions. Continued outreach to researcher groups and campus partners will be essential to the recognition of librarians as part of the solution. • Provide high-quality, reproducible search results For those librarians working on systematic reviews or providing in depth reference support, this is an opportunity to lead by example. Providing details of the search query along with specifics such as database coverage, search date and applied filters, raises subtle awareness of the necessity of these details. This may then provide the opportunity for discussion of similar essentials when reporting results. The majority of these high scoring responses reflect activities and services already underway, with some variation, at many academic libraries. By educating librarians and raising awareness among researchers in the specific areas of reproducible research, many library services can be easily adapted to address the reproducibility crises. Hosting additional symposiums and workshops around the topic will encourage other librarians to share the specific services and outreach initiatives that have been successful. Research Section Spotlight Hypothesis, vol. 29, no. 2, Fall/Winter 2017 38 References 1. Begley CG, Ellis LM. Drug development: Raise standards for preclinical cancer research. Nature. 2012;483(7391), 531-533. 2. Prinz F, Schlange T, Asadullah K. Believe it or not: how much can we rely on published data on potential drug targets? Nat Rev Drug Discov. 2011;10(9), 712-712. 3. Bollen K, Cacioppo JT, Kaplan RM, Krosnick JA, Olds JL, Dean H. Reproducibility, replicability, and generalization in the social, behavioral, and economic sciences. 2015. Retrieved from https://www.nsf.gov/sbe/SBE_Spring_2015_AC_Meeting_Presentations/Bollen_Report_on_Re plicability_SubcommitteeMay_2015.pdf 4. Baker M. 1,500 scientists lift the lid on reproducibility. Nature. 2016;533(7604), 452-454. doi:10.1038/533452a 5. Pritsker M. The Science Reproducibility Crisis: how librarians can solve it and why they should care? Paper presented at the The Librarian's Role in Reproducibity of Research Symposium, Seattle, WA. 2017. 6. Gronenschild EHBM, Habets P, Jacobs HIL, Mengelers R, Rozendaal N, van Os J, Marcelis M. The Effects of FreeSurfer Version, Workstation Type, and Macintosh Operating System Version on Anatomical Volume and Cortical Thickness Measurements. PLOS ONE. 2012;7(6), e38234. doi:10.1371/journal.pone.0038234 7. Kirtley S. Increasing value and reducing waste in biomedical research: librarians are listening and are part of the answer. The Lancet. 2016;387(10028), 1601. doi:http://dx.doi.org/10.1016/S0140-6736(16)30241-0 --- https://www.nsf.gov/sbe/SBE_Spring_2015_AC_Meeting_Presentations/Bollen_Report_on_Replicability_SubcommitteeMay_2015.pdf https://www.nsf.gov/sbe/SBE_Spring_2015_AC_Meeting_Presentations/Bollen_Report_on_Replicability_SubcommitteeMay_2015.pdf http://dx.doi.org/10.1016/S0140-6736(16)30241-0