C&RL News May 2018 248 Asystematic review is a type of review that “seeks to systematically search for, appraise and synthesis research evidence,”1 in- cluding results published in grey literature. For decades, systematic reviews have been widely used to synthesize evidence in the health sciences. More recently, other disciplines, such as agriculture and the social sciences, have seen a rise in systematic reviews and related research methodologies. In response to this development, both Cornell University2 and the University of Minnesota Libraries3 have launched systematic review services that explicitly cater to non-health-sciences researchers at their institutions. Because it is recommended that librarians play a part on systematic review teams,4 there is a need for resources and skill development in this area. While comprehensive and expert search- ing may be part of the traditional aspects of academic librarianship, systematic reviews also require transparency and reproducibility of search methodology. This work is sup- ported by use of reporting guidelines and related librarian expertise. This guide provides resources that are useful to librarians assisting with systematic reviews in a broad range of disciplines out- side the biomedical sciences. Because the bulk of published literature on systematic reviews is concentrated in the health sciences, some resources are subject-specific in title, but have broader applications. Research network organizations • Campbell Collaboration. Founded in 2000, Campbell Collaboration is an organiza- tion that supports research synthesis in edu- cation, criminal justice, and social welfare, whereas the Cochrane Collaboration sup- ports health science research synthesis. The stated mission of Campbell includes advanc- ing “positive social and economic change through the production and use of systematic reviews and other evidence synthesis for evidence-based policy and practice.” As of early 2018, 145 systematic reviews have been published via Campbell Collaboration. The Campbell Collaboration Online Library allows for advanced searching as well as brows- ing by coordinating group (e.g., education, nutrition) to locate systematic reviews, sum- maries, and protocols. Access: https://www. campbellcollaboration.org/. Reporting guidelines • EQUATOR (Enhancing the Quality and Transparency of health Research) Network. The EQUATOR initiative is fo- cused on improving reliability of published research literature. The EQUATOR Network hosts reporting guidelines in an effort to promote transparency and accuracy in schol- arly research. As of March 2018, EQUATOR Network contained 398 searchable reporting guidelines. Librarians could discover the most Megan Kocher is science librarian, email: mkocher@ umn.edu, and Amy Riegelman is social sciences librarian, email: aspringe@umn.edu, at the University of Minnesota Libraries © 2018 Megan Kocher and Amy Riegelman Megan Kocher and Amy Riegelman Systematic reviews and evidence synthesis Resources beyond the health sciences internet resources https://www.campbellcollaboration.org/ https://www.campbellcollaboration.org/ mailto:mkocher%40umn.edu?subject= mailto:mkocher%40umn.edu?subject= mailto:aspringe%40umn.edu?subject= May 2018 249 C&RL News appropriate reporting guidelines by selecting study type (e.g., systematic reviews/meta- analyses, study protocols), discipline, report section (e.g., data, appendix), or by keyword searching. When systematic reviews/meta- analyses is selected, 31 reporting guidelines appear in the results as of March 2018. Access: http://www.equator-network.org/. • PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Anal- ysis) Statement. PRISMA Statement is an example of reporting guidelines that are discoverable in the EQUATOR Network. As noted in the name, PRISMA guidance is spe- cific to systematic reviews and meta-analysis. The PRISMA flow diagrams are visible in many published systematic reviews and meta-analyses. The 2009 PRISMA Statement evolved from a previous iteration. In 2005, several entities gathered to revise and ex- pand the QUOROM statement (published in 1999), and they relied on evidence from an international survey of review authors. The PRISMA Checklist includes 27 required parts of the title, abstract, introduction, methods, results, discussion, and funding. For example, #8 of the checklist refers to the search: “Pres- ent full electronic search strategy for at least one database, including any limits used, such that it could be repeated.” The Flow Diagram is a transparent depiction of the studies dis- covered and screened at each stage. As of March 2018, there is a plan to update and revise the PRISMA Statement. Access: http:// www.prisma-statement.org/. Registering protocols • Open Science Framework (OSF) Registries. OSF was launched by the Center for Open Science. OSF gives any researcher the option to register a project, such as a systemat- ic review protocol. R e g i s t r y of a pro- tocol on OSF cre- ates a fro- zen, time-stamped record of the protocol, thus ensuring a level of transparency and accountability for the research. There are no limits to the types of protocols that can be hosted on OSF. Access: https://osf.io/ registries/. • Prospero: International Prospec- tive Register of Systematic Reviews. This is the primary database for registering sys- tematic review protocols and searching for published protocols. Prospero accepts pro- tocols from all disciplines (e.g., psychology, nutrition) with the stipulation that they must include health-related outcomes. Access: https://www.crd.york.ac.uk/PROSPERO/. Tools • Rayyan. Rayyan is a free web-based tool developed by the Qatar Computing Research Institute. It is designed to allow collaboration on systematic review projects through the article screening process. Rayyan supports blinding during screening to reduce bias, as well as coding and tagging. Data stored in Rayyan is held in a secure cloud location and is backed up daily. Access: https://rayyan.qcri.org/welcome. • SR Toolbox. This website is a search- able catalog of tools to support the entire systematic review process. Users can search by keyword, features, discipline, approach, and cost. Content is community-driven, but is edited by Chris Marshall at University of York. Access: http://systematicreviewtools.com/. Transparency and reproducibility of search methodology • “Reproducibility of search strat- egies is poor in systematic reviews published in high-impact pediatrics, cardiology and surgery journals: A cross-sectional study.” Jonathan B. Kof- fel and Melissa L. Rethlefsen studied the reproducibility of systematic review search strategies in high-impact pediatrics, car- diology, and surgery journals. The article explains the need for transparent reporting of methods (e.g., PRISMA) and that many systematic reviews have incomplete report- http://www.equator-network.org/ http://www.prisma-statement.org/ http://www.prisma-statement.org/ https://osf.io/registries/ https://osf.io/registries/ https://www.crd.york.ac.uk/PROSPERO/ https://rayyan.qcri.org/welcome http://systematicreviewtools.com/ C&RL News May 2018 250 ing. They found that librarian involvement in systematic reviews did influence repro- ducibility. Access: https://doi.org/10.1371 /journal.pone.0163309. • “Role of Expert Searching in Health Sciences Libraries.” In this 2003 policy statement by the Medical Library Associa- tion, the significance of expert retrieval and evaluation is explained. Especially helpful, this policy statement defines expert searching and lists the skills and knowledge needed for recognized experts. Expert searching plays an important role in systematic reviews and evidence-based decision making broadly. Access: https://www.ncbi.nlm.nih.gov/pmc /articles/PMC545120/. • “Systematic Reviews and Librar- ians: A Primer for Managers. Partner- ship.” This 2015 article from The Canadian Journal of Library and Information Practice and Research by Genevieve C. Gore and Julie Jones references recommendations from various authoritative groups regarding the value of librarians contributing to sys- tematic reviews. They quote the Campbell Collaboration, Cochrane Review Group, the Agency for Healthcare Research and Quality, and other entities. For example, this paper cites Campbell Collaboration, “[r] elevant methodological expertise includes: information/library science (searching and text retrieval).” This article would be helpful when making the case for librarian involve- ment on systematic review teams. Access: https://journal.lib.uoguelph.ca/index.php /perj/article/view/3343/3505#.WfCcM- mQzIU. • “Systematic Reviews need Systemat- ic Searchers.” In this 2005 article published in the Journal of the Medical Library Associa- tion, authors Jessie McGowan and Margaret Sampson published on the methods, skills, and knowledge of expert searchers contribut- ing to systematic reviews. While this article was written for a medical librarian audience, much of it translates to librarianship of other disciplines. This article also touches on ne- gotiating authorship and refers to existing guidelines for authorship credit based on substantial contritions, intellectual content, etc. Access: https://www.ncbi.nlm.nih.gov /pmc/articles/PMC545125/. • “Use of Recommended Sear ch Strategies in Systematic Reviews and the Impact of Librarian Involvement: A Cross-Sectional Survey of Recent Authors.” This study by Jonathan Koffel published in PLOS ONE looked at the im- pact of librarian involvement on systematic review teams. A survey was disseminated to a sample of systematic review authors from the Database of Abstracts of Reviews of Effects between 2012 and 2014. Survey questions inquired about search methods and librarian involvement. The study aimed to measure recommended systematic review search methods and librarian involvement as well as “whether librarian involvement predicts the use of recommended methods.” Among other revelations, the survey results revealed that 51% of the systematic reviews had librarian involvement while only 64% of those acknowledged a librarian in the text of the paper (e.g., acknowledgements) or as a listed coauthor. Access: https://doi. org/10.1371/journal.pone.0125931. Disciplinary resources • Collaboration for Environmental Evidence (CEE). CEE is a nonprofit organiza- tion that provides standards and guidelines for conducting systematic reviews that can impact environmental policy and practice. CEE assists organizations in commissioning systematic reviews and publishes syntheses in their journal, Environmental Evidence. Access: http://www.environmentalevidence. org/. • EPPI-Centre (Evidence for Policy and Practice Information). Started in 1993 and functioning as a specialist center in the United Kingdom, EPPI-Centre is based in the Social Science Research Unit of the Depart- ment of Social Science in University College London. Goals include informing evidence- based policy and practice in systematic reviews and research use (e.g., problem solv- ing, decision making). They support projects https://doi.org/10.1371/journal.pone.0163309 https://doi.org/10.1371/journal.pone.0163309 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC545120/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC545120/ https://journal.lib.uoguelph.ca/index.php/perj/article/view/3343/3505#.WfCcM-mQzIU https://journal.lib.uoguelph.ca/index.php/perj/article/view/3343/3505#.WfCcM-mQzIU https://journal.lib.uoguelph.ca/index.php/perj/article/view/3343/3505#.WfCcM-mQzIU https://www.ncbi.nlm.nih.gov/pmc/articles/PMC545125/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC545125/ https://doi.org/10.1371/journal.pone.0125931 https://doi.org/10.1371/journal.pone.0125931 http://www.environmentalevidence.org/ http://www.environmentalevidence.org/ May 2018 251 C&RL News of various disciplines (e.g., education and social policy, developing economies), and current projects can be explored on their website. Access: https://eppi.ioe.ac.uk/. • International Initiative for Impact Evaluation (3ie). This NGO supports and funds evidence synthesis in topics related to international development with the goal of influencing policy and improving effec- tiveness. Their outputs include systematic reviews as well as briefs, impact evaluations, and evidence gap maps. Their website in- cludes a database to search for systematic reviews. Access: http://www.3ieimpact.org /en/. • What Works Clearinghouse (WWC). WWC is managed by staff of the Institute of Education Sciences of the Department of Education (IES). IES work includes statistics, research, and evaluation to “provide scientific evidence on which to ground education prac- tice and policy and to share this information in formats that are useful and accessible.”5 Similarly the stated goal of WWC is “to provide educators with the information they need to make evidence-based decisions.” Navigating the interface includes filtering by topic (e.g., mathematics, literacy). After selecting filters, you are able to see effective- ness rankings by intervention. For example, the Early Childhood Education evidence review protocol assesses which interven- tions improve school readiness, emotional development, phonological processing, etc. WCC review protocols define the scope and parameters (e.g., population, outcomes) of a systematic review, and WCC provides authors with reporting guidelines to encourage clarity and transparency. Access: https://ies.ed.gov/ ncee/wwc/. Search tools • PRESS (Peer Review of Electronic Search Strategies). The PRESS Guideline includes recommendations for evaluating electronic search methodologies. The latest guideline from 2015 is hosted by the Cana- dian Agency for Drugs and Technologies in Health. Stated goals of PRESS include to 1) “formalize the peer-review process for librarians” and 2) provide “a second set of expert eyes . . . once a draft search strategy has been developed.” Guidance from a search strategy peer reviewer could include many things, such as recommenda- tions for translating the research question into a search strategy, Boolean and proxim- ity operators, subject headings, limits and filters, etc. Access: https://www.cadth.ca /resources/finding-evidence/press. • PRESS Forum. This closed wiki-based forum is a place for librarians to request reviews of systematic review search strate- gies, and to review the searches of others. Access: http://pressforum.pbworks.com/w /session/login. • Yale MeSH Analyzer. This tool allows users to enter up to 20 PubMed ID numbers, which it uses to aggregate the metadata from the associated articles into a spreadsheet. For systematic reviews, it is useful in search strategy development to quickly aggregate the Medical Subject Heading (MeSH) terms associated with relevant articles. While it only works for PubMed, it can be useful for de- veloping searches in medical-adjacent fields, such as psychology, nutrition, and animal health. Access: http://mesh.med.yale.edu/. Sources for grey literature • Open Grey. Open Grey is a venue for discovering grey literature that is produced in Europe. The types of documents discov- erable in Open Grey include conference papers, dissertations, preprints, technical reports, etc. It’s multidisciplinary, and as of March 2018, there were more than 1 million citations encompassing the sciences and technology, social sciences, and humanities. Access: http://www.opengrey.eu/. • OSF Preprints. OSF Preprints is a discovery platform hosted by the Open Sci- ence Framework. OSF Preprints uses open source infrastructure and a public API. When grey literature is desired for a systematic review, OSF Preprints could be searched to discover preprints and other document types. OSF Preprints is able to cross-search https://eppi.ioe.ac.uk/ http://www.3ieimpact.org/en/ http://www.3ieimpact.org/en/ https://ies.ed.gov/ncee/wwc/ https://ies.ed.gov/ncee/wwc/ https://www.cadth.ca/resources/finding-evidence/press https://www.cadth.ca/resources/finding-evidence/press http://pressforum.pbworks.com/w/session/login http://pressforum.pbworks.com/w/session/login http://mesh.med.yale.edu/ http://www.opengrey.eu/ C&RL News May 2018 252 OSF partner repositories, as well as several other repositories, including arXiv, bioRxiv, Cogprints, PeerJ, Preprints.org, and others. As of March 2018, the partner repositories hosted by OSF Preprints include AgriXiv, arabixiv, BITSS, Earth ArXiv, engrXiv, FocUS Archive, Frenxiv, INA-Rxiv, LawArXiv, LIS Scholar- ship Archive, MarXiv, MindRxiv, NutriXiv, Paleorxiv, PsyArXiv, SocArxiv, SportRxiv, and Thesis Commons. OSF Preprints demands no fees for deposits or access. Access: https:// osf.io/preprints/. Notes 1. Maria J. Grant and Andrew Booth, “A typology of reviews: an analysis of 14 review types and associated methodologies,” Health Information and Libraries Journal 26, no. 2 (2009): 91-108, https://doi.org/10.1111 /j.1471-1842.2009.00848.x. 2. Cornell University Library, “Cornell University Library systematic review service,” https://www.library.cornell.edu/services /systematic-review (accessed March 29, 2018). 3. Regents of the University of Minnesota, “Systematic review,” https://www.lib.umn. edu/researchsupport/systematic-review, ac- cessed March 29, 2018. 4. Genevieve C. Gore and Julie Jones, “Systematic reviews and librarians: a primer for managers,” Partnership: The Canadian Journal of Library and Information Practice and Research 10, no. 1 (2015), http://dx.doi. org/10.21083/partnership.v10i1.3343. 5. See https://ies.ed.gov/aboutus/. The award-winning source for jobs in Library and Information Science and Technology. JOB SEEKERS • Refine your search. Filter and sort hundreds of job ads by position type, employer, location, job title, and more. • Automate. Easily create customized email job alerts on the fly. EMPLOYERS • Strengthen your candidate pool. ALA reaches the diverse, engaged professionals you want to hire. • Simplify recruitment. Flat-rate pricing, with discounted multi- ad packages available. HRDR joblist.ala.org https://osf.io/preprints/ https://osf.io/preprints/ https://doi.org/10.1111/j.1471-1842.2009.00848.x https://doi.org/10.1111/j.1471-1842.2009.00848.x https://www.library.cornell.edu/services/systematic-review https://www.library.cornell.edu/services/systematic-review https://www.lib.umn.edu/researchsupport/systematic-review https://www.lib.umn.edu/researchsupport/systematic-review https://www.lib.umn.edu/researchsupport/systematic-review http://dx.doi.org/10.21083/partnership.v10i1.3343 http://dx.doi.org/10.21083/partnership.v10i1.3343 https://ies.ed.gov/aboutus/