Peer Reviewed Article Hypothesis , Vol. 35, No. 2, 2023 Methods for Evaluating Database Coverage Stacy Brodya, Rachel Brillb, Robyn Butcherc aReference and Instructional Librarian, Himmelfarb Library, The George Washington University, Washington, District of Columbia, https://orcid.org/0000-0002-2817-3849 , sbrody98@gwu.edu bReference and Instructional Librarian, Himmelfarb Library, The George Washington University, Washington, District of Columbia, https://www.orcid.org/0009-0005-8517-5401 , rgbrill@gwu.edu cResearch Information Specialist, Canadian Agency for Drugs and Technologies in Health, Ottawa, Canada,https://www.orcid.org/0000-0003-2385-5629 , RobynB@cadth.ca Purposes: In this article, we describe two methods to evaluate database content coverage, illustrating their use with the example of evaluating databases for coverage of medical education literature. Method Names: Though several methods for evaluating database content coverage have been described, the two used herein, referred to as the reference list search method and the journal method, use traditional librarian skills and tools and can be used across a variety of topic areas. Description: In this paper, we describe two approaches to evaluating database coverage. For each method, we use the example of evaluating databases for their coverage of medical education literature. In the reference list search method, we search for references from comprehensive reviews or bibliographies to determine their presence in databases. In the journal method, we describe the indexing status and coverage of key publications in databases. Strengths: These methods use established librarian skills and tools and can be applied to a wide variety of topics. Limitations: The results of each method depend on the quality and comprehensiveness of the original source material: comprehensive reviews and bibliographies for the reference list search method and publication lists for the journal method. Information professionals who conduct clinical evidence syntheses often refer to Cochrane for general guidance when selecting databases for searches.1 Campbell and other organizations provide guidance for selecting databases for other topic areas.2,3 More detailed evidence regarding database coverage of specific topics could assist searchers more effectively than the broad guidance currently available. Cochrane prescribes core databases to search for reviews of medical interventions. In other areas of research, lists of potentially relevant databases are provided, but decisions of which and how many databases to search are left to the searcher and review team. With little evidence available on the coverage of recommended databases and their utility in reviews, 1 https://www.orcid.org/0000-0002-2817-3849 https://orcid.org/ mailto:sbrody98@gwu.edu https://www.orcid.org/0009-0005-8517-5401 https://orcid.org/ mailto:rgbrill@gwu.edu https://www.orcid.org/0000-0003-2385-5629 https://orcid.org/ mailto:RobynB@cadth.ca Peer Reviewed Article Hypothesis , Vol. 35, No. 2, 2023 decisions may feel subjective. Selecting databases is only one part of the larger framework of decision-making processes in designing a search strategy.4 Current, relevant evidence to support that decision could facilitate more efficient, effective searching. Evaluating databases for comprehensiveness of domain-specific content can provide one piece of evidence to help guide those decisions. Several methods for evaluating database completeness have been described in the literature.5,6 We selected two methods to review based on our prior experience evaluating database coverage.7 They are the reference list search method, in which searches are conducted using a gold standard set of references on a topic, and the journal method, which relies upon the indexing status of relevant journals. These two methods rely on traditional library skills and tools and can be implemented across a variety of subject areas. Please note that this paper only addresses methods for assessing database coverage, not for evaluating database functionalities.8,9 To illustrate how each method is used to evaluate database coverage, we used the topic of medical education. Reference List Search Overview The reference list search method consists of five steps: selecting evidence sources, extracting references, searching databases, calculating proportions retrieved, and comparing proportions across databases (Figure 1). In step one, bibliographies, systematic reviews, or other evidence sources are selected for relevance to the research topic, comprehensiveness, and quality. In step two, references are extracted from these sources. These references represent the “reference standard”. Next, known item searches, using title, author and keyword, or other combinations, are conducted to determine the presence of these references in the databases of interest. In step four, the proportion of relevant references retrieved is calculated for each database by dividing the number of retrieved references by the total number of references in the reference standard. In step five, the proportions are compared across databases to identify those with greatest coverage. Figure 1: The five steps of the reference list search method for database evaluation. When using the reference list approach, criteria should be applied to the initial selection of systematic reviews or bibliographies. For instance, Butcher et al. describe criteria for selecting systematic reviews to capture references on COVID-19 interventions, diagnosis, and prediction.7 To be eligible, a systematic review had to a) “[report] searching at least four databases” and b) include a “minimum of five . . . studies . . . published or posted in 2020” or, if including fewer than five studies, “all must have been published or posted in 2020”.7 More generally, systematic reviews should provide at least one full search strategy and reference appropriate conduct and reporting guidelines. This helps readers evaluate how comprehensive and representative the result sets of these reviews may be. As more recent systematic reviews report following PRISMA-S, selection criteria can be updated.10 2 Peer Reviewed Article Hypothesis , Vol. 35, No. 2, 2023 When using this approach, it is important to consider the inclusion/exclusion criteria described in evidence sources. For example, if a reference standard is created from systematic reviews that exclude non-English language articles, conference abstracts, and qualitative research, the database coverage evaluation will reflect those parameters. The composition of the reference list dictates the utility of results. Alternative approaches have been used to create the reference standard. For example, collected works from specific evidence synthesis groups may provide a ready source for developing a reference set. Whiting et al. drew from systematic reviews with searches conducted by the Centre for Reviews and Dissemination at the University of York.11 Alternatively, Royle, Bain, and Waugh searched for relevant articles and consulted an expert on their usefulness.12 With the subject matter expert, they were able to include gray literature, research in progress, meeting abstracts, and conference proceedings. However, this approach requires considerable time and subject matter expertise. The results of the reference list search method can include raw numbers and proportions of references retrieved by known item searches, per database. Results can also be presented as references retrieved beyond those found in a core database. For instance, Butcher et al. calculated “the number of records found in [a] database and not found in PubMed”.7 Similarly, Hartling et al. “recorded how many of the studies not indexed in Medline were indexed in each of the additional databases” and further evaluated whether articles uniquely found in databases other than Medline significantly impacted the results of systematic reviews.13 For different topic areas, researchers may use databases other than MEDLINE as the standard of comparison. Our Experience with the Reference List Search We applied the reference list search method to evaluate databases for their comprehensiveness for medical education searches. To create the reference set, we used BEME (Best Evidence in Medical Education) reviews. The BEME collaboration is a group under the auspices of AMEE, an international association for health professions education. The BEME Collaboration has provided guidance for conducting reviews in medical education and has published reviews on their website. (Note: During the conduct of our research, their site moved behind a paywall.14,15 [Published Reviews (archive.org)].) Figure 2: Data validation and automatic formatting in Excel facilitated screening of reviews First, we selected relevant, comprehensive evidence sources. Following Butcher et al., we screened reviews using the following inclusion criteria: systematic type review, medical education focus, minimum three databases searched, and clearly reported search strategy.7 We conducted screening in Excel, using validation rules to guide data entry (Figure 2). Next, we recorded citation information; the numbers of databases searched and studies included; 3 https://amee.org/AMEE/Publications/Publications/BEME_Guides/AMEE/Publications/BEMELanding.aspx?Category=BEME_GUIDE&hkey=315583c4-5d6a-42c9-8f66-b647bf1ef75d https://amee.org/AMEE/Publications/Publications/BEME_Guides/AMEE/Publications/BEMELanding.aspx?Category=BEME_GUIDE&hkey=315583c4-5d6a-42c9-8f66-b647bf1ef75d http://web.archive.org/web/20221130150407/https:/bemecollaboration.org/Published+Reviews/ Peer Reviewed Article Hypothesis , Vol. 35, No. 2, 2023 and whether (yes-no) the review was systematic, on topic, and presented clear search strategies. When the BEME reviews site moved behind a paywall, the supplementary materials, search strategies and lists of included studies were not consistently posted on publisher sites accompanying the articles. Therefore, we had to use the Internet Archive’s Wayback Machine to locate supplementary materials originally found on the BEME reviews site. As screening continued, we started saving supplementary materials reporting search strategies and, when available, lists of included references to ensure we had consistent access to data. Following screening, we planned to extract included references to a citation manager; deduplicate; and search for references by DOI, author and title fragments, or combinations thereof. Data would be analyzed for proportion found per database and references retrieved beyond those found in PubMed (“additional yield”).11 At the time of this writing, we have paused screening BEME reviews. We may return to screening, looking for supplementary materials on the Wayback Machine, as necessary. Alternatively, we may curate a list of systematic reviews published in journals in the Annotated Bibliography of Journals for Educational Scholarship, created by the AAMC-Regional Groups on Educational Affairs. We encountered this resource during a webinar for authors seeking guidance on publishing medical education research (Scholarly Publishing Webinar Series - AAMC). While BEME has an explicit focus on systematic reviews for medical education, other journals may include similar publications. Limitations of the Reference List Search It may be important to consult subject matter experts to review the list of selected sources, as well as information professionals with evidence synthesis experience to assist in developing selection criteria and screening evidence sources. Time may be a limiting factor. For example, extracting and searching for items in the reference standard can take much time, depending on the size of the set. Newer librarians and library students could be involved in this task to distribute the workload. When determining whether to use this method, it is important to consider the availability of high-quality, comprehensive, relevant evidence sources. The reference standard, its size and composition, is at the core of this method. Criteria used in the evidence sources will be reflected in the reference set. For instance, if reviews only included randomized controlled trials, articles published in the last ten years, and articles written in the English language, these criteria are reflected in the composition of the reference standard. The potential influence of these criteria should be described in reporting the results of this method. Journal Method Overview The journal method for evaluating databases for completeness begins with developing a list of journals pertinent to the topic, reviewing database documentation to determine whether those journals are indexed in each database, and calculating and comparing proportions of relevant journals indexed across databases (Figure 3). 4 https://www.aamc.org/media/38166/download?attachment https://www.aamc.org/about-us/mission-areas/medical-education/scholarly-publishing-webinar-series https://www.aamc.org/about-us/mission-areas/medical-education/scholarly-publishing-webinar-series Peer Reviewed Article Hypothesis , Vol. 35, No. 2, 2023 Figure 3: The four steps of the journal method for database evaluation. The success of the journal method depends on the list of journals examined in much the same way that the success of the reference list method depends on the selection of a strong reference standard. Various tools can be used to select journals, including subscription-based resources like Web of Science and Ulrich’s, and freely available tools such as the Scopus source list and CiteScore.7,12,16 In addition to subject relevance, impact factors may be considered in building a journal list. Some disciplines may have bibliographies or core title lists, such as the Annotated Bibliography of Journals for Educational Scholarship, that could be used in this type of work. Our Experience with the Journal Method For comparison purposes, we used the same topic of medical education to illustrate how to use the journal method. To start, we generated a list of high-impact journals in the category “Education, Scientific Disciplines” from Journal Citation Reports (JCR). We also identified journals commonly encountered when responding to medical education-related reference questions. We found that not all journals with articles relevant to health sciences education are in JCR; of those that are, not all are categorized under “Education, Scientific Disciplines”. Some are, for instance, categorized under a specialty, e.g., pediatrics. In our sample, only 17 of 23 (74%) relevant journals in JCR are categorized under “Education”. This issue demonstrates that the categorization schema of tools involved in journal selection must be considered when conducting this work. We then collected data in Excel, using data validation tools and a README tab to guide data entry (Figure 4). For each journal, we collected the year the publication started, category in JCR, Journal Impact Factor (JIF), peer review information, and indexing status in the databases of interest. Figure 4: Excel data validation tools, for instance, input messages, as pictured above, helped improve consistency in data entry. We analyzed 26 journals across eight databases and found the highest number of journals were cited in PubMed and indexed in Scopus, 26 and 24, respectively. Furthermore, 21 5 https://www.scopus.com/sources https://www.scopus.com/sources https://www.aamc.org/media/38166/download?attachment Peer Reviewed Article Hypothesis , Vol. 35, No. 2, 2023 journals were indexed in MEDLINE and ten were included in PubMed Central. We noted lower representation of health sciences education journals in ERIC (4) and Education Source (8). No journals were uniquely indexed in ERIC and Education Source. Conversely, two journals were found only in PubMed and PubMed Central, reinforcing the importance of searching PubMed at the very least and the use of PubMed as a relative standard. We concluded that, while all databases may be included for thorough education-related searches, omitting ERIC and Education Source when time is limited is not likely to impact the comprehensiveness of a medical education literature search. After completing the journal analysis, we encountered additional sources for journal lists. The Annotated Bibliography of Journals for Educational Scholarship, created by the AAMC-Regional Groups on Educational Affairs, may have been a more comprehensive list for this project. Any future iteration of this project could use this list. Limitations of the Journal Method In our experience with the journal method, the categorization schema of tools used to develop the journal list is important to consider and report. For the topic of medical education, relying on JCR categorizations alone would have led us to miss journals that we, in our professional experience, understand to be relevant to this topic. Furthermore, when collecting data on indexing status, it is important to note selective indexing, as not all databases index all journals “cover to cover”. For instance, a database may selectively index certain topics or publication types (e.g., ERIC).17 Overall, this method was less time-consuming than the reference list method, provided evidence to support decision-making, and relied on traditional librarian skills and tools. Comparing and Contrasting the Reference List Search and Journal Methods The journal method is appropriate when time is limited and publishing on the topic is highly concentrated in a few key journals. When the topic is more complex and articles might be published in a diverse range of journals, we recommend using the reference list method. Lessons Learned & Practical Tips The two methods described rely on established librarian skills, e.g., searching, and tools, e.g. Excel and citation managers. Overall, we found the journal method faster and easier. The data could also be used to assist patrons in selecting journals for publication. Developing the initial journal list was perhaps the most challenging piece. When preparing for database evaluations, plan for data collection and reuse. Conducting sample searches can be helpful in anticipating which pieces of data to collect. Data validation tips and in-sheet data entry instructions help ensure consistency in data collection and save data collectors time in having to switch between the entry document and an instruction sheet (Figures 2 and 4). Finally, consider the potential for additional work. Manage data for reuse and provide sufficient documentation to enable reuse by your team or others, for instance in bibliometrics projects. Conclusion These methods can address practical challenges in selecting databases and provide evidence to support those decisions in evidence synthesis teams. We hope these methods empower librarians to conduct research using traditional librarian skills and resources. 6 https://www.aamc.org/media/38166/download?attachment https://www.aamc.org/media/38166/download?attachment Peer Reviewed Article Hypothesis , Vol. 35, No. 2, 2023 Though we have framed this article in terms of evidence synthesis searching, we invite our colleagues to comment on its applicability for purchasing and subscriptions decisions, identification of resources for research guides, and other practical uses. Additional Resources for Database Evaluation Research • Search Smart: https://www.searchsmart.org/ – “A free tool to give you the confidence to search the best databases available” – Filter 88 databases by coverage (topic, record type), access, search functions – See: “A free online guide to researchers’ best search options”18 • Online Databases: Advanced Search Techniques and Strategies for Graduate Students in Music and Music Education: https://researchguides.case.edu/c.php?g=17746&p=99778 – LibGuide by Case Western Reserve University, Kelvin Smith Library – Describes 6 quick ways to evaluate database content and 10 items for evaluating interfaces • Gusenbauer M. Search where you will find most: Comparing the disciplinary coverage of 56 bibliographic databases. Scientometrics. 2022;127(5):2683–745.19 – Gusenbauer uses a keyword approach and the 26 categories in the Scopus All Science Journal Classification Systems to compare subject coverage across databases • Bibliographies and other evidence sources for reference set development – Annotated Bibliography of Journals for Educational Scholarship • Library Carpentries Tidy Data for Librarians: Basic quality assurance and control – Lesson on data validation, conditional formatting, and other Excel tools for quality control in data entry CredIT Stacy Brody: Conceptualization, Formal Analysis, Data curation, Project administration, Writing - original draft Rachel Brill: Formal Analysis, Data curation, Writing - review & editing Robyn Butcher: Writing - review & editing References 1. Higgins J, Lasserson T, Chandler J, Tovey D, Thomas J, Flemyng E, et al. Methodological expectations of Cochrane Intervention Reviews (MECIR) [Internet]. London: Cochrane; 2022 Feb [cited 2020 Dec 11]. Report No.: Version February 2022. Available from: https://community.cochrane.org/mecir-manual 2. Kugley S, Wade A, Thomas J, Mahood Q, Jørgensen AK, Hammerstrøm K, et al. Searching for studies: a guide to information retrieval for Campbell systematic reviews. Campbell Syst Rev. 2017 Jan;13(1):1–73. 7 https://www.searchsmart.org/ https://researchguides.case.edu/c.php?g=17746&p=99778 https://www.aamc.org/media/38166/download?attachment https://librarycarpentry.org/lc-spreadsheets/04-quality-control.html https://librarycarpentry.org/lc-spreadsheets/04-quality-control.html https://community.cochrane.org/mecir-manual Peer Reviewed Article Hypothesis , Vol. 35, No. 2, 2023 3. Aromataris E, Munn Z. JBI manual for evidence synthesis. Adelaide, Australia: Joanna Briggs Institute; 2020. 4. Clark JM, Beller E, Glasziou P, Sanders S. The decisions and processes involved in a systematic search strategy: a hierarchical framework. J Med Libr Assoc [Internet]. 2021 Jul 20 [cited 2023 Jul 12];109(2). Available from: http://jmla.pitt.edu/ojs/jmla/article/view/1086 5. Ritchie SM, Young LM, Sigman J. Comparison of Selected Bibliographic Database Subject Overlap for Agricultural Information. Issues in Science and Technology Librarianship [Internet]. 2018 Jun 15 [cited 2022 Dec 6];(89). Available from: https://journals.library.ualberta.ca/istl/index.php/istl/article/view/1727 6. Tenopir C. Evaluation of database coverage: a comparison of two methodologies. Online Rev. 1982 Jan 1;6(5):423–41. 7. Butcher R, Sampson M, Couban RJ, Malin JE, Loree S, Brody S. The currency and completeness of specialized databases of COVID-19 publications. J Clin Epidemiol. 2022 Jul;147:52–9. 8. Gusenbauer M, Haddaway NR. Which academic search systems are suitable for systematic reviews or meta-analyses? evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources. Res Synth Methods. 2020 Mar;11(2):181–217. 9. Bethel A, Rogers M. A checklist to assess database-hosting platforms for designing and running searches for systematic reviews. Health Inf Libr J. 2014 Mar;31(1):43–53. 10. Rethlefsen ML, Kirtley S, Waffenschmidt S, Ayala AP, Moher D, Page MJ, et al. PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews. Syst Rev. 2021 Jan 26;10(1):39. 11. Whiting P, Westwood M, Burke M, Sterne J, Glanville J. Systematic reviews of test accuracy should search a range of databases to identify primary studies. J Clin Epidemiol. 2008 Apr;61(4):357–64. 12. Royle P, Bain L, Waugh N. Systematic reviews of epidemiology in diabetes: finding the evidence. BMC Med Res Methodol. 2005 Jan 8;5:2. 13. Hartling L, Featherstone R, Nuspl M, Shave K, Dryden DM, Vandermeer B. The contribution of databases to the results of systematic reviews: a cross-sectional study. BMC Med Res Methodol. 2016 Sep 26;16(1):127. 14. Haig A, Dozier M. BEME Guide No. 3: Systematic searching for evidence in medical education–Part 2: Constructing searches. Med Teach. 2003 Jan;25(5):463–84. 15. Haig A, Dozier M. BEME Guide No 3: Systematic searching for evidence in medical education–Part 1: Sources of information. Med Teach. 2003 Jan;25(4):352–63. 16. Elsevier. CiteScore [Internet]. 2023 [cited 2023 Jul 18]. Available from: https://www.elsevier.com/solutions/scopus/how-scopus-works/metrics/citescore 17. Education Resources Information Center (ERIC). ERIC Selection Policy [Internet]. Washington, DC: U.S. Department of Education; 2022 May [cited 2023 Jul 19]. Available from: https://eric.ed.gov/pdf/ERIC Selection 2022.pdf 18. Gusenbauer M. A free online guide to researchers’ best search options. Nature. 2023 Mar 23;615(7953):586–586. 19. Gusenbauer M. Search where you will find most: comparing the disciplinary coverage of 56 bibliographic databases. Scientometrics. 2022;127(5):2683–745. 20. Schaefer N, Morgan-Daniel J. Comparison of Databases for Complex Evidence Syntheses on Education for the Health Professions. Med Ref Serv Q. 2023 Jul 3;42(3):240–59. 8 http://jmla.pitt.edu/ojs/jmla/article/view/1086 https://journals.library.ualberta.ca/istl/index.php/istl/article/view/1727 https://www.elsevier.com/solutions/scopus/how-scopus-works/metrics/citescore https://eric.ed.gov/pdf/ERIC_Selection_2022.pdf