




































Peer Reviewed Article Hypothesis , Vol. 35, No. 2, 2023

Methods for Evaluating Database Coverage

Stacy Brodya, Rachel Brillb, Robyn Butcherc

aReference and Instructional Librarian, Himmelfarb Library, The George Washington
University, Washington, District of Columbia, https://orcid.org/0000-0002-2817-3849 ,

sbrody98@gwu.edu
bReference and Instructional Librarian, Himmelfarb Library, The George Washington

University, Washington, District of Columbia,
https://www.orcid.org/0009-0005-8517-5401 , rgbrill@gwu.edu

cResearch Information Specialist, Canadian Agency for Drugs and Technologies in
Health, Ottawa, Canada,https://www.orcid.org/0000-0003-2385-5629 ,

RobynB@cadth.ca

Purposes: In this article, we describe two methods to evaluate database content coverage,
illustrating their use with the example of evaluating databases for coverage of medical
education literature.

Method Names: Though several methods for evaluating database content coverage have
been described, the two used herein, referred to as the reference list search method and the
journal method, use traditional librarian skills and tools and can be used across a variety of
topic areas.

Description: In this paper, we describe two approaches to evaluating database coverage. For
each method, we use the example of evaluating databases for their coverage of medical
education literature. In the reference list search method, we search for references from
comprehensive reviews or bibliographies to determine their presence in databases. In the
journal method, we describe the indexing status and coverage of key publications in
databases.

Strengths: These methods use established librarian skills and tools and can be applied to a
wide variety of topics.

Limitations: The results of each method depend on the quality and comprehensiveness of
the original source material: comprehensive reviews and bibliographies for the reference list
search method and publication lists for the journal method.

Information professionals who conduct clinical evidence syntheses often refer to Cochrane for
general guidance when selecting databases for searches.1 Campbell and other organizations
provide guidance for selecting databases for other topic areas.2,3 More detailed evidence
regarding database coverage of specific topics could assist searchers more effectively than the
broad guidance currently available.

Cochrane prescribes core databases to search for reviews of medical interventions. In other
areas of research, lists of potentially relevant databases are provided, but decisions of which
and how many databases to search are left to the searcher and review team. With little
evidence available on the coverage of recommended databases and their utility in reviews,

1

https://www.orcid.org/0000-0002-2817-3849
https://orcid.org/
mailto:sbrody98@gwu.edu
https://www.orcid.org/0009-0005-8517-5401
https://orcid.org/
mailto:rgbrill@gwu.edu
https://www.orcid.org/0000-0003-2385-5629
https://orcid.org/
mailto:RobynB@cadth.ca


Peer Reviewed Article Hypothesis , Vol. 35, No. 2, 2023

decisions may feel subjective. Selecting databases is only one part of the larger framework of
decision-making processes in designing a search strategy.4 Current, relevant evidence to
support that decision could facilitate more efficient, effective searching.

Evaluating databases for comprehensiveness of domain-specific content can provide one piece
of evidence to help guide those decisions. Several methods for evaluating database
completeness have been described in the literature.5,6 We selected two methods to review
based on our prior experience evaluating database coverage.7 They are the reference list
search method, in which searches are conducted using a gold standard set of references on a
topic, and the journal method, which relies upon the indexing status of relevant journals.
These two methods rely on traditional library skills and tools and can be implemented across
a variety of subject areas. Please note that this paper only addresses methods for assessing
database coverage, not for evaluating database functionalities.8,9 To illustrate how each
method is used to evaluate database coverage, we used the topic of medical education.

Reference List Search

Overview
The reference list search method consists of five steps: selecting evidence sources, extracting
references, searching databases, calculating proportions retrieved, and comparing proportions
across databases (Figure 1). In step one, bibliographies, systematic reviews, or other evidence
sources are selected for relevance to the research topic, comprehensiveness, and quality. In
step two, references are extracted from these sources. These references represent the
“reference standard”. Next, known item searches, using title, author and keyword, or other
combinations, are conducted to determine the presence of these references in the databases of
interest. In step four, the proportion of relevant references retrieved is calculated for each
database by dividing the number of retrieved references by the total number of references in
the reference standard. In step five, the proportions are compared across databases to identify
those with greatest coverage.

Figure 1: The five steps of the reference list search method for database evaluation.

When using the reference list approach, criteria should be applied to the initial selection of
systematic reviews or bibliographies. For instance, Butcher et al. describe criteria for
selecting systematic reviews to capture references on COVID-19 interventions, diagnosis, and
prediction.7 To be eligible, a systematic review had to a) “[report] searching at least four
databases” and b) include a “minimum of five . . . studies . . . published or posted in 2020”
or, if including fewer than five studies, “all must have been published or posted in 2020”.7

More generally, systematic reviews should provide at least one full search strategy and
reference appropriate conduct and reporting guidelines. This helps readers evaluate how
comprehensive and representative the result sets of these reviews may be. As more recent
systematic reviews report following PRISMA-S, selection criteria can be updated.10

2



Peer Reviewed Article Hypothesis , Vol. 35, No. 2, 2023

When using this approach, it is important to consider the inclusion/exclusion criteria
described in evidence sources. For example, if a reference standard is created from systematic
reviews that exclude non-English language articles, conference abstracts, and qualitative
research, the database coverage evaluation will reflect those parameters. The composition of
the reference list dictates the utility of results.

Alternative approaches have been used to create the reference standard. For example,
collected works from specific evidence synthesis groups may provide a ready source for
developing a reference set. Whiting et al. drew from systematic reviews with searches
conducted by the Centre for Reviews and Dissemination at the University of York.11

Alternatively, Royle, Bain, and Waugh searched for relevant articles and consulted an expert
on their usefulness.12 With the subject matter expert, they were able to include gray
literature, research in progress, meeting abstracts, and conference proceedings. However, this
approach requires considerable time and subject matter expertise.

The results of the reference list search method can include raw numbers and proportions of
references retrieved by known item searches, per database. Results can also be presented as
references retrieved beyond those found in a core database. For instance, Butcher et al.
calculated “the number of records found in [a] database and not found in PubMed”.7

Similarly, Hartling et al. “recorded how many of the studies not indexed in Medline were
indexed in each of the additional databases” and further evaluated whether articles uniquely
found in databases other than Medline significantly impacted the results of systematic
reviews.13 For different topic areas, researchers may use databases other than MEDLINE as
the standard of comparison.

Our Experience with the Reference List Search
We applied the reference list search method to evaluate databases for their comprehensiveness
for medical education searches. To create the reference set, we used BEME (Best Evidence in
Medical Education) reviews. The BEME collaboration is a group under the auspices of
AMEE, an international association for health professions education. The BEME
Collaboration has provided guidance for conducting reviews in medical education and has
published reviews on their website. (Note: During the conduct of our research, their site
moved behind a paywall.14,15 [Published Reviews (archive.org)].)

Figure 2: Data validation and automatic formatting in Excel facilitated screening of
reviews

First, we selected relevant, comprehensive evidence sources. Following Butcher et al., we
screened reviews using the following inclusion criteria: systematic type review, medical
education focus, minimum three databases searched, and clearly reported search strategy.7

We conducted screening in Excel, using validation rules to guide data entry (Figure 2). Next,
we recorded citation information; the numbers of databases searched and studies included;

3

https://amee.org/AMEE/Publications/Publications/BEME_Guides/AMEE/Publications/BEMELanding.aspx?Category=BEME_GUIDE&hkey=315583c4-5d6a-42c9-8f66-b647bf1ef75d
https://amee.org/AMEE/Publications/Publications/BEME_Guides/AMEE/Publications/BEMELanding.aspx?Category=BEME_GUIDE&hkey=315583c4-5d6a-42c9-8f66-b647bf1ef75d
http://web.archive.org/web/20221130150407/https:/bemecollaboration.org/Published+Reviews/


Peer Reviewed Article Hypothesis , Vol. 35, No. 2, 2023

and whether (yes-no) the review was systematic, on topic, and presented clear search
strategies.

When the BEME reviews site moved behind a paywall, the supplementary materials, search
strategies and lists of included studies were not consistently posted on publisher sites
accompanying the articles. Therefore, we had to use the Internet Archive’s Wayback Machine
to locate supplementary materials originally found on the BEME reviews site.

As screening continued, we started saving supplementary materials reporting search strategies
and, when available, lists of included references to ensure we had consistent access to data.
Following screening, we planned to extract included references to a citation manager;
deduplicate; and search for references by DOI, author and title fragments, or combinations
thereof. Data would be analyzed for proportion found per database and references retrieved
beyond those found in PubMed (“additional yield”).11

At the time of this writing, we have paused screening BEME reviews. We may return to
screening, looking for supplementary materials on the Wayback Machine, as necessary.
Alternatively, we may curate a list of systematic reviews published in journals in the
Annotated Bibliography of Journals for Educational Scholarship, created by the
AAMC-Regional Groups on Educational Affairs. We encountered this resource during a
webinar for authors seeking guidance on publishing medical education research (Scholarly
Publishing Webinar Series - AAMC). While BEME has an explicit focus on systematic
reviews for medical education, other journals may include similar publications.

Limitations of the Reference List Search
It may be important to consult subject matter experts to review the list of selected sources,
as well as information professionals with evidence synthesis experience to assist in developing
selection criteria and screening evidence sources. Time may be a limiting factor. For example,
extracting and searching for items in the reference standard can take much time, depending
on the size of the set. Newer librarians and library students could be involved in this task to
distribute the workload.

When determining whether to use this method, it is important to consider the availability of
high-quality, comprehensive, relevant evidence sources. The reference standard, its size and
composition, is at the core of this method. Criteria used in the evidence sources will be
reflected in the reference set. For instance, if reviews only included randomized controlled
trials, articles published in the last ten years, and articles written in the English language,
these criteria are reflected in the composition of the reference standard. The potential
influence of these criteria should be described in reporting the results of this method.

Journal Method

Overview
The journal method for evaluating databases for completeness begins with developing a list of
journals pertinent to the topic, reviewing database documentation to determine whether those
journals are indexed in each database, and calculating and comparing proportions of relevant
journals indexed across databases (Figure 3).

4

https://www.aamc.org/media/38166/download?attachment
https://www.aamc.org/about-us/mission-areas/medical-education/scholarly-publishing-webinar-series
https://www.aamc.org/about-us/mission-areas/medical-education/scholarly-publishing-webinar-series


Peer Reviewed Article Hypothesis , Vol. 35, No. 2, 2023

Figure 3: The four steps of the journal method for database evaluation.

The success of the journal method depends on the list of journals examined in much the same
way that the success of the reference list method depends on the selection of a strong
reference standard. Various tools can be used to select journals, including subscription-based
resources like Web of Science and Ulrich’s, and freely available tools such as the Scopus
source list and CiteScore.7,12,16 In addition to subject relevance, impact factors may be
considered in building a journal list. Some disciplines may have bibliographies or core title
lists, such as the Annotated Bibliography of Journals for Educational Scholarship, that could
be used in this type of work.

Our Experience with the Journal Method
For comparison purposes, we used the same topic of medical education to illustrate how to
use the journal method. To start, we generated a list of high-impact journals in the category
“Education, Scientific Disciplines” from Journal Citation Reports (JCR). We also identified
journals commonly encountered when responding to medical education-related reference
questions. We found that not all journals with articles relevant to health sciences education
are in JCR; of those that are, not all are categorized under “Education, Scientific
Disciplines”. Some are, for instance, categorized under a specialty, e.g., pediatrics. In our
sample, only 17 of 23 (74%) relevant journals in JCR are categorized under “Education”.
This issue demonstrates that the categorization schema of tools involved in journal selection
must be considered when conducting this work.

We then collected data in Excel, using data validation tools and a README tab to guide
data entry (Figure 4). For each journal, we collected the year the publication started,
category in JCR, Journal Impact Factor (JIF), peer review information, and indexing status
in the databases of interest.

Figure 4: Excel data validation tools, for instance, input messages, as pictured above,
helped improve consistency in data entry.

We analyzed 26 journals across eight databases and found the highest number of journals
were cited in PubMed and indexed in Scopus, 26 and 24, respectively. Furthermore, 21

5

https://www.scopus.com/sources
https://www.scopus.com/sources
https://www.aamc.org/media/38166/download?attachment


Peer Reviewed Article Hypothesis , Vol. 35, No. 2, 2023

journals were indexed in MEDLINE and ten were included in PubMed Central. We noted
lower representation of health sciences education journals in ERIC (4) and Education Source
(8). No journals were uniquely indexed in ERIC and Education Source. Conversely, two
journals were found only in PubMed and PubMed Central, reinforcing the importance of
searching PubMed at the very least and the use of PubMed as a relative standard. We
concluded that, while all databases may be included for thorough education-related searches,
omitting ERIC and Education Source when time is limited is not likely to impact the
comprehensiveness of a medical education literature search.

After completing the journal analysis, we encountered additional sources for journal lists. The
Annotated Bibliography of Journals for Educational Scholarship, created by the
AAMC-Regional Groups on Educational Affairs, may have been a more comprehensive list for
this project. Any future iteration of this project could use this list.

Limitations of the Journal Method
In our experience with the journal method, the categorization schema of tools used to develop
the journal list is important to consider and report. For the topic of medical education,
relying on JCR categorizations alone would have led us to miss journals that we, in our
professional experience, understand to be relevant to this topic. Furthermore, when collecting
data on indexing status, it is important to note selective indexing, as not all databases index
all journals “cover to cover”. For instance, a database may selectively index certain topics or
publication types (e.g., ERIC).17 Overall, this method was less time-consuming than the
reference list method, provided evidence to support decision-making, and relied on traditional
librarian skills and tools.

Comparing and Contrasting the Reference List Search and Journal Methods
The journal method is appropriate when time is limited and publishing on the topic is highly
concentrated in a few key journals. When the topic is more complex and articles might be
published in a diverse range of journals, we recommend using the reference list method.

Lessons Learned & Practical Tips
The two methods described rely on established librarian skills, e.g., searching, and tools, e.g.
Excel and citation managers. Overall, we found the journal method faster and easier. The
data could also be used to assist patrons in selecting journals for publication. Developing the
initial journal list was perhaps the most challenging piece. When preparing for database
evaluations, plan for data collection and reuse. Conducting sample searches can be helpful in
anticipating which pieces of data to collect.

Data validation tips and in-sheet data entry instructions help ensure consistency in data
collection and save data collectors time in having to switch between the entry document and
an instruction sheet (Figures 2 and 4).

Finally, consider the potential for additional work. Manage data for reuse and provide
sufficient documentation to enable reuse by your team or others, for instance in bibliometrics
projects.

Conclusion

These methods can address practical challenges in selecting databases and provide evidence to
support those decisions in evidence synthesis teams. We hope these methods empower
librarians to conduct research using traditional librarian skills and resources.

6

https://www.aamc.org/media/38166/download?attachment
https://www.aamc.org/media/38166/download?attachment


Peer Reviewed Article Hypothesis , Vol. 35, No. 2, 2023

Though we have framed this article in terms of evidence synthesis searching, we invite our
colleagues to comment on its applicability for purchasing and subscriptions decisions,
identification of resources for research guides, and other practical uses.

Additional Resources for Database Evaluation Research

• Search Smart: https://www.searchsmart.org/

– “A free tool to give you the confidence to search the best databases available”

– Filter 88 databases by coverage (topic, record type), access, search functions

– See: “A free online guide to researchers’ best search options”18

• Online Databases: Advanced Search Techniques and Strategies for Graduate
Students in Music and Music Education:
https://researchguides.case.edu/c.php?g=17746&p=99778

– LibGuide by Case Western Reserve University, Kelvin Smith Library

– Describes 6 quick ways to evaluate database content and 10 items for evaluating
interfaces

• Gusenbauer M. Search where you will find most: Comparing the disciplinary coverage
of 56 bibliographic databases. Scientometrics. 2022;127(5):2683–745.19

– Gusenbauer uses a keyword approach and the 26 categories in the Scopus All
Science Journal Classification Systems to compare subject coverage across
databases

• Bibliographies and other evidence sources for reference set development

– Annotated Bibliography of Journals for Educational Scholarship

• Library Carpentries Tidy Data for Librarians: Basic quality assurance and
control

– Lesson on data validation, conditional formatting, and other Excel tools for
quality control in data entry

CredIT
Stacy Brody: Conceptualization, Formal Analysis, Data curation, Project administration,
Writing - original draft
Rachel Brill: Formal Analysis, Data curation, Writing - review & editing
Robyn Butcher: Writing - review & editing

References

1. Higgins J, Lasserson T, Chandler J, Tovey D, Thomas J, Flemyng E, et al. Methodological
expectations of Cochrane Intervention Reviews (MECIR) [Internet]. London: Cochrane; 2022
Feb [cited 2020 Dec 11]. Report No.: Version February 2022. Available from:
https://community.cochrane.org/mecir-manual
2. Kugley S, Wade A, Thomas J, Mahood Q, Jørgensen AK, Hammerstrøm K, et al.
Searching for studies: a guide to information retrieval for Campbell systematic reviews.
Campbell Syst Rev. 2017 Jan;13(1):1–73.

7

https://www.searchsmart.org/
https://researchguides.case.edu/c.php?g=17746&p=99778
https://www.aamc.org/media/38166/download?attachment
https://librarycarpentry.org/lc-spreadsheets/04-quality-control.html
https://librarycarpentry.org/lc-spreadsheets/04-quality-control.html
https://community.cochrane.org/mecir-manual


Peer Reviewed Article Hypothesis , Vol. 35, No. 2, 2023

3. Aromataris E, Munn Z. JBI manual for evidence synthesis. Adelaide, Australia: Joanna
Briggs Institute; 2020.
4. Clark JM, Beller E, Glasziou P, Sanders S. The decisions and processes involved in a
systematic search strategy: a hierarchical framework. J Med Libr Assoc [Internet]. 2021 Jul
20 [cited 2023 Jul 12];109(2). Available from:
http://jmla.pitt.edu/ojs/jmla/article/view/1086
5. Ritchie SM, Young LM, Sigman J. Comparison of Selected Bibliographic Database Subject
Overlap for Agricultural Information. Issues in Science and Technology Librarianship
[Internet]. 2018 Jun 15 [cited 2022 Dec 6];(89). Available from:
https://journals.library.ualberta.ca/istl/index.php/istl/article/view/1727
6. Tenopir C. Evaluation of database coverage: a comparison of two methodologies. Online
Rev. 1982 Jan 1;6(5):423–41.
7. Butcher R, Sampson M, Couban RJ, Malin JE, Loree S, Brody S. The currency and
completeness of specialized databases of COVID-19 publications. J Clin Epidemiol. 2022
Jul;147:52–9.
8. Gusenbauer M, Haddaway NR. Which academic search systems are suitable for systematic
reviews or meta-analyses? evaluating retrieval qualities of Google Scholar, PubMed, and 26
other resources. Res Synth Methods. 2020 Mar;11(2):181–217.
9. Bethel A, Rogers M. A checklist to assess database-hosting platforms for designing and
running searches for systematic reviews. Health Inf Libr J. 2014 Mar;31(1):43–53.
10. Rethlefsen ML, Kirtley S, Waffenschmidt S, Ayala AP, Moher D, Page MJ, et al.
PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in
Systematic Reviews. Syst Rev. 2021 Jan 26;10(1):39.
11. Whiting P, Westwood M, Burke M, Sterne J, Glanville J. Systematic reviews of test
accuracy should search a range of databases to identify primary studies. J Clin Epidemiol.
2008 Apr;61(4):357–64.
12. Royle P, Bain L, Waugh N. Systematic reviews of epidemiology in diabetes: finding the
evidence. BMC Med Res Methodol. 2005 Jan 8;5:2.
13. Hartling L, Featherstone R, Nuspl M, Shave K, Dryden DM, Vandermeer B. The
contribution of databases to the results of systematic reviews: a cross-sectional study. BMC
Med Res Methodol. 2016 Sep 26;16(1):127.
14. Haig A, Dozier M. BEME Guide No. 3: Systematic searching for evidence in medical
education–Part 2: Constructing searches. Med Teach. 2003 Jan;25(5):463–84.
15. Haig A, Dozier M. BEME Guide No 3: Systematic searching for evidence in medical
education–Part 1: Sources of information. Med Teach. 2003 Jan;25(4):352–63.
16. Elsevier. CiteScore [Internet]. 2023 [cited 2023 Jul 18]. Available from:
https://www.elsevier.com/solutions/scopus/how-scopus-works/metrics/citescore
17. Education Resources Information Center (ERIC). ERIC Selection Policy [Internet].
Washington, DC: U.S. Department of Education; 2022 May [cited 2023 Jul 19]. Available
from: https://eric.ed.gov/pdf/ERIC Selection 2022.pdf
18. Gusenbauer M. A free online guide to researchers’ best search options. Nature. 2023 Mar
23;615(7953):586–586.
19. Gusenbauer M. Search where you will find most: comparing the disciplinary coverage of
56 bibliographic databases. Scientometrics. 2022;127(5):2683–745.
20. Schaefer N, Morgan-Daniel J. Comparison of Databases for Complex Evidence Syntheses
on Education for the Health Professions. Med Ref Serv Q. 2023 Jul 3;42(3):240–59.

8

http://jmla.pitt.edu/ojs/jmla/article/view/1086
https://journals.library.ualberta.ca/istl/index.php/istl/article/view/1727
https://www.elsevier.com/solutions/scopus/how-scopus-works/metrics/citescore
https://eric.ed.gov/pdf/ERIC_Selection_2022.pdf

