Evidence Based Library and Information Practice Evidence Based Library and Information Practice 2011, 6.2 51 Evidence Based Library and Information Practice Evidence Summary PubMed is Slightly More Sensitive but More Time Intensive to Search than Ovid MEDLINE A Review of: Katchamart, W., Faulkner, A., Feldman, B., Tomlinson, G., & Bombardier, C. (2011). PubMed had a higher sensitivity than Ovid-MEDLINE in the search for systematic reviews. Journal of Clinical Epidemiology, 64(7), 805-807. doi:10.1016/j.jclinepi.2010.06.004 Reviewed by: Cari Merkley Associate Professor Library, Mount Royal University Calgary, Alberta, Canada Email: cmerkley@mtroyal.ca Received: 01 Mar. 2011 Accepted: 25 May 2011 2011 Merkley. This is an Open Access article distributed under the terms of the Creative Commons- Attribution-Noncommercial-Share Alike License 2.5 Canada (http://creativecommons.org/licenses/by-nc- sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. Abstract Objective — To compare the results of searching the MEDLINE database through Ovid and the free online version of PubMed administered by the National Library of Medicine for randomized controlled trials on the subject of the drug methotrexate (MTX) for patients suffering from rheumatoid arthritis. Design — Comparative analysis of search results. Setting — Searches conducted by researchers affiliated with Mahidol University in Bangkok, Thailand, and the University of Toronto and the University Health Network in Toronto, Ontario. Subjects — A total of 3966 search results obtained from Ovid MEDLINE and PubMed. Methods — This study employs an Ovid MEDLINE search strategy originally created for a published systematic review that identified randomized controlled trials on MTX and rheumatoid arthritis (Katchamart, Trudeau, Phumethum, & Bombardier, 2009). Two of the authors of the original systematic review (Katchamart and Bombardier) are among the authors of this current study. Appropriate medical subject heading (MeSH) terms and their synonyms were identified for the three main concepts (rheumatoid arthritis, MTX, and randomized controlled trials). The search was performed in Ovid MEDLINE, seeking articles in any language that met the mailto:cmerkley@mtroyal.ca� Evidence Based Library and Information Practice 2011, 6.2 52 search criteria, from the earliest date covered by MEDLINE to January 2009. Each MeSH or keyword term within a concept was searched separately, and then combined with other like terms using the Boolean operator OR. The searches for the three concepts were finally combined using AND. The Ovid MEDLINE search was then translated for use in PubMed by an information professional. The formatting and terminology used in some of the original Ovid MEDLINE search statements had to be changed so they would work in the new database environment, but the researchers tried to ensure that the two searches were as similar as possible. The translated search was then executed in PubMed. The final results, as well as the number of articles retrieved for each key search concept (rheumatoid arthritis, MTX, and randomized controlled trials), were then compared. The final results were further analyzed for measures of sensitivity, precision, and number needed to read. Sensitivity is calculated by the number of eligible studies found in a database divided by the “total number of eligible studies in the review” multiplied by 100 (Katchamart, Faulkner, Feldman, Tomlinson, & Bombardier, p. 806). Eligible studies were identified using the inclusion/exclusion criteria developed by Katchamart et al. The figure for “total number of eligible studies in the review” is taken from that same study, which forms the “gold standard” for this analysis (Katchamart et al., p. 806). Precision is calculated by dividing the total number of eligible citations from a database by the total number of citations returned by the database for the search multiplied by 100 (Katchamart et al., p. 806). The number needed to read (NNR) formula used by the authors is 1/precision, taken from a study by Bachman, Coray, Estermann, and Ter Riet (2002). Main Results — The PubMed search found more results than Ovid MEDLINE for each of the three key concepts – rheumatoid arthritis, MTX and randomized controlled trials. Once the three concepts were combined, PubMed found 106 more articles than Ovid MEDLINE (2036 vs. 1930). Once the review eligibility criteria were applied to the search results from PubMed, 18 eligible articles were identified, one more article than in Ovid MEDLINE. The authors indicated that the additional article located in PubMed was from a journal that was not yet indexed by MEDLINE at the time the relevant article was published. To determine database sensitivity, these numbers were then divided by 20, the total number of eligible studies located in the Katachamart et al. 2009 review, which employed tools like EMBASE and strategies like hand searching in addition to MEDLINE in order to identify relevant studies. Because of the additional study it located, the sensitivity of PubMed was determined to be slightly higher than Ovid MEDLINE (90% vs. 85%). There was little difference between the two databases in terms of precision and NNR. Precision for Ovid MEDLINE was calculated at 0.881% and at 0.884% for PubMed. The NNR was 114 for Ovid MEDLINE and 113 for PubMed. Conclusion — The authors state that while PubMed had a higher calculated sensitivity than Ovid MEDLINE in the context of this particular search because it contained content not indexed by Ovid MEDLINE that proved to be relevant for this topic, its precision and NNR were almost equal to MEDLINE’s. Some technical limitations of the PubMed interface were experienced by researchers during the study, such as periodic instability and the inability to save and modify searches and their results line by line. These same issues did not arise while using Ovid MEDLINE. The need for a skilled translation of Ovid MEDLINE searches for use in the PubMed interface was also emphasized by the authors, as differences in syntax and formatting that are not properly addressed could impact PubMed’s sensitivity and precision. Evidence Based Library and Information Practice 2011, 6.2 53 Commentary As the study opens, the authors remark on the fact that researchers are turning to PubMed because it is free. The results suggest that, in the context of this particular search, the free tool slightly outperformed the subscription database. The broader content of PubMed proved to be very significant for this topic area, and future researchers may want to compare the date coverage of the indexing of key journals in both databases because of this experience. However, the fact that PubMed returned higher results and a relevant article missed by MEDLINE is tempered by the list of the technical challenges the researchers experienced in using it. Reference to the PubMed user experience outside of the results themselves is very relevant to readers who are making decisions around the best use of their time and dollars, although the stability issues experienced by the authors may have been temporary in nature or addressed by the U.S. National Library of Medicine in subsequent updates. One wonders if the same critical eye could be applied to the OvidSP platform, even though the authors indicate that they did not experience the same challenges when using Ovid as they did with PubMed. Were there issues specific to Ovid MEDLINE that they did not encounter using PubMed, other than cost? Further research, as the authors point out, is also needed to compare PubMed’s and Ovid MEDLINE’s performance across a variety of topics, not just the one searched here. This study is inextricably linked with the systematic review which provided the authors with their search strategy and inclusion/exclusion criteria, and at times it relies heavily on that review to fill in details that are not provided here. This is problematic, particularly when figures are directly imported from the systematic review for use in calculating sensitivity or when it may be useful for readers to know the process by which citations were reviewed for relevance, something that is covered by a single sentence in this study. The inclusion/exclusion process is relevant to the PubMed and Ovid MEDLINE discussion as it feeds into measures of sensitivity, precision, and NNR. Other details glossed over here include the date coverage of the searches. One assumes that the Ovid MEDLINE search covers 1950 to January 2009 as it did in the systematic review, and that the same dates were covered in PubMed, but it is not explicitly stated. Finally, the author presented the formula for NNR, but did not explain what its value means in this particular study. For example, Bachmann et al. (2002) state that in the context of their study the “Number Needed to Read figure shows how many abstracts have to be read to identify one diagnostic study” (p. 656). How the 1/precision formula was determined for NNR is not clear in Bachmann et al. or in this study, and subsequent studies have used this same calculation but with slightly different understandings of what it means. For example, McKibbon, Wilczynski, and Haynes (2004) define NRR as “number of articles that are needed to be read to obtain one that is clinically relevant and has high-quality methods” (p. 13). It has also been proposed as an alternative to journal impact factor as an indicator of the quality of a particular journal (Toth, Muir, & Brice, 2005). The use of the existing methodology from the systematic review provides the authors with a model to compare their results to, but readers may wish to extend their reading to the original publication to get the most out of this more recent study. References Bachmann, L. M., Coray, R., Estermann, P., & Ter Riet, G. (2002). Identifying diagnostic studies in MEDLINE: Reducing the Number Needed to Read. JAMIA, 9(6), 653-658. doi: 10.1197/jamia.M1124 Katchamart, W., Trudeau, J., Phumethum, V., & Bombardier, C. (2009). Efficacy and toxicity of methotrexate (MTX) monotherapy versus MTX combination therapy with non- Evidence Based Library and Information Practice 2011, 6.2 54 biological disease-modifying antirheumatic drugs in rheumatoid arthritis: A systematic review and meta-analysis. Annals of the Rheumatic Diseases, 68, 1105- 1112. doi: 10.1136/ard.2008.099861 McKibbon, K. A., Wilczynski, N. L., & Haynes, R. B. (2004). What do evidence- based secondary journals tell us about the publication of clinically important articles in primary healthcare journals? BMC Medicine, 2(33), 1-13. doi:10.1186/1741-7015-2-33 Toth, B., Muir, J. A., & Brice, A. (2005). The number needed to read—a new measure of journal value. Health Information & Libraries Journal, 22(2), 81-82. doi:10.1111/j.1471- 1842.2005.00568.x / Evidence Based Library and Information Practice Evidence Summary