Evidence Based Library and Information Practice Evidence Based Library and Information Practice 2011, 6.2 39 Evidence Based Library and Information Practice Evidence Summary Residents and Medical Students Correctly Answer Clinical Questions More Often with Google and UpToDate than With PubMed or Ovid MEDLINE A Review of: Thiele, R. H., Poiro, N. C., Scalzo, D. C., & Nemergut, E. C. (2010). Speed, accuracy, and confidence in Google, Ovid, PubMed, and UpToDate: Results of a randomised trial. Postgraduate Medical Journal, 86(1018), 459-465. doi:10.1136/pgmj.2010.098053 Reviewed by: Theresa Arndt Associate Director for Research & Instructional Services Waidner-Spahr Library, Dickinson College Carlisle, Pennsylvania, United States of America Email: arndtt@dickinson.edu Received: 01 Mar. 2011 Accepted: 21 Apr. 2011 2011 Arndt. This is an Open Access article distributed under the terms of the Creative Commons-Attribution- Noncommercial-Share Alike License 2.5 Canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. Abstract Objective – To determine which search tool (Google, UpToDate, PubMed or Ovid- MEDLINE) produces more accurate answers for residents, medical students, and attending physicians searching on clinical questions in anesthesiology and critical care. Searcher confidence in the answers and speed with which answers were found were also examined. Design – Randomized study without a control group. Setting – Large university medical center. Subjects –Subjects included 15 fourth year medical students (third and fourth year), 35 residents, and 4 attending physicians volunteered and completed the study. One additional attending withdrew halfway through the study. The authors were unsuccessful in recruiting an equal number of subjects from each group. Methods – A set of eight anesthesia and critical care questions was developed, based on their commonality and importance in clinical practice and their answerability. Four search tools were employed: Google, UpToDate, PubMed, and Ovid MEDLINE. In part I, subjects were given a random set of four of the questions to answer with the search tool(s) of their choice, but could use only one Evidence Based Library and Information Practice 2011, 6.2 40 search tool per question. In part II, several weeks later, the same subjects were randomly assigned a search tool with which to answer all 8 questions. The authors state that “for data analysis, PubMed was arbitrarily chosen to be the “reference standard.”” Statistical analysis was used to identify significant differences between PubMed and the other search tools. Main Results – Part I: Subjects choosing a search tool were more likely to find a correct answer with Google or UpToDate. There were no statistically significant differences in confidence with answers between any of the search tools and PubMed. Part II: Though subjects were assigned a search tool, some questions were repeated from part I. For repeated questions, Ovid users (compared to PubMed users) were significantly less likely to find the correct answer for repeated questions. Otherwise, there was no statistically significant difference in questions answered correctly. Confidence did not differ. When asked to answer new questions, subjects using Google and UpToDate were significantly more likely to find a correct answer than PubMed users. UpToDate users were more confident. There was no statistical difference in primary outcome (correct answer with high confidence) between Google, Ovid, and PubMed. Pooled data from parts I and II, removing repeated questions: Subjects using Google and UpToDate were more likely to find correct answers. Confidence was highest among UpToDate users. Average search time per question (limited to 5 minutes per question) in ascending order of time spent was: UpToDate, Google, PubMed, and Ovid. Conclusion – While the number of participants is small, the results suggest that the popular search engine Google and the commercially produced secondary online source UpToDate are more useful and efficient for finding answers to questions arising in anesthesiology and critical care practice than tools focused exclusively on indexing the primary literature. Commentary This study provides further evidence of the value of general internet and specialized secondary online sources to clinical practice. While expert searchers may chafe at the use of Google in medical settings and the reliance on digested, commercial products such as UpToDate over the “premier biomedical database” MEDLINE, it is becoming more difficult to deny the effectiveness and efficiency of the new tools. The literature review by the authors of this study places their research in context with other similar studies. While many studies have focused on the search process, this study’s authors developed a set of realistic questions for which there were available evidence-based answers, and tested the ability of clinicians who were non-expert searchers to actually find the correct answer. Less compelling was the measure of self- reported confidence in the answer found. The authors express surprise at the lack of correlation between confidence and ability to correctly answer a question, although this phenomenon has been demonstrated in many studies across various disciplines including medical specialties. User choices of tools (in part I) and expressions of confidence were also contradictory, with Google being the most popular search tool despite users expressing most confidence in UpToDate. The study methods are quite complex, with multiple statistical analyses conducted. While the authors present various bar charts, the actual number of subjects behind the percentages presented is quite small. In addition to these charts and p values, it would have been helpful for the author to present an analysis of variance (ANOVA) table to help the reader judge the significance of the statistical findings. In addition to the small number of subjects, a significant study weakness, acknowledged by Evidence Based Library and Information Practice 2011, 6.2 41 the authors, is their failure to record the final source of the answer given for each question. The search tools used in this study are not directly comparable, nor are they discrete. Google is never an end source, but rather leads the user to other sources, which can include specific PubMed records or full-text open- access primary literature, such as that archived in PubMed Central. A Google search may also link directly to content within UpToDate, a web-based subscription source which itself cites the primary literature and incorporates MEDLINE records. This does not, however, challenge the finding of Google or UpToDate as the most efficient means to find an answer to a clinical question compared to PubMed, or to the commercial MEDLINE interface Ovid, which fared poorly in this study. Studies such as this one are useful for shedding light on the ways in which clinicians actually approach searching for answers to clinical questions and the high priority that is placed on efficiency in practice settings. Such understanding calls into question the emphasis many health librarians place on traditional databases, and can help librarians improve the ways in which they support this population of clients with research tools, training, and other services relevant to their needs. / Evidence Based Library and Information Practice Evidence Summary