Evidence Based Library and Information Practice Evidence Based Library and Information Practice 2011, 6.2 55 Evidence Based Library and Information Practice Evidence Summary Translation of Hedges in Medical Databases to Other Platforms’ Syntax May Cause Significantly Different Search Results A Review of: Bradley, S. M. (2010). Examination of the clinical queries and systematic review “hedges” in EMBASE and MEDLINE. Journal of the Canadian Health Libraries Association, 31, 27-37. Reviewed by: Heather Ganshorn Health Sciences Librarian University of Calgary Calgary, AB, Canada Email: Heather.Ganshorn@ucalgary.ca Received: 28 Feb. 2011 Accepted: 21 Apr. 2011 2011 Ganshorn. This is an Open Access article distributed under the terms of the Creative Commons- Attribution-Noncommercial-Share Alike License 2.5 Canada (http://creativecommons.org/licenses/by-nc- sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. Abstract Objective – To determine whether the methodological search filters in OvidSP MEDLINE and OvidSP EMBASE also known as Clinical Queries hedges had been modified from the originals which were written by the McMaster University Health Information Research Unit Hedges Group (the Haynes Group) and whether the translations of these hedges by the National Library of Medicine used in PubMed and EBSCO MEDLINE were reliable. The hedges examined are for the clinical categories of diagnosis, therapy, etiology, prognosis, clinical prediction guides, and reviews. The author also examined the translated National Library of Medicine (NLM) Systematic Reviews hedges in OvidSP MEDLINE and EBSCO MEDLINE. Design – Validity of hedges used in various databases. Setting – OvidSP MEDLINE, OvidSP EMBASE, EBSCO MEDLINE and PubMed were studied. Subjects – The Clinical Queries hedges designed to facilitate enhanced retrieval of particular types of studies in the above- mentioned databases were compared. Methods – The author ran the Clinical Queries hedges in OvidSP MEDLINE, OvidSP EMBASE and PubMed. Next, she manually entered the original Haynes Group published hedge search strings for each clinical query in these databases, and compared the results to the Clinical Queries. The author also mailto:Heather.Ganshorn@ucalgary.ca� Evidence Based Library and Information Practice 2011, 6.2 56 compared the results obtained from the Ovid MEDLINE Clinical Queries versus the hedges in PubMed and EBSCO MEDLINE. The percentage difference in number of hits between the Ovid platform and the other platform was calculated. Where the difference was greater than 10%, the author modified the search string and re-tested it. There was no gold standard for comparison, so it was not possible to make calculations such as sensitivity, specificity, precision, or accuracy. For the testing of the Review hedges, the author used the Cochrane Database of Systematic Reviews as a gold standard to compare search results. She also compared the results in OvidSP MEDLINE to the results in EBSCO MEDLINE and PubMed. Main Results – Comparing the 27 OvidSP Clinical Queries limits to the equivalent Haynes search strings, the author found identical results, suggesting that the OvidSP hedges have not been changed from Haynes’ original search strings. However, when the OvidSP MEDLINE hedges were compared to PubMed and EBSCO, there were discrepancies. If the hedges were translated exactly, one should expect the result sets to be nearly identical, with the exception of records that had not yet been uploaded to OvidSP and EBSCO (PubMed contains records that are not yet fully indexed). However, other problems became evident. While the majority of searches yielded similar numbers of records, there were discrepancies of >10% in the number of hits for five of the Clinical Queries. Some of the hedges involved truncated search terms that, in PubMed, generated a message indicating that only the first 600 variations of the word root would be used. The author modified these hedges in order to obtain potentially more accurate results, though as she does not have a gold standard set for comparison, the modified hedges could not be thoroughly evaluated. Three of the EBSCO MEDLINE Clinical Queries hedges also generated significantly different results from OvidSP MEDLINE. The author was able to modify these hedges to generate similar results to those found in PubMed. The author’s examination of the various systematic review hedges identified other problems. For these hedges, it was possible to use the Cochrane Database of Systematic Reviews as a simple gold standard to assess the reliability of these filters. The Haynes Clinical Queries Review hedge is used in OvidSP EMBASE. The author found that this hedge’s sensitive filter retrieved 100% of the Cochrane Reviews, while the optimized filter retrieved all reviews but one. However, the specific filter retrieved only 16% of the Cochrane reviews. The author notes that the Haynes hedges were developed using a subset of journals that did not include the Cochrane Database of Systematic Reviews. The Clinical Queries Review hedge in MEDLINE appeared to have better results. In OvidSP, the sensitive and optimized hedges found all but one record, while the specific hedge found 83% of the records, a result that was mirrored in EBSCO MEDLINE and PubMed. Conclusion - Users of OvidSP MEDLINE can be confident that the Clinical Queries limits are true translations of the hedges published by Haynes et al., as they were found to give identical results to manual entry of these hedges. However, users cannot be confident that these queries will give the same results in PubMed, due to differences in syntax between the two interfaces. Users of EBSCO MEDLINE can be less confident that the Clinical Queries have been perfectly translated from the original Haynes queries, as three of these queries were found to yield significantly different results from the OvidSP MEDLINE search. The author recommends that OvidSP be the search interface of choice when using these hedges in MEDLINE. The National Library of Medicine’s (NLM) Systematic Reviews hedge has been translated into OvidSP and EBSCO, but has never been validated. The author found significant errors in this hedge in the OvidSP version, which Evidence Based Library and Information Practice 2011, 6.2 57 were rectified after she contacted Ovid. However, Ovid was reluctant to share its translation of the hedge, as this is proprietary information. The author recommends that for this reason, it is best to use PubMed to search for systematic reviews, as the search string for its hedge is publicly available. The author also notes that this issue of proprietary information is very problematic for librarians, as it makes it impossible for them to assess the hedges they are using from vendors, or to identify the source of the problem when they get unusual results. Commentary This study raises some important concerns about the reliability of hedges in the various MEDLINE and EMBASE interfaces when searching the biomedical literature. The author’s approach to comparing these hedges is methodical and easily replicated. She has included several tables comparing results across interfaces in the print version of the article, and further supplementary data is available in the form of three supplementary appendices to the online version of the article. This makes is easy to view and analyze her work. The author also found some problems with OvidSP’s subject subheadings, which were reported to Ovid and subsequently rectified, so this research has already had a demonstrable beneficial effect. This research demonstrates that the closed- access nature of most proprietary database platforms can be problematic. The author contacted Ovid when she identified problems with their translation of the NLM Systematic Reviews hedge, requesting to view their version of it, and was told that the information was “proprietary,” though they eventually agreed to provide her with the hedge, only for use with her student coursework (p. 34). She also found problems with two other subheadings, which she reported to Ovid. However, her research clearly shows that users would benefit from a more open system in which the translations of these filters could be scrutinized by librarians, and improved upon/corrected if necessary. This study raises several issues that are worth pursuing further. The author only had access to EMBASE in the OvidSP version. It would be valuable if the original Elsevier version of EMBASE could also be included in a future comparison. Since PubMed and OvidSP have both updated their platforms since this work was done, it would be difficult to replicate the author’s work, but it is likely that many of the same issues persist, or that new ones may have arisen. Regular analysis of this sort should be carried out and reported on, ideally by more than one person or organization, and according to agreed-upon guidelines. This is information that medical librarians need on an ongoing basis. One would hope that the vendors themselves would shoulder some of this work. The author notes that she modified some of the problematic hedges, and that this appeared to improve her results, but without a gold standard set of articles, it is impossible to be certain. It would be interesting to see this study expanded to include such a gold standard set, but as this was a student project, and the work of one individual, it is understandable that the author chose not to expand the project in this direction. The author also notes that “although discrepancies in the number of hits would indicate a difference in the search results, the same number of hits would not necessarily mean identical search results” (p. 31). A gold standard set would go some way toward resolving the question of how similar different vendors’ result sets are. The use of the Cochrane Database of Systematic Reviews as a gold standard set for evaluating the Systematic Reviews hedges was very informative; it is doubtful that most searchers are aware that this hedge may not find relevant Cochrane Reviews. In general, when developing or modifying search strategies for testing, the best practice would be to have another information professional review and, if necessary, suggest Evidence Based Library and Information Practice 2011, 6.2 58 revisions to one’s work. The author does not expressly indicate that this was done, though as it was a student project, and she credits a faculty member in the acknowledgments, one can infer that there was some review of the author’s work. This study is a well executed evaluation of some key tools for medical librarians, and raises many issues that merit further investigation. / Evidence Based Library and Information Practice Evidence Summary