Evidence Based Library and Information Practice Evidence Based Library and Information Practice 2011, 6.4 72 Evidence Based Library and Information Practice Article Recording Database Searches for Systematic Reviews - What is the Value of Adding a Narrative to Peer-Review Checklists? A Case Study of NICE Interventional Procedures Guidance Jenny Craven Information Specialist National Institute for Health and Clinical Excellence Manchester, United Kingdom Email: jennifer.craven@nice.org.uk Paul Levay Information Specialist National Institute for Health and Clinical Excellence Manchester, United Kingdom Email: paul.levay@nice.org.uk Received: 25 August 2011 Accepted: 05 November 2011 2011 Craven and Levay. This is an Open Access article distributed under the terms of the Creative Commons- Attribution-Noncommercial-Share Alike License 2.5 Canada (http://creativecommons.org/licenses/by-nc- sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. Abstract This paper discusses the value of open and transparent methods for recording systematic database search strategies, showing how they have been applied at the National Institute for Health and Clinical Excellence (NICE, see Appendix C for definitions) in the United Kingdom (UK). Objective – The objectives are to: 1) Discuss the value of search strategy recording methods. 2) Assess any limitations to the practical application of a checklist approach. 3) Make recommendations for recording systematic database searches. Methods – The procedures for recording searches for Interventional Procedures Guidance at NICE were examined. A sample of current methods for recording systematic searches identified in the literature was compared to the NICE processes. The case study analyses the search conducted for evidence about an interventional procedure and shows the practical issues involved in recording the database strategies. mailto:jennifer.craven@nice.org.uk mailto:paul.levay@nice.org.uk Evidence Based Library and Information Practice 2011, 6.4 73 The case study explores why relevant papers were not retrieved by a search strategy meeting all of the criteria on the checklist used to peer review it. The evidence was required for guidance on non-rigid stabilisation techniques for the treatment of low back pain. Results – The analysis shows that amending the MEDLINE strategy to make it more sensitive would have increased its yield by 6614 articles. Examination of the search records together with correspondence between the analyst and the searcher reveals the peer reviewer had approved the search because its sensitivity was appropriate for the purpose of producing Interventional Procedures Guidance. The case study demonstrates the limitations of relying on a checklist to ensure the quality of a database search without having any contextual information. Conclusion – It is difficult for the peer reviewer to assess the subjective elements of a search without knowing why it has a particular structure or what the searcher intended. There is a risk that the peer reviewer will concentrate on the technical details, such as spelling mistakes, without having the contextual information. It is beneficial if the searcher records correspondence on key decisions and reports a summary alongside the search strategy. The narrative describes the major decisions that shaped the strategy and gives the peer reviewer an insight into the rationale for the search approach. Introduction Systematic reviews aim to provide more reliable conclusions than the individual studies that they contain by synthesising empirical evidence in a manner that minimises bias. Systematic reviews have become a standard tool in the healthcare sector, as the concept of evidence-based medicine has become increasingly popular (Bastian, Glasziou, & Chalmers, 2010). The Cochrane Collaboration, one of the leading producers of systematic reviews, states that their key characteristics include following an “explicit, reproducible methodology”. The manual for producing Cochrane Reviews goes on to show that the methodology relies on “a systematic search that attempts to identify all studies that would meet the eligibility criteria” (Higgins & Green, 2011, section 1.2.2). A rigorous search is required to produce a valid systematic review that is useful for clinicians and other decision makers. A full systematic search should employ a number of techniques for interrogating the literature, including citation searching, consulting experts and hand searching journals (Crumley, Wiebe, Cramer, Klassen, & Hartling, 2005). This case study only covers one aspect of the process, i.e, systematically searching the available electronic databases. The choice of database and search platform is an important issue which has been explored elsewhere (Booth, 2010; Younger & Boddy, 2009) and that issue is not covered here. There are two aspects to ensuring the systematic integrity of a database search. The first is to submit the search to a quality assurance process before it is run and the references downloaded. The second aspect is recording the actual search and presenting it alongside the review so that someone unfamiliar with the study can understand the methods. The purpose of this paper is to discuss the benefits of a checklist approach to peer reviewing the pre-search phase and to consider the implications of this for reporting at the post-search stage. The objectives are to assess the value of search strategy recording methods and to discuss any limitations to the Evidence Based Library and Information Practice 2011, 6.4 74 practical application of a checklist approach. Finally, it will make recommendations on recording systematic database searches. Background The case study explores the process that the United Kingdom (UK) National Institute for Health and Clinical Excellence (NICE) uses to record the searches for its Interventional Procedures Guidance (IPG). The searches are undertaken by a team of information professionals at NICE, based in the Information Services department. NICE is an independent organisation responsible for providing national guidance on the promotion of good health and the prevention and treatment of ill health (National Institute for Health and Clinical Excellence [NICE], 2010a). NICE Interventional Procedures Guidance advises the UK National Health Service (NHS) on whether interventional procedures used for diagnosis or treatment are safe enough and work well enough to be recommended for routine use (NICE, 2009a). Interventional procedures involve making cuts in the body, gaining access to body cavities or using electromagnetic radiation. More than four hundred pieces of guidance have been published since the programme was launched in 2003. Publication is one step in a longer process of encouraging the uptake of innovative technologies and procedures in clinical practice (Lourenco, Grant, Burr, & Vale, 2010). Interventional Procedures follow a series of structured steps, which take approximately nine months, to ensure that they are based on the best available evidence and meet the principles of openness and transparency (NICE, 2007; NICE, 2009a). Literature Review Literature Search Methods The case study reported in this paper originated with a literature search that aimed to understand current practice and to identify any evidence-based recommendations for recording search strategies. This was not intended to be a systematic literature review. The search, undertaken in March 2011, took an iterative approach and included a number of techniques: searching MEDLINE using the Ovid platform, the Cochrane Library using the Wiley platform, Library, Information Science and Technology Abstracts (LISTA ) using the EBSCO platform, Library and Information Science Abstracts (LISA ) using the ProQuest platform and Google Scholar; scanning the tables of contents for Health Information and Libraries Journal, Journal of the Medical Library Association and Evidence Based Library and Information Practice (EBLIP); consulting colleagues and systematic reviewers; and checking the reference lists of articles already identified. The searches combined free-text terms for systematic reviews, search strategies, search reporting, search recording, and checklists with subject headings, such as the MeSH headings Review Literature as Topic, Meta-Analysis as Topic, and Guidelines as Topic. Recording Search Strategies The purpose of recording a search strategy is to make the strategy used explicit and reproducible. The first step is to check the quality of the search and one method is to subject it to peer review. The Canadian Agency for Drugs and Technologies in Health (CADTH) investigated how consistently searches were being peer reviewed (Sampson, McGowan, Lefebvre, Moher, & Grimshaw, 2008a). The systematic review conducted for the CADTH report found that 26 tools to perform peer reviews were already in existence, although none of them had been validated against database searches (Sampson et al., 2008a, p. 31). The report identifies a number of elements that could affect a search and ranks them in three tiers according to the level of impact they can have on performance. The output from the CADTH report was a checklist called PRESS EBC: Peer Review of Electronic Search Strategies Evidence Based Checklist (PRESS) (McGowan, Sampson, & Lefebvre, 2010). This checklist can be used to assess whether a good quality search has been Evidence Based Library and Information Practice 2011, 6.4 75 performed, since the seven elements have been shown to reduce the precision or recall of a search if they have not been “skilfully executed” (McGowan et al., 2008, p. 150). Reporting Search Strategies Once the search has been peer reviewed, the references screened, and the review written, the methods must be reported to an external audience. The literature review confirmed that it is “good evidence-based practice” to record the search process used to prepare a systematic review (DeLuca, Mullins, Lyles, & Crepaz, 2008, p. 5). However, DeLuca et al. (2008) go on to argue that systematic reviews “often provide a limited explanation of the search methods used to capture the literature”, even though a poor search can have “critical implications” for the review (p. 5). Where search records do exist there is “no clear consensus regarding optimum reporting of a systematic review search methods and commonly recommended items show suboptimal reporting” (Sampson, McGowan, Tetzlaff, Cogo, & Moher, 2008b). In a more recent survey, Niederstadt and Droste (2010) conclude that “no generally accepted standard of reporting of [information retrieval] in [health technology assessment] exists”(p 450). The United States Agency for Healthcare Research and Quality has similarly concluded, in the context of comparative effectiveness research, that “better reporting and further research on search strategies is needed to develop additional evidence-based recommendations” (Relevo & Balshem, 2011, p. 1168). The value of consistent reporting procedures is highlighted by the work of Fehrmann and Thomas (2011) which shows that readers’ confidence in a review can be affected by how well the search methods have been reported. The Cochrane Handbook sets out requirements for documenting the search process (Lefebvre, Manheimer, & Glanville, 2011, sections 6.6.1-6.6.3). The handbook stresses that reviews are only reproducible if there is a record of the full search strategy that was actually run on each database. There have been several studies assessing compliance with the Cochrane search reporting standards, which are summarised in Yoshii, Plaut, McGraw, Anderson and Wellik (2009). The Centre for Reviews and Dissemination ([CRD], 2009, Appendix 3) has provided useful guidance on reporting the search process. The guidance makes a clear distinction between the very detailed search records that the information specialists should retain and the summary that should be reported in the published version (p. 249). The detailed records could still be made available as an appendix or online-only document. Guidelines for Systematic Reviews There are a number of guidelines available for reporting the entire systematic review process (Moher et al., 2011). The Preferred Reporting Items for Systematic Reviews and Meta- Analyses (PRISMA) instrument provides 27 elements to include in a systematic review report and two of these relate to searching (Liberati et al., 2009). PRISMA recommends using PRESS to check searches. The Institute of Medicine (2011) has issued guidance on conducting systematic reviews and standard 3.1 clearly states that one of the required elements for conducting a comprehensive search for evidence is to “use an independent librarian or other information specialist to peer review the search strategy” (p 84). The Appraisal of Guidelines Research and Evaluation (AGREE) Instrument can be used to assess the quality of clinical practice guidelines (AGREE Collaboration, 2001). AGREE includes a search dimension and assessors have to judge how far they agree with the statement that “systematic methods were used to search for evidence”(p 10). Systematic Reviews in Other Disciplines PRESS was developed for health technology assessments and it does not transfer directly to searches for qualitative evidence, which have Evidence Based Library and Information Practice 2011, 6.4 76 their own requirements and generally make more use of purposeful literature sampling. Booth (2006) has proposed the STARLITE checklist for qualitative searches. Reporting standards have been adapted to other disciplines and they should be appropriate to the types of evidence being used. The Social Care Institute for Excellence (Rutter, Francis, Coren, & Fisher, 2010) provides an example of procedures and guidance in a different sector. The Campbell Collaboration produces systematic reviews on education, criminal justice and social welfare topics, and its manual incorporates guidance on searching similar to Cochrane (Hammerstrøm, Wade, & Jørgensen, 2010). Limitations of the Current Recommendations A search strategy must be recorded properly before it can be peer reviewed with a checklist. A search would be rejected if any of the elements were missing, for example, if there were no indication of which database platform had been used. The current checklists are limited because they present the peer reviewer with the final search strategy without any background information. The checklists do not prompt the peer reviewer to consider how the strategy was developed, even though the final version may have changed considerably since its first draft. The various iterations could reveal important information about how the search for evidence has been conceptualised. This is not to suggest that the peer reviewer needs to see each draft of the search. The developmental process can be summarised into a narrative description that provides a rationale for the approach and is far richer than a search strategy seen in isolation. There are already some instructions for searchers to record this information. The CRD guidance encourages the searcher to record and explain any decisions made during searching (CRD, 2009, p. 21). The blank template provided in the Campbell manual encourages searchers to record the purpose of the search and advises keeping contemporaneous notes on any decisions (Hammerstrøm, Wade, & Jørgensen, 2010, pp. 31, 69). It would be helpful if these recommendations were given more prominence and if they were reported consistently to facilitate peer review. This paper describes the peer reviewing and recording of search strategies for Interventional Procedures Guidance at NICE. The case study will demonstrate the limits of peer reviewing with a checklist when the rationale has not been reported along with the search strategy. Methods A case study approach has been used to explore the process that NICE uses to record searches and to assess the benefits of the checklist approach for peer reviewing. The existing methods used to peer review searches and record search strategies identified in the literature search were compared to the processes at NICE and the Interventional Procedures programme in particular. The Information Services team has its own process documents detailing how it supports the various NICE programmes. The case study shows the similarities between the lessons learned at NICE and the experiences reported in the literature. Interventional Procedures Search Process The Information Services team at NICE works closely with analysts from the Centre for Health Technology Evaluation (CHTE) to locate the evidence for Interventional Procedures Guidance. The analyst uses the literature to write an Overview Document which forms the evidence presented to the Interventional Procedures Advisory Committee for use in its discussions. A search strategy is developed in MEDLINE, using the Ovid interface. The search might go through several iterations to ensure that it is retrieving the relevant literature. A senior member of the team peer reviews all searches against a checklist (see Appendix A), which has been tailored to the NICE process and pre- Evidence Based Library and Information Practice 2011, 6.4 77 dates PRESS by several years, although there are a number of similarities. The strategy is also run in a core set of databases, such as the Cochrane Library, Embase, and CINAHL. The MEDLINE strategy is translated into the appropriate vocabulary and syntax of the other databases and platforms. The quality of these search strategies is also assured through a process of peer review to maintain standards and minimise errors. A spreadsheet for recording the names of the searcher and the reviewer is useful for dealing with any subsequent enquiries from analysts or committee members. The process usually allows 14 days for designing the searches, peer reviewing and then actually performing the search and downloading the results. A template has been developed for recording the search information, which includes fields such as the name of the database or website searched, date searched, version used and the number of hits retrieved. Copies of all the search strategies, as they were actually run, are also retained. The template ensures that the information is recorded consistently across topics, regardless of who performed the search. The information is clearly laid out which makes it straightforward to reproduce the search if it needs to be updated at a later date. The completed templates form the appendix to the Overview Document presented to the Committee (see NICE, 2009b, Appendix C for an example). This process ensures that the committee members have both the evidence they need and an explanation of how that evidence was located. Results The case study concerned the production of Interventional Procedure Guidance number IPG 366, which was published in November 2010 (NICE, 2010b) and replaced IPG 183 from June 2006 (NICE, 2006). The final guidance document recommends that non-rigid implants between two or more vertebrae can be routinely offered as a treatment option to NHS patients with low back pain. The evidence on non-rigid implants was collected according to the process described above, with Information Services running the database searches and the analyst screening the abstracts. The evidence was collated into an Overview Document and this was presented to a committee including experts on spinal surgery. The committee members are a useful resource for identifying additional studies because they are familiar with the literature and the most up-to-date evidence in their field. Three papers identified at the Committee stage were not in the first draft of the Overview Document. The references, identified through this contact with experts, were included in the next version of the Overview Document (NICE, 2009b) and they were available when the Committee made its final recommendations. The searches met the criteria in the peer review checklist, however the three papers identified by the experts, still were not retrieved. The case study analysed why Plev and Sutcliffe (2005), Kanayama et al. (2005) and Grevitt et al. (1995) were missed by the database searches. The case study involved re- running the original search from October 2009 (reported in NICE 2009b) and then modifying it to see if it could have found the three papers. Date limits were applied to the new search, performed in July 2011, to give an approximation of the results that would have been available in October 2009. The full search strategy is reported in Appendix B. The obvious first question to ask was whether the articles were actually indexed in the databases that were searched. The Plev and Sutcliffe (2005) study is not indexed in any of the databases (MEDLINE, Embase, CINAHL and the Cochrane Library). The journal in question, Spine Journal: Official Journal of the North American Spine Society, is indexed in MEDLINE and Embase; however further investigation revealed that the article appeared in a special supplement reporting conference abstracts that had not been indexed in either database. Evidence Based Library and Information Practice 2011, 6.4 78 The initial check verified that the other two articles were available through the Ovid interface to MEDLINE [Appendix B, lines 58- 59]. The next question was whether the article was available in the database at the time of searching (Spreckelsen, Deserno, & Spitzer, 2010). The complete MEDLINE record for the Kanayama et al. (2005) paper shows that it was added to the database on 6 April 2006 and the search was conducted on 1 October 2009. Grevitt et al. (1995) was added to MEDLINE in November 1995. It has been established that two of the papers were available in MEDLINE at the time of the search. It is worth going back to the quality assurance checklist (whether the NICE or PRESS version) and retracing the peer review process. The checklist should be completed for each database to obtain the complete picture of what happened. The illustration here only considers MEDLINE because this is the key strategy used and any issues would have translated to the other databases. The spellings, syntax, line numbers, truncation, Boolean operators and proximity operators were applied correctly and the terms were all nested appropriately, in accordance with the criteria in Appendix A. A filter had been applied to remove animal studies but this was not responsible for the two papers being eliminated [lines 69-70]. The search used a combination of MeSH headings and natural language to capture relevant articles. The subject headings were exploded and appropriate sub-headings had been used. The free-text terms included synonyms and alternative spellings. The strategy in Appendix B appears to address the research question by combining terms for the procedure and condition. A closer look at how Grevitt et al. (1995) was indexed in MEDLINE reveals the problem. The article was indexed with the MeSH term low back pain and this was contained in the strategy [lines 35-36]. The search was structured [line 46] so that it retrieved articles with at least one term for the procedure and one for the condition (whether in the subject headings or free text). The problem must therefore lie with how the interventional procedure was described in the strategy. Several MeSH headings attached to the MEDLINE record appear in the search: (bone screws [line 11], lumbar vertebrae surgery [line 20], and internal fixators [line 12]). The issue is that these subject headings have been combined using a Boolean AND with the free text terms for flexible, dynamic, and non-rigid [lines 30-31]. None of the free-text terms for non-rigid appear in the title or abstract of Grevitt et al. (1995) thus the article was not found [line 61]. The same terms also account for the absence of Kanayama et al. (2005) from the final results [line 62]. This explains why the search missed the articles but what accounts for the search being constructed in this way? The reconstructed search in Appendix B shows that the two papers are retrieved if the free-text terms to describe non-rigid are removed from the strategy [lines 67-68]. The reconstructed search also shows that this line of free-text terms has a major impact on the yield; the original strategy retrieves 695 hits [line 57], whereas the version omitting the terms for non-rigid results in 7309 hits [line 66]. This means that an additional 6614 papers would have to be screened to locate Grevitt et al. (1995) and Kanayama et al. (2005). The reconstructed search shows that similar figures would have occurred at the time of the original search in October 2009 (with a difference of 6614 again between 554 [line 72] and 7168 [line 73]). These figures represent only the MEDLINE results and the number of additional papers required for screening would be much higher if the non-rigid terms were removed from the CINAHL, Embase and Cochrane database strategies. The line of non-rigid free-text terms increases the precision of the search and reduces the number of articles to be screened by over 90%. The line reduces the number of irrelevant hits on rigid stabilisation techniques that would otherwise have been screened. The changes to the search were balanced by the contact with Evidence Based Library and Information Practice 2011, 6.4 79 experts, and for this reason the other articles excluded by the line of free-text were not reviewed. The three papers confirm that contact with experts (McManus et al., 1998) and hand searching (Hopewell, Clarke, Lefebvre, & Scherer, 2007) can be useful supplements to database searches. Including a narrative explanation of the search would not have resulted in the articles being retrieved from MEDLINE, although it would have provided a clear explanation for why they had not been found. The search met the checklist criteria because the peer reviewer understood the context and the reasons for the strategy being made precise. NICE must issue timely guidance to the NHS and this affects the deadlines for reviews, the time available for screening references and the appropriate level of sensitivity in the searches. The original search was sufficiently robust for the purpose of producing Interventional Procedures Guidance. The same search might have failed the peer-review process if a different organisation, with more time available, had been undertaking the systematic review. Booth (2010) has discussed in more detail the implications of adopting the optimal approach to searching and the effects it can have on the comprehensiveness of results. The NICE template that is used to record the search strategies has since been amended to prompt the searcher to record a short narrative on the purpose and structure of the strategy. Discussion The case study illustrates the limitations of using a checklist for peer review without understanding why a strategy has a particular structure. A full peer review requires “an ascertainment that no technical errors have been made and a more subjective assessment of the adequacy of term selection” (Sampson et al., 2008a, p. 32). The CADTH report acknowledges that these two factors require different levels of knowledge to evaluate the search, with spelling mistakes a “largely mechanical” requirement, compared to the expertise required to judge how well the research question has been translated into a series of interlinked search concepts. The subjective elements are best judged when the searcher provides the contextual information that explains the structure of the search, as this case study has demonstrated. The searcher needs to retain correspondence regarding changes to the search, including the results from any test searches. The peer reviewer needs to understand the iterations that have been tried, the terms tested and removed, the reasons for terms being added, or the choice of a search filter. This background information could, for example, show that the search is sufficiently sensitive for its purpose and help the peer reviewer to avoid suggesting subject headings that have already been tested and removed from the search. It would not be feasible to report all of the communication or test results but a short paragraph summarising the decisions would be a valuable addition. The narrative could be useful for readers of the published review who need to decide whether it is a valid study. PRISMA (Liberati et al., 2009) and CRD (2009) acknowledge that the word limits in journals restrict the amount of information that can be reported but they both encourage authors to make their detailed records available, for example in online appendices. The narrative would be a useful addition to these search reports and provide the reader with the rationale for the search approach. The narrative would be an additional tool for facilitating critical appraisal and promoting transparent methods. Two searches could legitimately be structured quite differently, even if they were attempting to answer similar research questions. A NICE Interventional Procedure Guidance search has a different purpose than a Cochrane Review. The differences in the purpose of the search, the time available, and other external factors can influence the appropriate level of sensitivity in the search. The peer reviewer has to take this into account when assuring the quality of the strategy. This means that any checklists used at the pre-search stage will Evidence Based Library and Information Practice 2011, 6.4 80 need to be adapted to the needs of the organisation undertaking the search. It also means that any standardised checklists for peer reviewing published reviews, such as PRESS, will be limited if the strategy is not accompanied by a narrative justifying its structure. Conclusion The case study of Interventional Procedures Guidance at NICE has illustrated issues with the way that database searches are peer reviewed and reported. The checklist approach has been beneficial for facilitating consistency, accuracy and transparency in assessing searches. The risk with a checklist is that it focuses on the technical details of a search and quality assurance becomes something of a mechanical task. The peer reviewer performs a more effective role when contextual information is available to help judge the subjective elements of the search. The contextual information is only available if the searcher has recorded the decision-making process and presented it in a convenient format, such as a short narrative. The narrative should be recorded for the internal peer review process and it is also useful to report it to external readers of the report. The search strategy shows how the evidence was located and the narrative explains why it was done this way. A checklist for quality assurance and a narrative record of the major decisions are valuable for demonstrating that a search has been done systematically and that it followed an explicit, reproducible methodology. References AGREE Collaboration. (2001). Appraisal of guidelines for research & evaluation instrument. London: AGREE Collaboration. Retrieved 13 Nov. 2011 from http://apps.who.int/rhl/agreeinstrume ntfinal.pdf Bastian, H., Glasziou, P., & Chalmers, I. (2010). Seventy-five trials and eleven systematic reviews a day: How will we ever keep up? PLoS Medicine, 7(9), e1000326. doi:10.1371/journal.pmed.1000326 Booth, A. (2006). “Brimful of STARLITE”: Toward standards for reporting literature searches. Journal of the Medical Library Association, 94(4), 421- 429, e205. Booth, A. (2010). How much searching is enough? Comprehensive versus optimal retrieval for technology assessments. International Journal of Technology Assessment in Health Care, 26(4), 431-435. doi:10.1017/S0266462310000966 Centre for Reviews and Dissemination. (2009). Systematic reviews: CRD's guidance for undertaking reviews in health care. York: Centre for Reviews and Dissemination, University of York. Retrieved 13 Nov. 2011 from http://www.york.ac.uk/inst/crd/index_ guidance.htm Crumley, E., Wiebe, N., Cramer, K., Klassen, T., & Hartling, L. (2005). Which resources should be used to identify RCT/CCTs for systematic reviews: A systematic review. BMC Medical Research Methodology, 5, 24. doi:10.1186/1471-2288-5-24 DeLuca, J.B., Mullins, M.M., Lyles, C.M., Crepaz, N., Kay, L., & Thadiparthi, S. (2008). Developing a comprehensive search strategy for evidence based systematic reviews. Evidence Based Library and Information Practice, 3(1), 3- 32. Retrieved 13 Nov. 2011 from http://ejournals.library.ualberta.ca/ind ex.php/EBLIP/article/view/855 Fehrmann, P. & Thomas, J. (2011). Comprehensive computer searches and reporting in systematic reviews. Evidence Based Library and Information Practice 2011, 6.4 81 Research Synthesis Methods, 2(1), 15-32. doi:10.1002/jrsm.31 Grevitt, M.P., Gardner, A.D., Spilsbury, J., Shackleford, I.M., Baskerville, R., Pursell, L.M., Hassaan A.,& Mulholland, R.C. (1995). The Graf stabilisation system: Early results in 50 patients. European Spine Journal, 4(3), 169-175. doi:10.1007/BF00298241 Hammerstrøm, K., Wade, A., & Jørgensen, A.M.K. (2010). Searching for studies: A guide to information retrieval for Campbell Systematic Reviews. Oslo: Campbell Collaboration. Retrieved 13 Nov. 2011 from http://www.campbellcollaboration.org /news_/new_information_retrieval_gu ide.php Higgins, J.P.T., & Green, S. (Eds.). (2011). Cochrane handbook for systematic reviews of interventions version 5.1.0 [updated March 2011]. Oxford: The Cochrane Collaboration. Retrieved 13 Nov. 2011 from http://www.cochrane- handbook.org/ Hopewell, S., Clarke, M., Lefebvre, C., & Scherer, R. (2007). Handsearching versus electronic searching to identify reports of randomized trials. Cochrane Database of Systematic Reviews, 2007(Issue 2), MR000001. doi:10.1002/14651858.MR000001.pub2 Institute of Medicine (U.S.). Committee on Standards for Systematic Reviews of Comparative Effectiveness Research.. (2011). Finding what works in health care: Standards for systematic reviews. Washington, D.C.: National Academies Press. Retrieved 13 Nov. 2011 from http://www.iom.edu/Reports/2011/Fin ding-What-Works-in-Health-Care- Standards-for-Systematic- Reviews.aspx Kanayama, M., Hashimoto, T., Shigenobu, K., Oha, F., Ishida, T., & Yamane, S. (2005). Non-fusion surgery for degenerative spondylolisthesis using artificial ligament stabilization: Surgical indication and clinical results. Spine, 30(5), 588-592. doi:10.1097/01.brs.0000154766.74637.5e Lefebvre C., Manheimer E., & Glanville J. (2011). Chapter 6: Searching for studies. In Higgins, J.P.T., & Green, S. (Eds.), Cochrane handbook for systematic reviews of interventions version 5.1.0 [updated March2011]. Oxford: The Cochrane Collaboration. Retrieved 13 Nov. 2011 from http://www.cochrane- handbook.org/ Liberati, A., Altman, D.G., Tetzlaff, J., Mulrow, C., Gotzsche, P.C., Ioannidis, J.P.A., Clarke, M., Devereaus, P.J.,Kleiinen J., & Moher, D. (2009). The PRISMA Statement for reporting systematic reviews and meta-analysis of studies that evaluate health care interventions: Explanation and elaboration. PLoS Medicine, 6(7) e1000100. doi:10.1371/journal.pmed.1000100 Lourenco, T., Grant A.M., Burr, J.M., & Vale, L. (2011). The introduction of new interventional procedures in the British National Health Service - A qualitative study. Health Policy, 100, 35-42. doi:10.1016/j.healthpol.2010.08.012 McGowan, J., Sampson, M., & Lefebvre, C. (2010). An evidence based checklist for the peer review of electronic search strategies (PRESS EBC). Evidence Based Library and Information Practice, 5(1), 149-154. Retrieved 13 Nov. 2011 from http://ejournals.library.ualberta.ca/ind ex.php/EBLIP/article/view/7402 McManus, R.J., Wilson, S., Delaney, B.C., Fitzmaurice, D.A., Hyde, C.J., Tobias, R.S., Jowett, S,& Hobbs, F.D.R. (1998). Evidence Based Library and Information Practice 2011, 6.4 82 Review of the usefulness of contacting other experts when conducting a literature search for systematic reviews. BMJ, 317(7172), 1562-1563. Moher D., Weeks, L., Ocampo, M., Seely, D., Sampson, M., Altman, D.G., Schultz, K.F., Miller, D., Simera, I., Grimshaw, J, & Hoey, J. (2011). Describing reporting guidelines for health research: A systematic review. Journal of Clinical Epidemiology, 64(7), 718-742. doi:10.1016/j.clinepi.2010.09.013 National Institute for Health and Clinical Excellence. (2006). Non-rigid stabilisation techniques for the treatment of low back pain (Interventional Procedures Guidance IPG183). London: National Institute for Health and Clinical Excellence. Retrieved 13 Nov. 2011 from http://www.nice.org.uk/nicemedia/pd f/IPG183guidance.pdf National Institute for Health and Clinical Excellence. (2007). Interventional procedures programme methods guide. London: National Institute for Health and Clinical Excellence. Retrieved 13 Nov. 2011 from http://www.nice.org.uk/aboutnice/ho wwework/developingniceintervention alprocedures/interventionalprocedure sprogrammemanual/interventionalpro ceduresprogrammemanual.jsp?domed ia=1&mid=70C7F0D8-19B9-E0B5- D464C50580F580B1 National Institute for Health and Clinical Excellence. (2009a). Interventional procedures programme process guide. London: National Institute for Health and Clinical Excellence. Retrieved 13 Nov. 2011 from http://www.nice.org.uk/aboutnice/ho wwework/developingniceintervention alprocedures/interventionalprocedure sprogrammemanual/interventionalpro ceduresprogrammemanual.jsp?domed ia=1&mid=96186DBC-19B9-E0B5- D4B7ABF517DFC88B National Institute for Health and Clinical Excellence. (2009b) Interventional procedure overview of non-rigid stabilisation techniques for the treatment of low back pain. London: National Institute for Health and Clinical Excellence. Retrieved 13 Nov. 2011 from http://www.nice.org.uk/nicemedia/liv e/11204/49541/49541.pdf National Institute for Health and Clinical Excellence. (2010a). About NICE guidance: What does it mean for me? A guide for patients, carers and the public. London: National Institute for Health and Clinical Excellence. Retrieved 13 Nov. 2011 from http://www.nice.org.uk/media/B88/EF /NICEAboutGuidanceFeb2010.pdf National Institute for Health and Clinical Excellence. (2010b). Non-rigid stabilisation techniques for the treatment of low back pain (Interventional Procedures Guidance IPG366). London: National Institute for Health and Clinical Excellence. Retrieved 13 Nov. 2011 from http://www.nice.org.uk/ipg366 Niederstadt, C., & Droste, S. (2010). Reporting and presenting information retrieval processes: The need for optimizing common practice in health technology assessment. International Journal of Technology Assessment in Health Care, 26(4), 450-457. doi:10.1017/s0266462310001066 Plev D., & Sutcliffe, J.C. (2005). Outcome and complications using a dynamic neutralization and stabilization pedicle screw system (DYNESYS): Is this a "soft fusion"? Spine Journal, 5(4 Suppl. 1), S141-S142. doi:10.1016/j.spinee.2005.05.282 Evidence Based Library and Information Practice 2011, 6.4 83 Relevo, R. & Balshem, H. (2011). Finding evidence for comparing medical interventions: Agency for Healthcare Research and Quality (AHRQ) and the effective health care program. Journal of Clinical Epidemiology, 64(11), 1168- 1177. doi:10.1016/j.jclinepi.2010.11.022 Rutter, D., Francis, J., Coren, E., & Fisher, M. (2010). SCIE systematic research reviews: Guidelines, 2nd ed. London: Social Care Institute for Excellence. Retrieved 13 Nov. 2011 from http://www.scie.org.uk/publications/r esearchresources/rr01.pdf Sampson, M., McGowan, J., Lefebvre, C., Moher, D., & Grimshaw, J. (2008a). PRESS: Peer review of electronic search strategies. Ottawa: Canadian Agency for Drugs and Technologies in Health. Retrieved from http://www.cadth.ca/publication/781 Sampson, M., McGowan, J., Tetzlaff, J., Cogo E., & Moher, D. (2008b). No consensus exists on search reporting methods for systematic reviews. Journal of Clinical Epidemiology, 61(8), 748-754. doi:10.1016/j.clinepi.2007.10.009 Spreckelsen, C., Deserno, T.M., & Spitzer, K. (2010). The publication echo: effects of retrieving literature in PubMed by year of publication. International Journal of Medical Informatics, 79(4), 297-303. doi: 10.1016/j.ijmedinf.2010.01.007 Yoshii, A., Plaut, D.A., McGraw, K.A., Anderson, M.J., & Wellik, K.E. (2009). Analysis of the reporting of search strategies in Cochrane systematic reviews. Journal of the Medical Library Association, 97(1), 21-29. doi:10.3163/1536-5050.97.1.004 Younger, P. & Boddy, K. (2009). When is a search not a search? A comparison of searching the AMED complementary health database via EBSCOhost, OVID and DIALOG. Health Information & Libraries Journal, 26(2):126-135. doi:10.1111/j.1471-1842.2008.00785.x Evidence Based Library and Information Practice 2011, 6.4 84 Appendix A. NICE checklist to evaluate the quality of internal search strategies:  all relevant concepts are included in the strategy  all relevant MeSH and free text terms have been identified and included  Boolean logic is accurate, within and between concepts  proximity operators have been used appropriately  truncations are appropriate  brackets are in the right place  there are no spelling mistakes  device names have been included in the strategy as appropriate  there are no spurious results which may indicate an error, e.g. lines with zero hits  the filter for eliminating animal studies has been applied correctly  for review searches, the search has been limited by entry date from the date of the last search  any other limits have been applied as appropriate  the analyst has been consulted if anything is unclear in the literature search request form Evidence Based Library and Information Practice 2011, 6.4 85 Appendix B. Modified search strategy for NICE IPG 366 Database(s): Ovid MEDLINE(R) 1948 to July Week 1 2011 Date searched: 17 July 2011 1 (flexi$ adj3 (screw$ or implant$ or device$ or instrument$)).tw. (1110) 2 (rotat$ adj3 (screw$ or implant$ or device$ or instrument$)).tw. (1016) 3 (dynesis or dynesys).tw. (50) 4 dynamic neutrali?ation system$.tw. (8) 5 (dynamic adj2 (fus$ or stabili$)).tw. (1118) 6 or/1-5 (3222) 7 (interspin$ adj3 implant$).tw. (58) 8 (graf$ adj3 soft$ adj3 stabili$ adj3 system$).tw. (1) 9 orthopedic fixation devices/ or bone nails/ or bone plates/ or bone screws/ or bone wires/ or internal fixators/ or splints/ or suture anchors/ (43487) 10 (orthoped$ adj3 fix$ adj3 device$).tw. (25) 11 (bone$ adj3 (nail$ or plate$ or screw$ or wire$)).tw. (4388) 12 (internal adj3 fix$).tw. (9746) 13 splint$.tw. (9308) 14 (suture$ adj3 anchor$).tw. (1142) 15 exp arthrodesis/ (20186) 16 arthrodesis$.tw. (7188) 17 (Spin$ adj3 Fus$).tw. (4642) 18 exp laminectomy/ (7134) 19 laminectom$.tw. (5354) 20 exp Lumbar Vertebrae/su [Surgery] (8983) 21 (Lumbar$ adj3 Vertebr$).tw. (5118) 22 ((lumbar or pedicle) adj3 fus$).tw. (2324) 23 ((ligament$ or fusion$) adj3 (bone graft or pedical screw) adj3 lumbar).tw. (16) 24 Intervertebral Disk/ (9070) 25 "Prostheses and Implants"/ (34063) 26 24 and 25 (295) 27 (prosthet$ adj3 (Interverteb$ adj3 (Disc or disk))).tw. (6) 28 or/7-23 (90011) 29 26 or 27 or 28 (90124) [Terms for the interventional procedure] 30 (flexib$ or dynamic or non-rigid or non rigid).tw. (210337) [Free text terms for non-rigid] 31 29 and 30 (3592) [Terms for the interventional procedure combined with free text terms for non- rigid] 32 6 or 31 (6514) [Expansion of the terms for the interventional procedure] 33 exp Spinal Stenosis/ (3455) 34 (spin$ adj3 stenos$).tw (2891). 35 (low$ adj3 back$ adj3 pain$).tw. (14058) 36 Low Back Pain/ or failed back surgery syndrome/ (11335) 37 exp spondylolysis/ or spondylolisthesis/ (3618) 38 spondylolisthesis.tw. (2613) 39 spondylolysis.tw. (916) 40 (lumbar$ adj3 decompress$).tw. (413) 41 (lumbar adj3 dis$ adj3 disease$).tw. (1152) 42 degenerative dis$ disease$.tw. (911) 43 ((disc or disk) adj3 herniat$).tw. (5313) 44 listhesis$.tw. (80) Evidence Based Library and Information Practice 2011, 6.4 86 45 (flexion$ adj3 instab$).tw. (109) 46 or/33-45 (30123) [Terms for the condition] 47 32 and 46 (340) [Terms for the interventional procedure AND terms for the condition] 48 FASS.tw. (122) 49 diam implant$.tw. (4) 50 interspinous U.tw. (4) 51 x-stop.tw. (45) 52 mims.tw. (244) 53 (wallis adj5 stabili$).tw. (3) 54 or/48-53 (420) [Alternative names for the device] 55 47 or 54 (749) [Terms for the procedure and the condition combined with names for the device] 56 animals/ not humans/ (3533433) [Filter to exclude animal studies] 57 55 not 56 (695) [Search results with animal studies removed] 58 Graf stabilisation system: Early results in 50 patients.ti. (1) [Grevitt et al.] 59 Non-fusion surgery for degenerative spondylolisthesis using artificial ligament stabilization.ti. (1) [Kanayama et al.] 60 (Outcome and complications using a dynamic neutralization and stabilization pedicle screw system).ti. (0) [Plev et al. is not in Medline] 61 57 and 58 (0) [Grevitt et al. not retrieved by the original search] 62 57 and 59 (0) [Kanayama et al. not retrieved by the original search] 63 6 or 29 (92860) [Terms for the procedure with non-rigid free-text terms excluded] 64 63 and 46 (7137) [Terms for the procedure and condition with non-rigid free-text terms excluded] 65 64 or 54 (7512) [Terms for the procedure and condition combined with names for the device, with non-rigid free-text terms excluded] 66 65 not 56 (7309) [Animals filter applied to the new search with the non-rigid free-text terms excluded] 67 66 and 58 (1) [Grevitt et al. retrieved when non-rigid free-text terms are excluded] 68 66 and 59 (1) [Kanayama et al. retrieved when non-rigid free-text terms are excluded] 69 55 and 58 (0) [Grevitt et al. still not retrieved by the original search when the animals filter is not used] 70 55 and 59 (0) [Kanayama et al. still not retrieved by the original search when the animals filter is not used] 71 limit 57 to ed=20091001-20110717 (141) [Items added between the original search in October 2009 and the new search for this paper] 72 57 not 71 (554) [Approximation of the results in October 2009 when the search was originally run] 73 66 not 71 (7168) [Approximation of the search results if the non-rigid terms had been excluded in October 2009] 74 72 and 58 (0) [Grevitt et al. still not retrieved when strategy restricted to the dates of the original search] 75 72 and 59 (0) [Kanayama et al. still not retrieved when strategy restricted to the dates of the original search] 76 73 and 58 (1) [Grevitt et al. would have been retrieved if the non-rigid terms had been excluded in October 2009] 77 73 and 59 (1) [Kanayama et al. would have been retrieved if the non-rigid terms had been excluded in October 2009] Evidence Based Library and Information Practice 2011, 6.4 87 Appendix C. Abbreviations AGREE Appraisal of Guidelines Research and Evaluation CADTH Canadian Agency for Drugs and Technologies in Health CHTE Centre for Health Technology Evaluation (NICE) CRD Centre for Reviews and Dissemination HTA Health Technology Assessment IPG Interventional Procedures Guidance IPAC Interventional Procedures Advisory Committee MeSH Medical Subject Headings NICE National Institute for Health and Clinical Excellence NHS National Health Service PRESS Peer Review of Electronic Search Strategies PRESS EBC Peer Review of Electronic Search Strategies Evidence Based Checklist PRISMA Preferred Reporting Items for Systematic Reviews and Meta-Analyses STARLITE Sampling Type Approach Range Limits Inclusion Terms Electronic sources