E d u c a t i o n a l P e r s p e c t i v e How t o d o a S y s t e m a t i c L i t e r a t u r e R e v i e w ABSTRACT: Physiotherapists, whether serving individual patients or populations, always have to sought to base their decisions and actions on the best possible evidence. In making choices, health professionals m ay benefit fro m structured sum m aries o f the options and outcomes, system atic reviews o f the evidence and recom m endations regarding the best choices. The aim o f this p a p er is to present guidelines on how to conduct a system atic review. The structure and content o f a system atic review are being discussed, follo w in g a step-by-step approach. KEYWORDS: SYSTEM ATIC LITERATURE REVIEW, PHYSIOTHERAPY, METHODOLOGY, M ETA-ANALYSIS PAPADOPOULOS M, MSc (Physio)' RHEEDER P, (M Med)2 1 Physiotherapy Department, U niversity of Pretoria Clinical epidemiology, U niversity of Pretoria INTRODUCTION The problem o f how to keep abreast of the literature, both old and new, con­ fronts all health professionals, including physiotherapists. It is clear that most p h y sio th erap ists engaged in clinical practice get little or no tim e for reading during their working day. Because the volum e of info rm atio n available to health care professionals is so large, it is likely that most rely on reviews to deliver synopses of current knowledge in accessible and manageable formats. Reviews are pieces o f research in them ­ selves. They use prim ary research as their data; in other w ords they are research on research. H ealth pro fessio n als m ake m any patient m anagem ent decisions, on a daily basis. Some o f these decisions are relatively inconsequential w hile some others are vitally important. Decisions involve w eighing benefits and risks, and institute courses of action judged to be in the p a tie n t’s best interest. In m aking choices, health professionals may benefit from structured summaries of the options and outcomes, systematic reviews o f the evidence and recom m en­ dations regarding the best choices. W hen properly carried out on the high­ est possible proportion o f all relevant tri­ als, these systematic reviews provide the most accurate and authoritative guides to therapy. Systematic reviews of therapy is such a logical step in progress toward evidence-based health care that it has becom e the focus o f a rapidly growing international group o f clinicians, m etho­ dolo g ists and co n su m ers w ho have form ed the C ochrane C ollaboration. System atic reviews that are beginning to flow from this collaboration, and which are updated each tim e an im portant new trial is reported, are providing the high­ est levels o f evidence ever achieved on the efficacy o f preventative, therapeutic and rehabilitative regim ens (Cochrane D atabase of systematic reviews). TYPES OF LITERATURE REVIEWS T here are tw o types o f literatu re reviews: The narrative literature review and the system atic literature review. A narrative literature review typically p resen ts a series o f studies, w ith strengths and w eaknesses d iscussed selectively and inform ally by one or m ore acknowledged experts (Rosenfeld, 1996; Oxman et al, 1994).This unsys­ tematic approach o f perform ing a lite­ rature review represents the traditional strategy o f accum ulating and sum m aris­ ing evidence in an unsystem atic fashion and then applying im plicit preferences to arrive at treatm en t reco m m en d a­ tions. The approach is open to bias and is likely to lead to consistent, valid recom­ m endations only w hen the g radient between beneficial and adverse conse­ quences o f attentive actions is very large. The risk of such unsystem atic identification and collection of evidence is that treatm ent effects may be unde­ restim ated or, m ore commonly, over e stim ated . A d v erse effects m ay be exaggerated or ignored. One result of these unsystem atic approaches may be recom m endations advocating harm ful treatm ents and failin g to encourage effective therapy (Guyatt et al, 1999). F o r exam ple, the ob serv atio n that patients with ventricular ectopic beats follow ing m yocardial infarction were at high risk o f sudden death, coupled with the dem onstration that these extra beats could be suppressed by specific drugs, form ed a rationale for the w ide­ spread prescription o f these drugs to post-infarction patients w ith unstable cardiac rhythms. However, subsequent random ized controlled trials exam ined hard clinical outcom es, not physiologic processes, showed that several of these drugs increase, rather than decrease, the risk of death in such patients and their routine use is now strongly discouraged (Sackett et al, 1997). A more acceptable alternative would be to collect and read all the available evidence and perform a system atic re­ view. A system atic review is a means of achieving an adequate overview of the literature with respect to the quality of CORRESPONDENCE: M Papadopoulos D epartm ent of Physiotherapy University o f Pretoria PO B ox 667 Pretoria 0001 Tel: (012) 354-2023 SA J o u r n a l o f Physiotherapy 2 0 0 0 V o l 5 6 No 1 3 R ep ro du ce d by S ab in et G at ew ay u nd er li ce nc e gr an te d by th e P ub lis he r (d at ed 2 01 3. ) TABLE 1: Steps in Performing a Systematic Review STATE A CLEAR QUESTION □ c GIVE A SHORT BACKGROUND TO THE REVIEW □ i STATE CLEAR, SPECIFIC OBJECTIVES OF THE REVIEW --------- i c SET ELIGIBLITY CRITERIA FOR STUDIES TO BE RETRIEVED □ c PERFORM A SYSTEMATIC, COMPREHENSIVE SEARCH FOR STUDIES □ c APPLY ELIGIBLITY CRITERIA FOR STUDIES RETRIEVED - G IV IN G REASONS FOR EXCLUSIONS CH O O SE A N APPROPRIATE METHOD TO SYNTHESIZE DATA > COMPLETE SYNTHESIS OF DATA/METHODOLOGICAL QUALITY ASSESSMENT I I _ DISCUSS RESULTS SHORTLY IN TERMS OF PREVIOUSLY STATED OBJECTIVES □ c CONCLUDE INDICATING IMPLICATIONS FOR FUTURE RESEARCH A N D CLINICAL PRACTISE I I PREPARE A STRUCTURED REPORT OF THE REVIEW - CO NSISTIN G OF ALL OF THE ABOVE-MENTIONED STEPS A N D PUBLISH! the review ed papers and the outcom e of the interventions studied (De Bie, 1996). S ystem atic review s explicitly state in clu sio n and ex clusion criteria for evidence to be considered, conduct a com prehensive search for the evidence, and sum m arise the results according to ex p licit ru les (G uyatt et al, 1999). System atic reviews thus reduce large quantities o f research into key findings in a reliable way and offer a means of enabling health care professionals to keep abreast o f research; as such they are an essential step in the identification o f evidence - based health care (D roogan & Cullum , 1998). A systematic review provides strong evidence when the quality o f the pri­ mary studies is high and sample sizes are large and less strong evidence when designs are w eaker and sample sizes small. Because judgem ent is involved in m any steps in a sy stem atic review (in clu d in g specify in g in clusion and exclusion criteria, applying these criteria to potentially eligible studies, evaluating the m ethodological quality o f the pri­ mary studies, and selecting an approach to data analysis), systematic reviews are not im m une from bias. Nevertheless, in their rigorous approach to collecting and sum m arising data, system atic reviews reduce the likelihood o f bias in estim at­ ing the causal links betw een m anage­ m ent options and patien t outcom es (Guyatt et al, 1999). A IM OF THIS PAPER The aim of this paper is to present guide­ lines on how to conduct a systematic review. To study the quality of trails and sum m arize the results o f w ell-perform ed trials with high internal validity, and hence derive valid conclusions, is a rather new p h en o m en o n in physiotherapy. System atic reviewing could represent a useful tool for not only evaluating the effectiveness o f physiotherapy, but also for formulating new hypotheses and iden­ tifying areas w here lack o f knowledge hinders further developm ent (De Bie, 1996). S ystem atic review s can also have an educational function in that they could be used for theses or dissertations, thereby indicating w here future research can be im proved by avoiding flaw s from previous research. STRUCTURE AND CONTENT OF A SYSTEMATIC REVIEW A system atic review is com posed o f a num ber o f sections and steps (see Table 1). STATES A CLEAR QUESTION A specific question or set of questions should be stated. The questions should be w ell-defined and include references to the target population, the health care intervention and the outcom es o f inter­ est. A short background to the review should be given stating the im portance o f the review. Consequently, the objectives o f the review should be stated. O bjectives should be clear, specific and can include sub-group analysis eg w hether effects o f a certain intervention are dependent on age, gender or the co-existence of other illness or disease etc. For exam ple, to determ ine the effect o f a certain treat­ m ent m ethod in patients of different age groups. W ill the outcom es in the differ­ ent age groups be the same? On the basis o f the research question, eligibility criteria are form ulated on w hich studies are to be retrieved. These criteria apply to the participants o f stu­ dies (exclusion/inclusion criteria), the types o f disease or com plaint to be stu­ died, the study intervention and the o u tco m e m easu res to be rev iew ed . 4 SA J o u r n a l o f Physiotherapy 2000 V o l 56 No 1 R ep ro du ce d by S ab in et G at ew ay u nd er li ce nc e gr an te d by th e P ub lis he r (d at ed 2 01 3. ) FIGURE 1: The funnel plot for assessing publication bias. An odds ratio of one represents the line of null effect. Values <1 represent beneficial therapy while values > 1 represent harmful therapy. R equirem ents are also set for the study design i e random ised control trials (RCTs), control trials (CT) or pseudo­ random ised control trials (pseudo- RCTs). It should be noted though that the RCT is considered the most reliable m ethod o f assessing the efficacy of health care interventions. In the field of p h y siotherapy research, as in m any other disciplines, unfortunately not all trials live up to these standards. Although it rem ains questionable as to w hether evidence should be gathered from less adequately perform ed studies, it would be foolish to ignore the poten­ tial for obtaining valuable information from such investigations (De Bie, 1996). PERFORMS A SYSTEMATIC COMPREHENSIVE SEARCH STRATEGY This is im portant in order to identify as large as possible a proportion of all the prim ary studies. A review, which does not make strenuous efforts to identify all relevant studies, runs a great risk o f introducing bias. Published material can be found in databases such as the Cochrane library, M edline, Cinahl or Em base. Details of the search such as cut-off dates for studies, key words and M eSH terms should be stated clearly in order to make the search reproducible for other researchers. W hen reviewing physiotherapy research, a m ajor problem is that a num ber of trials appear in non­ indexed journals. There is also evidence to suggest that studies with “positive” results are more likely to appear in the published literature (Cook et al, 1993). This phenom enon, called “publication bias”, can yield false positive findings if one or m ore unpublished or unindexed “negative” articles are excluded from the system atic review. H andsearching of journals should be done as neglecting to do so may result in running the risk o f m issing important research. Lim iting searches to E nglish language articles only, may be a m ajor reason for non­ representativeness o f studies identified. W riting to experts, perform ing library handsearching in order to identify student projects, theses and conference proceed­ ings, are m easures required to identify unpublished material. After obtaining all relevant studies, criteria for eligibility, as stated before, should be applied. The presence of publication bias can easily be detected using a simple graph called a funnel plot (Figure 1). Smaller studies can be expected to yield more variable results and larger studies more precise estimates. If the right side o f the funnel only has a small num ber o f stud­ ies one can assume that a significant num ber o f negative studies has not been published. A nother statistical m ethod is som e­ times used to determ ine how important m issed negative studies would be. This so called “failsafe-n” gives the num ber o f unpublished studies that would be needed to yield the found pooled esti­ mate invalid ( G reenhalgh 1997). APPROPRIATE SYNTHESIS OF DATA In order to prevent bias a blinded assess­ m ent system can be used for the system ­ atic review. Authors, institutes or centres w here the studies have been performed and the journal in w hich the paper was published, can be masked. A t least two reviewers should independently study characteristics and outcom es, including inform ation on m orbidity and mortality, interventions, length o f trial follow-up etc. The final results can be reached by consensus am ongst reviewers. Experts from different areas such as statisticians and research m ethodologists can be used to rank data. W here there is m ore than one study sufficiently sim ilar in term s o f the inter­ vention studied and the outcom e(s) m ea­ sured, then statistical techniques may be used to synthesis (or pool) data from the prim ary studies (a process know n as m eta-analysis). Pooling o f data, how ­ ever is not always appropriate as in stud­ ies w hich are not sufficiently sim ilar in either a clinical or design sense as is very often the case in physiotherapy studies. In such a case a critical review w hich focuses on the quality o f existing studies and the strength o f the evidence they provide, will be m ore appropriate (De Bie, 1996). Reviewers can use a list o f param eters in order to m easure the m ethodological quality o f the studies. A w eighting scale can be applied to these param eters to discrim inate betw een m ajor and m inor flaws in trial design. Param eters that could be considered are listed in table 2. A score o f one point (+) can be given to each param eter fulfilled and zero (-) if not. Param eters can be graded as 0,5 points (_) if the descriptions are unclear or if incomplete for intervention, outcome m easures or data presentation (eg an unclear prescription o f blinding can be graded ±). In this w ay a m axim um m ethodological quality score can be cal­ culated and a hierarchical list, where higher scores indicate studies of higher methodological quality, can be generated. PRINCIPLES OF QUANTITATIVE SYNTHESIS: META-ANALYSIS The principle behind m eta-analysis is that the larger the sam ple size o f a study the m ore p recise the resu lts. M any SA J o u r n a l o f Physiotherapy 2000 V o l 56 No 1 5 R ep ro du ce d by S ab in et G at ew ay u nd er li ce nc e gr an te d by th e P ub lis he r (d at ed 2 01 3. ) TABLE 2: Suggested Parameters to determine the Methodological Quality of a Study. • W e ll d e s c rib e d inclusion c rite ria . • Presentation o f re le v a n t b a s e lin e characteristics. • R an d o m a llo c a tio n p ro c e d u re d e s c rib e d (a llo c a tio n co n c e a lm e n t). • Interventio ns w e ll d e s c rib e d (n a tu re , n u m b er, d u ra tio n o f treatm en t). • B lin d in g (patien ts, th e ra p is t, o b s e rv e r). • < 1 0 % w ith d r a w a ls a n d w ith d r a w a ls d e s c rib e d . • E ffe c t/o u tc o m e m easu rem en ts re le v a n t a n d w e ll d e s c rib e d . • A nalysis a n d presentation o f the results so that the analysis can b e assessed. statistically in sig n ifican t resu lts are found sim ply because the study was too small ( a so called type II error). By pooling different studies ( given that the m ethodologies and outcom es were approxim ately the same) the sam ple size is increased and a m ore precise estimate is obtained ( so called pooled estimate, often a pooled odds ratio ) (G reen- halgh,1997). Prior to pooling studies the reviewer must ensure that studies are homegenous (com bining apples with apples and not apples w ith pears). T his im plies that studies com e from the same population and that their results only differ because o f chance variation. By visually inspect­ ing the 95% confidence intervals of the estimates one can see w hether these overlap or not. If they do not overlap then these studies are hetergenous (implying that they do not com e from the same population). This is can also be shown with a statistical test called the chi-square test for hetergeneity with a p-value > 0.05 showing that the studies are indeed hom ogenous. This test how ­ ever has low pow er for detecting hetero­ geneity (Greenhalgh, 1997). The statistical method actually used for calcualting the “average”or pooled outcom e depends on whether hetero­ geneity was found or not. If they are ho m ogenous the outcom e is pooled across studies using a so called fixed- effects model and if they were heteroge­ nous using a random -effects model. The random-effects model gives a more conservative result (a larger p value). It is also im portant that heterogeneity be exam ined further and that results are not ju st sim ply pooled. Causes of hetero­ geneity m ight be related to age, stage of disease or in the case o f drugs, dosages used (Greenhalgh 1997). Often the statistical analysis includes a sensitivity analyis. In doing the effect on the outcom e is explored if you m anipulate certain variables such as quality w eights, inclusion and exclusion of weaker studies etc. The m ore robust the estim ate the better. If however the slightest change in quality weights for instance changes the estim ate signifi­ cantly then there are serious concerns about its validity (Greenhalgh, 1997). DISCUSSION OF FINDINGS The most im portant goal of systematic review ing is to sum m arize evidence from original research and thereby p ro ­ vide an adequate overview of the effect or effects o f the intervention of interest being studied (De Bie, 1996). In the sum m ary o f the sy stem atic review, m ethodologic lim itations of both the prim ary studies and the review process itself should be elaborated. T his section should th o roughly address the issue of bias in observational studies and the potential role o f the placebo effect when a control group is not present (Rosenfeld, 1996). A short discussion of the objectives initially stat­ ed, should com plete this section. The co nclusions should in clude a future research agenda including clinical and m eth o d o lo g ical req u irem en ts o f proposed studies. PUBLISHING RESULTS A properly perform ed system atic review will qualify for publication in m ost peer- reviewed journals. The abstract should be structured, stating clearly th at a protocol was follow ed, that specific objectives were set and that eligibility criteria for article selection w ere used. A table w ith key elem ents o f articles such as study design, sample size, treat­ m ent dose and duration, outcom e m ea­ sures, patients lost to follow-up etc, is a very helpful tool in supplem enting the text. A nother table, rep o rtin g rating m ethods and the m ethodological quality o f studies (as in table 2) can also be added. Reasons for excluding identified studies from the sy stem atic review should also be given. REFERENCES D e B ie R A ; 1 9 9 6 . M e th o d o lo g y o f sy ste m a tic review s: A n in trodu ction . P h y sic a l Therapy R e v ie w 1 :4 7 -5 1 . D r o o g a n J, C u llu m IV; 1 9 9 8 . S y ste m a tic re v ie w s in nursing. International Journal o f N u rsin g S tu d ie s 3 5 :1 3 -2 2 . C o ch ra n e datab ase o f sy ste m a tic rev iew s. A v a ila b le fr o m B M J P u b lis h in g gro u p , P O B o x 2 9 5 , L o n d o n W C 1 H 9 T E , U K . Tel: + 4 4 (0) 171 3 8 3 6 1 8 5 /6 2 4 5 ; F ax + 4 4 (0) 171 3 8 3 6 6 6 2 . C o o k D J, G uyatt G H , B y a n G , C lifto n J, B u c h in g h a m L , W illa n A , M c llr o y W, O xm an A D . ( 1 9 9 3 ). S h o u ld u n p u b lish e d data b e in clu d ed in m e ta -a n a ly se s? Current c o n v ic ­ tio n s and c o n tr o v e r s ie s . JA M A 2 6 9 (2 1 ): 2 7 4 9 - 2 7 5 3 . G reen h a lg h T ; 1 9 9 7 . H o w to read a paper: Papers that su m m a rise other papers (s y s te ­ m atic r e v ie w s and m e ta -a n a ly se s) B M J 315: 6 7 2 -6 7 5 G uyatt G H , S in cla ir J, C o o k D J, G la sz io u P; 1 9 9 9 . U s e r s ’ G u id e s to m e d ic a l literature: X V I. H o w to u se a treatm ent r e c o m m e n ­ dation. J A M A 2 8 1 ( 1 9 ) : 1 8 3 6 - 1 8 4 3 . R o s e n fe ld R M ; 1 9 9 6 . H o w to sy ste m a tic a lly re v ie w th e m e d ic a l literature. O ta la r y n g o lo g y - H ead and N e c k surgery 1 1 5 ( l): 5 3 -6 3 . O x m a n A D , C o o k D J, G uyatt G H ; 1994. U se r s G u id es to th e M e d ic a l Literature: VI. H o w to u se an o v erv iew . JA M A 2 7 2 (1 7 ): 1 3 6 7 -1 3 7 1 . S ack ett D L , R ich a rd so n W S , R o sen b e rg W, H a y n es R B ; 1 9 9 7 . E v id e n c e -B a s e d M ed icin e: H o w to p ractice and tea ch E B M . C h u rch ill- L iv in g sto n e , F irst p u b lic a tio n . 6 SA J o u r n a l o f Physiotherapy 2000 V o l 56 No 1 R ep ro du ce d by S ab in et G at ew ay u nd er li ce nc e gr an te d by th e P ub lis he r (d at ed 2 01 3. )