1http://dx.doi.org/10.20396/bjos.v20i00.8661701 Volume 20 2021 e211701 Original Article 1 Graduate Program in Dentistry, Meridional College/IMED, Passo Fundo, Brazil. 2 MSciD and PhD Graduate Program in Oral Science, Faculty of Dentistry, Federal University of Santa Maria (UFSM), Santa Maria, Rio Grande do Sul State, Brazil. Corresponding author: Rafael Sarkis-Onofre Graduate Program in Dentistry – Meridional Faculty/IMED, 304, Senador Pinheiro Street - Passo Fundo – Brazil 99070-220 e-mail: rafael.onofre@imed.edu.br Received: October 26, 2020 Accepted: December 16, 2021 How are meta-analyses being conducted and reported in dentistry? a meta-research study Letícia Tainá de Oliveira Lemes1 , Lara Dotto1 , Bernardo Antonio Agostini1 , Gabriel Kalil Rocha Pereira2 , Rafael Sarkis-Onofre1,* Aim: This study aimed to evaluate how meta-analyses are conducted and reported in dentistry. Methods: We conducted a search to identify dentistry-related Systematic Reviews (SRs) indexed in PubMed in 2017 (from January 01 until December 31) and published in the English language. We included only SRs reporting at least one meta-analysis. The study selection followed the 4-phase flow set forth in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Statement (PRISMA), and it was independently conducted by two researchers. Data extraction was performed by one of three reviewers, and data related to conducting and reporting of the meta-analysis were collected. Descriptive data analysis was performed summarizing frequencies for categorical items or median and interquartile range for continuous data. Results: We included 214 SRs with meta-analyses. Most of the studies reported in the title that a meta-analysis was conducted. We identified three critical flaws in the included studies: Ninety (90) meta-analyses (43.1%) did not specify the primary outcome; most of the meta-analyses reported that a measure of statistical heterogeneity was used to justify the use of a fixed-effect or random-effects meta-analysis model (n=114, 58.5%); and a great part did not assess publication bias (n=106, 49.5%). Conclusion: We identified deficiencies in the reporting and conduct of meta-analysis in dentistry, suggesting that there is room for improvement. Educational approaches are necessary to improve the quality of such analyses and to avoid biased and imprecise results. Keywords: Oral health. Systematic reviews as topic. Research report. 2 Lemes et al. Introduction Meta-analyses are an important component of systematic reviews and are a statisti- cal method to combine results from two or more independent studies1. This method enables improving the precision of estimates and answering conflicting questions or questions not discussed in individual studies2. However, meta-analyses might be misused and biased in a similar way to other research methods. Ioannidis3 (2016) demonstrated that 9,135 meta-analyses were published in PubMed in 2014, corresponding to an increase in the publication rate of 2,635% between 1991 and 2014. The author highlighted that many meta-analyses are redundant, unnecessary, or have methodological flaws. Page et al.4 investigated flaws in the application and inter- pretation in a sample of meta-analyses of therapeutic interventions. The findings demon- strated problems in aspects such as interpretation of the model used, subgroup analyses, and the minimum number of studies recommended to test funnel plot asymmetry. In dentistry, Saltaji et al.5 assessed 1,118 systematic reviews (SRs) published between 1991-2012; the findings demonstrated that almost 50% performed a meta-analysis, and the median of included studies in the largest meta-analysis was 9. Moreover, several arti- cles assessed the quality of meta-analyses performed in different oral health specializa- tions, demonstrating that the overall quality varies from low to medium6. However, there are no studies in dentistry evaluating reporting and conducting meta-analysis character- istics, which would enable highlighting areas for future improvement. Thus, this study aimed to evaluate how meta-analyses are conducted and reported in dentistry. Materials and Methods We used a dataset of SRs in dentistry indexed in PubMed in 2017. The reporting and conduct characteristics of these SRs were previously published7. We have only eval- uated data from SRs with meta-analysis and assessed how meta-analyses are con- ducted and reported. Search and Eligibility criteria A full description of the search strategy and eligibility criteria is available in the study of Bassani et al. (2019)7. First, we conducted a search in PubMed to identify dentistry-related SRs published in the English language indexed in 2017 (from Janu- ary 01 until December 31). We considered the article as a SR based on the Preferred Reporting Items for Systematic reviews and Meta-Analysis Protocols (PRISMA-P) defi- nition8. In this article, we have only included SRs reporting at least one meta-analysis. In addition, we included articles regardless of questions answered and study designs included (clinical studies, in vitro, in situ, etc.). The search strategy used is presented in Table 1 and was based on MeSH terms of PubMed and a specific filter (U.S. National Library of Medicine). Screening We selected the studies using a reference manager software (EndNote X7, Thomson Reuters, New York, USA). Details about the pilot test screening and study selection are 3 Lemes et al. available in the study by Bassani et al.7. The study selection followed the 4-phase flow set forth in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Statement9. Two researchers independently identified articles by reviewing titles and abstracts, and then screening the full text in a second phase. Data extraction A standardized form was created using Microsoft Excel based on the study by Page et al. 10 (2016). A pilot data extraction was performed to ensure consistency, and the details are available in the study by Bassani et al.7. Data related to the conduct and reporting of meta-analyses were extracted by one of three review- ers. Data regarding the journal category (general or specialized journal), number of authors, SR focus (Epidemiology, Diagnosis, Prevention, Prognosis, Treatment/ Therapeutic, Other, Unclear, or Mixed), dental specialization, funding (reported or not reported), and data related to conducting and reporting of meta-analysis such as details about the primary outcome, the model used, number of studies in the largest meta-analysis, statistical heterogeneity, publication bias, additional anal- ysis and software used to conduct the meta-analyses were extracted. The form used is available in the supplementary material. One author subsequently verified the data consistency, and the data were extracted again in the case of any doubt or inconsistency. Data analysis Descriptive analysis of the data was performed by summarizing frequencies for categorical items and calculating median and interquartile range for continuous data. All analyses were performed using Stata 14.2 software. Characteristics of the meta-analyses were assessed considering all included SRs and meta-analyses. Table 1. Search strategy “Oral Health”[Mesh] OR “Oral Health” OR “Health, Oral” OR “Dentistry”[Mesh] OR “Dentistry” OR “Dental Research”[Mesh] OR “Dental Research” AND (((systematic review [ti] OR meta-analysis [pt] OR meta-analysis [ti] OR systematic literature review [ti] OR this systematic review [tw] OR pooling project [tw] OR (systematic review [tiab] AND review [pt]) OR meta synthesis [ti] OR meta synthesis [ti] OR integrative review [tw] OR integrative research review [tw] OR rapid review [tw] OR consensus development conference [pt] OR practice guideline [pt] OR drug class reviews [ti] OR cochrane database syst rev [ta] OR acp journal club [ta] OR health technol assess [ta] OR evid rep technol assess summ [ta] OR jbi database system rev implement rep [ta]) OR (clinical guideline [tw] AND management [tw]) OR ((evidence based[ti] OR evidence-based medicine [mh] OR best practice* [ti] OR evidence synthesis [tiab]) AND (review [pt] OR diseases category[mh] OR behavior and behavior mechanisms [mh] OR therapeutics [mh] OR evaluation studies[pt] OR validation studies[pt] OR guideline [pt] OR pmcbook)) OR ((systematic [tw] OR systematically [tw] OR critical [tiab] OR (study selection [tw]) OR (predetermined [tw] OR inclusion [tw] AND criteri* [tw]) OR exclusion criteri* [tw] OR main outcome measures [tw] OR standard of care [tw] OR standards of care [tw]) AND (survey [tiab] OR surveys [tiab] OR overview* [tw] OR review [tiab] OR reviews [tiab] OR search* [tw] OR handsearch [tw] OR analysis [ti] OR critique [tiab] OR appraisal [tw] OR (reduction [tw]AND (risk [mh] OR risk [tw]) AND (death OR recurrence))) AND (literature [tiab] OR articles [tiab] OR publications [tiab] OR publication [tiab] OR bibliography [tiab] OR bibliographies [tiab] OR published [tiab] OR pooled data [tw] OR unpublished [tw] OR citation [tw] OR citations [tw] ,OR database [tiab] OR internet [tiab] OR textbooks [tiab] OR references [tw] OR scales [tw] OR papers [tw] OR datasets [tw] OR trials [tiab] OR meta-analy* [tw] OR (clinical [tiab] AND studies [tiab]) OR treatment outcome [mh] OR treatment outcome [tw] OR pmcbook)) NOT (letter [pt] OR newspaper article [pt]))) 4 Lemes et al. Results Figure 1 presents a flow diagram outlining the study selection process. The initial search in PubMed yielded 1375 records, and we included 214 SRs with meta-analysis after study screening of the title/abstract and full-text analysis. Table 2 presents the epidemiological characteristics of the SRs with meta-analyses included in the study. Most of the meta-analyses were published in specialized journals (n=159, 74,3%). Considering the country where those meta-analyses were produced, 4 countries produced 64.3% of SR with meta-analysis published, in which Brazil had the greatest contribution with 61 (28.5%). The main specialization was Oral and Maxillofacial Surgery (n=32, 14.9%), followed by Implantology (n=31, 14.5%) and Periodontics (n=29, 13.5%). The main focus of most of the SRs with meta-analyses was treatment/therapeu- tic (n=101, 47.2%), and a great number of the included meta-analyses reported no funding (n=80, 37.4%). The median number of authors was 5 (IQR 4-6). Figure 1. Flow diagram of study selection PRISMA 2009 Flow Diagram Records identified through database searching (n = 1375) Additional records identified through other sources (n = 0) Id en ti fi ca ti on S cr ee ni ng El ig ib ili ty In cl ud ed Meta-analyses (n = 214) SRs included (n = 495) Full-text articles assessed for eligibility (n = 616) Records screened (n = 1375) Records excluded (n = 759) Full-text articles excluded, with reasons (n = 121) See Bassani et al., 2019 Records after duplicates removed (n = 1375) 5 Lemes et al. Table 2. Epidemiological characteristics of meta-analyses included in the study Journal n % General 55 25.7% Specialty 159 74.3% Country Brazil 61 28.5% USA 24 11.2% China 18 8.4% United Kingdon 11 5.1% Switzerland 9 4.2% Germany 8 3.7% Sweden 7 3.2% Other 30 countries and unclear 76 35.7% Dental Specialties Oral and Maxillofacial Surgery 32 14.9% Implantology 31 14.5% Periodontics 29 13.5% Orthodontics 26 12.1% Oral and Maxillofacial Pathology 24 11.2% Restorative and Esthetic Dentistry 19 8.8% Pediatric Dentistry 17 7.9% Endodontics 13 6.1% Public Health 12 5.6% Prosthodontics 5 2.3% Radiology 5 2.3% Others 1 0.5% Focus Treatment/Therapeutic 101 47.20% Diagnosis 41 19.2% Epidemiology 29 13.5% Prognosis 22 10.3% Prevention 9 4.2% Other 6 2.8% Unclear 6 2.8% Funding No funding 80 37.4% Not reported 77 36% Non-profit sponsor 53 24.8% Unclear 3 1.4% Mixed 1 0.5% Number of authors Median 5 (IQR = 4-6) IQR – Interquartile range 6 Lemes et al. Table 3 presents the conduct and reporting characteristics of meta-analyses. Most of the SRs reported that a meta-analysis was conducted in the title (n=187, 87.4%). The “continuous variable” was the type of outcome most used (n=41, 34.4%), and the unit of measure of the first reported result of the primary outcome was “mean difference” (n=45, 38.5%). Most of the meta-analyses reported the statistical signif- icance of the first reported result of the primary outcome as “favorable, statistically significant” (n=60, 50.8%), and the majority of studies used a random effect model for all meta-analyses (n=130, 62.5%). Regarding the statistical heterogeneity of included studies in the meta-analysis, the majority of meta-analyses described some method to formally evaluate the statistical heterogeneity of included studies (n=180, 87.0%), and the method most used was the I2 test (n=161, 75.2%). When additional analyses were conducted, the most used was subgroup (n=65, 30.4%); however, most of the meta-analyses did not conduct additional analyses (n=121, 56.6%). The software most used to conduct meta-analyses was RevMan (n=73, 34.1%), followed by Stata (n=34, 15.9%). The median of studies included in the largest meta-analyses was 9 (IQR 5-16). Forty (40) meta-analyses (18.7%) presented a funnel plot graph; however, 10 did not report that the analysis was performed, and a further 3 did not report that publication bias was assessed. Table 3. Characteristics of conduct and reporting of meta-analyses Terms in the title n % Systematic review and meta-analysis 160 74.8% Only meta-analysis 27 12.6% Only systematic review 17 7.9% Neither 10 4.7% Did the review authors specify one or more primary outcome(s)? Yes 106 50.7% No 90 43.1% No but only one outcome reported 13 6.2% What type of outcome is the primary outcome? Continuous 41 34.4% Dichotomous 25 21% Both dichotomous and continuous 23 19.3% Not reported 15 12.6% Rate 12 10.1% Unclear 3 2.5% What is the unit of measure of the first reported result (effect estimate) of the primary outcome? Mean difference 45 38.5% Risk ratio 24 20.5% Odds ratio 18 15.4% Standardized mean difference 12 10.3% Prevalence 6 5.1% Continue 7 Lemes et al. Continuation Not reported or unclear 6 5.1% Likelihood ratios 5 4.3% Event Rate 1 0.8% What is the statistical significance of the first reported result (effect estimate) of the primary outcome? Favorable, statistically significant 60 50.8% Favorable, non-statistically significant 19 16.1% Unfavorable, non-statistically significant 14 11.9% Unfavorable, statistically significant 12 10.2% Non comparative 12 10.2% Not reported 1 0.8% Which meta-analysis model was used in the meta-analyses? Random-effects model for all meta-analyses 130 62.5% Varied 47 22.6% Fixed-effect model for all meta-analyses 21 10.1% Other 3 1.4% Not reported 7 3.4% Was the risk of bias (or quality) assessment incorporated into any meta-analyses in the review? No 119 59.9% Yes 79 40.1% Was any method described to formally evaluate statistical heterogeneity of included studies? No 25 12.1% Yes 180 87% Statistical heterogeneity was not taken into account using formal statistical evaluation, but heterogeneity of the studies was qualitatively assessed 2 0.9% Which methods were used to formally evaluate statistical heterogeneity of included studies? (*considering 214 studies) Chi-square or Cochran’s Q 109 50.9%* I^2 (I-square) 161 75.2%* tau^2 (tau-square) 17 7.9% Other 6 2.8%* Did the authors report that a measure of statistical heterogeneity was used to justify use of a fixed-effect or random-effects meta-analysis model? No 81 41.5% Yes 114 58.5% Did the authors report assessing (or an intent to assess) publication bias? No, publication bias was not assessed, and the authors did not report an intention to assess it 106 49.5% No, publication bias was not assessed, but the authors reported that they intended to assess it if they identified a sufficient number of studies 29 13.5% Yes, publication bias was assessed 79 36.9% Continue 8 Lemes et al. Continuation Which methods did the authors report using (or intending to use) to assess publication bias? (*considering the number total of studies) Begg’s test 6 2.8%* Egger’s test 25 11.7%* Funnel plot 64 30%* Sensitivity analysis comparing fixed-effect to random- effects model 3 1.4%* Subgroup analyses by sample size 2 0.9%* Other 2 0.9%* Funnel plot presented (*considering the number total of studies) 40 18.7% Which of the following additional analyses did the authors conduct? Meta-regression 21 9.8% Network meta-analysis 2 0.9% Sensitive 41 19.2% Subgroup 65 30.4% No additional analyses 121 56.6% What statistical software was used to perform meta-analyses? RevMan 73 34.1% Stata 34 15.9% Comprehensive Meta-Analysis 31 14.5% R 18 8.4% Not reported 18 8.4% Others 13 6.1% Two or more different programs 13 6.1% MedCalc 8 3.7% OpenMeta 6 2.8% Median of studies included in the largest meta-analysis 9 (IQR =16-5) For the primary outcome, 90 meta-analyses (43.1%) did not specify the primary out- come, and most of the meta-analyses did not incorporate the risk of bias assess- ment into them (n=119, 59.9%). Most of the meta-analyses reported that a mea- sure of statistical heterogeneity was used to justify the use of a fixed-effect or random-effects meta-analysis model (n=114, 58.5%) and did not assess publication bias (n=106, 49.5%). Discussion Our study is the first in dentistry to assess how meta-analyses are conducted and reported. We identified that the reporting and conduct characteristics of meta-analyses are varied. Most of the studies reported that a meta-analysis was conducted in the title and formally evaluated the statistical heterogeneity of included studies. However, most of the meta-analyses reported that a measure of statistical heterogeneity was 9 Lemes et al. used to justify the use of a fixed-effect or random-effects meta-analysis model and did not assess publication bias, thus demonstrating that there is room for improve- ment in conducting meta-analyses in dentistry. In addition, our estimates showed an increase rate of SRs with meta-analyses published in recent years compared to a previous study5. Several studies in dentistry assessed the quality of meta-analysis published in differ- ent dental specializations11,12. Both publications demonstrated that the meta-analysis quality can vary from medium to low and highlighted that important aspects such as publication bias assessment were not conducted in most meta-analyses; these findings were also corroborated by our study. Page et al.4 demonstrated that flaws in meta-analyses are not restricted to dentistry. The authors evaluated 110 statistical analyses in systematic reviews of therapeutic interventions in the biomedical field and pointed out that the involvement of statisticians in conducting systematic reviews could improve the quality of meta-analyses. Three important flaws in the conduct/report of meta-analyses were identified. First, almost 50% of the included studies did not assess the impact of publication bias. Publication bias occurs when there is a predilection for publishing particular results, such as in studies reporting high effect sizes which are more likely to be published than studies reporting lower effect sizes, and this type of bias can affect systematic reviews. Failure to assess the possible impact of publication bias on systematic reviews results can generate an invalid estimate, which could guide interventions or actions based on biased conclusions1,13. Koletsi et al. 14 (2016) assessed 162 sys- tematic reviews in dentistry, with the findings demonstrating a low rate of publica- tion bias analysis performed and a high rate of improperly conducted publication bias assessment. The second important flaw is the use of a statistical heterogeneity measure to jus- tify the use of a fixed- or random-effects meta-analysis model. This could be con- sidered a shortcoming since the choice of each effect only based on a statistical measure could exacerbate the effect of bias, thus producing a spurious estimate and an inappropriate conclusion. Recent recommendations clearly affirm that the decision between the use of the model is a topic of much debate. An alternative to the topic is a pragmatic approach planned to perform both analyses (fixed and random models), while the random effect model can be present in cases of not iden- tifying asymmetry in funnel plot, even knowing that funnel plot asymmetry suggests that both methods are problematic. In addition, the author pointed out that “the choice between a fixed-effect and a random-effects meta-analysis should never be made based on a statistical test for heterogeneity”1. Lastly, almost 50% of the meta-analyses included did not specify the primary out- come. Tricoo et al. 15 (2016) evaluated a sample of 96 systematic reviews and demonstrated that one-third of the studies did not specify or change the primary outcome. We did not assess if the studies included in our study were registered or did not compare the information reported in protocols and in the final publication, however it could be possible that study results were related to outcome reporting bias, generating imprecise results. 10 Lemes et al. There are some undeniable limitations of our study. We only considered one database and only studies published in English, and we performed a cross-sectional analysis only considering 2017. Also, our data extraction was based on the study report, and it is possible that some SRs were conducted more rigorously than was specified in the report. Additionally, we did not include meta-analyses without a systematic review, which could have led to selecting studies with high methodological quality, and we did not judge the study quality and if a correct methodology was used to perform the meta-analyses because this was not the aim of this study. Various approaches are encouraged to minimize the flaws identified in our analysis: 1) graduate program students should be receiving adequate training to perform and report meta-analyses; 2) involvement of statisticians on the systematic review team could improve analysis quality; 3) dissemination of PRISMA Statement extensions such as PRISMA for Individual Patient Data and Network Meta-Analyses could help in reporting adequate meta-analyses. In conclusion, we identified deficiencies in the reporting and conduct of meta-analyses in dentistry, suggesting that there is room for improvement. Educational approaches are necessary to improve the quality of such analyses and to avoid unbiased and imprecise results. Acknowledgements BAA, and RSO are funded in part by Meridional Foundation (Passo Fundo – Brazil). LL and LD are supported by CAPES, Brazil. This study was conducted in a Graduate Program sup- ported by CAPES, Brazil (Finance Code 001). The funders had no role in the study design, data collection and analysis, decision to publish or preparation of the manuscript. References 1. Page MJ, Higgins JPT, Sterne JAC. Assessing risk of bias due to missing results in a synthesis. In: Higgins J, Thomas J. Cochrane handbook for systematic reviews of interventions. Cochrane; 2019. Version 6.0. Chapter 13. 2. McKenzie JE, Beller EM, Forbes AB. Introduction to systematic reviews and meta-analysis. Respirology. 2016 May;21(4):626-37. doi: 10.1111/resp.12783. 3. Ioannidis JP. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Milbank Q. 2016 Sep;94(3):485-514. doi: 10.1111/1468-0009.12210. 4. Page MJ, Altman DG, McKenzie JE, Shamseer L, Ahmadzai N, Wolfe D, et al. Flaws in the application and interpretation of statistical analyses in systematic reviews of therapeutic interventions were common: a cross-sectional analysis. J Clin Epidemiol. 2018 Mar;95:7-18. doi: 10.1016/j.jclinepi.2017.11.022. 5. Saltaji H, Cummings GG, Armijo-Olivo S, Major MP, Amin M, Major PW, et al. A descriptive analysis of oral health systematic reviews published 1991-2012: cross sectional study. PLoS One. 2013 Sep;8(9):e74545. doi: 10.1371/journal.pone.0074545. 6. Jayaraman J, Nagendrababu V, Pulikkotil SJ, Innes NP. Critical appraisal of methodological quality of Systematic Reviews and Meta-analysis in Paediatric Dentistry journals. Int J Paediatr Dent. 2018 Nov;28(6):548-560. doi: 10.1111/ipd.12414. 11 Lemes et al. 7. Bassani R, Pereira GKR, Page MJ, Tricco AC, Moher D, Sarkis-Onofre R. Systematic reviews in dentistry: Current status, epidemiological and reporting characteristics. J Dent. 2019 Mar;82:71-84. doi: 10.1016/j.jdent.2019.01.014. 8. Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev. 2015 Jan;4(1):1. doi: 10.1186/2046-4053-4-1. 9. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JP, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ. 2009 Jul;339:b2700. doi: 10.1136/bmj.b2700. 10. Page MJ, Shamseer L, Altman DG, Tetzlaff J, Sampson M, Tricco AC, et al. Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study. PLoS Med. 2016 May;13(5):e1002028. doi: 10.1371/journal.pmed.1002028. 11. Kattan S, Lee SM, Kohli MR, Setzer FC, Karabucak B. methodological quality assessment of meta-analyses in endodontics. J Endod. 2018 Jan;44(1):22-31. doi: 10.1016/j.joen.2017.07.019. 12. El-Rabbany M, Li S, Bui S, Muir JM, Bhandari M, Azarpazhooh A. A Quality analysis of systematic reviews in dentistry, part 1: meta-analyses of randomized controlled trials. J Evid Based Dent Pract. 2017 Dec;17(4):389-98. doi: 10.1016/j.jebdp.2017.06.004. 13. Thornton A, Lee P. Publication bias in meta-analysis: its causes and consequences. J Clin Epidemiol. 2000 Feb;53(2):207-16. doi: 10.1016/s0895-4356(99)00161-4. 14. Koletsi D, Valla K, Fleming PS, Chaimani A, Pandis N. Assessment of publication bias required improvement in oral health systematic reviews. J Clin Epidemiol. 2016 Aug;76:118-24. doi: 10.1016/j.jclinepi.2016.02.019. 15. Tricco AC, Cogo E, Page MJ, Polisena J, Booth A, Dwan K, et al. A third of systematic reviews changed or did not specify the primary outcome: a PROSPERO register study. J Clin Epidemiol. 2016 Nov;79:46-54. doi: 10.1016/j.jclinepi.2016.03.025.