- Open Access
Increasing retractions of meta-analyses publications for methodological flaw
Systematic Reviews volume 10, Article number: 267 (2021)
Purpose of this letter was to explore the trends regarding methodological flaws of systematic review and meta-analyses (SRMAs) based on retraction notes in the past decades, and the categories of reasons for the retractions. Content analysis with descriptive statistics, Cochran Q test, and multinomial logistic regression were used. Based on 187 records of retracted SRMAs, retraction announcements can be categorized into academic ethical violation, methodological flaw, and writing or reporting problem. The numbers of academic ethical violation were significantly higher than those with methodological flaw (z = 3.51; p < 0.01) or writing problem (z = 8.58; p < 0.001). The numbers of methodological flaw were also higher than that with writing problem (z = 6.47; p < 0.001). Moreover, an increased proportion of methodological flaw was observed since 2006, and the retraction year was significantly associated with increased proportion of methodological flaw when academic ethical violation as the reference group.
There were significant increases in the publications of systematic reviews and meta-analyses (SRMAs) during past decade, but some meta-analyses seemed to be flawed and non-informative synthesizing [1, 2]. Mass productions of SRMAs raise concerns [1,2,3], and flaws of SRMAs have received scholarly attention . Inappropriate data syntheses may lead to research waste , and result in retraction due to unreliable findings and conclusions. It is the purpose of this letter to review the methodological flaws of SRMAs retracted and the retraction notes for the understanding of methodological flaws in syntheses.
The present letter searched references from three databases using relevant keywords before June 2021 (Additional file 1). Two authors independently double-checked the eligibility of references, information extraction, and classifications of reasons for retractions. They extracted authors’ name, retraction year, journal, and the reasons for retractions. There was no predefined frame for the classification of the retraction reasons, and the categories were constructed through content analysis. Since some retraction notes concurrently covered two or more categories, the classifications were recorded as multiple paired binary variables. R version 4.0.3 was used for analysis. Cochran Q test with adjusted P value was done for comparing counts among categories of the reasons of retractions using command “pairwiseMcNemar” in “RVAideMemoire” package. To test whether retraction year was accounted for the difference in reasons for retractions, multinomial logistic regression was further performed using the command “multinom” in “nnet” package.
A total of 198 retracted meta-analyses were identified (Additional file 2), but 11 retracted records were not included in the present analysis because there was no retraction reason (i = 6) or special cases (i = 5). In those five special cases, the SRMAs were not problematic because one or some of the original researches in those syntheses were retracted after the SRMA were published. Based on the remaining 187 records, retraction announcements can be categorized into academic ethical violation, methodological flaw, and writing or reporting problem (Additional files 3 and 4). The most common reason for retractions was academic ethical violation (i = 118, 69.82%) followed by methodological flaw (i = 72, 42.60%), and writing or reporting problem (i = 19, 11.24%; Additional file 5). The numbers of academic ethical violation were significantly higher than those with methodological flaw (z = 3.51; p < 0.01) or writing problem (z = 8.58; p < 0.001). The numbers of methodological flaw were also higher than that with writing problem (z = 6.47; p < 0.001; Additional file 6). Moreover, it is observed that there was an increased proportion of methodological flaw since 2006. In the other way, increased proportion of methodological flaw was significantly associated with the retraction year when academic ethical violation was used as the reference group (Fig. 1).
The most common reasons for retractions of meta-analyses were academic ethical violation and methodological flaw. Academic ethical violation for retraction of SRMAs publications is gradually decreasing, while methodological flaw for retractions is increasing in the recent years, particularly after 2015. These trends may be due to rapid development and maturing of methodology in evidence synthesis. For instance, methodology of SRMA becomes more familiar after 2009 due to newer version of the Cochrane Handbook and the PRISMA guideline [5, 6]. With the increased understanding of methodology of SRMA in academic society, scholars in recent years may identify methodological flaws in SRMAs more than before. Involvement of nonconflicting meta-analysis experts in research team or inviting nonconflicting meta-analysis experts to take part in the peer review are still important because experts in SRMA may improve the quality control of meta-analysis publications . Though the official announcement of retraction is objective, however, it is difficult to understand the complexity of the context behind the words. Therefore, the real reason behind the official statement may be limited in this study. Further qualitative or mixed method studies are foreseeably valuable in the future.
Availability of data and materials
All data generated or analyzed during this study are included in this published article.
Systematic review and meta-analysis
Ioannidis JP. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Milbank Q. 2016;94(3):485–514.
Niforatos JD, Weaver M, Johansen ME. Assessment of publication trends of systematic reviews and randomized clinical trials, 1995 to 2017. JAMA Intern Med. 2019;179(11):1593–4.
Wallach JD. Meta-analysis Metastasis. JAMA Intern Med. 2019;179(11):1594–5. https://doi.org/10.1001/jamainternmed.2019.2999.
Chalmers I, Bracken MB, Djulbegovic B, et al. How to increase value and reduce waste when research priorities are set. Lancet (London, England). 2014;383(9912):156–65.
Moher D, Liberati A, Tetzlaff J, et al. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097.
Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA. Cochrane Handbook for Systematic Reviews of Interventions. 2nd Edition. Chichester: Wiley; 2019.
This study did not receive any funding.
Ethics approval and consent to participate
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Additional file 1.
Database and search strategy.
Additional file 2.
Flow diagram of study selection.
Additional file 3.
Examples of category of problems.
Additional file 4.
Reference characteristics of retracted meta-analyses before June 2021 (i=198).
Additional file 5.
Doughnut plot of clarity of reasons for retractions of meta-analyses.
Additional file 6.
Plot of Cochran Q test for three categories of retraction reason.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
About this article
Cite this article
Chen, CY., Kang, YN., Kuo, K.N. et al. Increasing retractions of meta-analyses publications for methodological flaw. Syst Rev 10, 267 (2021). https://doi.org/10.1186/s13643-021-01822-2