Skip to main content

Increasing retractions of meta-analyses publications for methodological flaw

Abstract

Purpose of this letter was to explore the trends regarding methodological flaws of systematic review and meta-analyses (SRMAs) based on retraction notes in the past decades, and the categories of reasons for the retractions. Content analysis with descriptive statistics, Cochran Q test, and multinomial logistic regression were used. Based on 187 records of retracted SRMAs, retraction announcements can be categorized into academic ethical violation, methodological flaw, and writing or reporting problem. The numbers of academic ethical violation were significantly higher than those with methodological flaw (z = 3.51; p < 0.01) or writing problem (z = 8.58; p < 0.001). The numbers of methodological flaw were also higher than that with writing problem (z = 6.47; p < 0.001). Moreover, an increased proportion of methodological flaw was observed since 2006, and the retraction year was significantly associated with increased proportion of methodological flaw when academic ethical violation as the reference group.

Peer Review reports

Background

There were significant increases in the publications of systematic reviews and meta-analyses (SRMAs) during past decade, but some meta-analyses seemed to be flawed and non-informative synthesizing [1, 2]. Mass productions of SRMAs raise concerns [1,2,3], and flaws of SRMAs have received scholarly attention [1]. Inappropriate data syntheses may lead to research waste [4], and result in retraction due to unreliable findings and conclusions. It is the purpose of this letter to review the methodological flaws of SRMAs retracted and the retraction notes for the understanding of methodological flaws in syntheses.

Methods

The present letter searched references from three databases using relevant keywords before June 2021 (Additional file 1). Two authors independently double-checked the eligibility of references, information extraction, and classifications of reasons for retractions. They extracted authors’ name, retraction year, journal, and the reasons for retractions. There was no predefined frame for the classification of the retraction reasons, and the categories were constructed through content analysis. Since some retraction notes concurrently covered two or more categories, the classifications were recorded as multiple paired binary variables. R version 4.0.3 was used for analysis. Cochran Q test with adjusted P value was done for comparing counts among categories of the reasons of retractions using command “pairwiseMcNemar” in “RVAideMemoire” package. To test whether retraction year was accounted for the difference in reasons for retractions, multinomial logistic regression was further performed using the command “multinom” in “nnet” package.

Results

A total of 198 retracted meta-analyses were identified (Additional file 2), but 11 retracted records were not included in the present analysis because there was no retraction reason (i = 6) or special cases (i = 5). In those five special cases, the SRMAs were not problematic because one or some of the original researches in those syntheses were retracted after the SRMA were published. Based on the remaining 187 records, retraction announcements can be categorized into academic ethical violation, methodological flaw, and writing or reporting problem (Additional files 3 and 4). The most common reason for retractions was academic ethical violation (i = 118, 69.82%) followed by methodological flaw (i = 72, 42.60%), and writing or reporting problem (i = 19, 11.24%; Additional file 5). The numbers of academic ethical violation were significantly higher than those with methodological flaw (z = 3.51; p < 0.01) or writing problem (z = 8.58; p < 0.001). The numbers of methodological flaw were also higher than that with writing problem (z = 6.47; p < 0.001; Additional file 6). Moreover, it is observed that there was an increased proportion of methodological flaw since 2006. In the other way, increased proportion of methodological flaw was significantly associated with the retraction year when academic ethical violation was used as the reference group (Fig. 1).

Fig. 1
figure 1

Multinomial logistic regression of retraction year on reasons of retraction. CI, 95% confidence interval; E, academic ethical violation; M, methodological flaw; W, writing or reporting problem

Discussion

The most common reasons for retractions of meta-analyses were academic ethical violation and methodological flaw. Academic ethical violation for retraction of SRMAs publications is gradually decreasing, while methodological flaw for retractions is increasing in the recent years, particularly after 2015. These trends may be due to rapid development and maturing of methodology in evidence synthesis. For instance, methodology of SRMA becomes more familiar after 2009 due to newer version of the Cochrane Handbook and the PRISMA guideline [5, 6]. With the increased understanding of methodology of SRMA in academic society, scholars in recent years may identify methodological flaws in SRMAs more than before. Involvement of nonconflicting meta-analysis experts in research team or inviting nonconflicting meta-analysis experts to take part in the peer review are still important because experts in SRMA may improve the quality control of meta-analysis publications [3]. Though the official announcement of retraction is objective, however, it is difficult to understand the complexity of the context behind the words. Therefore, the real reason behind the official statement may be limited in this study. Further qualitative or mixed method studies are foreseeably valuable in the future.

Availability of data and materials

All data generated or analyzed during this study are included in this published article.

Abbreviations

SRMA:

Systematic review and meta-analysis

References

  1. Ioannidis JP. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Milbank Q. 2016;94(3):485–514.

    Article  Google Scholar 

  2. Niforatos JD, Weaver M, Johansen ME. Assessment of publication trends of systematic reviews and randomized clinical trials, 1995 to 2017. JAMA Intern Med. 2019;179(11):1593–4.

    Article  Google Scholar 

  3. Wallach JD. Meta-analysis Metastasis. JAMA Intern Med. 2019;179(11):1594–5. https://doi.org/10.1001/jamainternmed.2019.2999.

  4. Chalmers I, Bracken MB, Djulbegovic B, et al. How to increase value and reduce waste when research priorities are set. Lancet (London, England). 2014;383(9912):156–65.

    Article  Google Scholar 

  5. Moher D, Liberati A, Tetzlaff J, et al. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097.

    Article  Google Scholar 

  6. Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA. Cochrane Handbook for Systematic Reviews of Interventions. 2nd Edition. Chichester: Wiley; 2019.

Download references

Acknowledgements

Not applicable.

Funding

This study did not receive any funding.

Author information

Authors and Affiliations

Authors

Contributions

Conceptualization: Yi-No Kang and Kee-Hsin Chen. Data curation: Chia-Yun Chen and Kee-Hsin Chen. Formal analysis: Yi-No Kang. Investigation: Chia-Yun Chen, Yi-No Kang, and Kee-Hsin Chen. Methodology: Yi-No Kang, Ken N. Kuo, and Paul Glasziou. Supervision: Ken N. Kuo and Paul Glasziou. Visualization: Yi-No Kang. Writing—original draft: Chia-Yun Chen and Yi-No Kang. Writing—review and editing: Ken N. Kuo, Paul Glasziou, and Kee-Hsin Chen. The authors read and approved the final manuscript.

Corresponding authors

Correspondence to Paul Glasziou or Kee-Hsin Chen.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1.

Database and search strategy.

Additional file 2.

Flow diagram of study selection.

Additional file 3.

Examples of category of problems.

Additional file 4.

Reference characteristics of retracted meta-analyses before June 2021 (i=198).

Additional file 5.

Doughnut plot of clarity of reasons for retractions of meta-analyses.

Additional file 6.

Plot of Cochran Q test for three categories of retraction reason.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, CY., Kang, YN., Kuo, K.N. et al. Increasing retractions of meta-analyses publications for methodological flaw. Syst Rev 10, 267 (2021). https://doi.org/10.1186/s13643-021-01822-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s13643-021-01822-2