Does haste make waste? Prevalence and types of errors reported after publication of studies of COVID-19 therapeutics

Abstract

Background

The COVID-19 pandemic spurred publication of a rapid proliferation of studies on potential therapeutic agents. While important for the advancement of clinical care, pressure to collect, analyze, and report data in an expedited manner could potentially increase the rate of important errors, some of which would be captured in published errata. We hypothesized that COVID-19 therapeutic studies published in the early years of the pandemic would be associated with a high rate of published errata and that, within these errata, there would be a high prevalence of serious errors.

Methods

We performed a review of published errata associated with empirical studies of COVID-19 treatments. Errata were identified via a MEDLINE and Embase search spanning January 2020 through September 2022. Errors located within each published erratum were characterized by location within publication, error type, and error seriousness.

Results

Of 47 studies on COVID-19 treatments with published errata, 18 met inclusion criteria. Median time from publication of the original article to publication of the associated erratum was 76 days (range, 12–511 days). A majority of errata addressed issues with author attribution or conflict of interest disclosures (39.5%) or numerical results (25.6%). Only one erratum contained a serious error: a typographical error which could have misled readers into believing that the treatment in question had serious adverse effects when in fact it did not.

Conclusions

Despite accelerated publication times, we found among studies of COVID-19 treatments the majority of errata (17/18) reported minor errors that did not lead to misinterpretation of the study results. Retractions, an indicator of scientific misdirection even more concerning than errata, were beyond the scope of this review.

Introduction

Publication of biomedical research in scientific journals is the primary means of disseminating scientific findings to the research community and the public [1]. The goal is to accurately describe and interpret research findings, but mistakes are inevitable. Errors range from simple typographical glitches to more serious coding, arithmetic, or statistical errors that distort the study findings and lead to erroneous conclusions. Before publication, biomedical research reporting errors can be intercepted by authors, reviewers, or editors. Afterwards, the onus is on authors, journal editorial staff, and astute readers; noteworthy errors detected in this way are reported in the form of published errata, corrections, or corrigendum. While few studies have evaluated the incidence of such correctives, one report limited to general internal medicine and cardiology indicates they may be as frequent as 4.2 per 100 published articles [2].

In considering factors that may contribute to errors in the scientific record, one culprit may be pressure to communicate information rapidly, as might be seen when two competing research groups are rushing to claim a scientific discovery [3] or when external circumstances demand rapid reporting to support critical public health needs. The first 2 years of the COVID-19 pandemic created significant demand for research on therapies meant to inhibit the virus or mitigate the severity of clinical illness. Given the desperate need to understand the pathogenesis of COVID-19 and identify effective treatments, publications underwent rapid editorial review, with one international study reporting a nearly tenfold reduction in time from submission to acceptance for COVID-related compared with non-COVID related articles (11.3 versus 106 days) [4]. This accelerated schedule may have decreased the rigor of peer review.

Although several studies have evaluated retractions of COVID-19 publications [5,6,7,8], few have examined errors identified in published errata. We sought to characterize published errata related to COVID-19 treatment research and determine if compressed review times reported during the pandemic were associated with a higher prevalence and severity of error. We expected to find both a high rate of published errata and a high proportion of serious errors within published errata.

Methods

We conducted a systematic review of errata published between January 1, 2020, and September 1, 2022, that were associated with articles reporting empirical studies of COVID-19 treatments. Errata published in English and related to COVID-19 therapeutics were identified on MEDLINE and Embase using the search terms COVID-19, erratum, corrigendum, and correction (the complete list of search terms, prepared with the assistance of a reference librarian, is provided in Additional file 1). Article-erratum pairs were included if the article’s topic was an empiric evaluation of COVID-19 treatment including head-to-head comparisons of treatment modalities, treatment versus placebo, or treatment versus standard of care or supportive care. Study types included clinical trials, observational studies, and systematic reviews. Articles were excluded if they investigated the treatment of another disease process during the COVID-19 pandemic, COVID-19 diagnostics, epidemiologic studies of the prevalence of treatment, or were editorial or opinion pieces on COVID-19 that did not present an empirical evaluation of a treatment.

Errata were categorized by location of the error in the original publication (abstract, body of the article, tables or figures, references, or supplemental material) and error type (author disclosure or conflict of interest, author attribution, numerical or statistical error, textual error, or citation error). Errors were then categorized as serious or minor. Errors were defined as serious if they (1) affected the study results in a way that could lead to erroneous conclusions by the author(s) or readers or (2) were found in the abstract or title, increasing the risk of error propagation. Two independent reviewers screened articles for inclusion and categorized errors identified in the errata. When discrepancies between reviewers arose (n = 3), the articles were jointly re-reviewed; all three were excluded by consensus. Data was managed in a REDCap [9] database and descriptive statistics were used to analyze data.

Results

The initial search produced 171 results; however, on review, 124 were removed because they (1) lacked an associated erratum (n = 112), (2) represented duplicate entries (n = 8), or (3) consisted of abstracts without full articles (n = 4) (Fig. 1). Of the remaining 47 article-erratum pairs, 18 articles (19 associated errata) met inclusion criteria (1 article had 2 separate published errata). Fifteen of the 18 studies were randomized clinical trials [10,11,12,13,14,15,16,17,18,19,20,21,22,23,24], two were systematic reviews [25, 26], and one was an observational study [27].

Fig. 1

Flow diagram of article selection for study inclusion

Among the 19 errata, 55% addressed 1 error within the published article, 28% 2 to 4 errors, and 17% ≥ 5 errors. Higher impact journals appeared to be somewhat over-represented (Table 1). Two thirds had first authors based outside of Europe or North America. Errors were most often found in the abstract, title, or byline and frequently involved author misattribution (16.3%) or failure to disclose potential conflicts of interest (23.3%) (Table 2). Median time from publication of the original article to publication of the associated erratum was 76 days (range 12 to 511 days). Among the errors identified, only one was characterized as serious (Table 3). This serious error was a typographical error in the Results section of the abstract. The sentence read, “There was severe adverse event recorded in the study group”; however, the correct statement was “There was no severe adverse event recorded in the study group.” This error would have led readers to question the safety of the study treatment, if the reader did not further investigate the body of the article which contained the correct results for the treatment group.

Table 1 Characteristics of published articles and errata on COVID-19 therapeutics research articles

Table 2 Characteristics of errors reported in published errata on COVID-19 therapeutics research articles

Table 3 Sample of errors contained within errata for COVID-19 therapeutics articles

Discussion

In this study of published errata associated with articles on COVID-19 therapeutics, most reported errors were minor. Consistent with other studies, about half of detected errors were corrected in the original article’s online publication [2]. Many of these errors were related to author misattribution or failure to disclose conflict of interest. Others involved numerical errors that were primarily located in tables and figures. One serious error among 19 errata yields a serious error rate of 5%, which may lend cause for concern. This error, located in the abstract, misrepresented that the treatment group had serious adverse events, when in fact there were none. While we concede that therapeutic decisions are infrequently made on the basis of a single scientific study, a clinician quickly reviewing the abstract could have been dissuaded from pursuing what was found to be an effective COVID-19 therapeutic due to concerns of misreported adverse events with the treatment.

There have been several prior publications exploring reported error rates and seriousness of those errors. These studies have largely examined specific journals and included articles of all topics/types (Table 4). We observed a lower rate of serious errors (5%) than a study that reviewed randomized control trials from four high-impact journals (10%) [28]. When reviews expanded to include all study types, serious errors ranged from 14 to 25% of published errata [2, 29, 30]. One review of five radiology imaging journals found a very low overall errata rate (< 2%), with 6% of those errors found to be serious [31]. Another review dedicated to errors in authorship points out that each author is expected to proofread the manuscript prior to publication and thus errors related to name misspelling should be easily identified and corrected prior to publication [32]. That we found several errata related to misattribution of authorship—which ought to have been uncovered and corrected during the pre-publication process—raises the question as to whether other, less obvious errors were not identified. Additionally, this finding raises questions about the diligence of co-authors during the final pre-submission period.

Table 4 Summary of prior studies evaluating errata error rate in various journals and subject matters

The process of scientific publication is supposed to be self-correcting. Our findings suggest that despite the additional pressure to rapidly disseminate research on COVID-19 treatment, there is not a high rate of errors that would change interpretation of study results or conclusions. However, we are only able to capture errors that are published in errata; other errors may remain. Thus, our results might underestimate the true number of errors in publications related to COVID-19 therapeutics. In addition, article retractions, representing another major category of error (or malfeasance), were beyond the scope of this review. As reported by Peterson et al., COVID-19 articles have disproportionately retracted over the time span of the pandemic [34].

The COVID-19 pandemic was a sudden, disruptive force to the scientific publication process. We hypothesized that the unprecedented volume of research on a novel disease, the urgent need to disseminate findings, and necessity for accurate and high-quality data would have stress-tested the editorial and review processes leading to a noticeable increase in serious errors. Reassuringly, our findings align with error rates reported among studies examining targeted journals which included all study topics. This finding suggests that journals have been able to maintain publication standards for COVID-19 therapeutics during the pandemic. Increased transparency of changes to a journal’s editorial evaluation of COVID-19 research would be helpful in understanding the resources needed and potential for burnout among editors and reviewers. In addition, future research is warranted to evaluate the sustainability of the recent changes to the editorial and review processes and determine whether new approaches to the publication process could translate to non-COVID-19 research.