Skip to main content
Download PDF

Spin on adverse effects in abstracts of systematic reviews of orthodontic interventions: a cross-sectional study (part 2)

Systematic Reviews volume 12, Article number: 99 (2023) Cite this article

A Correction to this article was published on 19 March 2024

This article has been updated

Abstract

Background

It is critical that abstracts of systematic reviews transparently report both the beneficial and adverse effects of interventions without misleading the readers. This cross-sectional study assessed whether adverse effects of interventions were reported or considered in abstracts of systematic reviews of orthodontic interventions and whether spin on adverse effects was identified when comparing the abstracts with what was sought and reported in these reviews.

Methods

This cross-sectional study (part 2 of 2) used the same sample of 98 systematic reviews orthodontic interventions as used in part 1. Eligible reviews were retrieved from the Cochrane Database of Systematic Reviews and the 5 leading orthodontic journals between August 1 2009 and July 31 2021. Prevalence proportions were sought for 3 outcomes as defined in the published protocol. Univariable logistic regression models were built to explore associations between the presence of spin in the abstract and a series of predictors. Odds ratios (OR) 95% confidence intervals (95% CI) were used to quantify the strength of associations and their precision.

Results

76.5% (75/98) of eligible reviews reported or considered (i.e., discussed, weighted etc.) potential adverse effects of orthodontic interventions in the abstract and the proportion of spin on adverse effects was 40.8% (40/98) in the abstract of these reviews. Misleading reporting was the predominant category of spin, i.e., 90% (36/40). Our explorative analyses found that compared to the Cochrane Database of Systematic Reviews all 5 orthodontic journals had similar odds of the presence of spin on adverse effects in abstracts of systematic reviews of orthodontic interventions. The odds of the presence of spin did not change over the sampled years (OR: 1.03, 95% CI: 0.9 to 1.16) and did not depend on the number of authors (OR: 0.93, 95% CI: 0.71 to 1.21), or on the type of orthodontic intervention (OR: 1.1, 95% CI: 0.45 to 2.67), or whether conflicts of interests were reported (OR: 0.74, 95% CI: 0.32 to 1.68).

Conclusion

End users of systematic reviews of orthodontic interventions have to be careful when interpreting results on adverse effects in the abstracts of these reviews, because they could be jeopardized by uncertainties such as not being reported and misleading reporting as a result of spin.

Peer Review reports

Background

Abstracts should provide key information on a research study, which helps readers decide whether or not to access the full report [1]. It is therefore critical that abstracts transparently report the results of both the beneficial and adverse effects of interventions without misleading the readers. Misleading reporting, misleading interpretation, and misleading extrapolation of study results has been called “spin” [2, 3]. In this study, we assessed whether adverse effects of interventions were reported or considered in abstracts of both Cochrane and non-Cochrane reviews of orthodontic interventions and whether spin and what type of spin regarding adverse effects was present when comparing the abstracts with what was sought and reported in these reviews.

Titles and abstracts of publications of healthcare interventions are used for multiple purposes such as (1) an initial screening of the study type; (2) clarifying the included type of patients, interventions, comparators, outcomes, and settings; (3) obtaining a summary of the findings; and (4) an initial assessment of the validity of the study [1, 4, 5]. Titles and abstracts are the most and often only read sections of biomedical papers, because of a lack of time of readers, paywalls, or language issues [6]. It is therefore important that abstracts can be used as stand-alone documents that clearly and truthfully reflect what was reported in the full text [7]. The standard for Methodological Expectations of Cochrane Intervention Reviews (MECIR) [8] states under Item R13 that “The abstract of the review should aim to reflect a balanced summary of the benefits and harms of the intervention” and this is a “mandatory” Cochrane review standard. The inclusion of ‘adverse effects’ in this standard is crucial, because these effects are often poorly assessed and reported in clinical trials and systematic reviews of healthcare interventions [9,10,11,12,13,14,15,16].

In this context, it is important that findings on adverse effects are presented accurately in the abstract without misleading the reader. “A distorted presentation of study results” has been called “spin” [2, 3]. This definition and other commonly used definitions of spin and key terminology used in this article are listed in Table 1 [2, 3, 17,18,19,20,21,22,23,24,25]. Spin has been subdivided in 3 categories: “misleading reporting,” “misleading interpretations,” and “misleading extrapolations” of study results [2]. We adopted the definitions by Lazarus et al. [23] for these 3 categories of spin (Table 1). Controlling spin is important, because of its high prevalence and its consequences. For example, a randomized controlled trial (RCT) showed that spin in abstracts can influence the clinician’s interpretation of the results of a study [26]. Further, Yavchitz et al. [27] showed that the presence of spin in press releases and the mass media was related with spin in the conclusions of the pertinent abstracts of peer-reviewed RCTs. Our scoping searches showed that a high prevalence of spin has been recorded in abstracts of numerous research studies and for a wide variety of disciplines. For example, spin was present in 84% (107/128) of abstracts of reports of non-randomized studies assessing an intervention [23], 23% (24/105) of abstracts of RCTs in rheumatology [28], 57% (53/93) of abstracts of cardiovascular RCT reports [29], 34.2% (25/73) of abstracts of systematic reviews and meta-analyses related to treatment of proximal humerus fractures [30], 37.6% (27/72) of results, and 58.3% (42/72) of conclusions of abstracts of parallel-group RCTs with statistically non-significant results (P ≥ 0.05) [25]. Spin in abstracts of orthodontic studies was assessed in 2 recent publications [31, 32]. Guo et al. [31] found spin in 62.2% (69/111) of abstracts of parallel-group RCTs with clearly stated statistically non-significant primary outcomes and Makou et al. [32] identified spin in 48.6% (53/109) of abstracts of orthodontic meta-analyses.

Table 1 Glossary of terms
Full size table

This is part 2 of 2 cross-sectional studies on assessing and reporting of adverse effects in systematic reviews of orthodontic interventions published in 5 leading orthodontic journals and in the Cochrane Database of Systematic Reviews. Part 1 focused predominantly on seeking and reporting of adverse effects in the main text and supplementary files of these reviews [18, 33]. In part 2, we assessed whether adverse effects of orthodontic interventions were reported or considered (i.e., discussed, weighed etc.) in abstracts of these reviews. We further measured whether spin was introduced in the abstract regarding information on adverse effects as found and reported in these reviews. We also assessed the different categories of spin. The findings of this research study are important not only for patients and clinicians but also for researchers, peer reviewers, and editors because they have a crucial role in reducing the prevalence of spin [34].

Objectives

Our objectives were presented in the following 3 research questions [24]. Recent (up to October 31, 2021) scoping searches showed that these questions were not assessed previously.

  • Question 1. In abstracts of systematic reviews of orthodontic interventions, were potential adverse effects of these interventions reported or considered (i.e., discussed, weighed etc.)?

  • Question 2. Was spin identified on adverse effects of orthodontic interventions in the abstract?

  • Question 3. What type of spin was identified on adverse effects of orthodontic interventions in the abstract?

Methods

This manuscript reports the methods and results of part 2 of a cross-sectional study using the same 98 eligible reviews as in part 1 [33]. Additional information on the research methods and the characteristics of the included reviews can be found in part 1 [33] and in the published protocols of parts 1 and 2 [18, 24]. The protocol for this second cross-sectional study was published in “Research Integrity and Peer Review” [24] and can be consulted via the following link: https://researchintegrityjournal.biomedcentral.com/articles/10.1186/s41073-019-0084-4.

The checklist of the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement for cross-sectional studies [35] was included as Additional file 1. The differences between the methods planned in our protocol and those implemented in the final study were reported in Additional file 2. The rationales for these differences were also given. All raw data were reported in the Open Science framework Open Science Framework (https://osf.io/ka7mp/). There was no patient or public involvement during the development of the protocol or in the conduct of this study. The eligibility criteria, information sources, search strategy, and selection process used in part 1 of this cross-sectional study [33] were also used for part 2 of this study. To reduce the need of cross-checking between manuscripts, we reported these sections again.

Eligibility criteria

The eligibility criteria were published previously in our protocol [24] and in part 1 of this study [33] and were developed by two researchers (PS and RMR). These criteria are presented in Table 2 [36] and are further explained under here.

Table 2 Eligibility criteria
Full size table

Study designs

  • We included systematic reviews of orthodontic interventions. The definitions of the terms “systematic review,” “intervention review,” and “orthodontic interventions” listed in the Glossary of terms (Table 1) were used to assess eligibility.

  • The following reviews were excluded: (1) noninterventional reviews such as “Methodology,” “Diagnostic,” “Qualitative,” and “Prognostic”; (2) rapid and scoping reviews; (3) systematic reviews with Bayesian network meta-analysis; and (4) systematic reviews of interventions that did not find any eligible studies (empty reviews).

Participants

  • Systematic reviews of interventions on any type of patients undergoing orthodontic interventions, i.e., patients of any health status, sex, age, and demographics, and socio-economic status were eligible.

  • Intervention reviews that focused exclusively on patients with congenital anomalies, for example, with cleft lip and palate and systematic reviews of animal or laboratory studies were excluded.

Interventions

  • Systematic reviews on the following interventions were eligible: (1) systematic reviews that assessed the effects of clinical orthodontic interventions. Clinical orthodontic interventions refer to any type of orthodontic appliance that are used to move teeth or change the jaw size or position for orthodontic purposes; (2) systematic reviews of interventions with appliances to maintain or stabilize the outcomes of orthodontic treatment, for example, retainers; (3) systematic reviews of orthodontic interventions that compared the effects of orthodontic treatment with or without additional interventions such as pharmacological or small surgical interventions, e.g., periodontal or implant surgery; and (4) no exclusion criteria were applied to the characteristics of the operator who conducted the interventions.

  • Systematic reviews on the following interventions were excluded: (1) systematic reviews in which patients receive orthodontic treatment, but in which the effects of other interventions, e.g., periodontal surgery, were compared and not the effects of orthodontic interventions; (2) systematic reviews of interventions in which orthodontic appliances were specifically used for other purposes, e.g., changing jaw positions to treat respiration or temporomandibular disorders; (3) systematic review of orthodontic interventions that included orthognathic surgery; (4) systematic reviews that focused exclusively on adverse effects of interventions; and (5) systematic reviews that did not assess a specific orthodontic intervention but referred to orthodontic treatment as a whole.

Outcomes

  • Systematic reviews of orthodontic interventions that assessed any adverse effect of orthodontic interventions scored at any endpoint or timing were eligible. The effects of orthodontic interventions did not refer just to outcomes related to tooth and jaw size and positions but also to broader outcomes such as periodontal health, esthetic changes, the health of the temporomandibular joint, patient health experiences, and economic issues associated with the interventions. The reporting of outcomes on adverse effects did not determine eligibility of reviews for this cross-sectional study, i.e., reviews were not excluded because they did not report measured outcome data in a “usable” way [36].

  • No exclusion criteria regarding the outcomes of systematic reviews of orthodontic interventions were applied.

Setting

  • Systematic reviews of orthodontic interventions that reported on interventions conducted in any type of setting, i.e., university or private practice, were eligible.

Information sources and search strategy

The Cochrane Database of Systematic reviews [37] and the websites of 5 leading orthodontic journals were the information sources of this study. The journal selection of the latter journals was based on two criteria: (1) the journal has been published for 10 years or more and (2) the highest impact factor. The following 5 orthodontic journals fulfilled these criteria: European Journal of Orthodontics [EJO], American Journal of Orthodontics and Dentofacial Orthopedics [AJODO], Angle Orthodontist (AO), The Korean Journal of Orthodontics (KJO), and Orthodontics and Craniofacial Research (OCR). These journals were manually searched from August 1, 2009, until July 31, 2021, for systematic reviews that fulfilled the eligibility criteria. August 1, 2019, was chosen as the starting date, because it coincides with the launch of the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement on 21 July 2009 [38, 39].

Study records

Selection process

Two reviewers (PS and RMR) manually searched systematic reviews that fulfilled the eligible criteria. Pilot tests were conducted a priori to train both reviewers and to calibrate them. All titles and abstracts in the websites of the 5 orthodontic journals were hand-searched for eligible reviews. Eligible Cochrane reviews were searched in the section “Dentistry and Oral health” in the Cochrane library. Only the latest version of a review was eligible when review updates had been published. In the case of disagreement on the selection procedures, the following strategies were implemented and in this sequence: (1) discussions between these operators, (2) rereading the paper, (3) contacting of authors by email to clarify issues regarding a specific manuscript. Persistent disagreements were resolved through consultation with a methodologist. A total of 98 eligible systematic reviews of orthodontic interventions was identified in part 1 of this study [33]. This same sample of 98 studies was also used in this study.

Data collection procedures

All 98 eligible reviews together with their supplemental files were merged into binder PDFs, and according to protocol [24], pertinent search terms were linked to these documents to facilitate data extraction (Additional file 2). Our pilot-tested data collection forms were used to extract data and are given in Additional file 2. Data items were collected from the entire eligible review, i.e., the entire manuscript including the abstract, tables, figures, and additional files. We implemented this procedure for all eligible reviews but did not extract data from the plain language summary of eligible Cochrane reviews. Two calibrated authors (PS and RMR) independently collected data from the 98 eligible reviews to address the research questions. In the case of disagreement, we applied the same strategies as reported in the section “Selection process” and the third author (NDG) was consulted in the case of persistent disagreements.

Assessing adverse effects of orthodontic interventions

Pain and the various categories of adverse effects hypothetically linked to orthodontic interventions as defined by Preoteasa et al. [40] and modified by Steegmans et al. [33] were reported in a table in Additional file 2. This table was consulted as our reference to assess the reporting on adverse effects in the abstract. When additional adverse effects were identified that were not given in this table, we included them with rationale. Effects that could be labeled either as “beneficial” or “adverse” were not included unless the review authors labeled these ambiguous effects as “adverse.” Explanations for such decisions were given. Orthodontic interventions were classified in three types, i.e., type 1: orthodontic interventions to move teeth or change the jaw size or position for orthodontic purposes, type 2: orthodontic interventions with additional surgical, pharmacological, or vibratory interventions, and type 3: orthodontic interventions to maintain or stabilize orthodontic results.

Assigning spin of adverse effects of orthodontic interventions in abstracts of systematic reviews

Spin was assigned by comparing whether what was reported in the abstract on adverse effects of orthodontic interventions was congruent with the findings on these effects in the review. Three types of spin were assigned i.e., misleading reporting, misleading interpretation, and misleading (inappropriate) extrapolation on adverse effects of orthodontic interventions in abstracts of systematic reviews [27]. To facilitate this assignment and to reduce the risk of misinterpretation, each type of spin was subdivided in categories. The presence of spin was assigned when it was identified in one or more of these categories. Spin was assessed in all eligible reviews irrespective of whether these reviews sought adverse effects of interventions or not. Because the pilot tests for our protocol identified only 2 reviews with spin [24] and because assessing spin is not easy [22], we conducted additional pilot tests on 10 RCTs to further calibrate the operators (PS and RMR) that assigned spin and to fine-tune the descriptions of spin and the checklists for assigning spin. These fine-tuned descriptions of the different types of spin and the pertinent data collection forms to identify spin are reported respectively in Table 3 and Additional file 2. Definitions of spin were given for reviews that sought and those that did not seek adverse effects of orthodontic interventions (Table 3).

Table 3 Types of spin in reviews that did or did not seek adverse effects of interventions
Full size table

Power calculation

In our pilot sample, we identified an overall proportion of 14.3% (2/14) of spin of the adverse effects in the abstracts of 14 systematic reviews of orthodontic interventions. For this proportion, the Epitools software [41] calculated a required sample size of 48 studies (precision 0.1 and confidence level 0.95), which was fulfilled by our 98 eligible reviews.

Outcomes and statistical analyses

Outcomes

Prevalence proportions were calculated to quantify the answers to our 3 research questions in the 98 selected reviews. According to our published protocol, these proportions were also calculated separately for reviews that either did (n = 84) or did not (n = 14) seek any findings related to adverse effects of interventions in the included studies [33]. These statistics were calculated for (1) all journals as a one group, (2) the five leading orthodontic journals together and the Cochrane Database of Systematic Reviews separately, and (3) each eligible journal separately.

Explorative analyses

Univariable logistic regression models were built to determine the association between the presence of spin in the abstract and characteristics of the systematic review, i.e., journal, year of publication, number of authors, conflict of interest reported, conflict of interest present, funding reported, and type of orthodontic intervention. These analyses were not registered in the protocol and should therefore be interpreted as exploratory. The strengths of associations were quantified using odds ratios (OR) and 95% confidence intervals (95% CI). Multivariable models were built if multiple significant predictors were found in the univariable analysis. Analyses were performed with the use of commercial software (IBM SPSS 22.0, SPSS Inc, Chicago, IL). A two-sided P value of 0.05 was considered to be statistically significant.

Results

The results of the search for eligible reviews were reported previously in part 1 of this cross-sectional study [33] and identified 98 eligible reviews. The PRISMA flow diagram and all included reviews and all excluded studies with rationale were given again in Additional file 3. Figures 1 and 2 present the flow diagrams of the answers to the research questions, and Tables 4 and 5 report the pertinent proportion statistics. The number of identified systematic reviews and the number of eligible systematic reviews of orthodontic interventions given in these tables were published previously [33] and were reported again to give context to the outcomes to our research questions. In these tables, outcomes are further subdivided for eligible reviews that did (n = 84) or did not (n = 14) seek any findings related to adverse effects of interventions in the included studies. The initial inter-operator agreement between both operators for assigning spin was high (Cohen’s κ = 0.94), and complete agreement between operators was reached after discussion.

Fig. 1
figure 1

Reporting or considering adverse effects of orthodontic interventions in abstracts of systematic reviews

Full size image
Fig. 2
figure 2

Spin on adverse effects of orthodontic interventions in abstracts of systematic reviews

Full size image
Table 4 Outcomes on reporting or considering adverse effects and spin in abstracts of systematic reviews of orthodontic interventions
Full size table
Table 5 Outcome 3: types and categories of spin in abstracts of systematic reviews of orthodontic interventions
Full size table

Results for questions 1a and 1b

The results for questions 1a and 1b combined showed that the majority 76.5% (75/98) of eligible reviews reported or considered (i.e., discussed, weighted etc.) potential adverse effects of orthodontic interventions in the abstract (Fig. 1). This prevalence was much higher in the reviews that sought any findings related to adverse effects of interventions in the included studies (88.1% (74/84) than those reviews that did not seek these findings (7.1% (1/14) (Fig. 1).

Results for questions 2a and 2b

The results for questions 2a and 2b combined showed that the total proportion of the presence of spin on adverse effects in the abstract was 40.8% (40/98) in the eligible reviews (Table 4). This prevalence was considerable higher in the reviews that sought any findings related to adverse effects of interventions in the included studies (Question 2a), i.e., 46.4% (39/84) than those reviews that did not seek such findings 7.1% (1/14) (Fig. 2).

Results for questions 3a and 3b

For questions 3a and 3b combined, misleading reporting was the predominant type of spin i.e., 90% (36/40), which was subdivided in the categories of not reporting, 32.5% (13/40), and selective reporting 57.5% (23/40) (Table 5). Misleading interpretation and misleading (inappropriate) extrapolation types of spin were respectively 7.5% (3/40) and 2.5% (1/40).

Explorative analyses

The findings of our explorative analysis on the presence of spin on adverse effects in abstracts of systematic reviews of orthodontic interventions were reported in Table 6. Compared to the Cochrane Database of Systematic Reviews, the EJO (OR: 0.37, 95% CI: 0.08 to 1.63), the AJODO (OR: 0.29, 95% CI: 0.06 to 1.39), the AO (OR: 0.43, 95% CI: 0.09 to 1.95), the KJO (OR: 1.33, 95% CI: 0.09 to 20.11), and the O&C (OR: 0.67, 95% CI: 0.13 to 3.45) had similar odds of the presence of spin on adverse effects in abstracts of systematic reviews of orthodontic interventions. The odds of the presence of spin on adverse effects in abstracts of systematic reviews of orthodontic interventions did not change over the sampled years (OR: 1.03, 95% CI: 0.9 to 1.16). The odds of the presence of spin on adverse effects in abstracts of systematic reviews of orthodontic interventions did not change depending on the number of authors (OR: 0.93, 95% CI: 0.71 to 1.21). Compared to systematic reviews that did not report conflicts of interest, systematic reviews that reported conflicts of interest had similar odds (OR: 0.74, 95% CI: 0.32 to 1.68) of the presence of spin on adverse effects in abstracts of systematic reviews of orthodontic interventions. Systematic reviews on type 1 orthodontic interventions, i.e., orthodontic interventions to move teeth or change the jaw size or position for orthodontic purposes compared with systematic reviews on type 3 orthodontic interventions, i.e., orthodontic interventions to maintain or stabilize orthodontic results had similar odds (OR: 1.1, 95% CI: 0.45 to 2.67) of the presence of spin on adverse effects of orthodontic interventions in abstracts of systematic reviews (Table 6). Intervention type 2, i.e., orthodontic interventions with additional surgical, pharmacological, or vibratory interventions was not included in the analysis since there was only 1 systematic review of this type of intervention.

Table 6 Associations between presence of spin in the abstract and characteristics of the systematic review
Full size table

Discussion

Principal findings of the study

This cross-sectional study showed that the majority, i.e., 76.5% (75/98), of the eligible systematic reviews reported or considered (i.e., discussed, weighted) potential adverse effects of orthodontic interventions in the abstract (Table 4). In 40.8% (40/98), spin on adverse effects was found in the abstract of these reviews (Table 4). Spin related to misleading reporting was the predominant, i.e., 90.0% (36/40), type of spin (Table 5). No association was found between the presence of spin in the abstract and any of the predictors (Table 6).

Comparison with other studies

Item 1 of the 2004 CONSORT (Consolidated Standards of Reporting Trials) Harms extension [42] states that “if the study collected data on harms and benefits, the title or abstract should so state”. In this second part of our cross-sectional studies, 76.5% (75/98) of systematic reviews reported or considered adverse effects of interventions in the abstracts. This was lower than the 84.7% (83/98) of reviews that sought and reported findings on adverse effects in the main text or supplementary files of these reviews as we reported in part 1 [33]. A recent overview of systematic reviews by Junqueira et al. [43] found that harms were reported in 47% (258/552) of the abstracts of RCTs published prior to the CONSORT harms statement and in 54% (643/1201) of the abstracts of the RCTs published after the publication of this statement, indicating only a limited improvement in recent years. Qureshi et al. [14] found that most systematic reviews 81.4% (57/70) on interventions with gabapentin reported a statement on harms in the abstract. Different results in reporting of adverse effects in abstracts in this cross-sectional study compared with those in other studies could be the result of variables such as differences in (1) research design, i.e., systematic reviews versus RCTs, (2) sample size, (3) what is reported on harms in the abstract, e.g., specific versus more general statements, and (4) the field of research. For example, the relatively high prevalence of reporting of adverse effects in abstracts of systematic reviews in this study could be the result of having included only reviews of orthodontic interventions. Orthodontists might be more attentive in assessing adverse effects, because assessing adverse events such as undesired outcomes of orthodontic treatment and relapse is part of daily clinical practice.

A wide variety of prevalence proportions on spin has been identified in abstracts of systematic reviews of randomized-and non-randomized studies [23, 25, 28,29,30]. These studies identified proportions of spin that varied between 23% (24/105) of spin in abstracts of RCTs in rheumatology [28] and 84% (107/128) of spin in abstracts of non-randomized studies that assessed interventions [23]. According to our scoping searches, spin in the field of orthodontics has been assessed only in 2 recent studies [31, 32]. Spin was identified in 62.2% (69/111) of abstracts of parallel-group RCTs with clearly stated statistically non-significant primary outcomes [31] and in 48.6% (53/109) of abstracts of orthodontic meta-analyses [32]. In our study, none of the predictors assessed was associated with the presence of spin in this study. Similar findings were identified by Guo et al. [31] on the overlapping predictors with our study, i.e., “the year of publication” and “the number of authors.” Makou et al. [32] also found no association with the presence of spin for the overlapping predictors “journal” and “year of publication” but found a higher risk of spin in studies with a large number of authors (≥ 6). However, direct comparisons of our results on spin and those identified in other studies are often difficult because of differences in variables such as (1) the types and subtypes of spin and definitions of spin, (2) the research design, (3) the locations in the text where spin was assessed, (4) the field of research, (5) the types of interventions, (6) the journals included, and (7) the time point of the publication [42].

Strengths and limitations

This study has the following strengths: (1) the research methods were pilot tested on a series of systematic reviews and RCTs to consistently extract data and to calibrate data extractors; (2) the protocol of this study was published a priori; and (3) according to our scoping searches, this is the first study that assessed spin on adverse effects in abstracts of systematic reviews of interventions. These searches also showed that our protocol was the first article [24] that planned to assess spin in the field of orthodontics. Subsequently, 2 additional studies [31, 32] have assessed other types of spin in the orthodontic literature, indicating a growing interest in this topic. (4) All raw data were either included with this manuscript in additional files or registered in Open Science Framework (https://osf.io/ka7mp/).

This study also has limitations such as (1) the risk of inaccurate findings on reporting adverse effects of interventions in abstracts as a result of the inconsistent assessment and reporting of adverse effects in both primary research and in systematic reviews and (2) the assessment of spin is not completely objective [44]. However, our inter-operator agreement was high as indicated by a high Cohen’s κ (0.94), and disagreements were completely resolved through discussions. (3) The wide variety of different types and definitions of spin and the assessment of spin in different contexts often limits comparing findings on spin between studies [44]. (4) This study assessed a variety of proportions exclusively in Cochrane intervention reviews and in the 5 orthodontic journals with the highest impact factor. Our findings therefore probably underestimate the true magnitude of proportions on poor reporting and spin on adverse effects in the abstracts of the wider body of orthodontic systematic reviews, and (5) the true magnitude of some proportions could also be underestimated, because we assessed a recent sample (August 1, 2009, until July 31, 2021) of reviews. In this context, one should consider that poor reporting has decreased over time as was shown in a study that assessed the evolution of poor reporting in 20,920 RCTs included in a sample of Cochrane reviews [45].

Implications and future research

Our results imply that end-users of systematic reviews of orthodontic interventions have to be careful when interpreting the findings on adverse effects in abstracts of both Cochrane reviews and those published in the 5 leading orthodontic journals. This is particularly important, because the title and abstracts are often the only read sections of biomedical papers [6] and spin in abstracts can bias the clinician’s interpretations of the results [26]. Reading the full text of research studies is not a solution, because recent studies showed that the proportions of spin in abstracts are similar to those in the full text of the pertinent RCTs [29] and systematic reviews [46]. Guideline developers, researchers, peer reviewers, and editors have an important role in tackling poor reporting and spin regarding adverse effects in abstracts of systematic reviews of orthodontic interventions. Standards for reporting adverse effects in abstracts of systematic reviews of interventions have to be developed. Much research is ahead.

Availability of data and materials

Not applicable.

Change history

Abbreviations

AJODO:

American Journal of Orthodontics and Dentofacial Orthopedics

AO:

Angle Orthodontist

EJO:

European Journal of Orthodontics

KJO:

Korean Journal of Orthodontics

O&CR:

Orthodontics and Craniofacial Research

MECIR:

Methodological Expectations of Cochrane Intervention Reviews

PRISMA:

Preferred reporting items for systematic reviews and meta-analyses

RCT:

Randomized controlled trial

STROBE:

Strengthening the Reporting of Observational Studies in Epidemiology

References

  1. Beller EM, Glasziou PP, Altman DG, Hopewell S, Bastian H, Chalmers I, Gøtzsche PC, Lasserson T, Tovey D, PRISMA for Abstracts Group. PRISMA for Abstracts: reporting systematic reviews in journal and conference abstracts. PLoS Med. 2013;10(4):e1001419. https://doi.org/10.1371/journal.pmed.1001419. Epub 2013 Apr 9. PMID: 23585737; PMCID: PMC3621753.

    Article  PubMed  PubMed Central  Google Scholar 

  2. Haneef R, Lazarus C, Ravaud P, Yavchitz A, Boutron I. Interpretation of results of studies evaluating an intervention highlighted in Google health news: a cross-sectional study of news. PLoS One. 2015;10(10):e0140889. https://doi.org/10.1371/journal.pone.0140889. eCollection 2015.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  3. Haneef R, Yavchitz A, Ravaud P, Baron G, Oranksy I, Schwitzer G, Boutron I. Interpretation of health news items reported with or without spin: protocol for a prospective meta-analysis of 16 randomised controlled trials. BMJ Open. 2017;7(11):e017425. https://doi.org/10.1136/bmjopen-2017-017425.

    Article  PubMed  PubMed Central  Google Scholar 

  4. Ad Hoc Working Group for Critical Appraisal of the Medical Literature. A proposal for more informative abstracts of clinical articles. Ann Intern Med. 1987;106:598–604.

    Article  Google Scholar 

  5. Haynes RB, Mulrow CD, Huth EJ, Altman DG, Gardner MJ. More informative abstracts revisited. Ann Intern Med. 1990;113:69–76.

    Article  CAS  PubMed  Google Scholar 

  6. IslamajDogan R, Murray GC, Névéol A, Lu Z. Understanding PubMed user search behavior through log analysis. Database (Oxford). 2009;2009:bap018. https://doi.org/10.1093/database/bap018. Epub 2009 Nov 27. PMID: 20157491; PMCID: PMC2797455.

    Article  CAS  Google Scholar 

  7. Page MJ, Cumpston M, Chandler J, Lasserson T. Chapter III: Reporting the review. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA, editors. Cochrane handbook for systematic reviews of interventions version 6.2 (updated February 2021). Cochrane; 2021. Available from www.training.cochrane.org/handbook.

  8. Higgins JPT, Lasserson T, Chandler J, Tovey D, Thomas J, Flemyng E, Churchill R. Methodological expectations of Cochrane intervention reviews. London: Cochrane; 2021. Available from: MECIR Version February 2021 clean final_1.pdf (cochrane.org). Accessed 4 Dec 2021.

    Google Scholar 

  9. Bagul NB, Kirkham JJ. The reporting of harms in randomized controlled trials of hypertension using the CONSORT criteria for harm reporting. Clin Exp Hypertens. 2012;34(8):548–54. https://doi.org/10.3109/10641963.2012.681724. Epub 2012 May 9.

    Article  PubMed  Google Scholar 

  10. Carmichael K, Nolan SJ, Weston J, Tudur Smith C, Marson AG. Assessment of the quality of harms reporting in non-randomised studies and randomised controlled studies of topiramate for the treatment of epilepsy using CONSORT criteria. Epilepsy Res. 2015;114:106–13. https://doi.org/10.1016/j.eplepsyres.2015.04.019. Epub 2015 May 8. PMID: 26088893.

    Article  CAS  PubMed  Google Scholar 

  11. Hodkinson A, Kirkham JJ, Tudur-Smith C, Gamble C. Reporting of harms data in RCTs: a systematic review of empirical assessments against the CONSORT harms extension. BMJ Open. 2013;3(9):e003436. https://doi.org/10.1136/bmjopen-2013-003436.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Parsons R, Golder S, Watt I. More than one-third of systematic reviews did not fully report the adverse events outcome. J Clin Epidemiol. 2019;108:95–101. https://doi.org/10.1016/j.jclinepi.2018.12.007.

    Article  PubMed  Google Scholar 

  13. Pitrou I, Boutron I, Ahmad N, Ravaud P. Reporting of safety results in published reports of randomized controlled trials. Arch Intern Med. 2009;169(19):1756–61.

    Article  PubMed  Google Scholar 

  14. Qureshi R, Mayo-Wilson E, Rittiphairoj T, McAdams-DeMarco M, Guallar E, Li T. Harms in Systematic Reviews Paper 2: methods used to assess harms are neglected in systematic reviews of gabapentin. J Clin Epidemiol. 2022;143:212–23. https://doi.org/10.1016/j.jclinepi.2021.10.024. Epub 2021 Nov 3. PMID: 34742789; PMCID: PMC9875742.

    Article  PubMed  Google Scholar 

  15. Saini P, Loke YK, Gamble C, Altman DG, Williamson PR, Kirkham JJ. Selective reporting bias of harm outcomes within studies: findings from a cohort of systematic reviews. BMJ. 2014;349:g6501. https://doi.org/10.1136/bmj.g6501. PMID: 25416499; PMCID: PMC4240443.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Zorzela L, Golder S, Liu Y, Pilkington K, Hartling L, Joffe A, Loke Y, Vohra S. Quality of reporting in systematic reviews of adverse events: systematic review. BMJ. 2014;348:f7668. https://doi.org/10.1136/bmj.f7668. PMID: 24401468; PMCID: PMC3898583.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Cochrane library. About Cochrane reviews. Available from https://www.cochranelibrary.com/about/about-cochranereviews. Accessed 8 June 2023.

  18. Steegmans PAJ, Bipat S, MeursingeReynders RA. Seeking adverse effects in systematic reviews of orthodontic interventions: protocol for a cross-sectional study. Syst Rev. 2019;8(1):89. https://doi.org/10.1186/s13643-019-1000-1. PMID: 30953538; PMCID: PMC6449933.

    Article  PubMed  PubMed Central  Google Scholar 

  19. Glossary of terms in the Cochrane Collaboration. Version 4.2.5. Updated May 2005. Available from: http://aaz.hr/resources/pages/57/7.%20Cochrane%20glossary.pdf. Accessed 4 Dec 2021.

  20. Peryer G, Golder S, Junqueira D, Vohra S, Loke YK. Chapter 19: Adverse effects. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA, editors. Cochrane handbook for systematic reviews of interventions version 6.2 (updated February 2021). Cochrane; 2021. Available from www.training.cochrane.org/handbook.

  21. Boutron I, Ravaud P. Misrepresentation and distortion of research in biomedical literature. Proc Natl Acad Sci U S A. 2018;115(11):2613–9. https://doi.org/10.1073/pnas.1710755115.

    Article  ADS  CAS  PubMed  PubMed Central  Google Scholar 

  22. Boutron I. Spin in scientific publications: a frequent detrimental research practice. Ann Emerg Med. 2020;75(3):432–4. https://doi.org/10.1016/j.annemergmed.2019.11.002. Epub 2019 Dec 23. PMID: 31874770.

    Article  PubMed  Google Scholar 

  23. Lazarus C, Haneef R, Ravaud P, Boutron I. Classification and prevalence of spin in abstracts of non-randomized studies evaluating an intervention. BMC Med Res Methodol. 2015;13(15):85. https://doi.org/10.1186/s12874-015-0079-x.

    Article  Google Scholar 

  24. Steegmans PAJ, Di Girolamo N, MeursingeReynders RA. Spin in the reporting, interpretation, and extrapolation of adverse effects of orthodontic interventions: protocol for a cross-sectional study of systematic reviews. Res Integr Peer Rev. 2019;4:27. https://doi.org/10.1186/s41073-019-0084-4.

    Article  PubMed  PubMed Central  Google Scholar 

  25. Boutron I, Dutton S, Ravaud P, Altman DG. Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes. JAMA. 2010;303(20):2058–64. https://doi.org/10.1001/jama.2010.651.

    Article  CAS  PubMed  Google Scholar 

  26. Boutron I, Altman DG, Hopewell S, Vera-Badillo F, Tannock I, Ravaud P. Impact of spin in the abstracts of articles reporting results of randomized controlled trials in the field of cancer: the SPIIN randomized controlled trial. J Clin Oncol. 2014;32(36):4120–6. https://doi.org/10.1200/JCO.2014.56.7503. Epub 2014 Nov 17. PMID: 25403215.

    Article  PubMed  Google Scholar 

  27. Yavchitz A, Ravaud P, Altman DG, Moher D, Hrobjartsson A, Lasserson T, Boutron I. A new classification of spin in systematic reviews and meta-analyses was developed and ranked according to the severity. J Clin Epidemiol. 2016;75:56–65. https://doi.org/10.1016/j.jclinepi.2016.01.020. Epub 2016 Feb 2.

    Article  PubMed  Google Scholar 

  28. Mathieu S, Giraudeau B, Soubrier M, Ravaud P. Misleading abstract conclusions in randomized controlled trials in rheumatology: comparison of the abstract conclusions and the results section. Joint Bone Spine. 2012;79(3):262–7. https://doi.org/10.1016/j.jbspin.2011.05.008. Epub 2011 Jul 5. PMID: 21733728.

    Article  PubMed  Google Scholar 

  29. Khan MS, Lateef N, Siddiqi TJ, Rehman KA, Alnaimat S, Khan SU, Riaz H, Murad MH, Mandrola J, Doukky R, Krasuski RA. Level and prevalence of spin in published cardiovascular randomized clinical trial reports with statistically nonsignificant primary outcomes: a systematic review. JAMA Netw Open. 2019;2(5):e192622. https://doi.org/10.1001/jamanetworkopen.2019.2622. PMID: 31050775; PMCID: PMC6503494.

    Article  PubMed  PubMed Central  Google Scholar 

  30. Jones C, Rulon Z, Arthur W, Ottwell R, Checketts J, Detweiler B, Calder M, Adil A, Hartwell M, Wright DN, Vassar M. Evaluation of spin in the abstracts of systematic reviews and meta-analyses related to the treatment of proximal humerus fractures. J Shoulder Elbow Surg. 2021:S1058-2746(21)00016-1. https://doi.org/10.1016/j.jse.2020.11.026. Epub ahead of print. PMID: 33482369.

  31. Guo F, Fang X, Li C, Qin D, Hua F, He H. The presence and characteristics of ‘spin’ among randomized controlled trial abstracts in orthodontics. Eur J Orthod. 2021;43(5):576–82. https://doi.org/10.1093/ejo/cjab044. PMID: 34397084.

    Article  PubMed  Google Scholar 

  32. Makou O, Eliades T, Koletsi D. Reporting, interpretation, and extrapolation issues (SPIN) in abstracts of orthodontic meta-analyses published from 2000 to 2020. Eur J Orthod. 2021;43(5):567–75. https://doi.org/10.1093/ejo/cjab009. Epub ahead of print. PMID: 33740054.

    Article  Google Scholar 

  33. Steegmans PAJ, Di Girolamo N, Bipat S, Meursinge Reynders RA. Seeking adverse effects in systematic reviews of orthodontic interventions: a cross-sectional study (part 1). Accepted for publication in Systematic Reviews on 16 June 2023.

  34. Yavchitz A, Boutron I, Bafeta A, Marroun I, Charles P, Mantz J, Ravaud P. Misrepresentation of randomized controlled trials in press releases and news coverage: a cohort study. PLoS Med. 2012;9(9):e1001308. https://doi.org/10.1371/journal.pmed.1001308. Epub 2012 Sep 11. PMID: 22984354; PMCID: PMC3439420.

    Article  PubMed  PubMed Central  Google Scholar 

  35. von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP, STROBE Initiative. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Ann Intern Med. 2007;147(8):573–7.

    Article  Google Scholar 

  36. Lefebvre C, Glanville J, Briscoe S, Littlewood A, Marshall C, Metzendorf M-I, Noel-Storr A, Rader T, Shokraneh F, Thomas J, Wieland LS. Chapter 4: Searching for and selecting studies. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA, editors. Cochrane handbook for systematic reviews of interventions version 6.2 (updated February 2021). Cochrane; 2021. Available from www.training.cochrane.org/handbook.

  37. The Cochrane Database of Systematic Reviews (CDSR). Available from: https://www.cochranelibrary.com/cdsr/about-cdsr. Accessed 4 Dec 2021.

  38. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JP, Clarke M, Devereaux PJ, Kleijnen J, Moher D. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. J Clin Epidemiol. 2009;62(10):e1-34. https://doi.org/10.1016/j.jclinepi.2009.06.006. Epub 2009 Jul 23. PMID: 19631507.

    Article  PubMed  Google Scholar 

  39. Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. J Clin Epidemiol. 2009;62(10):1006–12. https://doi.org/10.1016/j.jclinepi.2009.06.005. Epub 2009 Jul 23. PMID: 19631508.

    Article  PubMed  Google Scholar 

  40. Preoteasa CT, Ionescu E, Preoteasa E. Chapter 18: Risks and complications associated with orthodontic treatment. In: Bourzgui F, editor. Orthodontics-basic aspects and clinical considerations. 2012. Under CC BY 3.0 license. www.intechopen.com. Available from: https://cdn.intechopen.com/pdfs/31388/InTech-Risks_and_complications_associated_with_orthodontic_treatment.pdf. Accessed 4 Dec 2021.

  41. Sergeant ESG. Sergeant, ESG, 2018. Epitools Epidemiological Calculators. Ausvet. Available from: http://epitools.ausvet.com.au. Accessed 4 Dec 2021.

  42. Ioannidis JP, Evans SJ, Gøtzsche PC, O’Neill RT, Altman DG, Schulz K, Moher D, CONSORT Group. Better reporting of harms in randomized trials: an extension of the CONSORT statement. Ann Intern Med. 2004;141(10):781–8.

    Article  PubMed  Google Scholar 

  43. Junqueira DR, Phillips R, Zorzela L, Golder S, Loke Y, Moher D, Ioannidis JPA, Vohra S. Time to improve the reporting of harms in randomized controlled trials. J Clin Epidemiol. 2021;136:216–20. https://doi.org/10.1016/j.jclinepi.2021.04.020. Epub 2021 May 10. PMID: 33984494.

    Article  PubMed  Google Scholar 

  44. Chiu K, Grundy Q, Bero L. ‘Spin’ in published biomedical literature: a methodological systematic review. PLoS Biol. 2017;15(9):e2002173. https://doi.org/10.1371/journal.pbio.2002173. PMID: 28892482; PMCID: PMC5593172.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  45. Dechartres A, Trinquart L, Atal I, Moher D, Dickersin K, Boutron I, Perrodeau E, Altman DG, Ravaud P. Evolution of poor reporting and inadequate methods over time in 20 920 randomised controlled trials included in Cochrane reviews: research on research study. BMJ. 2017;357:j2490. https://doi.org/10.1136/bmj.j2490. Erratum in: BMJ. 2017 Aug 8;358:j3806. PMID: 28596181.

    Article  PubMed  Google Scholar 

  46. Eleftheriadi I, Ioannou T, Pandis N. Extent and prevalence of spin in randomized controlled trials in dentistry. J Dent. 2020;100:103433. https://doi.org/10.1016/j.jdent.2020.103433. Epub 2020 Jul 17. PMID: 32687962.

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

Not applicable.

Funding

All expenses for developing and conducting of this cross-sectional study were paid evenly by each author.

Author information

Authors and Affiliations

  1. Department of Orthodontics, Academisch Centrum Tandheelkunde Amsterdam (ACTA), University of Amsterdam, Gustav Mahlerlaan 3004, Amsterdam, 1081 LA, The Netherlands

    Pauline A. J. Steegmans

  2. Department of Clinical Sciences, College of Veterinary Medicine, Cornell University, 930 Campus Rd, Ithaca, NY, 14853, USA

    Nicola Di Girolamo

  3. Department of Oral and Maxillofacial Surgery, Academic Medical Center, University of Amsterdam, Meibergdreef 9, Amsterdam, 1105 AZ, The Netherlands

    Reint A. Meursinge Reynders

  4. Studio Di Ortodonzia, Via Matteo Bandello 15, Milan, 20123, Italy

    Reint A. Meursinge Reynders

Authors
  1. Pauline A. J. Steegmans
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nicola Di Girolamo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Reint A. Meursinge Reynders
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

This cross-sectional study was conceived and designed by PS and RMR. These 2 researchers also conducted all data extraction procedures. PS, RMR, and NDG conducted the data analyses and statistical analyses. NDG provided support on methodological issues. All 3 authors revised the various draft versions of this paper and read and approved the final submitted manuscript. RMR is the guarantor.

Corresponding author

Correspondence to Reint A. Meursinge Reynders.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

Reint Meursinge Reynders is an associate editor for Systematic Reviews. All three authors declare that they have no further competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The original online version of this article was revised: The authors reported a typo error in Table 6 wherein the headings ‘Yes’ and ‘No’ were interchanged.

Supplementary Information

Additional file 1.

STROBE Checklist: Checklist of items that should be included in reports of cross-sectional studies [35].

Additional file 2.

2A. Differences between the protocol and the completed cross-sectional study. 2B. Search terms and their derivatives. 2C. Data collection forms. 2D. Adverse effects hypothetically linked to orthodontic interventions.

Additional file 3.

3A. PRISMA flow diagram. 3B. Included reviews. 3C. Excluded studies with rationale

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Steegmans, P.A.J., Di Girolamo, N. & Meursinge Reynders, R.A. Spin on adverse effects in abstracts of systematic reviews of orthodontic interventions: a cross-sectional study (part 2). Syst Rev 12, 99 (2023). https://doi.org/10.1186/s13643-023-02269-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s13643-023-02269-3

Keywords

Systematic Reviews

ISSN: 2046-4053