Skip to main content

Include or not to include conference abstracts in systematic reviews? Lessons learned from a large Cochrane network meta-analysis including 585 trials

Abstract

Background

Systematic reviews attempt to gather all available evidence. Controversy exists regarding effort and benefit of including study results presented at conferences only. We recently published a Cochrane network meta-analysis (NMA) including 585 randomized controlled trials comparing drugs for prevention of postoperative nausea and vomiting (PONV). Studies published as conference abstracts only were excluded. This study aimed to include all eligible studies published as abstracts only, assessing their added value regarding reporting quality and effect on the review’s interpretation.

Methods

Conference abstracts were searched in the review’s excluded studies and conference proceedings of anaesthesiologic societies. We assessed their reporting quality regarding review’s eligibility criteria, Cochrane ‘risk of bias’ assessment tool 1.0, and adherence to CONSORT (Consolidated Standards of Reporting Trials) for abstracts. Abstracts were included in sensitivity NMA, and impact on the NMA structure was investigated.

Results

We identified 90 abstracts. A total of 14% (13/90) were eligible. A total of 86% (77/90) are awaiting classification due to insufficient reporting of review’s eligibility criteria. In abstracts awaiting classification, sufficient information was missing on standardization of anaesthesia in 71% (55/77), age of participants in 56% (43/77), and outcome details in 46% (36/77). A total of 73% (66/90) of abstracts lacked sufficient information on 15/25 data extraction items. Reported study characteristics of abstracts were comparable to included studies of the review. A total of 62% (56/90) of abstract trials were assessed as overall high risk of bias due to poor reporting. Median adherence to CONSORT for abstracts was 24% (IQR, 18 to 29%). Six of the 13 eligible abstracts reported relevant outcome data in sufficient detail for NMA on seven outcomes of the Cochrane review. Inclusion of abstracts did not substantially change the network structure, network effect estimates, ranking of treatments, or the conclusion. Certainty of evidence for headache on palonosetron use was upgraded from very low to low.

Conclusions

Most conference abstracts on PONV were insufficiently reported regarding review’s narrow inclusion criteria and could not be included in NMA. The resource-intensive search and evaluation of abstracts did not substantially extent the full-text evidence base of the review, given the few adequately reported abstracts. Conferences should oblige authors to adhere to CONSORT for abstracts.

Peer Review reports

Background

Systematic reviews attempt to gather all available evidence, including unpublished studies on a specific research question. Controversy exists regarding the added value of including study results presented at conferences only [1]. While Cochrane highly desires searching relevant grey literature sources when generating a review [2], the US Agency for Healthcare Research and Quality only recommends that these searches be considered [3]. However, searching for and including conference abstracts are resource intensive [1]. In contrast to traditional systematic reviews with their comprehensive data pool, rapid reviews apply methodological short cuts, like limiting grey literature searches, to generate the review in a shorter time [4, 5].

Including conference abstracts in systematic reviews can increase its comprehensiveness and precision and can decrease the potential impact of publication bias [1, 6]. Other meta-epidemiologic studies have shown only minor differences in results between meta-analyses with and without conference abstracts [7, 8]. A Cochrane review has shown that one-third of the randomized controlled trials (RCTs) presented at conferences are subsequently not published in full [6]. They found evidence for publication bias as probability for full publication depended on the direction of the results [6].

Information in abstracts is often inadequately reported, and their dependability is questionable [1, 9,10,11,12,13,14]. When abstracts are the only record of the study results, providing all necessary information in abstracts to appraise the study and assess the eligibility for a systematic review is fundamental to increase the value for evidence synthesis. A crucial step towards higher reporting quality was the introduction of reporting guidelines like the CONSORT (Consolidated Standards of Reporting Trials) statement including a special version for journal and conference abstracts, published in 2008 [15].

The recent Cochrane review with network meta-analysis (NMA) from Weibel and colleagues compared the efficacy and safety of antiemetic drugs for the prevention of postoperative nausea and vomiting (PONV) in adults after general anaesthesia [16]. To enhance feasibility of the workload with an estimated number of more than 1000 trials for full-text screening, all trials published as conference abstracts only were excluded in the review protocol [16]. The current study addressed retrospectively whether inclusion of all eligible studies published as abstracts only is able to change the review’s interpretation. To this end, we searched for additional conference abstracts that were eligible for the Cochrane review and assessed abstracts’ reporting quality, risk of bias, and the impact on the conclusion of the review.

Methods

This study is a case study on the impact of randomized controlled studies published as conference abstracts only and which met otherwise the narrow eligibility criteria of the large Cochrane review published by Weibel et al. [16, 17]. This study has not been registered. A protocol for the retrospective case study has not been prepared. A completed PRISMA 2020 checklist is available in Additional file 1.

Eligibility criteria

For this study, RCTs reported as conference abstracts without a subsequent full-text publication were eligible only. No language restrictions were applied. The prespecified narrow eligibility criteria of the Cochrane review were applied for this study [16]. In brief, studies were required to investigate adult participants (i.e. the majority of participants over 18 years of age) undergoing any type of surgery with general anaesthesia and compare single or multiple pharmacological intervention(s) with antiemetic action belonging to one of the six drug classes (5-HT3 receptor antagonists, D2 receptor antagonists, NK1 receptor antagonists, corticosteroids, antihistamines (histamine 1 receptor antagonists), and anticholinergics) versus each other, versus no treatment, or versus placebo. All drugs had to be administered before or during anaesthesia to prevent PONV. Participants in all treatment arms within a study must have been subject to the same anaesthesia regimen. Primary outcomes of the review were vomiting within 24 h postoperatively, serious adverse events (SAEs), and any adverse event (AE) both within 7 days postoperatively. Secondary outcomes were drug class-specific side effects (e.g. headache, constipation, extrapyramidal symptoms, sedation, arrhythmia, QT prolongation, wound infection, and visual disturbances), early and late vomiting, nausea, and complete response (CR, no nausea and no vomiting, and no rescue antiemetic treatment for the first 24 h).

Information sources and search strategy

We searched the list of excluded studies from the Cochrane review for eligible abstracts. The review’s search included the Cochrane Central Register of Controlled Trials (CENTRAL), MEDLINE, Embase, CINAHL, and study registers (ClinicalTrials.gov, WHO ICTRP) and reference lists of relevant systematic reviews. Details of the search strategy are provided in the full Cochrane review [16].

Additionally, we searched the online available conference proceedings of the American Society of Anesthesiologists (ASA, 2000–2017), European Society of Anaesthesiology (ESA, 2004–2017), and International Anesthesia Research Society (IARS, 2012–2017) for eligible abstracts. Search keywords were ‘PONV’, ‘vomit’, ‘nausea’, ‘emesis’, and ‘emetic’.

Conference abstracts published up to November 2017, the primary search date of the Cochrane review, were eligible. If a full-text publication was found, we excluded the abstract. Duplicates were removed.

Study selection and data extraction

Abstracts were assessed for eligibility, and data were extracted by a single reviewer using Covidence [18]. Critical issues regarding eligibility or study details were discussed with a second reviewer.

If information on eligibility (PICO) criteria, including details on participants (P), intervention (I), comparator (C), and outcome (O), was missing or insufficiently reported, abstracts were categorized as awaiting classification. The number of missing or insufficient information regarding any eligibility criteria per abstract awaiting classification was determined. We compared the reasons for assessment as ‘awaiting classification’ before and after 2008, the year of the introduction of CONSORT for abstracts.

Data extraction followed a predefined and piloted data extraction form. In brief, we extracted the literature source, year of publication, publication language, the location of the study conduct/origin of authors (Africa, Asia, Australia, Europe, North America, South America), and setting (multicentre, single centre). Source of funding (sponsored by industry, nonindustry resources, not reported) and trial registration number were identified. Type of surgery, intervention details (inactive control, only active arms, single interventions, combination interventions), and the study’s outcomes with details were further extracted. We compared the characteristics of the eligible abstracts with those awaiting classification and with included studies of the Cochrane review.

Risk of bias

We assessed the risk of bias of eligible abstracts, and of those awaiting classification using the Cochrane ‘risk of bias’ assessment tool 1.0, and summarized the overall risk of bias for each study by reference to the judgements of the domains ‘sequence generation’, ‘blinding of participant, personnel, and outcome assessors’, and ‘incomplete outcome data’. We compared the risk of bias assessments of the abstracts with those of included studies in the Cochrane review. Methodological details of risk of bias assessment were described in the Cochrane review [16].

Assessment of reporting quality in abstracts

We assessed the reporting of data relevant for specific data extraction according to the Cochrane review extraction sheet with 25 relevant items in both eligible abstracts and abstracts awaiting classification. The analysed categories were study identification, study design and details regarding study populations, intervention, and outcomes.

We further assessed the reporting of relevant information necessary for risk of bias assessment. Information per item was judged as ‘sufficient’, ‘insufficient’, or ‘missing’.

Furthermore, we assessed the adherence to CONSORT for abstracts checklist items as ‘sufficient’, ‘insufficient’, or ‘missing’. We compared the reporting of data, risk of bias relevant information, and CONSORT for abstracts items in abstracts before and after 2008, the year of the introduction of CONSORT for abstracts.

Contribution of abstracts to network meta-analyses

Eligible abstracts with relevant outcome data that were reported in sufficient detail were assessed for their impact on the primary NMA. First, we investigated whether inclusion of abstracts changed the network information and structure, e.g. addition of new studies, new treatments, and new direct or indirect comparisons. Second, in sensitivity analyses, eligible abstracts were added to the primary NMAs published in the Cochrane review [16, 17]. Details for NMA were described elsewhere [16, 17]. We compared NMA effect estimates and ranking order of treatments of the primary NMA and the sensitivity analysis. We classified a relevant change in NMA effect estimates if the estimated effects differed with respect to the range of equivalence (i.e. was located outside the predefined range of equivalence), which was set as a risk ratio (RR) of 0.8 to 1.25 for PONV-related outcomes and a RR of 0.9 to 1.11 for all safety outcomes.

Results

We identified 151 potentially relevant abstracts by title screening from different sources (Fig. 1). After duplicates had been removed, 149 abstracts were assessed in detail for eligibility, and 59 abstracts were excluded with reasons (Fig. 1). Of the remaining 90 studies, 13 abstracts (14%) were eligible for inclusion, and 77 abstracts (86%) are awaiting classification due to lack of information regarding relevant eligibility (PICO) criteria. References to eligible abstracts and abstracts awaiting classification are available in Additional file 2. About half of all eligible abstracts (46%) and the abstracts awaiting classification (45%) were retrieved from the list of excluded studies in the Cochrane review (Additional file 3).

Fig. 1
figure 1

Study flow diagram. ASA (American Society of Anesthesiologists), ESA (European Society of Anaesthesiology), and the IARS (International Anesthesia Research Society)

In 86% of the abstracts (77/90), full eligibility could not be assessed due to insufficient or missing information on PICO criteria. Among the most frequent reasons leading to the assessment as ‘awaiting classification’ was a lack of information on the standardization of the anaesthesia regimen in the study arms (71%), age of participants (56%), and outcome details (47%) (Table 1). In 65% of all abstracts, awaiting classification (50/77) had more than one missing information regarding any eligibility criteria (Table 1). Comparing abstracts published before and after publication of the CONSORT statement for abstracts in 2008 revealed no difference in the availability of information on relevant eligibility criteria (Table 1).

Table 1 Reasons for awaiting classification (abstracts)

We compared the characteristics of studies published as eligible abstracts and abstracts awaiting classification with included studies of the Cochrane review (Additional file 3). Overall, the study characteristics were comparable with some minor differences pointed out in the following. Most studies were published in English and were primarily conducted in Asia, Europe, or North America. However, the majority of studies included in the Cochrane review were from Asia (51%), while eligible abstracts were mostly from North America (46%). Most studies investigated single drugs only, mostly ondansetron. The majority of the 13 eligible abstracts compared active interventions only (62%), whereas the majority of the 585 included studies in the Cochrane review had at least one inactive control arm (66%). Most participants were women. However, the proportion of women in the Cochrane review was higher (88%) than in abstracts (49% and 53%). The application of perioperative opioids was less frequently reported in abstracts than in studies included in the review. In the majority of abstracts, the type of general anaesthesia was not specified, in contrast to 88% using volatile anaesthetics in studies included in the review. Overall, most participants underwent gynaecological surgery. However, neurological procedures were more frequent in eligible abstracts (23% vs. 15% gynaecological procedures).

Overall, reporting of data relevant for data extraction according to the Cochrane review´s extraction sheet (25 items) was poor in eligible abstracts and abstracts awaiting classification. A total of 73% of abstracts (66/90) had missing or insufficient information on 15 out of 25 data extraction items. Eight of 25 categories were not addressed in more than 80% of all abstracts (Additional file 4). There is no overall improvement in reporting of data in abstracts published before and after 2008 (Additional file 4).

The risk of bias assessment of studies differed between studies published as abstracts, eligible or awaiting classification, and full-text publications, included in the Cochrane review (Fig. 2). None of the abstracts was assessed as overall low risk of bias compared to 27% of studies included in the Cochrane review. A total of 62% of all abstracts (56/90) were assessed as overall high risk of bias, including 7/13 of the eligible abstracts and 49/77 of the abstracts awaiting classification.

Fig. 2
figure 2

Comparison of Cochrane’s risk of bias assessment for abstracts and studies of the Cochrane review

Reporting of relevant information for risk of bias assessment on individual domains was inadequate. A total of 96% of all abstracts (86/90) had insufficient or missing information in at least six out of seven risk of bias domains resulting in an ‘unclear risk of bias’ assessment (Additional file 5). Most details were reported for the domain ‘incomplete outcome data’ with sufficient information in 38% of the abstracts. In general, there was no difference in the reporting of relevant information for risk of bias assessment in abstracts published before and after 2008 (Additional file 5).

Adherence to the CONSORT for abstracts reporting items was also suboptimal, with a median adherence per abstracts of 4 out of 17 items (24%) (interquartile range: 3 to 5 (1 to 29%)). None of the abstracts provided sufficient contact details, methods of sequence generation and allocation concealment, and adequate outcome results (Additional file 6). We also found very low adherence to describing blinding (2%) and trial registration (1%). There is no overall improvement in reporting according to CONSORT for abstract items in abstracts that were published before and after 2008 (Additional file 6).

Six of the 13 eligible abstracts reported relevant outcome data in sufficient detail for network meta-analyses on seven outcomes of the Cochrane review (Table 2). In the remaining seven studies, relevant information for data analysis was missing. Three studies could be included in the NMA of nausea within 24 h; two studies each were included in the NMA of vomiting within 24 h, early vomiting, complete response, and headache; and one study was included in the NMA of QT prolongation and vomiting late (Table 3, Additional file 7). The network structure of these outcomes was hardly affected. Inclusion of abstracts added one new treatment dolasetron-scopolamine in comparison with dolasetron to the outcomes early vomiting and late vomiting, and one new direct comparison droperidol-metoclopramide versus ondansetron was added to the outcome early vomiting (Table 3, Additional file 7). The number of additional comparisons per NMA due to inclusion of abstracts ranged between one and 11 (Table 3, Additional file 7). Effect estimates from single abstract studies in comparison with NMA effect estimates are shown in the Additional file 7. Inclusion of abstracts in sensitivity NMA analyses did not change the primary network effect estimates nor the ranking of treatments for any of the seven outcomes in a relevant manner (Table 3, forest plots are shown in Additional file 8). For most effect estimates, the clinical interpretation of effects did not change, with exception of palonosetron for vomiting within 24 h and headache. In case of vomiting, the clinical interpretation of the effect of palonosetron changed from uncertain (RR 0.62, 95% CI 0.48 to 0.80) to important benefit (RR 0.63, 95% CI 0.50 to 0.79), as the 95% CI did no longer include the range of equivalence (95% CI 0.80 to 1.25) (Table 3). However, certainty of evidence remained low. For headache, the certainty that palonosetron increases the risk for headache was upgraded from very low to low.

Table 2 Outcomes for analyses in eligible abstracts (n = 13)
Table 3 Sensitivity NMA analyses of NMA results primary and secondary outcomes including eligible abstracts

Discussion

Reporting in abstract publications in clinical PONV research is poor. Thus, we could not include the majority of abstracts into the recent Cochrane review on PONV with narrow eligibility criteria. In contrast, only 12% (206/1762) of full-text records assessed for eligibility in the Cochrane review were awaiting classification due to missing information in the study reports or study protocols [16]. Previous systematic reviews on PONV had broader inclusion criteria [19,20,21]. Therefore, more conference abstracts might be eligible, despite their deficiencies in reporting.

Overall, study characteristics of abstracts were mostly comparable to the included studies in the review. Interestingly, 62% of eligible abstracts compared active interventions only, whereas most of the studies (66%) in the review had an inactive control arm. This might indicate a publication bias since studies comparing active drugs might be prone to nonsignificant results. However, publication bias played a minor role in this field of clinical research as we suspected publication bias for only seven out of 64 direct comparisons with more than 10 RCTs in the NMA [16]. Moreover, considering both eligible abstracts and those awaiting classification, 49% compared active interventions only, while 51% had an inactive control arm. Therefore, the difference between eligible abstracts and studies in the review is probably only a random effect due to the limited data pool of 13 eligible abstracts. Abstracts were more frequently assessed as overall high risk of bias, due to poor reporting [16].

Complete, transparent, and accurate reporting in conference abstracts is fundamental as the conference abstract may remain the only available record of that trial [6]. However, reporting was inadequate across all areas analysed: eligibility criteria, study details for specific data extraction, risk of bias assessment, and CONSORT for abstracts checklist. Previous studies on reporting of abstracts in anaesthesia clinical trials mainly focused on the adherence to CONSORT of journal abstracts [22,23,24]. However, in the field of pain research, the reporting of conference abstracts has been assessed, from 2008 to 2014 [11]. With 26%, median adherence to CONSORT for abstracts was similar to this study (24%) [11]. In both studies, no abstract adequately reported contact details, and only one each mentioned trial registration [11]. The lack of e-mail address and study registration number is particularly concerning in conference abstracts, as they are important ways to obtain further study information [1, 25]. With 2% and 6% respectively, Saric and colleagues found very low adherence regarding randomization and recruitment status, comparable to ours (0% and 4%, respectively) [11]. Similar to our study, poor methodological reporting was found in 1992 and 2002 oncologic conference abstracts regarding method of allocation concealment (1%) and of blinding (12%) [14]. Overall, reporting needs to improve as fellow researchers have found [10, 14].

It is difficult to report in abstracts comprehensively when the total number of words or headings within an abstract is strictly limited. However, 250 to 300 words are suffice to address the CONSORT for abstracts checklist [15]. The 2020 instructions for authors of ASA, ESA, and IARS conferences permitted at least 2500 characters including spaces [26,27,28]. Previous of these gatherings, and conferences not explicitly referred to here, may have imposed different restrictions.

Overall, in our study, reporting did not improve after the publication of the CONSORT statement for abstracts in 2008. Our analysed “post-CONSORT” time period beginning in 2008 might be too early to capture the impact of CONSORT for abstracts. At that time, CONSORT might not have gained sufficient awareness yet. In addition, lack of improvement of reporting may be due to the missing requirement at conferences, for authors to adhere to CONSORT for abstracts. Still, 2020 instructions for authors of the ASA, ESA, and IARS conference proceedings do not mention any reporting guideline, in contrast to the journals Anesthesiology, European Journal of Anaesthesiology, and Anesthesia & Analgesia [26,27,28,29,30,31]. Only two of 819 instructions for abstracts on conferences had reporting instructions for RCT or systematic review authors [32]. Studies on implementation of CONSORT in journals have shown a potential positive influence on reporting [33, 34]. Thus, authors should be required to adhere to CONSORT for abstracts when submitting a conference abstract [11].

Inclusion of eligible abstracts in the NMA did not substantially change the network structure, network effect estimates, ranking of treatments, or the conclusion. The interpretation of the estimated benefit of palonosetron for vomiting within 24 h changed from ‘uncertain’ to ‘important’, but certainty of evidence remained low. Certainty of evidence for palonosetron boosting the risk of headache increased from ‘very low’ to ‘low’. All changes are of minor clinical relevance. As we could only include study results of six abstracts in the sensitivity NMA due to poor reporting in abstracts of inclusion criteria and outcome details, major differences to the original NMA with 585 studies could not have been expected. However, based on the amount of full-text studies, we did originally expect to find a greater number of eligible abstracts with a potentially greater impact on the NMA. Our case study supports the suggestion by Scherer et al. to consider availability of evidence informing a review, when deciding whether or not to search for conference abstracts [1]. As the full-text evidence on prevention of PONV research is plentiful [16], the added value of including conference abstracts appears to be limited. This finding may also be relevant to the generation of rapid reviews. Our study may indicate that limiting grey literature searches may be a reasonable methodological short cut, to generate a review in a shorter time. It is also supported by the guidance on rapid reviews from Cochrane and the WHO and Alliance for Health Policy and Systems Research [4, 5]. Our findings are consistent with a meta-epidemiological study identifying abbreviated literature searches as viable options for rapid evidence syntheses, when less certainty and a small risk for opposite conclusions can be accepted [35]. A study on the contribution of non-English reports and unpublished studies also showed only rarely an impact on the conclusion in child-relevant reviews [7].

This study has several limitations.

First, the study was performed in a single reviewer mode, including the abstract search in conference proceedings and assessment of eligibility and reporting. Despite all care and attention, including discussion with a second reviewer, studies might have been missed or have been inconsistently assessed. Assessment of reporting is always subjective, and identified inadequacies in studies’ reporting do not necessarily correspond to poor methodological study quality. Second, our study may underrepresent conference proceedings in a language other than English. We searched three English-language conference proceedings. However, most of the abstracts were retrieved from the list of excluded studies of the review, which did not apply any language restrictions. Third, the small number of sufficiently reported abstracts is the main limitation that prevents an impact on the conclusion of the NMA. We did not contact any author for missing abstract information, as reporting quality was a matter of analysis. Thus, more abstracts may be fully eligible, with a potentially greater impact on the NMA.

Conclusions

In summary, most conference abstracts on PONV were insufficiently reported regarding review’s narrow inclusion criteria and therefore could not be included in the NMA. The resource-intensive search and evaluation of abstracts did not substantially extend the already solid full-text evidence base of the review, given the small number of adequately reported abstracts. If in conference proceedings authors are obliged to adhere to CONSORT for abstracts, using clinical trial abstracts in systematic reviews may yield a more relevant evidence gain in the future.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Abbreviations

AE:

Adverse event

ASA:

American Society of Anesthesiologists

CENTRAL:

Cochrane Central Register of Controlled Trials

CONSORT:

Consolidated Standards of Reporting Trials

CI:

Confidence interval

CINAHL:

Cumulative index to nursing and allied health literature

CR:

Complete response

ESA:

European Society of Anaesthesiology

NMA:

Network meta-analysis

PICO:

Participants, intervention, comparator, and outcome

PONV:

Postoperative nausea and vomiting

RCT:

Randomized controlled trial

RR:

Risk ratio

sAE:

Serious adverse events

WHO:

World Health Organisation

References

  1. Scherer RW, Saldanha IJ. How should systematic reviewers handle conference abstracts? A view from the trenches. Syst Rev. 2019;8(1):264.

    Article  Google Scholar 

  2. Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al. Cochrane Handbook for Systematic Reviews of Interventions version 6.2 (updated February 2021). 2021. Cochrane, Available from www.training.cochrane.org/handbook.

  3. Balshem H, Stevens A, Ansari M, Norris S, Kansagara D, Shamliyan T, et al. Finding grey literature evidence and assessing for outcome and analysis reporting biases when comparing medical interventions: AHRQ and the effective health care program. In: Methods Guide for Effectiveness and Comparative Effectiveness Reviews. Rockville: Agency for Healthcare Research and Quality (US); 2013. Available from: https://www.ncbi.nlm.nih.gov/books/NBK174882/.

    Google Scholar 

  4. Garritty C, Gartlehner G, Kamel C, King V, Nussbaumer-Streit B, Stevens A, et al. Cochrane Rapid Reviews. Interim Guidance from the Cochrane Rapid Reviews Methods Group; 2020.

    Google Scholar 

  5. Tricco AC, Langlois EV, Straus SE, editors. Rapid reviews to strengthen health policy and systems: a practical guide. Geneva: World Health Organization; 2017.

    Google Scholar 

  6. Scherer RW, Meerpohl JJ, Pfeifer N, Schmucker C, Schwarzer G, von Elm E. Full publication of results initially presented in abstracts. Cochrane Database Syst Rev. 2018;(11):MR000005. https://doi.org/10.1002/14651858.MR000005.pub4.

  7. Hartling L, Featherstone R, Nuspl M, Shave K, Dryden DM, Vandermeer B. Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews. BMC Med Res Methodol. 2017;17(1):64.

    Article  Google Scholar 

  8. Schmucker CM, Blümle A, Schell LK, Schwarzer G, Oeller P, Cabrera L, et al. Systematic review finds that study data not published in full text articles have unclear impact on meta-analyses results in medical research. PLoS One. 2017;12(4):e0176210.

    Article  Google Scholar 

  9. van Driel ML, De Sutter A, De Maeseneer J, Christiaens T. Searching for unpublished trials in Cochrane reviews may not be worth the effort. J Clin Epidemiol. 2009;62(8):838–44.e3.

    Article  Google Scholar 

  10. Hopewell S, Clarke M, Askie L. Reporting of trials presented in conference abstracts needs to be improved. J Clin Epidemiol. 2006;59(7):681–4.

    Article  Google Scholar 

  11. Saric L, Vucic K, Dragicevic K, Vrdoljak M, Jakus D, Vuka I, et al. Comparison of conference abstracts and full-text publications of randomized controlled trials presented at four consecutive World Congresses of Pain: reporting quality and agreement of results. Eur J Pain. 2019;23(1):107–16.

    CAS  Article  Google Scholar 

  12. Scherer RW, Sieving PC, Ervin AM, Dickersin K. Can we depend on investigators to identify and register randomized controlled trials? PLoS One. 2012;7(9):e44183.

    CAS  Article  Google Scholar 

  13. Scherer RW, Huynh L, Ervin A, Taylor J, Dickersin K. ClinicalTrials.gov registration can supplement information in abstracts for systematic reviews: a comparison study. BMC Med Res Methodol. 2013;13:79.

    Article  Google Scholar 

  14. Hopewell S, Clarke M. Abstracts presented at the American Society of Clinical Oncology conference: how completely are trials reported? Clin Trials. 2005;2(3):265–8.

    Article  Google Scholar 

  15. Hopewell S, Clarke M, Moher D, Wager E, Middleton P, Altman DG, et al. CONSORT for reporting randomised trials in journal and conference abstracts. Lancet. 2008;371(9609):281–3.

    Article  Google Scholar 

  16. Weibel S, Rücker G, Jelting Y, Eberhart LHJ, Pace NL, Hartl HM, et al. Drugs for preventing postoperative nausea and vomiting in adults after general anaesthesia: a network meta-analysis. Cochrane Database Syst Rev. 2020;(10):CD012859. https://doi.org/10.1002/14651858.CD012859.pub2.

  17. Weibel S, Pace NL, Schaefer MS, Raj D, Schlesinger T, Meybohm P, et al. Drugs for preventing postoperative nausea and vomiting in adults after general anesthesia: An abridged Cochrane network meta-analysis. J Evid-Based Med. 2021;14(3):188–97.

    Article  Google Scholar 

  18. Veritas Health Innovation. Covidence systematic review software. Melbourne, Australia. https://www.covidence.org/. Accessed 5 May 2020.

  19. Carlisle JB, Stevenson CA. Drugs for preventing postoperative nausea and vomiting. Cochrane Database Syst Rev. 2006;(3):CD004125. https://doi.org/10.1002/14651858.CD004125.pub2.

  20. Tricco AC, Soobiah C, Blondal E, Veroniki AA, Khan PA, Vafaei A, et al. Comparative safety of serotonin (5-HT3) receptor antagonists in patients undergoing surgery: a systematic review and network meta-analysis. BMC Med. 2015;13:142.

    Article  Google Scholar 

  21. Tricco AC, Soobiah C, Blondal E, Veroniki AA, Khan PA, Vafaei A, et al. Comparative efficacy of serotonin (5-HT3) receptor antagonists in patients undergoing surgery: a systematic review and network meta-analysis. BMC Med. 2015;13:136.

    Article  Google Scholar 

  22. Janackovic K, Puljak L. Reporting quality of randomized controlled trial abstracts in the seven highest-ranking anesthesiology journals. Trials. 2018;19(1):591.

    Article  Google Scholar 

  23. Chow JTY, Turkstra TP, Yim E, Jones PM. The degree of adherence to CONSORT reporting guidelines for the abstracts of randomised clinical trials published in anaesthesia journals: A cross-sectional study of reporting adherence in 2010 and 2016. Eur J Anaesthesiol. 2018;35(12):942–8. https://doi.org/10.1097/EJA.0000000000000880. Epub ahead of print.

  24. Can OS, Yilmaz AA, Hasdogan M, Alkaya F, Turhan SC, Can MF, et al. Has the quality of abstracts for randomised controlled trials improved since the release of Consolidated Standards of Reporting Trial guideline for abstract reporting? A survey of four high-profile anaesthesia journals. Eur J Anaesthesiol. 2011;28(7):485–92.

    Article  Google Scholar 

  25. Young T, Hopewell S. Methods for obtaining unpublished data. Cochrane Database Syst Rev. 2011;(11):MR000027. https://doi.org/10.1002/14651858.MR000027.pub2.

  26. American Society of Anesthesiologists. Anesthesiology® 2020 annual meeting scientific abstract submission guidelines. https://www.asahq.org/annualmeeting/education/submissions. Accessed 5 May 2020.

  27. European Society of Anaesthesiology. Abstract submission guidelines Euroanaesthesia 2020. https://euroanaesthesia2020.org/abstract-submission/. Accessed 4 May 2020.

  28. International Anesthesia Research Society. Abstract submission guidelines and requirements. IARS AUA SOCCA annual meetings. https://meetings.iars.org/abstracts/submissions/. Accessed 4 May 2020.

  29. Anesthesiology Editorial Office. Instructions for authors anesthesiology. https://anesthesiology.pubs.asahq.org/public/InstructionsforAuthors.aspx. Accessed 3 May 2020.

  30. European Journal of Anaesthesiology Editorial Office. Guidance for authors on the preparation and submission of manuscripts to the European Journal of Anaesthesiology. http://edmgr.ovid.com/eja/accounts/ifauth.htm. Accessed 4 May 2020.

  31. Pittet JF, Anesthesia & Analgesia Editorial Board. Instructions for authors version 3.2 Anesthesia & Analgesia. http://edmgr.ovid.com/aa/accounts/ifauth.htm. Accessed 4 May 2020.

  32. Saric L, Dosenovic S, Mihanovic J, Puljak L. Biomedical conferences’ author instructions rarely mention guidelines for reporting abstracts of trials and systematic reviews. J Comp Eff Res. 2020;9(2):83–91.

    Article  Google Scholar 

  33. Turner L, Shamseer L, Altman DG, Weeks L, Peters J, Kober T, et al. Consolidated standards of reporting trials (CONSORT) and the completeness of reporting of randomised controlled trials (RCTs) published in medical journals. Cochrane Database Syst Rev. 2012;(11):MR000030. https://doi.org/10.1002/14651858.MR000030.pub2.

  34. Hopewell S, Ravaud P, Baron G, Boutron I. Effect of editors’ implementation of CONSORT guidelines on the reporting of abstracts in high impact medical journals: interrupted time series analysis. BMJ. 2012;344:e4178.

    Article  Google Scholar 

  35. Nussbaumer-Streit B, Klerings I, Wagner G, Heise TL, Dobrescu AI, Armijo-Olivo S, et al. Abbreviated literature searches were viable alternatives to comprehensive searches: a meta-epidemiological study. J Clin Epidemiol. 2018;102:1–11.

    Article  Google Scholar 

Download references

Acknowledgements

We would like to thank Elisabeth Friedrich-Würstlein for her help in organizing abstract publications. We would like to thank the author team of the Cochrane review for their support with literature screening, data extraction, critical appraisal, and analysis.

Funding

This publication was supported by the Open Access Publication Fund of the University of Wuerzburg. Open Access funding enabled and organized by Projekt DEAL. This work was supported by departmental resources only.

Author information

Authors and Affiliations

Authors

Contributions

SH, study design, acquisition of data, analysis, interpretation, and drafting the article; PK, study design, interpretation, and drafting the article; PM, interpretation and drafting the article; and SW, study design, acquisition of data, analysis, interpretation, and drafting the article. The authors read and approved the final manuscript.

Corresponding author

Correspondence to Stephanie Weibel.

Ethics declarations

Ethics approval and consent to participate

Not applicable

Consent for publication

Not applicable.

Competing interests

PK has financial relationships with the Teva ratiopharm GmbH, Fresenius Kabi GmbH Germany, and PAJUNK that might have an interest in the submitted work in the previous 3 years; he has no other relationships or activities that could appear to have influenced the submitted work. PK was involved in the conduct of studies related to the Cochrane review. He did not assess the relevant studies for in- or exclusion and were not allowed to extract data and critically appraise the quality of the relevant studies. The other authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1.

Completed PRISMA 2020 checklist.

Additional file 2.

References to eligible abstracts and abstracts awaiting classification.

Additional file 3.

Study characteristics of eligible abstracts, abstracts awaiting classification and included studies in the review; data sheet with the study characteristics of eligible abstracts, abstracts awaiting classification and included studies in the review.

Additional file 4.

Reporting of data for data extraction according to the Cochrane review’s extraction sheet (25 items); data sheet with the assessment of reporting quality of all abstracts regarding data extraction items.

Additional file 5.

Reporting of information for risk of bias assessment in abstracts; data sheet with the assessment of reporting quality of all abstracts regarding risk of bias relevant information.

Additional file 6.

Adherence to CONSORT for abstracts reporting items; data sheet with the assessment of reporting quality of all abstracts regarding the CONSORT for abstracts checklist.

Additional file 7.

Effect estimates of antiemetic treatments in the abstracts and the NMA; data sheet comparing effect estimates of antiemetic treatments in the abstracts vs. review (direct evidence/NMA).

Additional file 8.

Forest plots NMA; Sensitivity analyses including abstracts; graphical presentation of the NMA effect estimates of treatments per outcome with and without abstracts.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Hackenbroich, S., Kranke, P., Meybohm, P. et al. Include or not to include conference abstracts in systematic reviews? Lessons learned from a large Cochrane network meta-analysis including 585 trials. Syst Rev 11, 178 (2022). https://doi.org/10.1186/s13643-022-02048-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s13643-022-02048-6

Keywords

  • Randomized controlled trial
  • Conference abstract
  • Systematic review
  • Reporting quality
  • CONSORT
  • Postoperative nausea and vomiting