Towards core outcome set (COS) development: a follow-up descriptive survey of outcomes in Cochrane reviews
© Wuytack et al.; licensee BioMed Central. 2015
Received: 8 January 2015
Accepted: 8 May 2015
Published: 19 May 2015
A core outcome set (COS) can address problems of outcome heterogeneity and outcome reporting bias in trials and systematic reviews, including Cochrane reviews, helping to reduce waste. One of the aims of the international Core Outcome Measures in Effectiveness Trials (COMET) Initiative is to link the development and use of COS with the outcomes specified and reported in Cochrane reviews, including the outcomes listed in the summary of findings (SoF) tables. As part of this work, an earlier exploratory survey of the outcomes of newly published 2007 and 2011 Cochrane reviews was performed. This survey examined the use of COS, the variety of specified outcomes, and outcome reporting in Cochrane reviews by Cochrane Review Group (CRG). To examine changes over time and to explore outcomes that were repeatedly specified over time in Cochrane reviews by CRG, we conducted a follow-up survey of outcomes in 2013 Cochrane reviews.
A descriptive survey of outcomes in Cochrane reviews that were first published in 2013. Outcomes specified in the methods sections and reported in the results section of the Cochrane reviews were examined by CRG. We also explored the uptake of SoF tables, the number of outcomes included in these, and the quality of the evidence for the outcomes.
Across the 50 CRGs, 375 Cochrane reviews that included at least one study specified a total of 3142 outcomes. Of these outcomes, 32 % (1008) were not reported in the results section of these reviews. For 23 % (233) of these non-reported outcomes, we did not find any reason in the text of the review for this non-report. Fifty-seven percent (216/375) of reviews included a SoF table.
The proportion of specified outcomes that were reported in Cochrane reviews had increased in 2013 (68 %) compared to 2007 (61 %) and 2011 (65 %). Importantly, 2013 Cochrane reviews that did not report specified outcomes were twice as likely to provide an explanation for why the outcome was not reported. There has been an increased uptake of SoF tables in Cochrane reviews. Outcomes that were repeatedly specified in Cochrane reviews by CRG in 2007, 2011, and 2013 may assist COS development.
KeywordsCore outcome set Systematic review Outcome reporting bias
The Core Outcome Measures for Effectiveness Trials (COMET) Initiative, launched in 2010, seeks to advance the development of core outcome sets (COS) to address problems of outcome heterogeneity and outcome reporting bias [1, 2]. COS are minimum sets of outcomes that should be measured in any trial for a specific condition of interest, thus promoting consistency in the availability of information of the effects of interventions on key outcomes . The COMET Initiative aims to link the development and use of COS with the specification of outcomes for Cochrane reviews, including the outcomes listed in their summary of findings (SoF) tables. The Cochrane Collaboration is a not-for-profit organisation that produces systematic reviews (Cochrane reviews), which summarise information from individual studies to inform health decisions . Organisationally, it is structured into Cochrane Review Groups (CRGs) that are supported by a central administration. CRGs are disease and health condition-focused groups that work with Cochrane review authors and editors to develop Cochrane reviews. Summary of findings tables were developed by Grading of Recommendations Assessment, Development and Evaluation (GRADE) and were introduced into Cochrane reviews in 2008. These tables include a summary of the evidence for important outcome(s), as selected by the review authors, and the quality of this evidence . They improve readers’ understanding and speed of retrieval of the findings of Cochrane reviews, and thus, inclusion of the most relevant outcomes in these tables is important [6, 7]. As COS are key outcomes that should be measured, they may provide a useful way to select outcomes to include in SoF tables. In a survey of CRG co-ordinating editors, 73 % suggested that COS should be used in SoF tables .
As part of our work within the COMET Initiative, we performed an initial exploratory survey of outcomes in Cochrane reviews that were published in full for the first time in 2007 and 2011 . This survey explored the variety of outcomes used in these Cochrane reviews and the use of COS in reviews on specific conditions. The survey also identified the proportion of outcomes specified in the methods section that was reported in the results section of the Cochrane reviews and assessed the uptake of SoF tables in 2011 Cochrane reviews. Outcomes were examined by CRGs as a meaningful way to explore COS and outcome reporting in Cochrane reviews. The results of this initial survey showed that 37 % (1996/5363) of outcomes specified in the methods section of 702 Cochrane reviews that had studies in them were not reported in the results section of these reviews. For 14 % (732/5363) of cases, we did not find any reason in the text of the review for not reporting the outcome(s) in the review. None of the Cochrane reviews explicitly referred to a COS when specifying outcomes, and 31 % of 361 newly published 2011 Cochrane reviews that had studies in them had a SoF table. To determine possible changes in the use of COS in Cochrane reviews and changes in outcome reporting over time and to explore consistency in outcomes specified in reviews of individual CRGs (defined as being specified repeatedly by at least half of the new reviews in the individual CRGs in 2007, 2011, and 2013), we conducted a follow-up survey using Cochrane reviews published in full for the first time in 2013. As for the initial survey, this follow-up survey was concerned with examining what outcomes were examined in Cochrane reviews, rather than how these outcomes were measured.
The aim of this study was to survey the outcomes used in newly published 2013 Cochrane reviews as a follow-up to the survey of outcomes in newly published 2007 and 2011 Cochrane reviews .
To identify and highlight the use of COS in reviews from CRGs
To identify the number and variety (i.e. different types) of outcomes specified and reported in these Cochrane reviews by CRG
To identify the inclusion of SoF tables in these Cochrane reviews by CRG, including where SoF tables are being used, to examine the number of outcomes included in them and the quality of evidence (GRADE) for these outcomes
To compare the findings for objectives 1–3 with the findings of the initial survey of 2007 and 2011 newly published reviews
We conducted a descriptive survey of Cochrane reviews published in 2013 between March and August 2014. This follow-up survey was planned at the time of the initial survey to identify changes over time. The sample of Cochrane reviews in this repeat survey were thus published 5 and 2 years following the years of publication of the Cochrane reviews included in the initial survey, 3 years after the launch of the COMET Initiative, and 5 years after the introduction of SoF tables in Cochrane reviews.
Data that was extracted from included Cochrane reviews
• The number of included studies in the review
• The outcomes specified in the methods section of the review
• The outcomes reported in the findings section of the review (either narratively, numerically (e.g. using a risk ratio, odd ratio, risk difference) or in the meta-analysis)
• Possible reasons for not reporting an outcome in the review, for example when an outcome specified in the review had not been examined or reported in the studies included in that review
• Additional outcomes that were reported but that had not been specified in the methods section
• Inclusion of a SoF table in the review
• The number of outcomes in the SoF table (if applicable)
• The quality of evidence (GRADE) for the outcomes in the SoF table (if applicable)a
In the initial survey, 15 outcome categories had emerged from a scoping overview of the extracted data across the CRGs. These categories “Rather than attempting to define outcome domain systems, we used to extract and manage the data for analyses purposes” (, p. 239). In this sense, these categories are not definitive domains, nor do we mean them to be; rather, they are used in this survey to describe the types of categories of outcomes that are being used in Cochrane reviews and the variety of the types of individual outcomes that are reflective of these categories. These categories were: adverse events or effects (AE), mortality/survival, infection, pain, psychosocial, quality of life, activities of daily living (ADL), medication, economic, hospital, operative, compliance (with treatment), withdrawal (from treatment/study), satisfaction (patient/clinician/other healthcare provider), and other physiological or clinical (other than adverse events/effects, mortality/survival, pain, and operative outcomes). In this repeat survey, outcomes specified in the included reviews across all CRGs were categorised using these same categories so that comparisons to the initial survey could be made.
We used descriptive statistics to analyse the data. Spearman’s correlation was used to assess any link between the number of outcomes specified in a Cochrane review and the proportion of outcomes that were reported in the results section of the review and in the SoF table.
We identified 453 Cochrane reviews published in 2013 from 51 CRGs. After excluding 8 Cochrane reviews of diagnostic studies, 3 Cochrane overviews of reviews, 1 Cochrane review of qualitative studies, and all (2) Cochrane reviews by the methodology CRG, we included 439 Cochrane reviews from 50 CRGs in this survey. None of the Cochrane reviews stated explicitly that they used a COS in specifying their review outcomes.
Number of specified outcomes in reviews
The number of specified outcomes in each Cochrane review ranged from 1 (3 reviews) to 62 (1 review), with a median of 7 outcomes, and an interquartile range of 5–10 outcomes. Of the 3 Cochrane reviews that specified 1 outcome only, 1 review assessed a specific drug adverse effect , 1 review assessed interventions for smoking cessation , and 1 review examined the effectiveness of a drug for a specific type of poisoning with mortality as its only outcome . The review that specified 62 outcomes assessed different types of dietary advice to women with gestational diabetes and specified short and long-term outcomes for both mother and child .
Reporting of specified outcomes
There was a weak negative correlation between the number of outcomes specified and the proportion of the specified outcomes that were reported (Spearman’s correlation −0.3; p < 0.0001); Cochrane reviews that specified a larger number of outcomes in their methods section reported a smaller relative proportion of these outcomes in the results compared to Cochrane reviews that specified a smaller number of outcomes in their methods section. However, there was no correlation between the number of outcomes specified and the number of non-reported outcomes that did not have a clear reason for the non-report (Spearman’s correlation 0.079; p = 0.129).
When we examined primary outcomes only, we found that 84 % (698/829) of the specified primary outcomes were reported in the results section of the Cochrane reviews (range 22 % (1 CRG) to 100 % (16 CRGs) and median 89 % across CRGs). Eighty-nine percent of the 131 non-reported primary outcomes were not reported because the included studies did not either measure or report them. For the remaining 11 % of non-reported primary outcomes, we did not find any reasons in the text of the reviews for why these outcomes were not reported.
Outcome variation/consistency within Cochrane review groups
There was considerable variation in the outcomes specified in the Cochrane reviews of the individual CRGs, particularly in the category ‘other physiological/clinical’.
Additional file 1 shows the outcomes that were specified in at least half the 2013 reviews for each CRG and makes comparison with the 2007 and 2011 Cochrane reviews . The outcome ‘adverse events’ was specified in ≥50 % of reviews for 38 of the 50 CRGs, ‘quality of life’ was specified in ≥50 % of reviews for 17 of the 50 CRGs, and the outcome ‘mortality’ was specified in ≥50 % of reviews for 16 of the 50 CRGs. These were the 3 most commonly specified outcomes. Excluding CRGs with 2 or fewer new reviews in 2013 (because all specified outcomes are by default present in ≥50 % of reviews), the outcome ‘adverse events’ was specified in ≥50 % of reviews for 33 of 42 CRGs, the outcome ‘mortality’ was specified in ≥50 % of reviews for 15 of 42 CRGs, and ‘quality of life’ was specified in ≥50 % of reviews for 13 of 42 CRGs.
Summary of findings tables
Fifty-seven percent (216/375) of the Cochrane reviews that had studies included in them had a SoF table. Three Cochrane reviews that did not have any included studies also provided an empty SoF table, but these were excluded from the analysis. Of the 216 Cochrane reviews with a SoF table, 12.5 % (27/216) included 1 outcome in the table, 16.2 % (35/216) included 2 outcomes, 14.4 % (31/216) included 3 outcomes, 12.5 % (27/216) included 4 outcomes, 17.1 % (37/216) included 5 outcomes, 11.6 % (25/216) included 6 outcomes, and 14.8 % (32/216) included 7 outcomes. In addition, 1 Cochrane review had 8 outcomes and a second had 9 outcomes in their SoF tables. The number of outcomes in the SoF table was positively correlated with the number of outcomes specified in the methods section of the review (Spearman’s correlation 0.468, p < 0.0001). Sixty-nine of the 216 Cochrane reviews with a SoF table included sub-outcomes for at least 1 outcome in the SoF table. Furthermore, in 8 reviews, the SoF table listed outcomes for more than 1 follow-up period, and 8 reviews included outcomes for more than 1 intervention comparison. One review also presented outcomes for 2 different populations within the SoF table, and another review listed the outcomes separately according to the design of the studies.
The quality of evidence (GRADE) for the results for the outcomes listed in the SoF tables varied with a ‘low quality’ grade being the most common grade (30 %; 266/877 outcomes in all SoF tables). Twenty-one percent (181/877) of effect size estimates in SoF tables were based on ‘very low quality’ evidence, 20 % (178/877) on ‘moderate quality’ evidence, and 8 % (73/877) had a ‘high quality’ grade. For the remaining outcomes listed in the SoF tables (179/877), the outcomes had not been either measured or reported in the included studies (118/877), insufficient data were present to estimate an effect size (22/877), or there was no reason provided for not reporting a quality grade (29/877). One Cochrane review used a different evidence quality grading system (strong/moderate/limited/conflicting/inconclusive/no evidence).
In this follow-up survey of outcomes in Cochrane reviews, we found that the numbers of specified outcomes in the reviews were proportionately similar for 2007, 2011, and 2013 newly published Cochrane reviews, and the median number of outcomes specified in the methods section of the reviews was 7 for all years . However, the maximum number of outcomes was only 26 in 2007 and 2011, whereas 7 Cochrane reviews from 2013 specified a greater number of outcomes. Although larger numbers of specified outcomes were not related to non-reporting in the absence of a clear reason being given for this non-report, the selection of outcomes for Cochrane reviews should receive careful consideration .
In this follow-up survey, 68 % of all specified outcomes were reported in the results section of the Cochrane reviews. This is an increase when compared to the 63 % of specified outcomes that were reported in 2007 and 2011 Cochrane reviews . Moreover, 7 % of all specified outcomes were not reported, and we did not find any reason for this non-report in the text of these Cochrane reviews in 2013. This compares favourably to the earlier survey that found that 14 % of specified outcomes were not reported without reasons given for this non-report in the text of these Cochrane reviews . Outcome reporting bias, that is, the selection of reported outcomes based on the outcome results, may significantly impact on the findings of studies and Cochrane reviews . This observed reduction (50 %) over time in non-reporting of outcomes with no reason provided is welcome. Specified primary outcomes were also more frequently reported in 2013 than in the earlier survey, which is reassuring. However, in this survey, we compared the outcomes reported in the methods section of the Cochrane reviews with the outcomes reported in the results section. Ideally, the outcomes specified in the protocols would be compared with the outcomes reported in the results section of the completed Cochrane reviews; however, this was beyond the scope of the survey and is recognised, by the authors, as a limitation.
Outcome variation/consistency within Cochrane review groups
Considering the findings for the specified outcomes within each CRG (Additional file 1), 48 of the 50 CRGs had at least 1 outcome that was specified in more than half of its reviews. This could be seen as an increase in consistency in the specified outcomes within CRGs (i.e. outcomes that are specified repeatedly in at least half of the reviews of an individual CRG) compared to 2007 and 2011, when only 41 of 50 CRGs had at least 1 outcome specified in more than half of its reviews. However, caution should be taken when considering this finding as 7 CRGs published 2 or fewer new reviews in 2013, meaning that all specified outcomes for these CRGs must have been specified in at least half of their reviews.
The slight increase in CRGs that specified the outcome ‘adverse event/effects’ in at least half of the reviews from 71 % of CRGs in 2007 and 2011 to 76 % CRGs in 2013 is welcome. However, this is not yet in line with the Cochrane Handbook for Systematic Reviews of Interventions’ recommendation that all Cochrane reviews should try to include adverse effects of interventions .
Comparing the outcomes for each CRG that were specified in at least half of the CRGs’ reviews in 2013 to those published in 2007 and 2011  in greater detail (Additional file 1), consistency (defined as specified repeatedly in the methods section of at least half of the reviews in the CRG) across time varied from 1 CRG having no single outcome present in at least 50 % of reviews in 2007, 2011 and 2013, to 2 CRGs that had identical outcomes present in at least 50 % of reviews for the reviews published in 2007, 2011, and 2013. The outcomes that appear to have been consistently specified in Cochrane reviews over the last six years could possibly be considered for inclusion in a preliminary COS. However, the variation in the outcomes used in the reviews of a CRG will depend on the scope of the individual CRG. Logically, CRGs with a broader scope of reviews, covering multiple areas of health care, are likely to have a wider variety of outcomes.
Summary of findings tables
The guidance in the Cochrane Handbook for Systematic Reviews of Interventions strongly recommends that Cochrane reviews should present the most important outcomes in a SoF table . There has been an increased uptake of SoF tables in Cochrane reviews over time, with 57 % of the 2013 Cochrane reviews including a SoF table, compared to 31 % in 2011 . However, the uptake of SoF tables varies greatly across CRGs. This rise may be due to more CRG editors requesting it, as well as increased training amongst editorial staff and authors in producing SoF tables. For example, three Cochrane reviews that had no studies in them still included a SoF table. The corresponding CRGs appear to request a SoF table, as standard, for every review as all of their 2013 published reviews had a SoF table. With 73 % of surveyed CRG editors suggesting that COS should be used in a SoF table , the increasing trend of including a SoF table in Cochrane reviews could be an opportunity to promote the use of COS in Cochrane reviews. An examination of the Cochrane reviews for our survey, however, found that as yet, none of the included Cochrane reviews stated specifically that they were using a COS to guide the selection of their outcomes or their decisions on which outcomes to present in the SoF table.
In line with the recommendation of identifying a maximum of up to 7 outcomes for including in a SoF table , all but 2 reviews had 7 or fewer outcomes in their SoF table. Four reviews did not have a grade of the quality of the evidence for the outcomes in their SoF table, despite this being part of the standard content .
The proportion of specified outcomes that were reported in Cochrane reviews that had studies in them increased over time, from 61 % in 2007 to 65 % in 2011 and 68 % in 2013. Importantly, 2013 Cochrane reviews that did not report specified outcomes were twice as likely to provide an explanation for this non-report (i.e. 14 % in the 2007 and 2011 Cochrane reviews combined, compared to 7 % in the 2013 Cochrane reviews). Ideally, to reduce the potential for selection and reporting biases, a legitimate reason should be given for why all non-reported specified outcomes in Cochrane reviews are not being reported.
None of the Cochrane reviews stated explicitly that they used a COS in deciding which outcomes to specify in the methods sections of their reviews or to include in their SoF tables, although they may have considered a COS without clearly saying that specified outcomes came from it. Data on outcomes that are repeatedly being specified in the Cochrane reviews of a CRG at the 3 time points of the initial survey and this repeat survey (2007, 2011, 2013) may be useful for COS developers when considering what outcomes to include in a preliminary set of outcomes for a COS. It can provide another source of information in addition to the involvement of key stakeholders in COS development .
There has been an increased inclusion of SoF tables in Cochrane reviews, but these were still absent from nearly half of 2013 Cochrane reviews. As the use of SoF tables continues to be promoted, development of COS could go hand in hand with an increasing use of SoF tables in Cochrane reviews.
Core outcome measures for effectiveness trials
Core outcomes set
Cochrane Review Group
- SoF table:
Summary of findings table
The COMET Initiative is pleased to acknowledge funding from the European Union Seventh Framework Programme ([FP7/2007–2013] [FP7/2007–2011]) under grant agreement n°305081.
- Dwan K, Gamble C, Williamson PR, Kirkham JJ. Systematic review of the empirical evidence of study publication bias and outcome reporting bias—an updated review. PLoS One. 2013;8(7):e66844.View ArticlePubMedPubMed CentralGoogle Scholar
- Williamson PR, Altman DG, Blazeby JM, Clarke M, Devane D, Gargon E, et al. Developing core outcome sets for clinical trials: issues to consider. Trials. 2012;13:132.View ArticlePubMedPubMed CentralGoogle Scholar
- The COMET Initiative website. 2011–2015. http://www.comet-initiative.org. Accessed 15 June 2014.
- The Cochrane Collaboration website. 2015. http://www.cochrane.org/about-us. Accessed 20 May 2014.
- Higgins JPT, Green S. Cochrane handbook for systematic reviews of interventions version 5.1.0.: The Cochrane Collaboration; 2011.
- Rosenbaum SE, Glenton C, Nylund HK, Oxman AD. User testing and stakeholder feedback contributed to the development of understandable and useful summary of findings tables for Cochrane reviews. J Clin Epidemiol. 2010;63(6):607–19.View ArticlePubMedGoogle Scholar
- Rosenbaum SE, Glenton C, Oxman AD. Summary-of-findings tables in Cochrane reviews improved understanding and rapid retrieval of key information. J Clin Epidemiol. 2010;63(6):620–6.View ArticlePubMedGoogle Scholar
- Kirkham JJ, Gargon E, Clarke M, Williamson PR. Can a core outcome set improve the quality of systematic reviews?—a survey of the co-ordinating editors of Cochrane Review Groups. Trials. 2013;14:21.View ArticlePubMedPubMed CentralGoogle Scholar
- Smith V, Clarke M, Williamson P, Gargon E. Survey of new 2007 and 2011 Cochrane reviews found 37 % of prespecified outcomes not reported. J Clin Epidemiol. 2015;68(3):237–45.View ArticlePubMedGoogle Scholar
- Rizzuto I, Behrens Renee F, Smith Lesley A. Risk of ovarian cancer in women treated with ovarian stimulating drugs for infertility. In: Cochrane Database of Systematic Reviews. John Wiley & Sons, Ltd; 2013, Issue 8. Art. No.: CD008215. doi:10.1002/14651858. CD008215. Pub2.
- van der Meer Regina M, Willemsen Marc C, Smit F, Cuijpers P. Smoking cessation interventions for smokers with current or past depression. In: Cochrane Database of Systematic Reviews. John Wiley & Sons, Ltd; 2013, Issue 8. Art. No.: CD006102. doi:10.1002/14651858. CD006102. Pub2.
- Deng J, Huo D, Wu Q, Zhu L, Liao Y. Xuebijing for paraquat poisoning. In: Cochrane Database of Systematic Reviews. John Wiley & Sons, Ltd; 2013, Issue 7. Art. No.: CD010109. doi:10.1002/14651858. CD010109. Pub2.
- Han S, Crowther Caroline A, Middleton P, Heatley E. Different types of dietary advice for women with gestational diabetes mellitus. In: Cochrane Database of Systematic Reviews. John Wiley & Sons, Ltd; 2013, Issue 3. Art. No.: CD009275. doi:10.1002/14651858. CD009275. Pub2.
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.