Of 8434 studies included in 586 Cochrane reviews, 53 % were rated at low risk, 34 % were rated at unclear risk and 13 % were rated at high risk of bias due to selective reporting. We classified the reasons for high-risk judgements into nine categories. The most common reason was concern about outcome non-reporting bias (i.e. non-/partial reporting of at least one outcome). Few studies were rated at high risk because of concerns about bias in selection of the reported result (e.g. reporting of only a subset of measurements, analysis methods or subsets of the data that were pre-specified). Review authors often specified in RoB tables the study outcomes that were not reported (84 % of studies), but less frequently specified the outcomes that were partially reported (61 % of studies), or which were concerning for another reason (49 %). At least one study was rated at high risk of outcome non-reporting bias in 31 % of reviews. In a random sample of these reviews, only 30 % incorporated this information when interpreting results, by acknowledging that the synthesis of an outcome was missing data that were not/partially reported.
A strength of our study is that we examined a large cohort of Cochrane reviews, which comprised all reviews published during a specific period (rather than a non-randomly selected sample). Further, collection of data on judgements (low/unclear/high) and supporting text in RoB tables was automated by Cochrane database managers, which removed the potential for errors due to manual data extraction. There are also some limitations. Only one author classified reasons for the judgements of high risk of selective reporting bias, so there is potential for misclassification. However, category labels for many reasons were re-evaluated on multiple occasions, as modifications to categories were made whenever new examples were encountered; this may have reduced the potential for misclassification. Further, it is possible that some studies we examined were included in more than one of the included reviews. Therefore, our estimates of the number of studies at low/unclear/high risk of bias may have double-counted some studies. However, given that Cochrane strives to produce reviews addressing mutually exclusive questions, we suspect that the number of overlapping studies is low. Finally, we only examined Cochrane reviews so our findings may not generalise to non-Cochrane reviews which use the RoB tool.
The percentage of Cochrane reviews in our sample with at least one study suspected of outcome non-reporting bias (31 %) is lower than that observed in previous research. This bias was suspected in at least one study in 34 % of 283 Cochrane reviews published between 2006 and 2007 [2], but only the primary outcome in each review (rather than all outcomes) was assessed. When all outcomes were assessed in 46 Cochrane cystic fibrosis reviews, 100 % of reviews included at least one study suspected of outcome non-reporting bias [17]. Rather than use the risk of bias assessments by Cochrane reviewers, both investigations used a 9-point classification system to assess studies (ORBIT classification [2, 18]), and involved methodologists in the assessment. It is possible that ours is an underestimate of the true extent of the problem of outcome non-reporting bias, because of variation in how review authors interpret the guidance for the RoB tool, and in how Cochrane Review Groups enforce this guidance. In a 2014 survey of managing and coordinating editors of 42 Cochrane Review Groups, only 57 % expected review authors to search for trial protocols as a step in performing the assessment, and only 23 % considered their review authors to be moderately or largely competent in performing assessments [19]. Therefore, estimates of the frequency of biased studies based on routinely collected risk of bias assessments by Cochrane reviewers should be interpreted with caution [20].
Many of the reasons for high risk of bias judgements were poorly articulated in the RoB tables. For example, statements such as “Some outcomes were partially reported” (encountered in 39 % of studies) make it impossible for readers to know which outcomes to interpret with caution unless they retrieve the primary study report. Further, statements suggesting concern about how outcome data were analysed (e.g. “trialists reported change from baseline values” or “trialists reported unadjusted effect estimates”) are incomplete; it is unclear if review authors were concerned that the decision to report these effect estimates was data-driven or because they find such analytic strategies inappropriate in general. Also, rating a study at high risk of bias because “no protocol was available” means readers are left to guess whether the review authors suspect some outcomes are missing from the published report, or that the reported outcome data have been selected on the basis of the results, or both these reasons, or neither. Review authors often failed to acknowledge that a synthesis of an outcome was missing data that were not/partially reported, and this may have occurred for several reasons. It is possible that review authors believe that completing RoB tables is sufficient, without considering that readers may not refer to these tables [21]. Authors may believe readers are likely to ignore any narrative description of the risk of outcome non-reporting bias and instead just focus on the synthesised effect estimate. Further, the Cochrane Handbook currently does not provide a framework to guide review authors to consider the extent of missing outcome data within a synthesis, and whether its absence is likely to have biased the result (that is, the corresponding risk of bias in the systematic review effect estimate).
Developers of future risk of bias tools could address the problems discussed thus far by adopting the following suggestions. We believe that assessments should be directed at specific results rather than at the study as a whole, to account for that fact that risk of bias may not be the same for each result. Further, we propose that tools designed to assess the risk of bias in effect estimates of individual primary studies should assess bias in selection of the reported result but not outcome non-reporting bias. Outcome non-reporting bias could instead be appraised using a different mechanism, such as a tool to assess the risk that a synthesis (rather than an individual primary study) is affected by reporting biases; this tool could also address the risk of bias due to unpublished studies (“publication bias”) [12]. We are currently involved in projects to develop a reporting bias tool for systematic reviews, and to revise the Cochrane risk of bias tool for randomised trials in line with the suggestions outlined above. We anticipate that these initiatives will help review authors derive more appropriate conclusions about the benefits and harms of interventions.