Skip to main content

Table 2 Selected key points of major comments on the Cochrane risk of bias tool for randomized clinical trials: strengths, challenges and suggestions

From: Evaluation of the Cochrane tool for assessing risk of bias in randomized clinical trials: overview of published comments and analysis of user practice in Cochrane and non-Cochrane reviews

First authora Category Theme Key point
Armijo-Olivo Strengths   None mentioned
Challenges • Implementation (1.)
• Overall risk of bias (2.)
• Bias domains (3.)
1. “…the large number of trials classified as high or unclear RoB [risk of bias] casts doubts about the discrimination power of the RoB [risk of bias] tool to […] explain variability of treatments effects across studies…”
2. “…the overall assessment of the RoB [risk of bias] may not be useful to determine quality of individual trials.”
3. “…other methodological factors could be important for evaluating RoB and could be considered for inclusion in the RoB [risk of bias] tool after careful empirical evidence testing.”
Suggestions • Guidelines (1.) 1. “Improved guidelines to apply the RoB [risk of bias] tool and revisions to the tool for different health areas are needed.”
Bero Strengths   None mentioned
Challenges • Bias domains (1.) 1. “The current Cochrane risk of bias tool is insufficient to assess bias related to study funding sources.”
Suggestions • Funding (1.) 1. “…the Cochrane risk of bias tool should include funding source as a standard item because: 1. Funding source fits the definition of bias, 2. There is empirically-based evidence of bias related to funding source, 3. The observed bias related to funding source cannot be captured by the risk of bias criteria currently assessed with the risk of bias tool, 4. Risks of bias are not mutually exclusive, 5. Bias may be related to funding source even when all studies are industry-funded.”
Boutron Strengths • Aims (1. 2.)
• Improvement (3.)
• Transparency (1.)
1. “…the tool aims at being completely transparent, with a separation of the facts and reviewers’ judgments. This aim is particularly important because reviewers, editors, and readers can challenge the author on the judgment.”
2. “…the tool is intended to assess the risk of bias related to the design, conduct, and analyses of the trial and not the quality of reporting.”
3. “This tool has been an important step forward in the assessment of the risk of bias in systematic reviews and meta-analyses.”
Challenges   None mentioned
Suggestions   None mentioned
De Bruin Strengths   None mentioned
Challenges • Implementation (1.) 1. “…many do assess methodological quality, but very few incorporate them [/risk of bias assessments] in their analyses.”
Suggestions • Guidelines (1.) 1. “…systematic reviewers could consider adapting the risk-of-bias tool to the literature…”
Hartling Strengths   None mentioned
Challenges • Implementation (1.)
• Overall risk of bias (2.)
• Special situations (3.)
1. “Low agreement between reviewers suggests the need for more specific guidance regarding interpretation and application of the Risk of Bias (ROB) tool…”
2. “The majority of trials in the sample were assessed as high or unclear risk of bias…This raises concerns about the ability of the ROB [risk of bias] tool to detect differences across trials that may relate to biases in estimates of treatment effects.”
3. “…trials with different design features (e.g., crossover) or hypotheses (e.g., equivalence, non-inferiority), and those examining non-pharmacological interventions appear to create more ambiguity for risk of bias assessments.”
Suggestions • Guidelines (1.) 1. “There is a need for more detailed guidelines to apply […] the ROB [risk of bias] tool and […] further testing with the modified tool is warranted.”
Hróbjartsson Strengths • Aims (1.)
• Background (1.)
1. “The risk of bias tool provides a standardised approach, based on items selected on both theoretical and empirical grounds, and following broad consultations with clinical research methodologists.”
Challenges • Bias domains (2.)
• Implementation (1.)
1. “The risk of bias tool is a comparatively recent development that still likely needs refinement.”
2. “It is not clear that the risk of bias tool in its present version addresses this problem [of funding] adequately.”
Suggestions   None mentioned
Ivers Strengths   None mentioned
Challenges • Bias domains (1.)
• Implementation (2.)
• Overall risk of bias (3.)
1. “The risk of bias tool does not capture all sources of methodological bias and poor reporting interferes with the assessment of many domains.”
2. “While the overall risk of bias assessment using the Cochrane Risk of Bias Tool has been shown to differentiate effect sizes (i.e. higher risk of bias studies usually have larger effect sizes), 10 studies at high risk of bias may still offer valuable knowledge…”
3. “…assigning trials with high risk of bias in a single domain a status of high risk of bias overall may be arguable.”
Suggestions   None mentioned
Jefferson Strengths • Aims (1.) 1. “The real strength of the risk of bias tool appears not to be in the final judgements it enables, but rather in the process it helps facilitate: critical assessment of a clinical trial.”
Challenges • Bias domains (1. 3.)
• Implementation (2.)
1. “The current Cochrane risk of bias tool is not adequate for the task as it does not reliably identify all types of important biases, and nor does it organise and check the coherence of large amounts of information.”
2. “We found the Cochrane risk of bias tool to be difficult to apply to clinical study reports…[since]…its use lends itself to a checklist approach (in which each design item is sought and, if found, eliminated from the bias equation rather than with thought and consideration).”
3. “Many of the variables we found to be important when assessing the trial (e.g. date of trial protocol, date of un-blinding, date of participant enrolment) are simply not captured in the risk of bias tool…”
Suggestions   None mentioned
Katikireddi Strengths   None mentioned
Challenges • Implementation (1.) 1. “…reviewers are struggling to understand and/or operationalize current guidance on how to conduct and incorporate critical appraisal [/risk of bias] within synthesis.”
Suggestions • Guidelines (1.)
• Research (1.)
1. “Further research is required to establish the relative importance of different forms of bias and their likely impact […] and also to clarify how critical appraisals should be incorporated into SR [systematic review] findings.”
Morissette Strengths • Aims (1.) 1. “The Cochrane 'Risk of bias' tool differs from other quality appraisal tools because it questions the degree to which a study’s results should be believed…”
Challenges • Implementation (1.) 1. “The results of our review provide no clear guidance as to whether risk of bias assessments should be completed in a blind or un-blind manner.”
Suggestions • Research (1.) 1. “…we encourage further research in this area [of blind vs. un-blind risk of bias assessment] and recommend using all of the important components of the Cochrane 'Risk of bias' tool.”
Moustgaard Strengths   None mentioned
Challenges • Implementation (1.) 1. “No characterization of subjective vs. objective outcomes relevant to risk of performance bias is given explicitly in the Cochrane Handbook nor did we find it in the methodological articles or the clinical trial reports we reviewed.”
Suggestions   None mentioned
Roseman Strengths   None mentioned
Challenges • Bias domains (1.) 1. “…inclusion of risk of bias from conflicts of interest could reflect mechanisms through which industry involvement can influence study outcomes that are not fully captured by the current domains of the risk of bias tool.”
Suggestions • Funding (1.) 1. “…we recommend that the Cochrane Collaboration reconsider its position that trial funding and trial author-industry financial ties not be included in the risk of bias assessment.”
Savović Strengths • Aims (3.)
• Background (1.)
• Transparency (2.)
1. “…[the tool has] a standardized approach to bias assessments…”
2. “…[the tool has] transparency provided by requesting quotes…”
3. “…[the tool provides] a platform to encourage critical thinking.”
Challenges • Bias domains (1.)
• Implementation (2.)
1. “Some of the items that authors have included (such as sample size calculations and funding source) are explicitly discouraged in the Cochrane Handbook guidance. While there is evidence that some factors are empirically associated with effect estimates, such as single versus multicentre design, early stopping of trials and funding source [14-16], the extent to which these should be considered alongside the main bias domains is still a topic of debate.”
2. “The main purpose of this evaluation was to identify potential problems with the RoB [risk of bias] that can be rectified, and we suspect that users who encountered problems are more likely to have responded. This speculation is based on the high proportion of respondents who reported having problems with some aspects of the RoB tool, especially with individual RoB domains.”
Suggestions • Guidelines (1.) 1. “It is important that guidance and training materials continue to be developed for all aspects of the tool…”
Sterne Strengths   None mentioned
Challenges • Bias domains (1.) 1. “The current RoB [risk of bias] tool does not work well for assessment of selective reporting.”
Suggestions • Funding (1.) 1. “…the Cochrane risk of bias tool should not include funding source as a standard item.”
Vale Strengths   None mentioned
Challenges • Implementation (1. 2.)
• Bias domains (2.)
1. “The Cochrane Handbook states that because the ability to measure the true bias (or even the true risk of bias) is limited, then the possibility to validate a tool to assess that risk is also limited. Nevertheless, authors of Cochrane systematic reviews are required to use the Cochrane risk of bias tool.”
2. “Assessing risk of bias was particularly difficult for the more subjective domains [i.e. ‘selective outcome reporting’ and ‘other bias’].”
Suggestions   None mentioned
  1. Major comments were defined as longer comments with a substantial reflection (typically ≥100 words of text) on the strengths or challenges of the Cochrane risk of bias tool for randomized clinical trials (i.e. the tool)
  2. aSee Additional file 2 for references