Skip to main content

Table 4 User patterns of risk of bias implementations in Cochrane and non-Cochrane reviews—evaluation of the Cochrane tool for assessing risk of bias in randomized clinical trials

From: Evaluation of the Cochrane tool for assessing risk of bias in randomized clinical trials: overview of published comments and analysis of user practice in Cochrane and non-Cochrane reviews

Risk of bias implementation 100 Cochrane reviews (100 %) 100 non-Cochrane reviews (100 %) P value*
Risk of bias assessment in reviews    
 Any risk of bias (or quality) assessment 100 (100 %) 80 (80 %) 0.30
  Cochrane risk of bias tool 100 (100 %) 31 of 80 (39 %) 0.0002
  Jadad scale 0 (0 %) 19 of 80 (24 %) 0.0001
  PEDro scale 0 (0 %) 5 of 80 (6 %) 0.019
  Own construct or other scale 0 (0 %) 25 of 80 (31 %)c 0.0001
Descriptive use of risk of bias assessment    
 Explicit mentions risk of bias in abstract 80 (80 %) 18 of 31d (58 %) 0.42
 Explicit mentions risk of bias in discussion/conclusion 89 (89 %) 25 of 31d (81 %) 0.76
 Explicit mentions risk of bias in both abstract and discussion/conclusion 73 (73 %) 15 of 31d (48 %) 0.31
Sensitivity and subgroup analyses based on risk of bias    
 Review planned (in methods) to do sensitivity analyses 70 (70 %) 8 of 80 (10 %) 0.0001
 Review performed sensitivity analyses 19 (19 %) 11 of 80 (14 %) 0.55
  Based on overall risk of bias 2 of 19 (11 %) 9 of 11 (82 %) 0.015
  Based on individual risk of bias domains 9 of 19 (47 %) 2 of 11 (18 %) 0.45
  Unclear what analyses were based on 8 of 19 (42 %) 0 of 11 (0 %) 0.077
 Review performed, but did not plan sensitivity analyses 1 of 19 (5 %) 8 of 11 (72 %) 0.0084
 Review performed subgroup analysesa 2 (2 %) 0 of 80 (0 %) 0.50
 Review planned, but did not perform analyses 50 of 70 (71 %) 5 of 8 0.52
  Due to insufficient datab 41 of 50 (82 %) 3 of 5 0.73
  No explanation provided 9 of 50 (18 %) 2 of 5 0.33
GRADE
 Review incorporated GRADE 64 (64 %) 4 of 80 (5 %) 0.0001
  1. *P values were calculated with Fisher’s two-tailed exact test
  2. aAll subgroup analyses were based on “low” versus “high” risk of bias
  3. b“Insufficient data” was due to few trials included in the review or few trials judged as “low risk of bias”
  4. c15 non-Cochrane reviews made their own risk of bias construct/tool, eight incorporated two constructs/tools and the following constructs/tools (/methods) were used 18 times in total: CASP (×2), CEBM, Chalmers, CONSORT (×2), CTAM, Downs and Black criteria (×2), Evidence-based medicine toolkit, GRADE (×2), Methods Guide for Effectiveness and Comparative Effectiveness Reviews, MOOSE (×2), Newcastle Ottawa, QUOROM and STROBE
  5. d31 of 100 non-Cochrane reviews used the Cochrane risk of bias tool for randomized clinical trials (i.e. the tool) and were compared to the 100 Cochrane reviews that used the tool for randomized clinical trials