Skip to main content

Table 1 Items, response choices, and inter-rater reliability estimates for the Evidence Project risk of bias tool

From: The Evidence Project risk of bias tool: assessing study rigor for both randomized and non-randomized intervention studies

Risk of bias tool domains Items Response choices Kappa (κ)1
Dichotomous Categorical
 Study design 1: Cohort Yes, No 0.48  
2: Control or comparison group Yes, No 0.80  
3: Pre/post intervention data Yes, No 0.74  
 Participant representativeness 4: Random assignment of participants to the intervention Yes, No, NA 0.78 0.56
5: Random selection of participants for assessment Yes, No 0.41  
6: Follow-up rate of 80% or more Yes, No, NA, NR 0.67 0.55
 Equivalence of comparison groups 7: Comparison groups equivalent on sociodemographics Yes, No, NA, NR 0.65 0.56
8: Comparison groups equivalent at baseline on outcome measures Yes, No, NA, NR 0.59 0.50
Median kappa score (κ) across individual items 1–8   0.66  
Weighted kappa (κw) of the total count of items (sum of individual item dichotomous responses)   0.66  
  1. 1Kappa estimates are reported for dichotomous (Yes, No) and categorical ratings when appropriate. Categorical response sets further classify binary No ratings as: No (reported), NA (not applicable), NR (not reported). Agreement was categorized as poor (0.00), slight (0.01–0.20), fair (0.21–0.40), moderate (0.41–0.60), substantial (0.61–0.80), or almost perfect (0.81–1.00)