The Evidence Project risk of bias tool: assessing study rigor for both randomized and non-randomized intervention studies

Table 1 Items, response choices, and inter-rater reliability estimates for the Evidence Project risk of bias tool

Risk of bias tool domains	Items	Response choices	Kappa (κ)¹
Risk of bias tool domains	Items	Response choices	Dichotomous	Categorical
Study design	1: Cohort	Yes, No	0.48
	2: Control or comparison group	Yes, No	0.80
	3: Pre/post intervention data	Yes, No	0.74
Participant representativeness	4: Random assignment of participants to the intervention	Yes, No, NA	0.78	0.56
	5: Random selection of participants for assessment	Yes, No	0.41
	6: Follow-up rate of 80% or more	Yes, No, NA, NR	0.67	0.55
Equivalence of comparison groups	7: Comparison groups equivalent on sociodemographics	Yes, No, NA, NR	0.65	0.56
Equivalence of comparison groups	8: Comparison groups equivalent at baseline on outcome measures	Yes, No, NA, NR	0.59	0.50
Median kappa score (κ) across individual items 1–8			0.66
Weighted kappa (κ_w) of the total count of items (sum of individual item dichotomous responses)			0.66

¹Kappa estimates are reported for dichotomous (Yes, No) and categorical ratings when appropriate. Categorical response sets further classify binary No ratings as: No (reported), NA (not applicable), NR (not reported). Agreement was categorized as poor (0.00), slight (0.01–0.20), fair (0.21–0.40), moderate (0.41–0.60), substantial (0.61–0.80), or almost perfect (0.81–1.00)

ISSN: 2046-4053