Skip to main content

Table 8 Inter-rater agreement

From: A comparison of two assessment tools used in overviews of systematic reviews: ROBIS versus AMSTAR-2

ROBIS question No. of studies Gwet’s AC1/Gwet’s AC2 95% CI p-value
Domain 1: study eligibility criteria
 1.1 14 0.73 0.46, 1.00 < 0.001
 1.2 15 0.70 0.45, 0.95 < 0.001
 1.3 15 0.62 0.39, 0.84 < 0.001
 1.4 15 0.54 0.32, 0.76 < 0.001
 1.5 15 0.64 0.40, 0.88 < 0.001
Domain 1Concerns regarding specification of study eligibility criteria 15 0.61 0.29, 0.92 0.001
Domain 2: identification and selection of studies
 2.1 15 0.53 0.36, 0.69 < 0.001
 2.2 14 0.42 0.16, 0.69 0.005
 2.3 15 0.72 0.53, 0.92 < 0.001
 2.4 15 0.31 − 0.08, 0.70 0.110
 2.5 15 0.56 0.14, 0.99 0.013
Domain 2Concerns regarding methods used to identify and/or select studies 15 0.29 0.03, 0.55 0.031
Domain 3: data collection and study appraisal
 3.1 15 0.95 0.66, 1.00 < 0.001
 3.2 15 0.65 0.47, 0.84 < 0.001
 3.3 15 0.57 0.40, 0.74 < 0.001
 3.4 15 0.55 0.23, 0.88 0.003
 3.5 15 0.81 0.51, 1.00 < 0.001
Domain 3Concerns regarding methods used to collect data and appraise studies 15 0.52 0.19, 0.83 0.004
Domain 4: synthesis and findings
 4.1 15 0.55 0.33, 0.77 < 0.001
 4.2 13 0.55 0.29, 0.81 0.001
 4.3 15 0.80 0.62, 0.98 < 0.001
 4.4 15 0.13 − 0.19, 0.45 0.405
 4.5 14 − 0.10 − 0.52, 0.33 0.633
 4.6 15 0.23 − 0.17, 0.64 0.235
Domain 4Concerns regarding the synthesis and findings 15 0.18 − 0.08, 0.44 0.154
Risk of bias in the review
 A 15 0.10 − 0.25, 0.44 0.552
 B 15 0.61 0.40, 0.83 < 0.001
 C 15 0.39 0.01, 0.76 0.009
ROB 15 0.43 0.10, 0.77 0.015
  1. Six ratings missing