Skip to main content

Table 12 Inter-rater agreement

From: A comparison of two assessment tools used in overviews of systematic reviews: ROBIS versus AMSTAR-2

ROBIS question No. of studies Gwet’s AC1/Gwet’s AC2 95% CI p-value
Domain 1: study eligibility criteria
 1.1 16 0.57 0.17, 0.96 0.008
 1.2 16 0.71 0.55, 0.87 < 0.001
 1.3 16 0.76 0.61, 0.91 < 0.001
 1.4 16 0.71 0.54, 0.87 < 0.001
 1.5 16 0.49 0.20, 0.77 0.002
Domain 1Concerns regarding specification of study eligibility criteria 16 0.30 − 0.03, 0.63 0.072
Domain 2: identification and selection of studies
 2.1 16 0.54 0.34, 0.73 < 0.001
 2.2 16 0.64 0.37, 0.92 < 0.001
 2.3 16 0.57 0.34, 0.81 < 0.001
 2.4 16 0.50 0.27, 0.73 < 0.001
 2.5 14 0.61 0.18, 1.00 < 0.001
Domain 2Concerns regarding methods used to identify and/or select studies 16 0.43 0.13, 0.73 0.008
Domain 3: data collection and study appraisal
 3.1 14 0.82 0.51, 1.00 < 0.001
 3.2 16 0.70 0.44, 0.96 < 0.001
 3.3 16 0.72 0.52, 0.92 < 0.001
 3.4 16 0.92 0.83, 1.00 < 0.001
 3.5 15 0.66 0.21, 1.00 0.007
Domain 3Concerns regarding methods used to collect data and appraise studies 16 0.61 0.32, 0.89 < 0.001
Domain 4: synthesis and findings
 4.1 16 0.65 0.45, 0.86 < 0.001
 4.2 16 0.42 0.11, 0.73 0.011
 4.3 16 0.73 0.58, 0.88 < 0.001
 4.4 16 0.23 − 0.02, 0.48 0.072
 4.5 16 0.40 0.22, 0.57 < 0.001
 4.6 16 0.55 0.32, 0.77 < 0.001
Domain 4Concerns regarding the synthesis and findings 16 0.17 − 0.17, 0.50 0.305
Risk of bias in the review
 A 16 0.47 0.28, 0.65 0.015
 B 16 0.69 0.55, 0.82 < 0.001
 C 16 0.54 0.37, 0.72 < 0.001
ROB 16 0.47 0.17, 0.77 0.004
  1. Three ratings missing