Skip to main content

Table 8 Characteristics of primary methods studies and assessment of risk of bias

From: Toward a comprehensive evidence map of overview of systematic review methods: paper 2—risk of bias assessment; synthesis, presentation and summary of the findings; and assessment of the certainty of the evidence

  Study ID (first author, year)
Pieper 2014e [34] Whiting 2016 [15] Parmelli 2011 [19] Pieper 2014b [35]
Characteristics of the studies
 Title Impact of choice of quality appraisal tool for systematic reviews in overviews ROBIS: A new tool to assess risk of bias in systematic reviews was developed Using AMSTAR to assess the methodological quality of systematic reviews: An external validation study Systematic review finds overlapping reviews were not mentioned in every other overview
 Primary objective To examine reliability, validity and feasibility of four quality appraisal tools in an SR and explore how the choice of tool impacts the findings of the evidence synthesis To develop ROBIS, a new tool for assessing the risk of bias in systematic reviews (rather than in primary studies) To measure the reliability, construct validity and feasibility of AMSTAR on a sample of SRs in different medical fields To develop two measures to quantify the degree of overlap of primary studies across SRs and evaluate the validity of the measures
 Name of the included tools or measures AMSTAR [22, 23, 37], AQASR [117], OQAQ [28], RAPiD [88]a ROBIS [15] AMSTAR [22, 23], OQAQ CA and CCA [35]
 Type of assessment Assess reliability/ construct validity of the tool Assess content validity/reliability Assess reliability/ construct validity of the tool Construct validity testing of the measures
 Content validity—methods of item generation Not applicable—existing tool Content (domains and items) was based on a reporting standard for SRs (i.e. MECIR [118]) and an SR of 40 tools designed to assess the quality of SRs or meta-analyses Not applicable—existing tool Not applicable—not a tool
 Content validity—comprehensiveness Not applicable—existing tool Content experts (methodologists, systematic reviewers, guideline developers) reviewed the draft ROBIS tool in a face-to-face meeting and Delphi process Not applicable—existing tool Not applicable—not a tool
 Reliability—description of reliability testing Inter-rater reliability (agreement) between two review authors who independently applied AMSTAR, OQAQ, RAPiD and AQASR to 32 SRs. A 4-week interval separated assessment with each tool. Agreement was assessed at item level for AMSTAR and OQAQ, and domain level for RAP and AQASR (Cohen’s kappa) Inter-rater reliability (agreement) between two review authors who independently applied ROBIS to 8 SRs. Agreement was assessed at domain level (% agreement) Inter-rater reliability (agreement) between two review authors who independently applied AMSTAR to 54 SRs. Agreement was assessed at item level (Cohen’s weighted kappa) Not applicable
 Tests of validity—description of correlation coefficient testing Correlation between summary scores on OQAQ and RAPiD (not done for tools without summary scores). Qualitative assessment of whether assessment of SR quality with different tools altered overall conclusions about strength of association between volume and outcomes (where SR quality was one of four elements used to determine strength) Not assessed Correlation between scores on AMSTAR and scores from a similar measure, the OQAQ (Pearson’s rank correlation coefficient, results not reported in abstract) Correlation between measures (CA, CCA) calculated on a sample of overviews (Kendall τ-b) with each other, and each measure with the number of SRs and number of primary publications. Examined whether the measures were associated with publication source (HTA or journal publication), hypothesizing that HTA reports may have more overlap
 Other assessment (feasibility, acceptability, piloting) Time to complete Piloting involved three workshops on ROBIS where participants piloted the tools and provided feedback Time to complete Not reported
Risk of bias in the primary methods studies
 Existence of a protocol Not reported Not reported Not reported Not reported
 Method to select the sample of SRs to which the tool/measure was applied Convenience: SRs were included studies in an overview that examined associations between surgery volume and outcomes (not intervention effects) Convenience: SRs were included studies in an overview, being conducted by authors independent of the developers of ROBIS Convenience: SRs were in two different medical fields (hypertension, colorectal cancer), and described as a convenience sample but unclear how they were selected Census: All overviews identified from a literature search of five databases. Handsearching of websites of HTA agencies. Search restricted to articles published between 2009 and 2011.
 Process for selecting the raters/assessors who applied the tool/measureb Convenience: Raters were authors of an overview in which AMSTAR was used Convenience: Raters were authors of an overview in which ROBIS was piloted, and were independent of the tool developers. Unclear how they were recruited Unclear: No description of how raters were selected Not applicable
 Pre-specified hypotheses for testing of validity No: The expected direction or magnitude of correlation was not specified. ‘The Spearman’s rank correlation coefficient was calculated to compare the CATs [critical appraisal tool]’ Not applicable: no testing of validity No: The expected direction or magnitude of correlation was not specified. ‘Construct validity was investigated comparing the two instruments using Pearson’s rank correlation coefficient.’ Yes: ‘We hypothesized that the CA should have a strong (0.60–0.80) negative correlation with the number of included reviews and, compared to this, a lower negative correlation with the number of included primary publications. In contrast, we assumed that the CCA should have a very weak (0.00–0.20) or weak (0.20–0.40) negative correlation with the number of included reviews and the primary publications.’
  1. AMSTAR A Measurement Tool to Assess Systematic Reviews; AQASR Assessing the Quality and Applicability of Systematic Reviews; CA Covered Area; CCA Corrected Covered Area; HTA Health Technology Assessment; MECIR Methodological Expectations of Cochrane Intervention Reviews; OQAQ Overview Quality Assessment Questionnaire; RAPiD Rapid Appraisal Protocol internet Database; RoB risk of bias; ROBIS Risk of Bias In Systematic reviews; SRs systematic reviews
  2. aOQAQ [28] is also referred to as OQAC (Overview Quality Assessment Checklist), and RAPiD [88] is also referred to as RAP (Rapid Appraisal Protocol)
  3. bRoB in relation to any estimates of reliability and validity