Skip to main content

Table 3 Assessment of risk of bias in SRs and primary studies

From: Toward a comprehensive evidence map of overview of systematic review methods: paper 2—risk of bias assessment; synthesis, presentation and summary of the findings; and assessment of the certainty of the evidence

Step Sub-step Methods/approaches Sources (first author, year)
▪ Examples
1.0 Plan to assess risk of bias (RoB) in the included SRs§
  1.1 Determine how to assess RoB in the included SRs
      1.1.1 Select an existing RoB assessment tool for SRs Baker 2014 [43]; Becker 2008 [4]; Bolland 2014 [5]; Büchter 2011 [45, 65]; Caird 2015 [1]; Chen 2014 [46]; CMIMG 2012 [47]; Cooper 2012 [6]; Flodgren 2011 [49]; Foisy 2011 [50]; Foisy 2014 [52]; Hartling 2012 [53]; Hartling 2013 [54]; Jadad 1997 [29]; James 2014 [58]; JBI 2014 [39, 59]; Kovacs 2014 [60]; Kramer 2009 [61]; Li 2012 [62]; Pieper 2012 [3]; Pieper 2014c [66]; Pieper 2014d [68]; Pieper 2014a [17]; Robinson 2015 [24, 69,70,71,72]; Ryan 2009 [25]; Silva 2014 [75]; Singh 2012 [76]; Smith 2011 [77]; Thomson 2010 [26]; Whiting 2013 [12]
      1.1.2 Adapt an existing RoB tool (e.g. selecting or modifying items for the overview) CMIMG 2012 [47]; Hartling 2012 [53]; Jadad 1997 [29]; Pollock 2015 [31]
▪ Pollock 2015 assessed 4 (of 11) AMSTAR items thought to be the most important sources of bias, and developed sub-questions for each [31] ▪ Reporting selected items/domains modifies the tool, since some items/domains are ignored [53]
      1.1.3 Develop a RoB tool customised to the overview CMIMG 2012 [47]; Cooper 2012 [6]; Pieper 2012 [3]; Pieper 2014a [17]
      1.1.4 Use existing RoB assessments Baker 2014 [43]
▪ Use quality assessments of SRs published by Health EvidenceTM [27] or Health Systems Evidence [81]
      1.1.5 Describe characteristics of included SRs that may be associated with bias or quality without using or developing a tool Pieper 2014a [17]; Robinson 2015 [24, 69,70,71,72]
  1.2 Determine how to summarise or score the RoB assessments for SRs
      1.2.1 Report assessment for individual items or domains (with or without rationale for judgements) Hartling 2012 [53]
      1.2.2 Summarise assessments across items or domains by using a scoring system§§ JBI 2014 [39, 59]; Pieper 2014a [17]; Robinson 2015 [24, 69,70,71,72]; Whiting 2013 [12]; Ryan 2009 [25]; Silva 2014 [75]
▪ Sum items, assigning equal or unequal weight to each (JBI 2014 [39, 59])
▪ Calculate the mean score across items (JBI 2014 [39, 59])
      1.2.3 Summarise assessments across items or domains, then use cut-off scores or thresholds to categorise RoB using qualitative descriptors (e.g. low, moderate or high quality)§§ Crick 2013 [48]; JBI 2014 [39, 59]; Robinson 2015 [24, 69,70,71,72]; Ryan 2009 [25]; Silva 2014 [75]; Singh 2012 [76]
▪ Pollock 2015 [31] set cut-offs for rating an SR as having no serious limitations (‘yes’ response to 4/4 AMSTAR items), serious limitations (‘yes’ to 3/4 items and 1 ‘unclear’), or very serious limitations (‘yes’ to < 3/4)
▪ SRs that score < 3/10 on the AMSTAR scale might be considered low quality, 4-6/10 moderate quality, and 7–10/10 high quality (JBI 2014 [39, 59])
▪ All domains/items required (all domains/items required for SR to be deemed low RoB)
  1.3 Determine how to present the RoB assessments for SRs
      1.3.1 Display assessments in table(s) (e.g. overall rating in summary of findings table, and another table with RoB items for each SR) Aromataris 2015 [39, 59]; Becker [4]; Chen 2014 [46]; Hartling 2012 [53]; Smith 2011 [77]
      1.3.2 Display assessments graphically Crick 2015 [48]
▪ ROBIS RoB graph depicting authors’ judgments about each domain presented as percentages across all included SRs [15]
▪ Harvest plot, which depicts results according to study size and quality ([48])
      1.3.3 Report assessments in text Aromataris 2015 [39, 59]; Chen 2014 [46]; Hartling 2012 [53]; Li 2012 [62]
2.0 Plan how the RoB of primary studies will be assessed or re-assessed
  2.1 Determine how to assess the RoB of the primary studies in the included SRs (and any additional primary studies)
      2.1.1 Report RoB assessment of primary studies from the included SRs, using the approaches specified for data extraction to deal with missing, flawed assessments, or discrepant/discordant assessments of the same primary study (i.e. where two or more SRs assess the same study using different tools or report discordant judgements using the same tool) (See ‘Data extraction’ table in [10]). Aromataris 2015 [39, 59]; Becker 2008 [4]; Caird 2015 [1]; CMIMG 2012 [47]; Cooper 2012 [6]; Hartling 2012 [53]; Hartling 2014 [55]; Jadad 1997 [29]; Ioannidis 2009 [57]; Kramer 2009 [61]; Ryan 2009 [25]; Singh 2012 [76]; Thomson 2010 [26]
▪ Report RoB assessments of primary studies from the included SR(s), noting missing data and discrepancies (Hartling 2012 [53]; JBI 2014 [39]; Robinson 2015 [24, 69,70,71,72]
▪ Report RoB assessments from the highest quality SR (Jadad 1997 [29])
      2.1.2 Report RoB assessment of primary studies from the included SRs after performing quality checks to verify that the assessment method has been applied appropriately and consistently across a sample of primary studies Becker 2008 [4]; Hartling 2014 [55]; Ioannidis 2009 [57]; Jadad 1997 [29]; Kramer 2009 [61]; Moja 2012 [63]; Robinson 2015 [24, 69,70,71,72]; Thomson 2010 [26]
▪ Randomly sample a number of included RCTs, retrieve data from the original trial reports, and independently check 10% of RCT data from the included MAs to verify assessments were done without error and consistently
▪ Repeat RoB assessments on a sample of SRs to verify and check for consistency (Robinson 2015 [24, 69,70,71,72])
      2.1.3 (Re)-assess RoB of some or all primary studiesa CMIMG 2012 [47]; Cooper 2012 [6]; Hartling 2012 [53]; Jadad 1997 [29]; Moja 2012 [63]; Thomson 2010 [26]
▪ When two different tools are used, then assess the primary studies using one tool
▪ When two different tools are used (e.g. Cochrane RoB tool [67] and Jadad tool [29]; then re-assess RoB by standardising the assessments based on the Cochrane RoB domains, and match data from assessments from other tools to these domains)
      2.1.4 Don’t report or assess RoB of primary studies Inferred
  2.2 Determine how to summarise the RoB assessments for primary studies
      2.2.1 Report assessment for individual items or domains (with or without rationale for judgements)a JBI 2014 [39, 59]; Pieper 2014c [66]; Robinson 2015 [24, 69,70,71,72]; Ryan 2009 [25]; Silva 2014 [75]
      2.2.2 Summarise assessments across items or domains by using a scoring system§§ JBI 2014 [39, 59]; Pieper 2014c [66]; Robinson 2015 [24, 69,70,71,72]; Ryan 2009 [25]; Silva 2014 [75]
▪ Sum items, assigning equal or unequal weight to each (JBI 2014 [39, 59])
▪ Calculate the mean score across items (JBI 2014 [39, 59])
      2.2.3 Summarise assessments across items or domains, then use cut-off scores or thresholds to describe RoB (e.g. low, moderate and high quality)§§ JBI 2014 [39, 59]; Robinson 2015 [24, 69,70,71,72]; Ryan 2009 [25]; Silva 2014 [75]; Singh 2012 [76]
  2.3 Determine how to present the RoB assessments for primary studies
      2.3.1 Display assessments in table(s) (e.g. overall rating in summary of findings table, and another table with RoB items for each primary study)a Aromataris 2015 [39, 59]; Becker 2008 [4]; Chen 2014 [46]; Hartling 2012 [53]; Smith 2011 [77]; JBI 2014 [39, 59]
      2.3.2 Display assessments graphicallya Crick 2015 [48]
▪ Cochrane RoB graph depicting authors’ judgments about each domain presented as percentages across all included SRs [67]
▪ Harvest plot, which depicts results according to study size and quality ([48])
      2.3.3 Report assessments in texta Aromataris 2015 [39, 59]; Chen 2014 [46]; Hartling 2012 [53]; Li 2012 [62]; Smith 2011 [77]
3.0 Plan the process for assessing RoB
  3.1 Determine the number of overview authors required to assess studiesa
      3.1.1 Independent assessment by 2 or more authors Baker 2014 [43]; Becker 2008 [4]; Cooper 2012 [6]; JBI 2014 [39]; Li 2012 [62]; Ryan 2009 [25]
      3.1.2 One author assessment Inferred
      3.1.3 One assessment, 2nd confirmed Cooper 2012 [6]
      3.1.4 One assessment, 2nd confirms if uncertainty Cooper 2012 [6]
  3.2 Determine if authors (co-)authored one or several of the SRs included in the overview, and if yes, plan safeguards to avoid bias in RoB assessment Büchter 2011 [45, 65]
▪ Assessment of RoB of included SRs done by overview authors who were not authors of the SRs
  1. AMSTAR A Measurement Tool to Assess Systematic Reviews; CMIMG Comparing Multiple Interventions Methods Group; JBI Joanna Briggs Institute; OQAQ Overview Quality Assessment Questionnaire; RoB risk of bias; ROBIS Risk of Bias In Systematic reviews; SRs systematic reviews
  2. §We refer to ‘risk of bias’ assessment, since assessment of SR or primary study limitations should focus on the potential of those methods to bias findings. However, the terms quality assessment and critical appraisal are common, particularly when referring to the assessment of SR methods, and hence our analysis includes all relevant literature irrespective of terminology
  3. §§As is the case with assessment of RoB in primary studies, concerns have been raised about the validity of presenting a summary score or qualitative descriptors based on scores (e.g. low, moderate, high quality) [12, 17]
  4. aAdaptation of the step from SRs to overviews. No methods evaluation required, but special consideration needs to be given to unique issues that arise in conducting overviews