Table 3 Assessment of risk of bias in SRs and primary studies

      1.1.1 Select an existing RoB assessment tool for SRs

Baker 2014 [43]; Becker 2008 [4]; Bolland 2014 [5]; Büchter 2011 [45, 65]; Caird 2015 [1]; Chen 2014 [46]; CMIMG 2012 [47]; Cooper 2012 [6]; Flodgren 2011 [49]; Foisy 2011 [50]; Foisy 2014 [52]; Hartling 2012 [53]; Hartling 2013 [54]; Jadad 1997 [29]; James 2014 [58]; JBI 2014 [39, 59]; Kovacs 2014 [60]; Kramer 2009 [61]; Li 2012 [62]; Pieper 2012 [3]; Pieper 2014c [66]; Pieper 2014d [68]; Pieper 2014a [17]; Robinson 2015 [24, 69,70,71,72]; Ryan 2009 [25]; Silva 2014 [75]; Singh 2012 [76]; Smith 2011 [77]; Thomson 2010 [26]; Whiting 2013 [12]

      1.1.2 Adapt an existing RoB tool (e.g. selecting or modifying items for the overview)

CMIMG 2012 [47]; Hartling 2012 [53]; Jadad 1997 [29]; Pollock 2015 [31]

▪ Pollock 2015 assessed 4 (of 11) AMSTAR items thought to be the most important sources of bias, and developed sub-questions for each [31] ▪ Reporting selected items/domains modifies the tool, since some items/domains are ignored [53]

      1.1.3 Develop a RoB tool customised to the overview

CMIMG 2012 [47]; Cooper 2012 [6]; Pieper 2012 [3]; Pieper 2014a [17]

      1.1.4 Use existing RoB assessments

Baker 2014 [43]

▪ Use quality assessments of SRs published by Health Evidence^TM [27] or Health Systems Evidence [81]

      1.1.5 Describe characteristics of included SRs that may be associated with bias or quality without using or developing a tool

Pieper 2014a [17]; Robinson 2015 [24, 69,70,71,72]

      1.2.1 Report assessment for individual items or domains (with or without rationale for judgements)

Hartling 2012 [53]

      1.2.2 Summarise assessments across items or domains by using a scoring system^§§

JBI 2014 [39, 59]; Pieper 2014a [17]; Robinson 2015 [24, 69,70,71,72]; Whiting 2013 [12]; Ryan 2009 [25]; Silva 2014 [75]

▪ Sum items, assigning equal or unequal weight to each (JBI 2014 [39, 59])

▪ Calculate the mean score across items (JBI 2014 [39, 59])

      1.2.3 Summarise assessments across items or domains, then use cut-off scores or thresholds to categorise RoB using qualitative descriptors (e.g. low, moderate or high quality)^§§

Crick 2013 [48]; JBI 2014 [39, 59]; Robinson 2015 [24, 69,70,71,72]; Ryan 2009 [25]; Silva 2014 [75]; Singh 2012 [76]

▪ Pollock 2015 [31] set cut-offs for rating an SR as having no serious limitations (‘yes’ response to 4/4 AMSTAR items), serious limitations (‘yes’ to 3/4 items and 1 ‘unclear’), or very serious limitations (‘yes’ to < 3/4)

▪ SRs that score < 3/10 on the AMSTAR scale might be considered low quality, 4-6/10 moderate quality, and 7–10/10 high quality (JBI 2014 [39, 59])

▪ All domains/items required (all domains/items required for SR to be deemed low RoB)

      1.3.1 Display assessments in table(s) (e.g. overall rating in summary of findings table, and another table with RoB items for each SR)

Aromataris 2015 [39, 59]; Becker [4]; Chen 2014 [46]; Hartling 2012 [53]; Smith 2011 [77]

      1.3.2 Display assessments graphically

Crick 2015 [48]

▪ ROBIS RoB graph depicting authors’ judgments about each domain presented as percentages across all included SRs [15]

▪ Harvest plot, which depicts results according to study size and quality ([48])

      1.3.3 Report assessments in text

Aromataris 2015 [39, 59]; Chen 2014 [46]; Hartling 2012 [53]; Li 2012 [62]

      2.1.1 Report RoB assessment of primary studies from the included SRs, using the approaches specified for data extraction to deal with missing, flawed assessments, or discrepant/discordant assessments of the same primary study (i.e. where two or more SRs assess the same study using different tools or report discordant judgements using the same tool) (See ‘Data extraction’ table in [10]).

Aromataris 2015 [39, 59]; Becker 2008 [4]; Caird 2015 [1]; CMIMG 2012 [47]; Cooper 2012 [6]; Hartling 2012 [53]; Hartling 2014 [55]; Jadad 1997 [29]; Ioannidis 2009 [57]; Kramer 2009 [61]; Ryan 2009 [25]; Singh 2012 [76]; Thomson 2010 [26]

▪ Report RoB assessments of primary studies from the included SR(s), noting missing data and discrepancies (Hartling 2012 [53]; JBI 2014 [39]; Robinson 2015 [24, 69,70,71,72]

▪ Report RoB assessments from the highest quality SR (Jadad 1997 [29])

      2.1.2 Report RoB assessment of primary studies from the included SRs after performing quality checks to verify that the assessment method has been applied appropriately and consistently across a sample of primary studies

Becker 2008 [4]; Hartling 2014 [55]; Ioannidis 2009 [57]; Jadad 1997 [29]; Kramer 2009 [61]; Moja 2012 [63]; Robinson 2015 [24, 69,70,71,72]; Thomson 2010 [26]

▪ Randomly sample a number of included RCTs, retrieve data from the original trial reports, and independently check 10% of RCT data from the included MAs to verify assessments were done without error and consistently

▪ Repeat RoB assessments on a sample of SRs to verify and check for consistency (Robinson 2015 [24, 69,70,71,72])

      2.1.3 (Re)-assess RoB of some or all primary studies^a

CMIMG 2012 [47]; Cooper 2012 [6]; Hartling 2012 [53]; Jadad 1997 [29]; Moja 2012 [63]; Thomson 2010 [26]

▪ When two different tools are used, then assess the primary studies using one tool

▪ When two different tools are used (e.g. Cochrane RoB tool [67] and Jadad tool [29]; then re-assess RoB by standardising the assessments based on the Cochrane RoB domains, and match data from assessments from other tools to these domains)

      2.1.4 Don’t report or assess RoB of primary studies

Inferred

      2.2.1 Report assessment for individual items or domains (with or without rationale for judgements)^a

JBI 2014 [39, 59]; Pieper 2014c [66]; Robinson 2015 [24, 69,70,71,72]; Ryan 2009 [25]; Silva 2014 [75]

      2.2.2 Summarise assessments across items or domains by using a scoring system^§§

JBI 2014 [39, 59]; Pieper 2014c [66]; Robinson 2015 [24, 69,70,71,72]; Ryan 2009 [25]; Silva 2014 [75]

▪ Sum items, assigning equal or unequal weight to each (JBI 2014 [39, 59])

▪ Calculate the mean score across items (JBI 2014 [39, 59])

      2.2.3 Summarise assessments across items or domains, then use cut-off scores or thresholds to describe RoB (e.g. low, moderate and high quality)^§§

JBI 2014 [39, 59]; Robinson 2015 [24, 69,70,71,72]; Ryan 2009 [25]; Silva 2014 [75]; Singh 2012 [76]

      2.3.1 Display assessments in table(s) (e.g. overall rating in summary of findings table, and another table with RoB items for each primary study)^a

Aromataris 2015 [39, 59]; Becker 2008 [4]; Chen 2014 [46]; Hartling 2012 [53]; Smith 2011 [77]; JBI 2014 [39, 59]

      2.3.2 Display assessments graphically^a

Crick 2015 [48]

▪ Cochrane RoB graph depicting authors’ judgments about each domain presented as percentages across all included SRs [67]

▪ Harvest plot, which depicts results according to study size and quality ([48])

      2.3.3 Report assessments in text^a

Aromataris 2015 [39, 59]; Chen 2014 [46]; Hartling 2012 [53]; Li 2012 [62]; Smith 2011 [77]

      3.1.1 Independent assessment by 2 or more authors

Baker 2014 [43]; Becker 2008 [4]; Cooper 2012 [6]; JBI 2014 [39]; Li 2012 [62]; Ryan 2009 [25]

      3.1.2 One author assessment

Inferred

      3.1.3 One assessment, 2nd confirmed

Cooper 2012 [6]

      3.1.4 One assessment, 2nd confirms if uncertainty

Cooper 2012 [6]

  3.2 Determine if authors (co-)authored one or several of the SRs included in the overview, and if yes, plan safeguards to avoid bias in RoB assessment

Büchter 2011 [45, 65]

▪ Assessment of RoB of included SRs done by overview authors who were not authors of the SRs