Table 4 Synthesis, presentation and summary of the findings

      1.1.1 Include all SR results/MAs

Caird 2015 [1]; Cooper 2012 [6]

      1.1.2 Use decision rules or tools (e.g. Jadad tool [29]) to select results from a subset of SR/MAs

Caird 2015 [1]; Cooper 2012 [6]

▪ Select one SR result/MA from overlapping SR/MAs based on (a) the MA with the most complete information, and if that was equivalent, (b) the MA with the largest number of primary studies (Cooper 2012 [6])

      1.2.1 Describe and/or tabulate the characteristics of the included SRs in terms of PICO elements

Becker 2008 [4]; Cooper 2012 [6]; JBI 2014 [39, 59]; Pieper 2014c [66]; Robinson 2015 [24, 69,70,71,72]; Ryan 2009 [25]; Smith 2011 [77]; Thomson 2010 [26]

▪ Matrix of studies by PICO elements to allow comparison and assess important sources of heterogeneity across the SRs (Caird 2015 [1]; Kramer 2009 [61]; Smith 2011 [77]; Thomson 2010 [26])

      1.2.2 Describe and/or tabulate the results of the included SRs

Becker 2008 [4]; Caird 2015 [1]; Chen 2014 [46]; Cooper 2012 [6]; Hartling 2012 [53]; JBI 2014 [39, 59]; Pieper 2014c [66]; Robinson 2015 [24, 69,70,71,72]; Ryan 2009 [25]; Salanti 2012 [73]; Silva 2014 [75]; Singh 2012 [76]; Smith 2011 [77]; Thomson 2010 [26]

▪ Present pooled effect estimates and their confidence intervals (and associated statistics such as estimates of heterogeneity, I²), number and types of studies, number of participants, meta-analysis model and estimation method, authors conclusions

▪ Present the forest plots from the included SRs (Chen 2014 [46]; Pieper 2014c [66])

      1.2.3 Describe and/or tabulate the results of the included primary studies, including new or additional primary studies ^a

Caird 2015 [1]; Cooper 2012 [6]; O’Mara 2011 [64]; Robinson 2015 [24, 69,70,71,72]

▪ For example, summary data, effect estimates and their confidence intervals, study design, number of study participants (O’Mara 2011 [64])

      1.2.4 Summarise and/or tabulate RoB assessments of SRs and primary studies

Becker 2008 [4]; Caird 2015 [1]; Chen 2014 [46]; Hartling 2012 [53]; JBI 2014 [39, 59]; Li 2012 [62]; Ryan 2009 [25]; Robinson 2015 [24, 69,70,71,72]; Smith 2011 [77]

▪ For example, summarise the RoB/quality assessment methods used across the SRs

      1.2.5 Summarise and/or tabulate results from any investigations of statistical heterogeneity (e.g. results from subgroup analyses / meta-regression) within the included SRs

Cooper 2012 [6]; JBI 2014 [39, 59]; Smith 2011 [77]

      1.2.6 Summarise and/or tabulate results from any investigations of reporting biases (e.g. results from statistical tests for funnel plot asymmetry) within the included SRs

Singh 2012 [76]; Smith 2011 [77]

▪ Tabulate statistical tests of publication bias from the included MAs (Smith 2011 [77])

      1.2.7 Determine the order of reporting the results in text and tables (e.g. by outcome domain, by effectiveness of interventions)^a

Becker 2008 [4]; Bolland 2014 [5]; Salanti 2011 [73]; Smith 2011 [77]

      1.2.8 Determine methods for converting or standardising effect metrics (either from primary studies or meta-analyses) to the same scale (e.g. odds ratios to risk ratios)^a

Becker 2008 [4]; Cooper 2012 [6]; Thomson 2010 [26]

▪ Where a variety of summary statistics, such as odds ratios and risk ratios, are reported across SR/MAs, convert the results into one summary statistic to facilitate interpretation and comparability among results (Thomson 2010 [26])

      1.2.9 Determine methods to group results of specific outcomes (from either primary studies or MAs) into broader outcome domains^a

Ryan 2009 [25]; Thomson 2010 [26]

▪ Use an existing outcome taxonomy (e.g. Cochrane Consumers and Communication Review Group’s taxonomy). For example, results of an intervention on specific outcomes knowledge, accuracy, and risk of perception all map to the outcome domain consumer knowledge and understanding (Ryan 2009 [25])

  1.3 Determine graphical approaches to present the results^a

Becker 2008 [4]; Chen 2014 [46]; Crick 2015 [48]; Hartling 2014 [55]; JBI 2014 [39, 59]; Pieper 2014c [66]; Pieper 2014a [17]

▪ Use a forest plot to present MA effects (95% CI) from each SR sometimes referred to as ‘forest top plot’ (Becker 2008 [4]; Pieper 2014a [17])

▪ Use a harvest plot to present the direction of effect for trials or MAs or both, also depicting study size and quality (Crick 2015 [48])

▪ Use a bubble plot to display three dimensions of information, using colour to differentiate clinical indications: the x-axis (e.g. meta-analytic effect size), y-axis (e.g. SR quality), and the size of the bubble (e.g. number of included primary studies in a SR)

▪ Use a network plot to present the treatments that have been compared, with nodes representing treatments and links between nodes representing comparisons between treatments (Cooper 2012 [6])

  2.1 Do not conduct a new quantitative synthesis (e.g. because of lack of time or resources)

Salanti 2011 [73]

      2.2.1 Need to combine results from multiple MAs (with non-overlapping studies) for the same comparison and outcome

Robinson 2015 [24, 69,70,71,72]

      2.2.2 Need to incorporate additional primary studies; or, incorporate these studies under certain circumstances

Robinson 2015 [24, 69,70,71,72]; Pieper 2014a [17]

▪ When the identified SRs are out of date and more recent primary studies have been published (Robinson 2015 [24, 69,70,71,72])

▪ When inclusion of primary studies may change conclusions, strength of evidence judgements, or add new information (e.g. a trial undertaken in a population not currently included in the overview)

      2.2.3 Need to apply new meta-analysis methods, fitting a more appropriate meta-analysis method and model, or using a different effect metric

Robinson 2015 [24, 69,70,71,72]

▪ When a new meta-analysis method such as prediction intervals are required

▪ When a fixed effect model was fitted in a SR, but a random effects model was more appropriate

▪ When a risk ratio is used instead of an odds ratio

      2.2.4 Need to limit or expand the MAs into a new MA that meets the population, intervention and comparator elements of the overview

Thomson 2010 [26]; Whitlock 2008 [24, 69,70,71,72]

▪ Extracting the subset of trials that include only children and adolescents from a MA that includes trials with no restriction on age

      2.2.5 Need to undertake a new meta-analysis because of concerns regarding the trustworthiness of the SR/MA results

Robinson 2015 [24, 69,70,71,72]

▪ Concerns regarding data extraction errors

      2.2.6 Need to conduct a MA (if possible and makes sense to do so) because the SRs did not undertake MA

Inferred

      2.2.7 Need to conduct a MA to reconcile discordant findings of previous SRs

White 2009 [24, 69,70,71,72]

▪ If overview authors cannot determine reasons for the discordant findings among SRs, then they can regard this as an indication that they need to conduct a new MA (White 2009 [24, 69,70,71,72])

      2.3.1 Undertake a first-order meta-analysis of effect estimates (meta-analysis of the primary study effect estimates)^a

Becker 2008 [4]; Chen 2014 [46]; Cooper 2012 [6]; Pieper 2014a [17]; Robinson 2015 [24, 69,70,71,72]; Schmidt 2013 [74]; Tang 2013 [78]; Thomson 2010 [26]

▪ May re-extract data from the primary studies, or use the data reported in the reviews (see ‘Data extraction’ table in [10])

      2.3.2 Undertake a second-order meta-analysis of effect estimates (meta-analysis of meta-analyses) either ignoring the potential correlation across the meta-analysis estimates (arising from the same study included in more than one meta-analysis), or applying an adjustment to account for the potential correlation (e.g. inflating the variance of the meta-analysis)

Caird 2015 [1]; Chen 2014 [46]; Cooper 2012 [6]; Hemming 2012 [56]; Schmidt 2013 [74]; Tang 2013 [78]

▪ This issue of potential correlation (or non-independence) of the meta-analysis effect estimates may be more of a concern in overviews that seek to undertake a meta-analysis of the effects for the same intervention and same population, as compared with undertaking a meta-analysis of effects across populations (with the latter sometimes referred to as panoramic or multiple-indication reviews) (Chen 2014 [46]; Hemming 2012 [56])

▪ Refer to 5.1.4 for statistical approaches to dealing with overlap

      2.3.3 Undertake vote counting (e.g. based on direction of effect)^a

Becker 2008 [4]; Caird 2015 [1]; Flodgren 2011 [49]; Ryan 2009 [25]; Tang 2013 [78]; Thomson 2010 [26]

  2.4 Determine the method to convert effect metrics (either from primary studies or meta-analyses) to the same scale^a

Cooper 2012 [6]; Tang 2013 [78]; Thomson 2010 [26]

  2.5 Determine the meta-analysis model and estimation methods^a

Cooper 2012 [6]; Hemming 2012 [56]; Schmidt 2013 [74]

▪ For example, second order meta-analysis: fixed or random effects model to combine meta-analytic effects (Schmidt 2013 [74])

▪ For example, first-order meta-analysis across clinical conditions (multiple indication, panoramic review): three level hierarchical model, mixed effects model (Chen 2014 [46]; Hemming 2012 [56])

▪ For example, parametric or non-parametric methods (Cooper 2012 [6])

▪ For example, DerSimonian and Laird between-study variance estimator (Robinson 2015 [24, 69,70,71,72]; Tang 2013 [78])

  2.6 Determine graphical approaches^a

Becker 2008 [4]; Chen 2014 [46]; Crick 2015 [48]; Li 2012 [62]; Pieper 2014a [17]; Pieper 2014c [66]

▪ Use forest plots—either of meta-analysis results from each review, or results from individual studies (Becker 2008 [4]; Pieper 2014a [17]; Chen 2014 [46]; Pieper 2014c [66];

▪ Use a harvest plot, which depicts results according to study size and quality, noting the direction of effect (Crick 2015 [48])

      3.1.1 Tabulate results by modifying factors (e.g. study size, quality)^a

Caird 2015 [1]; Chen 2014 [46]; Hartling 2012 [53]; JBI 2014 [39, 59]; Singh 2012 [76]

▪ Graph or tabulate results of SRs by modifying factors (e.g. group by the type of included study design [SRs of RCTs, SRs of observational studies); group by methodological quality of the SRs, their completeness in evidence coverage, or how up-to-date they are) (Caird 2015 [1]; Chen 2014 [46]; Hartling 2012 [53]; JBI 2014 [39, 59])

      3.1.2 Graph results by modifying factors^a

(Caird 2015 [1]; Chen 2014 [46]; Hartling 2012 [53]; JBI 2014 [39, 59])

  3.2 Determine approach to identifying and quantifying heterogeneity^a

Cooper 2012 [6]

▪ Visual examination of overlap of confidence intervals in the forest plot, I² statistic, chi-squared test for heterogeneity

      3.3.1 Undertake a first-order subgroup analysis of primary study effect estimates^a

Becker 2008 [4]; Chen 2014 [46]; Cooper 2012 [6]; Singh 2012 [76]; Robinson 2015 [24, 69,70,71,72]; Thomson 2010 [26]

      3.3.2 Undertake a second-order subgroup analysis of meta-analysis effect estimates with moderators categorised at the level of the meta-analysis (e.g. SR quality). Issues of correlation across the meta-analysis estimates may occur (see 2.3.2)

Cooper 2012 [6]

  3.4 Determine the meta-analysis model and estimation methods^a

Refer to 2.5

▪ For example, random effects meta-regression

  4.1 Determine non-statistical approaches to assess missing SRs

Pieper 2014d [68]; Singh 2012 [76]

▪ Search SR registers (e.g. PROSPERO)

▪ Search for SR protocols

  4.2 Determine non-statistical approaches to assess missing primary studies

Bolland 2014 [5]

▪ Identify non-overlapping primary studies across SRs and examine reasons for non-overlap (e.g. different SR inclusion / exclusion criteria, different search dates, different databases) as a method for discovering potentially missing primary studies from SRs (Bolland 2014 [5])

▪ Conduct searches of trial registries to identify missing studies

  4.3 Determine statistical methods for detecting and examining potential reporting biases from missing primary studies or results within studies, or selectively reported results^a

Caird 2015 [1]; JBI 2014 [39, 59]; Singh 2012 [76]; Schmidt 2013 [74]; Smith 2011 [77]

▪ Visual assessment of funnel plot asymmetry of results from primary studies

▪ Statistical tests for funnel plot asymmetry using results from primary studies

  5.1 Determine methods for quantifying overlap

Cooper 2012 [6]; Pieper 2014b [35]

▪ Statistical measures to quantify the degree of overlap of primary studies across SRs (Pieper 2014b [35])

  5.2 Determine how to visually examine and present overlap of the primary studies across SRs

Caird 2015 [1]; Chen 2014 [46]; Cooper 2012 [6]; JBI 2014 [39, 59]; O’Mara 2011 [64]; Robinson 2015 [24, 69,70,71,72]; Thomson 2010 [26]

▪ Display a matrix comparing which primary studies were included in which SRs; or other visual approaches demonstrating overlap (e.g. Venn diagrams as referenced in Patnode [82])

      5.3.1 Use decision rules, or a tool, to select one (or a subset of) MAs with overlapping studies (see also 1.1.2 above)

Caird 2015 [1]; Chen 2014 [46]; Cooper 2012 [6]; O’Mara 2011 [64]; Pieper 2012 [3]; Robinson 2015 [24, 69,70,71,72]; Thomson 2010 [26]

▪ Choose the meta-analyses with the most complete information; methodologically rigorous; recentness of the meta-analysis; inclusion of certain study types (e.g. only randomised trials); publication status

▪ Exclude SRs that do not contain any unique primary studies, when there are multiple SRs (Pieper 2014a [17])

▪ Use a published algorithm or tool [Jadad 1997 [29]]

      5.3.2 Use statistical approaches to deal with overlap

Cooper 2012 [6]; Tang 2013 [78]

▪ Identify meta-analyses with 25% or more of their research in common and eliminate the one with the fewer studies in each comparison, except when multiple smaller meta-analyses (with little overlap) would include more studies if the largest meta-analysis was eliminated (Cooper 2012 [6])

▪ Sensitivity analyses (e.g. second-order MA including all MAs irrespective of overlap compared with second-order MA including only MAs where there is no overlap in primary studies) (Cooper 2012 [6])

▪ Inflate the variance of the meta-analysis estimate (Tang 2013 [78])

      5.3.3 Ignore overlap among primary studies in the included SRs

Cooper 2012 [6]; Caird 2015 [1]

      5.3.4 Acknowledge overlap as a limitation

Caird 2015 [1]

      6.1.1 Examine and record discordance among SRs addressing a similar question

Caird 2015 [1]; Chen 2014 [46]; Cooper 2012 [6]; Hartling 2012 [53]; JBI 2014 [39, 59]; Kramer 2009 [61]; Pieper 2014c [66]; Pieper 2012 [3]; Robinson 2015 [24, 69,70,71,72]; Smith 2011 [77]; Thomson 2010 [26]

▪ Discordance among SRs can arise from a lack of overlap in studies, or methodological differences

      6.1.2 Use decision rules or tools (e.g. Jadad 1997 [29]) to select one (or a subset of) SR/MAs

Bolland 2014 [5]; Caird 2015 [1]; Chen 2014 [46]; Cooper 2012 [6]; Hartling 2012 [53]; Jadad 1997 [29]; JBI 2014 [39, 59]; Kramer 2009 [61]; Moja 2012 [63]; Pieper 2012 [3]; Pieper 2014c [66]; Robinson 2015 [24, 69,70,71,72]; Smith 2011 [77]; Tang 2013 [78]; Thomson 2010 [26]

▪ Use a published algorithm based on whether the reviews address the same question, are of the same quality, have the same selection criteria (Jadad 1997 [29])

▪ Use an adapted algorithm (pre-existing algorithm adapted for the overview) (Bolland 2014 [5])

  6.2 Determine tabular or graphical approaches to present discordance

Inferred