Skip to main content

Table 4 Specification of eligibility criteria

From: Toward a comprehensive evidence map of overview of systematic review methods: paper 1—purpose, eligibility, search and data extraction

Step Sub-step Methods/approaches Sources
▪ Examples
1.0 Plan the eligibility criteria
  1.1 Determine PICO eligibility criteria for the overview (and setting and timing if applicable) Becker 2008 [1]; Cooper 2012 [32]; Ioannidis 2009 [38]; JBI 2015 [40, 41]; Li 2012 [44]; Thomson 2013 [59]
  1.2 Determine PICO eligibility criteria for SRs  
   1.2.1 Select only SRs that are similar (or narrower) in scope to the overview PICO elements (i.e. exclude SRs that include out-of-scope interventions/populations in addition to the intervention/population addressed by the overview) Becker 2008 [1]; Cooper 2012 [32]; Foisy 2011 [34]; JBI 2015 [40, 41]; Robinson 2016 [48,49,50,51,52]; Ryan 2009 [53, 54]; Thompson 2013 [59]
   1.2.2 Select all SRs that address the PICO elements, including those broader in scope than the overview (i.e. SRs that include the intervention/ population addressed by the overview, plus other out-of-scope interventions/ populations). This may involve selecting: (i) any SR, irrespective of whether separate data are available for the subgroup of interest or (ii) limiting to SRs that present separate data for the subgroup of interest Becker 2008 [1]; Cooper 2012 [32]; Kramer 2009 [43]; Ryan 2009 [53, 54]; Thompson 2013 [59]; Whitlock 2008 [48,49,50,51,52]
  1.3 Determine criteria (mechanisms) to select outcomes where there are multiple
   1.3.1 Include all outcomes reported in included SRs Becker 2008 [1]; Hartling 2012 [35]; Ryan 2009 [53, 54]; Thomson 2013 [59]
▪ Map the outcomes to a taxonomy (Ryan 2009 [53, 54])
   1.3.2 Select one or more outcomes using pre-specified criteria, for example: (i) outcomes judged important by subject specialists (e.g. consumers, policy makers), (ii) primary outcomes, and (iii) outcomes common to more than one SR Caird 2015 [31]; Hartling 2014 [37]; Ioannidis 2009 [38]; JBI 2015 [40, 41]; Smith 2011 [57]; Thomson 2013 [59]
▪ Report only those outcomes common to more than one SR (Caird 2015 [31]; Hartling 2014 [37])
   1.3.3 Select one or more outcomes using pre-specified decision rules (e.g. combine selection criteria in an algorithm) Inferred method
  1.4 Determine methodological eligibility criteria for SRs
   1.4.1 Include all SRs that meet the PICO criteria (i.e. no methodological criteria applied) Caird 2015 [31]
   1.4.2 Select SRs that meet minimum quality criteria or take a particular methodological approach.
Minimum criteria include:
(i) meets definition of an SR, (e.g. explicit search)
(ii) up-to-date
(iii) quality of the SR (e.g. based on selected criteria; cutoffs derived from AMSTAR score)
(iv) use of best practice methods (e.g. specific RoB tools; Cochrane or AHRQ’s EPC methods)
(v) free of conflicts of interest (e.g. no industry funding)
(vi) reports sufficient primary study characteristics to interpret results (e.g. PICO elements, RoB assessment)
Methodological approaches include:
(vii) type of included primary studies
(viii) type of data
(ix) type of synthesis (e.g. meta-analysis, narrative)
Becker 2008 [1]; Chen 2014 [2]; Cooper 2012 [32]; Foisy 2011 [34]; Hartling 2013 [36]; James 2014 [39]; JBI 2015 [40, 41]; Robinson 2016 [48,49,50,51,52]; Smith 2011 [57]; Thompson 2013 [59]
  1.5 Determine eligibility criteria to deal with SRs with overlap
   1.5.1 Include all SRs that meet the PICO, irrespective of overlap Cooper 2012 [32]; Whitlock 2008 [48,49,50,51,52]
   1.5.2 Select one SR from multiple addressing the same question using pre-specified methodological criteria as outlined in 1.4.2 Cooper 2012 [32]; Pieper 2014 [46]; Robinson 2015 [48,49,50,51,52]
▪ Select the highest quality SR (Cooper 2012 [32])
   1.5.3 Select one SR from multiple addressing the same question using pre-specified decision rules (e.g. combine one or more eligibility criteria in an algorithm) Cooper 2012 [32]
▪ Select the SR with the most complete information, and if these are equivalent, the M-A with the greatest number of primary studies (Cooper 2012 [32])
   1.5.4 Exclude SRs that do not contain any unique primary studies, when there are multiple SRs Pieper 2014 [46]
  1.6 Determine whether to consider additional primary studies for inclusion
   1.6.1 Do not include primary studies Becker 2008 [1]; Caird 2015 [31]; Thompson 2013 [59]; Whitlock 2008 [48,49,50,51,52]
   1.6.2 Include primary studies if pre-specified eligibility criteria are met, for example: (i) when a SR is not up-to-date, (ii) when a SR is inconclusive (i.e. new studies may overturn the findings of a SR), (iii) when the included SRs provide incomplete coverage of evidence in relation to the overview PICO (e.g. missing one or more interventions, population subgroup, study design), and (iv) when there are concerns about the methods SRs used to identify and select studies Baker 2014 [29]; Caird 2015 [31]; Cooper 2012 [32]; Pieper 2014 [46]; 2014 [47]; Thompson 2013 [59]; White 2009 [48,49,50,51,52]
▪ Include primary studies if the evidence in the SRs is inconclusive (e.g. when addition of a new primary study may overturn the findings) (Pieper 2014 [47])
▪ Include primary studies if the SRs are assessed as low quality (Pieper 2014 [47])
   1.6.3 Include primary studies using pre-specified decision rules to determine eligibility (e.g. combine one or more eligibility criteria in an algorithm for selection) Pieper 2014 [47]
2.0 Plan the study selection process
  2.1 Determine the number of overview authors required to select studiesa
   2.1.1 Independent screening all stages by 2 or more authors Becker 2008 [1]; Chen 2014 [2]; Hartling 2012 [35]; Li 2012 [44]; Pieper 2012 [6, 45]; 2014 [47]; Smith 2011 [57]
   2.1.2 One author screening at all stages Hartling 2012 [35]; Li 2012 [44]; Pieper 2014 [47]
   2.1.3 One author screening titles/abstracts, 2 or more screening full text Hartling 2012 [35]
   2.1.4 One screened at all stages, 2nd confirmed Hartling 2012 [35]
   2.1.5 One screened at all stages, 2nd confirms if uncertainty Hartling 2012 [35]
  1. AHRQ’s EPC Agency for Healthcare Research and Quality ‘s Evidence-based Practice Center, AMSTAR A MeaSurement Tool to Assess systematic Reviews, CMIMG Comparing Multiple Interventions Methods Group, JBI Joanna Briggs Institute, PICOs Population, Intervention, Comparison, Outcome, and Study design, RCT randomised controlled trial, SRs systematic reviews
  2. aAdaption of the step from SRs to overviews. No methods evaluation required, but special consideration needs to be given to unique issues that arise in conducting overviews