Skip to main content

Table 2 Characteristics of stage I studies and the extent to which each described (two ticks) or mentioned (one tick) methods pertaining to the latter steps in conducting an overview

From: Toward a comprehensive evidence map of overview of systematic review methods: paper 2—risk of bias assessment; synthesis, presentation and summary of the findings; and assessment of the certainty of the evidence

Citation Type of study Summary description of the article Latter steps in the conduct of an overview
Assessment of RoB in SRs and primary studies Synthesis, presentation and summary of the findings Assessment of the certainty of the evidence arising from the overview
Baker [43]
The benefits and challenges of conducting an overview of systematic reviews in public health: a focus on physical activity.
Article describing methods for overviews • Describes the usefulness of overviews for decision-makers and summarises some procedural steps to be undertaken
• Provides a case study of an overview on public health interventions for increasing physical activity
 
Becker [4]
Overviews of reviews.
Guidance for undertaking overviews • Early guidance providing the structure and procedural steps for the conduct of an overview
• Details the RoB/quality assessment of SRs, and describes how to present findings through tables and figures
Bolland [5]
A case study of discordant overlapping meta-analyses: vitamin D supplements and fracture.
Article describing methods for overviews • Describes criteria for explaining differences in SR/MAs addressing a similar question with discordant conclusions
• Builds on the guide to interpret discordant SRs proposed by Jadad 1997 [29]
• Suggests reporting items when an overview contains SR/MAs addressing a similar question
Brunton [44]
Putting the issues on the table: summarising outcomes from reviews of reviews to inform health policy.
Study examining methods used in a cohort of overviews • Presents a tabular method of vote counting (positive effect/negative effect/no change) for each reported outcome    
Büchter [45, 65]
How do authors of Cochrane Overviews deal with conflicts of interest relating to their own systematic reviews?
Study examining methods used in a cohort of overviews • Describes potential conflicts arising from dual authorship (where an overview author includes one or more SRs they authored) and suggests ‘safeguards’ to protect against potential bias
• Describes reporting items in relation to dual authorship
  
Caird [1]
Mediating policy-relevant evidence at speed: are systematic reviews of systematic reviews a useful approach?
Article describing methods for overviews • Describes the methodological challenges in the production of overviews aimed at translating the knowledge to policy makers
• Describes the trade-offs between producing a rapid overview and its comprehensiveness and reliability
Chen [46]
Scientific hypotheses can be tested by comparing the effects of one treatment over many diseases in a systematic review.
Study examining methods used in a cohort of overviews • Identifies possible aims of an overview as being to detect unintended effects, improve the precision of effect estimates or explore heterogeneity of effect across disease groups
• Describes the value and pitfalls of synthesis of MAs using three case studies
 
CMIMG [47]
Review type and methodological considerations.
Guidance for undertaking overviews • Builds on the Cochrane guidance for overviews by Becker 2008 [4]
• Describes the factors in the decision to conduct an overview vs. an SR
• Describes how to handle potential overlap in primary studies across SRs and RoB/quality of SRs
 
Cooper [6]
The overview of reviews: unique challenges and opportunities when research syntheses are the principal elements of new integrative scholarship.
Article describing methods for overviews • Describes steps in the conduct of an overview and methods to address challenges (e.g. dealing with overlap in primary studies across SRs)
• Describes methods for second order meta-analysis
Crick [48]
An evaluation of harvest plots to display results of meta-analyses in overviews of reviews: a cross-sectional study
Article describing methods for overviews • Examines the feasibility of using harvest plots as compared to summary tables to depict results of MAs in overviews  
Flodgren [49]
Challenges facing reviewers preparing overviews of reviews.a
Article describing methods for overviews • Mentions the issue of missing or inadequately reported data
• Mentions the challenges in summarising and evaluating large amounts of heterogeneous data across SRs
 
Foisy [50]
Mixing with the ‘unclean’: Including non-Cochrane reviews alongside Cochrane reviews in overviews of reviews.a
Article describing methods for overviews • Mentions some challenges inherent in defining AMSTAR scoring as inclusion criteria, and inclusion of non-Cochrane reviews alongside Cochrane reviews
• Mentions inclusion criteria to minimise overlap in primary studies across SRs
  
Foisy [51]
Grading the quality of evidence in existing systematic reviews: challenges and considerations.
Study examining methods used in a cohort of overviews • Mentions problems overview authors may encounter when applying GRADE to included SRs without going back to original data, and potential solutions   
Foisy [52]
Challenges and considerations involved in using AMSTAR in overviews of reviews.
Article describing methods for overviews • Describes differences in AMSTAR scores across Cochrane and non-Cochrane SRs, challenges in using AMSTAR to assess RoB of included SRs, and using the AMSTAR score as inclusion criterion   
Hartling [53]
A descriptive analysis of overviews of reviews published between 2000 and 2011.
Study examining methods used in a cohort of overviews • Describes methodological standards for SRs (PRISMA, MECIR) and their applicability to overviews
• Describes methods related to RoB of primary studies extracted from the SRs, RoB of the SRs, quantitative analysis and certainty of the evidence
Hartling [54]
Generating empirical evidence to support methods for overviews of reviews.a
Study examining methods used in a cohort of overviews • Identifies methodological issues when conducting overviews
• Mentions RoB of SRs, limitations of existing tools and challenges in assessing the certainty of the evidence
 
Hartling [55]
Systematic reviews, overviews of reviews and comparative effectiveness reviews: a discussion of approaches to knowledge synthesis.
Article describing methods for overviews • Briefly defines overviews, mentions the purposes in conducting an overview, and discusses some methodological challenges    
Hemming [56]
Pooling systematic reviews of systematic reviews: a Bayesian panoramic meta-analysis.
Article describing methods for overviews • Proposes a Bayesian method for meta-analysis (hierarchical meta-analysis) which uses uninformative priors as a means of pooling effect estimates in an overview    
Ioannidis [57]
Integration of evidence from multiple meta-analyses: a primer on umbrella reviews, treatment networks and multiple treatments meta-analyses.
Article describing methods for overviews • Defines ‘umbrella reviews’ as a pre-step to network meta-analysis
• Describes challenges of overviews
• Describes some reporting items for overviews
   
Jadad [29]
A guide to interpreting discordant systematic reviews.
Article describing methods for overviews • Seminal paper summarising the potential sources of discordance in results in a cohort of MAs, and types of discordance
• Presents an algorithm for interpreting discordant SR results
James [58]
Informing the methods for public health overview reviews: a descriptive analysis of Cochrane and non-Cochrane public health overviews a
Study examining methods used in a cohort of overviews • Briefly describes several steps in the conduct of overviews
• Compares Cochrane and non-Cochrane reviews in terms of tool used for assessing methodological quality, and certainty of the evidence
 
Joanna Briggs Institute (JBI) [39, 59]
Methodology for JBI umbrella reviews.
Guidance for undertaking overviews • Provides guidance as to what methods should be used at which step in the conduct of an overview
• Provides stylistic conventions for overviews to meet publication and reporting criteria for the JBI Database of Systematic Reviews and Implementation Reports
Kovacs [60]
Overviews should meet the methodological standards of systematic reviews.
Commentary that discusses methods for overviews • Mentions methodological challenges of overviews  
Kramer [61]
Preparing an overview of reviews: lessons learned. a
Article describing methods for overviews • Mentions the challenges encountered when the authors conducted three overviews including dealing with heterogeneity  
Li [62]
Quality and transparency of overviews of systematic reviews.
Article describing methods for overviews • Presents a pilot reporting/quality checklist
• Evaluates a cohort of overviews using the pilot tool, with the mean number of items but no details of the items
  
Moja [63]
Multiple systematic reviews: methods for assessing discordances of results.
Article describing methods for overviews • Describes methods to assess discordant finding among MAs based on the Jadad 1997 tool [29]  
O’Mara [64]
Guidelines for conducting and reporting reviews of reviews: dealing with topic relevances and double-counting.a
Article describing methods for overviews • Presents a ‘utility’ rating based on the SRs PICO compared to the overviews’ PICO question
• Mentions graphical and tabular approaches to establish the extent of overlap in primary studies across SRs
   
Pieper [3, 65]
Overviews of reviews often have limited rigor: a systematic review.
Study examining methods used in a cohort of overviews • Describes the methods used in overviews
• Recommends using validated search filters for retrieval of SRs
• Discusses whether to update the overview by including primary studies published after the most recent SR
Pieper [66]
Methodological approaches in conducting overviews: current state in HTA agencies.
Article describing methods for overviews • Describes the methods recommended in 8 HTA guideline documents related to overviews
• Compares the Cochrane Handbook guidance [67] to guidance produced by HTA agencies
Pieper [68]
Up-to-dateness of reviews is often neglected in overviews: a systematic review.
Study examining methods used in a cohort of overviews • Describes the process of searching for additional primary studies in an overview
• Presents decision rules for when to search for additional primary studies
• Describes sequential searching versus parallel searching for retrieving SRs and primary studies in an overview
 
Pollock [31]
An algorithm was developed to assign GRADE levels of evidence to comparisons within systematic reviews.
Article describing methods for overviews • Adapts GRADE guidance to assess the certainty of the evidence for a specific overview
• Presents an algorithm to assess certainty of the evidence
 
Robinson [24, 69,70,71,72]
Integrating bodies of evidence: existing systematic reviews and primary studies.
Article describing methods for overviews • Describes the steps and methods to undertake a complex review that includes multiple SRs, which is similar to methods used in overviews
• Describes methods for assessing risk of bias of primary studies in SRs, and methods to assess the certainty of the evidence
Ryan [25]
Building blocks for meta-synthesis: data integration tables for summarising, mapping, and synthesising evidence on interventions for communicating with health consumers.
Article describing methods for overviews • Presents tabular methods to deal with the preparation of overview evidence
• Discusses the data extraction process and organisation of data
• Presents a table of taxonomy of outcomes from the included SRs and a data extraction table based on this taxonomy
Salanti [73]
Evolution of Cochrane intervention reviews and overviews of reviews to better accommodate comparisons among multiple interventions.
Guidance for undertaking overviews • Defines overviews as integrating or synthesising (rather than summarising) evidence from SRs
• Describes methods for synthesising multiple-intervention SRs
   
Schmidt [74]
Methods for second order meta-analysis and illustrative applications.
Article describing methods for overviews • Describes statistical methods for second-order meta-analysis
• Provides examples of modelling between-MA variation
   
Silva [75]
Overview of systematic reviews—a new type of study. Part II
Study examining methods used in a cohort of overviews • Examines a cohort of Cochrane reviews for methods used
• Documented the sources and types of search strategies conducted
 
Singh [76]
Development of the Metareview Assessment of Reporting Quality (MARQ) Checklist.
Article describing methods for overviews • Presents a pilot reporting/quality checklist
• Evaluates four case studies using the pilot tool, with the mean number of items but no details of the items
 
Smith [77]
Methodology in conducting a systematic review of systematic reviews of healthcare interventions.
Article describing methods for overviews • Describes some steps and challenges in undertaking an overview, namely eligibility criteria, search methods, and RoB/quality assessment
• Presents tabular methods for presentation of results in an overview
Tang [78]
A statistical method for synthesizing meta-analyses.
Article describing methods for overviews • Describes statistical methods for second-order MA with examples    
Thomson [26]
The evolution of a new publication type: steps and challenges of producing overviews of reviews.
Article describing methods for overviews • Describes some steps in undertaking an overview and some of the challenges in conducting an overview
• Discusses that gaps or lack of currency in included evidence will weaken the overview findings
Thomson [79]
Overview of reviews in child health: evidence synthesis and the knowledge base for a specific population.
Study examining methods used in a cohort of overviews • Describes the process of including trials in overviews
• Discusses the challenge of overview PICO differing from the PICO of the included SRs
• Provides potential solutions as to what to do when mixed populations are reported in SRs and how to extract subgroup data
   
Wagner [80]
Assessing a systematic review of systematic reviews: developing a criteria.
Article describing methods for overviews • Presents a quality assessment tool for appraisal of overviews  
  1. AHRQ’s EPC Agency for Healthcare Research and Quality’s Evidence-based Practice Center; AMSTAR A Measurement Tool to Assess Systematic Reviews; CMIMG Comparing Multiple Interventions Methods Group; GRADE Grading of Recommendations Assessment, Development, and Evaluation; HTA health technology assessment; JBI Joanna Briggs Institute; MA meta-analysis; MECIR Methodological Expectations of Cochrane Intervention Reviews; PICO Population, Intervention, Comparison, Outcome; PRISMA Preferred Reporting Items for Systematic Reviews and Meta-Analyses; RoB risk of bias; R-AMSTAR revised AMSTAR; SR systematic review
  2. aIndicates a poster presentation
  3. Indicates a study describing one or more methods
  4. Indicates a study mentioning one or more methods