Table 2 Characteristics of stage I studies and the extent to which each described (two ticks) or mentioned (one tick) methods pertaining to the latter steps in conducting an overview
Citation | Type of study | Summary description of the article | Latter steps in the conduct of an overview | ||
|---|---|---|---|---|---|
Assessment of RoB in SRs and primary studies | Synthesis, presentation and summary of the findings | Assessment of the certainty of the evidence arising from the overview | |||
Baker [43] The benefits and challenges of conducting an overview of systematic reviews in public health: a focus on physical activity. | Article describing methods for overviews | • Describes the usefulness of overviews for decision-makers and summarises some procedural steps to be undertaken • Provides a case study of an overview on public health interventions for increasing physical activity | ✓ | ✓ | |
Becker [4] Overviews of reviews. | Guidance for undertaking overviews | • Early guidance providing the structure and procedural steps for the conduct of an overview • Details the RoB/quality assessment of SRs, and describes how to present findings through tables and figures | ✓✓ | ✓ | ✓✓ |
Bolland [5] A case study of discordant overlapping meta-analyses: vitamin D supplements and fracture. | Article describing methods for overviews | • Describes criteria for explaining differences in SR/MAs addressing a similar question with discordant conclusions • Builds on the guide to interpret discordant SRs proposed by Jadad 1997 [29] • Suggests reporting items when an overview contains SR/MAs addressing a similar question | ✓ | ✓ | ✓✓ |
Brunton [44] Putting the issues on the table: summarising outcomes from reviews of reviews to inform health policy. | Study examining methods used in a cohort of overviews | • Presents a tabular method of vote counting (positive effect/negative effect/no change) for each reported outcome | ✓ | ||
How do authors of Cochrane Overviews deal with conflicts of interest relating to their own systematic reviews? | Study examining methods used in a cohort of overviews | • Describes potential conflicts arising from dual authorship (where an overview author includes one or more SRs they authored) and suggests ‘safeguards’ to protect against potential bias • Describes reporting items in relation to dual authorship | ✓ | ||
Caird [1] Mediating policy-relevant evidence at speed: are systematic reviews of systematic reviews a useful approach? | Article describing methods for overviews | • Describes the methodological challenges in the production of overviews aimed at translating the knowledge to policy makers • Describes the trade-offs between producing a rapid overview and its comprehensiveness and reliability | ✓✓ | ✓✓ | ✓✓ |
Chen [46] Scientific hypotheses can be tested by comparing the effects of one treatment over many diseases in a systematic review. | Study examining methods used in a cohort of overviews | • Identifies possible aims of an overview as being to detect unintended effects, improve the precision of effect estimates or explore heterogeneity of effect across disease groups • Describes the value and pitfalls of synthesis of MAs using three case studies | ✓✓ | ✓✓ | |
CMIMG [47] Review type and methodological considerations. | Guidance for undertaking overviews | • Builds on the Cochrane guidance for overviews by Becker 2008 [4] • Describes the factors in the decision to conduct an overview vs. an SR • Describes how to handle potential overlap in primary studies across SRs and RoB/quality of SRs | ✓ | ✓ | |
Cooper [6] The overview of reviews: unique challenges and opportunities when research syntheses are the principal elements of new integrative scholarship. | Article describing methods for overviews | • Describes steps in the conduct of an overview and methods to address challenges (e.g. dealing with overlap in primary studies across SRs) • Describes methods for second order meta-analysis | ✓✓ | ✓✓ | ✓✓ |
Crick [48] An evaluation of harvest plots to display results of meta-analyses in overviews of reviews: a cross-sectional study | Article describing methods for overviews | • Examines the feasibility of using harvest plots as compared to summary tables to depict results of MAs in overviews | ✓✓ | ✓ | |
Flodgren [49] Challenges facing reviewers preparing overviews of reviews.a | Article describing methods for overviews | • Mentions the issue of missing or inadequately reported data • Mentions the challenges in summarising and evaluating large amounts of heterogeneous data across SRs | ✓✓ | ✓ | |
Foisy [50] Mixing with the ‘unclean’: Including non-Cochrane reviews alongside Cochrane reviews in overviews of reviews.a | Article describing methods for overviews | • Mentions some challenges inherent in defining AMSTAR scoring as inclusion criteria, and inclusion of non-Cochrane reviews alongside Cochrane reviews • Mentions inclusion criteria to minimise overlap in primary studies across SRs | ✓ | ||
Foisy [51] Grading the quality of evidence in existing systematic reviews: challenges and considerations. | Study examining methods used in a cohort of overviews | • Mentions problems overview authors may encounter when applying GRADE to included SRs without going back to original data, and potential solutions | ✓ | ||
Foisy [52] Challenges and considerations involved in using AMSTAR in overviews of reviews. | Article describing methods for overviews | • Describes differences in AMSTAR scores across Cochrane and non-Cochrane SRs, challenges in using AMSTAR to assess RoB of included SRs, and using the AMSTAR score as inclusion criterion | ✓✓ | ||
Hartling [53] A descriptive analysis of overviews of reviews published between 2000 and 2011. | Study examining methods used in a cohort of overviews | • Describes methodological standards for SRs (PRISMA, MECIR) and their applicability to overviews • Describes methods related to RoB of primary studies extracted from the SRs, RoB of the SRs, quantitative analysis and certainty of the evidence | ✓✓ | ✓ | ✓✓ |
Hartling [54] Generating empirical evidence to support methods for overviews of reviews.a | Study examining methods used in a cohort of overviews | • Identifies methodological issues when conducting overviews • Mentions RoB of SRs, limitations of existing tools and challenges in assessing the certainty of the evidence | ✓ | ✓ | |
Hartling [55] Systematic reviews, overviews of reviews and comparative effectiveness reviews: a discussion of approaches to knowledge synthesis. | Article describing methods for overviews | • Briefly defines overviews, mentions the purposes in conducting an overview, and discusses some methodological challenges | ✓ | ||
Hemming [56] Pooling systematic reviews of systematic reviews: a Bayesian panoramic meta-analysis. | Article describing methods for overviews | • Proposes a Bayesian method for meta-analysis (hierarchical meta-analysis) which uses uninformative priors as a means of pooling effect estimates in an overview | ✓✓ | ||
Ioannidis [57] Integration of evidence from multiple meta-analyses: a primer on umbrella reviews, treatment networks and multiple treatments meta-analyses. | Article describing methods for overviews | • Defines ‘umbrella reviews’ as a pre-step to network meta-analysis • Describes challenges of overviews • Describes some reporting items for overviews | ✓ | ||
Jadad [29] A guide to interpreting discordant systematic reviews. | Article describing methods for overviews | • Seminal paper summarising the potential sources of discordance in results in a cohort of MAs, and types of discordance • Presents an algorithm for interpreting discordant SR results | ✓ | ✓✓ | ✓✓ |
James [58] Informing the methods for public health overview reviews: a descriptive analysis of Cochrane and non-Cochrane public health overviews a | Study examining methods used in a cohort of overviews | • Briefly describes several steps in the conduct of overviews • Compares Cochrane and non-Cochrane reviews in terms of tool used for assessing methodological quality, and certainty of the evidence | ✓ | ✓ | |
Joanna Briggs Institute (JBI) [39, 59] Methodology for JBI umbrella reviews. | Guidance for undertaking overviews | • Provides guidance as to what methods should be used at which step in the conduct of an overview • Provides stylistic conventions for overviews to meet publication and reporting criteria for the JBI Database of Systematic Reviews and Implementation Reports | ✓✓ | ✓✓ | ✓ |
Kovacs [60] Overviews should meet the methodological standards of systematic reviews. | Commentary that discusses methods for overviews | • Mentions methodological challenges of overviews | ✓ | ✓ | |
Kramer [61] Preparing an overview of reviews: lessons learned. a | Article describing methods for overviews | • Mentions the challenges encountered when the authors conducted three overviews including dealing with heterogeneity | ✓ | ✓ | |
Li [62] Quality and transparency of overviews of systematic reviews. | Article describing methods for overviews | • Presents a pilot reporting/quality checklist • Evaluates a cohort of overviews using the pilot tool, with the mean number of items but no details of the items | ✓ | ||
Moja [63] Multiple systematic reviews: methods for assessing discordances of results. | Article describing methods for overviews | • Describes methods to assess discordant finding among MAs based on the Jadad 1997 tool [29] | ✓ | ✓✓ | |
O’Mara [64] Guidelines for conducting and reporting reviews of reviews: dealing with topic relevances and double-counting.a | Article describing methods for overviews | • Presents a ‘utility’ rating based on the SRs PICO compared to the overviews’ PICO question • Mentions graphical and tabular approaches to establish the extent of overlap in primary studies across SRs | ✓ | ||
Overviews of reviews often have limited rigor: a systematic review. | Study examining methods used in a cohort of overviews | • Describes the methods used in overviews • Recommends using validated search filters for retrieval of SRs • Discusses whether to update the overview by including primary studies published after the most recent SR | ✓ | ✓ | ✓ |
Pieper [66] Methodological approaches in conducting overviews: current state in HTA agencies. | Article describing methods for overviews | • Describes the methods recommended in 8 HTA guideline documents related to overviews • Compares the Cochrane Handbook guidance [67] to guidance produced by HTA agencies | ✓✓ | ✓ | ✓ |
Pieper [68] Up-to-dateness of reviews is often neglected in overviews: a systematic review. | Study examining methods used in a cohort of overviews | • Describes the process of searching for additional primary studies in an overview • Presents decision rules for when to search for additional primary studies • Describes sequential searching versus parallel searching for retrieving SRs and primary studies in an overview | ✓ | ✓✓ | |
Pollock [31] An algorithm was developed to assign GRADE levels of evidence to comparisons within systematic reviews. | Article describing methods for overviews | • Adapts GRADE guidance to assess the certainty of the evidence for a specific overview • Presents an algorithm to assess certainty of the evidence | ✓ | ✓✓ | |
Integrating bodies of evidence: existing systematic reviews and primary studies. | Article describing methods for overviews | • Describes the steps and methods to undertake a complex review that includes multiple SRs, which is similar to methods used in overviews • Describes methods for assessing risk of bias of primary studies in SRs, and methods to assess the certainty of the evidence | ✓✓ | ✓✓ | ✓✓ |
Ryan [25] Building blocks for meta-synthesis: data integration tables for summarising, mapping, and synthesising evidence on interventions for communicating with health consumers. | Article describing methods for overviews | • Presents tabular methods to deal with the preparation of overview evidence • Discusses the data extraction process and organisation of data • Presents a table of taxonomy of outcomes from the included SRs and a data extraction table based on this taxonomy | ✓✓ | ✓✓ | ✓✓ |
Salanti [73] Evolution of Cochrane intervention reviews and overviews of reviews to better accommodate comparisons among multiple interventions. | Guidance for undertaking overviews | • Defines overviews as integrating or synthesising (rather than summarising) evidence from SRs • Describes methods for synthesising multiple-intervention SRs | ✓ | ||
Schmidt [74] Methods for second order meta-analysis and illustrative applications. | Article describing methods for overviews | • Describes statistical methods for second-order meta-analysis • Provides examples of modelling between-MA variation | ✓✓ | ||
Silva [75] Overview of systematic reviews—a new type of study. Part II | Study examining methods used in a cohort of overviews | • Examines a cohort of Cochrane reviews for methods used • Documented the sources and types of search strategies conducted | ✓ | ✓✓ | |
Singh [76] Development of the Metareview Assessment of Reporting Quality (MARQ) Checklist. | Article describing methods for overviews | • Presents a pilot reporting/quality checklist • Evaluates four case studies using the pilot tool, with the mean number of items but no details of the items | ✓✓ | ✓ | |
Smith [77] Methodology in conducting a systematic review of systematic reviews of healthcare interventions. | Article describing methods for overviews | • Describes some steps and challenges in undertaking an overview, namely eligibility criteria, search methods, and RoB/quality assessment • Presents tabular methods for presentation of results in an overview | ✓✓ | ✓✓ | ✓✓ |
Tang [78] A statistical method for synthesizing meta-analyses. | Article describing methods for overviews | • Describes statistical methods for second-order MA with examples | ✓✓ | ||
Thomson [26] The evolution of a new publication type: steps and challenges of producing overviews of reviews. | Article describing methods for overviews | • Describes some steps in undertaking an overview and some of the challenges in conducting an overview • Discusses that gaps or lack of currency in included evidence will weaken the overview findings | ✓✓ | ✓✓ | ✓✓ |
Thomson [79] Overview of reviews in child health: evidence synthesis and the knowledge base for a specific population. | Study examining methods used in a cohort of overviews | • Describes the process of including trials in overviews • Discusses the challenge of overview PICO differing from the PICO of the included SRs • Provides potential solutions as to what to do when mixed populations are reported in SRs and how to extract subgroup data | ✓ | ||
Wagner [80] Assessing a systematic review of systematic reviews: developing a criteria. | Article describing methods for overviews | • Presents a quality assessment tool for appraisal of overviews | ✓ | ✓ | |