What guidance is available for researchers conducting overviews of reviews of healthcare interventions? A scoping review and qualitative metasummary
Systematic Reviews volume 5, Article number: 190 (2016)
Overviews of reviews (overviews) compile data from multiple systematic reviews to provide a single synthesis of relevant evidence for decision-making. Despite their increasing popularity, there is limited methodological guidance available for researchers wishing to conduct overviews. The objective of this scoping review is to identify and collate all published and unpublished documents containing guidance for conducting overviews examining the efficacy, effectiveness, and/or safety of healthcare interventions. Our aims were to provide a map of existing guidance documents; identify similarities, differences, and gaps in the guidance contained within these documents; and identify common challenges involved in conducting overviews.
We conducted an iterative and extensive search to ensure breadth and comprehensiveness of coverage. The search involved reference tracking, database and web searches (MEDLINE, EMBASE, DARE, Scopus, Cochrane Methods Studies Database, Google Scholar), handsearching of websites and conference proceedings, and contacting overview producers. Relevant guidance statements and challenges encountered were extracted, edited, grouped, abstracted, and presented using a qualitative metasummary approach.
We identified 52 guidance documents produced by 19 research groups. Relatively consistent guidance was available for the first stages of the overview process (deciding when and why to conduct an overview, specifying the scope, and searching for and including systematic reviews). In contrast, there was limited or conflicting guidance for the latter stages of the overview process (quality assessment of systematic reviews and their primary studies, collecting and analyzing data, and assessing quality of evidence), and many of the challenges identified were also related to these stages. An additional, overarching challenge identified was that overviews are limited by the methods, reporting, and coverage of their included systematic reviews.
This compilation of methodological guidance for conducting overviews of healthcare interventions will facilitate the production of future overviews and can help authors address key challenges they are likely to encounter. The results of this project have been used to identify areas where future methodological research is required to generate empirical evidence for overview methods. Additionally, these results have been used to update the chapter on overviews in the next edition of the Cochrane Handbook for Systematic Reviews of Interventions.
Systematic reviews (SRs) combine the results of multiple similar primary studies to answer a specific clinical question . With the exponential increase in the number of published SRs , overviews of reviews (overviews) have emerged as a logical solution to help manage this information overload. The purpose of overviews is to integrate information from multiple related SRs to provide a comprehensive synthesis of all SR evidence related to a specific clinical question . They are designed to be accessible, user-friendly documents that are typically broader in scope than any individual SR. Overviews are often conducted to address questions related to the efficacy, effectiveness, and/or safety of healthcare interventions—for example, examining multiple interventions for the prevention or treatment of a specific health condition . Table 1 describes the key characteristics of overviews.
Given their objective to synthesize extensive data in a user-friendly format, overviews have been gaining momentum as a valuable knowledge synthesis product to facilitate the uptake and application of knowledge by decision-makers. Thus, the number of published overviews has been steadily increasing in recent years [4–6]. This increase is at least partially due to the pioneering efforts of The Cochrane Collaboration, an international organization widely recognized as producing high-quality SRs of health evidence . In 2004, the Comparing Multiple Interventions Methods Group (originally called the Umbrella Reviews Methods Group) was established to develop general guidance for conducting overviews . This preliminary guidance was first published as a chapter in the Cochrane Handbook for Systematic Reviews of Interventions (Cochrane Handbook) in 2008 , and the first overview was published in the Cochrane Database of Systematic Reviews (CDSR) in 2009 . Today, Cochrane authors can publish overviews in the CDSR with a label that allows readers to distinguish them from standard SRs. Other research groups and organizations around the world have also adopted this research design as a valuable knowledge synthesis product [10, 11].
Overview methods evolved from SR methods for which there are well-established standards of conduct to ensure rigor, validity, and reliability of results . Overviews therefore aim to use explicit, reproducible, and systematic methods to search for, identify, and extract outcome data from SRs. However, since the unit of searching, inclusion, and data extraction is the SR, overview authors often encounter unique methodological challenges for which there are no obvious solutions or clear guidance. As a result, current practice when conducting overviews is driven largely by personal experience and trial and error, and published overviews show considerable variation in their methods and reporting [4–6]. In recent years, a number of overview authors have recognized the methodological challenges inherent in conducting overviews and expressed a need for comprehensive, up-to-date guidance for overviews [4, 5, 13].
The purpose of this scoping review was to identify and summarize all documents containing methodological guidance for conducting overviews examining the efficacy, effectiveness, and/or safety of healthcare interventions. The aims were as follows: (1) locate, access, compile, and create a map of documents that provide explicit methodological guidance for conducting overviews; (2) identify and describe areas where guidance for conducting overviews is clear and consistent, as well as areas where guidance is conflicting or missing; and (3) document common challenges involved in conducting overviews and determine whether existing guidance can help researchers overcome these challenges. We then used the results of this scoping review to update the chapter on overview methods appearing in the Cochrane Handbook.
To be included in the scoping review, documents had to meet one of two criteria: (1) provide explicit guidance for conducting overviews of healthcare interventions, defined as any guidance related to either the context or the process of conducting an overview or (2) describe an author team’s experience conducting one or more overviews of healthcare interventions. When selecting documents for inclusion, we used the definition of overviews provided in Table 1. We included guidance that applied to overviews examining the efficacy, effectiveness, and/or safety of healthcare interventions and excluded guidance that applied to other types of overviews (e.g., diagnostic test accuracy, prognostic, and qualitative overviews). We included documents produced in any language, year, or format.
Search methods for identification of documents
Our scoping review aimed to identify and include a wide range of document types, including unpublished documents such as internal documents, training manuals, and conference proceedings. We therefore conducted an iterative and extensive search to ensure breadth and comprehensiveness of coverage [14–17]. The search was conducted between January and March 2014 and involved reference tracking, database and web searches, handsearching of websites and conference proceedings, and contacting producers of overviews.
Our iterative reference tracking (“snowballing”) search [16, 17] was conducted by a research librarian (RF). The reference tracking search used a total of 30 target articles about overviews that were identified by the study authors prior to the start of the search and as the search progressed. For each target article, we searched for “citing” references using Google Scholar, “cited” references using Scopus and reference lists, and “similar articles” using PubMed. Database and web searches were conducted to supplement and enhance our reference tracking search (RF). We first updated the database searches reported in Hartling et al. . This involved searching MEDLINE (via Ovid), EMBASE (via Ovid), DARE (via Cochrane Library), and Scopus, for articles published between January 2010 and December 2013. We then augmented this search with two additional databases (MEDLINE via Web of Science, and Cochrane Methods Studies Database via Ovid) and one additional web search engine (Google Scholar). Relevant articles identified by the database and web searches were fed back into the reference tracking search and used as target articles to help locate additional relevant articles.
A number of additional sources were searched in an attempt to locate all unpublished and internal guidance documents (MP, RF). We handsearched the websites of 26 organizations that we knew had published at least one overview, and the conference proceedings (2000–2013) of three conferences: the International Cochrane Colloquium, Health Technology Assessment International, and the Canadian Agency for Drugs and Technologies in Health Symposium. Additionally, we contacted overview producers to ask if they had followed any specific guidance when conducting their overview(s): this involved contacting 20 Managing Editors of Cochrane Review Groups and Fields who oversaw the preparation of a combined 64 overviews published in the CDSR and Evidence-Based Child Health: A Cochrane Review Journal, and 110 authors who published a combined 148 overviews in journals other than the CDSR (lists of overviews obtained from  and ). We had satisfactory response rates (57% for authors of conference proceedings, 96% for Managing Editors of Cochrane Review Groups and Fields, and 55% for overview authors).
We updated select components of the search in November 2015. To ensure we continued to capture relevant documents published after our search dates, we used article alerts from MEDLINE (via Web of Science) and Google Scholar to monitor new articles between January 2014 and November 2015. Additionally, we searched conference proceedings for 2014 and 2015 and contacted an additional five Managing Editors of Cochrane Review Groups who oversaw the preparation of five overviews published in the CDSR in 2014 and 2015.
Finally, we handsearched the reference lists of the 52 guidance documents included in this scoping review. Due to the variability in terminology used to refer to overviews , we searched for and included terms such as “overview,” “overview of reviews,” “overview of systematic reviews,” “umbrella review,” “systematic review of systematic reviews,” and “metareview”. See Additional file 1 for complete search strategies.
Selection of documents
All titles and abstracts were independently screened by one reviewer (MP) and one research assistant. We kept those documents that were not overviews but that met the broad definition of “being about overviews” or “discussing some aspect of overviews.” We then retrieved the full text of all potentially relevant titles and abstracts. Full-text articles were assessed for inclusion by two independent reviewers (MP, LH) using the previously described eligibility criteria, with discrepancies resolved through discussion.
Data extraction and analysis
Relevant text contained within each included document was extracted and analyzed using a qualitative metasummary approach, which is an iterative, quantitatively oriented method of data analysis that involves aggregating textual data to identify and expose patterns of findings across groups of related documents [18, 19]. This involved extracting, editing, grouping, abstracting, and presenting findings (this work was completed using Microsoft Word and Excel). All data collection and analysis was conducted by one reviewer (MP) and checked for accuracy by a second reviewer (LH), with disagreements resolved through discussion. The qualitative metasummary process is described below.
First, we clearly specified the text eligible and not eligible for extraction using the criteria presented in Additional file 2 . For documents that provided explicit guidance for conducting overviews of healthcare interventions, we extracted text that provided guidance on how to conduct any part of an overview and text that described challenges involved when conducting overviews. For documents that described an author team’s experience conducting one or more overviews of healthcare interventions, we extracted only text that described challenges author teams encountered. We then separated guidance statements and challenges from all other text in the documents, and edited the guidance and challenges to ensure that they were presented in a way that was accessible to readers while preserving their underlying content and meaning . Guidance statements and challenges were then separated from each other and grouped, abstracted, and presented in parallel.
For both guidance statements and challenges, we used a two-stage approach to group similar findings together. First, we grouped all documents produced by the same research group to avoid giving extra weight to statements that were included in multiple documents produced by the same research group . Within each of these groupings, we further edited the findings to eliminate redundancies and duplicate text while leaving the meaning of the statements unchanged. Second, we grouped statements across research groups by stage of the overview process to ensure that all statements related to the same stage of the overview process appeared in the same place . The stages of the overview process included the following: deciding when and why to conduct an overview, specifying the scope, searching for and including SRs, quality assessment of SRs and their primary studies, collecting and analyzing data, grading quality of evidence, and drawing conclusions. These stages were identified iteratively: they were selected in advance using the stages presented in the Cochrane Handbook  and modified as needed to accommodate the specific guidance and challenges identified.
We then abstracted findings to summarize the content of each group of topically related guidance statements and challenges [18, 19]. For each stage of the overview process, we reworked our lists of guidance statements and challenges until we developed a new list of abstracted statements that captured the overall meaning of the original statements. This was done by eliminating redundancies, refining statements to ensure they were inclusive of the ideas presented by each research group, preserving ambiguity and contradictions, and ensuring clarity and accessibility.
Lastly, we provided a narrative summary of the abstracted guidance statements followed by a narrative summary of the abstracted challenges. For guidance statements only, we also calculated frequency and intensity effect sizes. These were used to extract more meaning from the narrative summaries by numerically describing the magnitude of the abstracted findings [18, 19]. Frequency effect sizes were calculated by dividing the number of research groups contributing guidance on a topic area by the total number of research groups. Intensity effect sizes were calculated by dividing the number of topic areas addressed by each research group by the total number of topic areas.
Results of the search
The literature search retrieved 2418 unique references. One hundred seventy-six references were identified as potentially eligible, and the full-text articles were assessed for inclusion. Of these, 124 documents were excluded (list available upon request). Fifty-two documents produced by 19 research groups were included; these documents are listed in Additional file 3 and are labeled “A1,” “A2,”…, “A52” in the text below. Figure 1 contains a flow diagram of documents through the review process. As anticipated, published articles that could be located through database searching comprised a minority (29%) of included documents; the majority (71%) were unpublished documents identified through other searching methods.
Summary of included guidance documents
Table 2 summarizes the characteristics of the included guidance documents and presents abbreviations for research groups that will be used throughout the remainder of the results.
Of the 52 included documents, 41 provided explicit methodological guidance for conducting overviews of healthcare interventions (Additional file 3, references A1–A41). These documents were produced between 2007 and 2015 by 12 research groups (range, 1–18 documents per group). The three most common types of documents were oral presentations (37%), journal articles (24%), and internal documents (22%). Four research groups (CHF, CMIMG, CPHG, EPOC) contributing 28 documents (68%) had primary affiliations associated with The Cochrane Collaboration. Eleven documents described an author team’s experience conducting one or more overviews of healthcare interventions (Additional file 3, references A42–A52). These documents (5 posters, 4 journal articles, 2 oral presentations) were produced between 2004 and 2015 by nine research groups (range, 1–2 documents per group). Six research groups (CCRG, CHF, CM, CS, DDC, EPOC) contributing 8 documents (73%) had primary affiliations associated with The Cochrane Collaboration. We first summarize the existing guidance for conducting overviews, with frequency and intensity effect sizes. We then summarize the challenges identified.
Guidance for conducting overviews
The guidance contained within the 41 documents that provided explicit methodological guidance fell into two broad categories: guidance related to the context for conducting overviews and guidance related to the process of conducting overviews. These categories could be further subdivided into 15 topic areas. The existing methods guidance for each topic area is summarized below; italicized terms are defined in Table 3.
Guidance related to the context for conducting overviews
Choosing between conducting an overview and a SR
Two groups provided guidance on this topic (CMIMG, EPPI). The CMIMG stated that authors should conduct an overview only when they intend to search for and extract data from SRs as opposed to primary studies. Authors should conduct a SR when they intend to search for or extract data from primary studies, conduct a network meta-analysis, or rank order interventions (A16). When choosing between both study designs, the scope of the research question should be taken into account (A12). See reference A23 for additional guidance on choosing between both study designs (CMIMG). EPPI-Centre stated that authors may consider conducting an overview when a broad research question co-occurs with a short time frame and limited resources (A30).
What types of questions about healthcare interventions can be answered using the overview format?
One group (CMIMG) provided guidance on this topic, though six additional groups (CHF: A2; DukeU: A29; JBI: A34; NOKC: A36; TCD: A37; UBirm: A38) referenced this guidance in their own documents. The CMIMG stated that overviews can summarize evidence from multiple SRs about “different interventions for the same condition; the same intervention for different conditions; the same intervention for the same condition where different outcomes are addressed in different SRs; or adverse effects of interventions” (A16).
Questions to consider before deciding to conduct an overview
Six groups (CHF, CMIMG, EPPI, JBI, TCD, WJNR) stated that the overview format must be suitable for the proposed research question. Questions to consider include the following: is the topic clinically relevant (CHF: A8); is the field too new or changing too rapidly to preclude the utility of an overview (EPPI: A30); are there enough relevant SRs on major interventions and/or disorders of interest (e.g., are SRs up-to-date and clinically and methodologically homogeneous) (CHF: A8; CMIMG: A16; WJNR: A39); have important organizational factors been considered (e.g., author team, time frame, and funding) (CHF: A8; CMIMG: A14; JBI: A34; TCD: A37); and does it make methodological sense to include all SRs in the same overview (e.g., has the transitivity assumption been met) (CHF: A3; CMIMG: A24)? The CHF states that proper planning is important and that authors should “beware of the common misperception that overviews are easy and straightforward” (A3).
Author team composition and roles
Four groups (CHF, CMIMG, JBI, WJNR) stated that a complete multidisciplinary author team is needed that ideally includes a project coordinator (CHF: A4), clinician or content expert (CHF: A9), researcher with methodological expertise (CHF: A9; CMIMG: A20; JBI: A34; WJNR: A39), statistician (as needed) (CHF: A9; CMIMG: A20), and information specialist (as needed) (CHF: A9). Additional members may also be required, and roles should correspond to each member’s area of expertise (CHF: A9). See reference A9 for additional detail on team member roles (CHF).
Target audience of the overview
Eight groups (CHF, CMIMG, DukeU, EPOC, EPPI, TCD, WHU, WJNR) stated that the target audience for overviews is healthcare decision-makers including clinicians (CHF: A10; CMIMG: A16; EPOC: A27; TCD: A37; WHU: A41; WJNR: A39), researchers (DukeU: A29; EPOC: A27; WJNR: A39), informed patients/consumers (CHF: A10; CMIMG: A16; WHU: A41), and policy-makers/commissioning agents (CHF: A10; CMIMG: A16; EPOC: A27; EPPI: A30; WHU: A41; WJNR: A39).
Guidance related to the process for conducting overviews
Specifying the scope
Ten groups provided guidance on this topic (CHF, CMIMG, CPHG, DukeU, EPOC, EPPI, JBI, NOKC, TCD, WJNR). They stated that authors should clearly specify and describe the clinical characteristics (e.g., populations, interventions, comparators, and outcomes) and study design information (e.g., SRs) of interest for the overview (CHF: A8; CMIMG: A16; CPHG: A28; EPOC: A27; JBI: A34; NOKC: A36; TCD: A37; WJNR: A39). Reference A9 contains additional detail about specifying outcomes of interest (CHF). Additionally, authors may wish to restrict their scope based on clinical or methodological characteristics (CHF: A6; CMIMG: A16; DukeU: A29; EPOC: A27; EPPI: A30; JBI: A33; NOKC: A36; TCD: A37).
Searching for SRs
Eleven groups provided guidance on this topic (CHF, CMIMG, CPHG, DukeU, EPOC, EPPI, JBI, NOKC, TCD, WHU, WJNR). They stated that authors should search the CDSR to locate Cochrane SRs (CHF: A8; CMIMG: A16; EPPI: A30; JBI: A34; TCD: A37). To locate non-Cochrane SRs, authors should search additional databases (e.g., MEDLINE, EMBASE) and SR registries (e.g., Epistemonikos) (CHF: A8; CMIMG: A26; CPHG: A28; DukeU: A29; EPOC: A27; EPPI: A30; JBI: A34; TCD: A37; WJNR: A39) and contact experts or conduct handsearching of sources relevant to the topic (JBI: A34; TCD: A37). Overview authors may choose to use SR-specific search terms and/or validated SR search filters (CHF: A8; DukeU: A29; EPOC: A27; JBI: A34; TCD: A37). They may also restrict their search by date, language, and/or publication status, if appropriate (CPHG: A28; DukeU: A29; EPOC: A27; JBI: A34; TCD: A37). Conflicting guidance was provided regarding whether or not overview authors should search for and include primary studies that are not contained within any included SR (CHF: A2; CMIMG: A16; CPHG: A28; DukeU: A29; EPPI: A30; NOKC: A36; WHU: A41). Different ways of searching for primary studies were described, for example, see reference A41 (WHU).
Selecting SRs for inclusion
Six groups (CHF, CMIMG, DukeU, EPOC, NOKC, TCD) indicated that authors should select SRs for inclusion using pre-defined inclusion criteria. The scopes of the SRs and overview may sometimes differ (DukeU: A29; NOKC: A36); in these cases, authors must assess the primary studies contained within each SR for inclusion, and they should only include the subset of primary studies that meet the overview’s inclusion criteria (CHF: A8; CMIMG: A16). Two groups (EPOC: A27; TCD: A37) recommended that documents be assessed for inclusion by two independent reviewers with consensus.
Should an overview include non-Cochrane SRs?
Nine groups provided guidance on this topic (CHF, CMIMG, DukeU, EPOC, EPPI, JBI, NOKC, TCD, WHU). Two groups affiliated with The Cochrane Collaboration (CHF: A8; CMIMG: A16) stated that authors of Cochrane overviews should include only Cochrane SRs, if possible, but they also stated that including non-Cochrane SRs has both advantages (e.g., greater topic coverage) and disadvantages (e.g., increases complexity of the overview). The groups provided conflicting guidance regarding whether or not overview authors should use SR quality as an inclusion criterion for non-Cochrane SRs (and if so, what procedure to follow and which tool to use) (CHF: A8; DukeU: A29; EPOC: A27; EPPI: A30; JBI: A34; NOKC: A36; TCD: A37; WHU: A40). There was uncertainty and conflicting guidance on the methods that should be used to manage overlapping SRs in overviews (e.g., should authors include only one SR per topic area, or should they include all relevant SRs regardless of overlap?) (CHF: A8; CMIMG: A26; DukeU: A29; EPPI: A30; TCD: A37; WHU: A41). See reference A40 (WHU) for ways to assess and report overlap in overviews, and references A8 (CHF) and A29 (DukeU) for ways to potentially manage overlap in overviews.
Assessing quality of included SRs
All 12 groups stated that quality assessment of SRs is important and should be done. Conflicting guidance was provided regarding the tool that should be used, though A MeaSurement Tool to Assess systematic Reviews (AMSTAR)  was mentioned most often, by seven research groups (CHF: A8; CMIMG: A16; DukeU: A29; EPOC: A27; JBI: A33; TCD: A37; WJNR: A39). Six groups recommended dual independent quality assessments with consensus (CMIMG: A16; DukeU: A29; EPOC: A27; JBI: A34; NOKC: A36; TCD: A37). No other guidance was provided describing the specific methods that should be used to assess SR quality (e.g., whether and how to modify the quality assessment tool for use in overviews).
Collecting and presenting data on descriptive characteristics of included SRs (and their primary studies)
Six groups (CHF, CMIMG, EPOC, JBI, TCD, WJNR) provided guidance on this topic. They stated that authors should extract information about the objectives, inclusion criteria, and methods of each included SR (CHF: A8; CMIMG: A16; EPOC: A27; JBI: A33; TCD: A37; WJNR: A39). Authors should also extract information about the primary studies included in each SR (CHF: A8; EPOC: A27; JBI: A33; TCD: A37; WJNR: A39).
Collecting and presenting data on quality of primary studies contained within included SRs
Seven groups provided conflicting guidance regarding how overview authors should collect and present data on primary study quality; methods proposed included extracting and reporting the quality assessments conducted within each SR or referring back to each primary study to conduct quality assessments (CHF: A8; CMIMG: A16; DukeU: A29; EPPI: A30; JBI: A34; NOKC: A36; WJNR: A39). Four groups explicitly recommended the former method over the latter, if possible (CHF: A8; CMIMG: A16; JBI: A34; NOKC: A36). No guidance was provided regarding the logistical concerns likely to be encountered (e.g., use of different quality assessment tools in different SRs).
Collecting, analyzing, and presenting outcome data
Seven groups provided guidance on this topic and described quantitative and narrative methods of presenting data (CHF, CMIMG, DukeU, EPOC, EPPI, JBI, UBirm). Three groups (CHF: A8; DukeU: A29; UBirm: A38) stated that outcome data could be extracted from SRs and analyzed or presented in a different way than the analyses contained within the SRs (e.g., using meta-analysis or other complex methods). Two groups (CMIMG: A16; JBI: A33) stated that outcome data could simply be presented in the overview as they were presented in SRs. Two groups (EPOC: A27; EPPI: A30) acknowledged both approaches without recommending one over the other. Research groups advised that the most appropriate method of data analysis may depend upon the overview’s research question and the amount of clinical, methodological, and/or statistical heterogeneity in the SRs (CHF: A9; CMIMG: A12; EPPI: A30). Three groups recommended dual independent data extraction with consensus (EPOC: A27; JBI: A33; UBirm: A38). Research groups provided limited guidance regarding the logistical concerns likely to be encountered when analyzing outcome data. For example, there is uncertainty regarding how to analyze data from overlapping SRs (though at a minimum, authors should acknowledge the overlap and potential for bias) (CHF: A9; CMIMG: A26; EPPI: A30; JBI: A33; also see WHU: A40).
Assessing quality of evidence of outcome data
Six groups stated that it is important to assess the quality of evidence, for example, using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) tool (CHF: A8; CMIMG: A16; EPPI: A30; JBI: A34; NOKC: A36; TCD: A37) . However, only two groups provided guidance regarding how to assess quality of outcome data in overviews; they stated that authors could either extract quality assessments from included SRs or conduct quality assessments themselves at the overview level (CHF: A8; CMIMG: A16). CMIMG recommended that two reviewers independently assess quality of evidence with a process for consensus (A16). No other guidance was provided regarding the logistical concerns likely to be encountered when conducting quality assessments (e.g., what if not all SRs assessed quality of evidence?).
Interpreting outcome data and drawing conclusions
Three groups (CMIMG, EPPI, WHU) provided guidance on this topic. They stated that authors must ensure that the conclusions they make are warranted based on the quality of the primary studies and SRs and the methods used to analyze data (CMIMG: A16; EPPI: A30). Authors should avoid making informal indirect comparisons across different interventions because the transitivity assumption will likely be problematic (CMIMG: A24). Authors should also state whether more research is likely to change the results of the overview (based on quality of evidence, if assessed) (WHU: A41).
Frequency and intensity effect sizes
The research groups that contributed the most guidance to this scoping review, as measured using intensity effect sizes (Table 4), were as follows: CMIMG (15/15 topics), CHF (13/15 topics), and EPPI and JBI (11/15 topics each). The topic areas that the most research groups discussed, as measured using frequency effect sizes, were as follows: “assessing quality of SRs” (12/12 groups), “searching for SRs” (11/12 groups), and “specifying the scope” (10/12 groups). Topics that the least number of research groups discussed were as follows: “choosing between conducting an overview and a SR” (2/12 groups), “interpreting outcome data and drawing conclusions” (3/12 groups), and “author team composition and roles” (4/12 groups each).
Challenges identified when conducting overviews
All 19 research groups contributing explicit guidance and/or author experiences identified at least one challenge involved when conducting overviews of healthcare interventions. These challenges are summarized in Table 5. Nine research groups also described limitations inherent to the overview format itself (CHF, CMIMG, CPHG, DukeU, EPOC, EPPI, JBI, WHU, WJNR). Specifically, they stated that overviews can be complex and resource intensive (CHF: A43; EPOC: A46; EPPI: A30; WHU: A40); susceptible to bias (CMIMG: A17; CPHG: A28; DukeU: A29; EPPI: A30; JBI: A34; WHU: A40); and dependent on (and limited by) the scope, inclusion criteria, methods, reporting, and coverage of their included SRs (CHF: A9; CPHG: A28; DukeU: A29; EPOC: A45; EPPI: A30; JBI: A34; WHU: A41; WJNR: A39). Few of the challenges identified when conducting overviews were adequately addressed by the methodological guidance previously summarized.
This scoping review found relatively consistent and comprehensive guidance for the first stages of the overview process, from choosing to conduct an overview through to selecting SRs for inclusion. Guidance for the latter stages was often conflicting and/or missing, and a number of outstanding challenges were identified. These latter stages included the following: deciding whether to include SRs published outside of the CDSR, assessing quality of SRs and their primary studies, collecting and analyzing data, and assessing quality of evidence of outcome data.
The shift from consistent to conflicting and/or missing guidance that occurs after the inclusion stage may be due to several factors. First, this is the point at which guidance for overviews takes on an additional level of complexity. Within an overview, there are two levels for assessing and reporting SR/study characteristics, quality/risk of bias, and outcome data (i.e., for both the SRs and their included primary studies). Existing methodological guidance does not yet adequately address how these stages of the overview process should occur relative to these two levels of information. Second, SRs are syntheses of pre-existing data, and we found that overviews are limited by the methods and reporting of their included SRs. Data may be missing, inadequately reported, or reported differently across included SRs, and it is currently unclear whether overview authors should rely solely on the SRs as they were conducted and presented, or whether and to what extent authors should refer back to the primary studies for additional information. Lastly, including SRs published outside of the CDSR can increase the complexity of the latter stages of the overview process due to greater variation in the methods and reporting of non-Cochrane SRs [22, 23] and the potential for topic overlap across multiple similar SRs . Limited guidance was available regarding the specific methods authors can use to address and manage these issues in overviews.
To circumvent some of the challenges authors are likely to encounter during the latter stages of the overview process, authors should first ensure that the overview format is appropriate for their question of interest. The CMIMG in particular provided comprehensive guidance regarding the context for conducting an overview (i.e., when and why to conduct an overview); however, much of this guidance is currently in the form of internal documents and conference proceedings that may be difficult for authors to access. Authors should also prepare a detailed protocol for their overview. Often overview authors describe their scope and inclusion criteria but provide less detail about methods to be used for quality assessments and data extraction and analysis. As well as reducing bias and promoting rigor and transparency of methods [11, 25], a protocol would allow overview authors to become familiar with the challenges they are likely to encounter and develop a priori decision rules to appropriately address those challenges. The guidance and challenges described in this paper will be useful for authors to consider when developing their protocols.
As is common when using a qualitative metasummary approach , an important insight emerged when we analyzed our data across topic areas, namely, that overviews are often conducted for one of two main purposes. The first purpose is to present and describe the complete body of SR evidence on a clearly defined topic [26, 27]. The second purpose is to address a question that differs from the question(s) in the underlying SRs and that often relates to a subset of the questions in the SRs (e.g., subpopulations, or subsets of interventions or outcomes) [28, 29]. Distinguishing between these two purposes, and recognizing that different methods may be used for each, can help resolve some discrepancies and challenges likely to be encountered during the latter stages of the overview process. For example, if the purpose is to answer a different question from those posed in the SRs, authors may wish to re-extract and re-analyze outcome data (e.g., using meta-analysis) from a set of non-overlapping SRs. However, if the purpose is to describe the complete body of SR evidence on a topic, authors may find it more appropriate to include all relevant SRs regardless of topic overlap and then present these results as they appeared in the SRs. Empirical evidence will be needed to determine whether these approaches affect the results or introduce bias at the overview level.
Ultimately, methodological guidance is required for those stages of the overview process where guidance is conflicting and/or missing and where outstanding challenges remain. This future guidance should be based on empirical evidence generated from well-conducted studies that evaluate methods for conducting overviews. While outside of the scope of the present paper, we identified several relevant methods studies (recently published, and in progress) when conducting this scoping review. These methods studies examined the following: implications of including multiple SRs published on the same topic area ; issues related to quality assessment of SRs [30–32]; different methods for presenting outcome data ; methods for assessing quality of evidence using GRADE [34, 35]; and reporting conflicts of interest in overviews . One additional study (in progress) was identified that will aim to summarize all empirical studies . Developing future methodological guidance for overviews based on the results of empirical studies will help ensure that guidance is based on sound evidence as opposed to personal experience or trial and error.
The current scoping review aimed to identify and collate all documents containing methodological guidance for conducting overviews of healthcare interventions. Due to the variety of publication formats for overview methods guidance and the difficulty in locating and accessing these documents, it is possible we may have missed relevant guidance documents. To maximize retrieval, our search utilized multiple complimentary methods in addition to database searching. We had satisfactory response rates (ranging from 55 to 96%) when locating and obtaining the full text of unpublished documents, and we were able to translate and extract data from all relevant non-English documents identified. We then used a rigorous process for identifying, extracting, and analyzing guidance statements and challenges from these documents. Importantly, we were interested in method guidance and challenges for overviews examining the efficacy, effectiveness, and/or safety of healthcare interventions and have excluded guidance and challenges specific to overviews that may address broader or different clinical questions. Guidance for conducting these other types of overviews is also needed but is outside the scope of the current project. It is important to note that the guidance and challenges summarized here was written by research groups with different organizational processes that likely produce overviews with differing purposes, scopes, target audiences, and/or resource requirements. Researchers should identify the purpose, scope, target audience, and resource requirements of their overview at the outset and determine how well the guidance and challenges presented here apply to their specific situation. Lastly, the guidance included in this scoping review came from documents that explicitly intended to provide methods guidance to readers: the methods presented here do not come from the actual methods used in published overviews. However, discussions with overview authors  and critical appraisal of published overviews [4, 5, 38, 39] indicates that the guidance and challenges in this scoping review are congruent with overview authors’ experiences.
This scoping review provides a systematic summary of existing methodological guidance for conducting overviews examining the efficacy, effectiveness, and/or safety of healthcare interventions. It highlights the stages of the overview process where guidance is consistent, conflicting, or missing, and it also provides a summary of the challenges involved in conducting overviews. This scoping review will serve as a useful resource for authors wishing to conduct overviews, as well as researchers wishing to conduct empirical research on overview methods. It is also a necessary first step to developing a cohesive methods guidance document that addresses relevant issues and areas of uncertainty when conducting overviews of healthcare interventions. Accordingly, the results of this scoping review were used to update the chapter on overview methods in the Cochrane Handbook. There has been a dramatic rise in the production of SRs and overviews in recent years. These syntheses are an important vehicle to increase the uptake and application of knowledge by clinical and policy decision-makers, and they can help address crucial health issues and ultimately improve health outcomes in diverse populations. Investing in strengthening the methods guidance for conducting overviews can help ensure a rigorous and valid evidence base for knowledge translation and dissemination.
A MeaSurement Tool to Assess systematic Reviews
Cochrane Consumers and Communication Review Group
Cochrane Database of Systematic Reviews
Cochrane Child Health Field
Cochrane Musculoskeletal Review Group
Cochrane Comparing Multiple Interventions Methods Group
Cochrane Public Health Group
Cochrane Stroke Group
Database of Abstracts of Reviews of Effects
Dutch Cochrane Centre
Excerpta Medica dataBASE
Cochrane Effective Practice and Organization of Care Review Group
Evidence for Policy and Practice Information and Co-ordinating Centre
Grading of Recommendations Assessment, Development and Evaluation
Joanna Briggs Institute Umbrella Reviews Methodology Group
Ludwig Boltzmann Institute for Health Technology Assessment
Norwegian Knowledge Centre for the Health Services
Overview of reviews
Pontifical Xavierian University
Trinity College Dublin
University of Birmingham
University of Dundee
Western Journal of Nursing Research
Green S, Higgins JPT, Alderson P, Clarke M, Mulrow CD, Oxman AD. Chapter 1: introduction. In: Higgins JPT, Green S, editors. Cochrane handbook for systematic reviews of interventions (version 5.1.0). The Cochrane Collaboration; 2011. www.handbook.cochrane.org.
Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 2010;7(9):e1000326.
Becker LA, Oxman AD. Chapter 22: overviews of reviews. In: Higgins JPT, Green S, editors. Cochrane handbook for systematic reviews of interventions (version 5.1.0). The Cochrane Collaboration; 2011. www.handbook.cochrane.org.
Hartling L, Chisholm A, Thomson D, Dryden DM. A descriptive analysis of overviews of reviews published between 2000 and 2011. PLoS One. 2012;7(11):e49667.
Pieper D, Buechter R, Jerinic P, Eikermann M. Overviews of reviews often have limited rigor: a systematic review. J Clin Epidemiol. 2012;65(12):1267–73.
Li L, Tian J, Tian H, Sun R, Liu Y, Yang K. Quality and transparency of overviews of systematic reviews. J Evid Based Med. 2012;5(3):166–73.
Cochrane: About us. http://www.cochrane.org/about-us (n.d.). Accessed 30 Nov 2015.
Cochrane Comparing Multiple Interventions Methods Group: About us. http://www.cmim.cochrane.org/ (2015). Accessed 30 Nov 2015.
Singh JA, Christensen R, Wells GA, Suarez-Almazor ME, Buchbinder R, Lopez-Olivo MA, et al. Biologics for rheumatoid arthritis: an overview of Cochrane reviews. Cochrane Database Syst Rev. 2009;4:CD007848.
Aromataris E, Fernandez R, Godfrey C, Holly C, Khalil H, Tungpunkom P, editors. The Joanna Briggs Institute reviewers’ manual 2014: methodology for JBI umbrella reviews. University of Adelaide: Joanna Briggs Institute; 2014.
Grimshaw J. A knowledge synthesis chapter. In: A guide to knowledge synthesis. Canadian Institutes of Health Research; 2010. http://www.cihr-irsc.gc.ca/e/41382.html. Accessed 30 Nov 2015
Higgins JPT, Green S, editors. Cochrane handbook for systematic reviews of interventions (version 5.1.0). The Cochrane Collaboration; 2011. www.handbook.cochrane.org.
Foisy M, Dryden DM, Fernandes RM, Hartling L, Thomson D. Advancing methods for overviews of reviews: a discussion of challenges and potential solutions. Abstracts of the 22nd Cochrane Colloquium, 21-26 Sept 2014. Hyderabad: Wiley; 2014.
Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8(1):19–32.
Levac D, Colquhoun H, O’Brien KK. Scoping studies: advancing the methodology. Implement Sci. 2010;5:69.
Horsley T, Dingwall O, Sampson M. Checking reference lists to find additional studies for systematic reviews. Cochrane Database Syst Rev. 2011;8:MR000026.
Greenhalgh T, Peacock R. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. BMJ. 2005;331(7524):1064–5.
Sandelowski M, Barroso J. Chapter 6: synthesizing qualitative research findings: qualitative metasummary. In: Sandelowski M, Barroso J, editors. Handbook for synthesizing qualitative research. New York: Springer Publishing Company; 2006. p. 151–97.
Sandelowski M, Barroso J. Creating metasummaries of qualitative findings. Nurs Res. 2003;52(4):226–33.
Shea BJ, Hamel C, Wells GA, Bouter LM, Kristjansson E, Grimshaw J, et al. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol. 2009;62(10):1013–20.
Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336(7650):924–6.
Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Med. 2007;4(3):e78.
Moseley AM, Elkins MR, Herbert RD, Maher CG, Sherrington C. Cochrane reviews used more rigorous methods than non-Cochrane reviews: survey of systematic reviews in physiotherapy. J Clin Epidemiol. 2009;62(10):1021–30.
Foisy M, Dryden DM, Fernandes RM, Hartling L. Multiple systematic reviews published on the same topic area: an analysis of systematic reviews that overlap in content. Abstracts of the 22nd Cochrane Colloquium, 21-26 Sep 2014. Hyderabad: Wiley; 2014.
Green S, Higgins JPT. Chapter 2: preparing a Cochrane review. In: Higgins JPT, Green S, editors. Cochrane handbook for systematic reviews of interventions (version 5.1.0). The Cochrane Collaboration; 2011. www.handbook.cochrane.org.
Worswick J, Wayne SC, Bennett R, Fiander M, Mayhew A, Weir MC, et al. Improving quality of care for persons with diabetes: an overview of systematic reviews—what does the evidence tell us? Syst Rev. 2013;2:26.
Farquhar C, Rishworth JR, Brown J, Nelen WL, Marjoribanks J. Assisted reproductive technology: an overview of Cochrane reviews. Cochrane Database Syst Rev. 2015;7:CD010537.
Foisy M, Boyle RJ, Chalmers JR, Simpson EL, Williams HC. The prevention of eczema in infants and children: an overview of Cochrane and non-Cochrane reviews. Evid Based Child Health. 2011;6(5):1322–39.
Cates CJ, Wieland LS, Oleszczuk M, Kew KM. Safety of regular formoterol or salmeterol in adults with asthma: an overview of Cochrane reviews. Cochrane Database Syst Rev. 2014;2:CD010314.
Pieper D, Mathes T, Eikermann M. Impact of choice of quality appraisal tool for systematic reviews in overviews. J Evid Based Med. 2014;7(2):72–8.
Foisy M, Hartling L. Challenges and considerations involved in using AMSTAR in overviews of reviews. Abstracts of the 22nd Cochrane Colloquium, 21-26 Sep 2014. Hyderabad: Wiley; 2014.
Jia P, Chen J, Zhang L, Zhao P, Zhang M. Challenges and considerations in assessing the reporting quality of systematic reviews in overviews using PRISMA. Abstracts of the 23rd Cochrane Colloquium, 03-07 Oct 2015. Vienna: Wiley; 2015.
Crick K, Wingert A, Williams K, Fernandes RM, Thomson D, Hartling L. An evaluation of harvest plots to display results of meta-analyses in overviews of reviews: a cross-sectional study. BMC Med Res Methodol. 2015;15:91.
Pollock A, Farmer SE, Brady MC, Langhorne P, Mead GE, Mehrholz J, et al. An algorithm was developed to assign GRADE levels of evidence to comparisons within systematic reviews. J Clin Epidemiol. 2015;doi:10.1016/j.jclinepi.2015.08.013.
Foisy M, Fernandes RM, Dryden DM, Hartling L. Grading the quality of evidence in existing systematic reviews: challenges and considerations. Abstracts of the 22nd Cochrane Colloquium, 21-26 Sep 2014. Hyderabad: Wiley; 2014.
Buchter RB, Pieper D. Most overviews of Cochrane reviews neglected potential biases from dual authorship. J Clin Epidemiol. 2016;doi:10.1016/j.jclinepi.2016.04.008.
Lunny C, McKenzie J, McDonald S, Brennan S. An evidence map of studies evaluating methods for conducting, interpreting, and reporting overviews of systematic reviews of interventions: rationale and design. Syst Rev. 2016;doi:10.1186/s13643-015-0178-0.
Thomson D, Foisy M, Oleszczuk M, Wingert A, Chisholm A, Hartling L. Overview of reviews in child health: evidence synthesis and the knowledge base for a specific population. Evid Based Child Health. 2013;8(1):3–10.
Cochrane: Special collection: Cochrane overviews. http://www.cochranelibrary.com/app/content/special-collections/article/?doi=10.1002/(ISSN)14651858(CAT)na(VI)SC000036 (2014). Accessed 30 Nov 2015.
The authors wish to thank Megan Nuspl for her assistance with searching and screening, and Sanja Schreiber and the Norwegian Knowledge Centre for the Health Services for assistance with German and Norwegian translations. We also thank all overview authors and Managing Editors of Cochrane Review Groups and Fields who responded to our requests for guidance documents, and all authors of the guidance documents included in this review who kindly shared their unpublished documents and/or responded to our requests for additional information. This project was funded in part by a Canadian Institutes of Health Research Operating Grant (Grant Number 119373). MP is supported by an Alberta Innovates Health Solutions Graduate Studentship and a Knowledge Translation Canada Graduate Student Fellowship. LH is supported by a Canadian Institutes of Health Research New Investigator Salary Award.
MP conceived of, designed, coordinated, and conducted all aspects of the study. RMF and LAB participated in the interpretation of the results. RF participated in the reference tracking, database searching, and web searching. LH conceived of the study and participated in its design and conduct. MP drafted the manuscript. All authors provided feedback and approved the final manuscript.
MP (former surname: Foisy), RMF, LAB and LH are Directors, Co-Conveners, and/or Members of both Cochrane Child Health and the Cochrane Comparing Multiple Interventions Methods Group. Because they have worked extensively to help advance overview methods, they have authored or contributed to 25 guidance documents included in this scoping review.
About this article
Cite this article
Pollock, M., Fernandes, R.M., Becker, L.A. et al. What guidance is available for researchers conducting overviews of reviews of healthcare interventions? A scoping review and qualitative metasummary. Syst Rev 5, 190 (2016). https://doi.org/10.1186/s13643-016-0367-5
- Overview of reviews
- Umbrella review
- Evidence synthesis
- Knowledge synthesis
- Evidence-based medicine
- Evidence-based practice
- Review methods
- Systematic reviews
- Scoping review