Evidence map of studies evaluating methods for conducting, interpreting and reporting overviews of systematic reviews of interventions: rationale and design
Systematic Reviews volume 5, Article number: 4 (2016)
Overviews of systematic reviews attempt to systematically retrieve and summarise the results of multiple systematic reviews into a single document. Methods for conducting, interpreting and reporting overviews of reviews are in their infancy. To date, there has been no systematic review or evidence map examining the range of methods for overviews nor of the evidence for using these methods. The objectives of the study are to develop and populate a framework of methods that have or may be used in conducting, interpreting and reporting overviews of systematic reviews of interventions (stage I); create an evidence map of studies that have evaluated these methods (stage II); and identify and describe unique methodological challenges of overviews.
The research will be undertaken in two stages. For both stages, we plan to search methods collections (e.g. Cochrane Methodology Register, Meth4ReSyn library, AHRQ Effective Health Care Program) to identify eligible studies. These searches will be supplemented by searching reference lists and citation searching. Stage I: Methods used in overviews will be identified from articles describing methods for overviews, methods studies examining a cross section/cohort of overviews, guidance documents and commentaries. The identified methods will populate a framework of available methods for conducting an overview. Two reviewers will independently code included studies to develop the framework. Thematic analysis of the coded data will be used to categorise and describe methods. Stage II: Evaluations of the performance of methods will be identified from systematic reviews of methods studies and methods studies. Evaluations will be described and mapped to the framework of methods identified in stage I.
The results of this process will be useful for mapping of methods for overviews of systematic reviews, informing guidance and identifying and prioritising method research in this field.
Overviews of systematic reviews (or umbrella reviews) attempt to systematically retrieve and summarise the results of multiple systematic reviews into a single document . The number of published overviews of systematic reviews (henceforth termed overviews) has increased steadily in recent years, in part due to the proliferation of systematic reviews . Methods for conducting, interpreting and reporting overviews are in their infancy . To date, there has been no systematic review or evidence map examining the range of methods for overviews (particularly those which are unique to overviews) nor of the evidence for using these methods.
In general, the steps for undertaking an overview mirror those of a systematic review, with many of the methods used in systematic reviews being directly transferrable to overviews (e.g. independent study selection and data extraction) . However, there are unique features of overviews that require the use of different or additional methods, for example, methods for assessing the quality or the risk of bias in systematic reviews; dealing with the inclusion of the same trial in multiple systematic reviews; dealing with out-of-date systematic reviews; and dealing with discordant results across systematic reviews .
Evidence maps provide a systematic method for mapping the evidence on a particular topic, with the resulting map facilitating identification of gaps in the literature [4, 5]. Bragge  defines evidence mapping as describing the yield, design and characteristics of research in broad topic areas, in contrast to systematic reviews, which usually address narrowly focused research questions. Evidence mapping has been primarily used to map the evidence for healthcare interventions [7, 8]; however, the approach may also be usefully applied for mapping the evidence on other topics. For example, evidence mapping may be useful for collating evidence on the range and performance of research methods. To our knowledge, evidence mapping has yet to be applied in this way .
Where possible, methodological guidance for conducting overviews should be based on empirical research; that is, based on methods that have been evaluated and shown to have better performance . It is therefore timely to systematically review and map the available methods literature to determine where there are gaps or areas of uncertainty and hence what methods research should be undertaken as a priority. We will apply the evidence mapping approach to studies evaluating the methods for conducting, interpreting and reporting overviews of systematic reviews of interventions.
The objectives of this study are to develop and populate a framework of methods that have been used, or may be used, in conducting, interpreting and reporting overviews of systematic reviews of interventions (stage I); (2) create an evidence map of studies that have evaluated these methods (stage II); and (3) identify and describe unique methodological challenges of overviews.
The methods used in this study are based on published methods for mapping the evidence in broad content areas [4, 6, 10–13]. Methods for developing an evidence map involve several steps, namely, defining a topic and specific questions to be answered by the evidence map; searching for and selecting relevant studies; and reporting on the yield and study characteristics.
Our mapping study will involve two stages. In stage I, we will use the following sources to identify and describe methods that have been used or recommended for use in overviews: articles describing methods for overviews, methods studies that have described the characteristics of a cross section or cohort of overviews; guidance documents for conducting overviews; and commentaries or editorials. The identified methods will populate a framework of methods relating to the steps involved in conducting an overview (e.g. defining objectives, searching for systematic reviews, selecting studies). The second stage will involve the development of an evidence map of studies that have evaluated any of the identified methods (stage II). The steps involved in these stages are depicted in Fig. 1. The evidence map will facilitate identification of methods that have had little or no evaluation.
We have used the Preferred Reporting Items for Systematic reviews and Meta-Analyses Protocols (PRISMA-P)  to develop this protocol. The PRISMA-P checklist was developed for the preparation of protocols of systematic reviews that summarise aggregate data from studies, particularly the evaluations of the effects of interventions, and therefore, not all PRISMA-P items were applicable (Additional file 1). We did not register our evidence map on PROSPERO, the international prospective register of systematic reviews (http://www.crd.york.ac.uk/PROSPERO/), since reviews of ‘methodological issues’ do not met the inclusion criteria for this registry (as specified 29 June 2015).
Several sources will be searched to identify available methods (stage I) and studies that have evaluated the performance of those methods (stage II). We will undertake searches of the following methods collections: Cochrane Methodology Register, Meth4ReSyn library, the Scientific Resource Center Methods library of the AHRQ Effective Health Care Program, and Cochrane Colloquium abstracts. In addition, we will undertake a search of Ovid MEDLINE from 2000 onwards. We will use variations of the following search terms: overview, umbrella reviews, overviews of reviews, overviews of systematic reviews, meta-review or review of reviews (Additional file 2).
We will conduct purposive searches to locate evaluations of the methods we have identified, where the above searches are unlikely to have located these evaluations. For example, although searching for systematic reviews is a fundamental step of overviews, there are other purposes aside from overviews (e.g. developing guidelines) for which retrieving systematic reviews is necessary. Therefore, papers describing the development and evaluation of search strategies for systematic reviews may reasonably not have mentioned ‘overviews’ (or its synonyms) and thus would not be identified in the searches we describe above. Details of these additional search strategies will be documented and reported.
Given the potential difficulties in locating methods literature on overviews, we will supplement our searches by examining the reference lists of all included articles and undertaking citation searches of seminal papers using Google Scholar, Scopus and Web of Science. All three of these databases will be searched for citations because searches run in each database have been shown to return unique material . In addition, as part of a related research project to develop a search strategy to identify overviews in MEDLINE, we set aside any methods papers that we identified through screening citations . Finally, we will contact relevant experts in the field to help identify additional methods papers.
Separate eligibility criteria will be applied for stages I and II of the development of the evidence map (Fig. 1).
Stage I: populating the framework—identification of methods used in overviews of systematic reviews of interventions
Articles describing methods for overviews
Studies examining methods used in a cross section or cohort of overviews
Guidance (e.g. handbooks and guidelines) for undertaking overviews
Commentaries or editorials that discuss methods for overviews
Articles published in languages other than English
Studies describing methods for network meta-analysis
Articles exclusively about methods for overviews of other review types (i.e. not of interventions)
If necessary, the framework (stage I) will be further refined to include methods identified from studies in stage II, if these have not already been identified in stage I.
Stage II: populating the evidence map—identification of evaluations of methods for overviews of systematic reviews of interventions
Systematic reviews of methods studies that have evaluated methods for overviews
Methods studies that have evaluated methods for overviews
Articles published in languages other than English
Methods studies that have evaluated methods for network meta-analysis
In cases where we identify systematic reviews in which the search for studies was conducted before 2013, we will search for more recent methods studies and include the results of these in the evidence map, in addition to including the results of the systematic review.
We plan to populate the evidence map with evaluations of methods that are different or additional to those required to undertake a systematic review of primary research. For example, methods for screening articles for inclusion in an overview (e.g. whether one or more reviewers are required) do not differ to those used in undertaking a systematic review of primary research, so would not be included. However, evaluations of methods used to assess the quality or risk of bias of systematic reviews are of relevance to overviews, and not systematic reviews of primary research, and so would be included. It may be the case that methods have been evaluated in the context of other ‘overview’ products, such as guidelines, and if these methods are of relevance to overviews, they will be included.
The eligibility criteria will be pilot tested by three independent reviewers on a sample of articles retrieved from the search to ensure consistent application.
Citations retrieved from the searches will be imported into EndNote. Duplicates will be identified and removed. Two reviewers will independently review titles and abstracts for their potential inclusion against the selection criteria. Full-text articles will be retrieved when both reviewers agree that inclusion criteria have been met or when there is uncertainty. Two reviewers will then independently assess the articles for inclusion; any disagreement will be resolved by discussion or by arbitration of a third reviewer. In instances where there is limited or incomplete information regarding a study’s eligibility (e.g. when only an abstract is available), the authors will be contacted to request the full text or further details.
Assessment of the risk of bias of evaluation studies (stage II)
There is no tool available to assess the risk of bias of studies that have evaluated methods (i.e. studies meeting the criteria for stage II). We will therefore report characteristics of the evaluation studies that may plausibly be associated with bias and have been used in other methodology reviews (e.g. [17–19]). These characteristics will include study design, method to select the cohort of studies included in the evaluation, process used to extract data and existence of a protocol. For systematic reviews of methods studies, we will use a tool to assess the risk of bias in systematic reviews (ROBIS) .
Data extraction, data coding and analysis
Developing the coding framework
The coding framework will be developed using an iterative process. Two reviewers (CL, SB) will independently code three articles using a simple framework comprised of codes representing the main steps in conducting an overview as outlined in the Cochrane Handbook (e.g. specifying eligibility criteria, data extraction) . Additional codes will be generated inductively by the two reviewers from the text of the three articles. The independently developed frameworks will then be compared and combined into a single integrated framework. The resulting framework will be reviewed by all authors, then tested on three further articles purposively selected to reflect the diversity of article types (i.e. study examining methods in a cohort of overviews, guidance, editorial) included at stage I. Further refinements will be discussed and agreed by all authors. The preliminary framework of parent and illustrative child codes, with descriptions for interpreting each code, is presented in Table 1.
Stage I: populating the framework
Data will be coded from articles meeting the inclusion criteria of stage I using NVivo software. Coded data will be analysed thematically to further refine and populate the framework with descriptions of each method. This thematic analysis will be used to (1) determine the most appropriate categorisation of methods (i.e. the final structure and terminology used in the framework); (2) determine whether or not methods are distinct (i.e. identifying whether methods described using different terminology are indeed different and vice versa); and (3) write a description of each method by synthesising key concepts across included studies. In addition, data will be coded to identify the type of paper (article describing method for overviews, cohort/cross-sectional; commentary or editorial; guidance document); noted advantages/disadvantages of the method; examples of use; at which step in the overview process the method is used; and unique methodological considerations in conducting overviews.
Stage II: populating the evidence map
We plan to extract the following data from methods evaluation studies using a data extraction form: characteristics of the article (e.g. publication year, journal); characteristics of the study (study design, type of sample selection (e.g. random, consecutive, others), process to extract data, existence of a protocol); primary objective(s); secondary objective(s); methods evaluated for primary objective; evaluation analysis methods; and the quantitative results relating to the primary objective.
Coding and data extraction for both stages of the study will be undertaken independently by two reviewers. We anticipate that it will be difficult to pre-specify response categories for each data extraction item due to the potential variability in the type of evaluations. Therefore, much of the extracted data will be verbatim free text, which will be categorised following discussion between the reviewers. Discrepancies in extracted data will be discussed between the reviewers until consensus is reached or by arbitration of a third reviewer if required.
We will describe the evaluations that have been undertaken and map these evaluations to the framework of methods identified in stage I. The description of evaluations will include the yield and characteristics of the available evidence, the use, advantages and disadvantages of each method and a summary of the findings of the evaluations.
Methods for conducting, interpreting and reporting overviews are in their infancy. To date, there has been no systematic review or evidence map examining either the range of methods for overviews or the evidence behind those methods. We plan to use evidence mapping methods to develop and populate a framework of methods used in conducting, interpreting and reporting overviews of systematic reviews of interventions (stage I) and create an evidence map of studies that have evaluated these methods (stage II). This will be a novel application of the evidence mapping methodology in a context where, to our knowledge, this method has not been used.
The results of the evidence map may influence the use of particular methods in overviews, either directly or by influencing guidance for overview methods. However, perhaps the most notable use of the evidence map will be in directing methods research to areas where there is limited evidence for the methods that are being used. Such a map may also be useful for national funding agencies in considering what methods research should be funded.
Strengths and limitations
We have developed a protocol to guide our research and reduce post hoc decision making. Two reviewers will undertake the screening, data coding and data extraction of all methods studies. Searching for methodological papers can prove very difficult in databases other than specialist methodology collections, such as the Cochrane Methodology Register and the Meth4ReSyn library . In addition, while the coverage of two of the specialist methodology collections extends beyond the health and medical literature (Meth4ReSyn and Scientific Resource Center Methods library), the focus of our search is within health. These factors might mean that some relevant methodology papers are missed. However, we have included reference checking, citation searching and contacting relevant experts in the field to minimise the impact of these limitations.
At the time of submitting this protocol, we have undertaken the searching and screening for stage I. Two reviewers have independently coded three methods papers for stage I and developed an initial framework (Table 1). During the peer-review process, we amended the inclusion criteria to clarify that our focus was on methods for overviews of systematic reviews of interventions. We also revised our search strategies in response to suggestions from the reviewer.
Our results will lead to an inventory of evaluation studies of methods for overviews of systematic reviews of interventions. The evidence map will aid in the development and implementation of methods for overviews which will be relevant to a wide range of knowledge users, including researchers, funders and journal editors. The evidence map will help to prioritise the future research agenda in this field.
Population, Intervention, Comparison and Outcomes
Preferred Reporting Items for Systematic Reviews and Meta-Analyses for Protocols
a tool for assessing the risk of bias in systematic reviews
Caldwell DM, Welton NJ, Ades AE. Mixed treatment comparison analysis provides internally coherent treatment effect estimates based on overviews of reviews and can reveal inconsistency. J Clin Epidemiol. 2010;63:875–82.
Pieper D, Buechter R, Jerinic P, Eikermann M. Overviews of reviews often have limited rigor: a systematic review. J Clin Epidemiol. 2012;65(12):1267–73.
Edwards P, Clarke M, DiGuiseppi C, Pratap S, Roberts I, Wentz R. Identification of randomized controlled trials in systematic reviews: accuracy and reliability of screening records. Stat Med. 2002;21(11):1635–40.
Snilstveit B, Vojtkova M, Bhavsar A, Gaarder M. Evidence gap maps—a tool for promoting evidence-informed policy and prioritizing future research. Policy Research working paper. In. Washington DC: World Bank Group Contract no. WPS 6725. http://documents.worldbank.org/curated/en/2013/12/18648542/evidence-gap-maps-tool-promoting-evidence-informed-policy-prioritizing-future-research. Accessed 13 July 2015.; 2013.
Pham MT, Rajić A, Greig JD, Sargeant JM, Papadopoulos A, McEwen SA. A scoping review of scoping reviews: advancing the approach and enhancing the consistency. Res Synth Methods. 2014;5(4):371–85.
Bragge P, Clavisi O, Turner T, Tavender E, Collie A, Gruen RL. The global evidence mapping initiative: scoping research in broad topic areas. BMC Med Res Methodol. 2011;11(1):92.
Schneider C, Mohsenpour A, Joos S, Bozorgmehr K. Health status of and health-care provision to asylum seekers in Germany: protocol for a systematic review and evidence mapping of empirical studies. Syst Rev. 2014;3:139.
Althuis MD, Weed DL. Evidence mapping: methodologic foundations and application to intervention and observational research on sugar-sweetened beverages and health outcomes. Am J Clin Nutr. 2013;98(3):755–68.
McKenzie JE, Clarke MJ, Chandler J. Why do we need evidence-based methods in Cochrane? Cochrane Database Syst Rev. 2015;7:ED000102.
Arksey H, O'Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8(1):19–32.
Armstrong R, Hall BJ, Doyle J, Waters E. ‘Scoping the scope’ of a Cochrane review. J Public Health. 2011;33(1):147–50.
Daudt HM, Van Mossel C, Scott SJ. Enhancing the scoping study methodology: a large, inter-professional team’s experience with Arksey and O’Malley’s framework. BMC Med Res Methodol. 2013;13(1):48.
Levac D, Colquhoun H, O’Brien KK. Scoping studies: advancing the methodology. Implement Sci. 2010;5(1):1–9.
Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P) 2015 statement. Syst Rev. 2015;4(1):1.
Bakkalbasi N, Bauer K, Glover J, Wang L. Three options for citation tracking: Google Scholar, Scopus and Web of Science. Biomed Digit Libr. 2006;3:7.
Lunny C, McKenzie JE, McDonald S. Retrieval of overviews of systematic reviews in MEDLINE was improved by a validated search strategy. J Clin Epidemiol. Article in Press; http://dx.doi.org/10.1016/j.jclinepi.2015.12.002.
Page MJ, McKenzie JE, Kirkham J, Dwan K, Kramer S, Green S, et al. Bias due to selective inclusion and reporting of outcomes and analyses in systematic reviews of randomised trials of healthcare interventions. Cochrane Database Syst Rev. 2014;10:MR000035.
Lundh A, Sismondo S, Lexchin J, Busuioc OA, Bero L. Industry sponsorship and research outcome. Cochrane Database Syst Rev. 2012;12:Mr000033.
Lundh A, Jørgensen AW, Bero L. Association between personal conflicts of interest and recommendations on medical interventions. Cochrane Database Syst Rev. 2013;6(MR000040). doi: 10.1002/14651858.MR000040.
Whiting P, Savovic J, Higgins JP, Caldwell DM, Reeves BC, Shea B, et al. ROBIS: a new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2015. doi:10.1016/j.jclinepi.2015.06.005.
Higgins JPT, Green, S. (editors). Cochrane handbook for systematic reviews of interventions Version 5.1. 0 [updated March 2011]. Cochrane Collab. 2011. Available from www.cochrane-handbook.org. Accessed 13 July 2015. In.; 2011.
Golder S, Loke Y. Search strategies to identify information on adverse effects: a systematic review. J Med Libr Assoc. 2009;97(2):84–92.
Wells G, Shea B, O’connell D, Peterson J, Welch V, Losos M, Tugwell P. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. Ottawa: Ottawa Hospital Research Institute. http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp. Accessed 13 July 2015 2000.
Shea BJ, Hamel C, Wells GA, Bouter LM, Kristjansson E, Grimshaw J, et al. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. In: J Clin Epidemiol. 2009;62:1013–20.
NICE. The guidelines manual. London: National Institute for Health and Clinical Excellence. http://www.nice.org.uk/article/pmg6b/chapter/Appendix%20B%3a%20Methodology%20checklist%3a%20systematic%20reviews%20and%20metaanalyses. Accessed 13 July 2015
Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336(7650):924–6.
Harbour R, Miller J. A new system for grading recommendations in evidence based guidelines. BMJ. 2001;323(7308):334–6.
Hillier S, Grimmer-Somers K, Merlin T, Middleton P, Salisbury J, Tooher R, et al. FORM: an Australian method for formulating and grading recommendations in evidence-based clinical guidelines. BMC Med Res Methodol. 2011;11:23.
This work was conducted as part of a PhD undertaken by Carole Lunny, who is funded by an Australian Postgraduate Award and an International Postgraduate Research Scholarship administered through Monash University, Australia. Joanne McKenzie holds a National Health and Medical Research Council (NHMRC) Australian Public Health Fellowship (1072366).
The authors declare that they have no competing interests.
CL participated in the design of the study, contributed to the design of the data coding and analysis methods, developed the coding framework and drafted the manuscript. JM conceived the study and drafted the manuscript. SM participated in the design of the study and developed and refined the search strategies. SB contributed to the design of the data coding and analysis methods and developed the coding framework. All authors provided critical review of the manuscript. All authors read and approved the final manuscript.
PRISMA-P: recommended items to address in a systematic review protocol. The Preferred Reporting Items for Systematic reviews and Meta-Analyses Protocols (PRISMA-P)  was used to develop this protocol. The PRISMA-P checklist was developed for the preparation of protocols of systematic reviews that summarise aggregate data from studies, particularly the evaluations of the effects of interventions, and therefore, not all PRISMA-P items were applicable. (DOC 54 kb)
Search strategies. The search strategies that will be used to identify eligible studies. (DOC 24 kb)
About this article
Cite this article
Lunny, C., Brennan, S.E., McDonald, S. et al. Evidence map of studies evaluating methods for conducting, interpreting and reporting overviews of systematic reviews of interventions: rationale and design. Syst Rev 5, 4 (2016). https://doi.org/10.1186/s13643-015-0178-0