Toward a comprehensive evidence map of overview of systematic review methods: paper 1—purpose, eligibility, search and data extraction

Background Overviews of systematic reviews attempt to systematically retrieve and summarise the results of multiple systematic reviews. Methods for conducting, interpreting and reporting overviews are in their infancy. To date, there has been no evidence map of the methods used in overviews, thus making it difficult to determine the gaps and priorities for methods research. Our objectives were to develop and populate a comprehensive framework of methods for conducting, interpreting and reporting overviews (stage I) and to create an evidence map by mapping studies that have evaluated overview methods to the framework (stage II). Methods We searched methods collections (e.g. Cochrane Methodology Register, Meth4ReSyn library, AHRQ Effective Health Care Program) to identify eligible studies for both stages of this research. In stage I, cross-sectional studies, guidance documents and commentaries that described methods proposed for, or used in, overviews were used to develop and populate the framework of methods. Drafts and multiple iterations of the framework were discussed and refined by all authors. In stage II, we identified and described studies evaluating overview methods and mapped these evaluations to the framework. Results In this paper, we present results for the four initial steps of conducting an overview: (a) specification of the purpose, objectives and scope, (b) specification of the eligibility criteria, (c) search methods and (d) data extraction. Twenty-nine studies mentioned or described methods relevant to one or more of these steps. In the developed framework, identified methods and approaches were grouped according to the steps an overview author would need to undertake. Fifteen studies evaluated identified methods, all of which mapped to the search methods step. These studies either reported the development and evaluation of a new search filter to retrieve systematic reviews or compared the performance of multiple filters. Conclusion Gaps in the evaluation of methods were found for the majority of steps in the framework. More empirical studies are needed to evaluate the methods outlined and provide a comprehensive evidence map. The framework is useful for planning these evaluations and for planning methods required to deal with challenges that arise when conducting an overview. Electronic supplementary material The online version of this article (10.1186/s13643-017-0617-1) contains supplementary material, which is available to authorized users.


Background
Overviews of systematic reviews synthesise the results of multiple systematic reviews. Overviews are typically broader in scope than systematic reviews (SRs) and may examine different interventions for the same condition, the same intervention for different conditions, or the same intervention for the same condition but focusing on different outcomes [1][2][3][4].
The number of published overviews has increased steadily in recent years largely in response to the increasing number of SRs [5,6]. The main steps and many of the methods used in the conduct of SRs are directly transferrable to overviews, such as independent study selection and data extraction [7]. However, many features are unique to overviews and require the application of different or additional methods. For example, methods for assessing the quality or the risk of bias of SRs, dealing with the inclusion of the same trial in multiple SRs, dealing with out-of-date SRs, and dealing with discordant results across SRs [6].
Despite the growth in overviews, there has been no evidence map identifying the range of methods for overviews and examining the evidence for using these methods. Evidence mapping is a systematic method used to characterise and catalogue a body of literature pertaining to evidence on a topic and is useful for identifying gaps in the literature [8,9]. Evidence mapping has been commonly used to map the effects of healthcare interventions; however, the approach may also be applied for mapping the evidence on other topics, such as collating and synthesising evidence on the range and performance of research methods.
It is critical to determine whether there is evidence to support the use of methods for overviews because the validity and reliability of the findings from overviews depend on the performance of the underlying methods. This research aims to provide a comprehensive framework of overview methods and the evidence underpinning these methods-an evidence map of overview methods. In doing so, we aim to help overview authors plan for common scenarios encountered when conducting an overview and enable prioritisation of methods development and evaluation.

Objectives
The objectives of this study were to (a) develop and populate a comprehensive framework of methods that have been used, or may be used, in conducting, interpreting and reporting overviews of systematic reviews of interventions (stage I); (b) map studies that have evaluated these methods to the framework (creating an evidence map of overview methods) (stage II); and (c) identify unique methodological challenges of overviews and methods proposed to address these. This paper is the first of two companion papers. In this first paper, we present the methods framework for the four initial steps of conducting an overview: (a) specification of the purpose, objectives and scope of the overview; (b) specification of the eligibility criteria; (c) search methods and (d) data extraction methods (stage I). We then map studies evaluating methods to this framework (stage II). In a second paper, we will present the methods framework, and a map of evaluation studies, for the subsequent steps in conducting an overview: assessing risk of bias of primary studies and SRs; certainty of evidence arising from the overview; synthesis, presentation and summary of findings; and interpretation of findings and drawing conclusions (Fig. 1).
We use the term 'methods framework' (or equivalently, 'framework of methods') to describe the organising structure we have developed to group related methods and against which methods evaluations can be mapped. The highest level of this structure is the broad steps of conducting an overview (e.g. search methods). The methods framework, together with the studies that have evaluated these methods, form the evidence map of overview methods.

Methods
A protocol for this study has been published [10]. The methods for the two stages ( Fig. 2) are now briefly described, along with deviations from the planned methods.
Stage I: development and population of the framework of methods

Search methods
We searched MEDLINE from 2000 onwards and the following methods collections: Cochrane Methodology Register, Meth4ReSyn library, Scientific Resource Center Methods library of the AHRQ Effective Health Care Program, and Cochrane Colloquium abstracts. Searches were last run on December 2, 2015 (see Additional file 1 for search strategies). We also set aside any methods articles that we identified through screening citations as part of a related research project to develop a search strategy to identify overviews in MEDLINE [5]. To identify other potentially relevant studies, we examined the reference lists of included studies and undertook forward citation searches of seminal articles using Google Scholar, Scopus and Web of Science. We contacted authors of posters to retrieve the poster, or the full report of the study, and to ask if they were aware of any related methods articles. We planned to contact researchers with expertise in methods for overviews to identify articles missed by our search, but did not undertake this step due to time constraints.

Eligibility criteria
For the development and population of the framework, we identified articles describing methods used, or recommended for use, in overviews of systematic reviews of interventions.
Inclusion criteria: i. Articles describing methods for overviews of systematic reviews of interventions ii. Studies examining methods used in a cross-section or cohort of overviews iii. Guidance (e.g. handbooks and guidelines) for undertaking overviews iv. Commentaries or editorials that discuss methods for overviews Exclusion criteria: i. Articles published in languages other than English ii. Studies describing methods for network meta-analysis iii. Articles exclusively about methods for overviews of other review types (i.e. not of interventions) We populated the framework with methods that were different or additional to those required to undertake a SR of primary research. Methods evaluated in the context of other 'overview' products, such as guidelines, which were of relevance to overviews, were included.
The eligibility criteria were piloted by three reviewers independently on a sample of articles retrieved from the search to ensure consistent application.

Study selection
Two reviewers independently reviewed titles and abstracts for their potential inclusion against the eligibility criteria. Full-text articles were retrieved when both reviewers agreed that inclusion criteria were met or when there was uncertainty. Any disagreement was resolved by discussion or by arbitration of a third reviewer. In instances where there was limited or incomplete information regarding a study's eligibility (e.g. when only an abstract was available), the study authors were contacted to request the full text or further details.

Data extraction, coding and analysis
One author collected data from all included articles using a pre-tested form; a second author collected data from a 50% sample of the articles.
Data collected on the characteristics of included studies We collected data about: (i) the type of article (coded as per our inclusion criteria), (ii) the main contribution(s) of the article (e.g. critique of methods), (iii) the extent to which each article described methods or approaches pertaining to each step of an overview (e.g. mention without description, described-insufficient detail to implement, described-implementable), (iv) a precis of the methods or approaches covered and (v) the data on which the article was based (e.g. audit of methods used in a sample of overviews, author's experience).
Coding and analysis to develop and populate the framework of methods We planned to code articles in NVivo software, applying a coding frame to extract descriptions of methods pertaining to each step of an overview [10]. However, during the initial phases of analysis, we found the extracts difficult to interpret when read out of context because many methods were either sparsely described or were inferred rather than explicit. As a consequence of the difficulty coding these data, we revised our analytic approach. We separated studies that described a method pertaining to a step in the overview process, from those that made cursory mention of a method. The subset of articles coded as providing description were read by two authors (CL and SB, JM or SM) who independently drafted the framework for that step to capture and categorise all identified or inferred methods. To ensure comprehensiveness of the framework, methods were inferred when a clear alternative existed to a reported method (e.g. using decision rules or an algorithm to combine eligibility criteria was rarely mentioned, but was clearly an option for multiple sub-steps).
The drafts and multiple iterations of the framework were discussed and refined by all authors, during which we delineated unique decision points faced when planning each step of an overview (e.g. determining eligibility criteria to deal with SRs with overlap, determining how discrepant data across SRs will be handled) and the methods/options available for each. We grouped conceptually similar approaches together and extracted examples to illustrate the options. For example, we categorised all approaches that involved specifying criteria to select one SR from multiple overlapping SRs together, and then listed examples of criteria suggested in included studies (e.g. select most recent SR, highest quality, most comprehensive).

Search methods
In addition to the main searches outlined in the 'Search methods' section for Stage I, we planned to undertake purposive searches to locate evaluations of methods where the main searches were unlikely to have located these evaluations. For this paper, we undertook a purposive search to locate evaluations of search filters for the retrieval of SRs (Additional file 2) since articles describing the development and evaluation of search strategies for SRs may reasonably not have mentioned 'overviews' (or its synonyms) and thus would not be identified in the main searches. For the other steps, the identified methods were specific to overviews, so evaluations were judged likely to be retrieved by our main search.

Eligibility criteria
To create the evidence map, we identified articles describing evaluations of methods for overviews of systematic reviews of interventions.
Inclusion criteria: i. SRs of methods studies that have evaluated methods for overviews ii. Methods studies that have evaluated methods for overviews Exclusion criteria: i. Articles published in languages other than English ii. Methods studies that have evaluated methods for network meta-analysis We added the additional criterion that methods studies had to have a stated aim to evaluate methods, since our focus was on evaluation and not just application of a method.

Study selection
We used the same process for determining which studies met the inclusion criteria for stage II as for stage I ('Study selection' section Stage I).

Data extraction
The only methods evaluations identified were evaluations of search filters for SRs, from which we extracted the data listed in Table 1. We had originally planned to extract quantitative results from the methods evaluations relating to the primary objectives; however, on reflection, we opted not to do this since we felt this lay outside the purpose of the evidence map. Data were extracted independently by two authors (CL, JEM) from four (of 15) studies. The remaining data were extracted by one author (CL).

Assessment of the risk of bias
We planned to report the characteristics of the stage II evaluation studies that may plausibly be associated with bias. For methods evaluations of search filters for identifying SRs, we used assessment criteria informed by Harbour [11]. The assessment criteria included existence of a protocol and validation of the filter on a data set distinct from the derivation set (external validation).

Analysis
The yield and characteristics of the methods evaluation studies were described and mapped to the framework of methods.

Results of the search
We retrieved 1850 records through searching databases and methods collections. A further 1384 records were identified through other sources (methods articles identified as part of a related research project [5], reference checking, and forward citation searching). After removal of duplicate records, 1179 records remained (Fig. 3). From screening titles and abstracts, we excluded 1092 records that were ineligible. We assessed 87 full-text reports for eligibility and excluded 21, with reasons noted in Additional file 3. Of the remaining 66, 42 were included in stage I and 24 in stage II.
Our purposive search strategy (dated May 2016) to identify studies evaluating search filters for the retrieval of SRs resulted in the inclusion of three more stage II studies (see Fig. 4 for details), bringing the total number of methods evaluations to 27. Of the 42 stage I and 27 stage II studies, 29 and 15, respectively, pertained to one or more of the four initial steps in conducting an overview and so are included in this first paper; the remainder will be included in our second companion paper. All 15 stage II studies were evaluations of search filter studies for retrieval of SRs. We first describe the characteristics of the included articles (see 'Characteristics of included articles'; Table 2), followed by presentation of the developed methods framework. This presentation is organised into sections representing the broad steps of conducting an overview (sections 'Specification of purpose, objectives and scope' , 'Specification of eligibility criteria' , 'Search methods' and 'Data extraction'; Tables 3, 4, 5 and 6). In each section, we orient readers to the structure of the methods framework, which includes a set of steps and sub-steps (e.g. under 'Search methods' , the steps are 'plan the sources to search' , 'plan the search strategy for retrieval of SRs' , and 'plan how primary studies will be retrieved'). Components within the tables are referred to using labels and numbers (e.g. 2.1.3). We highlight methods/approaches to deal with commonly encountered scenarios for which overview authors need to plan (see ' Addressing common scenarios unique to overviews'; Table 7). Our description is focused on methods/options that are distinct, have added complexity, compared with SRs of primary studies, or have been proposed to deal with major challenges in undertaking an overview. Importantly, the methods/approaches and options reflect the ideas presented in the literature and should not be interpreted as endorsement for the use of the methods. Reporting considerations for all steps are reported in Additional file 4.

Characteristics of included articles
The characteristics of the included articles and the extent to which each described methods or approaches pertaining to the initial steps of an overview are indicated in Table 2. The majority of articles were published as full reports (n = 24/29; 83%). The most common type of article was one in which methods for overviews were described (n = 16/29; 55%), followed by articles that examined the methods used in a crosssection of overviews (n = 8/29; 28%), guidance documents (n = 4/29; 14%) and commentaries and editorials (n = 1/29; 3%). Methods for the specification of purpose, objectives and scope (n = 22); specification of eligibility criteria (n = 21); search methods (n = 18) and methods for data extraction (n = 17) were similarly mentioned or   Integration of evidence from multiple meta-analyses: a primer on umbrella reviews, treatment networks and multiple treatments meta-analyses.
Article describing methods for overviews • Defines umbrella reviews as a pre-step to network meta-analysis • Describes challenges of overviews and a checklist of pitfalls  The evolution of a new publication type:

✓✓
Steps and challenges of producing overviews of reviews.
Article describing methods for overviews • Describes some steps in undertaking an overview and the challenges inherent in production of overviews • Discusses that gaps or lack of currency in included evidence will weaken the overview findings ✓✓ ✓✓  described. Relatively, few articles described methods across all of the initial steps in conducting an overview (n = 6).

Specification of purpose, objectives and scope
The two steps in the framework under 'specification of purpose, objectives and scope' were 'determine stakeholder involvement in planning the overview (1.0)' and 'define the purpose, objectives and scope (2.0)' (Table 3). In the following, we focus on the methods/approaches and options for the step 'define the purpose, objectives and scope (2.0)'. Other methods/approaches are similar to those in planning a SR, but have been included in the framework for completeness. We identified different purposes for undertaking an overview (2.1), with some of these purposes being 'map the type and quantity of available evidence (2.1.1)' , 'compare multiple interventions with the intent of drawing inferences about the comparative effectiveness of the interventions for the same condition (2.1.2)' and 'summarise the effects of an intervention across different conditions, populations, or problems (2.1.4)'. The latter borrows strength when there is sparse data for a single condition and a similar mechanism of action for the intervention is predicted across conditions. Options for confirming that an overview is the appropriate type of study for addressing the purpose and objectives (as compared with an intervention review or network meta-analysis) (2.2), included the 'use of a decision tool (2.2.1)' or 'use other reasoning (2.2.2)'. A further identified sub-step was to 'determine any constraints that will restrict the scope of the overview (2.3)'. Considerations arising from sub-steps 2.1-2.3 will influence whether an overview is conducted to address a narrow or broad question (2.4). This decision is then operationalised in the final identified sub-step 'define the objectives using Population, Intervention, Comparison, Outcome (PICO) elements (or equivalent) to develop an answerable question (2.5)'.

Specification of eligibility criteria
The two steps in the framework under 'specification of eligibility criteria' were 'plan the eligibility criteria (1.0)' and 'plan the study selection process (2.0)' (Table 4). In the following, we focus on the step 'plan the eligibility criteria (1.0)' , which covers methods that are key to dealing with common scenarios and challenges that arise in overviews (Table 7). A unique decision in planning overviews is to 'determine methodological eligibility criteria for SRs (1.4)'. Multiple criteria were identified, including approaches for selecting reviews that meet minimum quality criteria, or reviews that take a particular methodological approach (1.4.2). These criteria underpin many of the identified approaches for dealing with SRs with overlap in information and data (1.5). Overlap can arise when SRs with similar topics include one or more identical primary studies. One identified option was to include all SRs that meet the PICO criteria irrespective of overlap, that is, ignore overlap, note overlap, or deal with overlap using other methods (e.g. data extraction, synthesis) (1.5.1). However, other approaches aim to minimise overlap by specifying criteria to select one SR from multiple (1.5.2). These approaches include selecting one SR based on methodological criteria for SRs (see options in 1.4.2), selecting the most comprehensive SR, or excluding SRs that do not contain any unique primary studies (1.5.4). The latter approach may still result in inclusion of multiple overlapping SRs. An inherent complexity in using eligibility criteria to deal with overlap is that using single criteria can result in unintended loss of information through exclusion of important SRs (for example, the most recent SR could be excluded if only the highest quality SR is selected). An approach that overcomes this is to combine multiple criteria in an algorithm (1.5.3).
Another identified decision was whether to include additional primary studies (1.6). One option was to include primary studies only if pre-specified eligibility criteria are met (1.6.2). Circumstances that may prompt inclusion of primary studies are outlined in 1.6.2. Adaption of the step from SRs to overviews. No methods evaluation required, but special consideration needs to be given to unique issues that arise in conducting overviews

Search methods
The three steps in the framework under 'search methods' were 'plan the types of sources to search (1.0)' , 'plan the search strategy for retrieval of SRs (2.0)' , and 'plan how primary studies will be retrieved, if eligibility criteria determines that primary studies should be included (3.0)' (Table 5). Search methods for overviews largely parallel those used in a SR of primary studies. Unique considerations relate to the option to restrict searches to SR databases (1.1.1), the use of filters developed to retrieve SRs (2.1), and approaches to searching for additional primary studies. If additional primary studies are eligible for the overview, authors will need to determine the sequence of searching for SRs and primary studies. The search for primary studies may be done in parallel with the search for SRs (3.1.1), or in sequence, searching first for SRs then for primary studies (3.1.2). The latter strategy focuses on retrieving primary studies where evidence is missing (i.e. where SRs are not up-to-date or where the SRs provide incomplete coverage of the overview question).

Data extraction
The two steps in the framework under 'data extraction' were 'plan the data elements to extract (1.0)' and 'plan the data extraction process (2.0)' (Table 6). We now highlight methods/approaches for dealing with these two steps, with a focus on methods for dealing with scenarios described in Table 7. Adaption of the step from SRs to overviews. No methods evaluation required, but special consideration needs to be given to unique issues that arise in conducting overviews  Adaption of the step from SRs to overviews. No methods evaluation required, but special consideration needs to be given to unique issues that arise in conducting overviews Table 7 Methods and approaches for addressing common scenarios unique to overviews Methods/approaches proposed in the literature a Scenario for which authors need to plan Eligibility criteria (Table 4) Data extraction (Table 6) 1 Reviews include overlapping information and data (e.g. arising from inclusion of the same primary studies)    An identified sub-step in planning the data elements to extract (1.0) was determining the data to extract about the results from the SRs (1.3). For overviews, this will be driven by the purpose of the overview (e.g. whether the aim of the overview is to summarise results narratively from included SRs, or synthesise the results from component trials, or meta-analyses, from the included SRs). In addition to determining the data to extract about results from SRs, if the eligibility criteria of the overview include primary studies, then the data to extract from primary studies will also need to be determined (1.4).
A complexity that arises when undertaking an overview is the challenge of how to deal with overlapping (2.2) and discrepant (2.3) information and data across SRs. Identified options include extraction of information from all SRs, noting any discrepancies (2.2.1, 2.3.1), or extraction of data and information from only one SR (2.2.2, 2.3.2) based on pre-specified criteria, such as using the most recent SR, or the SR of the highest quality. Or, when there are discrepancies, different data elements (e.g. effect estimates, quality assessments) might be extracted from different SRs that meet certain decision rules (2.3.3), such as the SR that reports the most complete information on effect estimates. Methods for dealing with variation in the information reported and missing data are outlined in sub-step 2.4. In overviews, compared with SRs, there is additional complexity in resolving variation in information reported and missing data since there is an additional source of information (SRs in addition to primary studies).

Addressing common scenarios unique to overviews
Many of the identified methods were proposed to overcome common methodological challenges unique to overviews. Table 7 summarises these scenarios, showing methods that could be used to address each. While the literature reviewed often suggested a single method or step at which a scenario should be dealt with, Table 7 shows that there are multiple options, some of which can be combined.

Stage II: identification and mapping of evaluations of methods
We found no studies that had evaluated methods in the steps of the framework for 'specification of purpose, objectives and scope' , 'specification of eligibility criteria' and 'data extraction'. Fifteen studies, published between 1998 and 2016, evaluated search filters for the retrieval of SRs (Table 8). One study [12], evaluated the performance of seven bibliographic databases to determine their coverage of SRs. This evaluation mapped to the option 'select the types of databases to search' (1.1.1) of the 'search methods' step of the framework (Table 5). Of the remaining 14 studies, two compared the performance of multiple published filters [13,14], four developed new search filters and compared their performance against other published filters [15][16][17][18], and eight developed and evaluated new search filters (but without comparison with other published filters) [19][20][21][22][23][24][25][26]. These evaluations mapped to the option 'select a published SR filter' (2.1.1) of the 'search methods' step of the framework ( Table 5).
The filters were designed to retrieve SRs across a range of databases (CINAHL, DARE, EMBASE, PsycINFO, Epistemonikos, MEDLINE, Simple Web Indexing System for Humans (SWISH) and TRIP). Seven studies developed the gold standard by handsearching journals, three used a combination of handsearching journals and database searches and five used only database searches. The performance measures used included sensitivity/recall, specificity, precision, accuracy and the number needed to read. In terms of risk of bias, none of the evaluation studies referred to a study protocol or noted the existence of one, and only three validated their search filter on a data set distinct from the derivation set [13,16,17].

Discussion
Despite the emergence of overviews as a common form of evidence synthesis, to date, there has been no comprehensive map of overview methods or the evidence underpinning these methods. We aimed to address this gap. A framework was developed for the initial steps in the conduct, interpretation and reporting of an overview (specification of the purpose, objectives and scope; specification of the eligibility criteria; search methods; and data extraction methods) with associated methods/ approaches and options. The framework makes explicit large number of steps and methods that need to be considered when planning an overview and demonstrates some of the added complexity in an overview compared with a SR of primary studies. The framework also demonstrates that challenges in undertaking an overview, such as dealing with overlapping information across SRs, may be dealt with at different steps of the overview process (e.g. specification of eligibility criteria or data extraction). Fifteen evaluation studies were found in stage II, all of which mapped to the 'search methods' step of the framework. These studies either developed and evaluated a new search filter or compared the performance of existing search filters to retrieve SRs.
What this study adds to guidance and knowledge about overview methods?
Our analysis aligns with findings of other recent reviews in identifying important gaps in guidance on the conduct of overviews [27,28]. These gaps include patchy coverage of methods, wherein guidance covers selected options but not alternatives and insufficient description to operationalise many methods (Table 2). While others have concluded there is a lack of consensus over many methods [28], overviews serve many purposes, and different approaches are needed for different purposes. Recognising this, the framework attempts to capture the spectrum of options available to overview authors, providing a tool for systematic consideration of alternative approaches. We highlight the scenarios for which overview authors need to plan and identify methods proposed to tackle each scenario (Table 7). While these contributions help to address the patchy coverage of methods, the framework cannot address the lack of operational detail in current guidance. A forthcoming update of the Cochrane Handbook should help [28], but other guidance will be needed to cover the many methods not applicable to Cochrane overviews. For authors writing guidance, the framework could serve as a checklist to ensure comprehensive coverage of the methods proposed in the literature.
The lack of evaluation studies identified in stage II indicates that there is limited evidence to inform methods decision-making in overviews. For each of the steps in the framework, there is often a range of different methods to use, which could conceivably impact on the results and conclusions of the overview, their utility for decision-makers, and the time/resources required to complete the overview. This lack of evaluation of methods means [28] there may be inappropriate variability in the methods employed across overviews (as has been observed [6]). Further, overviews that seek to address the same research question, but which are undertaken using different methods, may reach discordant conclusions.
How might the framework be used by overview authors and methodologists?
The framework may be useful to researchers conducting overviews and methodologists. As highlighted above, the framework is useful for making explicit the decisions overview authors need to make when planning an overview. Using the framework as a checklist to plan methods for dealing with common scenarios should lessen the challenges that arise when conducting an overview. Using the framework during protocol development may also lead to less post-hoc decision-making that can arise from not being aware of the decisions that need to be made before commencing the overview. Less post hoc decision-making may limit potential bias in the process of undertaking the overview. For overview methodologists, the provision of comparative options for each of the steps of the framework facilitates identification (and prioritisation) of methods evaluations that might be undertaken. For example, examining those steps where selection of a different option is hypothesised to importantly impact on the results and conclusions of the overview (discussed below under 'Future research to refine and populate the framework and evidence map').

Strengths and limitations
To our knowledge, this is the first attempt to create a comprehensive framework of the many methods proposed for use in overviews. It is also the first study, of which we are aware, that has used evidence mapping in the context of methods research. A protocol of this investigation has been published [10] and any post hoc decisions have been documented. During our analysis, we developed an organising structure to group-related methods and used consistent language to synthesise the varied descriptions encountered in the literature. We also made inferences to ensure that where a clear alternative to a described method existed, it was captured in the framework. Both steps helped generate a more uniform and complete inventory of methods than would have been possible through simply collating methods as described.
Methods studies related to overviews are challenging to find other than in specialist methodology registers, such as the Cochrane Methodology Register and the Meth4ReSyn library, meaning that some methods articles may have been missed. We conducted reference checking and forward citation searching in three databases to minimise the number of missed articles. Further, we focused our search on locating articles that used the term 'overview' (or related terminology). However, methods that may be applicable to overviews, such as those used in clinical practice guidelines, may not have been located. We did not broaden our search, or specifically examine guidance documents for producing guidelines, to keep the project containable. Our analysis involved piecing together information spread across multiple sources, and 'translating' varied descriptions of methods into a common language. This process, and the many decisions involved in structuring our framework required considerable judgement. While the process led to a more complete and uniform description of methods than we identified in any other source, the subjective nature of this analysis means that other researchers may have made different decisions.
Future research to refine and populate the framework and evidence map Future research will involve seeking input on the framework from methodologists and researchers conducting overviews in terms of their face validity, that is, the structure of the framework and the comprehensiveness of the steps and identified methods. Hence, the framework will likely be refined and evolve over time. Further, as methods for overviews are evaluated, the evidence map can be further populated. While there is currently too little methods evaluation for a visual representation (or map) of the evidence to be useful, the framework provides the structure for creating this map. Some priority areas requiring evaluation, which we encourage methodologists to consider, relate to the decisions around eligibility and data extraction. For example, what is the effect of selecting one SR from multiple SRs addressing the same topic versus including all SRs? Outcomes of interest may include proximal measures, such as whether eligible primary studies or important data are missed. More distal measures include time taken to complete the overview, utility for decision-makers, and whether the findings and conclusions of the review change. Additionally, researchers could examine whether observed effects vary when different eligibility criteria are used to select one SR from a multiple. Similar questions can be posed about the effects of extracting data from one versus multiple SRs, from primary studies to SRs only, and so on. Evidence arising from these evaluations should lead to further refinement of the framework and, more importantly, empirical data about the trade-offs associated with alternative methodological approaches.