Skip to main content

The impact of different inclusion decisions on the comprehensiveness and complexity of overviews of reviews of healthcare interventions

Abstract

Background

Overviews of reviews (overviews) compile information from multiple systematic reviews (SRs) to provide a single synthesis of relevant evidence for decision-making. Overviews may identify multiple SRs that examine the same intervention for the same condition and include some, but not all, of the same primary studies. There is currently limited guidance on whether and how to include these overlapping SRs in overviews. Our objectives were to assess how different inclusion decisions in overviews of healthcare interventions affect their comprehensiveness and results, and document challenges encountered when making different inclusion decisions in overviews.

Methods

We used five inclusion decisions to conduct overviews across seven topic areas, resulting in 35 overviews. The inclusion decisions were (1) include all Cochrane and non-Cochrane SRs, (2) include only Cochrane SRs, or consider all Cochrane and non-Cochrane SRs but include only non-overlapping SRs, and in the case of overlapping SRs, select (3) the Cochrane SR, (4) the most recent SR (by publication or search date), or (5) the highest quality SR (assessed using AMSTAR). For each topic area and inclusion scenario, we documented the amount of outcome data lost and changed and the challenges involved.

Results

When conducting overviews, including only Cochrane SRs, instead of all SRs, often led to loss/change of outcome data (median 31% of outcomes lost/changed; range 0–100%). Considering all Cochrane and non-Cochrane SRs but including only non-overlapping SRs and selecting the Cochrane SR for groups of overlapping SRs (instead of the most recent or highest quality SRs) allowed the most outcome data to be recaptured (median 42% of lost/changed outcome recaptured; range 28–86%). Across all inclusion scenarios, challenges were encountered when extracting data from overlapping SRs.

Conclusions

Overlapping SRs present a methodological challenge for overview authors. This study demonstrates that different inclusion decisions affect the comprehensiveness and results of overviews in different ways, depending in part on whether Cochrane SRs examine all intervention comparisons relevant to the overview. Study results were used to develop an evidence-based decision tool that provides practical guidance for overview authors.

Peer Review reports

Background

Systematic reviews (SRs) of healthcare interventions aim to assess an intervention’s efficacy or effectiveness by using explicit and reproducible methods to combine the results of all relevant primary studies [1]. By synthesizing all available data, SRs attempt to explore and ultimately resolve discrepancies among primary studies that may have different, and sometimes contradictory, results of an intervention’s effect. However, as the number of published SRs steadily increases [2], it is becoming increasingly common to find multiple SRs that address the same, or very similar, clinical questions [3]. We refer to these as “overlapping SRs,” and they may include many, but not necessarily all, of the same primary studies, due to differences in methods used for inclusion criteria, search strategies, study selection, and data extraction and analysis [3].

Researchers conducting overviews of reviews of healthcare interventions (overviews) often encounter overlapping SRs. Overviews use explicit and systematic methods to compile data from multiple, related SRs to provide a single synthesis of evidence for healthcare decision-making [4]. They are typically broader in scope than any individual SR and often examine the efficacy or effectiveness of multiple interventions for preventing or treating a specific clinical condition [4]. Overview authors can encounter overlapping SRs when they decide to include both SRs published in and outside of the Cochrane Database of Systematic Reviews (“Cochrane SRs” and “non-Cochrane SRs”). This is because Cochrane attempts to avoid duplication of effort by publishing only one SR on healthcare interventions for a specific condition or illness, whereas multiple non-Cochrane SRs can exist to address the same, or very similar, clinical questions. Researchers that choose to include overlapping SRs in overviews will encounter important methodological considerations [5,6,7,8]. Overview authors should properly assess the amount of overlap in the primary studies contained within the overview’s included SRs. If overlap exists, and the outcome data from some primary studies contribute more than once to the analyses, bias can be introduced into the overview as disproportionate weight has been given to some of the data [5]. Researchers may also find it difficult to appropriately extract and analyze outcome data from overlapping SRs if their conduct, quality, and/or reporting differs between SRs [6]. Further, if overlapping SRs included in the overview have discordant results and/or conclusions, researchers need to decide how they will synthesize and discuss these differences [6]. Despite these methodological considerations, only half of the overviews that contain overlapping SRs currently acknowledge and discuss the overlap [5].

To date, researchers have used several approaches to manage overlapping SRs in overviews [6, 7]. Some researchers have included all relevant Cochrane and non-Cochrane SRs and avoided overlap by extracting outcome data for each primary study only once (regardless of how many SRs contained that study’s data) [9, 10]. Others have avoided overlap by restricting the overview to synthesizing only Cochrane SRs [4, 8, 11], while others have included Cochrane and non-Cochrane SRs and avoided overlap by using specific criteria to prioritize SR inclusion when confronted with multiple, overlapping SRs (e.g., only include the Cochrane, most recent, or highest quality SR) [8, 9]. Currently, there is no empirical evidence on the impact of these different inclusion decisions, and no guidance for how to choose one method of inclusion over another [6, 7].

The purpose of this study was to provide empirical evidence examining the inclusion of overlapping SRs in overviews of reviews of healthcare interventions. Specifically, we assessed how different decisions surrounding the inclusion and exclusion of overlapping SRs in overviews affect the comprehensiveness and results of overviews, and documented challenges encountered when using different inclusion criteria in overviews. Results of this study were then used to develop an evidence-based decision tool to help overview authors make inclusion decisions in overviews. This tool is presented in a companion paper by Pollock et al. [12].

Methods

Study procedures

This was a multiple case study [13]. Each “case” was an overview of reviews conducted by the Alberta Research Centre for Health Evidence between 2010 and 2016 that examined a question related to the efficacy or effectiveness of multiple healthcare interventions for preventing or treating a clinical condition related to pediatric health. Seven cases [14,15,16,17,18,19,20] were included in the study based on convenience sampling [21]: acute asthma [14], acute otitis media [15], bronchiolitis [16], croup [17], eczema [18], gastroenteritis [19], and procedural sedation [20]. The inclusion criteria (populations, interventions, comparators, outcome measures, and study designs) for each case are provided in Additional file 1. For feasibility, we used clinical judgment to restrict the inclusion criteria of four cases, compared to the inclusion criteria used in the published overviews (see footnotes in Additional file 1). We then conducted each of the seven overview cases using five different inclusion decisions (described in detail below). This resulted in 35 overviews of healthcare interventions. We assessed the impact of the different inclusion decisions on the comprehensiveness and results of the overviews, both within and across overview cases.

Conducting the overviews

For each overview, all published, English-language Cochrane and non-Cochrane SRs that met the overview’s inclusion criteria were identified from its reference list. All seven overviews searched for Cochrane SRs; four also searched for non-Cochrane SRs. For the three overviews that did not search for non-Cochrane SRs [15,16,17], we conducted additional literature searches to locate and include non-Cochrane SRs that met the overview’s inclusion criteria. An information specialist conducted the literature searches using the inclusion criteria and search dates from each overview (AM). The search strategies for all overview topics are available in published overviews and upon request. Screening non-Cochrane SRs for inclusion was conducted independently by two reviewers, with discrepancies resolved by consensus or third party adjudication (AC, AM, DO, JS, MM, MO). At the end of the literature identification stage, each of the seven overview cases consisted of a published overview along with all published English-language Cochrane and non-Cochrane SRs that met that overview’s inclusion criteria. Two reviewers independently assessed the methodological quality of each SR in each overview using A MeaSurement Tool to Assess systematic Reviews (AMSTAR) [22], with discrepancies resolved via consensus or third party adjudication (MP, LH, AC, AM, IS, MO, SS). AMSTAR scores (/11) were summarized using means and standard deviations.

The seven overview cases were conducted sequentially, according to five different inclusion scenarios, for a total of 35 overviews. The five inclusion scenarios were chosen because they are commonly cited in the literature as potentially appropriate ways to manage overlapping SRs in overviews [6, 7, 9]. The inclusion scenarios guided the decisions on which SRs and outcome data to include in each overview, as follows:

  • Full inclusion scenario: All eligible outcome data were extracted from all eligible Cochrane and non-Cochrane SRs. We ensured accuracy of effect estimates by making sure that each primary study’s outcome data were extracted only once (regardless of how many SRs contained that study’s data). This involved extracting data from the Cochrane SR (if present), followed sequentially by the most recent and/or highest quality SRs that most closely matched the overview’s scope for each intervention comparison.

  • Restricted scenario 1: All eligible outcome data were extracted from all Cochrane SRs.

  • Restricted scenarios 2 to 4: All eligible outcome data were extracted from all non-overlapping SRs, and in the case of groups of overlapping SRs, we included outcome data from the Cochrane SR (restricted scenario 2), most recent SR (restricted scenario 3), or highest quality SR (restricted scenario 4). For restricted scenario 2, if there was no Cochrane SR within a group of overlapping SRs, no outcome data were extracted. For restricted scenario 3, the most recent SR was defined as the SR with the most recent year of publication (for Cochrane SRs, we used the “year last assessed as up-to-date”). If two SRs were tied for “most recent,” we included the one with the most recent search date. For restricted scenario 4, the highest quality SR was defined as the SR with the highest AMSTAR score (/11). If two SRs were tied for “highest quality,” we noted this in the results files and did not extract data, as there are currently no accepted criteria for objectively choosing between two SRs with the same quality scores. It was possible for different restricted scenarios to end up including the same SRs if, for example, the Cochrane SR was also the most recent and/or the highest quality SR.

Matrices showing which comparisons and SRs were included in the overviews are provided in Additional file 2. Because many SRs examined multiple interventions and comparators, we assessed overlap within SRs for each individual comparison.

Data extraction and analysis for the 35 overviews adhered to standard methods [4]. The following data were extracted for each of the 35 overviews: (1) descriptive characteristics of the SRs (e.g., Cochrane or non-Cochrane, first author, year of publication, populations, and included comparisons), (2) descriptive characteristics of the included primary studies contained within the SRs (e.g., first author, year of publication, study design, and sample size, for studies that matched the relevant overview’s inclusion criteria), and (3) outcome data. We extracted outcome data from all relevant primary studies for all primary, secondary, adverse effects, and supplemental outcome measures, as specified in the corresponding overviews (Additional file 1). When raw outcome data were reported in SRs, numerical data were extracted from SRs and re-analyzed in Review Manager 5.3 (The Cochrane Collaboration, Copenhagen, Denmark), using standard meta-analysis techniques [23]. Outcome data were expressed using the measures of effect used in the corresponding overviews (risk ratios, odds ratios, and/or risk differences for dichotomous outcomes, and mean differences and/or standardized mean differences for continuous outcomes), with 95% confidence intervals. We conducted all analyses using random effects modeling and the Mantel-Haenszel method (dichotomous data) or inverse variance method (continuous). When meta-analyzed data were reported in SRs but raw study data were not provided, or when only narrative data were provided, the data were extracted and reported based on statistical significance or the SR authors’ description as “significant in favor of intervention,” “not significant,” or “significant in favor of comparator.” For additional methodological decisions unique to each case, we adhered to the decision rules contained within the “Methods” section of the published overviews (though for feasibility, we did not conduct any subgroup or sensitivity analyses). Outcome data from the SRs contained within the procedural sedation overview case were not extracted, because data for the comparator groups were often not available.

We classified all outcome data using published criteria as “favorable” (p ≤ 0.05 in favor of the intervention, or finding described as “significant”), “neutral” (p > 0.05, or finding described as “not different between groups”), or “unfavorable” (p ≤ 0.05 in favor of the comparator, or finding described as “favoring non-intervention comparator”) [24, 25]. We classified outcome data as “unknown” when the effect estimate was not estimable (due to no events in either group) or when the “full inclusion scenario” contained discordant outcome data from multiple overlapping SRs. One reviewer (MP) extracted and analyzed the data, and two additional reviewers (RMF, LH) oversaw this process and provided clinical and methodological input as needed. One reviewer (MP) also documented the challenges encountered when conducting the overviews according to the different inclusion scenarios, and discussed these challenges with two additional reviewers (RMF, LH).

Data analysis

Data analysis consisted of both within-case analyses and cross-case syntheses [13, 26]. For each of the seven overview topics, we used three complimentary methods to visualize and describe the “full inclusion scenario”: (1) we reported characteristics of the included SRs and their primary studies; (2) we generated a citation matrix [5] to show which SRs (columns) included which primary studies (rows), with sample sizes of primary studies reported in relevant cells; and (3) we used the citation matrix to calculate the “corrected covered area” (CCA) [5] to assess the extent of primary study overlap between the SRs included in the overview. The CCA represents “the area [of the citation matrix] that is covered after eliminating the inclusion of all primary studies the first time they are counted” [5]. The formula is, CCA = \( \frac{N-r}{rc-r} \), where N = total number of included primary studies (number of non-empty cells), r = total number of unique primary studies (number of rows), and c = total number of SRs (number of columns). The amount of overlap could range from 0 to 100 and was categorized using published criteria as “slight” (0–5), “moderate” (6–10), “high” (11–15), or “very high” (> 15). Detailed instructions for creating citation matrices and calculating the CCA can be found in Pieper et al. [5].

For each of the six overview topics for which outcome data were extracted, we systematically compared “restricted scenarios 1 to 4” to the full inclusion scenario and documented the extent of data loss and change. We calculated the number and percentage of SRs, intervention comparisons, primary studies, and subjects that were lost in each restricted scenario. For the overviews’ outcome data, we compared the result classifications obtained in each of the four restricted scenarios to those obtained in the full scenario. Each outcome was described as “no change” (the result classification remained the same in both the restricted and full scenarios), “change” (the result classification differed in the restricted compared to the full scenario), or “data lost” (all data for that outcome were lost in the restricted scenario). We then calculated the number and percentage of primary, secondary, adverse effect, and supplemental outcomes that were lost and changed in each restricted scenario. These data were organized into a case-ordered descriptive matrix to permit within-case and cross-case analyses [13, 26].

As is standard with a multiple case study, we aimed to demonstrate replication logic across cases [13]. We summarized the effects of the five inclusion scenarios on the comprehensiveness and results of each overview, examined the patterns and themes that emerged across overviews, identified groups of similar and contrasting overviews, and narratively described these different groups of overviews [13, 26]. We then provided a narrative summary of challenges encountered when making different inclusion decisions in overviews, along with the number of overview topics affected, potential implications, and examples.

Results

Description of overview cases

The seven overviews included in this study contained 6–19 SRs (range 0–7 Cochrane SRs, 2–13 non-Cochrane SRs). The SRs had a median publication year of 2008 (range 1989–2013) and a mean AMSTAR score of 7.0/11 (SD 2.8). Compared to the non-Cochrane SRs included in the overviews, the Cochrane SRs were more recent (2010 vs. 2007) and of higher quality (9.6 vs. 5.7). Of the 30 Cochrane SRs, three were new publications and 27 were updates (median 2 updates; range 2–5 updates). Across the overviews, 43% of the included primary studies appeared in multiple SRs (range 23–53% per overview topic), and 53% and 77% were included in Cochrane and non-Cochrane SRs, respectively. Across the overviews, the study overlap between the SRs ranged from slight (CCA 3.3) to high (CCA 14.9). The characteristics of the SRs included in the seven overviews are presented in Table 1. To maintain consistency with subsequent results tables, the table is organized using the categorization scheme described in the next paragraph.

Table 1 Characteristics of the systematic reviews included in each overview

Effect of different inclusion scenarios on comprehensiveness and results of overviews

When analyzing study results, we identified two distinct groups of overviews that showed similar patterns of outcome data loss and change: overviews for which Cochrane SRs did, and did not, examine all relevant intervention comparisons. All study results are presented according to this grouping. The impact of the different inclusion scenarios on the comprehensiveness and results of overviews is displayed in Table 2, summarized in Table 3, and described below.

Table 2 Impact of the full and restricted inclusion scenarios on the comprehensiveness and results of overviews
Table 3 Summary table: amount of outcome data included in each overview when comparing different inclusion scenarios

Overviews for which Cochrane SRs examined all relevant intervention comparisons

In the bronchiolitis, croup, and gastroenteritis overviews, all relevant intervention comparisons were examined in the Cochrane SRs. For the bronchiolitis and gastroenteritis overviews, though all non-Cochrane SRs overlapped with Cochrane SRs, the non-Cochrane SRs sometimes contributed additional primary studies, outcomes, and/or time points that were not included in the Cochrane SRs. Thus, when restricting to Cochrane SRs only (restricted scenario 1), these additional non-Cochrane data, which contributed to 13% (bronchiolitis) and 31% (gastroenteritis) of all outcomes, were lost. When reintroducing all non-overlapping SRs to the Cochrane SRs (restricted scenario 2), these outcome data remained lost. For the croup overview, the non-Cochrane SRs did not contribute any unique outcome data not already contained within the Cochrane SRs, so data loss was 0%. For all three overviews, the outcome data in restricted scenarios 1 and 2 were the same.

For the bronchiolitis and croup overviews, the Cochrane SRs (restricted scenario 2) were always the most recent SRs (restricted scenario 3) and the highest quality SRs (restricted scenario 4), making restricted scenarios 1–4 the same in terms of comprehensiveness and results. For the gastroenteritis overview, the Cochrane SRs were always the highest quality SRs (restricted scenario 4), making restricted scenarios 1, 2, and 4 the same. Including Cochrane SRs (restricted scenarios 1, 2, and 4) compared to the most recent SRs (restricted scenario 3) led to less data loss and change.

Overviews for which Cochrane SRs did not examine all relevant intervention comparisons

In the acute asthma, acute otitis media, eczema, and procedural sedation overviews, not all relevant intervention comparisons were examined in the Cochrane SRs. For the acute asthma, eczema, and acute otitis media overviews in particular, when restricting to Cochrane SRs only (restricted scenario 1), the non-Cochrane outcome data, which contributed to 28% (acute asthma), 54% (acute otitis media), and 67% (eczema) of all outcomes, were lost. When reintroducing all non-overlapping SRs to the Cochrane SRs (restricted scenario 2), all non-Cochrane data for unique intervention comparisons were recaptured. In restricted scenario 2, data remained lost or changed for 4% (acute asthma) and 39% (acute otitis media, eczema) of outcomes. Thus, the outcome data in restricted scenario 2 were always more comprehensive than those in restricted scenario 1, and data were recaptured for 86% (acute asthma), 28% (acute otitis media), and 42% (eczema) of lost or changed outcomes. We were unable to extract outcome data for the procedural sedation overview because data for the comparator groups were often unavailable. However, no Cochrane SRs were included in this overview. Thus, in restricted scenario 1, all outcome data would have been lost.

For the acute asthma, acute otitis media, and eczema overviews, including Cochrane SRs (restricted scenario 2) compared to the most recent SRs (restricted scenario 3) led to less or the same amount of data loss and change. For the acute asthma overview, the Cochrane SRs (restricted scenario 2) were always the highest quality SRs (restricted scenario 4), making restricted scenarios 2 and 4 the same in terms of comprehensiveness and results. For the eczema and acute otitis media overviews, we were unable to calculate the amount of data loss and change for restricted scenario 4, because SRs were sometimes “tied” for highest quality. Notably, it was always a Cochrane SR and a most recent SR that were tied for highest quality (Additional file 2). For the procedural sedation overview, it is unclear what would have happened in restricted scenarios 2–4.

Challenges related to including overlapping SRs in overviews

When conducting the seven overviews and analyzing their outcome data, we identified and documented challenges related to identifying overlapping SRs (two challenges), making different inclusion decisions in overviews (seven challenges), and extracting and analyzing outcome data from overlapping SRs (three challenges). These challenges, along with potential implications and examples, are presented in Table 4 and described below.

Table 4 Challenges encountered when including overlapping systematic reviews in overviews

Identifying groups of overlapping SRs was challenging when the SRs examined all interventions (as opposed to one specific intervention) for the condition of interest and when the primary studies contained within the SRs had multiple related publications that were referenced differently across SRs. Overview authors may need to closely examine the content of the SRs and their included primary studies to accurately assess the extent and nature of the overlap.

All five decisions for SR inclusion presented their own challenges. When including all Cochrane and non-Cochrane SRs in the overviews, many primary studies that were included in the non-Cochrane SRs were identified by, but excluded from, the Cochrane SRs, for being outside their scope or having methodological deficiencies (median 38%; range 8–100%). When including only Cochrane SRs, clinical input was often required to assess whether or not the Cochrane SRs comprehensively examined all relevant intervention comparisons. When including only one SR per group of overlapping SRs, the following challenges were encountered: Cochrane SRs sometimes overlapped; not all groups of overlapping SRs included a Cochrane SR; some groups of overlapping SRs contained multiple SRs that were “tied” for most recent or highest quality; and reporting of SRs sometimes made it challenging to use recency and quality as inclusion criteria. In all cases, overview authors may need additional decision rules to appropriately address these challenges.

Extracting and analyzing outcome data from overlapping SRs often proved challenging. Overlapping SRs often analyzed or presented the same outcome data in different ways or had discordant results for the same outcomes, potentially due to differences in SRs’ inclusion criteria or methods of analysis. Including overlapping SRs in overviews also meant that we encountered older and/or low-quality non-Cochrane SRs that had gross deficiencies in conduct and/or reporting. These commonly encountered challenges may increase the complexity of the data extraction process and make it difficult for overview authors to extract outcome data in a systematic and transparent way.

Discussion

The current study involved conducting each of seven overviews using five different sets of inclusion criteria, to provide empirical evidence examining the inclusion of overlapping SRs in overviews. This study found that different inclusion decisions led to different amounts of outcome data loss and change across overviews. Specifically, including only Cochrane SRs (i.e., not considering non-Cochrane SRs) often, but not always, led to a loss of outcome data. For groups of overlapping SRs (i.e., when considering both Cochrane and non-Cochrane SRs), selecting the Cochrane SR, as opposed to the most recent or highest quality SR, maximized the amount of outcome data included in the overview. This study also identified challenges associated with identifying, including, and extracting outcome data from overlapping SRs in overviews.

Examining the different inclusion scenarios across overview topics revealed that including only Cochrane SRs, compared to all Cochrane and non-Cochrane SRs, often, but not always, led to a loss of outcome data. The data loss always occurred for one of two reasons. First, when overviews had Cochrane SRs that examined all relevant intervention comparisons, all data loss occurred because the overlapping non-Cochrane SRs contributed additional primary studies, outcomes, and/or time points for existing intervention comparisons. It is unclear whether these additional outcome data are of clinical importance. On the one hand, these results data are lost. In the current study, this led to the complete loss of some outcomes, and changes in the statistical significance of some outcomes. On the other hand, some of these lost outcome data came from primary studies in non-Cochrane SRs that were identified by, but excluded from, the Cochrane SRs, for being outside their scope or having methodological deficiencies. If researchers agree with the inclusion decisions made in the Cochrane SRs, it may not be appropriate to include this subset of excluded primary studies in the overview, especially if these studies are of lower quality or do not increase the certainty of evidence (i.e., GRADE). Second, when overviews had Cochrane SRs that did not examine all relevant intervention comparisons, data loss also occurred because the non-Cochrane SRs contributed outcome data for unique intervention comparisons not examined in any Cochrane SR. These additional data fell within the scope of the overview and were likely of clinical importance, and restricting to only Cochrane SRs led to the exclusion of relevant intervention comparisons from the overview. However, reintroducing all non-overlapping SRs to the Cochrane SRs allowed non-Cochrane data for the non-overlapping intervention comparisons to be recaptured.

Examining the different inclusion scenarios also revealed that for groups of overlapping SRs, selecting the Cochrane SR, as opposed to the most recent or highest quality SR, maximized the amount of outcome data included in the overview. Across overview topics, the Cochrane SRs were sometimes the most recent SRs and were often or always the highest quality SRs. Thus, researchers may often end up selecting Cochrane SRs for inclusion in overviews regardless of which inclusion criteria are used. Further, including Cochrane SRs, even when there are more recent or higher quality non-Cochrane SRs available, may result in more outcome data included in the overview, potentially because Cochrane SRs consistently present raw study-level data in well-reported narrative summaries or meta-analyses. To capture outcome data from groups of overlapping SRs that do not contain a Cochrane SR, researchers may choose to include one of the non-Cochrane SRs. In these cases, our results suggest that selecting the highest quality, as opposed to the most recent, non-Cochrane SR, may minimize data loss.

When conducting the overviews, we often encountered practical challenges related to overlapping SRs. In fact, it was sometimes challenging to simply identify groups of overlapping SRs. We found that overlap can occur at the level of the SRs, but also within SRs at the level of the included primary studies, intervention comparisons, or outcome data. Thus, the issue of “overlap” may be more complex than previously envisioned. There were challenges associated with all five inclusion decisions examined in this study. Notably, many challenges related to extracting and analyzing outcome data from overlapping SRs, especially when multiple SRs contained the same or similar outcome data, and when non-Cochrane SRs were poorly conducted and/or reported. A clear understanding of these challenges, combined with consistent application of appropriate decision rules, can help overview authors successfully manage overlapping SRs in overviews.

The results of this study were used to develop an evidence-based decision tool to help researchers make informed inclusion decisions in overviews. This decision tool is presented and described in a companion paper by Pollock et al. [12]. The tool contains four questions: (1) Do Cochrane SRs likely examine all relevant intervention comparisons?; (2) Do the Cochrane and non-Cochrane SRs overlap?; (3) Do the non-Cochrane SRs overlap with each other; and (4) Are researchers prepared and able to avoid double-counting outcome data from overlapping SRs, by ensuring that each primary study’s outcome data are extracted from overlapping SRs only once? Each yes/no question is answered sequentially, as needed, and the tool is structured so that different answers correspond to appropriate inclusion decisions. Guidance is provided to help researchers answer each question, and the tool provides empirical evidence regarding the impact, advantages, disadvantages, and potential trade-offs of the different inclusion decisions. The decision tool can provide overview authors with the knowledge and means to make informed inclusion decisions in overviews, by helping them determine which inclusion decision may be best suited to their specific situation.

The findings of this study should be considered in light of four methodological considerations. First, this study used one of two standard methods of outcome data analysis that involved extracting and reanalyzing the data from SRs (as opposed to presenting the data exactly as they appear in the SRs) [6]. This was done to avoid double-counting outcome data from multiple overlapping SRs. However, there is currently no evidence regarding whether or to what extent the two methods of outcome data analysis may affect the results of overviews. Second, this study operated under the simplified assumption that within each overview, all outcome data, intervention comparisons, primary studies, and SRs were equally relevant. As judgments about “relevance” would have been difficult to incorporate into the analysis in an objective and systematic way, we weighted all outcome data equally and did not comment on the clinical relevance of the specific data that were lost or changed. For similar reasons, we were also unable to account for potential differences in reporting of SRs (specifically selective outcome reporting) that may have affected the comprehensiveness and results of the overview cases. Third, though we extracted, analyzed, and presented data for a number of potentially relevant variables of interest, we focused our results on the variable that helped explain the different patterns of outcome data loss (i.e., the number of intervention comparisons included in Cochrane and non-Cochrane SRs) [13]. We opted not to discuss the other variables in detail, as they did not contribute to the overall pattern of findings in a cohesive or consistent way. For example, we hypothesized that differences in amounts of primary study overlap may lead to systematic differences in comprehensiveness and results of overviews across various inclusion scenarios, but found that this was not the case. Lastly, the inclusion decisions examined in this study are commonly cited as practical ways to manage overlapping SRs in overviews while avoiding issues related to double-counting outcome data [6, 7, 11]. However, real-life inclusion decisions are not always as straightforward as those examined in this study. For example, there are no accepted criteria for selecting a single SR when two or more SRs are “tied” for highest quality, and researchers may also manage groups of overlapping SRs by choosing to include the “most comprehensive SRs” or the “most relevant SRs.” These subjective decisions were not examined in the current methods study, as they may be operationalized in different ways depending on the author group or overview topic.

As is standard with a multiple case study, we aimed to establish generalizability of our findings by demonstrating replication logic across cases [13]. We described individual cases, looked for patterns across cases, identified similar and contrasting cases, and described groups of similar cases together. Our main study findings remained stable across overviews with a range of characteristics. For example, the overviews included different numbers of SRs (6–19) with various publication dates (1989–2013) and quality scores (1–11 out of 11), had “slight” to “high” primary study overlap between SRs, and had non-Cochrane SRs that contributed 0–100% of outcome data. Achieving replication across multiple cases with different characteristics helps establish robustness of the findings and suggests that the patterns observed within and across cases are coherent, systematically related, and unified [13]. This strengthens the generalizability of the patterns of knowledge gained from the study [13]. However, for coherence, we necessarily used a convenience sample of overviews that posed unique clinical questions within the bounds of certain pre-defined limits. Future research may examine whether or to what extent the observed patterns of findings generalize to a broader group of overviews of healthcare interventions. Findings should not be generalized to overviews that address different clinical questions (e.g., qualitative, diagnostic test accuracy, or prognostic overviews).

Though the number of published SRs will likely keep increasing over time, ongoing developments in the research community may alter the extent and nature of future SR duplication. Core outcome sets for effectiveness trials and for specific health conditions may help offset challenges related to variations in outcome measures across multiple overlapping SRs (www.comet-initiative.org). Perhaps more importantly, prospective systematic review registers—and the increasing expectation for authors to register their protocols—may help reduce the number of overlapping SRs being conducted by different author groups at the same time (www.crd.york.ac.uk/prospero/). These ongoing developments may have the potential to impact methodological decisions relating to inclusion of overlapping SRs in overviews and should be considered going forward.

Conclusions

There is currently limited guidance available for researchers conducting overviews of healthcare interventions. For example, there are challenges and uncertainty regarding the methods that should be used to manage overlapping SRs in overviews. The current study helps address this gap in guidance by contributing empirical evidence examining the impact of different inclusion decisions on the comprehensiveness and results of overviews. Our results highlight practical challenges related to inclusion of overlapping SRs in overviews and show that different inclusion decisions affect the comprehensiveness and results of overviews in different ways. The results were used to develop an evidence-based decision tool to help researchers make transparent and well-informed inclusion decisions in overviews. This decision too, presented and described in Pollock et al. [12], provides practical guidance for overview authors and warrants further evaluation.

Abbreviations

AMSTAR:

A MeaSurement Tool to Assess systematic Reviews

CCA:

Corrected covered area

Overview:

Overview of reviews

SR:

Systematic review

References

  1. Green S, Higgins JPT, Alderson P, Clarke M, Mulrow CD, Oxman AD. Chapter 1: introduction. In: Higgins JPT, Green S, editors. Cochrane handbook for systematic reviews of interventions (version 5.1.0): The Cochrane Collaboration; 2011. http://www.handbook.cochrane.org.

  2. Page MJ, Shamseer L, Altman DG, Tetzlaff J, Sampson M, Tricco AC, et al. Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study. PLoS Med. 2016;13(5):e1002028.

    Article  Google Scholar 

  3. Jadad AR, Cook DJ, Browman GP. A guide to interpreting discordant systematic reviews. Can Med Assoc J. 1997;156(10):1411–6.

    CAS  Google Scholar 

  4. Becker LA, Oxman AD. Chapter 22: overviews of reviews. In: Higgins JPT, Green S, editors. Cochrane handbook for systematic reviews of interventions (version 5.1.0): The Cochrane Collaboration; 2011. http://www.handbook.cochrane.org.

  5. Pieper D, Antoine SL, Mathes T, Neugebauer EA, Eikermann M. Systematic review finds overlapping reviews were not mentioned in every other overview. J Clin Epidemiol. 2014;67(4):368–75.

    Article  Google Scholar 

  6. Pollock M, Fernandes RM, Becker LA, Featherstone R, Hartling L. What guidance is available for researchers conducting overviews of reviews of healthcare interventions? A scoping review and qualitative metasummary. Syst Rev. 2016;5:190.

    Article  Google Scholar 

  7. Ballard M, Montgomery P. Risk of bias in overviews of reviews: a scoping review of methodological guidance and four-item checklist. Res Synth Methods. 2017;8(1):92–108.

    Article  Google Scholar 

  8. Pollock A, Campbell P, Brunton G, Hunt H, Estcourt L. Selecting and implementing overview methods: implications from five exemplar overviews. Syst Rev. 2017;6:145.

    Article  Google Scholar 

  9. Cooper H, Koenka AC. The overview of reviews: unique challenges and opportunities when research syntheses are the principal elements of new integrative scholarship. Am Psychol. 2012;67(6):446–62.

    Article  Google Scholar 

  10. Caird J, Sutcliffe K, Kwan I, Dickson K, Thomas J. Mediating policy-relevant evidence at speed: are systematic reviews of systematic reviews a useful approach? Evid Policy. 2015;11(1):81–97.

    Article  Google Scholar 

  11. Thomson D, Foisy M, Oleszczuk M, Wingert A, Chisholm A, Hartling L. Overview of reviews in child health: evidence synthesis and the knowledge base for a specific population. Evid Based Child Health. 2013;8(1):3–10.

    Article  Google Scholar 

  12. Pollock M, Fernandes RM, Newton AS, Scott SD, Hartling L. A decision tool to help researchers make decisions about including systematic reviews in overviews of reviews of healthcare interventions. Syst Rev. 2019. https://doi.org/10.1186/s13643-018-0768-8.

  13. Yin RK. Case study research: design and methods. 5th ed. Thousand Oaks: SAGE Publications; 2013.

    Google Scholar 

  14. Pollock M, Sinha I, Hartling L, Rowe BH, Schrieber S, Fernandes RM. Inhaled short-acting bronchodilators for managing emergency childhood asthma: an overview of reviews. Allergy. 2017;72(2):183–200.

    Article  CAS  Google Scholar 

  15. Oleszczuk M, Fernandes RM, Thomson D, Shaikh N. The Cochrane Library and acute otitis media in children: an overview of reviews. Evid Based Child Health. 2012;7(2):393–402.

    Article  Google Scholar 

  16. Bialy L, Foisy M, Smith M, Fernandes RM. The Cochrane Library and the treatment of bronchiolitis in children: an overview of reviews. Evid Based Child Health. 2011;6(1):258–75.

    Article  Google Scholar 

  17. Bjornson C, Russell K, Foisy M, Johnson DW. The Cochrane Library and the treatment of croup in children: an overview of reviews. Evid Based Child Health. 2010;5(4):1555–65.

    Article  Google Scholar 

  18. Foisy M, Boyle RJ, Chalmers JR, Simpson EL, Williams HC. The prevention of eczema in infants and children: an overview of Cochrane and non-Cochrane reviews. Evid Based Child Health. 2011;6(5):1322–39.

    Article  Google Scholar 

  19. Freedman SP, Ali S, Oleszczuk M, Gouin S, Hartling L. Treatment of acute gastroenteritis in children: an overview of systematic reviews of interventions commonly used in developed countries. Evid Based Child Health. 2013;8(4):1123–37.

    Article  Google Scholar 

  20. Hartling L, Milne A, Foisy M, Lang E, Sinclair D, Klassen TP, et al. What works and what’s safe in pediatric emergency procedural sedation: an overview of reviews. Acad Emerg Med. 2016;23(5):519–30.

    Article  Google Scholar 

  21. Etikan I, Musa SA, Alkassim RS. Comparison of convenience sampling and purposive sampling. Am J Theor Appl Stat. 2016;5(1):1–4.

    Article  Google Scholar 

  22. Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7:10.

    Article  Google Scholar 

  23. Deeks JJ, HJP T, Altman DG. Chapter 9: analysing data and undertaking meta-analyses. In: Higgins JPT, Green S, editors. Cochrane handbook for systematic reviews of interventions (version 5.1.0): The Cochrane Collaboration; 2011. http://www.handbook.cochrane.org.

  24. Tricco AC, Tetzlaff J, Pham B, Brehaut J, Moher D. Non-Cochrane vs. Cochrane reviews were twice as likely to have positive conclusion statements: cross-sectional study. J Clin Epidemiol. 2009;62(4):380–6.

    Article  Google Scholar 

  25. Lai NM, Teng CL, Lee ML. Interpreting systematic reviews: are we ready to make our own conclusions? A cross-sectional study. BMC Med. 2011;9:30.

    Article  Google Scholar 

  26. Miles MB, Huberman AM, Saldana J. Qualitative data analysis: a methods sourcebook. 3rd ed. Thousand Oaks: SAGE Publications; 2014.

    Google Scholar 

Download references

Acknowledgements

The authors wish to thank the following individuals for assistance with searching, screening, inclusion, and/or AMSTAR assessments: Annabritt Chisholm, Andrea Milne, Melanie Muise, Dunsi Oladele, Marta Oleszczuk, Sanja Schreiber, Jennifer Seida, and Ian Sinha.

Funding

This project was funded in part by a Canadian Institutes of Health Research (CIHR) Operating Grant (Grant Number 119373). MP was supported by an Alberta Innovates Health Solutions Graduate Studentship and a Knowledge Translation Canada Graduate Student Fellowship. LH and ASN were supported by CIHR New Investigator Salary Awards.

Availability of data and materials

The datasets analyzed in the study are available from the corresponding author on request.

Author information

Authors and Affiliations

Authors

Contributions

MP and LH conceived of the study. MP, RMF, SDS, and LH designed the study. MP, RMF, and LH conducted the study. MP, RMF, ASN, and LH interpreted the results of the study. MP drafted the manuscript, and all authors provided feedback and approved the final manuscript.

Corresponding author

Correspondence to Lisa Hartling.

Ethics declarations

Ethics approval and consent to participate

Not applicable (the study did not involve human participants).

Consent for publication

Not applicable (the study did not involve human participants).

Competing interests

Together, MP (former surname: Foisy), RMF, and LH are authors on the seven overviews, and three of the systematic reviews, included in this study.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional files

Additional file 1:

Inclusion criteria used in each overview, stratified by overview topic. (DOCX 17 kb)

Additional file 2:

Comparisons and systematic reviews included in different inclusion scenarios, for each overview topic. (DOCX 54 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pollock, M., Fernandes, R.M., Newton, A.S. et al. The impact of different inclusion decisions on the comprehensiveness and complexity of overviews of reviews of healthcare interventions. Syst Rev 8, 18 (2019). https://doi.org/10.1186/s13643-018-0914-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s13643-018-0914-3

Keywords