Methodological areas: assessment of areas of need
Locating existing systematic reviews
Available guidance
Several organizations present guidance on locating existing systematic reviews, including the EPC program and the Cochrane Collaboration (see Table 3). These groups recommend using specific databases and search filters to aid in locating existing systematic reviews. Commonly recommended databases include: Database of Abstracts of Reviews of Effects (DARE), Cochrane Database of Systematic Reviews (CDSR), Health Technology Assessment Database, MEDLINE and Embase. Some organizations promote limiting searches for existing systematic reviews to selected sources (for example, CDSR), with the idea that these systematic reviews would be expected to meet sufficient quality standards. We identified no criteria for selecting systematic reviews.
Evidence-based practice center discussions
EPC members expressed concerns about locating reviews based on very limited sources, for example, only searching for EPC or Cochrane reviews. Some EPC members were uncomfortable with any type of selective search instead of doing a broader, if not comprehensive, search for existing systematic reviews. However, other EPC members felt it could be appropriate to selectively use one or two earlier reviews without having to review all of the available prior reviews, pointing out that the scientific rationale and purpose in conducting a systematic search for existing systematic reviews is different than when searching for primary studies. Currently, there is a lack of consensus and limited guidance concerning how to adequately locate and transparently select and use only a subset of reviews.
Assessment
Current EPC guidance on locating existing systematic reviews states that EPCs should conduct a targeted search of a higher yield database, which includes output from the EPC program, MEDLINE’s Top 120 Index Medicus Journals, Health Technology Assessments, CDSR and DARE. EPC program guidance suggests that identifying existing systematic reviews may be done separately or be completed as part of the broad search for primary literature to answer key questions. EPC members shared concern about the extensive effort that may be required to search for and locate all reviews and assess their quality. Instead, given the purpose of locating existing systematic reviews, the ideal search might locate only highly relevant, well-done, very recent, existing systematic reviews that are more likely to allow the current reviewer to leverage the prior work. Some members proposed that future guidance further limit the search for earlier reviews to sources that may have greater likelihood of identifying higher quality and better reported reviews, such as the EPC and Cochrane databases. This would also, theoretically, negate the need to determine if a report is a systematic review.
Assessing relevance
Available guidance
Several organizations present guidance on assessing the relevance of existing systematic reviews. European Collaboration for Health Technology Assessment (ECHTA) and Danish Centre for Health Technology Assessment (DACEHTA) provide general guidance for assessing the relevance of all literature, but no guidance specific to existing reviews. Cochrane provides guidance for assessing the relevance of existing systematic reviews for use in an overview of reviews and specifically recommends that reviewers consider if a review is up-to-date (up-to-date is not defined in this context, but for Cochrane reviews is considered as a review with a search conducted within last 2 years). The EPC program provides EPCs with some guidance for assessing whether an existing review can answer key questions or sub-questions of a review. The EPC program recommends consideration of how recently the review’s primary literature search ended and how well the PICOTS-SD elements are reported and match with the current review, noting that partially relevant reviews may be useful for background or checking references. As described in Table 3, specific steps recommended in EPC guidance include conducting an initial screening for relevance while considering the timeliness of the review’s literature search. In the second stage of screening, the review’s PICOTS-SD elements should be compared to those in the new review protocol for relevance. If these elements are poorly reported, the new reviewers should not consider incorporating the existing review. If a review is outdated, but otherwise on point, an update of the search could be done.
Evidence-based practice center discussions
EPC members raised concerns about deciding which existing reviews, if any, to use in a new review, especially in situations when many reviews of varying quality or scope are available. It can often be difficult to find reviews that match all elements of question. Still, existing reviews may be sufficiently similar to consider using, although criteria for all situations are not easy to define. Thus, determining which reviews are ‘close enough’ is inherently subjective and can be time consuming.
Assessment
Guidance exists for assessing the relevance of existing reviews for conducting overviews of reviews and for integrating existing reviews into a new review. Even so, EPC members expressed concerns about selecting which reviews to use when many are available. They reported rarely finding reviews that match directly all elements of questions. A clearer understanding of the factors (such as clinical area, review purpose, volume of literature, type of review) that have been correlated with successful (and unsuccessful) attempts to incorporate prior systematic reviews into new reviews would be helpful.
Assessing review quality
Available guidance
Guidance is available on assessing the overall quality of systematic reviews (see Table 3). ECHTA and DACEHTA provide general guidance for assessing the quality of all literature, but do not specifically address assessing the quality of systematic reviews. In contrast, Institute for Quality and Efficiency in Health Care (IQWiG) and York Centre for Reviews and Dissemination (CRD) present guidance on assessing the quality of systematic reviews and recommend the use of a quality rating tool or checklist such as Oxman and Guyatt’s quality index or the AMSTAR Instrument [4, 5]. NICE suggests using existing reviews as evidence in developing a new guideline if they are up-to-date, have been assessed as high quality using the NICE methodology checklist for systematic reviews [15], and include the accompanying evidence statements and evidence tables. EPC guidance recommends using AMSTAR as a starting point for quality assessment and supplementing as deemed appropriate for specific reviews. The Cochrane Collaboration is currently developing a new tool (ROBIS) to assess risk of bias in systematic reviews. In addition, the IOM report Finding What Works in Health Care: Standards for Systematic Reviews may provide the basis for the development of instruments to assess quality of systematic reviews [11].
Evidence-based practice center discussions
EPC members raised concerns about using AMSTAR to rate the quality of reviews, specifically, as to how accurate AMSTAR is in differentiating between poorer quality and better quality reviews. This is important since EPCs generally agreed that evidence from existing reviews should only be used if the review can be relied upon to substitute for the methods for the conduct of reviews espoused by the EPC program. A suggestion was raised for a minimum standard of quality to be set for earlier reviews that are included as evidence in new reviews.
Assessment
Guidance is available for quality rating of reviews with available tools and checklists (for example, Oxman and Guyatt, AMSTAR, and National Institute for Health and Care Excellence (NICE)) [5, 15, 16]. However, EPC members noted that these quality assessment tools have limitations. Further, it is important to remember that these instruments do not assess the quality of primary evidence, which must also be taken into consideration. More explicit consideration of the quality rating approaches available for systematic reviews, and empiric evidence to support these, would help in the consideration and selection of existing systematic reviews for use in new reviews.
Determining appropriate use for relevant existing systematic reviews: scanning references
Available guidance
Cochrane, the EPC program, and CRD provide guidance on scanning the reference lists of existing systematic reviews (see Table 3). These organizations suggest scanning the reference lists of existing reviews as a supplementary method to find relevant studies for a new systematic review. Prior EPC program guidance suggested cross-checking primary studies from one or more existing systematic reviews to confirm adequacy of primary searches or to help select the most comprehensive existing review for incorporation.
Evidence-based Practice Center Discussions
Overall, most EPC members had experience using existing reviews for scanning references for relevant articles. No major concerns over the use of existing reviews in this manner were presented.
Assessment
Some guidance is available for using existing reviews as a potential source of relevant studies for a new review. As noted below, incorporating search results from a prior systematic review to substitute for some or all of the new review’s search efforts is different from simply scanning references as an additional source of information to support other efforts. More commonly, the latter occurs (that is, reference lists from existing reviews are used to augment or check comprehensiveness of the search yield used in the current review). In general, EPC members felt comfortable using existing reviews as a way to identify possible additional articles for a review. Thus, this does not appear to be an area where future guidance is needed except for the general methodological question of when this type of hand-searching has reached saturation (that is, when no further relevant studies are being identified and searching may stop).
Determining use: search strategy and results of existing searches
Available guidance
Cochrane and the EPC program provide guidance on using the search strategy from existing reviews (see Table 3. The EPC program recommends using part or all of the search strategy from an existing review, particularly in the case of updating. The EPC program also indicates when it is acceptable to incorporate results of searches from existing systematic reviews to locate all primary studies - that is, when the search methods (for example, search strategy, inclusion/exclusion criteria, literature screening methods) are consistent with EPC program methods for finding evidence. The EPC program recommends supplementing primary studies from the existing systematic review by performing bridge searches whenever the search date for the existing review ended more than 1 year earlier than searches for the new review are being conducted [2]. Data from primary studies gathered through these combined search efforts are then abstracted, followed by a new analysis and synthesis of data. Cochrane recommends utilizing information about search strategies from existing reviews and guidelines when possible in a new review, but does not address using search results in entirety.
Evidence-based practice center discussions
EPC members reported sometimes using the search strategy from an existing review, particularly when search strategies are reviewed and approved as robust by their medical librarians. EPCs also mentioned that, once they have determined it is methodologically appropriate to incorporate an existing review’s search results, they go beyond simply including the primary studies from the existing search. Instead, they check the primary studies against their inclusion/exclusion criteria, and some go further by also checking all studies from the excluded studies table in the previous systematic review, particularly in cases where there are slight differences in the questions or eligibility criteria.
Assessment
Some guidance is available on using the search strategy and search results from existing reviews. The EPC program provides some guidance for when to trust a search strategy from a prior review. Given the developing practice of requesting peer review for their search strategies, medical librarian review prior to incorporating an existing search strategy could be warranted [2]. Existing guidance related to when and how to incorporate the results of previous search strategies is relatively robust in the current EPC guidance; incorporation of existing search results is also the most common way EPCs report incorporating existing systematic reviews into their work. Further guidance should be based on empiric evidence to establish whether any additional value is added by 1) considering included studies from more than one existing systematic review and 2) considering excluded studies from the previous systematic review upon which the search results are being based.
Determining use: data abstraction
Available guidance
Cochrane, NICE, and the EPC program provide guidance on using the data abstraction from existing reviews (see Table 3). When using existing reviews in a new guideline, NICE recommends creating new evidence summaries but using and directly referencing the original evidence tables from the prior review. For an overview of reviews, Cochrane guidance recommends using the abstracted data presented in the existing reviews and seeking additional information from the authors of systematic reviews or from primary study authors only if necessary. Though there is ongoing work in this area such as the Systematic Review Data Repository (SRDR) project, which will likely affect how abstracted data are used in the future, current guidance from the EPC program suggests using data abstraction tables from an existing review only if the methods used are deemed to be high quality and consistent with the EPC program methods for abstracting evidence.
Evidence-based practice center discussions
Most EPC members had concerns over whether or not data abstracted from an existing review are trustworthy, comprehensive, or accurate. Additionally, it was often reported that people tend to have to go back and abstract at least some data themselves from the primary studies due to slightly different outcomes, missing data or improper abstraction. This is particularly true when access to full evidence tables is limited, as can be the case with some peer-reviewed articles. For any data abstraction that is incorporated, spot-checking for quality assurance was recommended, though there was no consensus on the minimum level required.
Assessment
Some guidance is available for using the data abstraction from existing reviews. However, there are still issues to be resolved when considering using abstracted data from an existing review. Additionally, ongoing work such as the SRDR project could obviate the need for separate guidance if pre-abstracted, high-quality data become available for many reviews. At the least, tools like SRDR will result in the need for updated guidance which incorporates such new data resources for reviewers. Guidance should specifically address various possible data abstraction scenarios. One such example is guidance related to including all versus only some of the abstracted data from an existing review. Because of different review scopes, PICOTS-SD, quality criteria of included studies, etcetera, systematic reviewers may determine that they will only need to use a portion of the data included in an existing review. Conversely, systematic reviewers may need to add to existing abstracted data either by abstracting additional data from previously included studies or by adding entirely new studies. Finally, as noted in the EPC discussion themes, systematic reviewers are often concerned about the trustworthiness or accuracy of abstracted data. Therefore, guidance is needed to establish consistent, reliable methods for conducting data checking or confirming the abstracted data to assure accuracy.
Determining use: study-level risk of bias assessments
Available guidance
Guidance is available on assessing the risk of bias of individual studies, though little guidance specifically addresses how to incorporate this information from existing reviews into a new systematic review. AS noted in Table 3, Cochrane recommends conducting quality assessments of evidence for all outcomes addressed in an overview of reviews and evaluating the quality assessment judgments from existing reviews to ensure consistency among included reviews. If an existing review does not include a quality assessment, Cochrane recommends performing a quality assessment of the primary studies.
Evidence-based practice center discussions
EPC members noted that quality assessment of a systematic review to determine appropriateness for inclusion in a new systematic review is not sufficient to represent the quality of individual studies included in the existing review. When a review is incorporated as more than a source of primary studies, it raises a number of issues about how to proceed in terms of quality rating. EPC members expressed strong concerns about relying on the risk of bias assessments from prior reviews; in particular, risk of bias assessments are known to be poorly reproducible, and often lack transparency and rationale for ratings. No guidance is available on how to integrate information from reviews that do not have risk of bias assessments, or used risk of bias assessment tools different from those used in the new review. Overwhelmingly, EPC members felt that the primary studies included in an earlier review needed to be quality rated in some way that is sufficiently consistent with that applied to newly identified primary studies included in the new review. Even if existing reviews include risk of bias assessments, an overarching theme from the discussion with EPC members was that it may be difficult for reviewers to trust the reliability of other reviewers’ risk of bias assessments. Some EPC members felt that you may be able to trust particular sources (for example, Cochrane or EPCs) using particular risk of bias assessment tools.
Assessment
Guidance is readily available for assessing the risk of bias of primary studies with currently available tools and checklists (for example, Cochrane, EPC program) [2, 8, 17, 18]. However, the lack of guidance about when to accept or when to repeat such assessments when conducted by others as part of prior reviews was of particular concern to EPCs. Other issues of concern for EPCs include how to handle risk of bias assessment from prior reviews that are based on different tools from those used in a new review. Lack of clear guidance in this area is a barrier since the possible need to assess risk of bias for all primary studies introduces resource demands that reduce the likelihood of efficiency gains from incorporating existing systematic reviews.
Determining use: synthesis, including strength of evidence assessment
Available guidance
Cochrane and the EPC program provide some guidance on using the synthesis from existing systematic reviews (see Table 3). Cochrane provides guidance on using prior synthesis in an update of a review and in an overview of reviews. In an update, Cochrane recommends rerunning the quantitative synthesis if data from new studies are found. In an overview of reviews, Cochrane recommends relying on the previous analysis as much as possible and only reanalyzing data if necessary (for example, if different populations or subgroups are analyzed in different reviews). The EPC program recommends incorporating an existing review in its entirety, including evidence summaries, only if it is high quality, the key questions are very similar to the new review’s questions, and the methods used are consistent with EPC program methods. However, current EPC program guidance does not explicitly address methods for incorporating prior syntheses. The EPC program does address dealing with discordant results from existing reviews and recommends making an effort to determine reasons for the disagreements both as part of the new review planning and in discussion of its findings. At the review planning stage, uncertainty of which discrepant results to trust can provide a strong rationale for conducting a new review. No current guidance is available on performing strength of evidence assessments for systematic reviews, particularly as one component of a larger body of evidence answering a key question.
Evidence-based practice center discussions
Most EPC members had concerns over how and if you can trust a prior review and whether or not you can trust the data synthesis. The lack of guidance available for assessing strength of evidence when integrating an existing systematic review into a new review poses particular challenges. Despite current guidance, EPC members had additional concerns about how to deal with discordant results from reviews.
Assessment
Limited guidance is available on using the synthesis from existing reviews and this is an area that EPC members struggle with when considering using existing reviews. EPCs particularly struggle when asked to assess overall strength of evidence for mixed bodies of evidence (that is, those representing both primary studies and prior systematic reviews). The difficulty is compounded by the need for different guidance, depending on the circumstances surrounding the prior review(s) and the current review. We identified four scenarios that might require different guidance as to how the synthesis may be used from an existing review: 1) up-to-date and has a meta-analysis; 2) up-to-date and presents a narrative summary; 3) not up-to-date and has a meta-analysis; and 4) not up-to-date and presents a narrative summary. These scenarios obviously apply only when one or more existing reviews are being included in a new review based on meeting certain criteria (for example, scope of key questions, PICOTS, and quality).
Methods and results reporting
Available guidance
IQWiG, NICE, Cochrane and the EPC program present guidance on reporting results when utilizing existing reviews (see Table 3). The Annals of Internal Medicine has also published methodological reporting requirements for authors [19]. For a benefit assessment based on systematic reviews, IQWiG recommends summarizing the results from existing reviews in tables and noting any differences in results between reviews. In a guideline document using existing reviews, NICE recommends referencing evidence tables from existing guidelines with a direct link to the source. Cochrane provides guidance on how to present information in an update of a review, including how to highlight information from new studies. When integrating existing reviews, the EPC program recommends including a summary table of included systematic reviews, including an assessment of the overlap in primary studies between reviews. Additionally, the EPC program recommends justifying the use of existing reviews in a new review and commenting on advantages and disadvantages of using existing reviews in the discussion session. In an update of a review, the EPC program recommends explicitly showing what has changed from the previous report.
Evidence-based practice center discussions
EPC members had concerns about how to show evidence from reviews, including how to highlight the primary evidence when using a prior review. Challenges have surfaced when users of EPC reports are unable to identify specific details about primary studies because they have been included as a part of a systematic review instead of highlighted individually. Some EPC members have presented varying levels of detail based on how they used existing reviews but had no guidance on the best approaches. Other EPC members have presented existing systematic reviews in tables and pulled out elements for discussion, even when the existing reviews are not otherwise integrated into a report.
Assessment
While some guidance is available for how to report the methods to locate and decide how to use an existing review, issues still may arise when reporting results from existing reviews. Specifically, it can be difficult to highlight primary studies or isolate results from individual studies when integrating results from existing reviews. It is also important to not double count studies, such as through inclusion of primary studies also included in existing systematic reviews or by considering as separate evidence systematic reviews with overlapping primary studies. Guidance is needed on the level of detail needed on all aspects, such as how new data were incorporated into evidence tables. Future examination of various approaches recommended by different groups and development of a set of templates with worked examples could be very useful.