Guidance on review type selection for health technology assessments: key factors and considerations for deciding when to conduct a de novo systematic review, an update of a systematic review, or an overview of systematic reviews

Abstract

Background

A systematic review (SR) helps us make sense of a body of research while minimizing bias and is routinely conducted to evaluate intervention effects in a health technology assessment (HTA). In addition to the traditional de novo SR, which combines the results of multiple primary studies, there are alternative review types that use systematic methods and leverage existing SRs, namely updates of SRs and overviews of SRs. This paper shares guidance that can be used to select the most appropriate review type to conduct when evaluating intervention effects in an HTA, with a goal to leverage existing SRs and reduce research waste where possible.

Process

We identified key factors and considerations that can inform the process of deciding to conduct one review type over the others to answer a research question and organized them into guidance comprising a summary and a corresponding flowchart. This work consisted of three steps. First, a guidance document was drafted by methodologists from two Canadian HTA agencies based on their experience. Next, the draft guidance was supplemented with a literature review. Lastly, broader feedback from HTA researchers across Canada was sought and incorporated into the final guidance.

Insights

Nine key factors and six considerations were identified to help reviewers select the most appropriate review type to conduct. These fell into one of two categories: the evidentiary needs of the planned review (i.e., to understand the scope, objective, and analytic approach required for the review) and the state of the existing literature (i.e., to know the available literature in terms of its relevance, quality, comprehensiveness, currency, and findings). The accompanying flowchart, which can be used as a decision tool, demonstrates the interdependency between many of the key factors and considerations and aims to balance the potential benefits and challenges of leveraging existing SRs instead of primary study reports.

Conclusions

Selecting the most appropriate review type to conduct when evaluating intervention effects in an HTA requires a myriad of factors to be considered. We hope this guidance adds clarity to the many competing considerations when deciding which review type to conduct and facilitates that decision-making process.

Background

A systematic review (SR) helps us make sense of a body of research, while minimizing bias, by identifying, analysing, appraising, synthesizing, and interpreting research findings using a standardized, pre-defined method and an a priori protocol [1]. In the context of health care, the SR methodology provides an essential tool for summarizing the available published knowledge around a specific research question to support organizations and professionals who make recommendations and decisions about wide-ranging issues [2, 3]. In health technology assessments (HTAs), which use explicit methods to determine the value of a health technology at different points in its lifecycle [4], SRs are routinely conducted to answer research questions around the effectiveness and safety of drugs, medical devices, and clinical interventions and provide clinical evidence to support their appropriate use. This paper focuses on such use of SRs of quantitative research in HTAs, that is, in the evaluation of interventions to prevent, diagnose, or treat diseases or conditions.

The traditional de novo SR combines the results of multiple primary studies to answer a specific research question [5, 6]. There are, however, two alternative review types that use systematic methods and leverage existing SRs, namely an update of an SR and an overview of SRs. An update of an SR incorporates new evidence (by extending the source- or time-related domains of its search, including additional databases or longer periods), new methods, or new analyses into a previously completed SR [6,7,8]. An overview of SRs—also known as a review of reviews, an umbrella review, a meta-review, or other variations of these terms—compiles data from multiple SRs to provide a single summary of relevant evidence [5, 9]. These three review types—de novo SRs, updates of existing SRs, and overviews of existing SRs—are most often used to evaluate intervention effects and form the basis of clinical evidence in HTAs. As such, other review types (e.g., scoping reviews, integrative reviews, and realist reviews) [10, 11] were out of scope for this paper. Rapid reviews [12] and living reviews [13] were also out of scope for this paper, as these approaches can be used with any review type (e.g., an overview may be conducted rapidly with a single reviewer and updated regularly and frequently as a living document).

Definitions

de novo SR: uses systematic and explicit methods* to combine the results of multiple primary studies to answer a specific research question [5, 6].

Update of an SR: uses systematic and explicit methods* to incorporate new evidence, new methods, or new analyses into a previously completed SR [6,7,8].

Overview of SRs: uses systematic and explicit methods* to compile data from multiple SRs to provide a single summary of relevant evidence [5, 9].

*Systematic and explicit methods usually involve: publishing a priori protocols that are detailed enough to be reproducible by others; searching multiple databases and grey literature to identify as much of the available relevant data as possible; and engaging two reviewers for each step during the review process to minimize bias.

Given the exponential growth in the number of published SRs [14, 15] and high-resource requirements for completing new SRs [15, 16], conducting updates or overviews of existing SRs, instead of de novo SRs, may be considered by reviewers under the right circumstances. Using existing SRs in updates or overviews can help minimize duplication of effort and redundancy in research and can contribute to a more efficient use of resources and a further synthesis of the extant literature [17]. However, there are unique challenges associated with conducting updates or overviews of existing SRs, arising from the need to rely on the work of the SR authors, which can be resource-intensive to address [15]. In other words, there are trade-offs. Therefore, potential benefits and challenges of leveraging existing SRs instead of primary study reports must both be considered. While there is some guidance on how to select a review type to conduct [2, 15, 18,19,20], we are not aware of guidance that incorporates the myriad of factors that we have come to consider when making that decision in our work.

The objective of this paper is to share our guidance that is based on our work. It can be used when selecting the most appropriate review type that uses systematic methods to answer a research question about intervention effects in an HTA, while ensuring an appropriate balance between evidentiary needs and methodological rigour. The specific goals were as follows: (1) to identify factors that should be considered when deciding to conduct a de novo SR, an update of an existing SR, or an overview of existing SRs, and (2) to develop an algorithm to help with that decision-making process. The intent of this paper is not to provide a definitive answer on which review type to conduct in a specific scenario or to offer guidance on how to conduct a review using a specific review type. Rather, this paper identifies key factors that should be considered and organizes them into a summary and a flowchart that can be used as a decision tool to identify the right circumstances in which one review type over the others may be favoured or ruled out.

Process

This work consisted of three steps. First, a guidance document was drafted by methodologists from two Canadian HTA agencies based on their experience. Next, the draft guidance was supplemented with a literature review. Lastly, broader feedback from HTA researchers across Canada was sought and incorporated into the final guidance.

Step 1. Draft guidance

The Canadian Agency for Drugs and Technologies in Health (CADTH) and the Institute of Health Economics (IHE) have conducted both updates and overviews of existing SRs, instead of de novo SRs, to answer clinical research questions in several HTAs (e.g., see references [21,22,23,24,25]). Methodologists at CADTH (JK, LW) and IHE (MP), who had been involved in deciding which review types to conduct for the abovementioned projects, reflected on the key factors and considerations that informed, or could have informed, those decisions and organized them into a guidance document that comprised a summary and a corresponding flowchart outlining the decision-making process. The guidance was originally drafted by the methodologists at CADTH and then shared with the methodologist at IHE for review and input. This collaboration across the two HTA organizations helped ensure that the guidance was both methodologically robust and applicable across different decision-making contexts. Thus, the draft guidance reflected the experiences and expertise of two Canadian HTA agencies (i.e., CADTH and IHE).

Step 2. Literature review

A literature review was conducted to identify any existing related guidance from academia, SR organizations, and other HTA agencies.

Search strategy

A targeted literature search was performed by an information specialist at CADTH. Published literature was identified by searching the MEDLINE database via Ovid. The search strategy was comprised of both controlled vocabulary, such as the National Library of Medicine’s MeSH (Medical Subject Headings), and keywords. Retrieval was not limited by publication date but was limited to the English language. The search was completed on March 5, 2019. Regular alerts updated the search until manuscript submission. The complete search strategy is presented in Appendix 1.

Grey literature (literature that is not commercially published) was identified by searching sources listed in relevant sections of the Grey Matters checklist (https://cadth.ca/grey-matters-practical-tool-searching-health-related-grey-literature) [26], primarily the websites of Canadian and major international HTA agencies. Google was used to search for additional web-based materials. These searches were supplemented by reviewing bibliographies of key papers. More information on the grey literature search strategy is presented in Appendix 2.

Screening and selection

Literature identified through the database and grey literature searches were screened by a single reviewer (JK) at CADTH. Guidance for deciding when to conduct a de novo SR, an update of an existing SR, or an overview of existing SRs to evaluate intervention effects was in scope. Further, the following eligibility criteria were applied:

• Documents of the following nature were in scope and included:

  • ○ Provide explicit guidance for selecting one review type over another

  • ○ Describe when it is or is not appropriate to use a specific review type

  • ○ Help identify, define, or set thresholds for factors and considerations important for deciding to conduct a specific review type (e.g., how to judge relevance, quality, or currency of an existing SR)

• Documents of the following nature were not in scope and excluded:

  • ○ Papers on review types that are not intended to answer research questions about intervention effects (e.g., scoping reviews, integrative reviews, or realist reviews)

  • ○ Papers on methodological approaches (e.g., rapid reviews or living reviews) that can be used with any review type

  • ○ Primary studies, SRs, or clinical practice guidelines on specific topics and their protocols

  • ○ Guidance for conducting a review using a specific review type

  • ○ Guidance for conducting SRs in specific fields (e.g., anatomy or nursing)

  • ○ Guidance for integrating quantitative and qualitative evidence (e.g., mixed methods or for knowledge translation)

  • ○ Anecdotal, editorial, or opinion-based literature

Data extraction and analysis

Text that provided explicit guidance for selecting one review type over another, described when it is or is not appropriate to use a specific review type, or helped identify, define, or set thresholds for factors and considerations important for deciding to conduct a specific review type was deemed relevant. Excerpts of relevant text from the included documents were extracted and organized by a single reviewer (JK) at CADTH according to the factors and considerations identified in Step 1. All excerpts of relevant text are presented in Appendix 3.

Synthesis

The excerpts of relevant text in Appendix 3 were used to confirm, support, complement, or refine the draft guidance from Step 1. Details and citations from the literature review were incorporated into the guidance to help ensure that our guidance agreed with existing guidance from the literature. The updated guidance was then shared a second time with the methodologist at IHE (MP) for review and input.

Step 3. Broader discussion and feedback

The updated guidance from Step 2, which incorporated CADTH’s and IHE’s experiences as well as existing guidance from the literature, was shared more broadly with HTA researchers across Canada who have experience with deciding which review types to conduct and with conducting clinical reviews in HTAs. Input was collected from an additional nine researchers at CADTH and IHE as well as from 13 researchers at the Institut national d'excellence en santé et en services sociaux (INESSS) and the National Advisory Committee on Immunization (NACI). This input was collected through written feedback and/or small group discussions and further incorporated into the updated guidance from Step 2. Thus, the final guidance reflected the experiences and expertise of several Canadian HTA agencies (i.e., CADTH, IHE, INESSS, and NACI). All authors reviewed and approved the final version of the guidance, which appears in this paper.

Insights

This section has two parts. The first part identifies key factors and considerations that can inform the decision to select the most appropriate review type to answer a research question about intervention effects in an HTA. The second part organizes those key factors and considerations into a summary and a flowchart that can guide the decision-making process.

Part 1. Key factors and considerations

In total, nine key factors and six considerations that are important for selecting a specific review type to conduct for evaluating intervention effects were identified through reflections on experiences with clinical reviews in past HTA projects at CADTH and IHE. The literature review found these factors and considerations to be in good agreement with existing guidance from academia, SR organizations, and other HTA agencies. In total, 19 papers were identified as relevant through the literature review. They were from Joanna Briggs Institute [27], the Cochrane Collaboration [6, 9], the Agency for Healthcare Research and Quality [15, 18, 19, 28], the Belgian Health Care Knowledge Centre [29], the World Health Organization [30], eight different HTA agencies [31], the Centre for Reviews and Dissemination at the University of York [3], and other research groups [2, 8, 10, 17, 32,33,34,35] (see Appendix 3). The feedback received from other HTA researchers across Canada also found these factors and considerations to be representative of the experiences of different organizations and applicable across different decision-making contexts.

The nine key factors and six considerations each fell into one of two categories: the evidentiary needs of the planned review and the state of the existing literature. The key factors and considerations are presented in Table 1 and described below, with supporting citations. Of note, not all three review types are discussed or discussed to the same extent under each factor, as not all factors and considerations are relevant or deterministic in favouring or ruling out each review type. Also, many of the key factors and considerations are interdependent, as the evidentiary needs of a planned review must be balanced with the state of the existing literature. Therefore, many of the factors and considerations are not sufficient on their own to help select one review type over another, which motivated us to organize them into a summary and a corresponding flowchart in the following section. Further, we kept in mind the unique challenges that overviews present, stemming from having to rely on SRs, instead of primary studies, as the unit of analysis, and therefore possibly requiring more resources, compared to SRs [15, 19], and tried to balance them with the potential benefits obtained (e.g., summarizing the existing literature at higher levels or avoiding or minimizing duplication or redundancy in research).

Table 1 Key factors and considerations for selecting to conduct one review type over another

Reviewers who plan to use these key factors and considerations to help select the most appropriate review type for their HTA will need an understanding of both the evidentiary needs of the planned review and the state of the existing literature. Specifically, there must be a clear research question with a well-defined scope. Further, a scoping exercise should be undertaken to gain a general idea of the quantity, quality, and other characteristics of relevant SRs and primary studies that are available. While it may be difficult to obtain a comprehensive understanding of the literature through the scoping exercise, and reviewers will likely work with incomplete information, the guidance provided in this paper is intended to help reviewers think through, and plan for, the issues that they are likely to encounter when deciding which review type to conduct. To that end, reviewers may find it difficult to answer the questions provided below with certainty (e.g., instead of “yes” or “no,” “likely yes,” “likely no,” or even “unclear”), especially when the existing literature is vast or complex. Nevertheless, the scoping exercise could be viewed as an “upfront” investment in the conduct of an HTA that need not be wasted (e.g., information learned through the scoping exercise can be “reused” during the review process) and may end up saving resources (e.g., by identifying an existing SR and eliminating the need for a new SR) or achieving other meaningful goals (e.g., summarizing the existing literature at higher levels or avoiding or minimizing duplication or redundancy in research).

• I. Evidentiary needs of the planned review

  • 1. Review scope: Is the scope of the research question broad?

    A de novo SR and an update of an SR are identical or similar in scope [8], which tends to be focused and narrow [2, 34]. An overview of SRs, on the other hand, tends to have a broader scope [2, 3, 10, 32, 33, 35]. When expressed in terms of the Population, Intervention, Comparator, and Outcome (PICO) framework, a de novo SR or an update of an SR tends to have narrowly defined PICO elements, while an overview tends to have expanded upon one of the PICO elements. As described by the Cochrane Collaboration, overviews are appropriate for addressing research questions relating to:

    • The same intervention used for different conditions or populations (i.e., broader P);

    • Different interventions used for the same condition or population (i.e., broader I);

    • Different approaches to the application of the same intervention used for the same condition or population (i.e., broader I);

    • The same intervention used for the same condition or population but for different outcomes or time points (i.e., broader O); or

    • Adverse effects of an intervention for one or more conditions or populations (i.e., broader O or P) [27, 32, 34].

    In other words, for an overview, there are two or more relevant SRs available that each covers one “part” of a broad research question, with all SRs combined resulting in perfect or almost perfect coverage [2, 9, 10, 27, 32,33,34,35].

  Consideration I: If the planned review has narrowly defined PICO elements, a de novo SR or an update of an SR may be the most appropriate review type. If the planned review has a broader scope that expands upon one of the PICO elements, an overview of SRs may be the most appropriate review type.

  • 2. Review objective and analytic approach: Is quantitative combination of findings needed to provide a summary measure or to rank interventions?

    A de novo SR, an update of an SR, and an overview of SRs all intend to answer a specific research question [5], using explicit, systematic methods [34]. However, the unit of searching, inclusion, data extraction, analysis, and synthesis varies from the primary study—for a de novo SR and an update—to the SR—for an overview.

    A de novo SR or an update of an SR—having access to primary study-level data—is best suited for quantitative combination of findings through meta-analyses (MAs), indirect treatment comparisons (ITCs), or network meta-analyses (NMAs) to provide a summary measure or answer a question about which intervention works best or is the safest.

    With the SR as the unit of analysis, there are limitations in the analytic capabilities of an overview, and hence the research objectives it can fulfill. In an overview, quantitative combination of findings is more difficult, compared to narrative incorporation of findings, without going back to the primary study reports [15, 27]. This is especially true if relevant SRs are of low quality in methodology or reporting (leading to errors or missing data) or if they overlap significantly in their included primary studies (giving too much statistical power to certain studies) [15, 27]. Further, unless reviewers have thoroughly examined the transitivity assumption (i.e., that the studies making different direct comparisons are sufficiently similar in all respects other than the treatments being compared) and found it to be valid, which is very difficult to do using SR reports, ITCs, or NMAs should not be conducted in overviews [9, 15, 31, 34]. Therefore, an overview of SRs—relying on review-level data—is best suited for examining a body of SR evidence to provide overall trends in research findings or answer a question about which interventions are effective or safe or for exploring if and why the evidence base on a topic or question is heterogeneous [27]. ITCs and NMAs in overviews are explicitly discouraged, and informal indirect comparisons should also be avoided [9].

  Consideration II: If the planned review requires quantitative combination of findings through MAs, ITCs, or NMAs to provide a summary measure or to rank interventions, a de novo SR or an update of an SR should be considered instead of an overview of SRs. In an overview, a narrative, not quantitative, synthesis should be conducted.

• II. State of the existing literature

  • 3. Relevance: Are there one or more relevant SRs available?

    To answer this question, database searches with appropriate SR filters or targeted searches for potentially relevant SRs may be needed. Potential SRs of interest could then be assessed on whether they meet eligibility criteria of the planned review, as specified by the PICO elements [28, 29]. If the scope of an SR is identical to or narrower than that of the planned review, the entire SR would be relevant. If its scope is broader, the subset of its findings that meets the eligibility criteria of the planned review would be relevant, if reviewers are able to extract the relevant results separately (e.g., from a subgroup analysis).

    If there is no relevant SR, conducting a de novo SR is likely necessary.

    For an update, there must be one or more relevant SRs available that each perfectly or almost perfectly covers the research question. It may be possible to integrate multiple SRs in an update [15, 19], but this is discouraged due to logistical and methodological complexities, as it would likely require merging existing SRs first with each other and secondly with newly identified primary studies [5, 9]. Therefore, our recommendation is to handle multiple relevant SRs through either an overview of two or more SRs or an update of a single SR that is “best” in terms of relevance, quality, comprehensiveness, and currency [18].

    For an overview, there should be two or more relevant SRs available that each covers at least one “part” of a research question, with all SRs combined resulting in perfect or almost perfect coverage. There is no accepted minimum or maximum number of SRs required for an overview, although recommendations range from including a minimum of “two or more” SRs [9] to “5–10 or more” SRs [29].

  Consideration III: The quantity of relevant SRs available may rule out certain review types for the planned review. If there is no relevant SR, no overview or update could be conducted. If there is only one relevant SR, no overview should be conducted.

  • 4. Methodological quality: Are the relevant SRs of sufficiently high quality in methodology?

    While it may sometimes be appropriate to include all relevant SRs in the planned review for completeness, it is often advised to assess the methodological quality of existing SRs and only use high-quality SRs in updates or overviews to build on high-quality literature [3, 6, 9, 15, 19, 28, 31]. In the absence of high-quality SRs, replication in the form of a de novo SR may be deemed appropriate to produce a higher-quality SR. [6, 17, 36]

    To assess methodological quality, relevant SRs should be critically appraised for their methodological rigour [17, 29]. Established critical appraisal tools, such as AMSTAR 2 [37] or ROBIS [38], can be used [17, 29]. There is no broadly accepted threshold for what is considered sufficiently high quality in methodology for a relevant SR to be used in a planned review, although there is guidance for what is good, fair, or poor quality [3, 18]. The threshold for sufficiently high quality in methodology may vary depending on the purpose of the review, how the relevant SRs will be used, whether there are high-quality SRs available, and what resources are available for the planned review. Nevertheless, criteria for determining whether relevant SRs are of sufficiently high quality in methodology should be established a priori for each planned review to minimize bias [28].

  • 5. Reporting quality: Are the relevant SRs of sufficiently high quality in reporting?

    Poor reporting in existing SRs would make it difficult not only to extract data but also to decide whether the SRs are relevant, of high methodological quality, comprehensive, and up to date, and hence, useful for the planned review. Therefore, high quality in reporting is desirable [15, 19], especially if the SRs are to be used in overviews without going back to primary study reports.

    To assess quality in reporting, reporting guidelines may be used [32], supplemented with additional criteria required for the planned review (e.g., exact data needed, such as point estimates and associated variability in a specific format). Available reporting guidelines include PRISMA Statement (for all SRs) [39], SWiM (for SRs with narrative syntheses) [40], and MOOSE (for SRs with MAs of observational studies in epidemiology) [41]. There is no broadly accepted threshold for what is considered sufficiently high quality in reporting for an existing SR to be used. However, at the least, it should be well written, describe the PICO elements and methods clearly, and report all necessary data in an easily extractable form.

  Consideration IV: Ideally, only SRs of high quality in both methodology and reporting should be used for updates and, in many cases, for overviews. In the absence of high-quality SRs, replication in the form of a de novo SR may be appropriate.

  • 6. Comprehensiveness: Are the relevant SRs comprehensive?

    To answer this question, relevant SRs should be examined on whether they include all relevant PICO elements of the planned review either individually (i.e., a single SR is comprehensive on its own) or together (i.e., two or more SRs are comprehensive when combined). If they do not, using them in an overview may result in research gaps in the final review, as it is not recommended that an overview of SRs be supplemented with primary studies due to logistical and methodological complexities [5, 9]. Therefore, a research question that cannot be fully addressed with existing SRs due to important research gaps in those SRs may be best addressed by a de novo SR or an update of an SR (e.g., with a search for additional data from an expansion of one or more of the PICO elements or a re-analysis of the available data for a different outcome) if primary studies are available to fill those gaps. On the other hand, if two or more relevant SRs are available that when combined result in perfect or almost perfect coverage [2, 9, 10, 27, 32,33,34,35], an overview of SRs may be appropriate.

  • 7. Currency: Are the relevant SRs up to date?

    In general, an SR may be considered to be up to date when (1) there is no new evidence or (2) there is new information, but it is unlikely to change the review conclusions [34]. However, the definition of currency varies from one field to another, as the acceptable time period from the last date of search will depend on various factors, such as the activity or advances in the field or in methods, the importance and urgency of the research question being addressed, and the level of uncertainty around the evidence base [3, 8, 19, 42]. Criteria for determining the currency or outdatedness of relevant SRs need to be established a priori for each planned review to minimize bias [28, 43].

    For those overviews that have the objective to answer a research question about intervention effects (as opposed to the objective to describe a body of literature), all relevant SRs should be sufficiently up to date [9]. Whether an SR is sufficiently up to date should be determined with the help of experts in the field of research who have the knowledge to speak to the various factors that define currency mentioned above. Therefore, a research question that cannot be fully addressed with existing SRs because they are out of date may be best addressed by an update of an SR—if there is a high-quality SR available—or by a de novo SR—if there is no high-quality SR available—that is, if new evidence is available to bring it up to date [3].

  • 8. New evidence: Are there additional relevant primary studies missing from the relevant SRs?

    Additional primary studies that are relevant to the research question but not captured by relevant SRs may be identified during scoping or through expert input. They may be (1) newly published studies or (2) studies not identified by relevant SRs because the SRs had narrower scopes, searched fewer databases, or had limited information sources (e.g., no grey literature). Since an overview of SRs should not be supplemented with primary studies [5, 9], a research question that cannot be fully addressed with existing SRs because additional primary studies would be needed for comprehensiveness or currency may be best addressed by an update of an SR—if there is a high-quality SR available—or by a de novo SR—if there is no high-quality SR available.

  Consideration V: If relevant SRs are not comprehensive or are outdated and need to be supplemented with additional primary studies that are available, an overview of SRs may not be appropriate. Instead, an update of an SR—if there is a high-quality SR available—or a de novo SR—if there is no high-quality SR available—is recommended for consistency in the analytic approach at the study level.

  • 9. Discordance in results: Are the findings of the relevant SRs discordant for unknown reasons?

    If there are two or more relevant SRs addressing the same research question with matching PICO elements, and if they are of sufficiently high quality in both methodology and reporting, their results could then be compared. Results are concordant if they match with respect to the direction, magnitude, and statistical significance of the estimated intervention effects for the outcome(s) of interest and in terms of the interpretations and inferences made by the SR authors [44]. Conversely, results are discordant if they differ in any of those aspects [44]. Such differences may or may not be important enough to lead to different health care decisions [44].

    If the results of relevant SRs with matching PICO elements are discordant and if the differences matter for the planned review, reasons for discordance should be explored [15, 18, 19, 30, 35, 44]. Potential reasons for discordance include differences in the methods or timing of different SRs that lead to different sets of studies being included, data being extracted, analyses being conducted, or syntheses or interpretations being made [44]. If reasons for discordant findings are clear, and if there is certainty around which discrepant results to trust [15, 30, 35]—likely those of high quality in both methodology and reporting and most up to date—reviewers may choose one or more SRs that are most appropriate for the planned review and conduct an update or an overview. If reasons for discordant findings are unclear and if there is uncertainty around which discrepant results to trust [15, 30, 35], a de novo SR may be indicated, instead of using one or more of the conflicting SRs [15, 19].

  Consideration IV: If the results of two or more relevant SRs with matching PICO elements are discordant, and reasons for discordance cannot be reliably determined, a de novo SR may be needed.

Part 2. Making the decision

A flowchart is presented in Fig. 1 and described below. This flowchart depicts the nine key factors and six considerations outlined above in the order that we see as helpful for selecting a specific review type to conduct for evaluating intervention effects in an HTA. The flowchart should be followed from top to bottom, first with Part A and then with Part B. Each question is indicated with a “Q” followed by a number that corresponds to the numbering of the key factors identified above. Both Part A and Part B begin with Q1 and Q2 (in blue boxes), which address the evidentiary needs of the planned review, and then proceed to Q3 through Q9 (in orange boxes), which address the state of the existing literature. Some of the questions from Q3 to Q9 are combined or modified for further investigation. Depending on the responses to the questions (“yes” or “no” in black circles), some review types are eliminated through this decision-making process (in yellow boxes), leading to one decision at the end (in green boxes).

Fig. 1

Flowchart for choosing one review type over another

If there is only a single relevant SR that is high quality in both methodology and reporting, comprehensive, and up to date, there is likely no need for a new SR of any kind for the specific research question (see D5 in Fig. 1 in both Part A and Part B). In this case, the existing SR can and should be used instead of conducting a new SR. Similarly, if there are multiple such SRs without discordant results for unknown reasons, no new SR should be conducted, and the best existing SR can be used (see D7 in Fig. 1 in both Part A and Part B). In all other cases, a decision on which review type to conduct must be made.

For some situations, there are considerations regarding the key factors identified above that are absolutely deterministic in ruling out certain review types, including the following:

• If the planned review requires quantitative combination of findings through MAs, ITCs, or NMAs, an overview should not be conducted (Consideration II; see D2 in Part A of Fig. 1).

• If there is no relevant SR, an update or an overview could not be conducted (Consideration III; see D3 in Part A and D9 in Part B of Fig. 1).

• If there is only one relevant SR, an overview should not be conducted (Consideration III; see D10 in Part B of Fig. 1).

For most situations, however, it is the interplay of multiple considerations regarding the key factors that informs the decision on which review type to conduct. In these cases, while no review types can be completely ruled out, one review type may be favoured over the others, as follows:

• If the planned review has a narrow scope (Consideration I), instead of an overview (see D1 in Part A of Fig. 1), it may be more appropriate to conduct a de novo SR (see D4 in Part A of Fig. 1) or an update of an SR (see D6 and D8 in Part A of Fig. 1), depending on whether one or more relevant SRs of high quality are available (Consideration IV) without discordant results for unknown reasons (Consideration VI). For example, if there is a single “best” relevant SR that is high quality in both methodology and reporting and comprehensive but needs an update with newly available data, new methods, or new analyses, with concordant results with other relevant SRs, an update may be the most appropriate review type. If there is no relevant SR at all or no relevant SR that is high quality in both methodology and reporting and comprehensive, a de novo SR may be needed.

• Conversely, if the planned review has a broader scope (Consideration I), and there are two or more relevant SRs available (Consideration III), an overview may be the most appropriate review type, especially if the relevant SRs are high quality in both methodology and reporting, comprehensive when combined, up to date, and concordant in their findings, with no need to be supplemented with additional primary studies (Considerations IV, V, and VI; see D12 in Part B of Fig. 1). If one or more of these conditions on the existing literature are not met, reviewers should carefully consider whether an overview of SRs is the most appropriate review type. It may be more appropriate to conduct a de novo SR, especially if the number of relevant primary studies is manageable with given resources (see D4 and D13 in Part B of Fig. 1), or an update if there is a single “best” relevant SR that is high quality in both methodology and reporting and comprehensive on its own (see D 6 and D8 in Part B of Fig. 1).

The flowchart in Fig. 1 outlines a decision-making process for selecting a review type to conduct, capturing all of the considerations from above.

Discussion

This paper shares guidance that can be used when selecting the most appropriate review type to conduct when evaluating intervention effects in an HTA. It identified nine key factors and six considerations concerning the evidentiary needs of the planned review and the state of the existing literature. Those factors and considerations were then organized into a summary and a corresponding flowchart that can be used as a decision tool, balancing potential benefits of leveraging existing SRs against any drawbacks of not starting a new SR from scratch. With that, a reviewer may choose to conduct no new SR at all, update an existing SR, use multiple existing SRs in an overview, or conduct a de novo SR. This guidance is aligned with the literature from various organizations [3, 6, 9, 15, 18, 19, 27,28,29,30,31] and other research groups [2, 8, 10, 32,33,34,35, 45], including a recent publication that highlights eliminating research waste by avoiding unnecessary duplication of SRs [17].

To fully contemplate these key factors and considerations for a planned review, there must be a clear research question as well as a good understanding of the quantity, quality, and other characteristics of relevant SRs and primary studies that are available. Therefore, this approach may require detailed scoping and perhaps a process change as well as additional resources as an “upfront” investment in the conduct of an HTA. This challenges the notion that leveraging existing SRs necessarily saves resources, and it reinforces the idea that existing SRs should be leveraged for other reasons, such as summarizing the existing literature at higher levels and avoiding or minimizing duplication or redundancy in research, all of which had been raised by other groups as well [15, 17, 19, 46]. Further, any additional work performed to follow the guidance presented here and apply the decision tool need not be wasted, as it could identify existing SRs that fully meet the needs of the planned review and actually free up and save resources or it could be used during the review process, regardless of the review type chosen.

The decision tool presented here depicts general guidance and should be contextualized to each case. For example, it might not always be clear if the research scope for the planned review is broad or narrow (Q1 in Fig. 1 in both Part A and Part B). Further, even for a narrow topic, if there are multiple relevant SRs, an overview may be considered as long as there is no need for quantitative combination of findings. In another example, if there is a high-quality SR that is almost but not fully comprehensive (Q3-6 in Fig. 1 in both Part A and Part B), the reviewer may decide to update that SR with revised PICO elements, instead of conducting a de novo SR, although this would generally be considered a purposeful replication and perhaps more of a de novo SR instead of an update in any case [6,7,8, 17]. Therefore, the decisions outlined in the tool should be viewed as examples in ideal situations instead of a collection of definitive and exhaustive possibilities, and going against those decisions is possible and may even be appropriate in certain cases. The flowchart should be seen as simply highlighting the many interacting considerations that need be contemplated during the decision-making process. We suspect that others who conduct evidence syntheses are also working through these decisions, perhaps on an ad hoc basis, and present this work to share our current decision-making process and invite commentary and reflections.

Regardless of which review type is used, it is essential that the project protocol and report provide a list of existing relevant SRs identified in the literature through the scoping activity (e.g., as an appendix) and the rationale behind selecting a specific review type over others for transparency purposes. During scoping, effort should be made to also identify ongoing SRs (e.g., through PROSPERO). If ongoing or planned reviews are identified, opportunities for brokering or collaborating or other ways to incorporate them into the decision-making process for selecting a review type should be explored.

We considered incorporating into the flowchart rapid approaches [12] and living approaches [13], either or both of which could be overlaid on each review type (e.g., rapid SR, rapid update, or rapid overview; living SR or living overview; living rapid SR or living rapid overview). A rapid approach alters, simplifies, or omits components of a systematic process to produce information in a shorter period of time [12]. This approach may involve searching fewer databases or engaging a single reviewer instead of dual reviewers and may be used under resource constraints (e.g., short timelines or limited funding) [20], especially if there is already some high-quality evidence with one or more relevant SRs available and no unexplained discrepant findings across the SRs. A living approach continually updates an SR or an overview, incorporating relevant new evidence as it becomes available [13]. This approach may be used if a priority research question cannot be fully addressed because there is an important level of uncertainty in the existing evidence and if new evidence that will impact the current conclusions is likely to emerge soon in a research field that is moving relatively quickly [43]. As the literature on these approaches is still actively emerging, we did not incorporate these review approaches into the flowchart but identify them as potential additional considerations, as also noted by others [17]. In other words, reviewers could use the flowchart provided here to first select the most appropriate review type to conduct and then decide to apply rapid and/or living approaches to the chosen review type as needed.

Conclusions

This work provides guidance on how to balance a myriad of factors and considerations to help select the most appropriate review type to conduct when evaluating intervention effects in an HTA. The decision-making process presented is undoubtedly complicated, yet fundamental for every review, given the exponential growth in the number of published SRs [14, 15] and high-resource requirements for completing new SRs [15]. While leveraging existing SRs may or may not save resources, doing so will likely contribute to a better synthesis of the existing literature and a reduction of research waste and therefore lead to a more efficient use of resources in HTAs.

We hope this guidance adds clarity to the many competing considerations when deciding which review type to conduct and facilitates that decision-making process. As we continue to test and update this decision tool, we invite feedback from others who may have their own guidance or apply ours in their work. While we focused on SRs of intervention effects, the application of this guidance to other types of reviews, such as qualitative or mixed methods reviews, should be explored in the future.

Appendices

Appendix 1

Table 2

Table 2 Database search strategy

Appendix 2

Table 3

Table 3 Grey literature search strategy

Relevant websites from the following sections of the CADTH grey literature checklist Grey Matters: A Practical Tool for Searching Health-Related Grey Literature (https://www.cadth.ca/grey-matters) were searched:

• Health Technology Assessment Agencies

• Databases (free)

• Internet Search

Appendix 3

Table 4

Table 4 Excerpts of relevant text from the literature review