Table 4 Reported challenges related to conducting overviews

 Choosing between conducting an overview and a systematic review

2 (CHF, CMIMG)

It is not clear how to decide when it is better to perform an intervention systematic review versus an overview of reviews. It can be difficult to compare multiple interventions in the overview format, and it is often not feasible or appropriate to conduct a network-meta-analysis within an overview of reviews.

 What types of questions about healthcare interventions can be answered using the overview format?

2 (CCG, CMG)

Methodological approaches may differ depending on the type of question that the overview of reviews aims to answer.

 Questions to consider before deciding to conduct an overview

8 (CCRG, CHF, CMIMG, EPOC, JBI, UBirm, UCalg, UDun)

Overviews of reviews can be time consuming to produce, so there is a need to think about time and resource limitations and the need to balance flexibility with rigor. Authors need to think about the coverage and up-to-dateness of the available systematic reviews and decide whether an overview of reviews should be conducted if key primary studies or important interventions are missing from the available systematic reviews. Authors need to think about whether it would be feasible within time and resource constraints to update any systematic reviews that are out of date.

 Author team composition and roles

2 (CMIMG, UCalg)

Authors are challenged with determining the size, composition, and skillset of the team members. A larger team than originally thought might be needed when individual contributors are limited in time.

 Target audience of the overview

1 (CCG)

Approaches to preparing the overview of reviews may need to be adapted depending on the intended audience.

 Specifying the scope

9 (CHF, DCC, EPPI, LBI, RRI, UCalg, UBirm, UDun, WJNR)

Defining the scope and selecting and prioritizing populations and outcomes of interest can be difficult. The scope of available systematic reviews may be broader or narrower than the scope of the overview of reviews, and the available systematic reviews might not present data that are most relevant to the objective of the overview of reviews. When the scope is broad but time is limited, important outcomes might need to be prioritized.

 Searching for systematic reviews

7 (CHF, CPHG, DukeU, EPOC, LBI, UBirm, UCalg)

Developing searches and deciding which index terms to use, which sources to search, and what restrictions should be placed on the search (e.g., language, date) can be challenging and need to be well thought out to avoid missing important systematic reviews. There is debate about the need to also search for primary studies that are not contained in any included systematic reviews, or when searches should be updated to find new primary studies. This adds complexity to the search.

 Selecting systematic reviews for inclusion

15 (ARCHE, CHF, CMIMG, CSG, CSU, DukeU, EPOC, EPPI, GCU, JBI, RRI, UCalg, UBirm, UDun, WHU)

There are many decision points in selecting systematic reviews for inclusion that can be challenging and time-consuming. Authors need to decide how to define a ‘systematic review’, for which there is no single agreed upon definition. Authors then need to plan how they will handle systematic reviews that are out of date. They can update these themselves, add relevant primary studies, or concede that the findings of recent trials will be omitted. This can be a trade-off between amount and quality of evidence included. Finally, authors are challenged with identifying and handling primary study overlap at the selection level when many overlapping systematic reviews may exist. This can be time-intensive and challenging because of variable reporting across the available systematic reviews (e.g., may not transparently report all associated publications, may include different arms of the same trials).

 Should an overview include non-Cochrane systematic reviews

3 (ARCHE, CHF, CMIMG)

The decision about whether to only include Cochrane systematic reviews or to also include non-Cochrane systematic reviews can be a balance between ensuring quality and coverage of all important interventions. Though non-Cochrane reviews can be of poorer methodological quality and have less detailed reporting, Cochrane reviews alone may not cover all relevant interventions or be adequately up to date. If authors choose to include both Cochrane and non-Cochrane systematic reviews, it is likely that they will need to deal with primary study overlap. However, this may occur even if only Cochrane systematic reviews are included.

 Assessing the quality of included systematic reviews

10 (ARCHE, CCRG, CHF, CMIMG, EPPI, EPOC, GCU, PXU, RRI, UDun)

There is no agreement on which tool might be best to use (e.g., AMSTAR, AMSTAR 2, or ROBIS) to assess methodological quality, or how to use them in the context of an overview of reviews. It can be difficult to distinguish between methodological quality and the quality of reporting, and poor reporting in the systematic reviews can make assessment challenging. Authors often have difficulty interpreting and coming to agreement with assessments on the available tools. It is unclear whether authors should assess systematic reviews in their entirety or only the components that are relevant to the overview question, and what to do with systematic reviews that include other embedded reviews. When overview quality is being used to choose between overlapping systematic reviews, authors need to be careful to not exclude potentially relevant information. When overlapping systematic reviews use different methodologies and come to discordant conclusions, it can be hard to tell whether their methods are appropriate.

 Collecting and presenting data on descriptive characteristics of included systematic reviews (and primary studies)

11 (CCRG, CHF, CMIMG, CMG, DCC, DukeU, EPOC, JBI, LBI, NOKC, UCalg)

Overview authors are challenged with data extraction at two levels, first the level of the systematic review, and then potentially the level of the primary study. When relying on the reporting of the included systematic reviews, authors may struggle when these are poorly reported and missing important details. Overview authors need to carefully check systematic reviews for errors in data extraction, as these errors will lead to errors in the overview of reviews. They also need to decide how to deal with systematic reviews with missing information of relevance to the overview of reviews. Going back to the primary studies can be time consuming, but not doing so can lead to a loss of information.

 Collecting and presenting data on quality of primary studies contained within included systematic reviews

7 (CCRG, CHF, CSG, DCC, EPOC, EPPI, JBI)

Collecting and presenting information on the quality of the primary studies can mean relying on the appraisals of the original systematic review authors, which may be flawed, inconsistent, or poorly reported. Some systematic reviews may only report a summary of appraisals, rather than the risk of bias or quality of individual studies or outcomes of interest. Comparisons across systematic reviews can be difficult if different tools are used in each, because using different methods of assessing risk of bias can lead to disparate judgments.

 Collecting, analyzing, and presenting outcome data

23 (ARCHE, CChile, CCRG, CMG, CMIMG, CHF, DCC, DukeU, EPOC, EPPI, JBI, KCL, LBI, NOKC, PXU, RRI, TCD, UAuck, UBirm, UCalg, UDun, WHU, WJNR)

Many difficulties may arise when collecting, analyzing, and presenting findings at the overview level, because of inconsistency in methodology and reporting of findings across systematic reviews. For example, the included systematic reviews and their primary studies may use heterogeneous outcome measures. Additionally, the included systematic reviews may be incompletely reported, or may not report data on subgroups of interest. Overlapping systematic reviews might present discordant results or present similar data in different ways (e.g., different summary measures), and it can be complex and time-consuming to ensure that data from single studies are not over-represented. Interpretation of measures of overlap (e.g., matrices and corrected covered area) can be a challenge when the number of primary studies is large. To perform analyses of interest, overview authors might need to go back to individual studies, or concede that the available information is incomplete. It may not always be appropriate or feasible to conduct meta-analyses in overviews, and network meta-analyses and informal indirect comparisons are usually not appropriate. However, narrative synthesis can become complex and open to bias if not adequately described. There is a concern that synthesis errors at the SR level could result in errors at the overview level.

 Assessing quality of evidence of outcome data

11 (CCRG, CHF, CMG, CMIMG, CSG, DCC, EPOC, Glasgow, PXU, RRI, UDun)

It may not be possible or appropriate to simply extract existing GRADE appraisals from the included systematic reviews. The reviews might not include GRADE appraisals for the outcomes or populations of interest or be missing details on each of the GRADE considerations. Different systematic reviews with the same studies that have made different decisions about handling data (analysis) and appraising study quality may come to different GRADE conclusions, especially related to the study limitations, consistency, and precision domains. Different raters across systematic reviews could come to different conclusions, due to the subjectivity of the GRADE approach. If re-doing the GRADE for each systematic review, authors are likely to encounter difficulty due to an absence of guidance on how to apply GRADE in the context of an overview, incomplete reporting at the level of the systematic review, and a lack of familiarity with the contributing primary studies.

 Interpreting outcome data and drawing conclusions

9 (CHF, CMIMG, DCC, DukeU, EPOC, GCU, LBI, URRI, UCalg)

Interpreting data and drawing conclusions can be difficult. The included systematic reviews (and their included primary studies) may use heterogeneous outcome measures which can limit the ability to draw useful conclusions. Procedural variation at the systematic review and overview levels (e.g., study selection, data extraction) can lead to different conclusions from the same set of data. It can be difficult to provide interpretation of analyses of multiple interventions; multiple comparisons from different systematic reviews that are included in the same overview; discordant results and conclusions across the included systematic reviews. Authors need to consider the methods used in the systematic reviews and overview, and decide how best to highlight uncertainties and gaps that remain.