Table 5 Common challenges involved in conducting overviews

 Choosing between conducting an overview and a SR

1 (CMIMG)

Network meta-analyses are very difficult to conduct in overviews and should likely not be conducted within overviews. It may be difficult to determine whether it is more appropriate to conduct an overview, or a systematic review with or without network meta-analysis.

 What types of questions about healthcare interventions can be answered using the overview format?

2 (CCRG, CM)

Methods used to conduct overviews may vary according to the type of question (e.g., scope, clinical characteristics) being posed in the overview.

 Questions to consider before deciding to conduct an overview

5 (CHF, CMIMG, DCC, JBI, UDun)

Should authors conduct an overview if there are not enough relevant SRs (e.g., if SRs do not address all important interventions)?

 Author team composition and roles

2 (CHF, CMIMG)

Overview authors often have limited time. What skills are required for authors wishing to conduct overviews?

 Target audience of the overview

0

No challenges identified.

 Specifying the scope

4 (EPPI, LBI, UBirm, UDun)

Defining the scope, and selecting and prioritizing outcomes, can be difficult. The scope of the overview may have almost complete overlap, or very limited overlap, with the scope of the relevant SRs.

 Searching for SRs

5 (CHF, CPHG, EPOC, LBI, UBirm)

Search strategies can be complex. It is unclear whether government reports that include both primary studies and SRs should be included in an overview. It is unclear whether and how overview authors should search for primary studies that are not contained within any included SR.

 Selecting SRs for inclusion

8 (CHF, CMIMG, DukeU, EPPI, JBI, UBirm, UDun, WHU)

It is unclear whether lower-quality SRs or older SRs should be included or excluded. Decisions surrounding inclusion and exclusion can affect the efficiency, utility, and breadth of the overview.

 Should an overview include non-Cochrane SRs?

9 (CHF, CMIMG, CPHG, DukeU, EPOC, EPPI, TCD, WHU, WJNR)

Including non-Cochrane SRs can be difficult and will increase the complexity of the overview process. Non-Cochrane SRs can be of low methodological quality and may be poorly reported. Additionally, some Cochrane and non-Cochrane SRs will have overlap in their clinical questions, inclusion criteria, and/or included primary studies, and may have discordant results and/or conclusions. Overlapping SRs can be problematic, and there are potential challenges involved in assessing the amount of overlap in included SRs. Additionally, methods for choosing between overlapping SRs have not yet been developed; for example, it is unclear whether authors should include only one SR per topic area (and if so, which one?), or if they should include all SRs regardless of overlap (and if so, how will overlap be managed?). Authors including non-Cochrane SRs also have to clearly define what counts as a SR.

 Assessing quality of included SRs

9 (CCRG, CHF, CMIMG, CPHG, EPOC, EPPI, PXU, UBirm, UDun)

Assessing quality of SRs can be difficult and time-consuming. Many different tools could be used to assess SR quality, and some tools designed to assess quality may also assess reporting. There is also uncertainty surrounding how to interpret and apply the results of quality assessments in the context of overviews.

 Collecting and presenting data on descriptive characteristics of included SRs (and their primary studies)

11 (CCRG, CHF, CM, CMIMG, DCC, DukeU, EPOC, JBI, LBI, UDun, NOKC)

Data may be missing, inadequately reported, or reported differently across included SRs, and it is unclear what to do when reporting is incomplete (e.g., should the data be extracted from primary studies?). Additionally, data extraction errors in SRs could lead to errors in the overview.

 Collecting and presenting data on quality of primary studies contained within included SRs

7 (CCRG, CHF, CM, DCC, EPOC, EPPI, UDun)

Collecting and presenting quality of primary studies can be difficult and time-intensive. Information about the quality of primary studies included in SRs may be missing, inadequately reported, or reported differently across included SRs. For example, different SRs may use different tools to assess quality of primary studies.

 Collecting, analyzing, and presenting outcome data

15 (CCRG, CHF, CM, CMIMG, DCC, DukeU, EPOC, EPPI, JBI, LBI, NOKC, UBirm, UDun, WJNR, WHU)

Collecting, analyzing and presenting outcome data can be difficult, especially when the scope, methods, or results of the included SRs are heterogeneous. Outcome data may be missing, inadequately reported, or reported inconsistently across included SRs, and it is unclear what to do in these situations (e.g., should the data be extracted from primary studies instead?). It is also unclear how best to summarize and report outcome data that comes from overlapping (and potentially discordant) SRs. It may not always be possible or appropriate to conduct meta-analyses in overviews or to directly compare results across different SRs. Similarly, network meta-analyses are often not appropriate in overviews. Additionally, overviews may not accurately capture information about adverse effects or cost-effectiveness of interventions, and data extraction errors in SRs could lead to errors in the overview.

 Assessing quality of evidence of outcome data

9 (CCRG, CHF, CM, CPHG, DCC, EPOC, PXU, UDun, WHU)

It may not be possible to simply extract existing GRADE assessments from SRs. However, it may be challenging to conduct (or re-do) GRADE assessments at the overview level, using data from SRs. For example: data needed to assess quality of evidence in SRs may be missing, inadequately reported, and/or reported differently across included SRs; the “study quality” domain may be assessed differently across similar SRs (e.g., different tools used, same tool used but different assessments obtained, only summary assessments reported); and the “consistency” and “precision” domains may be affected if different methodological decisions are made in similar SRs (e.g., pooling versus not pooling data). Additionally, achieving consensus may be difficult. The GRADE tool may need to be modified for use in overviews.

 Interpreting outcome data and drawing conclusions

6 (CHF, CMIMG, DukeU, EPOC, LBI, WJNR)

Interpreting outcome data and drawing conclusions can be difficult. There is uncertainty surrounding how to interpret outcome data in overviews. It can be difficult to form a coherent judgment when multiple different comparisons from multiple SRs are included in the same overview, and/or when overlapping SRs report discordant results. It can also be difficult to determine implications for research. Additionally, there is concern that the methods used to conduct overviews might affect the conclusions reached.