Table 4 Summary of common features and differences between the exemplar overviews, and implications for development of protocols for future overviews

Clearly defined aim or question.

The aim or question can vary considerably depending on the focus of the overview.

1. Be clear about the purpose of the overview and ensure clear definition of aim or question.

2. Involve key stakeholders at the development stage to ensure the planned overview is relevant and useful. Resources being compiled by Cochrane Training will inform methods of involving people [35, 36].

Includes relevant reviews.

The reviews may be reviews of randomised or quasi-randomised trials, or may be reviews of other study designs, including non-randomised, qualitative or mixed methods studies.

3. Ensure clear definition and justification of scope of overview.

4. Include a clear definition of what criteria constitute a “systematic review”. These criteria may include; a pre-defined protocol detailing the scope and methods to be employed in conducting the review; a specific and detailed search strategy giving at least one example of the exact terms used within the search; a clear selection process in line with pre-specified criteria for inclusion in the systematic review; assessment of included studies for various types of bias including publication bias; synthesis of research findings in order to address the original research question; discussion of results in relation to existing evidence, limitations of the research and included studies and contribution to the field of study [52, 53].

Some Cochrane overviews only include Cochrane systematic reviews, whilst other Cochrane overviews include any systematic reviews.

Some overviews include primary research studies in addition to review. Additional searches for new primary studies have been considered when identified systematic review evidence has been judged to be out of date, or there has been known gaps within systematic review evidence.

5. Plan, and clearly state within the overview protocol, how multiple overlapping reviews (i.e. systematic reviews which address the same, or similar, research questions, including the same, or similar, primary research studies [54] will be dealt with. Consensus is yet to be reached on the best way to deal with overlapping review evidence, but there is a clear need for guidance and improved transparency in the way in which authors deal with this problem [55].

6. Clearly state plans for action to be taken if identified systematic review evidence is out of date or gaps in the evidence (i.e. absence of reviews) are identified) [56]. If searching for primary studies is to be considered, full details of the scope and methods for this should be provided. Consider contacting authors of original reviews to see if they would update this. Note: Pieper [25] proposes and discusses two different approaches for searching for primary studies to ensure an overview is up-to-date; one approach involves searching for reviews and primary studies in parallel, whilst the other approach involves identifying the most up-to-date review and updating the searches from the date of the last search [56].

A clearly defined selection criteria for included reviews.

The parameters or domains which are defined reflect the aims/focus of the overview.

7. Ensure definition of selection criteria which are relevant to the aims/focus of the overview.

A structured search strategy to identify relevant reviews

The search strategy may be limited to databases of reviews (e.g. CDSR, DARE^a), or may involve searching a wide variety of electronic databases.

8. Consider and justify whether overview is limited to Cochrane reviews only or whether non-Cochrane reviews are to be included, with reflection on the available time and resources, and the potential added value of including non-Cochrane reviews and searching additional databases. If non-Cochrane reviews are to be included, state databases to be searched. With the discontinuation of the DARE database, authors should consider which key databases to search, the potential added value of searching additional databases, and the use of a validated search filter for systematic reviews. Authors should remain aware of any new database options, as there is ongoing work to develop comprehensive databases of systematic reviews (e.g. EPISTEMONIKOS [57] and PDQ-Evidence [58].

The search strategy may have date restrictions. The use of date restrictions has been proposed as an approach to searching for literature within limited timescales, justification provided often relates to improvement in the quality and consistency of systematic review evidence over time (including improvements from 1999 following release of the QUality Of Reporting Of Meta-analyses (QUOROM) [31] statement was released.

Two independent overview authors apply inclusion criteria and select reviews.

–

9. Plan for two independent overview authors to apply inclusion criteria and select reviews.

Systematic extraction and reporting of key characteristics of included reviews.

–

10. Clearly state data to be extracted, including the characteristics of included reviews to be extracted.

Systematic extraction and reporting of results of meta-analyses

Some overviews extract and report key statistical data (e.g. effect size, confidence intervals), whilst others only summarise the direct of effect.

11. Clearly state the statistical data which is to be extracted, and how this will be summarised. Detail any re-analysis if this is planned.

Some overviews carry out some degree of re-analysis of the results presented in the reviews.

Assessment of methodological quality of included reviews.

The tool which is used to assess methodological quality or risk of bias varies, and may be the AMSTAR or ROBIS tool. The method of applying and reporting the results from these tools can vary.

12. Select and justify a tool to assess methodological quality of included reviews, according to up-to-date evidence relating to available tools, and with consideration of the need to assess quality of methods and/or risk of bias. Note: The issues raised in relation to the AMSTAR have been debated by other authors [59,60,61], and the use of a priori decision tools has been proposed as a method to support the valid, reliable use of the AMSTAR tool within overviews [13].

13. Plan to have two independent reviewers assessing methodological quality, and state how any disagreements will be addressed.

14. Plan to report the results of individual questions or domains (not just a summary score).

Assessment of quality of evidence within reviews.

If quality of evidence within reviews is assessed the GRADE approach (for reviews of effectiveness) and the QUADAS-2 [39] (for reviews of diagnostic test accuracy) have been used. The method of applying the GRADE approach varies between overviews, and an algorithmic approach has been explored.

15. Consider best evidence and guidance relating to the application of tools to assess quality of evidence within reviews. Be aware of new developments and guidance in this field, and build on methods in previous overviews.

Data synthesis and summary of key findings.

While the need for clear, accessible summaries of key findings is common across all overviews, the methods of summarising key findings vary.

16. Clearly state how key findings will be summarised, with consideration of summarising the findings in relation to the quality of evidence (e.g. GRADE), and the populations, interventions and outcomes which this evidence relates to (for evidence relating to effectiveness), and the audience to whom the overview is of relevance. A number of different templates for summarising the findings of an overview have been proposed, including simple ‘traffic light’ graphics to illustrate evidence of effective, non-effective or detrimental interventions [4] and structured summary of results tables [8, 62, 63].