Table 1 Coding framework for methods used in overviews of systematic reviews of interventions (stage I)

1. Attributes of the study

Information about the study being coded; includes codes for the purpose and aims and methods of the study.

2. Definition of an overview

Information about the concept and origins of overviews; includes codes for definitions, terminology, rationale and key references for overviews.

3. Terminology for overviews

Description of the terms used for overviews (e.g. ‘review of reviews’). Terms used in the title, abstract and body of the study are coded separately.

4. Methodological considerations (independent of the step or stage of the overview)

Information about the challenges and decisions faced when planning the methods of an overview. Considerations that are unique to overviews should be coded, such as dealing with reviews that (a) overlap in scope, (b) use different methods for assessing bias in included studies, (c) report different outcomes for similar comparisons and (d) have discrepant findings for the same outcomes/comparisons.

 4.a. Dealing with currency (or lack of currency) of reviews

Considerations relating to whether a review is up-to-date and the implications of including reviews with older search or publication dates.

 4.b. Dealing with reviews of different methodological quality

Considerations relating to the inclusion and interpretation of reviews assessed as being at a high risk of bias (e.g. because of the methods used to identify and select studies, synthesis approach). This may include discussion about exclusion of poorer quality reviews, decisions to weight the findings of reviews based on risk of bias and consideration of risk of bias when grading the evidence arising from a review.

 4.c. Dealing with overlap (and differences in scope)

Considerations related to dealing with reviews that include the same studies and overlapping information/data from those studies. Methods relevant to dealing with overlap may apply to one or more steps/stages of the review (e.g. Population, Intervention, Comparison, Outcome (PICO), assessment of quality of included studies, summary and synthesis).

 4.d. Dealing with gaps in review coverage

Considerations relating to aspects of the overview question that have not been addressed in reviews (e.g. particular comparisons, types of interventions, subgroups).

 4.e. Dealing with missing (or loss of) information about study characteristics

Considerations relating to the information available in reviews about the primary study characteristics (PICO elements, study design). Reviewers report selected information, focusing on that most relevant to their question. This may or may not be congruent with the overview question, and the information may be difficult to interpret out of context, resulting in loss of information.

 4.f. Dealing with inconsistent or incomplete outcome reporting across reviews

Considerations when reviews report different outcomes, outcome measures, follow-up times, analyses or data from studies. This may include considerations relating to loss of information if reviews that report unique outcome data are excluded for other reasons (e.g. on the basis of date or quality).

 4.g. Dealing with discrepant methods of quality appraisal across reviews

Considerations when reviews use different methods to assess the risk of bias or quality of included studies (e.g. Cochrane Risk of Bias tool [21], Newcastle-Ottawa Quality Assessment Scale [23]).

 4.h. Dealing with missing information from the synthesis in reviews

Considerations relating to the type and nature of syntheses (including meta-analyses) reported in the review. Reviewers conduct and report syntheses most pertinent to their question, which may mean syntheses for particular subgroups, interventions or outcomes are not available. Additional considerations may relate to missing information about heterogeneity, sensitivity analyses or other analyses that might be needed to interpret the results of a review.

 4.i. Dealing with summary and synthesis of multi-faceted interventions

Considerations relating to the summary and synthesis of findings for multi-faceted interventions. Reviewers may take different approaches to synthesising findings for multi-faceted interventions, driven in part by the question they aim to answer. For example, they might examine potential additive effects of each intervention component or they might decide that interventions are too heterogeneous for meaningful synthesis. Across reviews, this can lead to quite different summaries and syntheses.

 4.j. Dealing with discrepant findings across reviews

Considerations when two or more reviews estimate or describe different effects (quantitatively or qualitatively) based on similar studies or data. The findings of two reviews that are similar in scope may differ (a) because their results are different or (b) because the reviewers’ interpretation of the results is different. This node is intended to capture the former.

 4.k. Dealing with discrepant interpretation of similar findings across reviews

Considerations when two or more reviews interpret similar results differently. The findings of two reviews that are similar in scope may differ (a) because their results are different or (b) because the reviewers’ interpretation of the results is different. This node is intended to capture the latter.

5. Methods (specific to step or stage of the overview)

Identification, description and other information about the methods used to conduct an overview, grouped by the steps (or stages) described in the Cochrane Handbook [21].

 5.a. Overall steps, sequence or stages

Identification or summary of the key steps or stages involved in producing an overview of reviews.

 5.b. Specification of scope, questions, objectives

Methods for determining, defining and reporting the scope of an overview; includes factors that influence how broad or narrow questions should be, the ways in which questions might be split (e.g. by condition, by population subgroup, by intervention) and the implications of doing so. This node relates to the specification of eligibility criteria but differs in that it covers conceptual issues and the methodological implications of lumping/splitting.

 5.c. Specification of eligibility criteria

Methods for determining and specifying eligibility criteria (i.e. PICO); includes restrictions on eligibility (e.g. publication status, year) that may be used to deal with overlap; includes codes for outcome selection.

  5.c.i. Outcome selection mechanisms

Methods for specification and selection of outcomes. Note that while information about outcome selection methods can be reported under eligibility criteria, reviews are not necessarily excluded from the overview if they do not report a specified outcome. Instead, the outcomes reported may determine inclusion in the synthesis (or summary) of effects. In circumstances where reviews are included irrespective of outcomes, information relating to outcome specification may need to be coded under other relevant nodes (e.g. ‘Outcome selection mechanisms’).

  5.c.ii. Decision rules for selecting a review from multiple overlapping reviews

Methods for dealing with multiple reviews that include the same studies and overlapping data from those studies. Several methods have been proposed to deal with overlap and are represented in the subnodes. These include (a) the review with the largest amount of studies, (b) most recent/up-to-date information, (c) the highest quality studies, (d) most complete reporting, (e) by publication status, (f) one review per author, (g) eliminate the review with the least amount of studies and (h) ignore overlap.

 5.d. Search methods

Methods for searching for reviews in an overview; includes codes for specific search filters used, such as those provided in PubMed.

 5.e. Selection of reviews

Methods for selecting reviews for inclusion in the overview, such as the use of two reviewers to independently screen reviews for inclusion in the overview.

 5.f. Data extraction and coding

Methods for data extraction in an overview and coding of information from the included reviews; includes codes for extracting data for subgroups, extracting data from reviews versus trials, and methods used to select outcomes for extraction.

 5.g. Assessment of risk of bias (methodological quality) of primary studies included in the reviews

Methods for dealing with reviews that have used different methods or tools for assessing risk of bias (or quality) of primary studies.

 5.g. Assessment of risk of bias arising from the methods of the review

Methods for assessing the risk of bias arising from the design, conduct and reporting of reviews. Sometimes referred to as methodological quality of the review. Example tools include ROBIS [20], A Measurement Tool to Assess Systematic Reviews (AMSTAR) [24], National Institute for Health and Care Excellence (NICE) [25].

 5.g. Assessment of the overall quality of the evidence arising from the overview

Methods for assessing the overall quality of the evidence for each comparison/outcome in the overview. These methods are likely to be the same as used for any synthesis (e.g. Grades of Recommendation, Assessment, Development and Evaluation (GRADE) [26], Scottish Intercollegiate Guidelines Network (SIGN) [27], FORM [28]) but may require adaptation to synthesise evidence across reviews instead of primary studies. For example, should the assessment of risk of bias in the included review be considered when grading the evidence and, if so, how?

 5.h. Synthesis

Methods for analysing and synthesising the data in an overview; includes approaches for dealing with meta-analyses with overlapping studies from different reviews (i.e. those for the same comparison and outcomes), exploring heterogeneity, etc.

  5.h.i. Quantitative synthesis

Methods for quantitative synthesis in an overview; includes codes for synthesising meta-analyses in an overview, exploring heterogeneity among included reviews, considerations of how to include meta-analyses and how to include subgroup analyses and synthesis and summary without meta-analysis.

  5.h.ii. Synthesis and summary without meta-analysis

Methods for synthesis and summary that do not include a meta-analysis (e.g. plotting and tabulating data, vote counting).

 5.i. Presentation and summary of findings

Methods for presenting and summarising the findings of an overview. May include methods for efficient depiction of the overlap of studies across included reviews, methods for summarising findings when the same study is included in more than one review, etc.

 5.j. Interpreting findings and drawing conclusions

Methods for interpreting results and drawing conclusions in an overview.

 5.k. Reporting

Recommendations about the preferred reporting items for overviews, which may involve using standards or guidelines for reporting overviews, checklists or reporting tools.

6. Pros and cons of method

Known or anticipated benefits and disadvantages of using different methods in an overview. These might relate to efficiency of production, utility of the overview for decision makers and the validity of findings (bias in estimates of intervention effects).