Skip to main content

Table 1 Criteria for determining that a conclusion is out-of-date

From: A surveillance system to assess the need for updating systematic reviews

Ottawa’s label

Ottawa method

 

Qualitative criteria for potentially invalidating signals

A1

Opposing findings: a pivotal* trial or systematic review (or guidelines) including at least one new trial that characterized the treatment in terms opposite to those used earlier

A2

Substantial harm: a pivotal trial or systematic review (or guidelines) whose results called into question the use of the treatment based on evidence of harm or that did not proscribe use entirely but did potentially affect clinical decision-making

A3

A superior new treatment: a pivotal trial or systematic review (or guidelines) whose results identified another treatment as significantly superior to the one evaluated in the original review, based on efficacy or harm

 

Qualitative criteria for signals of major changes

A4

Important changes in effectiveness short of ‘opposing findings’

A5

Clinically important expansion of treatment

A6

Clinically important caveat

A7

Opposing findings from discordant meta-analysis or non-pivotal trial

 

Quantitative criteria signals of changes in evidence

B1

A change in statistical significance (from nonsignificant to significant)

B2

A change in relative effect size of at least 50 percent

RAND’s label

RAND method indications for the need for an update

1

Original conclusion is still valid and this portion of the original report does not need updating. This conclusion was reached if we found no new evidence or only confirmatory evidence and all responding experts assessed the CER conclusion as still valid, we classified the CER conclusion as still valid

2

Original conclusion is possibly out-of-date and this portion of the original report may need updating. This conclusion was reached if we found some new evidence that might change the CER conclusion, and/or a minority of responding experts assessed the CER conclusion as having new evidence that might change the conclusion, then we classified the CER conclusion as possibly out-of-date

3

Original conclusion is probably out-of-date and this portion of the original report may need updating. This conclusion was reached if we found substantial new evidence that might change the CER conclusion, and/or a majority of responding experts assessed the CER conclusion as having new evidence that might change the conclusion, then we classified the CER conclusion as probably out-of-date

4

Original conclusion is out-of-date. This conclusion was reached if we found new evidence that rendered the CER conclusion out-of-date or no longer applicable; we classified the CER conclusion as out-of-date. Recognizing that our literature searches were limited, we reserved this category only for situations where a limited search would produce prima facie evidence that a conclusion was out-of-date, such as the withdrawal of a drug or surgical device from the market, a black box warning from FDA, and so on

  1. Abbreviation: CER comparative effectiveness review, FDA Food and Drug Administration.
  2. *a pivotal trial is defined as trial that is published in one of the top five general medical journals or a trial whose sample size is at least triple that of the largest trial in the original systematic review.
  3. Legend: Table 1 presents the criteria used to determine if a conclusion is out of date within an SR (here CER). Criteria A1 to B2 come from the Ottawa method and criteria 1 to 4 are based on the RAND method.