Skip to main content

Table 1 Criteria for determining that a conclusion is out-of-date

From: A surveillance system to assess the need for updating systematic reviews

Ottawa’s label Ottawa method
  Qualitative criteria for potentially invalidating signals
A1 Opposing findings: a pivotal* trial or systematic review (or guidelines) including at least one new trial that characterized the treatment in terms opposite to those used earlier
A2 Substantial harm: a pivotal trial or systematic review (or guidelines) whose results called into question the use of the treatment based on evidence of harm or that did not proscribe use entirely but did potentially affect clinical decision-making
A3 A superior new treatment: a pivotal trial or systematic review (or guidelines) whose results identified another treatment as significantly superior to the one evaluated in the original review, based on efficacy or harm
  Qualitative criteria for signals of major changes
A4 Important changes in effectiveness short of ‘opposing findings’
A5 Clinically important expansion of treatment
A6 Clinically important caveat
A7 Opposing findings from discordant meta-analysis or non-pivotal trial
  Quantitative criteria signals of changes in evidence
B1 A change in statistical significance (from nonsignificant to significant)
B2 A change in relative effect size of at least 50 percent
RAND’s label RAND method indications for the need for an update
1 Original conclusion is still valid and this portion of the original report does not need updating. This conclusion was reached if we found no new evidence or only confirmatory evidence and all responding experts assessed the CER conclusion as still valid, we classified the CER conclusion as still valid
2 Original conclusion is possibly out-of-date and this portion of the original report may need updating. This conclusion was reached if we found some new evidence that might change the CER conclusion, and/or a minority of responding experts assessed the CER conclusion as having new evidence that might change the conclusion, then we classified the CER conclusion as possibly out-of-date
3 Original conclusion is probably out-of-date and this portion of the original report may need updating. This conclusion was reached if we found substantial new evidence that might change the CER conclusion, and/or a majority of responding experts assessed the CER conclusion as having new evidence that might change the conclusion, then we classified the CER conclusion as probably out-of-date
4 Original conclusion is out-of-date. This conclusion was reached if we found new evidence that rendered the CER conclusion out-of-date or no longer applicable; we classified the CER conclusion as out-of-date. Recognizing that our literature searches were limited, we reserved this category only for situations where a limited search would produce prima facie evidence that a conclusion was out-of-date, such as the withdrawal of a drug or surgical device from the market, a black box warning from FDA, and so on
  1. Abbreviation: CER comparative effectiveness review, FDA Food and Drug Administration.
  2. *a pivotal trial is defined as trial that is published in one of the top five general medical journals or a trial whose sample size is at least triple that of the largest trial in the original systematic review.
  3. Legend: Table 1 presents the criteria used to determine if a conclusion is out of date within an SR (here CER). Criteria A1 to B2 come from the Ottawa method and criteria 1 to 4 are based on the RAND method.