Skip to main content

Table 1 Proposed steps to develop QA tools with examples drawn from existing tool development

From: A proposed framework for developing quality assessment tools

Stage QUADAS-2[8] ROBIS [9] PROBAST [10] ROBINS-I [11] RoB 2.0 [12]
Stage 1: initial steps
 1.1 Identify need Update for QUADAS New tool specifically for RoB in SR New tool specifically for prediction models New tool specifically for NRS of interventions Update to Cochrane RoB tool
 1.2 Obtain funding Part of MRC methodology programme grant MRC methodology project grant No external funding MIF grant from Cochrane and MRC methodology project grant MRC hubs project grant
 1.3 Assemble team Larger group (n = 18) Steering group (n = 10) + larger group (n = 19) Larger group (n = 40) Domain-based working groups and others (n = 35) Domain-based working groups and others (n = 38)
 1.4 Manage project Steering group (n = 9) Project group (n = 5) Steering group (n = 9) Steering group (n = 5) Steering group (n = 8)
 1.5 Define scope Diagnostic test accuracy Systematic reviews Prediction modelling (prognostic and diagnostic) Non-randomised studies of interventions Randomised controlled trials
  Bias + applicability
Domain based + signalling questions
Signalling questions phrased so yes indicates low risk of bias
Bias + relevance
Domain based + signalling questions
Signalling questions phrased so yes indicates low risk of bias
Bias + applicability
Domain based + signalling questions
Signalling questions phrased so yes indicates low risk of bias
Bias
Domain based + signalling questions
Bias
Domain based + signalling questions + algorithm for domain level ratings
Stage 2: tool development
 2.1. Generate items Starting point original QUADAS tool [7]
Evidence review of sources of bias and variation
Review of existing QA tools + how QA handled in existing overviews
Classification of MECIR guidance to identify sources of bias
Key publications on prognostic modelling studies
Work on TRIPOD [21], CHARMS [22] and QUADAS-2 [8]
Survey of Cochrane review groups and face-to-face meeting. Domain-based working groups Starting point draft tool based on original Cochrane RoB [23] tool. Evidence review of sources of bias.
 2.2. Agree items Face-to-face meeting Face-to-face meeting Web-based survey Face-to-face meeting + domain-based working group Face-to-face meeting + domain-based working group
 2.3. Produce first draft Steering group Steering group Steering group Domain-based working groups + steering group Domain-based working groups + steering group
 2.4. Pilot and refine Web-based survey + piloting in reviews Web-based survey + piloting in reviews and by steering group members on key reviews Web-based survey + piloting in reviews Domain-based working groups, web survey, cognitive interviews, piloting by working groups and external reviewers on key papers, event to pilot version 1.0 Domain-based working groups + 3 day piloting event + additional piloting of refined version
Stage 3: Dissemination
 3.1 Publication Single publication + guidance document as web supplement Single publication + guidance document as web supplement Primary publication + E&E document (in process) Single publication + guidance document on website Not yet published, guidance document on website
 3.2 Website www.quadas.org www.robis-tool.info Not yet available www.riskofbias.info www.riskofbias.info
 3.3 Uptake Recommended by multiple organisations including Cochrane, NICE and AHRQ, Recommended by Cochrane, NICE and Estonia Health Insurance Fund Recommended by Cochrane Prognosis Group Not yet, only recently available Not yet, only recently available
 3.4 Translations Italian and Japanese versions Estonian and Portuguese versions in preparation None to date None to date None to date