Skip to main content

Table 1 Data extracted from methods studies evaluating tools for assessing risk of bias in SRs

From: Toward a comprehensive evidence map of overview of systematic review methods: paper 2—risk of bias assessment; synthesis, presentation and summary of the findings; and assessment of the certainty of the evidence

Study design
Category
Data extracted
Primary methods studies
 Study characteristics First author, year
Title
Primary objective
 Description of primary methods studies Name of the included tools or measures
Type of assessment (e.g. assessment of reliability, content validity)
Content validity—methods of item generation
Content validity—comprehensiveness
Reliability—description of reliability testing
Tests of validity description of correlation coefficient testing
Other assessment (feasibility, acceptability, piloting)
 Risk of bias criteria Existence of a protocol
Method to select the sample of SRs to which the tool/measure was applied
Process for selecting the raters/assessors who applied the tool/measure
Pre-specified hypotheses for testing of validity
Systematic reviews of methods studies
 Study characteristics First author, year
Title
 Description of SRs of methods studies Primary objective
Number of included tools
Number of studies reporting on the included tools
Name of the included tools or measures (unnamed tools are identified by first author name and year of publication)
Content validity—reported method of development (e.g. item generation, expert assessment of content)
Reliability—description of reliability testing
Construct validity—description of any hypothesis testing. For example, how assessments from two or more tools relate, whether assessments relate to other factors (e.g. effect estimates or findings)
Other assessment (feasibility, acceptability, piloting)
 Risk of bias criteria (using three domains from the ROBIS tool [15]) Domain 1—study eligibility criteria: concerns regarding specification of eligibility criteria (low, high or unclear concern)
Domain 2—identification and selection of studies: concerns regarding methods used to identify and/or select studies (low, high or unclear concern)
Domain 3—data collection and study appraisal: concerns regarding methods used to collect data and appraise studies (low, high or unclear concern)
Overall judgment: Interpretation addresses all concerns identified in Domains 1–3, relevance of studies was appropriately considered, reviewers avoided emphasising results based on statistical significance.