Table 1 Data extracted from methods studies evaluating tools for assessing risk of bias in SRs
Study design Category | Data extracted |
|---|---|
Primary methods studies | |
Study characteristics | First author, year |
Title | |
Primary objective | |
Description of primary methods studies | Name of the included tools or measures |
Type of assessment (e.g. assessment of reliability, content validity) | |
Content validity—methods of item generation | |
Content validity—comprehensiveness | |
Reliability—description of reliability testing | |
Tests of validity description of correlation coefficient testing | |
Other assessment (feasibility, acceptability, piloting) | |
Risk of bias criteria | Existence of a protocol |
Method to select the sample of SRs to which the tool/measure was applied | |
Process for selecting the raters/assessors who applied the tool/measure | |
Pre-specified hypotheses for testing of validity | |
Systematic reviews of methods studies | |
Study characteristics | First author, year |
Title | |
Description of SRs of methods studies | Primary objective |
Number of included tools | |
Number of studies reporting on the included tools | |
Name of the included tools or measures (unnamed tools are identified by first author name and year of publication) | |
Content validity—reported method of development (e.g. item generation, expert assessment of content) | |
Reliability—description of reliability testing | |
Construct validity—description of any hypothesis testing. For example, how assessments from two or more tools relate, whether assessments relate to other factors (e.g. effect estimates or findings) | |
Other assessment (feasibility, acceptability, piloting) | |
Risk of bias criteria (using three domains from the ROBIS tool [15]) | Domain 1—study eligibility criteria: concerns regarding specification of eligibility criteria (low, high or unclear concern) |
Domain 2—identification and selection of studies: concerns regarding methods used to identify and/or select studies (low, high or unclear concern) | |
Domain 3—data collection and study appraisal: concerns regarding methods used to collect data and appraise studies (low, high or unclear concern) | |
Overall judgment: Interpretation addresses all concerns identified in Domains 1–3, relevance of studies was appropriately considered, reviewers avoided emphasising results based on statistical significance. | |