1. Comparison to an ‘ideal’ randomized controlled trial (RCT) | |
RCTs are not available for exposure studies and, therefore, not relevant to decision makers who must rely on observational studies of exposures. | |
Assessing observational studies based on RCTs results in a default rating of high risk of bias | |
Some of the questions derived from evaluating RCTs of interventions are inappropriate or impossible to apply for observational studies. | |
Sources of bias specific to observational studies may not be captured by comparison to an RCT. | |
2. Inadequate assessment of bias related to confounding | |
Does not capture bias related to over-adjustment for confounders or inappropriate modelling of confounders. | |
Does not capture advantages of newer statistical methods used for control for confounding. | |
Clearer guidance is needed on method for identifying confounders. | |
Does not differentiate between confounders, co-exposures and complex exposures. | |
3. Inadequate assessment of bias related to measurement of exposure | |
Assessment is limited to validity and reliability of the measurement, and these concepts are not clearly defined. | |
4. Use of an overall risk of bias rating | |
Does not distinguish between studies that have a ‘serious’ risk of bias in one domain and those that have multiple ‘serious’ risks of bias. | |
Assumes all risk of bias domains are weighted equally. | |
5. Additional risks of bias relevant to observational studies are not assessed (e.g. funding source) | |
6. Signalling questions | |
Do not consistently help the raters come to a consensus on how to rate a bias domain. | |
Specific questions unclear or confusing. | |
7. Practical considerations | |
Time-consuming to use. | |
There are limitations of using a single tool to rate different study designs. |