Skip to main content

Table 5 Risk of bias for non-randomised studies using the Quality Assessment Tool for Quantitative Studies (QATSQ) tool

From: An updated systematic review of interventions to increase awareness of mental health and well-being in athletes, coaches, officials and parents

Study

Selection bias

Study design

Confounders

Blinding

Data collection methods

Withdrawals and dropouts

Summary

Bapat et al. (2009) [59]

2

2

3

3

3

3

 

Participants are very likely to be representative Cannot tell the percentage of participants who agreed

Study is designated as a cohort analytic study

There were gender and age differences that may have influenced the outcomes between participants and these were not controlled for in analysis

Outcome assessors knew intervention status, and blinding of participants to research question is not described

The validity and reliability of the instruments are not described

Withdrawals and dropouts were not described

Weak quality: as this study scored four weak ratings, the overall judgement is weak quality

Breslin et al. (2017) [8, 60]

2

1

1

2

1

3

 

Participants are very likely to be representative Cannot tell percentage of participants who agreed

Study is designated as a controlled clinical trial

Confounders (gender, sport type) were similar across control and intervention groups

Cannot tell if outcome assessors were aware of intervention status and cannot tell if intervention participants were aware of research question

Tools were shown to be valid and reliable

Cannot tell if there were withdrawals or dropouts

Moderate quality: As this study scored one weak rating the overall judgement is moderate quality

Breslin et al. (2018)

1

1

2

2

1

3

 

Participants are very likely to be representative. All participants agreed to participate

Study is designated as a controlled clinical trial

Age differences between groups may have acted as confounder. Other significant demographic differences were controlled for

Cannot tell if outcome assessors were aware of intervention status and cannot tell if intervention participants were aware of research question

Tools were shown to be valid and reliable

Significant drop out rate was described and reasons provided

Moderate quality; As this study scored one weak rating the overall judgement is moderate quality

Chow et al. (2020) [46]

1

2

2

3

1

1

 

Participants are very likely to be representative. All participants agreed to participate

Study is designated as a cohort analytic study

No significant baseline differences between those who had mental health experience and those who had not therefore groups were combined for primary analysis

Outcome assessors knew intervention status and blinding of participants to research question is not described

Tools were shown to be valid and reliable

100% completion rate at follow-up

Moderate quality; as this study scored one weak rating the overall judgement is moderate quality

Donohue et al. (2015) [62]

1

2

3

3

1

2

 

Participants are very likely to be representative All participants agreed to participate

Study is designated as a cohort analytic study

There were gender, ethnic and age differences that may have influenced the direction of result. These were not controlled for in the analysis

Outcome assessors knew intervention status, and the participants knew intended outcome of the research (i.e. developing intervention)

The validity and reliability of the instruments is described

There was a 70% follow-up rate from those that consented and completed the intervention

Weak quality: as this study scored two weak ratings, the overall judgement is weak quality

Dowell et al. (2020) [48]

2

2

3

3

2

3

 

Participants are somewhat likely to be representative, fee required may influence sample. All participants agreed to participate

Study is designated as a cohort analytic study

The requirement to control confounders was alluded to but the rationale behind adjustment was not sufficiently transparent

Outcome assessors knew intervention status and blinding of participants to research question is not described

Some tools were shown to be valid and reliable, low internal consistency was observed for measuring conduct problems

Less than 50% of initial sample completed intervention

Weak quality; as this study scored three weak ratings the overall judgement is weak quality

Dubuc-Charbonneau and Durand-Bush (2015) [49]

1

2

3

3

1

1

 

Participants are very likely to be representative. All participants agreed to participate

Study is designated as a cohort analytic study

Confounding variables were not discussed

Outcome assessors knew intervention status and blinding of participants to research question is not described

Tools were shown to be valid and reliable

100% completion rate at follow-up

Weak quality; as this study scored two weak ratings, the overall judgement is weak quality

Fogaca (2019) [50]

2

1

3

3

1

2

 

Participants are somewhat likely to be representative. Risk of selection bias by removal of one team from intervention group data. Above 80% of participants agreed to participate

Study is designated as a controlled clinical trial

Study showed that there were no significant differences between intervention and control for mental health measures pre-test with the exception of depression, as a result the outlying team was removed from the data. No discussion of demographic differences (potential confounders) between intervention and control

Outcome assessor knew intervention status and blinding of participants to research question is not described

Tools were shown to be valid and reliable

60–79% completion rate

Weak quality; as this study scored two weak ratings, the overall judgement is weak quality

Hurley et al. (2018) [18]

1

1

2

3

1

1

 

Participants are very likely to be representative. All participants agreed to participate

Study is designated as a controlled clinical trial

The study deploys a ‘matched’ control group to attempt to control for confounding variables but no mention of whether this holds true

Outcome assessors knew intervention status and blinding of participants to research question is not described

Tools were shown to be valid and reliable

> 80% completion rate at follow-up

Moderate quality; as this study scored one weak rating, the overall judgement is moderate quality

Hurley et al. (2020) [52]

1

1

1

2

1

3

 

Participants are very likely to be representative. All participants agreed to participate

Study is designated as a controlled clinical trial

Matched control trial to account for confounding variables. Covariates are adjusted for

Cannot tell if outcome assessors were aware of intervention status and blinding of participants to research question is not described

Tools were shown to be valid and reliable

Retention of participants was low particularly in the control group

Moderate quality; as this study scored one weak rating, the overall judgement is moderate quality

Laureano et al. (2014) [53]

2

1

2

3

2

1

 

Participants are somewhat likely to be representative. All participants agreed to participate

Study is designated as a controlled clinical trial

Study corrected for pre-test differences. However, extraneous variables impacting cannot be ruled out

Outcome assessors knew intervention status and blinding of participants to research question is not described

Some tools were shown to be valid and reliable, FORQ results should be treated tentatively due to low internal consistency

100% completion rate at follow-up of intervention and control groups

Moderate quality; as this study scored one weak rating, the overall judgement is moderate quality

Longshore and Sachs (2015) [64]

1

1

1

3

3

1

 

Participants are very likely to be representative Above 80% of participants agreed to participate

Study is designated as a controlled clinical trial.

No significant differences were found between the groups before the intervention

Outcome assessors knew intervention status, and the participants knew intended outcome of the research (i.e. benefits of mindfulness)

The validity and reliability of the instruments is not described

There was a > 80% follow-up rate from those that consented and completed the intervention

Weak quality: as this study scored two weak ratings, the overall judgement is weak quality

Mohammed et al. (2018) [54]

2

1

1

3

2

1

 

Participants are somewhat likely to be representative. >80% of participants agreed to participate

Study is designated as a controlled clinical trial

Confounders were similar across intervention and control group

Outcome assessors knew intervention status and participants were not blinded to research question

The tools deployed displayed varied levels of validity and reliability

> 80% completion rate at follow-up

Moderate quality; as this study scored one weak rating, the overall judgement is moderate quality

Pierce et al. (2010) [65]

2

2

3

3

3

2

 

Participants are very likely to be representative Cannot tell the percentage of participants who agreed

Study is designated as a cohort analytic study

There were age and education differences that may have influenced the direction of result these were not controlled for in the analysis

Outcome assessors knew intervention status, and the participants knew intended outcome of the research (i.e. respond to mental health problems)

The validity and reliability of the instruments is not described

There was a 66% follow-up rate from those that consented and completed the intervention

Weak quality: as this study scored three weak ratings, the overall judgement is weak quality

Sebbens et al. (2016) [66]

1

1

1

3

3

1

 

Participants are very likely to be representative Above 80% of participants agreed to participate

Study is designated as a controlled clinical trial

No significant demographic differences were found between the groups before the intervention

Outcome assessors knew intervention status, and blinding of participants to research question is not described

The validity and reliability of the instruments is not described

There was a > 80% follow-up rate from those that consented and completed the intervention

Weak quality: As this study scored two weak ratings, the overall judgement is weak quality

Slack et al. (2015) [67]

1

2

3

3

3

1

 

Participants are very likely to be representative

Above 80% of participants agreed to participate

Study is designated as a cohort analytic study

Confounding variables were not discussed

Outcome assessors knew intervention status, and blinding of participants to research question is not described

While one measure was referenced as valid and reliable, no information was reported on validity and reliability of another measure (RSMT)

There was a > 80% follow-up rate from those that consented and completed the intervention

Weak quality: As this study scored three weak ratings, the overall judgement is weak quality

Shannon et al. (2019) [56]

1

1

1

3

1

2

 

Participants are very likely to be representative. All participants agreed to participate

Study is designated as a controlled clinical trial

Baseline measurements indicated that there were no significant differences between control and intervention group for study outcomes or gender. Age was significantly different but analysis showed it did not have a significant effect on outcomes

Outcome assessors knew intervention status and blinding of participants to research question is not described

Tools were shown to be valid and reliable

There is no information provided about withdrawals or dropouts but Little’s MCAR analyses revealed data was missing at random and the expectation maximisation algorithm was used to estimate missing values

Moderate quality; as this study scored one weak rating, the overall judgement is moderate quality

Tester et al. (1999) [68]

2

2

3

2

1

3

 

Participants are very likely to be representative Cannot tell the percentage of participants who agreed

Study is designated as a cohort analytic study

Confounding variables were not discussed

Cannot tell if outcome assessors were aware of intervention status Cannot tell if intervention participants were aware of research question

Tools were referenced as valid and reliable

Cannot tell if there were withdrawals or dropouts

Weak quality: As this study scored two weak ratings, the overall judgement is weak quality

Vella et al. (2020) [57]

1

1

1

2

2

3

 

Participants are very likely to be representative. All participants agreed to participate

Study is designated as a controlled clinical trial

Matched control to account for confounding variables. Baseline differences are highlighted and adjusted for

Cannot tell if outcome assessors were aware of intervention status and blinding of participants to research question is not described

Majority of tools were shown to be valid and reliable except low internal consistency for implicit beliefs scale

A small proportion of participants completed the entire intervention per protocol

Moderate quality; as this study scored one weak rating, the overall judgement is moderate quality

Vidic et al. (2018) [58]

1

2

3

3

1

3

 

Participants are very likely to be representative. All participants agreed to participate

Study is a cohort design

Did not control for confounding variables

Outcome assessor knew intervention status and blinding of participants to research question is not described

Tool used was shown to be valid and reliable

There is no information provided about withdrawals or dropouts

Weak quality; as this study scored more than two weak ratings, the overall judgement is weak quality

Summary of bias across the studies

Twelve studies were of strong quality and controlled for selection bias, the remaining 8 were of moderate quality

Eleven studies were of strong quality for study design and the remaining 9 were of moderate quality

There was a mixture of strong (n = 7), weak (n = 9) and moderate (n = 4) information provided on confounders

Fifteen of the non-randomised studies were of weak quality for blinding participants and outcome assessors. 5 were of moderate quality

Eleven of the non-randomised studies were of strong quality and referenced adequate reliability and validity for outcome measures, while 9 studies used tools of varied validity

There was a mixture of strong (n = 8), weak (n = 8) and moderate (n = 4) for the researchers’ disclosure of follow-up rates and dropouts

Nine studies were deemed to be of moderate quality and 11 were of weak quality

  1. 1 = strong; 2 = moderate; 3 = weak