Skip to main content

Table 5 Risk of bias for non-randomised studies using the Quality Assessment Tool for Quantitative Studies (QATSQ) tool

From: An updated systematic review of interventions to increase awareness of mental health and well-being in athletes, coaches, officials and parents

Study Selection bias Study design Confounders Blinding Data collection methods Withdrawals and dropouts Summary
Bapat et al. (2009) [59] 2 2 3 3 3 3  
Participants are very likely to be representative Cannot tell the percentage of participants who agreed Study is designated as a cohort analytic study There were gender and age differences that may have influenced the outcomes between participants and these were not controlled for in analysis Outcome assessors knew intervention status, and blinding of participants to research question is not described The validity and reliability of the instruments are not described Withdrawals and dropouts were not described Weak quality: as this study scored four weak ratings, the overall judgement is weak quality
Breslin et al. (2017) [8, 60] 2 1 1 2 1 3  
Participants are very likely to be representative Cannot tell percentage of participants who agreed Study is designated as a controlled clinical trial Confounders (gender, sport type) were similar across control and intervention groups Cannot tell if outcome assessors were aware of intervention status and cannot tell if intervention participants were aware of research question Tools were shown to be valid and reliable Cannot tell if there were withdrawals or dropouts Moderate quality: As this study scored one weak rating the overall judgement is moderate quality
Breslin et al. (2018) 1 1 2 2 1 3  
Participants are very likely to be representative. All participants agreed to participate Study is designated as a controlled clinical trial Age differences between groups may have acted as confounder. Other significant demographic differences were controlled for Cannot tell if outcome assessors were aware of intervention status and cannot tell if intervention participants were aware of research question Tools were shown to be valid and reliable Significant drop out rate was described and reasons provided Moderate quality; As this study scored one weak rating the overall judgement is moderate quality
Chow et al. (2020) [46] 1 2 2 3 1 1  
Participants are very likely to be representative. All participants agreed to participate Study is designated as a cohort analytic study No significant baseline differences between those who had mental health experience and those who had not therefore groups were combined for primary analysis Outcome assessors knew intervention status and blinding of participants to research question is not described Tools were shown to be valid and reliable 100% completion rate at follow-up Moderate quality; as this study scored one weak rating the overall judgement is moderate quality
Donohue et al. (2015) [62] 1 2 3 3 1 2  
Participants are very likely to be representative All participants agreed to participate Study is designated as a cohort analytic study There were gender, ethnic and age differences that may have influenced the direction of result. These were not controlled for in the analysis Outcome assessors knew intervention status, and the participants knew intended outcome of the research (i.e. developing intervention) The validity and reliability of the instruments is described There was a 70% follow-up rate from those that consented and completed the intervention Weak quality: as this study scored two weak ratings, the overall judgement is weak quality
Dowell et al. (2020) [48] 2 2 3 3 2 3  
Participants are somewhat likely to be representative, fee required may influence sample. All participants agreed to participate Study is designated as a cohort analytic study The requirement to control confounders was alluded to but the rationale behind adjustment was not sufficiently transparent Outcome assessors knew intervention status and blinding of participants to research question is not described Some tools were shown to be valid and reliable, low internal consistency was observed for measuring conduct problems Less than 50% of initial sample completed intervention Weak quality; as this study scored three weak ratings the overall judgement is weak quality
Dubuc-Charbonneau and Durand-Bush (2015) [49] 1 2 3 3 1 1  
Participants are very likely to be representative. All participants agreed to participate Study is designated as a cohort analytic study Confounding variables were not discussed Outcome assessors knew intervention status and blinding of participants to research question is not described Tools were shown to be valid and reliable 100% completion rate at follow-up Weak quality; as this study scored two weak ratings, the overall judgement is weak quality
Fogaca (2019) [50] 2 1 3 3 1 2  
Participants are somewhat likely to be representative. Risk of selection bias by removal of one team from intervention group data. Above 80% of participants agreed to participate Study is designated as a controlled clinical trial Study showed that there were no significant differences between intervention and control for mental health measures pre-test with the exception of depression, as a result the outlying team was removed from the data. No discussion of demographic differences (potential confounders) between intervention and control Outcome assessor knew intervention status and blinding of participants to research question is not described Tools were shown to be valid and reliable 60–79% completion rate Weak quality; as this study scored two weak ratings, the overall judgement is weak quality
Hurley et al. (2018) [18] 1 1 2 3 1 1  
Participants are very likely to be representative. All participants agreed to participate Study is designated as a controlled clinical trial The study deploys a ‘matched’ control group to attempt to control for confounding variables but no mention of whether this holds true Outcome assessors knew intervention status and blinding of participants to research question is not described Tools were shown to be valid and reliable > 80% completion rate at follow-up Moderate quality; as this study scored one weak rating, the overall judgement is moderate quality
Hurley et al. (2020) [52] 1 1 1 2 1 3  
Participants are very likely to be representative. All participants agreed to participate Study is designated as a controlled clinical trial Matched control trial to account for confounding variables. Covariates are adjusted for Cannot tell if outcome assessors were aware of intervention status and blinding of participants to research question is not described Tools were shown to be valid and reliable Retention of participants was low particularly in the control group Moderate quality; as this study scored one weak rating, the overall judgement is moderate quality
Laureano et al. (2014) [53] 2 1 2 3 2 1  
Participants are somewhat likely to be representative. All participants agreed to participate Study is designated as a controlled clinical trial Study corrected for pre-test differences. However, extraneous variables impacting cannot be ruled out Outcome assessors knew intervention status and blinding of participants to research question is not described Some tools were shown to be valid and reliable, FORQ results should be treated tentatively due to low internal consistency 100% completion rate at follow-up of intervention and control groups Moderate quality; as this study scored one weak rating, the overall judgement is moderate quality
Longshore and Sachs (2015) [64] 1 1 1 3 3 1  
Participants are very likely to be representative Above 80% of participants agreed to participate Study is designated as a controlled clinical trial. No significant differences were found between the groups before the intervention Outcome assessors knew intervention status, and the participants knew intended outcome of the research (i.e. benefits of mindfulness) The validity and reliability of the instruments is not described There was a > 80% follow-up rate from those that consented and completed the intervention Weak quality: as this study scored two weak ratings, the overall judgement is weak quality
Mohammed et al. (2018) [54] 2 1 1 3 2 1  
Participants are somewhat likely to be representative. >80% of participants agreed to participate Study is designated as a controlled clinical trial Confounders were similar across intervention and control group Outcome assessors knew intervention status and participants were not blinded to research question The tools deployed displayed varied levels of validity and reliability > 80% completion rate at follow-up Moderate quality; as this study scored one weak rating, the overall judgement is moderate quality
Pierce et al. (2010) [65] 2 2 3 3 3 2  
Participants are very likely to be representative Cannot tell the percentage of participants who agreed Study is designated as a cohort analytic study There were age and education differences that may have influenced the direction of result these were not controlled for in the analysis Outcome assessors knew intervention status, and the participants knew intended outcome of the research (i.e. respond to mental health problems) The validity and reliability of the instruments is not described There was a 66% follow-up rate from those that consented and completed the intervention Weak quality: as this study scored three weak ratings, the overall judgement is weak quality
Sebbens et al. (2016) [66] 1 1 1 3 3 1  
Participants are very likely to be representative Above 80% of participants agreed to participate Study is designated as a controlled clinical trial No significant demographic differences were found between the groups before the intervention Outcome assessors knew intervention status, and blinding of participants to research question is not described The validity and reliability of the instruments is not described There was a > 80% follow-up rate from those that consented and completed the intervention Weak quality: As this study scored two weak ratings, the overall judgement is weak quality
Slack et al. (2015) [67] 1 2 3 3 3 1  
Participants are very likely to be representative
Above 80% of participants agreed to participate
Study is designated as a cohort analytic study Confounding variables were not discussed Outcome assessors knew intervention status, and blinding of participants to research question is not described While one measure was referenced as valid and reliable, no information was reported on validity and reliability of another measure (RSMT) There was a > 80% follow-up rate from those that consented and completed the intervention Weak quality: As this study scored three weak ratings, the overall judgement is weak quality
Shannon et al. (2019) [56] 1 1 1 3 1 2  
Participants are very likely to be representative. All participants agreed to participate Study is designated as a controlled clinical trial Baseline measurements indicated that there were no significant differences between control and intervention group for study outcomes or gender. Age was significantly different but analysis showed it did not have a significant effect on outcomes Outcome assessors knew intervention status and blinding of participants to research question is not described Tools were shown to be valid and reliable There is no information provided about withdrawals or dropouts but Little’s MCAR analyses revealed data was missing at random and the expectation maximisation algorithm was used to estimate missing values Moderate quality; as this study scored one weak rating, the overall judgement is moderate quality
Tester et al. (1999) [68] 2 2 3 2 1 3  
Participants are very likely to be representative Cannot tell the percentage of participants who agreed Study is designated as a cohort analytic study Confounding variables were not discussed Cannot tell if outcome assessors were aware of intervention status Cannot tell if intervention participants were aware of research question Tools were referenced as valid and reliable Cannot tell if there were withdrawals or dropouts Weak quality: As this study scored two weak ratings, the overall judgement is weak quality
Vella et al. (2020) [57] 1 1 1 2 2 3  
Participants are very likely to be representative. All participants agreed to participate Study is designated as a controlled clinical trial Matched control to account for confounding variables. Baseline differences are highlighted and adjusted for Cannot tell if outcome assessors were aware of intervention status and blinding of participants to research question is not described Majority of tools were shown to be valid and reliable except low internal consistency for implicit beliefs scale A small proportion of participants completed the entire intervention per protocol Moderate quality; as this study scored one weak rating, the overall judgement is moderate quality
Vidic et al. (2018) [58] 1 2 3 3 1 3  
Participants are very likely to be representative. All participants agreed to participate Study is a cohort design Did not control for confounding variables Outcome assessor knew intervention status and blinding of participants to research question is not described Tool used was shown to be valid and reliable There is no information provided about withdrawals or dropouts Weak quality; as this study scored more than two weak ratings, the overall judgement is weak quality
Summary of bias across the studies Twelve studies were of strong quality and controlled for selection bias, the remaining 8 were of moderate quality Eleven studies were of strong quality for study design and the remaining 9 were of moderate quality There was a mixture of strong (n = 7), weak (n = 9) and moderate (n = 4) information provided on confounders Fifteen of the non-randomised studies were of weak quality for blinding participants and outcome assessors. 5 were of moderate quality Eleven of the non-randomised studies were of strong quality and referenced adequate reliability and validity for outcome measures, while 9 studies used tools of varied validity There was a mixture of strong (n = 8), weak (n = 8) and moderate (n = 4) for the researchers’ disclosure of follow-up rates and dropouts Nine studies were deemed to be of moderate quality and 11 were of weak quality
  1. 1 = strong; 2 = moderate; 3 = weak