Skip to main content

Table 2 Quality appraisal of included studies of process

From: Interventions integrating health and academic interventions to prevent substance use and violence: a systematic review and synthesis of process evaluations

Intervention name Site Methods included steps to minimise bias in Findings Overall rating
Sampling methods Data collection Data analysis Supported by data Have breadth and depth Privilege young people’s perspectives Overall reliability and trustworthiness Overall usefulness answering our research questions
4Rs (Reading Writing, Respect and Resolution) New York, USA [39] No
Purposeful sampling but of only high-performing classrooms
Yes
Different instruments piloted and used; findings triangulated
Yes
Author verified data through ‘reflexive conversations’ and member-checking
Yes
Clear results followed methods
Yes
Very comprehensive data collected from a number of classrooms
No Medium
Selection of only high performers limits transferability of findings
High
Detailed information about implementation provided
DRACON Brisbane, Queensland and New South Wales, Australia [37] No
No detail provided
No
No detail provided on increasing rigour
No
No details were provided
No
No quotations present to support qualitative data and no links to questionnaire data
No
Good breadth of findings, but limited depth
Yes Low
Limited data on methods and links to results
Low
Limited detail on implementation
English classes (no name) Houston, USA [43] No
No detail provided
No
No detail provided
No
No detail provided
Yes
Survey results followed clearly; qualitative results presented without supporting quotations
Yes
Mixed methods enabled exploration of both breadth and depth
No Low
Limited detail on the rigour of methods used
Medium
One of the few studies in which integration was core to the study’s design and some good detail around implementation is provided
Hashish and Marijuana Haifa, Israel [40] No
No detail provided
No
No detail provided
No
No detail provided
No
Scant data were provided, and it was unclear how these were produced
No
Minimal findings reported
Yes Low
Poor reporting of methods and minimal results
Low
Lack of detail in findings restricted the use of this study
Infused-Life Skills Training PA, USA [42] No
No detail provided
Yes
Multiple methods and instruments used; findings triangulated
No
No detail provided
No
No primary data provided, only authors’ accounts of the data
Yes
Different aspects of implementation explored from students, teachers and administrators
No Low
Poor reporting of methods and minimal results
Medium
Paper provides interesting insights and is the only one to compare with non-integrated curriculum implementation, but detail on methods is lacking
Kids, Adults Together (KAT) Southeast Wales, UK [33] No
No detail provided
Yes
Multiple methods used at different data points to ensure comprehensive perspectives
Yes
Comparative coding used to refine analytical framework
Yes
Clear results followed methods
No
Good depth around acceptability, limited detail on other aspects of implementation
Yes Medium
Insufficient detail to determine possible bias introduced in sampling, but data collection and analysis seem appropriate
Low
Nothing about the integration of academic and health curricula in findings
Southeast Wales, UK
[44]
No
No detail provided
Yes
Comprehensive qualitative data was collected
Yes
Data were triangulated; constant comparison of data was done; and authors increased validity of instruments
Yes
Although actual quotations and results from process evaluation were limited
Yes
Data were collected on many aspects of implementation
No Medium
A lack of data on methods makes reliability impossible to ascertain
Medium
This study has interesting findings but would be better to see them grounded in primary data
Peaceful Panels Athens, USA [45] No
Convenience sample drawn from the author’s classroom
Yes
A range of methods used to collect data and an independent peer audited the author’s methods
Yes
Author employed reflexivity, debriefs with peers, and member-checking to increase robustness
Yes
Clear results followed methods
Yes
Considerable detail on a number of implementation factors reported
Yes Medium
Convenience sampling and (opinion of the study team) less-robust than possible analyses may limit trustworthiness
Medium
Detailed information about implementation processes, but limited information about influencing factors
Positive Action Hawaii [31] Yes
Sampling of schools was random, and there was an attempt to reach a census of all participating students
Yes
Validated tools that collected data around a variety of measures of implantation were used
Yes
Data were analysed using statistically appropriate methods
Yes
Clear results followed methods
Yes
Various features of implementation were explored in detail. These were generated from a large sample of diverse students. No qualitative data, however
Yes High
Methods were appropriate, efforts were made to increase rigour and the findings and interpretations lead clearly from the methods used
High
This paper gives good information about important aspects of implementation
Hawaii [32] No
Census of teachers attempted without success and no explanation provided
Yes
High reliability of tools used
Yes
Analysis were appropriate, and data were entirely quantitative
Yes
Clear results followed methods
Yes
Good range of process measures covered in considerable depth
No High
Study was well-conducted and statistically robust
High
Useful discussion of key implementation factors including the perspectives of implementers
Chicago [41] Yes
Relevant sampling criteria used with a very high response rate
Yes
Multiple data sources used and triangulated
Yes
Analytical approach was appropriate and robust
Yes
Clear results followed methods
Yes
Multiple data sources provided information about many aspects of implementation, in detail, with description of relationships between these
No Medium
Although methods were robust, there was no qualitative data to answer the more useful ‘why’ questions, particularly behind the relationships between implementation factors
High
Good data provided around multiple aspects of implementation
Promoting Alternative Thinking Strategies PA, USA [38] Yes
Sufficient detail provided; high (85%) response rate
Yes
Alpha-reliability coefficients acceptable and provided; other measures of validity lacking
Yes
Data analyses were appropriate
Yes
Clear results follow methods
No
Study limited to teachers’ psychological factors
No High
This is a well-conducted study
Medium
Although methodologically sound, comprehensive results are lacking
Roots of Empathy Western Australia [34] No
No detail provided
No
No detail provided
No
No detail provided
Yes
Clear results followed methods
No
Breadth around implementation from a teacher perspective, but little depth
No Low
A lack of methodological detail make trustworthiness questionable
Medium
Useful data on some aspects of implementation provided, but lacking methodological rigour
Western Canada and the Isle of Man, UK
[30]
Yes
Participants were from an ongoing RCT
Yes
Reliability of instruments was good
Yes
Data were merged to increase study power
Yes
Constructs were well-defined and studied. Slight bias to Canadian results
No
Lack of qualitative data
No High
This is a methodologically rigorous study
Medium
Focus on teacher characteristics and implementation is valuable, but qualitative findings are limited
Steps to Respect CA, USA
[36]
Yes
Participants are from an ongoing RCT; high response rate
Yes
Questionnaire had high face validity and reliability
Yes
Data analysis were appropriate
Yes
Although qualitative exploration was lacking
No
Concepts explored were limited
No High
This was a methodologically sound study
Medium
Useful data, but qualitative findings are limited
The Gatehouse Project Victoria, Australia
[35]
No
No detail provided on how participants were selected
Yes
Multiple methods used to collect data at multiple points in the year
No
No detail provided
Yes
Clear results followed methods
Yes
Multiple aspects of implementation were explored from multiple stakeholder perspectives
No Medium
More detail on methodological rigour would be required to make a fair assessment of robustness
High
Very useful data provided around implementation characteristics