Table 2 Quality appraisal of included studies of process
Intervention name | Site | Methods included steps to minimise bias in | Findings | Overall rating | |||||
|---|---|---|---|---|---|---|---|---|---|
Sampling methods | Data collection | Data analysis | Supported by data | Have breadth and depth | Privilege young people’s perspectives | Overall reliability and trustworthiness | Overall usefulness answering our research questions | ||
4Rs (Reading Writing, Respect and Resolution) | New York, USA [39] | No Purposeful sampling but of only high-performing classrooms | Yes Different instruments piloted and used; findings triangulated | Yes Author verified data through ‘reflexive conversations’ and member-checking | Yes Clear results followed methods | Yes Very comprehensive data collected from a number of classrooms | No | Medium Selection of only high performers limits transferability of findings | High Detailed information about implementation provided |
DRACON | Brisbane, Queensland and New South Wales, Australia [37] | No No detail provided | No No detail provided on increasing rigour | No No details were provided | No No quotations present to support qualitative data and no links to questionnaire data | No Good breadth of findings, but limited depth | Yes | Low Limited data on methods and links to results | Low Limited detail on implementation |
English classes (no name) | Houston, USA [43] | No No detail provided | No No detail provided | No No detail provided | Yes Survey results followed clearly; qualitative results presented without supporting quotations | Yes Mixed methods enabled exploration of both breadth and depth | No | Low Limited detail on the rigour of methods used | Medium One of the few studies in which integration was core to the study’s design and some good detail around implementation is provided |
Hashish and Marijuana | Haifa, Israel [40] | No No detail provided | No No detail provided | No No detail provided | No Scant data were provided, and it was unclear how these were produced | No Minimal findings reported | Yes | Low Poor reporting of methods and minimal results | Low Lack of detail in findings restricted the use of this study |
Infused-Life Skills Training | PA, USA [42] | No No detail provided | Yes Multiple methods and instruments used; findings triangulated | No No detail provided | No No primary data provided, only authors’ accounts of the data | Yes Different aspects of implementation explored from students, teachers and administrators | No | Low Poor reporting of methods and minimal results | Medium Paper provides interesting insights and is the only one to compare with non-integrated curriculum implementation, but detail on methods is lacking |
Kids, Adults Together (KAT) | Southeast Wales, UK [33] | No No detail provided | Yes Multiple methods used at different data points to ensure comprehensive perspectives | Yes Comparative coding used to refine analytical framework | Yes Clear results followed methods | No Good depth around acceptability, limited detail on other aspects of implementation | Yes | Medium Insufficient detail to determine possible bias introduced in sampling, but data collection and analysis seem appropriate | Low Nothing about the integration of academic and health curricula in findings |
Southeast Wales, UK [44] | No No detail provided | Yes Comprehensive qualitative data was collected | Yes Data were triangulated; constant comparison of data was done; and authors increased validity of instruments | Yes Although actual quotations and results from process evaluation were limited | Yes Data were collected on many aspects of implementation | No | Medium A lack of data on methods makes reliability impossible to ascertain | Medium This study has interesting findings but would be better to see them grounded in primary data | |
Peaceful Panels | Athens, USA [45] | No Convenience sample drawn from the author’s classroom | Yes A range of methods used to collect data and an independent peer audited the author’s methods | Yes Author employed reflexivity, debriefs with peers, and member-checking to increase robustness | Yes Clear results followed methods | Yes Considerable detail on a number of implementation factors reported | Yes | Medium Convenience sampling and (opinion of the study team) less-robust than possible analyses may limit trustworthiness | Medium Detailed information about implementation processes, but limited information about influencing factors |
Positive Action | Hawaii [31] | Yes Sampling of schools was random, and there was an attempt to reach a census of all participating students | Yes Validated tools that collected data around a variety of measures of implantation were used | Yes Data were analysed using statistically appropriate methods | Yes Clear results followed methods | Yes Various features of implementation were explored in detail. These were generated from a large sample of diverse students. No qualitative data, however | Yes | High Methods were appropriate, efforts were made to increase rigour and the findings and interpretations lead clearly from the methods used | High This paper gives good information about important aspects of implementation |
Hawaii [32] | No Census of teachers attempted without success and no explanation provided | Yes High reliability of tools used | Yes Analysis were appropriate, and data were entirely quantitative | Yes Clear results followed methods | Yes Good range of process measures covered in considerable depth | No | High Study was well-conducted and statistically robust | High Useful discussion of key implementation factors including the perspectives of implementers | |
Chicago [41] | Yes Relevant sampling criteria used with a very high response rate | Yes Multiple data sources used and triangulated | Yes Analytical approach was appropriate and robust | Yes Clear results followed methods | Yes Multiple data sources provided information about many aspects of implementation, in detail, with description of relationships between these | No | Medium Although methods were robust, there was no qualitative data to answer the more useful ‘why’ questions, particularly behind the relationships between implementation factors | High Good data provided around multiple aspects of implementation | |
Promoting Alternative Thinking Strategies | PA, USA [38] | Yes Sufficient detail provided; high (85%) response rate | Yes Alpha-reliability coefficients acceptable and provided; other measures of validity lacking | Yes Data analyses were appropriate | Yes Clear results follow methods | No Study limited to teachers’ psychological factors | No | High This is a well-conducted study | Medium Although methodologically sound, comprehensive results are lacking |
Roots of Empathy | Western Australia [34] | No No detail provided | No No detail provided | No No detail provided | Yes Clear results followed methods | No Breadth around implementation from a teacher perspective, but little depth | No | Low A lack of methodological detail make trustworthiness questionable | Medium Useful data on some aspects of implementation provided, but lacking methodological rigour |
Western Canada and the Isle of Man, UK [30] | Yes Participants were from an ongoing RCT | Yes Reliability of instruments was good | Yes Data were merged to increase study power | Yes Constructs were well-defined and studied. Slight bias to Canadian results | No Lack of qualitative data | No | High This is a methodologically rigorous study | Medium Focus on teacher characteristics and implementation is valuable, but qualitative findings are limited | |
Steps to Respect | CA, USA [36] | Yes Participants are from an ongoing RCT; high response rate | Yes Questionnaire had high face validity and reliability | Yes Data analysis were appropriate | Yes Although qualitative exploration was lacking | No Concepts explored were limited | No | High This was a methodologically sound study | Medium Useful data, but qualitative findings are limited |
The Gatehouse Project | Victoria, Australia [35] | No No detail provided on how participants were selected | Yes Multiple methods used to collect data at multiple points in the year | No No detail provided | Yes Clear results followed methods | Yes Multiple aspects of implementation were explored from multiple stakeholder perspectives | No | Medium More detail on methodological rigour would be required to make a fair assessment of robustness | High Very useful data provided around implementation characteristics |