Skip to main content

Advertisement

Interventions integrating health and academic interventions to prevent substance use and violence: a systematic review and synthesis of process evaluations

Abstract

Background

Within increasingly constrained school timetables, interventions that integrate academic and health education to reduce substance use and violence may hold promise as a category of intervention that can positively affect both academic and health outcomes. There are no current systematic reviews exploring the effectiveness of such interventions or factors that affect their implementation.

Methods

A total of 19 bibliographic databases and 32 websites were searched. References were also extracted from the reference lists of included studies, and experts and authors were contacted to identify relevant studies.

We included reports with no restrictions on language or date. References were screened on title/abstract and those not thus excluded were screened on full report. Data extraction and appraisal followed the Critical Appraisal Skills Programme, Evidence for Policy and Practice Information and Co-ordinating Centre and Cochrane tools. Extracted process data were qualitatively meta-synthesised for common themes.

Results

Seventy-eight thousand four hundred fifty-one unique references were identified, and 62 reports were included. A total of 16 reports (reporting on 15 studies of 12 interventions) evaluated process. Key facilitators of integrated academic and health curricula were supportive senior management and alignment of the intervention with school ethos; a positive teaching environment, including positive perceptions around the ability to be flexible in the adaptation and delivery of integrated academic and health curricula; positive pre-existing student and teacher attitudes towards intervention content; and parental support of interventions, largely through reinforcement of messaging at home. Important barriers were over-burdened teachers, with little time to learn and implement integrated curricula.

Conclusion

Several useful facilitating and inhibiting factors linked to the implementation of interventions that integrate academic and health education for reduced substance use and/or violence were identified, providing tentative but insightful evidence of context-specific issues that may impact intervention success. However, overall, there is still a considerable gap in our understanding of how to achieve the successful implementation of these interventions.

Background

Schools have long played a role in promoting health among students [1,2,3,4,5]. However, schools in many countries now dedicate less curriculum time to health-related programming due to increasing pressures to meet academic performance standards, which place constraints on school schedules [6,7,8]. One way to maintain health-promoting programmes that is receiving increasing attention is through the integration of health and academic education curricula [9, 10]. To assess our current understanding of these curricula, we carried out a systematic review of such interventions aiming to prevent tobacco, drug or alcohol use (henceforth referred to as ‘substance use’) and/or violence. To our knowledge, this review is the first of its kind. As part of the systematic review, theories of change, process and outcome evaluations were synthesised.

Our ongoing synthesis of outcome evaluations will assess the effectiveness of interventions that integrate academic and health education in reducing substance use and violence outcomes. Our synthesis of theories of change is reported in detail elsewhere (currently under review). Briefly, the theory synthesis established that this category of intervention aims not only to integrate the teaching of health and academic education but also to bridge the relationships between staff and students so that affective bonds are strengthened, teachers serve more effectively as role models and students become more engaged in school. Many interventions also strive to generalise learning beyond the classroom to ensure that messages about health and academic education coming from the wider school and families are consistent with those taught in class and for reinforcement of knowledge and skills at multiple levels. The curricula and associated intervention components are further intended to normalise students’ positive behaviours to influence the development of social and emotional skills. These include, for example, self-management, empathy, communication and conflict resolution. Through these mechanisms, it is hypothesised that students will be less inclined to use substances, violence and aggression will decrease and academic performance will improve. To extend the usefulness of our review and to facilitate the design of future interventions that integrate academic and health education, a synthesis of factors affecting implementation of these interventions, documented in process evaluations, was undertaken.

Recent UK Medical Research Council guidance on process evaluations of complex interventions [11] stresses that these are useful in exploring what factors facilitate success. The process of designing more theoretically driven health improvement interventions has been hindered by the dominant paradigm within evidence syntheses, which is to focus on synthesising only quantitative studies answering questions about ‘what works’ [12]. Through synthesis of evidence on intervention processes, evaluators can develop hypotheses about the contexts within which interventions might be implemented and in which intervention mechanisms of action may produce intended outcomes, alongside findings about what works [13].

Although there are no existing syntheses of process data focused specifically on school-based interventions that integrate health and academic education, those examining the delivery of school-based health promotion interventions more generally can be found in the literature [14,15,16,17,18,19,20,21]. These identify constraining and facilitating factors operating at the school and class level, including the acceptability of the intervention to school staff and the adequacy of support for delivery. However, these factors are inconsistently defined and explored, making synthesis across studies challenging [22]. Theoretical frameworks also offer some suggestions as to what factors are likely to determine successful implementation. For example, May and Finch present normalisation process theory as a framework for understanding the sustainability of intervention implementation, suggesting a number of key factors: intervention coherence (people can make sense of a new practice), cognitive participation (people are willing to participate in a new practice), collective action (people are willing to take on the work required for the new practice) and reflexive monitoring (people are prepared to monitor the practice) [23].

However, no existing syntheses or theoretical frameworks have identified the factors that are likely to determine successful implementation of interventions integrating health and academic education in schools. This gap is likely because such integration is not seen as a focal component of the design of many interventions that use it, but rather, something that has emerged due to practical considerations. This may be one of the main reasons why these interventions remain under-developed. Therefore, we aimed to identify, appraise and synthesise available evidence from process evaluations to address the following research question: what characteristics of interventions, deliverers, participants and school contexts facilitate or limit successful implementation and receipt of interventions integrating health and academic interventions to prevent substance use and violence?

Methods

Review methods

Our overall review synthesised evidence on the theory of change, implementation and outcomes of interventions integrating health and academic interventions to prevent substance use and violence. Full methods are reported in a protocol included as a web appendix. The review followed PRISMA guidelines [24]. This paper reports on the synthesis of evidence on implementation. To be included in this synthesis, studies evaluated interventions delivered in classroom settings within mainstream public or private schools in regular school hours and integrating academic and health education to prevent substance use or violence among general populations of students aged 4–18 years. Included studies reported on the planning, delivery, receipt or causal pathways of interventions using quantitative and/or qualitative data. In October and November 2015, we consulted experts in the field of health education and social-emotional learning in schools to obtain their suggestions for possible included interventions or individual studies. From 18 November to 22 December 2015, we searched 19 health, social science and education databases. Searching of 32 websites and reference lists of relevant studies for further references followed between 12 and 23 January 2016. After carrying out the sample screen of 100 references to ensure more than 90% agreement, four reviewers independently screened the complete list of all possible included records on title and abstract. The full text of all records retained after this process were read in full by two reviewers to generate a final list of included studies that could answer at least one of our research questions (see Additional file 1 for further details).

Data extraction and quality appraisal

We extracted data using a modified version of an existing tool [25] including items on study location; intervention/components, development and delivery; timing of delivery and evaluation; provider characteristics; target population; sampling and sample characteristics; data collection and analysis; and findings relevant to our review including verbatim quotes, author descriptions and interpretations of the findings. After piloting and refinement, two reviewers working independently extracted data from study reports and then met to agree on coding.

The reliability and usefulness of process evaluations was assessed by two reviewers using a standard tool for process studies—which has been widely applied in systematic reviews and informed by principles of qualitative research—[26] including the following six criteria: whether the sampling strategy was indicated; whether data collection methods were indicated—including any statements around increasing rigour of data collection; the degree of data analysis—including any statements around efforts made to improve reliability of findings and reduce bias; the extent to which the study findings were grounded in the data; the extent to which the study privileged the perspectives of intervention participants; and the breadth and depth of findings. Studies were assigned two types of ‘weight of evidence’ based on the reliability or trustworthiness of the findings and the usefulness of the findings for shedding light on factors relating to the research questions. Study reliability was judged as high when steps were taken to ensure rigour in at least four criteria, as medium when addressing only three and low when addressing two or fewer. To achieve a rating of ‘high’ usefulness, studies needed to be judged to have privileged the perspectives of intervention participants and to present findings that achieve both breadth and depth. Studies that were rated as ‘medium’ usefulness only partially met this criterion, and ‘low’ rated studies were judged to have sufficient but limited findings. Quality was used to determine the qualitative weight given to findings in our synthesis, with none of the themes represented solely by studies judged as low on both dimensions.

Process evaluation data synthesis

Process evaluations commonly report qualitative, quantitative or mixed results. We anticipated that the quantitative data presented in included studies would address diverse questions and would therefore be too heterogeneous to meta-analyse statistically. Instead, textual reports of quantitative results were subject to thematic synthesis [27,28,29] after first checking that they were consistent with the quantitative data presented in the study reports. Studies were first read and re-read by two reviewers. The two reviewers then carried out line-by-line coding of process data in NVivo 11, developing inductive codes from these process data. Coding focused on textual reports which included verbatim qualitative data excerpts and author interpretations of these. Summaries of quantitative results were also coded in this manner after first checking that they were consistent with the quantitative data presented in the studies.

Each reviewer developed an emerging coding structure of hierarchically arranged codes applied in the course of the analysis. The two reviewers then compared their coding to agree on a common structure that formed the basis for the synthesis. As the overall analysis was developed, the reviewers referred to tables summarising the methodological quality of each study to ensure the synthesis reflected study quality.

Results

Search results

As per Fig. 1, after removing duplicates, 76,971 references were identified from the search (Additional file 2). From these, we included 16 relevant process evaluation reports that answered our research question on characteristics affecting implementation. These 16 reports presented data on15 empirical studies. One report (Hanson [30]) presented data on two separate studies. Two studies were each reported via two linked studies (Beets 2007 and 2008 [31, 32]; and Rothwell and Segrott and Segrott et al [33, 34]). There were 12 interventions reported on within these papers. A summary of all included studies of process and interventions is given in Table 1.

Fig. 1
figure1

Flow of studies in the review

Table 1 Summary of interventions reporting on process

Characteristics and quality of process evaluations

Of the 15 empirical studies, eight were conducted in the USA, three in Australia, two in the UK, one in Canada and one in Israel. Of the 12 interventions these studies summarised, four took place in primary or elementary schools, five in high or secondary schools and three in both (Table 1). Quality assessment is detailed in Table 2. Study reliability and usefulness varied. Only five reports were judged highly reliable and trustworthy, and five reports provided insights of ‘high’ value in answering our research questions. Six and five reports were respectively judged of ‘medium’ and ‘low’ reliability and trustworthiness.

Table 2 Quality appraisal of included studies of process

Thematic synthesis of process evaluations

Five overarching thematic areas emerged, with one or more sub-themes related to implementation. These areas are support from senior school staff, teachers’ immediate working environment, teacher attitudes towards intervention characteristics, student attitudes towards intervention characteristics and parental support. These themes and their sub-themes are described below.

Support from senior school staff

Support from school managers and other senior staff, including administrators, was cited as a key driver of successful intervention implementation by eight authors (reporting on seven interventions) [30, 34, 32, 35,36,37,38,39], and two sub-themes were identified in the data.

A positive, supportive school climate that aligns with intervention goals was a facilitator

A supportive school climate is not only one in which school and intervention ethos overlap, but also one in which school managers are invested and consistently active in the intervention. In the case of the latter, provision of mentoring and coaching to teachers involved in intervention curricula and committing dedicated time and resources to the curricula was important. Consequently, teachers were more likely to feel a sense of support and connectedness to the school.

Although of medium reliability, findings from the Gatehouse Project process evaluation were deemed highly useful.

Ongoing practical support from leadership has been acknowledged as important for mainstreaming the promotion of emotional well-being through promoting greater connections between learning, classroom practices, and student well-being. [One teacher recalls], ‘The support of my principal has to come number one…getting the time on the timetable, setting up a team, [that] can’t happen unless you’ve got someone in administration that thinks it’s a great idea.’ [35], p.378

Furthermore, a sense of connectedness to the school meant that teachers felt aligned with the school’s decision to engage with the intervention in question, which helped to encourage teachers’ beliefs in and acceptance of their responsibility to teach the respective curriculum.

From one of the three highly reliable and useful studies of the Positive Action intervention in the USA, Beets et al. reported that:

school leadership should develop a culture that encourages a shared and collective vision among staff and administration, is supportive of new innovations, and is aligned with the core values and concepts a given program is promoting … Perceptions of school climate were directly related to the beliefs teachers held about prevention/[social-emotional learning] and the attitudes teachers had towards [Positive Action]. [34], pp. 272–73

A political and administrative environment that is amenable to an integrated curriculum is necessary

Support from ‘higher-up’ in terms of intervention alignment with political priorities, leading to dedicated policies and funding to facilitate an intervention’s implementation, was important for its success.

Despite being judged a lower quality study within our appraisal, implementers of Roots of Empathy in Western Australia noted that:

Because of the financial support of [the Department of Education and Training] and its coordination of training, the program was successfully implemented. It is essential, however, that there is a strong policy and resourcing commitment to effectively sustain [the program]. [34], p.68

Together, the factors identified in the two sub-themes were regarded as promoting greater implementation fidelity of integrated academic and health curricula.

Teachers’ immediate working environment

Teachers’ perceptions of their teaching environment as one that would be amenable to the intervention increased their own motivation for intervention delivery, with direct impact on implementation. Three sub-themes on this subject emerged from 10 studies of nine interventions [34, 40, 41, 35, 37,38,39, 42,43,44].

Teachers working collaboratively and learning from one another was a facilitator

For example, within the Positive Action intervention, [31, 32, 41] successful implementation was associated with teacher perceptions of their schools having an innovative culture and strong relationships between teaching staff. Authors suggested that these findings were due to schools with a capacity for innovation being perceived as more open to change and to new approaches, such as those championed by new interventions, which gave teachers more freedom to explore new programmatic areas [32, 41]. Strong relationships between teachers in Positive Action and other interventions were linked to a sense of mutual support and connectedness that teachers felt would help them to optimise intervention delivery [35, 42, 43].

Despite its lower quality rating in our appraisal, Bechtel et al.’s evaluation of the Infused-Life Skills Training intervention in the USA raised a number of useful insights including reports that:

the first year participating teachers were especially helpful with recruiting and supporting new teachers in the program … they informally shared their experience with their fellow teachers, increasing interest and awareness of the program. They also gave examples of their lesson plans and discussed the importance of coaching and behavioral rehearsal in helping students master the life skills. [42], p. 224

Teachers feeling well-prepared to deliver the curriculum was essential

This sub-theme was raised more than any other (across six different interventions) and related to teachers feeling properly prepared and supported to deliver the curriculum [34, 37,38,39, 42, 44]. The consistency with which this issue recurred suggests this is essential to successful intervention delivery. This sub-theme was linked to the first theme of supportive schools as, often, much of this feeling of preparedness and confidence in delivery among teachers was instilled through support from management and other senior staff. More practically, the availability of intervention resources such as an easy-to-follow curriculum, adequate training and pre-prepared materials was highlighted as being very useful to teachers.

Within the Infused-Life Skills Training intervention:

teachers reported that the training was critical in adequately preparing them to integrate [life skills] components into their curriculum. They indicated that the training was especially effective in their development and implementation of infused lessons, and that the step-by-step process and manual were valuable in guiding the development of their lesson plans. [42], p. 224

Teacher workloads and burnout is a barrier that should be overcome with administrative supports

Five studies identified teacher workload and/or burnout as a barrier to intervention implementation [37,38,39, 42, 43]. This may be partly addressed via school management support as above.

Limited methodological detail was provided by Holcomb and Denk in their study of English Classes, although many important aspects of implementation were explored. For example, they highlighted that:

research [to implement integrated curricula] consumed additional time in the teachers’ already busy schedules and required teachers to ‘learn’ some of the materials before presenting them to their students. Teachers’ lack of time or access to information, in some cases, may have limited the amount of health content applied to individual lessons. Thus, interdisciplinary lessons sometimes were not as detailed as they could have been. [43], p. S-39.

This sub-theme links clearly to the one above, as teacher preparation for an additional curriculum responsibility contributed to burnout. With adequate training and administrative support—or collaboration with other teachers—teachers experienced less burnout and were more likely to implement the intervention successfully.

Within Ransford et al.’s high-quality study of Promoting Alternative Thinking Strategies:

teachers who perceived their school administration as more supportive reported higher implementation quality, and positive perceptions of training and coaching were associated with higher levels of implementation dosage and quality. Teachers who reported the highest levels of burnout and the most negative perceptions of curriculum supports reported the lowest levels of implementation dosage and quality. [38], p. 510

Teacher attitudes towards intervention characteristics

Linked to teachers’ views on how supportive the school climate was for the implementation of these interventions, a key theme in several reports [32, 34, 35, 37, 40, 41] concerned the acceptability to teachers of the interventions themselves. This sub-theme was emphasised across studies as a factor enabling successful implementation to a greater extent than the acceptability of the intervention to students (see below), likely because teachers were typically the primary deliverers of the interventions.

Teacher belief in their responsibility to teach and own the integrated curriculum was a facilitator

Teacher uptake of interventions’ objectives was found to be linked to their attitudes towards the curriculum, their beliefs in their responsibility to teach social and emotional curricula and a sense of ownership of the integrated curriculum.

For example, Beets et al. reported from an evaluation of the Positive Action intervention in American primary schools that:

teacher beliefs regarding their responsibility to teach [social-emotional learning] concepts were significantly…related to their attitudes towards Positive Action…[which] were positively related to the amount of the Positive Action curriculum delivered…and the amount of the curriculum delivered was positively related to material utilization in both the classroom…and school-wide. [32], p. 217

Positive teacher attitude towards and belief in the potential of the integrated curriculum was a facilitator

Teacher perceptions of the role of social and emotional learning—which was a part of the curriculum in all but two interventions included in the review overall—influenced their internalisation and subsequent role-modelling of the behaviours promoted within the curriculum [32, 34, 35, 38, 39, 41].

Emphasised in the Roots of Empathy intervention:

all participants were committed to the importance of [social and emotional learning] in their teaching…they considered [it] essential to the academic learning that underpinned the teaching philosophy of all participants. The pedagogical understandings in the…program were consistent with each participant’s philosophy of learning and teaching. [34], p. 63

Conversely, teachers’ initial scepticism to new interventions or their feeling that these were a disruption to learning was barriers to implementation. Although the methods of this evaluation were poorly reported, the drama-based DRACON intervention in Australia experienced this barrier and the process evaluation explored this.

A few [teachers] start with stronger reservations or resistance [to the programme], and some of these have chosen to withdraw from the project. These reservations are usually expressed as: not trusting drama to achieve its purpose, sometimes because it is perceived to potentially disrupt an orderly classroom, or to be too time-consuming in a full syllabus. [37], p. 279

Teachers’ freedom to be innovative and have flexibility within the curriculum was a strength

Curricula that were perceived by teachers to be adaptable to their classroom settings were generally implemented to a greater extent [40, 41, 43]. Some curricula were designed to be flexible, allowing teachers to adapt components of the intervention in line with the goals of their classroom and the topical interests of students.

One example of this type of flexible curriculum was found in the English Classes intervention in secondary Israeli schools reported by Holcomb and Denk:

teachers reported that the program’s greatest strengths were its flexibility, its infusion of new material into their classrooms, and its interest to students … Autonomy allowed by the program was a significant strength noted by all the teachers, not only for the convenience it provided, but for the respect it displayed for their professionalism. [43], p. S-39

Student attitudes towards intervention characteristics

Students’ positive perception of the integrated curriculum was a facilitator

The acceptability of the intervention to students was reported as facilitating implementation, particularly where students saw the curriculum’s messages as relevant [31, 33, 40, 43,44,45].

Holcomb and Denk suggested that, ‘it was generally believed that the high level of interest among students was generated by the relevance of the health topics.’ [43], p. S-39

Students’ pre-existing attitudes aligning with intervention ideals was a facilitator

Not unlike teachers’ views about social-emotional learning, students’ pre-existing views of intervention messaging, if positive, were helpful in implementation. In Low et al.’s high-quality study of the Steps to Respect intervention in the USA, ‘significant positive associations with students’ engagement in the [Steps to Respect] lessons were found for classroom average levels of student support [of the programme], [and pre-existing] student attitudes against bullying, student climate and school connectedness.’ [36], p. 171.

Integrative interventions involving activities were regarded positively by students

Acceptability was greater where the learning activities that the interventions required were perceived to be relevant to students and fun to learn. For example, Wales et al. reported from an evaluation of the Peaceful Panels intervention in secondary schools in the United States that:

although the students were not unanimous in positive feelings about the program, the great majority of them stated that they enjoyed it and that they felt that it helped them understand violence prevention … The students’ positive feelings implied that students enjoyed learning through comics and it is possible that this was this helped them retain what they learned. [45], p. 143

Students were particularly positive where a health/academic integration intervention encouraged teachers to focus on topics that were judged more relevant to students than traditional academic content or to use more participative learning methods than would usually be the case. An example from Bechtel et al. suggests that:

students responded with interest and enthusiasm to the infused approach, liked the integration of substance abuse prevention into other subject areas, and were more engaged and eager to participate in class. Moreover, their students especially liked the facilitative classroom environment and the hands-on approaches of behavioral rehearsal and role playing. [42], p. 224

Parental support

A lack of parental participation and positive role-modelling of intervention concepts was a barrier

Parental involvement was in some cases a direct component of the intervention [33, 44] and thus a part of intervention implementation fidelity. Indirectly, parental involvement through reinforcement and role-modelling of curriculum messaging was sometimes part of the processes through which the intervention was hypothesised to work [33, 39, 44]. The role of parents could therefore be positive or negative and more often was indicated by authors as a barrier. For example, Sung reported in her account of implementation of the 4Rs intervention in primary schools in the USA, which was rated as highly useful, that:

[an implementing teacher] … viewed inconsistency between the way students are taught at school and at home as an impediment. For example, whereas she taught children to ‘talk things out’ without using violence in a conflict, some parents encouraged their children to use violence as means of solving social conflicts at school. [39], p. 100

Discussion

Summary of key findings

Although factors that influenced implementation varied widely depending on the intervention, several—often linked—themes did emerge from our synthesis, namely around the necessity of senior management support, having a positive teaching environment, positive pre-existing teacher and student attitudes towards integrated health and academic interventions and favourable opinions about the autonomy and innovation that the interventions enabled, and parental support of interventions.

It is worth noting that many of the themes above relate to factors affecting implementation which might apply generally to school-based health promotion and social and emotional learning interventions. Here, we aim to draw out what our synthesis suggests about factors affecting the implementation of our specific category of interventions which integrate health and academic education. First, this category of intervention particularly benefits from consistent cross-school support from administrators and colleagues in integrating health across the curriculum. Strong networks, continuous training and shared understanding about the overall aims of integration take time to build and effort to sustain. Thus, ongoing support from administrators, both practically and in terms of morale, is crucial.

Second, interventions need to be flexible and locally adaptable if they are to mesh with the existing teaching environment and curriculum. Third, interventions that integrate academic and health education are innovative and challenging and so require teachers and staff to believe in, and commit to, integration as a longer term aim to improve students’ health and social and emotional learning. Such support appears to be promoted both because teachers value the scope they provide for local adaptation and professional autonomy and because students value the chance to learn using methods that are more participative and topics that appear more relevant to students’ lives than in standalone academic subjects.

To our knowledge, there are no existing reviews of interventions that integrate academic and health education. However, reviews of related interventions can help in contextualising our findings. In their review of health promotion interventions in schools, Chilton et al. [16] similarly noted that school (and teacher) cultural norms concerning substance use affected the extent to which interventions addressing this were successfully implemented. Staff investment overall was critical, including support from administrators. Likewise, Pearson et al. echoed the importance of engaging staff and suggested that, ‘implementation hinges on negotiation and programme delivery and the acceptability (or otherwise) of the programme to those who deliver it.’ (p. 17) They further commented on the importance of deliverers’ enthusiasm for the intervention and the need to root it in their perceived responsibility for its success [18]. Bonell et al.’s review of process evaluations of interventions aiming to increase the healthiness of school environments reported on the importance of a health intervention’s alignment with school ethos as a predictor of its success, as well as the importance to good implementation of the broad participation of all staff and support from administrators [15]. Rimm-Kaufmann and Hulleman noted similar factors in a review of social and emotional learning interventions in primary schools, emphasising teachers’ enthusiasm for interventions as being pivoted on their overall culture of education on these subjects and of these skills. Coupled with school-wide support and ongoing mentoring from higher-level staff, a supporting ethos enhanced teachers’ commitment to interventions and was thus crucial to their success [21].

Indeed, our review provides evidence that teachers’ perceptions of their school’s teaching culture was a key determinant of successful implementation, something which did not emerge as a key theme in the other reviews cited above. This factor may reflect the importance of genuine integration between health and mainstream academic elements when delivering this particular category of intervention [46].

Considering May and Finch’s normalisation theory [23] again as a framework to understand factors affecting the potential sustainability of these interventions, the roles of the teacher in understanding/internalising key components of the curriculum and enthusiastically taking responsibility for the intervention were facilitators of these interventions. Likewise, collective action through whole-school engagement and administrator support were also notable facilitators. There was no mention of reflexive monitoring in reports of process. However, reporting on these individual elements was inconsistent across interventions. Therefore, while some interventions like Positive Action, 4Rs, the Gatehouse Project and Roots of Empathy, which reported positively on these factors, seem conducive to sustainability, the potential sustainability of other interventions remains questionable.

Implications for research and policy

Our findings suggest that integrative interventions, while attractive as ways to deliver some health, social and emotional learning in the context of school systems overwhelmingly focused on educational attainment, are not a panacea, as their implementation poses particular challenges. Proper integration requires that teachers believe in the interventions and have the time and resources to reflect and build-in a seamless integration, that interventions have enough flexibility to be applied effectively in diverse contexts and that the baseline teaching culture of a school is conducive to this type of intervention. This category of interventions will not flourish in instances where staff are demoralised and change jobs frequently, where they are sceptical about integrating health into their lessons or where managerial and collegial support for this challenging work of integration is perceived an issue.

Unfortunately, moderating factors—for example, the effects of gender, socioeconomic status, and so forth—were not examined by the authors of the studies we have included in this review, so they could not be included in our analysis. From other studies and reviews specific to interventions that emphasised social-emotional learning (which represents the majority of interventions included in our review), identified moderating factors include universal versus targeted interventions, the influence of the overall risk level of a school, the quality of schools’ interactions with students, students’ family environments and differential impact on boys versus girls and younger versus older age groups [14, 17, 19, 20]. It would be of interest to explore if and how such factors may play a role in the implementation and uptake of the integrated academic and health interventions, which may be of value in future research.

Many interventions in our broader systematic review prioritised reporting on outcomes over process (35 outcome evaluations were included, whereas only 16 relevant process evaluations could be found). This critique can be applied to the reporting of interventions more broadly. A review of implementation data by Michie et al. found that only 5–30% publications of experimental studies had detailed intervention descriptions at all [47]. This lack of detail presents issues when trying to produce replicable interventions. The paucity of intervention description is compounded further by a lack of explanation of the mechanisms by which interventions achieve outcomes and the contextual factors that may influence both implementation and outcomes [22, 48]. This gap has been noted by several authors, and across disciplines [22, 47,48,49,50,51].

Our current review confirms that process data were certainly less prioritised by authors, but further, that the quality of this reporting was poor. There was limited discussion of context in particular. Especially within complex public health interventions that aim to bring about behaviour change, both implementation and outcomes are inevitably influenced by context. Furthermore, understanding implementation through establishing how it was impacted by context, among other central implementation processes and factors, can prevent ‘type III’ errors—the wrongful attribution of intervention outcomes to an incorrectly implemented intervention [52].

A thorough exploration of implementation makes it possible to know whether an intervention has been implemented as intended and what considerations must be applied prior to its replication elsewhere. Realist approaches are helpful here, as they aim to thoughtfully test hypotheses around intervention implementation, noting explicitly the mechanisms that lead to outcomes and the contexts that influence these [48, 53]. Future research should therefore aim to incorporate a study of process alongside outcomes for a more robust understanding of intervention effectiveness.

Additionally, there is increasing recognition that evaluation needs to move away from accrediting specific interventions as effective/not and towards developing and refining theory of implementation. To do so, there needs to be sound documentation of how context influences intervention implementation alongside theories of change to detail how context interacts with intervention mechanisms to produce outcomes [48, 53]. Thus, process evaluations or other studies reporting on implementation must empirically examine how context influenced implementation to develop an implementation theory that is evidence-based [23]. Therefore, our synthesis of process data is a contribution to this literature, focused on a particular category of school-based health-promoting interventions. Simultaneously, however, it highlights a pressing need for further research into the processes of such interventions for their applicability to be fully appreciated.

Method’s strengths and limitations

Our study involved a comprehensive search of available literature on the implementation of interventions that aim to integrate academic and health education to reduce substance use and/or violence. Given our robust searching methods, it is likely that we have captured, to the greatest extent possible, what is published about the implementation of these interventions. Our use of a standard tool to assess quality also added to the review’s rigour.

Although our analysis sought to employ a systematic and in-depth approach to synthesising the findings of process evaluation, it was somewhat limited by the paucity of relevant findings. While reporting on conventional, largely quantitative measures of implementation fidelity and acceptability, many studies failed to report on how implementation was affected by characteristics of interventions, deliverers, participants or school contexts and so contributed little to our synthesis.

Conclusion

Several factors facilitating and inhibiting the implementation of interventions that integrate academic and health education to reduce substance use and/or violence are described here, providing tentative but insightful evidence of context-specific issues that may impact intervention success. However, overall, there is still a considerable gap in our understanding of how to achieve the successful implementation of these. With a view to promoting sustainability of these interventions within ever-changing socio-political and economic circumstances, more detail about context, moderating factors and facilitators and barriers at the individual, school-wide and community levels will be necessary. Our synthesis of effects of these interventions on violence and substance use is currently being completed for publication.

References

  1. 1.

    Faggiano F, Minozzi S, Versino E, Buscemi D. Universal school-based prevention for illicit drug use. Cochrane Database Syst Rev. 2014;(12):CD003020. https://doi.org/10.1002/14651858.CD003020.pub3.

  2. 2.

    Farrington DP, Ttofi MM. School-based programs to reduce bullying and victimization. Washington: U. S. Department of Justice; 2009.

  3. 3.

    Foxcroft DR, Tsertsvadze A. Cochrane Review: universal school-based prevention programs for alcohol misuse in young people. Cochrane Database Syst Rev. 2012;7(2):450–575.

  4. 4.

    Thomas RE, Perera R. School-based programmes for preventing smoking. Cochrane Database Syst Rev. 2006;4:CD001293.

  5. 5.

    Vreeman RC, Carroll AE. A systematic review of school-based interventions to prevent bullying. Arch Pediatr Adolesc Med. 2007;161(1):78–88.

  6. 6.

    Demissie Z, Brener N, McManus T, Shanklin S, Hawkins J, Kann L. School health profiles 2012: characteristics of health programs among secondary schools. Atlanta: Centers for Disease Control and Prevention; 2013.

  7. 7.

    Frey KS, Hirschstein MK, Guzzo BA. Second step preventing aggression by promoting social competence. J Emot Behav Disord. 2000;8(2):102–12.

  8. 8.

    Visscher AJ. Public school performance indicators: problems and recommendations. Stud Educ Eval. 2001;27(3):199–214.

  9. 9.

    Formby E, Coldwell M, Stiell B, Demack S, Stevens A, Shipton L, Wolstenholme C, Willis B. Personal, social, health and economic (PSHE) education: a mapping study of the prevalent models of delivery and their effectiveness. Sheffield: Sheffield Hallan University; 2011.

  10. 10.

    Wright G, Ainsworth P. Plastered evaluation: part of It’s not OK! Violence prevention education programme. Liverpool: Ariel Trust; 2008.

  11. 11.

    Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, Moore L, O’Cathain A, Tinati T, Wight D. Process evaluation of complex interventions: Medical Research Council guidance. BMJ. 2015;350:h1258.

  12. 12.

    Petticrew M. Time to rethink the systematic review catechism? Moving from ‘what works’ to ‘what happens. Syst Rev. 2015;4(1):36.

  13. 13.

    Fletcher A, Jamal F, Moore G, Evans RE, Murphy S, Bonell C. Realist complex intervention science: applying realist principles across all phases of the Medical Research Council framework for developing and evaluating complex interventions. Evaluation. 2016;22(3):286–303.

  14. 14.

    Aber JL, Pedersen S, Brown JL, Jones SM, Gershoff ET. Changing children’s trajectories of development. National Center for Children and Poverty Columbia University. New Yord: Mailman School of Public Health; 2003.

  15. 15.

    Bonell C, Farah J, Harden A, Wells H, Parry W, Fletcher A, Petticrew M, Thomas J, Whitehead M, Campbell R. Systematic review of the effects of schools and school environment interventions on health: evidence mapping and synthesis. Public Health Res. 2013;1(1):1-317.

  16. 16.

    Chilton R, Pearson M, Anderson R. Health promotion in schools: a scoping review of systematic reviews. Health Educ. 2015;115(3/4):357–76.

  17. 17.

    McCormick MP, Cappella E, O’Connor EE, McClowry SG. Context matters for social-emotional learning: examining variation in program impact by dimensions of school climate. Am J Commun Psychol. 2015;56(1–2):101–19.

  18. 18.

    Pearson M, Chilton R, Wyatt K, Abraham C, Ford T, Woods HB, Anderson R. Implementing health promotion programmes in schools: a realist systematic review of research and experience in the United Kingdom. Implement Sci. 2015;10(1):1.

  19. 19.

    Raimundo R, Marques-Pinto A, Lima ML. The effects of a social–emotional learning program on elementary school children: the role of pupils’ characteristics. Psychol Schools. 2013;50(2):165–80.

  20. 20.

    Sancassiani F, Pintus E, Holte A, Paulus P, Moro MF, Cossu G, Angermeyer MC, Carta MG, Lindert J. Enhancing the emotional and social skills of the youth to promote their wellbeing and positive development: a systematic review of universal school-based randomized controlled trials. Clin Pract Epidemiol Ment Health. 2015;11(1):21-40.

  21. 21.

    Rimm-Kaufman SE, Hulleman CS. Social and emotional learning in elementary school settings: identifying mechanisms that matter. In: The handbook of social and emotional learning: research and practice; 2015. p. 151–66.

  22. 22.

    Proctor EK, Powell BJ, McMillen JC. Implementation strategies: recommendations for specifying and reporting. Implement Sci. 2013;8(1):1.

  23. 23.

    May C, Finch T. Implementing, embedding, and integrating practices: an outline of normalization process theory. Sociology. 2009;43(3):535–54.

  24. 24.

    Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA group. Preferred reporting items for systematic reviews and meta analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097.

  25. 25.

    Peersman G, Oliver S, Oakley A. EPPI-center review guidelines: data collection for the EPIC database. London: EPPI-Centre Social Science Research Unit; 1997.

  26. 26.

    Mukdarut B, Dickson K, Felix L, Chiumento A. The impact of mental health and psychosocial support interventions on people affected by humanitarian emergencies: a systematic review. Oxford: Oxfam GB; 2017.

  27. 27.

    Barnett-Page E, Thomas J. Methods for the synthesis of qualitative research: a critical review. BMC Med Res Methodol. 2009;9(1):1.

  28. 28.

    Popay J, Roberts H, Sowden A, Petticrew M, Arai L, Rodgers M, Britten N, Roen K, Duffy S. Guidance on the conduct of narrative synthesis in systematic reviews. A product from the ESRC methods programme Version. Lancaster: University of Lancaster; 2006.

  29. 29.

    Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol. 2008;8(1):1.

  30. 30.

    Hanson JL. Teachers’ beliefs about emotions in the classroom: relations to teacher characteristics and implementation of a social-emotional learning program. Vancouver: University of British Columbia; 2012.

  31. 31.

    Beets M. Factors associated with the implementation fidelity of a school-based social and character development program: findings from the Positive Action Program, Hawai’i. Corvallis: University of Oregon; 2007.

  32. 32.

    Beets MW, Flay BR, Vuchinich S, Acock AC, Li K-K, Allred C. School climate and teachers’ beliefs and attitudes associated with implementation of the positive action program: a diffusion of innovations model. Prev Sci. 2008;9(4):264–75.

  33. 33.

    Rothwell H, Segrott J. Preventing alcohol misuse in young people aged 9-11 years through promoting family communication: an exploratory evaluation of the Kids, Adults Together (KAT) Programme. BMC Public Health. 2011;11(1):1.

  34. 34.

    Segrott J, Rothwell H, Hewitt G, Playle R, Huang C, Murphy S, Moore L, Hickman M, Reed H. Preventing alcohol misuse in young people: an exploratory cluster randomised controlled trial of the Kids, Adults Together (KAT) programme. Public Health Res. 2015;3(15):1–188.

  35. 35.

    Cain G, Carnellor Y. ‘Roots of Empathy’: a research study on its impact on teachers in Western Australia. J Stud Wellbeing. 2008;2(1):52-72.

  36. 36.

    Bond L, Glover S, Godfrey C, Butler H, Patton GC. Building capacity for system-level change in schools: lessons from the gatehouse project. Health Educ Behav. 2001;28(3):368–83.

  37. 37.

    Low S, Van Ryzin MJ, Brown EC, Smith BH, Haggerty KP. Engagement matters: lessons from assessing classroom implementation of steps to respect: a bullying prevention program over a one-year period. Prev Sci. 2014;15(2):165–76.

  38. 38.

    O'Toole J, Burton B. Acting against conflict and bullying. The Brisbane DRACON project 1996–2004—emergent findings and outcomes. Res Dram Educ. 2005;10(3):269–83.

  39. 39.

    Ransford CR, Greenberg MT, Domitrovich CE, Small M, Jacobson L. The role of teachers’ psychological experiences and perceptions of curriculum supports on the implementation of a social and emotional learning curriculum. School Psychol Rev. 2009;38(4):510.

  40. 40.

    Sung J-A. Teachers’ perceptions and experiences of the implementation of a social-emotional learning program in an inner-city public elementary school. New York: Columbia University; 2015.

  41. 41.

    Zoller U, Weiss S. “Hashish and marijuana”—an innovative, interdisciplinary drug education curricular program for high schools. J Drug Educ. 1981;11(1):37–46.

  42. 42.

    Malloy M, Acock A, DuBois DL, Vuchinich S, Silverthorn N, Ji P, Flay BR. Teachers’ perceptions of school organizational climate as predictors of dosage and quality of implementation of a social-emotional and character development program. Prev Sci. 2015;16(8):1086–95.

  43. 43.

    Bechtel LJ, Vicary JR, Swisher JD, Smith EA, Hopkins AM, Henry KL, Minner D. An interdisciplinary approach for the integration and diffusion of substance abuse prevention programs. Am J Health Educ. 2006;37(4):219–25.

  44. 44.

    Holcomb JD, Denk JP. An interdisciplinary English/health connection: promoting health awareness and healthy behaviors. J Health Educ. 1993;24(sup1):S–36-S-41.

  45. 45.

    Wales A. Peaceful panels: using comics to teach anti-bullying to junior high school students. Prescott Valley: Northcentral University; 2013.

  46. 46.

    Evans-Whipp T, Beyers JM, Lloyd S, Lafazia AN, Toumbourou JW, Arthur MW, Catalano RF. A review of school drug policies and their impact on youth substance use. Health Promot Int. 2004;19(2):227–34.

  47. 47.

    Michie S, Fixsen D, Grimshaw JM, Eccles MP. Specifying and reporting complex behaviour change interventions: the need for a scientific method. Implement Sci. 2009;4(1):1.

  48. 48.

    Bonell C, Fletcher A, Morton M, Lorenc T, Moore L. Realist randomised controlled trials: a new approach to evaluating complex public health interventions. Soc Sci Med. 2012;75(12):2299–306.

  49. 49.

    Diez-Roux AV. Bringing context back into epidemiology: variables and fallacies in multilevel analysis. Am J Public Health. 1998;88(2):216–22.

  50. 50.

    Fixsen DL, Naoom SF, Blase KA, Friedman RM. Implementation research: a synthesis of the literature. Tampa: University of South Florida; 2005. http://ctndisseminationlibrary.org/PDF/nirnmonograph.pdf. Accessed 12 Jan 2016

  51. 51.

    Langford R, Bonell CP, Jones HE, Pouliou T, Murphy SM, Waters E, Komro KA, Gibbs LF, Magnus D, Campbell R. The WHO Health Promoting School framework for improving the health and well-being of students and their academic achievement. Cochrane Database Syst Rev. 2014. https://doi.org/10.1002/14651858.CD008958.pub2.

  52. 52.

    Basch CE, Sliepcevich EM, Gold RS, Duncan DF, Kolbe LJ. Avoiding type III errors in health education program evaluations: a case study. Health Educ Behav. 1985;12(3):315–31.

  53. 53.

    Pawson R, Tilley N. Realistic evaluation. London: Sage; 1997.

Download references

Acknowledgements

The authors wish to acknowledge Clare Stansfield who carried out the database searches of studies included in this review. We are also grateful for the technical support we received from the staff at the EPPI-Centre throughout.

Funding

This project is funded by a grant from the National Institute for Health Research Public Health Research programme (grant PHR 14/52/15). This report presents independent research commissioned by the National Institute for Health Research (NIHR). The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, MRC, CCF, NETSCC, the Public Health Research programme or the Department of Health.

Availability of data and materials

Data sharing is not applicable to this article as no data sets were generated or analysed during the current study.

Author information

CB led the design and conduct of the study. AF, GM, JT and RC contributed to the design of this review. TT and CB data extracted and quality assessed all included studies. TT and SP led the analysis of all process data, overseen by CB. TT wrote the first draft of this paper. All authors contributed to editing further drafts. All authors read and approved the final manuscript.

Correspondence to Chris Bonell.

Ethics declarations

Ethics approval and consent to participate

This project was approved by the research ethics committee of UCL Institute of Education (ethics approval reference REC 746). The project complied with the Social Research Association’s ethical guidelines and guidance from the National Coordinating Centre for Public Engagement.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional files

Additional file 1:

PRISMA Checklist. (DOC 412 kb)

Additional file 2:

Example search strategy. (DOCX 123 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Keywords

  • Process evaluation
  • Systematic literature review
  • Health education
  • Tobacco use
  • Substance use
  • Violence

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate. Please note that comments may be removed without notice if they are flagged by another user or do not comply with our community guidelines.