Theoretical approaches to process evaluations of complex interventions in health care: a systematic scoping review protocol

Abstract

Background

Complex interventions in health care are characterized by multiple interacting components as well as by numerous nonlinear interactions with the social systems within which they are being implemented. The process of developing, evaluating and implementing complex interventions is therefore challenging. Established guidance such as the MRC (Medical Research Council) framework for developing and evaluating complex interventions refers to process evaluations as an integral part of the development of complex evidence-based interventions. Even though the need for process evaluations is recognized, the realization of such approaches is challenging because methodological instruction is sparse, and the phenomenon of interest is complex. A number of theoretical approaches indicating how to conduct process evaluations of complex interventions in health care exist, but a systematic and comprehensive overview of these is missing. Thus, the objective of the systematic scoping review described herein is to provide an overview and analysis of theoretical approaches suitable for the planning and conducting of process evaluations.

Methods

The design and conduct of this review will follow the procedures of a systematic scoping review. The search strategy will be developed following the BeHEMoTh (Behaviour of interest; Health context; Exclusions; Models or Theories) template which has been conceptualized for structured reviews of theory. The systematic search of the MEDLINE (via PubMed), CINAHL (via EBSCO) and PsycInfo (via EBSCO) electronic databases will be complemented by “hand searching” techniques. Study selection, data extraction, and data analysis will be performed by tandems of two researchers independently of each other. Divergent decisions and judgements between the two researchers will be discussed by the whole review team.

Discussion

The findings from this scoping review will provide an overview and comparison of theoretical approaches suitable for process evaluations of complex interventions in health care. The review results will support researchers in choosing the theoretical approach that best fits the respective focus of their process evaluation study.

Systematic review registration

This study has been registered with PROSPERO (International Prospective Register of Systematic Reviews) under registration number CRD42020211732.

Background

Efforts to improve healthcare delivery and outcomes often require complex interventions. Complex interventions were originally defined as interventions with multiple interacting components [1, 2], indicating that complexity is particularly linked to the intervention itself. Ongoing debates about which characteristics constitute complex interventions [3,4,5] have complemented this original definition with the awareness that complexity is given “…primarily due to the social systems within which these actions occur, the contextually contingent nature of impacts, and the agency of the groups and individuals whose behaviours they aim to influence” [3].

Due to this changing definition, it is increasingly recommended to integrate process evaluations in the development and evaluation of complex evidence-based interventions to assess implementation aspects and to gain a better understanding of causal mechanisms and contextual factors associated with variation in outcomes [1, 6,7,8]. Process evaluations can especially help to distinguish between those interventions that are inherently faulty (failure of intervention) and those interventions that are delivered inaccurately (implementation failure) [9]. Thus, they can prevent a so-called type III error, which refers to the question of whether the intervention actually has been delivered or if there is in fact something measured that does not exist [10].

The MRC (Medical Research Council) framework for developing and evaluating complex interventions provides a well-established guide for the development of complex interventions in healthcare [1, 2]. The revised version from 2008 [1, 11] for the first time took into account that process evaluations are highly recommendable to understand processes and to obtain explanations if interventions fail or have unanticipated consequences [1]. Nevertheless, the 2008 framework provided only sparse information on how to integrate and conduct process evaluations. Process evaluation is described as being “nested within a trial” that in the first place focuses on outcome evaluation [11].

In recent years, methodological aspects of process evaluations have been increasingly discussed; thus, the current MRC provides comprehensive guidance for process evaluations of complex interventions [6, 12]. This guidance refers to [1] implementation, [2] mechanisms of impact, and [3] context as key aspects to be examined [6, 12]. Furthermore, it “provides researchers, practitioners, funders, journal editors and policy-makers with guidance in planning, designing, conducting and appraising process evaluations of complex interventions” ([12] , p. 9}. It comprises process evaluation theory as well as practical information [12].

Regardless of the scientific discourse in the context of the MRC framework, the concept of process evaluation is not new to the field of health care research; it had already emerged by the 1960s [13]. Suchmann [14], as one of the first to emphasize the need to evaluate health interventions/programmes in relation to their practical setting, identified [1] the attributes of the programme, [2] the population exposed to the programme, [3] the situational context in which the programme takes place, and [4] the different kinds of effects produced by the programme as obligatory domains for “making sense” of the evaluative findings. Although these basic assumptions of Suchmann [14] remain relevant for the context of process evaluation, a wide range of developments have produced a diverse research field with many underlying concepts.

Commonly quoted concepts are recruitment (of cluster/of individuals), maintenance, context, resources, implementation, reach (of individuals), barriers, exposure, initial use, continued use, contamination, dose delivered, dose received, fidelity (degree to which an intervention is implemented in accordance with the original protocol), delivery (to clusters/to individuals), response (of clusters/of individuals), effectiveness, unintended consequences, theory, efficacy, and adoption [13, 15,16,17]. Many of these concepts are also closely linked to the broad field of implementation research [18]. Thus, the underlying theoretical allocation and a clear distinction of these concepts is challenging.

The existing theoretical approaches relevant for process evaluations set varying priorities and organize their concepts in different ways (e.g. [19]). In addition, concepts termed identically in different theoretical approaches are not always defined in the same way, and they are not always selective between the different theoretical approaches. The MRC guidance for process evaluation of complex interventions [12] presents “an overview of a number of core frameworks” ([12], p. 18) relevant to process evaluation, but the authors explicitly state that they “make no claims to exhaustiveness” ([12], p. 18). While the theoretical approaches considered in the MRC guidance for process evaluation include many of the concepts mentioned above, the guidance does not provide a thorough analysis or comparison of the concepts the theoretical approaches refer to. Therefore, it remains unclear why the proposed theoretical approaches were selected and if there may be other theoretical approaches of relevance. This complicates the suggestion of the MRC guidance to combine concepts from different theoretical approaches for the development and conduct of process evaluations [12]. However, given the variety of theoretical approaches that are of relevance for process evaluations, it remains challenging to select and combine theoretical approaches or single concepts that fit the requirements and aims of a specific process evaluation approach.

Therefore, we will perform a comprehensive and systematic scoping review and analysis of the theoretical approaches suitable for process evaluations of complex interventions in health care. For this purpose, we understand theoretical approaches as interrelated propositions that intend to explain and predict general sets of phenomena [20]. Accordingly theoretical approaches help to structure our knowledge, understanding, explanation, and prediction of the world and they allow to generate new research hypotheses as well [20,21,22]. In the context of process evaluation, a theoretical foundation is essential to better understand and systematically investigate the black box of implementing complex interventions in complex social settings [22]. At the same time, theoretical approaches are not uniform but might be e.g. implicit or explicit, descriptive, explanatory or predictive [22] or address high, middle or low abstraction levels [21]. Moreover, a range of heterogenous terms is being used in order to consider theoretical aspects in research projects and real life, e.g. theoretical/conceptual model, (theoretical/conceptual) framework, theoretical concepts [21, 22] These terms are often not clearly distinguished and used in an interchangeable way [20, 21]. Based on the above-noted explanations, we will therefore consider any kind of theoretical approach suitable for process evaluation in order to achieve a comprehensive overview.

The review results will further support researchers in considering, selecting and combining the available theoretical approaches (or just single concepts) and thus promote tailored and theory-informed process evaluation approaches.

Objectives

The following study objectives were defined:

• To identify theoretical approaches suitable for process evaluations of complex interventions in health care;

• To describe, analyse and compare the identified theoretical approaches and single concepts that these theoretical approaches comprise of.

Methods

We will perform a review of theoretical approaches. This kind of review differs substantially from a review of empirical data. Methodological issues related to such approaches have been rarely discussed in the literature [23], and they are not taken into account in overviews of review types [24, 25]. We will therefore consider the methodological issues related to this review carefully and ensure its transparency and comprehensiveness. The development and reporting of the review will follow the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) extension for Scoping Reviews (PRISMA-ScR). The PRISMA extension, PRISMA-P [26] (see Additional file 1) has been used for the preparation of this review protocol.

Important protocol amendments will be reported with the review results.

Searches

We will use the BeHEMoTh (Behaviour of interest; Health context; Exclusions; Models or Theories) template only to develop our search strategy. This template has been recently conceived to inform the development of procedures for the structured review and identification of theory [27]. It refers to the following categories:

• Be – Behaviour of interest ➔ ways in which the population interacts with the health context;

• H – Health context of interest;

• E – Exclusions ➔ to exclude, e.g. non-theoretical/technical models/frameworks; and

• MoTh – Models or Theories ➔ generic terms for models/frameworks, as well as named frameworks/models, if appropriate.

Table 1 gives an overview of the initial search terms which will be refined during the search process.

Table 1 Initial search terms based on the BeHEMoTh framework [27]

Based on the final search strategy, systematic electronic searches of the following databases relevant to health care research will be performed: MEDLINE (via PubMed), CINAHL and PsycInfo (both via EBSCO).

Furthermore, iterative, pragmatic approaches, in addition to keyword searches alone, are recommended as an effective method for qualitative reviews [28, 29] and seem to suit the needs of reviews of theory as well.

Thus, following the iterative approach, the systematic search will be complemented by “hand-searching” techniques such as expert consultations, trawls through specific journals, and backward as well as forward citation tracking, which is frequently recommended as being especially valuable for reviews of theory [23].

The development and application of the search strategy will be continuously discussed by the project team, which comprises all the authors of this study protocol.

Study criteria

The search results will be screened according to defined inclusion criteria and thereby resulting exclusion criteria. An initial set of inclusion criteria is outlined in Table 2.

Table 2 Outline of initial inclusion criteria of the review

For the purpose of our review, precise criteria definitions based on established frameworks, such as PICO [30] or its equivalent for reviews of qualitative studies, SPIDER [28], developed in the context of health research are not suitable, since relevant theoretical approaches possibly do not originate from the specific field of process evaluation or from health care research [23]. Further, one of the key challenges in this review will be to define which criteria constitute a “process evaluation” since a consistent terminology and definition is missing. Thus, from the beginning of the review process, we have discussed within the review team which criteria could define “process evaluation”. We started out with a literature-based working definition (additional file 2), which we will continuously develop throughout the review process in iterative loops. The inclusion and exclusion criteria will be adapted accordingly. Following the iterative approach described above, amendments and specifications of the initial set of criteria will be discussed and decided throughout the review process. All decisions and amendments are being documented in a study manual, which will be provided in the final review paper to ensure transparency.

Data management

Data management will be organized using the EPPI-Reviewer4 software [31], which has been developed for managing and analysing data in all types of systematic reviews, such as meta-analysis, framework synthesis and thematic synthesis [32]. Features of the software support reference management, study classification, data extraction, and synthesis. The software is recommended by the Cochrane Collaboration [33].

Selection process

Duplicates will be removed before starting the selection process. This step, as well as the other steps of the selection process, will be supported by the EPPI-Reviewer4 software.

Based on the final set of inclusion and exclusion criteria, all titles and abstracts will be screened independently by each researcher paired up in teams of two (tandems) (all authors of this study protocol). Full texts will be obtained and assessed where the inclusion criteria seem to be met but where ratings for the title and abstract screenings differ between reviewers and no consensus can be achieved. Cases that remain unclear after a full text assessment will be discussed within the tandems and agreed upon between both reviewers. In case of non-agreement within one tandem, the publication will be discussed within the project team for a final decision.

Inclusion and exclusion of publications reporting studies that are not labelled as “process evaluations”, but that actually meet the criteria of a process evaluation and that apply a theoretical approach will be thoroughly discussed within the project team. This discussion will be based on a working definition of “process evaluation” that will be developed consecutively throughout the review process.

Publications describing the same framework will be consolidated before data extraction since the framework/model is the unit of interest and not the single report [33]. Different versions and developments of frameworks/models will be distinguished.

Data extraction strategy

The EPPI-Reviewer4 software [32] will be used for data extraction. This software allows flexible coding schemes for classifying studies with regard to eligibility criteria and for capturing detailed information from the included references. We will extract data based on the following coding categories: author, year, name of theoretical approaches, scope, process of development, historical roots/theoretical basis, dimensions and constructs included in the theoretical approaches, and guidance on the utilization of the theoretical approaches.

The data extraction will again be performed independently by each researcher paired up in tandems. The EPPI-Reviewer4 software [32] allows the classification of theoretical approaches by multiple users, the comparison of results, and the documentation of the process of agreeing upon final decisions.

Data synthesis and presentation

The identified theoretical approaches will be presented in a table format. These tables will provide an overview and content-wise sorting of the identified theoretical approaches. The tables will also present the data extracted with regard to each of the aforementioned categories (see data extraction). A main focus will be the display and allocation of the single concepts based on the identified theoretical approaches. They will also be presented in a table. However, since concepts used in the context of process evaluation often overlap and are inconsistently defined, we will also consider divergent definitions and meanings.

The review results will be published in a peer-reviewed journal and presented at relevant conferences.

Discussion

The relevance of process evaluations is being increasingly discussed in the scientific discourse on the development and evaluation of complex interventions and is now widely recognized [1, 6, 8, 15, 16]. Although an increasing number of process evaluations are being conducted, these approaches are often not well founded on theory since there is little guidance available on how to derive concrete process evaluation procedures from theoretical allocation [16]. Frequently, this leads to arbitrary methodological procedures and subsequently to low-quality results. Furthermore, the phenomenon investigated is very demanding, since the focus is not only on the intervention to be developed but also on the implementation, the mechanisms of impact, and above all on the context [1, 6], which makes it necessary to consider complex social systems [3]. A theoretical foundation is therefore essential for the planning and realization of process evaluations. The results of this review will provide a systematic and critical overview and comparison of existing theories for process evaluations and thus facilitate the targeted selection of suitable theoretical approaches for the respective process evaluation approach. The review will build upon and further contribute to the scientific discourse on process evaluations that has been inspired by the MRC and its guidance on process evaluation.

From a methodological point of view, there will be some challenges related to this review since little literature exists with regard to literature reviews aiming at identifying theory. However, we expect that we will achieve high-quality results with the methodological approach described in this protocol, which is based on the guidelines for conducting systematic scoping reviews and the demonstrated iterative, inductive and reflexive approaches of qualitative research.