- Open Access
- Open Peer Review
Use of programme theory to understand the differential effects of interventions across socio-economic groups in systematic reviews—a systematic methodology review
Systematic Reviewsvolume 6, Article number: 266 (2017)
Systematic review guidance recommends the use of programme theory to inform considerations of if and how healthcare interventions may work differently across socio-economic status (SES) groups. This study aimed to address the lack of detail on how reviewers operationalise this in practice.
A methodological systematic review was undertaken to assess if, how and the extent to which systematic reviewers operationalise the guidance on the use of programme theory in considerations of socio-economic inequalities in health. Multiple databases were searched from January 2013 to May 2016. Studies were included if they were systematic reviews assessing the effectiveness of an intervention and included data on SES. Two reviewers independently screened all studies, undertook quality assessment and extracted data. A narrative approach to synthesis was adopted.
A total of 37 systematic reviews were included, 10 of which were explicit in the use of terminology for ‘programme theory’. Twenty-nine studies used programme theory to inform both their a priori assumptions and explain their review findings. Of these, 22 incorporated considerations of both what and how interventions do/do not work in SES groups to both predict and explain their review findings. Thirteen studies acknowledged 24 unique theoretical references to support their assumptions of what or how interventions may have different effects in SES groups. Most reviewers used supplementary evidence to support their considerations of differential effectiveness. The majority of authors outlined a programme theory in the “Introduction” and “Discussion” sections of the review to inform their assumptions or provide explanations of what or how interventions may result in differential effects within or across SES groups. About a third of reviews used programme theory to inform the review analysis and/or synthesis. Few authors used programme theory to inform their inclusion criteria, data extraction or quality assessment. Twenty-one studies tested their a priori programme theory.
The use of programme theory to inform considerations of if, what and how interventions lead to differential effects on health in different SES groups in the systematic review process is not yet widely adopted, is used implicitly, is often fragmented and is not implemented in a systematic way.
A key challenge facing systematic reviewers when complying with recent guidance [1,2,3,4,5,6] on incorporating considerations of socio-economic health inequalities, is to determine not only if but also how the interventions being reviewed may work differently across socio-economic status (SES) groups. An understanding of how socio-economic health inequalities may impact on intervention effectiveness can help reviewers to decide whether interventions are likely to have either a positive or negative effect on the health inequality gap . This may then influence their decision on whether or not to include considerations of socio-economic health inequalities in the review.
Guidance on incorporating considerations of health inequalities in systematic reviews recognises the limitations of using traditional approaches to formulate review questions . While the traditional ‘PICO’ (population, intervention, comparison and outcome) framework, and subsequent derivatives , can help reviewers to clarify the specific components under review, they are not designed to help to identify explanatory relationships for if and how interventions may have differential effects on health across different SES groups . For example, when defining the ‘P’ in PICO (i.e. population group), the emphasis is on describing what population characteristics are under review (e.g. condition, age), rather than the way different people experience the healthcare system within which an intervention is delivered.
As a consequence, equity review guidance recommends using additional methods such as programme theory, logic models or theories of change to understand the assumptions behind if and how the intervention may work differently across SES groups [1, 5]. When considering the need to incorporate health inequalities in systematic reviews, therefore, reviewers need to know if, what and how interventions designed to improve the health of a population may have differential effects across different SES groups. Little is known, however, on if and how reviewers operationalise the equity review guidance when deciding whether or not to incorporate considerations of socio-economic health inequalities in systematic reviews.
Defining programme theory
Programme theory is the overarching theory or model of how an intervention is expected to work . There is, however, a lack of consistency in the way in which the terms relating to programme theory are applied in the literature, with some authors using them synonymously. Others note that while an overlap between the terms exists, a distinction can nevertheless be made between them [11, 12] (see Table 1).
The use of programme theory in guiding the conduct of systematic reviews is not new [13,14,15]. New theory-informed approaches to systematic reviews (e.g. realist reviews, ‘best fit’ framework synthesis) have increased awareness of the use of theory within the systematic review process . More recently, programme theory has been advocated as a tool to help reviewers of complex interventions to better understand ‘what works, for whom and under what circumstance’ [9, 17]. The Cochrane Collaboration have recently published guidance on the choice and use of theory in complex intervention reviews .
Limitations of the systematic review guidance in helping reviews to operationalise a programme theory
While current equity guidance clearly presents the rationale for incorporating considerations of health inequalities in systematic reviews, it offers little practical guidance on how to operationalise a programme theory to inform an understanding of if, what and how interventions work for different SES groups .
A study of systematic review guidance for incorporating health inequalities  found that of 20 guidance documents, only one  looked at how reviewers were operationalising such decisions. Although data were collected on whether reviewers operationalised their decisions by using theory, empirical evidence or personal experience, no information was sought on which theories or evidence was used, or how these were used to inform the review process.
Much of the guidance on incorporating considerations of health inequalities in systematic reviews is written either from the perspective that health inequalities have already been identified as the focus of the review or written with an underlying assumption that reviewers already have a good understanding of health inequalities and how they could impact on their review findings . Furthermore, the use of terms, such as ‘programme theory’, ‘logic model’, etc., may neither be widely accepted nor understood by reviewers [1, 19]. These challenges make operationalising the guidance difficult for both expert and novice reviewers who either do not have a health inequality background or are unfamiliar with the use of programme theory to inform systematic reviews.
Perceived value of programme theory use in systematic reviews
The perceived value of using programme theory to inform systematic reviews is well documented in the literature (see Table 2). From an implementation perspective, programme theory may help to identify the elements of an intervention that may be more effective for given populations, therefore increasing the applicability and usefulness in translating the review findings into practice.
This is particularly important for considerations of health inequalities. Given the diverse nature of health problems and the necessary interactions required between what are often complex interventions and individuals, it is likely that the underlying mechanisms supporting or undermining the effectiveness of interventions will vary and be context dependent . In the event of a deficiency or absence of evidence from review findings, programme theory can help reviewers to make assumptions about whether and how the intervention may indirectly result in differential effectiveness, which can then better inform the direction of future health inequalities research .
Operationalising programme theory in systematic reviews
Few empirical papers examine how reviewers utilise programme theory. A recent study by Kneale et al.  on the use of programme theory found that only five Cochrane Reviews published between September 2013 and September 2014, and 13 reviews published in the 3ie (International Initiative for Impact Evaluation) database of systematic reviews in 2013, mentioned use of either a logic model or theory of change. All of the reviews included in the Kneale et al.  study used programme theory to describe how the intervention might work a priori, but relatively few used it to inform other elements of the review process such as guiding selection criteria or to structure the synthesis. The study authors identified a need to develop good practice on how to use programme theory, logic models and theory of change in systematic reviews to avoid their use becoming merely a ‘tick-box exercise’. The conclusions of their study also support calls made elsewhere for researchers to develop a better understanding of the use and value of theory within the systematic review process .
However, the Kneale et al. review  offers only limited assistance to reviewers who seek to operationalise the use of programme theory, being based upon a relatively small sample of Cochrane and 3ie reviews and relying upon the included reviews explicitly articulating the terms ‘logic model’ or ‘theory of change’. Programme theory may be invoked either explicitly or implicitly without the use of such terms .
Using programme theory to guide action on health inequalities
One example of how programme theory can be operationalised to guide research on health inequalities is the typology of actions to tackle social inequalities in health . Acknowledging the lack of evidence in primary research on the differential effects of interventions on health across SES groups, Whitehead  calls for it to be ‘absolutely imperative’ that a theory-based approach is adopted to guide actions on reducing health inequalities.
In particular, Whitehead (, p. 476) calls for the best use of ‘intervention programme theories, to come up with plausible mechanisms for bringing about the desired change’. Using programme theory to understand how interventions may work to bring about an improvement in health across disadvantaged populations, Whitehead  suggests four levels of action in tackling the underlying causes of health inequalities (see Table 3).
Even though health inequalities research includes examples of the use of programme theory, there has been no exploration, to date, of its use in informing considerations of socio-economic inequalities in health in systematic reviews.
The purpose of this study is to assess if, how and the extent to which systematic reviewers operationalise the guidance on the use of programme theory in considerations of socio-economic inequalities in health.
The objectives are as follows:
To identify the extent to which reviewers operationalise the equity guidance in articulating considerations of whether and how differences in intervention effectiveness on health may be expected across SES groups
To identify how reviewers rationalise an understanding of what and how interventions have differential effects in or across SES populations (e.g. use of programme theory terminology and tools, authority for their decision based on theory, empirical evidence, personal experience)
To identify the extent to which reviewers are using programme theory to inform the review process (e.g. to predict or explain a change in health status, to inform the approach to the methods).
A systematic methodology review was undertaken. A methodology review is defined by the Cochrane Methodology Reviews Group  as ‘examining the evidence on methodological aspects of systematic reviews, randomised trials and other evaluations of health and social care’. The PRISMA guidance for the conduct and reporting of systematic reviews was adhered to during the review process .
Table 4 outlines the inclusion and exclusion criteria.
A systematic methodology review requires a different approach to the identification of the literature compared with conventional systematic reviews of empirical research . Methods for undertaking methodological reviews are undefined, but the focus of the search should aim for a systematic rather than exhaustive approach . Databases and websites searched were selected for their potential relevance in indexing records that were relevant to the review aims and objectives (see Table 5).
The search strategy was developed by one author (MM) with expertise in information retrieval in Ovid MEDLINE and adapted for other electronic databases. A second information scientist reviewed the search strategy. Full-text searches were undertaken in Google Scholar (Additional file 1).
Data collection and analysis
A two-stage process to filter studies was undertaken. Stage one involved an initial screening of titles and abstracts against the inclusion criteria. Studies were then categorised into (1) ‘probable’ studies that appeared to meet the inclusion criteria; (2) ‘possible’ studies that may be eligible but further information was required; and (3) excluded studies. Studies in the first two categories were taken forward to stage two where the full text of the study was retrieved and assessed for eligibility. Any studies not meeting the inclusion criteria after stage two were excluded with reasons noted. At least two reviewers (MM, NM, GC) independently screened all studies. Disagreements were resolved by discussion. Covidence.org was used to screen the studies. Multiple publications were analysed as one study.
The approach to quality assessment was guided by the aims of the review and follows the advice of Snilstveit (, p. 400) that ‘authors should systematically assess the quality of all studies included in their review, adopting criteria that are sensible for the question it is being used to answer’. No formal criteria exist to assess the use of programme theory in systematic reviews; therefore, all included studies were assessed against PRISMA Equity Extension criteria [1, 5] for reporting use of programme theory (see Table 6).
Data extraction and synthesis
Two review authors (MM, AC) extracted data independently from the included studies using pre-determined criteria. The data extraction form was piloted. Disagreements were resolved by discussion. Study characteristics extracted included author, year of publication, review topic, type of synthesis, types of studies included in the reviews, whether the study had an SES focus (where the primary aim related to effects within or across SES groups) or SES was accounted for (e.g. SES data were collected, subgroup analysis was undertaken on SES characteristics), intervention type, population, outcomes (relevant to SES), programme theory terminology used in the review, and use of programme theory to inform the review process. A template, adapted from that of Kneale et al. , was used to extract data on the use of programme theory. As this study is an exploration of the use of programme theory, a narrative approach to synthesis was undertaken.
A total of 5058 references were identified from the literature search. One hundred and eight references incorporated considerations of SES, either by collecting data relating to SES variables or by undertaking data analysis on SES variables. Forty references [20, 28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66] reporting on 37 studies (40%) articulated considerations of if, what or how interventions designed to improve the health of a population may have differential effects across different SES groups in systematic reviews and were included in this study (Fig. 1).
Included study characteristics
Table 7 highlights the characteristics of the included studies. The most reviewed topics were obesity- and diet-related issues. Twenty-eight reviews had an SES focus, whereby the aim or an objective of the review related to assessing either differential effectiveness of interventions across SES populations, or the effectiveness of interventions within a targeted group of socio-economically disadvantaged populations. Nine reviews accounted for effectiveness targeted at, or across, socio-economically disadvantaged populations (e.g. using subgroup analysis) but did not report it as being a specific aim or objective of the review.
Moving beyond simply aiming to measure the effectiveness of an intervention, 18 studies specifically aimed to examine which characteristics relating to the intervention may have different effects in and across SES populations. The majority of reviews undertook a narrative synthesis (n = 18) with the number of studies included in the reviews ranging from 5 to 463.
Despite articulating a programme theory on how they expected the intervention to work differently for SES populations, three reviews [58, 59, 61] reported only that data were collected on SES characteristics and did not offer any analysis of data by SES. Five reviews [36, 45, 64,65,66] reported either a lack of data on differential effects by SES within the included studies in the review or a lack of availability of studies for inclusion in the review.
Table 8 presents the results of the quality assessment. Only six reviews met all three quality criteria. One review  reported using a logic model but did not include it in the review. Of the eight reviews that reported adhering to the PRISMA Equity Extension guidance , only one  met all three criteria.
How reviewers rationalise an understanding of if, what and how/why interventions have differential effects in or across socio-economic populations
Defining programme theory terminology
Ten studies were explicit in the use of terminology for ‘programme theory’; however, not a single review mentioned the term ‘programme theory’. Two reviews [37, 66] referred to a ‘logic model’, three describe a conceptual model [45, 48, 65], while others referred to a logic pathway , conceptual framework , casual modelling framework  or, simply, framework (, child and adult reviews). The remaining studies were implicit, rather than explicit in their use of programme theory, describing their assumptions about what and how interventions may work differently for different SES populations without labelling it as ‘programme theory’.
‘If’ interventions work/do not work for different socio-economic groups
With the exception of three studies [53, 58, 63], all studies considered whether it was likely that interventions may have differential effects in health within or across SES populations a priori in the “Introduction” section. Such a verdict was largely made on the basis of the burden of disease in different SES groups. For example, the risk or prevalence of the disease was greater for lower SES populations compared with higher SES populations . Others described the burden in terms of higher rates of unhealthy behaviours, such as cigarette consumption , or lower rates of healthy behaviours, such as adherence to medications, among lower SES groups .
‘What’ interventions work/do not work for different socio-economic groups
All included studies considered what types of interventions are likely to work or not work for different SES groups. For example, Bambra et al.  suggest that tailored weight-loss interventions worked equally well or better in children in low SES groups. In a review of interventions to improve medication adherence, Laba et al.  found that physician- and patient-targeted interventions were most effective in socioeconomically disadvantaged populations. This was in contrast to a previous systematic review  which found larger improvements in medication adherence among the general population in physician only-targeted interventions.
Three of these studies [45, 59, 61] incorporated these considerations to inform only their a priori assumptions of what works for different SES groups, and five studies [37, 39, 46, 55, 63] incorporated these considerations only to explain what interventions work based on their review findings. The remaining 29 studies considered what types of interventions work or do not work both in their a priori assumptions and in explaining their review findings. Studies used programme theory to inform considerations of what works, as well as what does not work.
‘How’ interventions work/do not work for different socio-economic groups
All included studies also considered how interventions may or may not work for different SES groups. Of these, six reviews [45, 47, 53, 59, 61, 65] incorporated such considerations only a priori, and nine [37,38,39, 41, 43, 44, 46, 55, 63] discussed how interventions may have differential effects only to explain their review findings.
Twenty-two studies considered how interventions work or do not work both in their a priori assumptions and in their explanations of the review findings. For example, Laba et al.  suggest that differences in adherence behaviour between social groups can help explain why interventions which target both patient and physician are more effective for lower SES groups. Whereas in a systematic review of obesity-related lifestyle interventions, Magnee et al.  suggest that greater effects may be seen in higher SES groups because lower SES groups participate less. They go on to say that lower participation rates in lower SES groups may be due to either intervention design (e.g. recruitment strategies not reaching lower SES groups) or participant response (e.g. lower SES groups may not prioritise participating in interventions if they are experiencing other material and psychosocial problems).Overall, 22 studies incorporated considerations of both what and how interventions work or do not work in and across SES groups to both predict and explain their review findings.
Legitimisation of programme theory in systematic reviews
Thirteen studies (see Table 9) referenced the theoretical literature to inform an understanding of what or how/why interventions may lead to differential effectiveness within or across SES groups. One study mentioned the Oxford Food and Activity Behaviours taxonomy but did not provide a reference . The theoretical literature was used to inform both a priori assumptions (n = 10) ([20, 29] (child and adult reviews), [33, 36, 40, 49, 52, 54, 62]) and explanations of review findings (n = 7) [33, 40, 44, 51, 52, 54, 55]. In four studies [33, 40, 52, 54], the theoretical literature informed both a priori assumptions and explanations of review findings.
Collectively, the included studies acknowledged 24 unique theoretical references to support a priori assumptions and explanations of the review findings of what or how/why interventions may have different effects within and across SES groups (see Table 9). The most referenced were intervention theories: ‘How and why do interventions that increase health overall widen inequalities within populations?’  and ‘A typology of actions to tackle social inequalities in health’ .
The majority of the included studies (n = 36) used supplementary evidence to support their considerations of differential effectiveness. Supplementary evidence included empirical (qualitative or quantitative), descriptive, or policy-related evidence. Of note here is the use of supplementary evidence to examine intervention-generated inequalities (e.g. [69, 70]). In 32 of the included studies, the authors’ review findings were used to inform explanations of the review findings of differential effectiveness. Only two studies [20, 66] mentioned the involvement of other stakeholders in developing their logic models or programme theory. In some studies, only partial support for the programme theory was derived either from the literature or from the review findings.
Extent of use of programme theory to inform the review process
Table 10 outlines the extent to which programme theory is used to inform the review process within the included studies. The majority of authors outlined a programme theory in the “Introduction” and “Discussion” sections of the review to inform their assumptions (n = 32) or to provide explanations (n = 34) of what or how interventions may result in differential effects within or across SES groups. Despite not always being explicit in their use of programme theory, 29 review teams used this approach to inform both their a priori assumptions and explanations of review findings.
Twenty-one studies (see Table 10) tested their a priori programme theory of how they expected interventions to have different effects on health within or across SES populations and revisited or revised their programme theory to explain their review findings.
Interest in the use of programme theory to inform systematic reviews is increasing. In an attempt to enhance the applicability of review findings, reviewers are being encouraged to extend consideration beyond whether an intervention is effective or not, towards examining ‘what interventions work, for whom, and how’ .
While the relatively small proportion of systematic reviews (n = 108) incorporating considerations of socio-economic inequalities in health is in line with that reported elsewhere , it is clear that the push to move systematic reviews away from only considering if an intervention works towards a better understanding of what works and how, is slowly starting to emerge in the literature (see Table 10).
The lack of reference to the equity guidance within systematic reviews incorporating considerations of socio-economic inequalities in health may reflect the short interval between the publication of the guidance and the systematic reviews included in this study. However, given that eight studies in this sample did reference the PRISMA Equity guidance between 2013 and 2016, it may also suggest a lack of awareness of the guidance, or consideration of its relevance and importance, not only among systematic reviewers, but also among journal editors and peer reviewers. Therefore, to deliver better evidence on equity within research syntheses and systematic reviews, not only does the health research community need to increase awareness of equity guidance, but journal editors and peer reviewers also need to be proactive in encouraging reviewers to adopt the equity guidance when undertaking and reporting such reviews .
Alternatively, a lack of reference to the equity guidance in the reviews may indicate that reviewers are unsure about how to operationalise the guidance with respect on how the intervention may be expected to work within or across SES populations. Little empirical research has been undertaken on reviewers’ understanding of how to operationalise equity guidance for systematic reviews.
In studies that do use programme theory terminology (e.g., logic model, conceptual framework), the findings here agree with commentators who note that these terms are often inconsistently applied in the literature [16, 18]. No studies explicitly applied the term ‘programme theory’ to describe their assumptions. This supports assertions made elsewhere  that the use of theory to inform systematic reviews may not be explicitly articulated.
While it may not necessarily matter whether or not reviewers explicitly label a ‘programme theory’ to describe their understanding of how interventions may or may not work, not explicitly labelling it makes it harder to ascertain the extent to which reviewers are either consciously or unconsciously using programme theory to inform systematic reviews. The majority of the reviews included in this study described their programme theory in the narrative of the review without explicitly labelling it as a theory or using more graphical representations such as logic models or analytical frameworks. If reviewers are either considering this detail irrelevant to the methods section of their reports, or unconsciously using programme theory, then there is a need for greater clarity on operationalising the use of theory in systematic reviews.
In most of the included studies, programme theory was informed by low-level theory (i.e. assumptions based on supplementary evidence, e.g. empirical or policy). This is consistent with the findings of a previous study on the use of theory in systematic reviews . The lack of reference to more formal theory (e.g. intervention theories such as Whitehead’s  typology of policies and interventions, and behaviour change theories such as the Theory of Planned Behavior ) to inform reviewers’ assumptions or explanations of whether and how interventions may have differential effects may suggest that reviewers are using programme theory unconsciously. Among the included reviews that used formal theory, the most popular were intervention theories based on the target of the intervention (e.g. individual, community, societal).
While this study set out neither to examine the quality or richness of the programme theory used in systematic reviews nor to establish the fidelity or utility of use of theory, analysing interventions based only on the target of the intervention (i.e. universal versus individual) may not offer sufficient explanation of which components of the intervention process may work better for different SES groups and why.
Using only supplementary evidence to explain how interventions may work differently across SES populations may weaken the applicability of the review, especially given that it was often unclear as to how the supplementary evidence was identified. This is not limited to systematic reviews with a socio-economic health inequality focus. MacLure  observes that there is a strict inclusion and exclusion process set up for the main body of the review, only for other evidence to be brought into play in the discussion in an unsystematic way to interpret the findings.
While Anderson et al.  warn that, ‘conclusions drawn about intervention effects based simply on ad hoc criteria, rather than a theoretical understanding of the putative mechanisms of action of the intervention, can sometimes obscure aspects of the intervention that contribute to its effect’, using programme theory in an ad hoc and supplementary way to explain review findings may, therefore, result in conclusions based on poor-quality studies that may have little direct relevance to the topic under review.
The results of this study demonstrate that the use of programme theory to inform socio-economic health inequality considerations in the systematic review process remains in its infancy, is used implicitly, is often fragmented and is not implemented in a systematic way. The PRISMA Equity Extension explanation and elaboration document (, ‘Item 3: rationale’) highlights that the explicit reporting of programme theory can guide the reviewer in the choice of methods and synthesis. However, the findings of this study agree with Kneale et al. , in suggesting that programme theory is not yet seen as a tool that is integral to the whole review process.
Instead, reviewers are more likely to use programme theory in an ad hoc way at the start (a priori) and end of the review using supplementary evidence rather than to use formal theory or to test their assumptions or explanations of how interventions may or may not work for different SES groups. This may be due to the fact that many reviewers are using programme theory implicitly and therefore are unaware of its potential value in guiding the whole review process.
Using programme theory to inform only an a priori understanding of how reviewers expect the intervention to work across SES populations allows reviewers to ‘tick a box’ in the PRISMA Equity Extension guidance . However, integration of programme theory within the whole review process would provide a more systematic ‘uncovering’ of possible explanations that emerge a posteriori from the review findings for how interventions may work differently for different populations.
Establishing review intentions a priori has always been the approach in clinical effectiveness reviews in order to minimise bias . However, a priori assumptions of how an intervention may work may not necessarily be supported by the review findings. This may lead to an uninformed interpretation of the problem being imposed at the outset of the review. Furthermore, reviewers may not necessarily identify all the issues relating to how an intervention is expected to work a priori, and therefore, a revision of the programme theory may be required .
In avoiding the use of programme theory simply becoming what Kneale et al.  have described as a ‘tick-box’ exercise in demonstrating compliance with the PRISMA Equity Extension criteria, reviewers need to understand how programme theory can help in moving beyond simply basing their systematic reviews on theory towards securing a theoretical underpinning of the review analysis and synthesis.
The value of a programme theory approach lies in its ability to allow an acceptable, systematic, tested and refined a posteriori reasoning rather than post hoc assumption of how interventions may work. Twenty-one reviews in this study tested their programme theories; in doing so, they present review findings that are tested interpretations or explanations, rather than unbiased observations, thus strengthening the validity and applicability of the review findings.
Using programme theory to articulate considerations of if, what and how interventions work for different populations will require reviewers to include more diverse forms of evidence beyond randomised controlled trials. Only 12 reviews in this study included evidence other than quantitative research to support their programme theory, largely using a narrative synthesis approach.
If reviewers of effectiveness studies are to be encouraged to consider health inequalities in systematic reviews, this will require a paradigm shift: they will need to move from a positivist stance in considering only ‘if and what works’ towards a more realist informed way of thinking to consider ‘what works, for whom, and under what circumstance’. It requires a different approach to analysis moving away from a purely observational approach (i.e. how often, how much) to one that incorporates a more interpretive approach. This shift is hugely challenging, requiring skills in understanding theoretical sensitivity and in being able to generate, test and refine possible explanations.
While there is a greater need to understand the advantages and disadvantages of different synthesis approaches, the decision on what synthesis approach to undertake and whether to conduct separate parallel reviews has to be balanced with considerations of time, resources and the skills and expertise within the review team . Given that it is unrealistic to expect all reviewers to accompany each systematic review with a parallel realist review, the challenge is how reviewers of effectiveness studies can incorporate realist principles within reviews of effectiveness to inform considerations of what and how interventions may work differently across SES populations within a single review.
Establishing an a priori programme theory of if, what and how an intervention is expected to work across different SES groups, testing and refining it based on the review findings may offer reviewers a way forward. Further research is currently underway by the author (MM) on a theory-led meta-framework of factors associated with differential effects of interventions across SES groups.
Strengths and limitations of the methodological review
A key strength of this study was the inclusion of full-text searches rather than simply undertaking title and abstract searches alone. Full-text searching took place in order to overcome the limitations associated with the fact that SES analysis may have been undertaken but not reported in journal abstracts. In addition, during the screening phase, if no mention was made in the title or abstract of SES, the full text of the paper was examined.
A further strength was to include studies that were either explicit or implicit in their consideration of what and how interventions may have different effects within and across SES populations. This extends a previous study on the use of programme theory  and recognises that the articulation of what works, for whom and under what circumstance may not necessarily be recognised as ‘programme theory’ or labelled as a ‘logic model’ or ‘conceptual framework’. Indeed, as this study demonstrates, reviewers rarely identify with these labels even when outlining a programme theory. The involvement of two reviewers to independently extract the data on programme theory aimed to reduce the chances of missing relevant information.
Only systematic reviews published since the introduction of the PRISMA Equity Guidance  were included in this study. The short interval between the publication of the guidance and the systematic reviews included in this study may result in a more modest indication of the extent to which reviewers are operationalising the PRISMA guidance in exploring how interventions may result in differential effectiveness within or across SES populations. Studies in which it was not possible to separate out the analysis for SES were excluded; therefore, it may be that other theories relating to what works and how/why they work may have been missed.
Unless explicitly stated in the included studies, this study was unable to assess the way in which different members of the review team or stakeholders contributed to the programme theory. Furthermore, this study only assessed the extent of use of programme theory if reviewers were explicit in reporting their use of it to inform their review processes.
This study did not set out to examine the fidelity, utility or richness of the programme theory. For example, Magnee et al. (, e62) described why they assume that differential effectiveness across SES populations may arise, because “more highly educated people may be better equipped to benefit from interventions”, yet they failed to explain in what way or why highly educated people are ‘better equipped’. In another example, Mizdrak et al., (, Introduction) stated that they expected differential effectiveness to occur because “low income purchasers may react differently to changes in food price than high income purchasers” but it is not clear in what way or why low income purchasers ‘may react differently’. Therefore, the use of the PRISMA Equity Extension criteria  in the quality assessment only allows for an assessment of the quality of the reporting of programme theory.
Given the lack of evidence in primary research on the differential effects of interventions on health across SES groups, Whitehead  considers it ‘absolutely imperative’ that a theory-based approach is adopted to guide actions on reducing health inequalities. Despite the PRISMA Equity Extension guidance recommending the use of programme theory, this study demonstrates that use of the guidance to inform considerations of if, what and how interventions lead to differential effects on health within and across SES groups in the systematic review process is not yet widely adopted and is fragmented.
Encouraging reviewers of effectiveness studies to consider health inequalities in systematic reviews requires a paradigm shift in thinking, from a positivist (i.e. ‘if, and what works’), towards a realist informed way of thinking (i.e. ‘what works, for whom, and under what circumstance’). The fact that reviewers are more likely to use programme theory implicitly, in an ad hoc descriptive way, and use supplementary evidence to support their assumptions of how interventions work rather than use more ‘formal’ theories, suggests that reviewers are unconsciously using programme theory and are not fully exploiting its potential in informing the review process.
International Initiative for Impact Evaluation database
Population, intervention, comparator and outcome
Preferred Reporting Items for Systematic Reviews and Meta-Analysis
Welch V, Petticrew M, Tugwell P, Moher D, O'Neill J, Waters E, White H. PRISMA-Equity 2012 Extension: reporting guidelines for systematic reviews with a focus on health equity. PLoS Med. 2012;9(10):e1001333.
Tripney J, Roulstone A, Hogrebe N, Vigurs C, Schmidt E, Stewart R: Interventions to improve the labour market situation of adults with physical and/or sensory disabilities in low- and middle-income countries: a systematic review. 2015.
Tugwell P, Petticrew M, Kristjansson E, Welch V, Ueffing E, Waters E, Bonnefoy J, Morgan A, Doohan E, Kelly MP. Assessing equity in systematic reviews: realising the recommendations of the Commission on Social Determinants of Health. BMJ. 2010;341:c4739.
Ueffing E TP, Welch V, Petticrew M, Kristjansson E for the Campbell and Cochrane Equity Methods Group.: Equity checklist for systematic review authors. 2012:Version 2012-2010-2002.
Welch V, Petticrew M, Petkovic J, Moher D, Waters E, White H, Tugwell P. Extending the PRISMA statement to equity-focused systematic reviews (PRISMA-E 2012): explanation and elaboration. J Clin Epidemiol. 2016;70:68–89.
Welch V, Brand K, Kristjansson E, Smylie J, Wells G, Tugwell P. Systematic reviews need to consider applicability to disadvantaged populations: inter-rater agreement for a health equity plausibility algorithm. BMC Med Res Methodol. 2012;12:187.
MDR M. Health inequalities and evidence synthesis as part of the CLAHRC NWC. In: Position Statement. Liverpool: Evidence Synthesis Theme, CLARHC NWC, University of Liverpool; 2015.
Davies KS. Formulating the evidence based practice question: a review of the frameworks. Evid Based Libr Inf Pract. 2011;6(2).
Anderson LM, Oliver S, Michie S, Rehfuss E, Noyes J, Shemilt I. Investigating complexity in systematic reviews of interventions using a spectrum of methods. J Clin Epidemiol. 2013;66:1223–9.
Davidoff F, Dixon-Woods M, Leviton L, Michie S. Demystifying theory and its use in improvement. BMJ Qual Saf. 2015;24(3):228–38.
Funnells SCRP, J. Purposeful program theory: effective use of theories of change and logic models. Jossey-Bass: California; 2011.
Kneale D, Thomas J, Harris K. Developing and optimising the use of logic models in systematic reviews: exploring practice and good practice in the use of programme theory in reviews. PLoS One. 2015;10(11):e0142187.
Popay J, Sowden A, Petticrew M, Arai L, Rodgers M, Britten N, Roen K, Duffy S.: Guidance on the conduct of narrative synthesis in systematic reviews. A product from the ESRC methods programme. 2006.
Noyes JL, Lewin S: Supplemental guidance on selecting a method of qualitative evidence synthesis, and integrating qualitative evidence with Cochrane intervention reviews. In: Supplementary guidance for inclusion of qualitative research in Cochrane systematic reviews of interventions version 1 (updated August 2011). edn. Edited by Noyes J, HannesK, Harden A, Harris J, Lewin S, Lockwood C.: Cochrane Collaboration Qualitative Methods Group; 2011.
Tranfield D, Denyer D, Smart P. Towards a methodology for developing evidence-informed management knowledge by means of systematic review. Br J Manag. 2003;14:207–22.
Noyes J, Hendry M, Booth A, Chandler J, Lewin S, Glenton C, Garside R. Current use was established and Cochrane guidance on selection of social theories for systematic reviews of complex interventions was developed. J Clin Epidemiol. 2016;75:78–92.
Petticrew M. Time to rethink the systematic review catechism? Moving from ‘what works’ to ‘what happens’. Syst Rev. 2015;4:36.
Maden M. Consideration of health inequalities in systematic reviews: a mapping review of guidance. Syst Rev. 2016;5(1):202.
Guise JM, Chang C, Viswanathan M, Glick S, Treadwell J, Umscheid C, Whitlock E, Fu R, Berliner E, Paynter R, Anderson J, Motu’apuaka M, Trikalinos T. Systematic reviews of complex multicomponent health care interventions. Research white paper. Rockville, MD: Agency for Healthcare Research and Quality; 2014.
McLean S, Gee M, Booth A, Salway S, Nancarrow S, Cobb M, Bhanbhro S. Targeting the Use of Reminders and Notifications for Uptake by Populations (TURNUP): a systematic review and evidence synthesis (structured abstract). In: Health technology assessment database: Health Services and Delivery Research; 2014.
Baxter S, Killoran A, Kelly MP, Goyder E. Synthesizing diverse evidence: the use of primary qualitative data analysis methods and logic models in public health reviews. Public Health. 2010;124(2):99–106.
Whitehead M. A typology of actions to tackle social inequalities in health. J Epidemiol Community Health. 2007;61(6):473–8.
Cochrane methodology reviews [methodology.cochrane.org].
Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred Reporting Items for Systematic Reviews and Meta-analyses: the PRISMA statement. BMJ. 2009;339:b2535.
Gentles SJ, Charles C, Nicholas DB, Ploeg J, McKibbon KA. Reviewing the research methods literature: principles and strategies illustrated by a systematic overview of sampling in qualitative research. Syst Rev. 2016;5(1):172.
Lilford RJ, Richardson A, Stevens A, Fitzpatrick R, Edwards S, Rock F, Hutton JL. Issues in methodological research: perspectives from researchers and commissioners. Health Technol Assess. 2001;5(8):1–57.
Snilstveit B. Systematic reviews: from ‘bare bones’ reviews to policy relevance. J Dev Effect. 2012;4:388–408.
Backholer K, Sarink D, Beauchamp A, Keating C, Loh V, Peeters A. The effect of a sugar sweetened beverage tax among different socioeconomic groups: a systematic review. Obes Res Clin Pract. 2014;8:4–5.
Bambra C, Hillier F, Cairns-Nagi J, Kasim A, Moore H, Summerbell C. How effective are interventions at reducing socioeconomic inequalities in obesity among children and adults? Two systematic reviews (structured abstract). In: Health technology assessment database: Public Health Research; 2015.
Hillier-Brown FC, Bambra CL, Cairns JM, Kasim A, Moore HJ, Summerbell CD. A systematic review of the effectiveness of individual, community and societal-level interventions at reducing socio-economic inequalities in obesity among adults. Int J Obes. 2014;38(12):1483–90.
Hillier-Brown FC, Bambra CL, Cairns J-M, Kasim A, Moore HJ, Summerbell CD: A systematic review of the effectiveness of individual, community and societal level interventions at reducing socioeconomic inequalities in obesity amongst children. BMC Public Health 2014, 14(1):834-834.
Cairns J-M, Bambra C, Hillier-Brown FC, Moore HJ, Summerbell CD. Weighing up the evidence: a systematic review of the effectiveness of workplace interventions to tackle socio-economic inequalities in obesity. J Public Health. 2015;37(4):659–70.
Beauchamp A, Backholer K, Magliano D, Peeters A. The effect of obesity prevention interventions according to socioeconomic position: a systematic review (provisional abstract). Obes Rev. 2014;15:541–54.
Boelsen-Robinson T, Peeters A, Beauchamp A, Chung A, Gearon E, Backholer K. A systematic review of the effectiveness of whole-of-community interventions by socioeconomic position. Obes Rev. 2015;16(9):806–16.
Brown TJ, Todd A, O'Malley C, Moore HJ, Husband AK, Bambra C, Kasim A, Sniehotta FF, Steed L, Smith S, et al. Community pharmacy-delivered interventions for public health priorities: a systematic review of interventions for alcohol reduction, smoking cessation and weight management, including meta-analysis for smoking cessation. BMJ Open. 2016;6(2):e009828.
Brown TJ, Todd A, O'Malley CL, Moore HJ, Husband AK, Bambra C, Kasim A, Sniehotta FF, Steed L, Summerbell CD. Community pharmacy interventions for public health priorities: a systematic review of community pharmacy-delivered smoking, alcohol and weight management interventions (structured abstract). In: Health technology assessment database: Public Health Research; 2016.
Brown T, Platt S, Amos A. Equity impact of population-level interventions and policies to reduce smoking in adults: a systematic review. Drug Alcohol Depend. 2014;138:7–16.
Brown T, Platt S, Amos A. Equity impact of European individual-level smoking cessation interventions to reduce smoking in adults: a systematic review. Eur J Pub Health. 2014;24(4):551–6.
Brown T, Platt S, Amos A. Equity impact of interventions and policies to reduce smoking in youth: systematic review. Tob Control. 2014;23(S1):e98–e105.
Bull ER, Dombrowski SU, McCleary N, Johnston M. Are interventions for low-income groups effective in changing healthy eating, physical activity and smoking behaviours? A systematic review and meta-analysis. BMJ Open. 2014;4(11):e006046.
Cleland V, Granados A, Crawford D, Winzenberg T, Ball K. Effectiveness of interventions to promote physical activity among socioeconomically disadvantaged women: a systematic review and meta-analysis. Obes Rev. 2013;14(3):197–212.
Everson-Hock ES, Johnson M, Jones R, Woods HB, Goyder E, Payne N, Chilcott J. Community-based dietary and physical activity interventions in low socioeconomic groups in the UK: a mixed methods systematic review. Prev Med. 2013;56(5):265–72.
Gardner MP, Adams A, Jeffreys M. Interventions to increase the uptake of mammography amongst low income women: a systematic review and meta-analysis. PLoS One [Electronic Resource]. 2013;8(2):e55574.
Hill S, Amos A, Clifford D, Platt S. Impact of tobacco control interventions on socioeconomic inequalities in smoking: review of the evidence. Tob Control. 2014;23(S1):e89–97.
Hollands GJ, Shemilt I, Marteau TM, Jebb SA, Lewis HB, Wei Y, Higgins JP, Ogilvie D. Portion, package or tableware size for changing selection and consumption of food, alcohol and tobacco. Cochrane Database Syst Rev. 2015;(9):Cd011045.
Kader M, Sundblom E, Elinder LS. Effectiveness of universal parental support interventions addressing children's dietary habits, physical activity and bodyweight: a systematic review. Prev Med. 2015;77:52–67.
Kendrick D, Young B, Mason-Jones AJ, Ilyas N, Achana FA, Cooper NJ, Hubbard SJ, Sutton AJ, Smith S, Wynn P, et al. Home safety education and provision of safety equipment for injury prevention (review). Evid Based Child Health Cochrane Rev J. 2013;8(3):761–939.
Kristjansson E, Francis DK, Liberato S, Benkhalti Jandu M, Welch V, Batal M, Greenhalgh T, Rader T, Noonan E, Shea B et al: Food supplementation for improving the physical and psychosocial health of socio-economically disadvantaged children aged three months to five years. In: Cochrane Database Syst Rev: John Wiley & Sons, Ltd; 2015.
Laba TL, Bleasel J, Brien JA, Cass A, Howard K, Peiris D, Redfern J, Salam A, Usherwood T, Jan S. Strategies to improve adherence to medications for cardiovascular diseases in socioeconomically disadvantaged populations: a systematic review (provisional abstract). Int J Cardiol. 2013;167:2430–40.
Laws R, Campbell KJ, van der Pligt P, Russell G, Ball K, Lynch J, Crawford D, Taylor R, Askew D, Denney-Wilson E: The impact of interventions to prevent obesity or improve obesity related behaviours in children (0-5 years) from socioeconomically disadvantaged and/or indigenous families: a systematic review. BMC Public Health 2014, 14(1):779–779
Magnee T, Burdorf A, Brug J, Kremers SP, Oenema A, van Assema P, Ezendam NP, van Genugten L, Hendriksen IJ, Hopman-Rock M, et al. Equity-specific effects of 26 Dutch obesity-related lifestyle interventions. Am J Prev Med. 2013;44(6):e57–66.
McGill R, Anwar E, Orton L, Bromley H, Lloyd-Williams F, O'Flaherty M, Taylor-Robinson D, Guzman-Castillo M, Gillespie D, Moreira P et al: Are interventions to promote healthy eating equally effective for all? Systematic review of socioeconomic inequalities in impact. BMC Public Health 2015, 15(1):457-457.
Mizdrak A, Scarborough P, Waterlander WE, Rayner M. Differential responses to food price changes by personal characteristic: a systematic review of experimental studies. PLoS ONE [Electronic Resource]. 2015;10(7):e0130320.
Moore GF, Littlecott HJ, Turley R, Waters E, Murphy S. Socioeconomic gradients in the effects of universal school-based health behaviour interventions: a systematic review of intervention studies. BMC Public Health. 2015;15(1):1–15.
Moredich CA, Kessler TA. Physical activity and nutritional weight loss interventions in obese, low-income women: an integrative review (provisional abstract). J Midwifery Womens Health. 2014;59:380–7.
Rojas-Garcia A, Ruiz-Perez I, Rodriguez-Barranco M, Goncalves Bradley DC, Pastor-Moreno G, Ricci-Cabello I. Healthcare interventions for depression in low socioeconomic status populations: a systematic review and meta-analysis. Clin Psychol Rev. 2015;38:65–78.
Sarink D, Peeters A, Freak-Poli R, Beauchamp A, Woods J, Ball K, Backholer K. The impact of menu energy labelling across socioeconomic groups: a systematic review. Appetite. 2016;99:59–75.
Ciciriello S, Johnston Renea V, Osborne Richard H, Wicks I, de Kroo T, Clerehan R, O'Neill C, Buchbinder R. Multimedia educational interventions for consumers about prescribed and over-the-counter medications. In: Cochrane Database Syst Rev: John Wiley & Sons, Ltd; 2013.
Ejemot-Nwadiaro Regina I, Ehiri John E, Arikpo D, Meremikwu Martin M, Critchley Julia A. Hand washing promotion for preventing diarrhoea. In: Cochrane database of systematic reviews: John Wiley & Sons, Ltd; 2015.
Gittelsohn J, Lee-Kwan SH, Batorsky B. Community-based interventions in prepared-food sources: a systematic review. Prev Chronic Dis. 2013;10:E180.
Gurol-Urganci I, de Jongh T, Vodopivec-Jamsek V, Atun R, Car J. Mobile phone messaging reminders for attendance at healthcare appointments. In: Cochrane Database Syst Rev: John Wiley & Sons, Ltd; 2013.
Hartmann-Boyce J, Jebb SA, Fletcher BR, Aveyard P. Self-help for weight loss in overweight and obese adults: systematic review and meta-analysis. Am J Public Health. 2015;105(3):e43–57.
Kroon Féline PB, van der Burg Lennart RA, Buchbinder R, Osborne Richard H, Johnston Renea V, Pitt V. Self-management education programmes for osteoarthritis. In: Cochrane Database Syst Rev: John Wiley & Sons, Ltd; 2014.
Lutge Elizabeth E, Wiysonge Charles S, Knight Stephen E, Sinclair D, Volmink J. Incentives and enablers to improve adherence in tuberculosis. In: Cochrane Database Syst Rev: John Wiley & Sons, Ltd; 2015.
Pega F, Carter K, Blakely T, Lucas Patricia J. In-work tax credits for families and their impact on health status in adults. In: Cochrane Database Syst Rev: John Wiley & Sons, Ltd; 2013.
Augustincic Polec L, Petkovic J, Welch VA, Ueffing E, Ghogomu ET, Pardo JP, Grabowsky M, Attaran A, Wells GA, Tugwell P: Strategies to increase the ownership and use of insecticide-treated bednets to prevent malaria. In Cochrane Database Syst Rev: John Wiley & Sons, Ltd; 2015
Cutrona SL, Choudhry NK, Stedman M, Servi A, Liberman JN, Brennan T, Fischer MA, Brookhart MA, Shrank WH. Physician effectiveness in interventions to improve cardiovascular medication adherence: a systematic review. J Gen Intern Med. 2010;25(10):1090–6.
White M, Adams J, Heywood P. How and why do interventions that increase health overall widen inequalities within populations? In: Barbones S, editor. Health, Inequality and Public Health. Bristol: Policy Press; 2009.
Lorenc T, Petticrew M, Welch V, Tugwell P. What types of interventions generate inequalities? Evidence from systematic reviews. J Epidemiol Community Health. 2013;67(2):190–3.
Arblaster L, Lambert M, Entwistle V, Forster M, Fullerton D, Sheldon T, Watt I. A systematic review of the effectiveness of health service interventions aimed at reducing inequalities in health. J Health Serv Res Policy. 1996;1(2):93–103.
Petticrew M, Welch V, Tugwell P. ‘It is surely a great criticism of our profession...’ the next 20 years of equity-focused systematic reviews. J Epidemiol Community Health. 2014;68(4):291–2.
Welch V, Petticrew M, Ueffing E, Benkhalti Jandu M, Brand K, Dhaliwal B, Kristjansson E, Smylie J, Wells GA, Tugwell P. Does consideration and assessment of effects on health equity affect the conclusions of systematic reviews? A methodology study. PLoS One. 2012;7(3):e31360.
Azjen I. The theory of planned behavior. Organ Behav Hum Decis Process. 1991;50:179–211.
MacLure M. ‘Clarity bordering on stupidity’: Where’s the quality in systematic review? In: British Educational Research Association annual conference. Manchester; 2004.
Cochrane Handbook for Systematic Reviews of Intervetions.: The Cochrane Collaboration; 2011.
Rohwer A, Booth, A., Pfadenhauer, L., Brereton, L., Gerhardus, A., Mozygemba, K., Oortwijn, W., Tummers, M., Van Der Wilt, G.J., Rehfuess, E.: Guidance on the use of logic models in health technology assessments of complex interventions. 2016.
Dahlgren G, Whitehead, M.: Tackling inequalities in health: what can we learn from what has been tried? In: Working paper prepared for the Kings Fund International Seminar on Tackling Inequalities in Health. Ditchley Park, Oxfordshire; 1993.
Sutton S: Determinants of health related behaviours: theoretical and methodological issues. In: The SAGE handbook of Health Psychology. Edited by Sutton S, Baum, A., Johnston, M. London: SAGE Publications Ltd; 2004: 94-126.
Thanks to Eleanor Kotas, Information Scientist, Liverpool Reviews and Implementation Group, University of Liverpool, for reviewing the search strategy.
This review was undertaken as part of a wider PhD (MM) examining the incorporation of health inequalities in evidence synthesis funded by The National Institute for Health Research Collaboration for Leadership in Applied Health Research and Care North West Coast (NIHR CLAHRC NWC). NMcM is a PhD student funded by The NIHR CLAHRC NWC. AB is funded by the University of Sheffield to collaborate via the Translating Knowledge into Action (TK2A) theme, NIHR CLAHRC Yorkshire and Humber. MG and RD are part funded by The NIHR CLAHRC NWC.
Availability of data and materials
Ethics approval and consent to participate
The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health.
AB is an editor for the Systematic Reviews journal.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Search strategy. (DOCX 15 kb)