Skip to main content
Download PDF

Protocol for a systematic review of the use of qualitative comparative analysis for evaluative questions in public health research

Systematic Reviews volume 8, Article number: 252 (2019) Cite this article

Abstract

Background

There is an increasing recognition that health intervention research requires methods and approaches that can engage with the complexity of systems, interventions, and the relations between systems and interventions. One approach which shows promise to this end is qualitative comparative analysis (QCA), which examines casual complexity across a medium to large number of cases (between 10 and 60+), whilst also being able to generalise across those cases. Increasingly, QCA is being adopted in public health intervention research. However, there is a limited understanding of how it is being adopted. This systematic review will address this gap, examining how it is being used to understand complex causation; for what settings, populations and interventions; and with which datasets to describe cases.

Methods

We will include published and peer-reviewed studies of any public health intervention where the effects on population health, health equity, or intervention uptake are being evaluated. Electronic searches of PubMed, Scopus, Web of Science (incorporating Social Sciences Citation Index and Arts & Humanities Citation Index), Microsoft Academic, and Google Scholar will be performed. This will be supplemented with reference citation tracking and personal contact with experts to identify any additional published studies. Search results will be single screened, with machine learning used to check these results, acting as a ‘second screener’. Any disagreement will be resolved through discussion. Data will be extracted from full texts of eligible studies, which will be assessed against inclusion criteria, and synthesised narratively, using thematic synthesis methods.

Discussion

This systematic review will provide an important map of the increasing use of QCA in public health intervention literature. This review will identify the current scope of research in this area, as well as assessing claims about the utility of the method for addressing complex causation in public health research. We will identify implications for better reporting of QCA methods in public health research and for reporting of case studies such that they can be used in future QCA studies.

Systematic review registration

PROSPERO, CRD42019131910

Peer Review reports

Background

Identifying the pathways that lead to the success of health interventions is crucial for public health research. However, assessing causal relationships between interventions and outcomes across contexts is challenging [1,2,3], as multiple interacting components come together in often complex ways [4,5,6] and ‘inevitable adaptations’ take place as health interventions get implemented within local contexts [3]. Recent work [7] has pointed to the limitations of using randomised control trials (RCTs) to examine complex health interventions, as they are often unrepresentative of everyday practices across contexts: there is increased input to achieve fidelity, participants in trials may be more committed to the intervention, and context plays an important role in shaping the intervention. Understanding complexity thus requires approaches that draw on real world case studies [2, 8] and can identify pathways to successful implementation of, and effective outcomes from, health interventions across diverse contexts.

Qualitative comparative analysis (QCA) is one approach that has been increasingly adopted in public health research and evaluation [8,9,10,11,12] to examine complexity. Developed by Charles Ragin [13, 14], QCA is a set-theoretic method, which ‘bridges’ quantitative (large-n statistical analyses) and qualitative (small-n case study) techniques to analyse complex causation across contexts [15]. A QCA analysis can involve a medium to large number of cases (between 10 and 60+), whilst also being able to generalise across those cases [16, 17]. QCA is often described as being suitable for ‘small-n’ scenarios and is less frequently used in large datasets, though some have used it with some success [18].

A QCA approach provides a tool for examining what conditions (alone or in combination with other conditions) are ‘necessary’ or ‘sufficient’ to produce some outcome [19]. QCA involves using Boolean algebra and formal logic to examine the presence or absence of conditions, and the effect on an outcome. QCA uses ‘truth tables’, which ‘list all the possible combinations of casual conditions and the empirical outcome associated with each configuration … to identify explicit connections between combinations of casual conditions and outcomes’ [19]. When first developed degree of membership was limited by nominal-scale variables, crisp sets, but has since been revised by Ragin to include ‘fuzzy sets’, which scale degree of membership from 0.0 to 1.0 [13, 19].

Warren et al. have argued that QCA offers an important contribution to mixed method public health evaluation methodologies [20]. Others have identified opportunities to use QCA for informing the translation of research/interventions into practice and policy [21, 22]. There is a growing body of scholarship in public health that does use QCA methods for these, and other, aims. However, to date, there has been no systematic review of the use of QCA in public health literature and how it is being adopted. This systematic review will address this gap, examining how it is being used in public health research, including the methods and cases used to explore complexity, the populations it is being applied to, and the types of intervention research that QCA has been used to examine. This is a timely point to assess the field, in order to identify strengths and weaknesses, gaps, and implications for how to better report QCA methods and the case studies on which future QCA studies might be based.

Study aim

The primary aim of the review is to understand what is known about the purpose, role, and use of qualitative comparative analysis (QCA) in research on the uptake and public health effects of interventions.

The proposed systematic review will answer the following research questions:

  1. 1.

    How is QCA used for public health evaluation? What populations, settings, methods used in source case studies, unit/s and level of analysis (‘cases’), and ‘conditions’ (often referred to as ‘variables’ in public health) have been included in QCA studies?

  2. 2.

    What strengths and weaknesses have been identified by researchers who have used QCA to understand complex causation in public health evaluation research?

  3. 3.

    How is QCA used to understand the contexts in which interventions increase or decrease health inequities?

  4. 4.

    What are the existing gaps in, and strengths and weakness of, the QCA literature, and what implications do these have for future QCA studies for public health?

The study will also identify implications for better reporting of QCA methods in public health research and for reporting case studies in general.

Methods/design

Our systematic review protocol was registered with the International Prospective Register of Systematic Reviews (PROSPERO) on 29 April 2019 (CRD42019131910). The protocol was prepared using the Preferred Reporting Items for Systematic Reviews and Meta-Analysis Protocols (PRISMA-P) 2015 statement [23].

Eligibility criteria

Papers included in the systematic review will be published, be peer reviewed, and report on studies that have undertaken a qualitative comparative analysis (QCA) to examine casual complexity in the implementation, uptake, and/or effects of public health interventions. There will be no date restrictions.

Inclusion criteria

Types of participants

Studies will include any population (including national, geographical communities, population sub-groups). As the review will be examining the types of populations included in QCA studies, all study participants will be included. This includes youth and adult populations, their families, and public health professionals.

Types of interventions

QCA studies that incorporate case studies of any kind of public health interventions will be included. Public health interventions are defined here as those focused on the promotion or protection of health, or the prevention of ill health for populations. This is in contrast to clinical interventions, which focus on the prevention or treatment of illness in individuals [24]. This will include any interventions for where benefits to population health are the stated outcomes.

Types of outcome measures

To be included, QCA studies must identify an outcome (the phenomenon that the QCA is aiming to explain) that is relevant to public health, in which the effects on population health, health equity, or intervention uptake are being evaluated.

Types of studies

Papers reporting studies that use crisp and/or fuzzy sets will be included. QCA can be included as part of the analysis, or as the only tool used in analysing data.

Papers reporting both primary studies and systematic reviews which use QCA in their analysis will be included.

Exclusion criteria

Publications not in English.

Studies where no intervention is evaluated, including studies that use QCA to examine public health infrastructure (i.e. staff training for low-skilled health staff), or papers that report on prevalence of health issues (i.e. prevalence of child mortality).

Search strategy

The search strategy will use the following phrases ‘Qualitative Comparative Analysis’ and ‘QCA’, which will be combined with keywords (i.e. ‘health’, ‘public health’, ‘intervention’, and ‘wellbeing’). Database searches will be conducted in PubMed, Scopus, Web of Science (incorporating Social Sciences Citation Index and Arts & Humanities Citation Index), Microsoft Academic, and Google Scholar. Study design must be QCA; no date limits will be imposed on the search. The search will be updated at the end of the review to ensure inclusion of all relevant studies. This will also be supplemented by reviewing reference lists of primary studies identified in the review, reviewing studies that cite these primary studies, as well as inviting experts to comment on whether the final list of studies omits any published studies known to them.

Study selection and data collection

Data and records will be maintained by the lead investigator (BH) and stored on a shared secure platform for access by all investigators.

Selection process

Titles and abstracts identified by the bibliographic search and additional sources will be single screened against the criteria above, with machine learning employed to check these results, acting as a ‘second screener’ [25]. ‘EPPI-Reviewer’ Version 4 software, or equivalent machine learning software, will be used to screen titles and abstracts. After removing duplicates, each study will be categorised as ‘include’, ‘exclude’, or ‘unsure’. The full texts of articles categorised as ‘include’ and ‘unsure’ will be retrieved and then assessed against inclusion criteria by reviewers. Those categorised as ‘unsure’ and/or where there are disagreements will be resolved through discussion, and if necessary, a third reviewer will mediate any unresolved differences.

Data extracted will be tabulated, using a standardised form approved by all investigators to ensure consistency (see Table 1). This will include extracting the following information for each included paper: title, author, rationale for QCA, context (setting, case level of analysis, and population characteristics), health intervention, health outcome/s, methods of data generation pertaining to the cases included (qualitative and/or quantitative), findings, and study conclusions. The reviewer/s will also extract any strengths and limitations outlined in each paper. Study authors will be contacted regarding any uncertainties.

Table 1 Study data collection
Full size table

Assessment of risk of bias

Quality assessment

The objective of this review is to examine how QCA is currently used in the public health literature. There are no established reporting criteria for quality, so no studies will be excluded on the basis of quality. However, there are suggestions as to how to best report QCA [15, 16]. We will utilise four key criteria from these recommendations to map current reporting of the method. These will be evidence of familiarity with cases, justification for selection of cases, availability of raw data matrix and ‘truth table’ outcomes, and solution formula reported.

Data synthesis

Data will be synthesised narratively, using thematic synthesis methods [26,27,28]. Final studies included will, following a descriptive analysis, be grouped by outcome measures, as well as context of intervention (e.g. school). We will assess claims about the utility of the method for addressing complex causation in public health research by identifying how far the published QCA study was able to identify necessary and sufficient conditions for the selected outcome. We will analyse separately (as a sub sample) those studies with an outcome related to health equity or inequalities, to assess how far QCA has been used to date in this area.

Discussion

QCA has been increasingly used in public health research, although the utility of the method for producing policy-relevant findings has been questioned [29]. The use of QCA in public health research therefore warrants further attention. We will map current use of the method, including identifying what types of intervention and setting have been addressed and what strengths and weaknesses researchers have identified in using QCA to address questions of complex causation. We will assess the extent to which current reporting in public health matches recommendations for best practice within the QCA research community, and the implications of these recommendations and current practice for the public health research community. As QCA relies on detailed knowledge of cases, we will also comment on the implications for public health researchers publishing stand-alone case studies, which might be used in future QCA studies. At present, there has been no systematic review of the use of QCA in public health literature and how it is being adopted; this systematic review will fill this gap, with the aim to inform, guide, and assess use of QCA in future complex health systems research.

There are several limitations to this study. Including only published studies from a rapidly emerging field may not be representative of the most recent applications of the method. In addition, we are only able to include those in English, which may underrepresent high-quality research from some regions.

Availability of data and materials

All relevant data are within the paper and its supporting information files.

Abbreviations

QCA:

Qualitative comparative analysis

RCT:

Randomised control trial

References

  1. Greenhalgh T, Papoutsi C. Studying complexity in health services research: desperately seeking an overdue paradigm shift. BMC Med. 2018;16(1):95.

    Article  Google Scholar 

  2. Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ. 2015;350:h1258.

    Article  Google Scholar 

  3. Cohn S, Clinch M, Bunn C, Stronge P. Entangled complexity: why complex interventions are just not complicated enough. J Health Serv Res Policy. 2013;18(1):40–3.

    Article  Google Scholar 

  4. Kelly MP, Russo F. Causal narratives in public health: the difference between mechanisms of aetiology and mechanisms of prevention in non-communicable diseases. Sociol Health Illn. 2018;40(1):82–99.

    Article  Google Scholar 

  5. Shiell A, Hawe P, Gold L. Complex interventions or complex systems? Implications for health economic evaluation. BMJ. 2008;336(7656):1281–3.

    Article  Google Scholar 

  6. Rutter H, Savona N, Glonti K, Bibby J, Cummins S, Finegood DT, et al. The need for a complex systems model of evidence for public health. Lancet. 2017;390(10112):2602–4.

    Article  Google Scholar 

  7. Craig P, Ruggiero ED, Frohlich KL, Mykhalovskiy E, White M. Taking account of context in population health intervention research: guidance for producers, users and funders of research: Canadian Institutes of Health Research (CIHR) & National Institute for Health Research (NIHR); 2018. Available from: https://www.ncbi.nlm.nih.gov/books/NBK498645/. Accessed Jan 2019.

  8. Kane H, Hinnant L, Day K, Council M, Tzeng J, Soler R, et al. Pathways to program success: a qualitative comparative analysis (QCA) of communities putting prevention to work case study programs. J Public Health Manag Pract. 2017;23(2):104–11.

    Article  Google Scholar 

  9. Hartmann-Boyce J, Bianchi F, Piernas C, Riches SP, Frie K, Nourse R, et al. Grocery store interventions to change food purchasing behaviors: a systematic review of randomized controlled trials. Am J Clin Nutr. 2018;107(6):1004–16.

    Article  Google Scholar 

  10. Matheson A, Walton M, Gray R, Lindberg K, Shanthakumar M, Fyfe C, et al. Evaluating a community-based public health intervention using a complex systems approach. J Public Health. 2018;40(3):606–13.

    Article  Google Scholar 

  11. Roberts MC, Murphy T, Moss JL, Wheldon CW, Psek W. A qualitative comparative analysis of combined state health policies related to human papillomavirus vaccine uptake in the United States. Am J Public Health. 2018;108(4):493–9.

    Article  Google Scholar 

  12. van der Kleij RMJJ, Crone MR, Paulussen TGWM, van de Gaar VM, Reis R. A stitch in time saves nine? A repeated cross-sectional case study on the implementation of the intersectoral community approach Youth At a Healthy Weight. BMC Public Health. 2015;15:1032.

    Article  Google Scholar 

  13. Ragin CC. The comparative method: moving beyond qualitative and quantitative strategies. Berkeley: University of California Press; 2014. p. 218.

  14. Ragin CC. Using qualitative comparative analysis to study causal complexity. Health Serv Res. 1999;34(5 Pt 2):1225–39.

    CAS  PubMed  PubMed Central  Google Scholar 

  15. Wagemann C, Schneider CQ. Qualitative comparative analysis (QCA) and fuzzy-sets: agenda for a research approach and a data analysis technique. Comp Sociol. 2010;9(3):376–96.

    Article  Google Scholar 

  16. Schneider CQ, Wagemann C. Set-theoretic methods for the social sciences: a guide to qualitative comparative analysis. Cambridge: Cambridge University Press; 2012.

  17. Ragin CC, Shulman D, Weinberg A, Gran B. Complexity, generality, and qualitative comparative analysis. Field Methods. 2003;15(4):323–40.

    Article  Google Scholar 

  18. Cooper B, Glaesser J. Exploring the robustness of set theoretic findings from a large n fsQCA: an illustration from the sociology of education. Int J Soc Res Methodol. 2016;19(4):445–59.

    Article  Google Scholar 

  19. Ragin CC. Redesigning social inquiry: fuzzy sets and beyond. Chicago: The University of Chicago Press; 2008.

  20. Warren J, Wistow J, Bambra C. Applying qualitative comparative analysis (QCA) to evaluate a public health policy initiative in the north east of England. Polic Soc. 2013;32(4):289–301.

    Article  Google Scholar 

  21. Kane H, Lewis MA, Williams PA, Kahwati LC. Using qualitative comparative analysis to understand and quantify translation and implementation. Transl Behav Med. 2014;4(2):201–8.

    Article  Google Scholar 

  22. Hanckel B, Ruta D, Scott G, Green J. The Daily Mile as a public health intervention: a rapid ethnographic assessment of uptake and implementation in South London, UK. BMC Public Health. 2019;19(1):1167.

  23. Shamseer L, Moher D, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ. 2015;349:g7647.

    Article  Google Scholar 

  24. Rychetnik L, Frommer M, Hawe P, Shiell A. Criteria for evaluating evidence on public health interventions. J Epidemiol Community Health. 2002;56(2):119–27.

    Article  CAS  Google Scholar 

  25. O’Mara-Eves A, Thomas J, McNaught J, Miwa M, Ananiadou S. Using text mining for study identification in systematic reviews: a systematic review of current approaches. Syst Rev. 2015;4(1):5.

    Article  Google Scholar 

  26. Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol. 2008;8(1):45.

    Article  Google Scholar 

  27. Barnett-Page E, Thomas J. Methods for the synthesis of qualitative research: a critical review. BMC Med Res Methodol. 2009;9(1):59.

    Article  Google Scholar 

  28. Popay J, Roberts H, Sowden A, Petticrew M, Arai L, Rodgers M, et al. Guidance on the Conduct of Narrative Synthesis in Systematic Reviews. A Product from the ESRC Methods Programme. Lancaster: Institute of Health Research; 2006.

  29. Tanner S. QCA is of questionable value for policy research. Polic Soc. 2014;33(3):287–98.

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to thank and acknowledge the support of Sara Shaw, PI of MR/S014632/1.

Funding

Developed under MRC MR/R014094/1 ‘Developing a theory of change, outcome measures and evaluation design for an evaluation of the impact of the Daily Mile on obesity and health’, and conducted with support from MRC: MR/S014632/1 ‘Case study, context and complex interventions (Triple C): development of guidance and publication standards to support case study research’.

Author information

Authors and Affiliations

  1. King’s College London, SUPHI, London, UK

    Benjamin Hanckel & Judith Green

  2. London School of Hygiene and Tropical Medicine, London, UK

    Mark Petticrew

  3. University College London, London, UK

    James Thomas

Authors
  1. Benjamin Hanckel
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mark Petticrew
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. James Thomas
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Judith Green
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

BH, MP, JT, and JG conceived and designed the study. All authors drafted this manuscript, revised it for important content, and have provided the final approval of this version. BH, the corresponding author, is the guarantor of the review.

Corresponding author

Correspondence to Benjamin Hanckel.

Ethics declarations

Ethics approval and consent to participate

N/A

Consent for publication

N/A

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hanckel, B., Petticrew, M., Thomas, J. et al. Protocol for a systematic review of the use of qualitative comparative analysis for evaluative questions in public health research. Syst Rev 8, 252 (2019). https://doi.org/10.1186/s13643-019-1159-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s13643-019-1159-5

Keywords

Systematic Reviews

ISSN: 2046-4053