Protocol for a systematic review of the use of qualitative comparative analysis for evaluative questions in public health research
Systematic Reviews volume 8, Article number: 252 (2019)
There is an increasing recognition that health intervention research requires methods and approaches that can engage with the complexity of systems, interventions, and the relations between systems and interventions. One approach which shows promise to this end is qualitative comparative analysis (QCA), which examines casual complexity across a medium to large number of cases (between 10 and 60+), whilst also being able to generalise across those cases. Increasingly, QCA is being adopted in public health intervention research. However, there is a limited understanding of how it is being adopted. This systematic review will address this gap, examining how it is being used to understand complex causation; for what settings, populations and interventions; and with which datasets to describe cases.
We will include published and peer-reviewed studies of any public health intervention where the effects on population health, health equity, or intervention uptake are being evaluated. Electronic searches of PubMed, Scopus, Web of Science (incorporating Social Sciences Citation Index and Arts & Humanities Citation Index), Microsoft Academic, and Google Scholar will be performed. This will be supplemented with reference citation tracking and personal contact with experts to identify any additional published studies. Search results will be single screened, with machine learning used to check these results, acting as a ‘second screener’. Any disagreement will be resolved through discussion. Data will be extracted from full texts of eligible studies, which will be assessed against inclusion criteria, and synthesised narratively, using thematic synthesis methods.
This systematic review will provide an important map of the increasing use of QCA in public health intervention literature. This review will identify the current scope of research in this area, as well as assessing claims about the utility of the method for addressing complex causation in public health research. We will identify implications for better reporting of QCA methods in public health research and for reporting of case studies such that they can be used in future QCA studies.
Systematic review registration
Identifying the pathways that lead to the success of health interventions is crucial for public health research. However, assessing causal relationships between interventions and outcomes across contexts is challenging [1,2,3], as multiple interacting components come together in often complex ways [4,5,6] and ‘inevitable adaptations’ take place as health interventions get implemented within local contexts . Recent work  has pointed to the limitations of using randomised control trials (RCTs) to examine complex health interventions, as they are often unrepresentative of everyday practices across contexts: there is increased input to achieve fidelity, participants in trials may be more committed to the intervention, and context plays an important role in shaping the intervention. Understanding complexity thus requires approaches that draw on real world case studies [2, 8] and can identify pathways to successful implementation of, and effective outcomes from, health interventions across diverse contexts.
Qualitative comparative analysis (QCA) is one approach that has been increasingly adopted in public health research and evaluation [8,9,10,11,12] to examine complexity. Developed by Charles Ragin [13, 14], QCA is a set-theoretic method, which ‘bridges’ quantitative (large-n statistical analyses) and qualitative (small-n case study) techniques to analyse complex causation across contexts . A QCA analysis can involve a medium to large number of cases (between 10 and 60+), whilst also being able to generalise across those cases [16, 17]. QCA is often described as being suitable for ‘small-n’ scenarios and is less frequently used in large datasets, though some have used it with some success .
A QCA approach provides a tool for examining what conditions (alone or in combination with other conditions) are ‘necessary’ or ‘sufficient’ to produce some outcome . QCA involves using Boolean algebra and formal logic to examine the presence or absence of conditions, and the effect on an outcome. QCA uses ‘truth tables’, which ‘list all the possible combinations of casual conditions and the empirical outcome associated with each configuration … to identify explicit connections between combinations of casual conditions and outcomes’ . When first developed degree of membership was limited by nominal-scale variables, crisp sets, but has since been revised by Ragin to include ‘fuzzy sets’, which scale degree of membership from 0.0 to 1.0 [13, 19].
Warren et al. have argued that QCA offers an important contribution to mixed method public health evaluation methodologies . Others have identified opportunities to use QCA for informing the translation of research/interventions into practice and policy [21, 22]. There is a growing body of scholarship in public health that does use QCA methods for these, and other, aims. However, to date, there has been no systematic review of the use of QCA in public health literature and how it is being adopted. This systematic review will address this gap, examining how it is being used in public health research, including the methods and cases used to explore complexity, the populations it is being applied to, and the types of intervention research that QCA has been used to examine. This is a timely point to assess the field, in order to identify strengths and weaknesses, gaps, and implications for how to better report QCA methods and the case studies on which future QCA studies might be based.
The primary aim of the review is to understand what is known about the purpose, role, and use of qualitative comparative analysis (QCA) in research on the uptake and public health effects of interventions.
The proposed systematic review will answer the following research questions:
How is QCA used for public health evaluation? What populations, settings, methods used in source case studies, unit/s and level of analysis (‘cases’), and ‘conditions’ (often referred to as ‘variables’ in public health) have been included in QCA studies?
What strengths and weaknesses have been identified by researchers who have used QCA to understand complex causation in public health evaluation research?
How is QCA used to understand the contexts in which interventions increase or decrease health inequities?
What are the existing gaps in, and strengths and weakness of, the QCA literature, and what implications do these have for future QCA studies for public health?
The study will also identify implications for better reporting of QCA methods in public health research and for reporting case studies in general.
Our systematic review protocol was registered with the International Prospective Register of Systematic Reviews (PROSPERO) on 29 April 2019 (CRD42019131910). The protocol was prepared using the Preferred Reporting Items for Systematic Reviews and Meta-Analysis Protocols (PRISMA-P) 2015 statement .
Papers included in the systematic review will be published, be peer reviewed, and report on studies that have undertaken a qualitative comparative analysis (QCA) to examine casual complexity in the implementation, uptake, and/or effects of public health interventions. There will be no date restrictions.
Types of participants
Studies will include any population (including national, geographical communities, population sub-groups). As the review will be examining the types of populations included in QCA studies, all study participants will be included. This includes youth and adult populations, their families, and public health professionals.
Types of interventions
QCA studies that incorporate case studies of any kind of public health interventions will be included. Public health interventions are defined here as those focused on the promotion or protection of health, or the prevention of ill health for populations. This is in contrast to clinical interventions, which focus on the prevention or treatment of illness in individuals . This will include any interventions for where benefits to population health are the stated outcomes.
Types of outcome measures
To be included, QCA studies must identify an outcome (the phenomenon that the QCA is aiming to explain) that is relevant to public health, in which the effects on population health, health equity, or intervention uptake are being evaluated.
Types of studies
Papers reporting studies that use crisp and/or fuzzy sets will be included. QCA can be included as part of the analysis, or as the only tool used in analysing data.
Papers reporting both primary studies and systematic reviews which use QCA in their analysis will be included.
Publications not in English.
Studies where no intervention is evaluated, including studies that use QCA to examine public health infrastructure (i.e. staff training for low-skilled health staff), or papers that report on prevalence of health issues (i.e. prevalence of child mortality).
The search strategy will use the following phrases ‘Qualitative Comparative Analysis’ and ‘QCA’, which will be combined with keywords (i.e. ‘health’, ‘public health’, ‘intervention’, and ‘wellbeing’). Database searches will be conducted in PubMed, Scopus, Web of Science (incorporating Social Sciences Citation Index and Arts & Humanities Citation Index), Microsoft Academic, and Google Scholar. Study design must be QCA; no date limits will be imposed on the search. The search will be updated at the end of the review to ensure inclusion of all relevant studies. This will also be supplemented by reviewing reference lists of primary studies identified in the review, reviewing studies that cite these primary studies, as well as inviting experts to comment on whether the final list of studies omits any published studies known to them.
Study selection and data collection
Data and records will be maintained by the lead investigator (BH) and stored on a shared secure platform for access by all investigators.
Titles and abstracts identified by the bibliographic search and additional sources will be single screened against the criteria above, with machine learning employed to check these results, acting as a ‘second screener’ . ‘EPPI-Reviewer’ Version 4 software, or equivalent machine learning software, will be used to screen titles and abstracts. After removing duplicates, each study will be categorised as ‘include’, ‘exclude’, or ‘unsure’. The full texts of articles categorised as ‘include’ and ‘unsure’ will be retrieved and then assessed against inclusion criteria by reviewers. Those categorised as ‘unsure’ and/or where there are disagreements will be resolved through discussion, and if necessary, a third reviewer will mediate any unresolved differences.
Data extracted will be tabulated, using a standardised form approved by all investigators to ensure consistency (see Table 1). This will include extracting the following information for each included paper: title, author, rationale for QCA, context (setting, case level of analysis, and population characteristics), health intervention, health outcome/s, methods of data generation pertaining to the cases included (qualitative and/or quantitative), findings, and study conclusions. The reviewer/s will also extract any strengths and limitations outlined in each paper. Study authors will be contacted regarding any uncertainties.
Assessment of risk of bias
The objective of this review is to examine how QCA is currently used in the public health literature. There are no established reporting criteria for quality, so no studies will be excluded on the basis of quality. However, there are suggestions as to how to best report QCA [15, 16]. We will utilise four key criteria from these recommendations to map current reporting of the method. These will be evidence of familiarity with cases, justification for selection of cases, availability of raw data matrix and ‘truth table’ outcomes, and solution formula reported.
Data will be synthesised narratively, using thematic synthesis methods [26,27,28]. Final studies included will, following a descriptive analysis, be grouped by outcome measures, as well as context of intervention (e.g. school). We will assess claims about the utility of the method for addressing complex causation in public health research by identifying how far the published QCA study was able to identify necessary and sufficient conditions for the selected outcome. We will analyse separately (as a sub sample) those studies with an outcome related to health equity or inequalities, to assess how far QCA has been used to date in this area.
QCA has been increasingly used in public health research, although the utility of the method for producing policy-relevant findings has been questioned . The use of QCA in public health research therefore warrants further attention. We will map current use of the method, including identifying what types of intervention and setting have been addressed and what strengths and weaknesses researchers have identified in using QCA to address questions of complex causation. We will assess the extent to which current reporting in public health matches recommendations for best practice within the QCA research community, and the implications of these recommendations and current practice for the public health research community. As QCA relies on detailed knowledge of cases, we will also comment on the implications for public health researchers publishing stand-alone case studies, which might be used in future QCA studies. At present, there has been no systematic review of the use of QCA in public health literature and how it is being adopted; this systematic review will fill this gap, with the aim to inform, guide, and assess use of QCA in future complex health systems research.
There are several limitations to this study. Including only published studies from a rapidly emerging field may not be representative of the most recent applications of the method. In addition, we are only able to include those in English, which may underrepresent high-quality research from some regions.
Availability of data and materials
All relevant data are within the paper and its supporting information files.
Qualitative comparative analysis
Randomised control trial
Greenhalgh T, Papoutsi C. Studying complexity in health services research: desperately seeking an overdue paradigm shift. BMC Med. 2018;16(1):95.
Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ. 2015;350:h1258.
Cohn S, Clinch M, Bunn C, Stronge P. Entangled complexity: why complex interventions are just not complicated enough. J Health Serv Res Policy. 2013;18(1):40–3.
Kelly MP, Russo F. Causal narratives in public health: the difference between mechanisms of aetiology and mechanisms of prevention in non-communicable diseases. Sociol Health Illn. 2018;40(1):82–99.
Shiell A, Hawe P, Gold L. Complex interventions or complex systems? Implications for health economic evaluation. BMJ. 2008;336(7656):1281–3.
Rutter H, Savona N, Glonti K, Bibby J, Cummins S, Finegood DT, et al. The need for a complex systems model of evidence for public health. Lancet. 2017;390(10112):2602–4.
Craig P, Ruggiero ED, Frohlich KL, Mykhalovskiy E, White M. Taking account of context in population health intervention research: guidance for producers, users and funders of research: Canadian Institutes of Health Research (CIHR) & National Institute for Health Research (NIHR); 2018. Available from: https://www.ncbi.nlm.nih.gov/books/NBK498645/. Accessed Jan 2019.
Kane H, Hinnant L, Day K, Council M, Tzeng J, Soler R, et al. Pathways to program success: a qualitative comparative analysis (QCA) of communities putting prevention to work case study programs. J Public Health Manag Pract. 2017;23(2):104–11.
Hartmann-Boyce J, Bianchi F, Piernas C, Riches SP, Frie K, Nourse R, et al. Grocery store interventions to change food purchasing behaviors: a systematic review of randomized controlled trials. Am J Clin Nutr. 2018;107(6):1004–16.
Matheson A, Walton M, Gray R, Lindberg K, Shanthakumar M, Fyfe C, et al. Evaluating a community-based public health intervention using a complex systems approach. J Public Health. 2018;40(3):606–13.
Roberts MC, Murphy T, Moss JL, Wheldon CW, Psek W. A qualitative comparative analysis of combined state health policies related to human papillomavirus vaccine uptake in the United States. Am J Public Health. 2018;108(4):493–9.
van der Kleij RMJJ, Crone MR, Paulussen TGWM, van de Gaar VM, Reis R. A stitch in time saves nine? A repeated cross-sectional case study on the implementation of the intersectoral community approach Youth At a Healthy Weight. BMC Public Health. 2015;15:1032.
Ragin CC. The comparative method: moving beyond qualitative and quantitative strategies. Berkeley: University of California Press; 2014. p. 218.
Ragin CC. Using qualitative comparative analysis to study causal complexity. Health Serv Res. 1999;34(5 Pt 2):1225–39.
Wagemann C, Schneider CQ. Qualitative comparative analysis (QCA) and fuzzy-sets: agenda for a research approach and a data analysis technique. Comp Sociol. 2010;9(3):376–96.
Schneider CQ, Wagemann C. Set-theoretic methods for the social sciences: a guide to qualitative comparative analysis. Cambridge: Cambridge University Press; 2012.
Ragin CC, Shulman D, Weinberg A, Gran B. Complexity, generality, and qualitative comparative analysis. Field Methods. 2003;15(4):323–40.
Cooper B, Glaesser J. Exploring the robustness of set theoretic findings from a large n fsQCA: an illustration from the sociology of education. Int J Soc Res Methodol. 2016;19(4):445–59.
Ragin CC. Redesigning social inquiry: fuzzy sets and beyond. Chicago: The University of Chicago Press; 2008.
Warren J, Wistow J, Bambra C. Applying qualitative comparative analysis (QCA) to evaluate a public health policy initiative in the north east of England. Polic Soc. 2013;32(4):289–301.
Kane H, Lewis MA, Williams PA, Kahwati LC. Using qualitative comparative analysis to understand and quantify translation and implementation. Transl Behav Med. 2014;4(2):201–8.
Hanckel B, Ruta D, Scott G, Green J. The Daily Mile as a public health intervention: a rapid ethnographic assessment of uptake and implementation in South London, UK. BMC Public Health. 2019;19(1):1167.
Shamseer L, Moher D, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ. 2015;349:g7647.
Rychetnik L, Frommer M, Hawe P, Shiell A. Criteria for evaluating evidence on public health interventions. J Epidemiol Community Health. 2002;56(2):119–27.
O’Mara-Eves A, Thomas J, McNaught J, Miwa M, Ananiadou S. Using text mining for study identification in systematic reviews: a systematic review of current approaches. Syst Rev. 2015;4(1):5.
Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol. 2008;8(1):45.
Barnett-Page E, Thomas J. Methods for the synthesis of qualitative research: a critical review. BMC Med Res Methodol. 2009;9(1):59.
Popay J, Roberts H, Sowden A, Petticrew M, Arai L, Rodgers M, et al. Guidance on the Conduct of Narrative Synthesis in Systematic Reviews. A Product from the ESRC Methods Programme. Lancaster: Institute of Health Research; 2006.
Tanner S. QCA is of questionable value for policy research. Polic Soc. 2014;33(3):287–98.
The authors would like to thank and acknowledge the support of Sara Shaw, PI of MR/S014632/1.
Developed under MRC MR/R014094/1 ‘Developing a theory of change, outcome measures and evaluation design for an evaluation of the impact of the Daily Mile on obesity and health’, and conducted with support from MRC: MR/S014632/1 ‘Case study, context and complex interventions (Triple C): development of guidance and publication standards to support case study research’.
Ethics approval and consent to participate
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Hanckel, B., Petticrew, M., Thomas, J. et al. Protocol for a systematic review of the use of qualitative comparative analysis for evaluative questions in public health research. Syst Rev 8, 252 (2019). https://doi.org/10.1186/s13643-019-1159-5