Table 1 Challenges of applying systematic review methods to a review of theory

Inclusion criteria

Emphasis on setting a priori criteria specifying population, intervention, comparison, outcome, and study (PICOS) design; or SPIDER [20] for reviews of qualitative data.

Defining a theory. Subject-specific inclusion criteria may exclude relevant theories that originated in different subject areas.

To be included, a theory had to define a mechanism linking financial resources and health. Subject experts identified influential theories that originated from other subject areas.

Literature search

Generally strong focus on searches of electronic bibliographic databases, requiring clear search terms.

Vast literature and intentionally broad review question including literature from different subject areas, using different terminologies.

Review included multiple forms of both formal electronic searches and hand searches and citation tracking to follow how theories develop and influence later literature. This is less of an issue if the search relates to a narrower review question, e.g. focusing on a specific intervention and its mechanism.

Data extraction

Standardised forms and software (e.g. RevMan) available

Determining what information is required to extract and how best to extract this uniformly. Various methods possible.

Developed a spreadsheet for extracting data about theories from a large number of included studies. This process could have been improved by using qualitative methods and software.

Quality appraisal

Appraisal of clear reporting of empirical study methods to enable assessment of potential biases, and the generalisability of the study results.

Tools developed for appraising conventional systematic reviews focus on internal validity and study design.

The theory review did not include a standardised critical appraisal of the theories. In retrospect, it may have been useful to have attempted to grade theories by relevance to the review question and, possibly, by level of detail or originality [23].

Synthesis

Various methods, including meta-analysis of suitably homogeneous data and narrative synthesis to explore heterogeneity between studies.

Summarising theories rather than empirical findings.

Interpretative collation of key concepts and relations to create a causal map. Iterative process of checking between this mapping process and the themes emerging from the data. Consideration of how competing theories may be genuinely oppositional, mutually inclusive, and/or represent different links in a longer and more complex causal chain (similar to meta-ethnographic concepts of reciprocity, refutation, and line of argument).