Cochrane economics methods: 1993 to 2013 and after 2013
As we reflect on the first 20 years of development of economics methods for Cochrane reviews, it is apposite to acknowledge the insight of those early leaders within The Cochrane Collaboration who recognised a need for the organisation to consider both efficiency and effectiveness perspectives. We should also acknowledge the legacy of those health economists who responded to this challenge (named in the legend to Figure 1) by forming, in 1993, the informal discussion group that evolved into the Campbell and Cochrane Economics Methods Group. However, the development and application of economics methods in Cochrane reviews has progressed at a much slower rate in comparison to the rapid growth in production of Cochrane reviews more generally. Barriers to this progress include: the specificity of economics methods applicable for use in Cochrane intervention reviews due to their global audience; limited availability of resources and capacity to support Cochrane methods development; and limited capacity and expertise to support the application of economics methods in reviews. In this context, major challenges remain to building capacity (through the training of Cochrane authors, editors and methodologists), to establishing a wider economics methods network to support production of economics components of Cochrane reviews, and to securing funds to support further economics methods development.
A number of notable milestones have been passed during the first 20 years of the Cochrane Collaboration with respect to economics. The NHS EED and HEED databases have been established as key resources for the economics components of reviews (both currently free at the point of use to Cochrane contributors). The Methods Group was co-registered with The Campbell Collaboration in 2003, expanding its scope to cover the applied fields of crime and justice, education, social welfare and (latterly) international development alongside health. Methods guidelines were first published in the Cochrane Handbook in 2008 . A free online tool to automate the adjustment of estimates of costs for currency and price year was published in 2010 (http://eppi.ioe.ac.uk/costconversion/default.aspx). A book describing state-of-the-art approaches to evidence synthesis that combine economics and systematic review methods is now in its second edition [19, 20]. The CCEMG has developed a new methods framework for brief economic commentaries (http://www.c-cemg.org). Most recently, methods guidelines have been published on the use of the GRADE system to rate quality of evidence for resource use and costs, which will facilitate the incorporation of economic evidence into Summary of Findings tables. These tables are increasingly used in Cochrane to summarise principal findings and quality of evidence for important outcomes .
Looking ahead to the next 20 years, the volume of available economic evaluation will increase as economic evaluations increasingly become required as part of trials and other comparative studies by funding and regulatory bodies. This growth of the evidence base indicates the increasing need to consider economic evidence by decision-makers. Methods for incorporating economic perspectives and evidence into Cochrane intervention reviews need to continue to evolve so that they better meet decision-makers needs . More Cochrane reviews that utilise the methods frameworks we have outlined here are therefore urgently needed. The rate of progress will hinge on levels of organisational leadership, commitment to and investment in the production of economic components of Cochrane reviews, alongside levels of capacity and resources in the CCEMG, author teams, editorial bases and other Cochrane entities. Inevitably, further, sustained economics methods research and development are also needed. One major issue yet to be addressed is the development of methods guidelines for incorporating economic perspectives and evidence into Cochrane screening and diagnostic test accuracy reviews.
We next discuss some challenges to be addressed in the further development of economics methods for Cochrane intervention reviews (and methods for systematic reviews of economic evaluations more generally).
In recent work, Anderson and Shemilt discuss the possibility of producing pooled estimates of costs and cost-effectiveness when conducting systematic reviews of economic evaluations . Arguing for a more explanatory approach to such reviews, they propose that the real contribution of a systematic review of economic evidence may not be to produce a single authoritative result, but to help decision-makers understand the structure of the resource allocation problem addressed and the impact on the overall result of key determinants of costs and cost-effectiveness. The methods frameworks described above in this paper are consistent with this view. Anderson and Shemilt further argue that systematic reviews of economic evaluations are likely to be most useful in: (i) identifying the most relevant study (for the decision problem in hand) for a particular setting; (ii) understanding the key economic trade-offs and causal relationships in a decision problem or policy area; or (iii) justifying and informing decision model development.
A consideration of each of these three points in turn demonstrates that further investigation is needed to clarify how best to identify key studies with results that are applicable or transferable to particular jurisdictions and key economic trade-offs. With respect to key studies the value of meta-analytic techniques to explore the impact of factors likely to explain variation (that is, investigate heterogeneity) in estimates of resource use, costs and effects between studies remains under-explored for economic data. A multivariate meta-regression analysis, in principle, allows the effects of multiple explanatory factors to be investigated simultaneously. Brunetti and colleagues have recently published brief guidance notes to inform judgements about whether generating and presenting pooled estimates of resource use and costs is likely to be appropriate, with a view to investigating pre-specified factors that may drive between-study heterogeneity in such estimates . They suggest that meta-analysis of estimates of specific items of resource use may be judged appropriate provided that the metric used to quantify such estimates is common between studies (or a common metric can be derived), and that meta-analysis of estimates of costs may be judged appropriate in a more limited set of circumstances, and even then, only after estimates derived from different studies have first been adjusted to a common currency and price year. However, we are not aware of any examples which have used standard meta-analytic techniques to pool, and investigate 'between-study’ heterogeneity in published estimates of resources use and costs within a systematic review framework. The feasibility and usefulness of this approach therefore warrants further study.
It is clear that many health technology assessment organisations, for example, NICE in England , now rely on decision-analytic models to help assess the effectiveness and cost-effectiveness of interventions. Cochrane intervention reviews remain important inputs to this process but are not sufficient. This is because individual reviews do not include all relevant comparators, and Cochrane reviews do not include the further level of evidence synthesis that is provided by a decision-analytic model. To ensure that Cochrane reviews remain relevant to decision-making, the CCEMG needs to grapple with this issue, which has at its heart consideration of the transferability of findings. Some elements of a decision model are more likely to be transferable than others. A decision model describes two related processes - the disease pathway and a prevention/care pathway. The disease pathway is determined by underlying biology and may be more transferable, whilst the prevention/care pathway may be less transferable; but this does not preclude the formulation of illustrative pathways that could inform the development of context specific models in particular settings. Such an approach might be further enhanced should the concept of modular reviews be adopted; illustrative models might be presented within a module to assist in the development of models applicable to particular end users. Furthermore, whilst cost and utility data might have limited transferability (and relevance to some decision makers ) the results of an illustrative model might be specified in natural units (for example, number of visits, length of stay) and resulting health states (probabilities of death, survival impaired, or with no problems).
Multi-level modelling is an alternative analytic technique that may hold some promise for investigation of factors likely to explain variation in estimates of costs, effects and cost-effectiveness within a systematic review framework. Economic evaluations typically provide multiple estimates of resource use, costs and effects in the form of both a 'base case’ analysis and often extensive sensitivity and sub-group analyses. Boehler , used such techniques to explore the relative importance of different predictive factors for the costs, effects and cost-effectiveness of statins both within and between studies and also between countries. The analytical approaches are complex and results may be biased if sensitivity and sub-group analyses are selectively reported in the index studies. Therefore, as with use of meta-regression techniques, further exploration of multi-level modelling approaches is needed.