- Open Access
- Open Peer Review
Early economic evaluation of emerging health technologies: protocol of a systematic review
Systematic Reviewsvolume 3, Article number: 81 (2014)
The concept of early health technology assessment, discussed well over a decade, has now been collaboratively implemented by industry, government, and academia to select and expedite the development of emerging technologies that may address the needs of patients and health systems. Early economic evaluation is essential to assess the value of emerging technologies, but empirical data to inform the current practice of early evaluation is limited. We propose a systematic review of early economic evaluation studies in order to better understand the current practice.
This protocol describes a systematic review of economic evaluation studies of regulated health technologies in which the evaluation is conducted prior to regulatory approval and when the technology effectiveness is not well established. Included studies must report an economic evaluation, defined as the comparative analysis of alternatives with respect to their associated costs and health consequences, and must evaluate some regulated health technology such as pharmaceuticals, biologics, high-risk medical devices, or biomarkers. We will conduct the literature search on multiple databases, including MEDLINE, EMBASE, the Centre for Reviews and Dissemination Databases, and EconLit. Additional citations will be identified via scanning reference lists and author searching. We suspect that many early economic evaluation studies are unpublished, especially those conducted for internal use only. Additionally, we use a chain-referral sampling approach to identify authors of unpublished studies who work in technology discovery and development, starting out with our contact lists and authors who published relevant studies. Citation screening and full-text review will be conducted by pairs of reviewers. Abstracted data will include those related to the decision context and decision problem of the early evaluation, evaluation methods (e.g., data sources, methods, and assumptions used to identify, measure, and value the likely effectiveness and the costs and consequences of the new technology, handling of uncertainty), and whether the study results adequately address the main study question or objective. Data will be summarized overall and stratified by publication status.
This study is timely to inform early economic evaluation practice, given the international trend in early health technology assessment initiatives.
Economic evaluation is the comparative analysis of alternative technologies with respect to their costs and consequences . It is often used late in the evaluation of drugs, devices, and other technologies to inform coverage decisions . This is problematic because the evidence portfolio at the time of the reimbursement decision is often incomplete. Cost and preference data are often not gathered prospectively within clinical trials designed to evaluate effectiveness. This means that analysts cannot directly use trial data and must struggle to assemble evidence from a variety of sources and settings to evaluate the cost-effectiveness of new interventions . Published data suggest that considerable economic uncertainty existed in about half of the submissions for reimbursement decisions . Also, clinical trial design is rarely informed by ex ante economic modeling. This means that optimal strategies and the full range of subgroups may not have been fully considered prior to commencing the clinical research program.
Economic evaluation can be used early in the process of technology evaluation. As an example, Vallejo-Torres et al. illustrate a three-stage cost-effectiveness analysis of a new medical device: an early phase in which simple methods are used to estimate the maximum price attainable for the technology, a mid-stage which synthesizes data into a cost-effective model and identifies which information is most valuable to reduce decision uncertainty, and a late stage in which all relevant information is synthesized . The results suggest that early and iterative economic evaluation could be useful to inform decisions along the technology development process.
Early evaluation can contribute to decision-making by both industry and government [6, 7]. From an industry's perspective, early evaluation may be used for early market assessment, managing research and development portfolios, and informing pricing and reimbursement scenarios. From the policy perspective, decision makers may benefit from information supplied by early evaluation. Thus, there is interest from both innovators and payers in early health technology assessment and, more specifically, early economic evaluation, to inform planning and development decisions by industry and to inform the potential of new technologies that may meet health system needs .
Though interest in early economic evaluation is considerable, much applied research is still at the pilot stage. There is uncertainty about which technologies can benefit from early evaluation, what methods are appropriate, and the contribution of early economic evaluation to health technology assessment . The aim of this systematic review is to describe the characteristics of early economic evaluation of emerging technologies and to understand current methods to early evaluation.
This protocol describes a systematic review of economic evaluation studies that are conducted to inform the development and planning of early evaluation of emerging health technologies.
An economic evaluation is defined as the comparative analysis of alternatives with respect to their associated costs and health consequences. A study is included if it reports an early economic evaluation of a regulated health technology in order to inform the development of the technology (e.g., pre-clinical research or phase I or II studies). For this review, regulated health technologies include pharmaceuticals, biologics, medical devices, or biomarkers. The term ‘biologic’ refers to a class of therapeutics that are produced by means of biological processes, including, for example, the blood production-stimulating protein erythropoietin, a growth-stimulating hormone, a gene therapy, a monoclonal antibody, or a vaccine . The term ‘medical device’ refers to a class II device (e.g., blood pressure monitors, contact lenses, pregnancy test kits, single-use surgical instruments, catheters), a class III device (e.g., ventilators, cardiac monitors, hip implants, knee implants, lasers, chlamydia test kits, glucose meters), or a class IV device (e.g., defibrillators, pacemakers, coronary stents, HIV test kits, neurosurgical shunts) that requires product licensing for general marketing purposes (examples are from a presentation by Ms. Sarah Chandler of Health Canada to MaRS-Excite, September 2012, with permission). The term ‘biomarker’ (or a biological marker) refers to an objectively measured and evaluated indicator of a normal biological process, a pathogenic process (e.g., a DNA sequence that is associated with susceptibility to disease), or a pharmacologic response to a therapeutic intervention .
The literature search will be conducted on multiple databases, including MEDLINE (Ovid), MEDLINE In-Process (Ovid), EMBASE (Ovid), the Centre for Reviews and Dissemination: National Health Services Economic Evaluation Database (NHS EED), the Cochrane Library: Health Technology Assessment Database (HTA Database), and EconLit. The first four sources are standard repositories of economic evaluation studies. The last database is a reliable source of citations and abstracts of economic research . All databases will be searched from the date of inception until present.
The search used both Medical Subject Headings (MeSH) as well as keyword variants of all relevant terms, including keyword terms derived from MeSH scope notes to increase retrieval of citations from Medline In-Process. The search developed in MEDLINE will be adopted with the syntax and subject headings appropriate for the other databases mentioned. In addition to early economic evaluation keywords and MeSH terms generated through scoping search (below), the final search strategy will include the validated NHS EED search filter and a published MEDLINE G and EMBASE G filter to identify potentially relevant citations of economic evaluation studies in MEDLINE and EMBASE . Both MEDLINE G and EMBASE G filters were tested and showed that they improve the search precision ; however, both have been modified to conform to this study's needs. Namely, the subject heading for randomized control studies (RCTs) was removed from original filters, since we did not want to limit the retrieved studies only to RCTs. Limits that are part of NHS EED, MEDLINE G, and EMBASE G filters included humans and publication type. The final search was not limited by language.
Since the early economic evaluation concept is not well defined and it is not included in the MeSH controlled vocabulary, an initial scoping search was performed in order to identify potential candidate keyword terms as well as MeSH subject headings. A set of seed articles was used to develop the MeSH terms and keywords used in the literature search [2, 5–8, 13–22]. These articles were selected because they are commonly cited and highly relevant to early health technology assessment or early economic evaluation. The relevant MeSH terms harvested from the seed articles include ‘device approval,’ ‘diffusion of innovation,’ and ‘decision support techniques.’
From the seed articles, we extracted the cited keywords and other keywords that are deemed relevant to the aspects of early economic evaluation. Table 1 describes these aspects and the associated keywords. The first aspect is that the evaluation is conducted at an early phase in the development of the emerging technology. The objective of an early evaluation is to examine the likely cost-effectiveness of the technology. The evaluation is to provide decision support for product and policy development. There is limited supporting evidence regarding the effectiveness of the technology. Lastly, early evaluation may have an influence on the design and future performance of the technology.
The search strategy has been developed by an information analyst with the review team and will be peer-reviewed by another information analyst from a systematic review research group . Peer-reviewing of the literature search strategy will be conducted according to the Peer Review of Electronic Search Strategies (PRESS) statement . The protocol for this systematic review is registered at PROSPERO, the International prospective register of systematic reviews (registration number CRD42013004636) .
Additional citations will be identified via scanning the reference lists, author searching of potentially relevant studies, and forward citation searching in Scopus. We suspect that many early economic evaluation studies are unpublished, especially those conducted by industry for internal use only. There appears to be no straightforward way to identify authors of these unpublished studies. We will therefore consider a chain-referral sampling approach to sample ‘hidden populations’ in our attempt to contact these authors .
In the chain-referral sampling approach, a hidden population is one in which a sampling frame (i.e., a list of all the members of the population) cannot be constructed in advance. As an alternative, new members are selected from the social network of existing members of the sample. In this approach, first a number of seeds are selected . These seeds are members of the hidden population that have been identified: authors of published early economic evaluation studies who work for industry, and similar individuals identified from our contact files. The seeds are contacted and form the first stage (stage 0) of the sampling process. The seeds identify other members of the population. The members identified in the next stage (stage 1) are approached and then asked to identify other members. This process is continued until no new members could be identified.
We fully recognize that the above sampling scheme may not yield a comprehensive collection of unpublished study reports. The key barrier to access to internal reports of early evaluation may be the reluctance to share proprietary information. Possible authors will be contacted by email. We will follow up by phone with authors who contributed to unpublished studies in order to discuss their involvement and, if possible, request information about the unpublished material. For this approach, non-responders are defined as a lack of response after three attempts to contact a member of the sampling population.
Due to the broad inclusion criteria, screening of potentially relevant citations will be conducted by pairs of reviewers who are formally trained and experienced in economic evaluation. Full-text reports of citations that are deemed relevant by one screener will be obtained. Reasons for exclusion will be captured for studies that are deemed to be not relevant for the review. Full-text review will be conducted in pairs; disagreement will be discussed and, if necessary, adjudicated by another health economist.
Table 2 describes data elements that are going to be extracted from the included studies. First, understanding the decision context is necessary to discern the contribution of early evaluation to decision-making in product and policy development. Data pertaining to the decision context will be abstracted (e.g., who initiates the evaluation, who commissions it, the funding sources, why an evaluation is needed right now, who conducts the evaluation, and the primary target audience). Other abstracted data include the decision problem, disease classification, and target population, technology type (e.g., pharmaceuticals, medical devices, biomarkers), and technology stage (e.g., pre-clinical, phase I or II studies). Data pertaining to the methodological aspects of an early evaluation will be abstracted, especially how authors of included studies handled uncertainty in the early evaluation (e.g., probabilistic sensitivity analysis, value of information analysis).
Open text coding will be used to capture the following descriptions: the likely effectiveness of a new technology and how the associated health consequences and resources used were measured and valued. These descriptions will be used to discern the data sources, methods, and assumptions involved in the early evaluation. We will assess as to whether the results of an included study are adequate to answer the main study questions or objective, according to authors of included studies and according to the reviewers. A pair of reviewers using the Dedoose software for qualitative literature review will abstract each included study independently .
Characteristics of early economic evaluation studies will be summarized according to the various aspects that define early evaluation (Table 2). Given the qualitative nature of this systematic review, there will be no quantitative evidence synthesis. Unpublished studies obtained from the chain-referral sampling will be summarized separately because the representativeness of this sample of unpublished studies (relative to the population of early economic evaluation conducted to support internal decisions) is uncertain.
This protocol describes a systematic review of published and unpublished studies of early economic evaluation of regulated health technologies, when the supporting evidence for their effectiveness is not fully established. The aim is to understand current practice regarding the use of early economic evaluation to inform decisions in the development of emerging technologies, including decisions regarding the allocation of private and public resources for the development of these technologies and decisions regarding the planning and design of research programs to collect additional evidence to support regulatory and reimbursement requirements. This systematic review will summarize the methods used in early economic evaluation and provide data regarding the contribution of early economic evaluation to decision-making for product and policy development.
Our systematic review is timely. Conceptual discussion of early HTA over a decade ago has evolved into concrete implementations over recent years [6, 21, 32–34]. As we understand it, early HTA initiatives now exist in the UK, the Netherlands, Austria, and Ontario [35, 36, 21, 37]. These initiatives facilitate the collaboration between private enterprises, public sectors, and academia in the development of selective technologies that address the needs of patients and health systems and, to varied extent, investment expectations . Early economic evidence may facilitate decisions regarding the selection of promising technologies and the complex trade-offs between public and private investments .
Economic evaluation is essential for health technology assessment. Ijzerman and colleagues review previous work and methods of early health technology assessment and propose a conceptual framework for early assessment that acknowledges the different phases in technology development and the specific decision problems faced by different decision makers . They review a range of tools to identify and evaluate technologies in development, most notably early health economic modeling. They suggest that early HTA is an emerging field of research that will likely become more important because of the increasingly complex trade-offs between investments and access in medical product development.
The method outlined in this protocol is sometimes referred to as a meta-epidemiological investigation. In particular, there are well-established precedence regarding the use of systematic review to investigate factors affecting the conduct and reporting of trials, systematic reviews, and economic evaluations of health interventions [38–45]. By adapting a systematic approach to our investigation, our aim is to collect relevant data from a representative sample of all early economic evaluation studies and, therefore, reliably discern the contribution of early economic evaluation to product and policy development.
Drummond M, Sculpher M, Torrance G, O'Brien B, Stoddard G: Methods for the Economic Evaluation of Health Care Programmes. 2005, Oxford: Oxford University Press, 3
Hartz S, John J: Contribution of economic evaluation to decision making in early phases of product development: a methodological and empirical review. Int J Technol Assess Health Care. 2008, 24 (4): 465-472.
Lehoux P, Williams-Jones B, Miller F, Urbach D, Tailliez S: What leads to better health care innovation? Arguments for an integrated policy-oriented research agenda. J Health Serv Res Policy. 2008, 13 (4): 251-254.
Clement FM, Harris A, Jing JL, Yong K, Lee KM, Manns BJ: Using effectiveness and cost-effectiveness to make drug coverage decisions: a comparison of Britain, Australia, and Canada. JAMA. 2009, 302 (13): 1437-1443.
Vallejo-Torres L, Steuten L, Parkinson B, Girling AJ, Buxton MJ: Integrating health economics into the product development cycle: a case study of absorbable pins for treating hallux valgus. Med Decis Making. 2011, 31 (4): 596-610.
Retel VP, Joore MA, Linn SC, Rutgers EJ, van Harten WH: Scenario drafting to anticipate future developments in technology assessment. BMC Res Notes. 2012, 5: 442-
Shah SGS, Robinson I, Alshawi S: Developing medical device technologies from users' perspectives: a theoretical framework for involving users in the development process. Int J Technol Assess Health Care. 2009, 25 (4): 514-521.
Ijzerman MJ, Steuten LM: Early assessment of medical technologies to inform product development and market access: a review of methods and applications. Appl Health Econ Health Policy. 2011, 9 (5): 331-347.
Center for Biologics Evaluation and Research: What is a biological product 2007. [http://www.fda.gov/AboutFDA/Transparency/Basics/ucm194516.htm]
Foundation for the National Institutes of Health: The Biomarkers Consortium 2011. [http://www.biomarkersconsortium.org/]
EBSCO Industries I: Source of references to economic literature. 2012. [http://www.ebscohost.com/academic/econlit]
Glanville J, Fleetwood K, Yellowlees A, Kaunelis D, Mensinkai S: Development and Testing of Search Filters to Identify Economic Evaluations in MEDLINE and EMBASE. 2009, Canadian Agency for Drugs and Technologies in Health: Ottawa
Hartz S, John J: Public health policy decisions on medical innovations: what role can early economic evaluation play?. Health Policy. 2009, 89 (2): 184-192.
Pietzsch JB, Pate-Cornell ME: Early technology assessment of new medical devices. Int J Technol Assess Health Care. 2008, 24 (1): 36-44.
Dong H, Buxton M: Early assessment of the likely cost-effectiveness of a new technology: a Markov model with probabilistic sensitivity analysis of computer-assisted total knee replacement. Int J Technol Assess Health Care. 2006, 22 (2): 191-202.
Yao GL, Novielli N, Manaseki-Holland S, Chen YF, van der Klink M, Barach P, Chilton PJ, Lilford RJ, European HANDOVER Research Collaborative: Evaluation of a predevelopment service delivery intervention: an application to improve clinical handovers. BMJ Qual Saf. 2012, 21 (Suppl 1): i29-i38.
McAteer H, Cosh E, Freeman G, Pandit A, Wood P, Lilford R: Cost-effectiveness analysis at the development phase of a potential health technology: examples based on tissue engineering of bladder and urethra. J Tissue Eng Regen Med. 2007, 1 (5): 343-349.
Postmus D, de Graaf G, Hillege HL, Steyerberg EW, Buskens E: A method for the early health technology assessment of novel biomarker measurement in primary prevention programs. Stat Med. 2012, 31 (23): 2733-2744.
Garrison LP, Veenstra DL: The economic value of innovative treatments over the product life cycle: the case of targeted trastuzumab therapy for breast cancer. Value Health. 2009, 12: 1118-1123.
Retel VP, Bueno-de-Mesquita JM, Hummel MJ, van de Vijver MJ, Douma KF, Karsenberg K, van Dam FS, van Krimpen C, Bellot FE, Roumen RM, Linn SC, van Harten WH: Constructive Technology Assessment (CTA) as a tool in coverage with evidence development: the case of the 70-gene prognosis signature for breast cancer diagnostics. Int J Technol Assess Health Care. 2009, 25 (1): 73-83.
Girling AJ, Lilford RJ, Young TP: Pricing of medical devices under coverage uncertainty-a modelling approach. Health Econ. 2012, 21 (12): 1502-1507.
Vallejo-Torres L, Steuten LM, Buxton MJ, Girling AJ, Lilford RJ, Young T: Integrating health economics modeling in the product development cycle of medical devices: a Bayesian approach. Int J Technol Assess Health Care. 2008, 24 (4): 459-464.
St. Michael's Hospital: Knowledge translation in the Keenan Research Centre of the Li Ka Shing Knowledge Institute of St. Michael's Hospital. 2013. [http://www.stmichaelshospital.com/research/kt.php]
Sampson M, McGowan J, Cogo E, Grimshaw J, Moher D, Lefebvre C: An evidence-based practice guideline for the peer review of electronic search strategies. J Clin Epidemiol. 2009, 62 (9): 944-952.
Han D: Early economic evaluation of emerging health technologies: protocol of a systematic review. PROSPERO 2013:CRD42013004636 2013. [http://www.crd.york.ac.uk/PROSPERO/display_record.asp?ID=CRD42013004636]
Salganik MJ, Heckathorn DD: Sampling and estimation in hidden populations using respondent-driven sampling. Socio Meth. 2004, 34: 193-239.
Carlsson P, Jørgensen T: Scanning the horizon for emerging health technologies: conclusions from a European workshop. Int J Technol Assess Health Care. 1998, 14 (4): 695-704.
NICE: Guide to the Methods of Technology Appraisal. 2008, London: National Institute for Health and Clinical Excellence
CADTH: Guidelines for the Economic Evaluation of Health Technologies. 2006, Ottawa, [http://www.cadth.ca/media/pdf/186_EconomicGuidelines_e.pdf]
World Health Organization: International Classification of Diseases (ICD). 2013. [http://www.who.int/classifications/icd/en/]
SocioCultural Research Consultants L: Qualitative Research Data Analysis Software from Dedoose. 2011. [http://www.dedoose.com/]
Bartelmes M, Neumann U, Luhmann D, Schonermark MP, Hagen A: Methods for assessment of innovative medical technologies during early stages of development. GMS Health Technol Assess. 2009, 5: Doc15-[http://www.ncbi.nlm.nih.gov/pubmed/21289902]
Retel VP, Hummel MJ, van Harten WH: Review on early technology assessments of nanotechnologies in oncology. Mol Oncol. 2009, 3 (5–6): 394-401.
Retel VP, Hummel MJ, van Harten WH: Early phase technology assessment of nanotechnology in oncology. Tumori. 2008, 94 (2): 284-290.
MaRS-EXCITE: MaRS excellence in clinical innovation and technology evaluation. 2012. [http://www.excite.marsdd.com/]
MATCH: MATCH - Multidisciplinary Assessment of Technology Centre for Healthcare. 2013. [http://www.match.ac.uk/index.php]
Oncotyrol: Oncotyrol – Center for Personalized Cancer Medicine. 2011. [http://www.biocenter.i-med.ac.at/oncotyrol]
Schulz KF, Chalmers I, Hayes RJ, Altman DG: Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA. 1995, 273 (5): 408-412.
Wood L, Egger M, Gluud LL, Schulz KF, Juni P, Altman DG, Gluud C, Martin RM, Wood AJG, Sterne JAC: Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ. 2008, 336 (7644): 601-605.
Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, Tugwell P, Klassen TP: Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses?. Lancet. 1998, 352 (9128): 609-613.
Balk EM, Bonis PA, Moskowitz H, Schmid CH, Ioannidis JP, Wang C, Lau J: Correlation of quality measures with estimates of treatment effect in meta-analyses of randomized controlled trials. JAMA. 2002, 287 (22): 2973-2982.
Savovic J, Jones H, Altman D, Harris R, Jűni P, Pildal J, Als-Nielsen B, Balk E, Gluud C, Gluud L, Ioannidis J, Schulz K, Beynon R, Welton N, Wood L, Moher D, Deeks J, Sterne J: Influence of reported study design characteristics on intervention effect estimates from randomised controlled trials: combined analysis of meta-epidemiological studies. Health Technol Assess. 2012, 16 (35): 1-82.
Bell CM, Urbach DR, Ray JG, Bayoumi A, Rosen AB, Greenberg D, Neumann PJ: Bias in published cost effectiveness studies: systematic review. Br Med J. 2006, 332 (7543): 699-701.
Sculpher MJ, Pang FS, Manca A, Drummond MF, Golder S, Urdahl H, Davies LM, Eastwood A: Generalisability in economic evaluation studies in healthcare: a review and case studies. Health Technol Assess. 2004, 8 (49): iii-iv.
Drummond MF, Iglesias CP, Cooper NJ: Systematic reviews and economic evaluations conducted for the National Institute for Health and Clinical Excellence in the United Kingdom: a game of two halves?. Int J Technol Assess Health Care. 2008, 24: 146-150.
Funding for this study is provided in part by Health Quality Ontario and the Ontario Ministry of Health and Long-Term Care to the Toronto Health Economics and Technology Assessment (THETA) Collaborative. The THETA Collaborative provides technology assessment support for the Ontario Health Technology Advisory Committee at Health Quality Ontario. THETA is a designated methods center for early HTA projects with MaRS-EXCITE  and has received infrastructure funding for early HTA from the Canadian Foundation for Innovation (2012 Leading Edge and New Initiatives Funds, University of Toronto, Project No. 31001). ACT is funded by a new investigator award in knowledge synthesis from the Canadian Institutes for Health Research Drug Safety and Effectiveness Network.
The authors declare they have no competing interests in relation to the planned systematic review.
BP participated in conception and design of the project, piloting the protocol with a literature search from January 2010 to March 2013, preparing the results of the pilot review for a presentation at the panel discussion on early Health Technology Assessment at the Canadian Drugs and Technologies in Health Symposium 2013 in St John’s Newfoundland and Labrador Canada, drafting and revising the manuscript. HATT participated in designing the project, piloting the protocol, preparing the presentation, drafting and revising the manuscript. DH participated in designing the project, piloting the protocol, preparing the presentation, drafting and revising the manuscript. PP participated in conception and design of the project, piloting the protocol, preparing the presentation, drafting and revising the manuscript. JB participated in designing the project, piloting the protocol, obtaining peer-review of the literature search, conducting the literature search, registering the protocol, and drafting and revising the manuscript. FM participated in conception and design of the project and revising the manuscript. VR participated in conception and design of the project and revising the manuscript. WC participated in conception and design of the project and revising the manuscript. ACT participated in conception and design of the project and revising the manuscript. MP participated in conception and design of the project and revising the manuscript. MK participated in conception and design of the project, revising the manuscript and being responsible for the direction and quality of the project. All authors approved of the final manuscript.
About this article
- Economic Evaluation
- Health Technology Assessment
- Early Evaluation
- Economic Evaluation Study
- National Health Service Economic Evaluation