Open Access
Open Peer Review

This article has Open Peer Review reports available.

How does Open Peer Review work?

Effects and repercussions of local/hospital-based health technology assessment (HTA): a systematic review

  • Marie-Pierre Gagnon1, 2Email author,
  • Marie Desmartis1,
  • Thomas Poder3, 4 and
  • William Witteman1
Systematic Reviews20143:129

DOI: 10.1186/2046-4053-3-129

Received: 14 July 2014

Accepted: 13 October 2014

Published: 28 October 2014

Abstract

Background

Health technology assessment (HTA) is increasingly performed at the local or hospital level where the costs, impacts, and benefits of health technologies can be directly assessed. Although local/hospital-based HTA has been implemented for more than two decades in some jurisdictions, little is known about its effects and impact on hospital budget, clinical practices, and patient outcomes. We conducted a mixed-methods systematic review that aimed to synthesize current evidence regarding the effects and impact of local/hospital-based HTA.

Methods

We identified articles through PubMed and Embase and by citation tracking of included studies. We selected qualitative, quantitative, or mixed-methods studies with empirical data about the effects or impact of local/hospital-based HTA on decision-making, budget, or perceptions of stakeholders. We extracted the following information from included studies: country, methodological approach, and use of conceptual framework; local/hospital HTA approach and activities described; reported effects and impacts of local/hospital-based HTA; factors facilitating/hampering the use of hospital-based HTA recommendations; and perceptions of stakeholders concerning local/hospital HTA. Due to the great heterogeneity among studies, we conducted a narrative synthesis of their results.

Results

A total of 18 studies met the inclusion criteria. We reported the results according to the four approaches for performing HTA proposed by the Hospital Based HTA Interest Sub-Group: ambassador model, mini-HTA, internal committee, and HTA unit. Results showed that each of these approaches for performing HTA corresponds to specific needs and structures and has its strengths and limitations. Overall, studies showed positive impacts related to local/hospital-based HTA on hospital decisions and budgets, as well as positive perceptions from managers and clinicians.

Conclusions

Local/hospital-based HTA could influence decision-making on several aspects. It is difficult to evaluate the real impacts of local HTA at the different levels of health care given the relatively small number of evaluations with quantitative data and the lack of clear comparators. Further research is necessary to explore the conditions under which local/hospital-based HTA results and recommendations can impact hospital policies, clinical decisions, and quality of care and optimize the use of scarce resources.

Keywords

Health technology assessment Local/hospital-based HTA Impact of HTA activities Hospital budget Perceptions of stakeholders HTA units Internal committee Mini-HTA Ambassador model

Background

While health technology assessment (HTA) is often done at a national or international level, many local health services and hospitals consider that it makes sense to move the assessment closer to the point of care, where the costs, impacts, and benefits of technologies can be directly assessed. This is justified by the fact that many decisions regarding health technologies (prioritization, investment, adoption, and disinvestment) are made at the local/hospital level[1]. This is also the result of an increasing awareness that specific organizational contexts should be taken into account when assessing health technologies[2].

With the emergence of HTA activities in hospitals, the Hospital Based HTA Interest Sub-Group was created within the HTAi—the international scientific and professional society for HTA—in 2006. In 2008, this sub-group elaborated a conceptual model to classify the different approaches for performing HTA within hospitals around the world[1]. Four different approaches were described: 1) ambassador model, 2) mini-HTA, 3) internal committee, and 4) HTA unit. The ambassador model seeks to promote changes in practice through a specific HTA dissemination approach. In this approach, interested clinicians who are recognized as opinion leaders play the role of ambassadors of the HTA message within health-care organizations at regional and local levels. Mini-HTA is a management and decision support tool that consists of questions about the technology, the patient, the organization, and the financial aspects[3]. The mini-HTA is usually done by a single professional who often participates in the assessment process, collecting data at the hospital level in order to inform decision makers. The internal committee consists in many cases of an ad hoc multidisciplinary group representing different perspectives, in charge of reviewing evidence and making recommendations to the health-care organization. The HTA unit represents the most structured model for hospital-based HTA. It is a formal organizational structure with specialized HTA personnel working on a full-time basis on the production of HTA material of high scientific quality.

According to a recent survey, the number of hospitals performing HTA is growing around the world[4]. While local/hospital-based HTA has been in place for a few decades in some jurisdictions, there is very limited knowledge of its effects on decisions regarding health technologies. Thus, it is important to review current evidence about local/hospital-based HTA in order to inform future initiatives.

This paper presents a systematic review of the effects and impact of local/hospital-based HTA reported in the literature on decision-making or management. The main questions that guided this review are the following: 1) Have HTA recommendations been accepted and implemented? 2) What expenses and savings are related to HTA activities and their recommendations? 3) What are the perceptions of various stakeholders towards local/hospital HTA? We also compiled information about strengths and weaknesses of different approaches of local/hospital-based HTA, as well as barriers and facilitators with respect to the implementation of their recommendations, as secondary outcomes.

Methods

This review adheres to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist[5] (see Additional file1).

Screening and selection

A review protocol was established, based on a previously published knowledge synthesis[6]. We searched for qualitative, quantitative, or mixed-methods studies with empirical data about the effects or impact of local/hospital-based HTA. We did not use date or language limits. Studies were excluded if they were not about local/hospital HTA activities; did not provide data concerning the impact of HTA activities on decision-making or management, budget, or perceptions of stakeholders as regards HTA activities; and were not based on an empirical analysis of the effects or impact of HTA activities or recommendations.

An information specialist (WW) developed the search strategy and performed literature search in two databases (PubMed and Embase; see Additional file2). Other literature were identified through citation tracking of the included studies. Two members of the team (WW, MD) independently reviewed titles and abstracts for relevance, with the intervention of a third reviewer (MPG) in case of discrepancy. Teams of two reviewers (MD, TP, WW, and MPG) then independently conducted full-text reviews for eligibility; any disagreement was resolved by discussion among reviewers.

Extraction of data

The same teams of two reviewers independently extracted the following information from the selected studies: general characteristics of the studies (country, methodological approach); local/hospital HTA approach and activities; use of conceptual framework or model; reported effects and impacts of local/hospital-based HTA; factors facilitating/hampering the use of hospital-based HTA recommendations; perceptions of stakeholders regarding local/hospital HTA activities (benefits and concerns); and resources needed for implementing hospital-based HTA. We also assessed study quality using the Mixed Methods Appraisal Tool (MMAT)[7]. Due to the great heterogeneity among studies, we conducted a narrative synthesis of their results[8].

Results

A total of 707 potentially relevant articles were identified from the main search strategy, of which 15 met the eligibility criteria and were included. Other search strategies identified four additional eligible publications, but one of them was an update of an included study. Consequently, a total of 18 studies were included, described in 19 papers. The study selection process is presented in Additional file3.

Overview of studies

More than half of the studies were from the USA (n = 7) and Canada (n = 5). Other studies were from Denmark (n = 2), Austria (n = 2), and Australia and France (n = 1 each). Over half of the articles (10/18) were published since 2005 and a third (6/18) since 2010. The six articles published prior to 2000 were all from the USA, but one study was conducted in Canada[9].

Purpose and methods of the studies

Four studies surveyed hospitals in a country or in part of a country to explore how HTA was used in decision-making for the introduction of technological innovations. They used questionnaires[911] or semi-structured interviews[12]. Five studies used a case study to report the experience of use of HTA in a particular hospital[1315] or in a few hospitals[16, 17]. In all of these studies, with the exception of two[15, 16], HTA was performed by committees of different types, often of an ad hoc nature.

Six studies reported an analysis of the outcomes of an HTA program or unit and the impact of their reports and recommendations. In Canada, Poulin et al.[18] conducted a retrospective analysis of the outcomes of an HTA program (over a 5-year period) and of the kinds of decisions that were made based on local committee recommendations. McGregor and Brophy[19, 20] evaluated the impact of 55 HTA reports that their Technology Assessment Unit (TAU) produced during 8 years of service. They used interviews with local administrative and clinical decision makers as well as document analysis to evaluate the impact of HTA reports and recommendations on hospital policy decision-making and hospital spending. Other impact studies focused on an HTA unit or committee that covered many hospitals. For instance, Bodeau-Livinec et al.[21] studied the Committee for the Assessment and Dissemination of Technological Innovations (CEDIT in French), an HTA unit that covers a network of 39 university hospitals located in the Paris region. Similarly, the study by Lee et al.[22] illustrated the impact of the activities of an HTA unit in a health region in Alberta (Canada). Finally, Schumacher and Zechmeister[23, 24] focused on the impact of an HTA research program that produces various types of reports for decision makers at different levels of the health-care system in Austria, including hospitals.

Using the survey approach, two studies assessed the use of mini-HTA in the introduction of new technologies in the Danish hospital sector[3] or in a major public hospital in Denmark[25]. The study by Rashiq et al.[26] evaluated the Alberta Ambassador Program put in place to inform clinicians about current research evidence on the management of chronic non-cancer pain, using a pre- and post-session questionnaire.

Quality of the studies

The assessment of the quality of studies can be found in Additional file4. Even if the assessment of the quality of studies was conducted using the MMAT, we decided not to take this into account in the interpretation of our results due to the exploratory aim of the review. Overall, studies conducted after 2000 were of better quality than earlier studies.

Reported effects and impacts of local/hospital-based HTA activities

Given the great heterogeneity among studies, notably in terms of approaches for performing HTA, we report their effects and impacts according to the four categories proposed by the Hospital Based HTA Interest Sub-Group[1] (ambassador model, mini-HTA, internal committee, and HTA unit). However, we found a wide range of “committees” in the studies reviewed. Their composition and structure varied across studies (from ad hoc committees to very structured ones). In this review, we distinguish committees from HTA units that are formal organizational structures with dedicated HTA personnel.

HTA committees

Reported effects and impacts of activities of different types of HTA committees on decision-making (or management) are presented in Table 1. As the only financial aspect reported in some of these studies[12, 13] is the minimum cost related to a technology for deciding whether to assess them, we will not create a specific section here for financial impact.
Table 1

Reported effects and impact—HTA committees

Reference and country

Methods/participants

Type of committee (structured or unstructured (ad hoc))

Type of impact

Cram et al. 1997 USA[10]

Survey/33 clinical engineering departments throughout the USA

Not specified

On decision-making (or management)

23/27 committees doing HTA are multidisciplinary

  • Several respondents used HTA to cut costs and provide more standardization.

  • The HTA process was seen by some respondents as allowing broader input into decision-making processes.

On clinicians’ or other stakeholders’ perceptions

  • 20/25 (80%) who used an HTA system felt it was a useful tool.

  • Main problems perceived in HTA processes: internal politics; lack of understanding that could lead committees to make poor decisions

Luce and Brown 1995 USA[12]

Interviews/48 participants from 30 organizations (hospitals, health maintenance organizations, and third-party payers)

Not specified

On decision-making (or management)

For hospitals: multidisciplinary committees; formulary committees; department chiefs

  • Hospital decision makers used HTA almost exclusively for making purchasing decisions and as a means of controlling expenditures.

  • Decisions were based on financial assessment with little or no formal evaluation of changes in patient outcomes or medical practice patterns.

  • Purchase or non-purchase recommendations were rarely contravened by management and were distributed to relevant departments throughout the organization.

Financial

  • New technologies priced over a predetermined threshold (US $100,000 or $250,000) were all assessed prior to purchase.

Menon and Marshall 1990 Canada[9]

Survey/50 (59.5%) teaching hospitals across Canada

Structured: 23/50

On decision-making (or management)

  • 34/43 hospitals practicing HTA stated that information produced was used in making decisions about new technology acquisition.

On clinicians’ or other stakeholders’ perceptions

  • 76% of respondents thought that a formal management structure for HTA should exist in teaching hospitals.

Patail and Aranha 1995 USA[13]

Case study/1 major teaching hospital

Structured

On decision-making (or management)

  • Of 16 technologies formally approved in 1988–1993, 13 were implemented.

  • HTA allowed engineers and decision makers not to take the information provided by manufacturers and vendors for granted.

Financial

  • Technologies over $500,000 were assessed.

Poulin et al. 2012 Canada[18]

Case study of HTA program outcomes

Structured

On decision-making (or management)

  • Of the 68 technologies for which a HTA was requested, 15 were incomplete and dropped, 12 were approved, 3 were approved on an urgent/emergent basis, 21 were approved for “clinical audit” on a restricted basis, 14 were approved for research use only, and 3 were referred to additional review bodies.

  • Decisions based on local HTA program recommendations were rarely “yes” or “no”. Many technologies were given restricted approval, with full approval contingent on satisfying certain conditions such as clinical outcomes review, training protocol development, or funding.

Financial

  • Cost was the first reason to reject a technology, followed by health gain.

Rosenstein et al. 2003 USA[11]

Survey/19 hospitals in western USA

Structured: 42%

On decision-making (or management)

Ad hoc: 48%

  • 28% of HTA committees had direct responsibility for approval.

  • While committees did not have final decision-making power, their recommendations were appropriate and well integrated with the hospital’s overall mission and strategic plan.

Saaid 2011 Australia[17]

Multicase study/4 hospitals (3 private for-profit, 1 public)

1/4 has a formal committee

On decision-making (or management)

3/4 have a product review committee

  • The impact of HTA as a support tool for decision makers was minimal.

  • Decisions in private for-profit hospitals were informal and driven by business strategy and cost-effectiveness of the technology.

  • For the public hospital, HTA was a requirement in decision-making, but the process was new.

On clinicians’ or other stakeholders’ perceptions

  • Ignorance/unfamiliarity with HTA.

Weingart 1995 USA[14]

Case study/1 major teaching hospital

Ad hoc

On decision-making (or management)

  • The technology assessed was qualified as an engineering disaster for various reasons:

  • Decision makers did not go far enough in their discussions to evaluate the institutional strategy or strategic implications of the technology. They lacked expertise in assessing feasibility and profitability.

  • Members of the committee (only physicians) were too optimistic despite limited data.

  • The mandate of the committee was too narrow and did not include comparison with alternative technology.

   

  • The process was not structured enough (ad hoc structure), and there was no official strategic plan in place at the hospital.

Impact on hospital policies and management

A survey conducted in 30 organizations (including hospitals, health maintenance organizations (HMOs), and third-party payers) in the USA in 1995[12] showed that decision makers used HTA to inform purchasing or coverage decisions regarding new and expensive technologies and drugs, as a means to better use resources. Decision makers usually followed the purchase or no purchase recommendation made by the assessment committee. However, with the exception of pharmacy committees, decisions were based on financial evaluation with little or no formal evaluation of changes in patient outcomes or medical practice patterns. A survey in teaching hospitals across Canada conducted in 1990[9] found that HTA was common in teaching hospitals (43/50), although it took various forms. Thirty-four of the 43 hospitals practicing HTA stated that information from HTA was used in decision-making about new technology acquisition. However, only 23 hospitals had a formal management structure for HTA. In the Cram et al. study[10] of clinical engineering departments throughout the USA conducted in 1997, 80% of the respondents (20/25) who used an HTA process felt that it was a useful tool and several of them had used it to cut costs and provide more standardization. The benefits of different HTA processes varied from hospital to hospital, but some respondents stressed that the HTA process informed purchase requests to be evaluated by a multifunctional team, which allows for broadening the input involved in decision-making processes[10].

In a case study from 1995 concerning the impact of the assessment of biliary lithotripsy by a task force of physicians in a major teaching hospital in the USA, Weingart[14] reported that the works of this committee have led to purchasing of the technology, which was later described as an engineering disaster. The author outlined many reasons for this failure: the exaggerated optimism of physicians in the task force, despite limited data in published reports; a mandate that was too narrow and did not include comparisons with alternative technologies; the lack of expertise of decision makers in assessing feasibility and profitability; and an assessment process that was not structured enough.

Patail and Aranha[13] reported the impact of the work of a multidisciplinary assessment team, including managers and biomedical engineers, in a US hospital in 1995. Their results showed that of 16 technologies formally approved by the assessment team between 1988 and 1993, a total of 13 had been implemented. According to these authors, HTA made it possible for decision makers to avoid taking the information provided by manufacturers and vendors for granted.

In 2003, a survey among 19 hospitals in the western part of the USA[11] revealed that although 90% of hospitals reported the existence of a specific staff responsible for providing formal reviews, only 42% had a dedicated technology assessment committee. Among these committees, 28% had direct responsibility for technology approval. Even if committees did not make the final decision, respondents reported that their recommendations were well integrated into the hospital’s mission and strategic plan.

Recently, the multiple case study by Saaid[17] examined the use of HTA in decision-making processes for acquiring new health technologies in four selected hospitals (three not-for-profit private hospitals and one public hospital) of southeast Queensland (Australia). The results showed that the impact of HTA as a support tool for decision makers was relatively minimal. Decisions in private hospitals were informal and driven by business strategy and the cost-effectiveness of the technologies and also significantly influenced by physicians. In the public hospital, HTA was a requirement in decision-making, and a formal committee was in place, but it was at an early stage of development.

For their part, Poulin et al.[18] analyzed the outcomes of a local HTA program in Alberta (Canada) over a 5-year period. They reported that decisions based on recommendations of this program were rarely “yes” or “no” but offered many approval options between full acceptance and rejection. Many technologies have received restricted approval, with full approval contingent on satisfying conditions such as clinical outcomes review, training protocol development, or funding.

HTA unit

Impact on hospital policies and management
Reported effects and impact of activities of HTA units are presented in Table 2. Four studies of this category analyzed the impact of HTA reports and recommendations on hospital policies and implementation. The study by McGregor and Brophy[19, 20] evaluated the impact of 55 reports issued by the TAU of the McGill University Health Center (MUHC) between 2004 and 2011. Of the 63 recommendations produced in these reports, 45 (71%) have been accepted and incorporated into hospital policy. The most frequent reason for recommendations not being accepted was failure to identify administrative responsibility to carry this out.
Table 2

Reported effects and impact—HTA units

References and country

Methods

Type of impact

Bodeau-Livinec et al. 2006 France[21]

Semi-directive interviews and survey

On decision-making (or management)

  • 10 of 13 recommendations had an impact on theintroduction of the technology in health organizations.

  • One recommendation appears not to have had an impact. The impact of two technologies was impossible to assess.

Financial

  • The main criterion upon which to base a new technology introduction decision on HTA is the cost. Some medical specialties were more concerned by CEDIT’s work than others—cardiology and medical imaging, for instance.

On clinicians’ or other stakeholders’ perceptions

  • Interviewees viewed the CEDIT as very scientifically reputable. HTA recommendations were used as decision-making tools by administrative staff and as negotiating instruments by physicians in dealing with management.

Lee et al. 2003 Canada[22]

Case study (review of document and structured consultation)

Financial

  • Example of one evaluation to address the issue of arthroplasty operations. Savings were estimated at CAN $1 million annually through orthopedic supply standardization and a new contract with vendors.

On clinicians’ or other stakeholders’ perceptions

  • High level of interest for a locally focused HTA and implementation unit.

McGregor 2012 Canada[19]

Impact study using mixed methods (interviews and financial analysis)

On decision-making (or management)

  • Impact of 55 HTA reports produced (2004–2011): Of 63 recommendations, 45 (71%) have been accepted and incorporated into hospital policy.

Update of McGregor and Brophy 2005 Canada[20]

  • Most frequent reasons for recommendations not being accepted: failure to identify administrative responsibility to carry this out, lack of funds, complex administrative changes, technology already implanted, technology which would potentially render the hospital vulnerable to legal action.

Financial

  • 19 accepted reports have resulted in conservation of hospital resources.

  • The extent of these savings could be estimated in the case of 15 reports: estimated overall savings of CAN$ 9,840,270.

  • Over the 8 years of full functioning of the HTA unit: average annual quantifiable savings have been CAN$ 1,140,958.

Mitchell 2010 USA[15]

Case studies

On decision-making (or management)

  • Two examples of local data integrated into hospital-based HTA. In both case studies, important differences were found among the hospitals. These differences affected the prioritization of different attributes of a technology and could result in different conclusions being drawn about how the technology should be used at each hospital, even within the same health-care network.

Veluchamy and Alder 1989 USA[16]

Case study

On decision-making (or management)

  • The HTA units helped decision makers integrate patient needs and medical staff interests and capabilities with the hospital’s resources (i.e., staff, facilities, financing).

  • It speeded up the delivery of newly developed treatment technologies (9–12 months as compared to 24–36 months before HTA implementation). It identified the most promising technologies and coordinated their acquisition and implementation.

  • It provided better access to these technologies for patients and reduced length of stay (42% reduction for laser angioplasty).

On clinicians’ or other stakeholders’ perceptions

  • Physicians derived personal and professional satisfaction from participation in the HTA units. These units have improved relations between medical staff and hospital management (better communication and physicians’ needs better fulfilled).

Schumacher and Zechmeister 2013 Austria[23]

Impact study using mixed methods (interviews, questionnaire, download analysis, etc.)

On decision-making (or management)

  • Hospital associations used HTA for investment/reimbursement decisions, treatment guidelines, and budget allocation, as well as for the preparation of negotiations.

  • Various pressure groups, such as the pharmaceutical industry and the professionals’ association, could explain the inability to implement some HTA recommendations.

  • With the exception of the rapid technology assessment program for single hospital procedures, selective use of HTA reports was identified, rather than standardized inclusion of HTA into the processes.

Financial

  • Several technologies, identified as showing patterns of over-usage, were used more restrictively after the HTA report was published, leading to a decrease in expenditure. Expenditure decrease accounted for at least several million euros for single hospital associations.

On clinicians’ or other stakeholders’ perceptions

  • Clearest evidence was available for the “awareness” impact category, while references regarding “acceptance” were rarely mentioned. The LBI-HTA was usually seen as a vehicle for simple cost containment and rationing, rather than a tool supporting redistribution of resources into evidence-based technologies.

Zechmeister and Schumacher 2012 Austria[24]

Impact study using mixed methods (administrative data analysis and interviews)

On decision-making (or management)

  • 5 full HTA reports and 56 rapid technology assessments were used for reimbursement decisions, while 4 full HTA reports and 2 rapid assessments were used for disinvestment decisions and resulted in reduced volumes and expenditure. There were 2 full HTA reports showing no impact on decision-making. Impact was most evident for hospital technologies.

  • In 48% of reports produced for reimbursement/investment decisions, the recommendation and decision were totally consistent. In 40% of reports, technologies that were not recommended were included on certain conditions, while the decision was more restrictive than the recommendation for 12% of reports.

Financial

  

  • Several millions of euros were saved due to HTA recommendations. For disinvestment decisions, cost savings were about 3 million euros per report, with huge variation (0–12 million). Savings were frequently for more than one hospital (regional hospital associations).

In France, Bodeau-Livinec et al.[21] assessed the perceptions of various stakeholders regarding the use of HTA recommendations produced by the CEDIT. Decision makers found these recommendations very useful and reported a good match between the recommendations and their implementation. Of the 13 recommendations produced, ten had an impact on the introduction of the technology in health organizations and only one did not have an impact; the impact of the two remaining recommendations was impossible to assess.

In two studies[23, 24], Schumacher and Zechmeister analyzed the impact of the HTA research program of the Institute of Technology Assessment (ITA) and the Ludwig Boltzmann Institute for HTA (LBI-HTA) on the Austrian health-care system. In one study, they used a multidimensional framework based on seven impact categories: awareness, acceptance, policy process, policy decision, practice (clinical, reimbursement), final outcomes and economic impact, and, lastly, enlightenment[23]. Their results showed evidence of impact for all of the predefined categories, but particularly on hospitals where HTA is used for investment/reimbursement decisions, treatment guidelines, budget allocation, and the preparation of negotiation. For example, authors reported that the recommendation and decision were totally consistent for 48% of the reports produced for reimbursement/investment decisions. For 40% of the reports, technologies that had not been recommended were included based on certain conditions, while in 12% of the reports, the decision was more restrictive than the recommendation[24].

Veluchamy and Alder[16] described the many positive effects of an HTA unit in two hospitals of the Mount Carmel Health Region (USA). In fact, their case study reported integration of patient needs and medical staff interests and capabilities with the hospital’s resources, higher speed of delivery of newly developed treatment, better access for patients to these technologies, and reduced length of stay as consequences of the presence of the HTA unit. Finally, the study by Mitchell et al.[15] described two examples of how local data integrated into hospital-based HTA were used at the institutional level to inform decisions. In the first example, qualitative local data (staffing patterns and local preferences) had considerable bearing on technology choice (the selection of a new cardiac catheterization lab). In the second example, local outcomes data from administrative records were decisive in the decision on whether or not to continue telemedicine services in critical care units. In both case studies, important differences were found among the hospitals.

Financial impact

Among studies that reported financial impact of HTA activities, McGregor’s study[19] demonstrated that 19 accepted reports have resulted in conservation of hospital resources. To measure this impact, they considered that without a negative recommendation their hospital would have authorized the technology; hence, the recommendation translates into cost saving for the hospital. The extent of savings could only be estimated for 15 reports, which accounted for overall estimated savings of CAN$ 9,840,270. Over the course of 8 years in this HTA unit, the average annual quantifiable saving was CAN$ 1,140,958.

The financial impact of the Calgary Health Technology Implementation Unit was illustrated through an example of an evaluation that addressed the issue of arthroplasty operations in a health region of Alberta (Canada)[22]. The authors estimated that savings reached CAN$ 1 million annually through orthopedic supply standardization and a new contract with vendors.

The Schumacher and Zechmeister studies[23, 24] showed that using HTA in decision-making resulted in various economic outcomes. First, several hospital technologies that had been assessed as showing patterns of over-usage were used more restrictively after the HTA report had been published, leading to a decrease in expenditure[23]. Interviews and analysis of administrative data showed that the impact of HTA recommendations translated into a global cost saving of several million euros for single hospital associations[24].

Mini-HTA

Impact on hospital policies and management
Although mini-HTA is widely used in hospitals in Denmark as the principal basis for decision-making, the Ehlers et al. study[3] reported that no decision makers based their decisions exclusively on them. Mini-HTA could ease technology implementation to a considerable degree; through their local participation in the analysis, stakeholders may acquire a better understanding of the new technology and become more willing to implement it[3]. The study by Folkersen and Pedersen[25] showed similar positive effects of the use of mini-HTA in one major Danish hospital: a greater level of contact between doctors and administrative staff and improved relationships between health professionals and economists, which have often been problematic due to the perception of competing priorities (quality vs. budget). A satisfaction rate of 77% with the HTA method among respondents has also been found in this study (Table 3).
Table 3

Reported effects and impact—mini-HTA and ambassador model

References and country

Methods/participants

Mini-HTA or ambassador

Type of impact

Ehlers and Jensen 2006 Denmark[3]

Survey/140 Danish hospitals

Mini-HTA

On decision-making (management)

  • Mini-HTA is used as a decision support tool at all decision-making levels within the Danish hospital sector.

  • No decision makers based their decisions exclusively on mini-HTAs (but always used them as a supplement).

  • In hospital management sectors, the mini-HTA was often the principal basis for decision-making.

  • A majority of decision makers stated that the mini-HTA eased implementation to a considerable or fair degree.

On clinicians’ or other stakeholders’ perceptions

  • Advantages of using mini-HTAs:

    Based on HTA principles

    The form of the tool, be it a tabular form or a checklist

    The way the form or checklist was being used (flexibility, openness, and timing)

  • Disadvantages mentioned typically centered on insufficiency of the evaluation of the evidence base and the lack of quality control.

Folkersen and Pedersen 2006 Denmark[25]

Survey/1 of Denmark’s main public hospital

Mini-HTA

On decision-making (management)

  • The HTA method has improved the relationships between health professionals and economists, which were previously problematic due to the perception of different or opposing priorities (quality vs. budget). Both parties have become more understanding towards the roles and tasks of the other party due to the implementation of mini-HTA.

On clinicians’ or other stakeholders’ perceptions

  • Overall, 77% of respondents were satisfied with the HTA method.

  • Some dissatisfaction concerning the HTA method included:

    Causing a too great and troublesome administrative burden

    Placing too much or exclusive emphasis on financial factors, while neglecting professional and technical aspects

    Causing limits on budgets, which in turn prevented the purchase of new equipment

   

    Financial questions were too difficult to answer for some hospital staff

Rashiq et al 2006 Canada[26]

Pre- and post-session questionnaires/pre: 130 participants; post: 79 (60.8%)

Ambassador model

On decision-making (management)

  • The ambassador program was successful in increasing awareness of the best evidence in chronic non-cancer pain management and positively influenced treatment decisions.

  • Some participants (35%) reported practice changes as a result of the workshops.

  • 70% indicated that an action plan has been developed following the workshop.

  • 80% indicated that they disseminated the material to other practitioners.

On clinicians’ or other stakeholders’ perceptions

   

  • 99% indicated that the workshops had been a useful way of linking research to practice.

Ambassador model

The Rashiq et al. study[26] evaluated the Alberta Ambassador Program, which was seeking to promote changes in practice in rural areas and among isolated practitioners in Alberta, on the topic of chronic non-cancer pain (CNCP) management. The results of a pre- and post-evaluation of 2-h interactive sessions provided by the HTA ambassador showed that it was successful in increasing awareness of the best evidence in CNCP management and positively influenced treatment decisions. The evaluation showed that 35% of participants reported practice changes as a consequence of the workshops, 70% indicated that an action plan has been developed as a result of the program, and 80% indicated that they shared the material with other practitioners[26].

Barriers and facilitators with respect to the success of local/hospital-based HTA and the uptake of recommendations

HTA committee

In a study of proto-HTA—that is the application of an HTA approach without using the term HTA—in the USA, internal politics was one of the most important hindrances in successfully applying the HTA process[10]. The large amount of time required to perform HTA was also seen as a problem by some respondents[10]. Furthermore, lack of understanding also led committees to make poor decisions on some occasions[10]. Another study from the USA[12] reported that hospitals used technology assessment to control expenditures, and there was little or no evaluation of patient outcomes or medical practice patterns. This study also mentioned the importance of producing a credible and useful assessment. Patail and Aranha[13], Luce and Brown[12], Menon and Marshall[9], and Weingart[14] underscored the importance of a multidisciplinary team with strong corporate leadership and the use of a structured process.

According to a study by Rosenstein et al.[11], organizations that demonstrated success in evaluating technology had the following two characteristics: a multidisciplinary composition of the committee that included physician representation and an organizational commitment to dedicating resources to support the technology assessment program. Technology assessment should also be an important component of the hospital’s strategic plan. According to Poulin et al.[18], the use of multicriteria decision tools and patient/public input were also important factors facilitating the use of recommendations. Saaid[17] suggested that the HTA committee or unit needed to be independent to prevent too close a connection between HTA actors and decision makers.

HTA unit

According to Mitchell et al.[15], clinicians should be consulted before decision-making criteria are finalized because they have crucial insight into how a device or technology can increase or decrease the efficiency of care, which ensures better “buy-in” of the finished product[15]. They also mentioned the importance of documenting costs and benefits and considering the attributes that are important for doctors and managers. Among factors explaining uptake of the recommendations produced by their HTA unit, McGregor[19] mentioned the role of a policy committee that includes representatives of nurses, physicians, allied health professionals, patients, and administrators and that reflects the values of institutional members. McGregor and Brophy[19, 20] also mentioned transparency, relevance of topics for hospital management, fairness, and timeliness of reports (ideally delivered within 6 months) as factors increasing the uptake of HTA recommendations. Clear identification of the authorities responsible for the initiation of the report and for its acceptance, as well as the individuals in charge of carrying out its recommendations, was also recommended[19]. Lastly, Lee et al.[22] underscored local focus, involvement in implementation of HTA recommendations, and collaboration with academia as factors related to the success of the Calgary Health Technology Implementation Unit in Alberta.

Among the barriers to the uptake of HTA recommendations, Bodeau-Livinec et al.[21] mentioned time taken to complete investigative procedures, poor knowledge of recommendations, and recommendations becoming obsolete as a result of developments in knowledge and technology. These authors also stressed that some respondents felt that the HTA unit (CEDIT) was too closely connected with decision-making departments[21]. For their part, Schumacher and Zechmeister[23] reported the lack of acceptance of HTA based on stakeholders’ perception that it was more a vehicle for cost containment and rationing rather than a tool supporting redistribution of resources based on scientific evidence for a more efficient use of resources.

Mini-HTA

According to Ehlers et al.[3], mini-HTA facilitated implementation of recommendations because key stakeholders’ participation in the assessment favored a higher degree of ownership and willingness to implement the new technologies. Advantages of using mini-HTAs included the form of the tool, be it a tabular form or a checklist, and the way which it could be used (flexibility, openness, and timing). Disadvantages of mini-HTA typically centered on insufficient evaluation of the evidence base and the lack of quality control, too much emphasis on financial factors, and difficulty answering financial questions for some hospital staff[3].

Ambassador model

The Rashiq et al. study[26] demonstrated that developing a teaching strategy by a multidisciplinary team is a condition for the success of the ambassador approach addressed to rural health practitioners and administrators. The perceived credibility of the ambassadors as content and methodology experts has been shown to play a major role in the uptake of HTA recommendations. To provide one-page summaries of HTA evidence and to deliver locally interactive sessions are other factors facilitating the success of the ambassador approach as a way to transfer HTA knowledge. Good communication skills are also a factor of success[26].

Discussion

Although HTA has been conducted at the local and hospital level for more than two decades, evidence from the scientific literature is very limited regarding its effects on decision-making as well as its impact on costs. This is mainly due to the fact that few evaluations have been conducted by those who are involved in local/hospital-based HTA (internal evaluation) and even fewer by people outside these organizations (external evaluation).

Nevertheless, most studies reviewed reported a positive impact of local/hospital-based HTA on decisions regarding the acquisition or withdrawal of health technologies in hospitals, as well as positive perceptions from managers and clinicians[3, 911, 13, 16, 1825].

Each of the four HTA approaches for performing HTA (ambassador model, mini-HTA, internal committee, and HTA unit) corresponds to specific needs and structures and has its strengths and limitations. The literature shows that the ambassador model can impact clinicians’ decisions[27], but it remains a strategy that relies upon individual clinicians whose influence, interest, and availability may vary. However, this model is associated with minimal costs, essentially limited to support for training and networking of the ambassadors.

The structure (form or checklist) of mini-HTAs and their features (flexibility, openness, and timing) are greatly appreciated by decision makers[3]. However, insufficient evaluations and lack of quality control could be important disadvantages. Consequently, there may be some concern about transparency and partiality[3]. The costs of performing mini-HTAs in Denmark and elsewhere have not been documented in the scientific literature.

With respect to internal HTA committees, some authors expressed concerns about the fact that these committees may not have the expertise to appraise or synthesize scientific evidence adequately[10, 14]. Moreover, risk for conflicts of interests may exist when evaluations are performed at the level of a clinical department rather than at the hospital level[28]. In this case, evaluations may be too narrow in scope and biased towards interventions performed by that department. Although the composition of internal committees varies from one hospital to another, members of such committees are already employed by the health-care organization, which limits the operating costs of these committees. According to several authors[914, 1820], the most efficient structure for an HTA committee would be a single, multidisciplinary committee that includes physician and nursing representation, members of the administration and finance sectors, and, eventually, patient representatives[18, 20].

Lastly, the formal HTA unit, which is the most complex organizational structure for hospital HTA, presents several advantages, such as depth, high quality, and scientific rigor of the HTA process[6, 15, 16]. The fact that the HTA unit works in partnership with all stakeholders interested in the technology, and its relative independence from clinicians or hospital management, is also highlighted as a benefit of this kind of structure[20]. Nevertheless, its main disadvantage is the fact that the HTA unit requires investments in terms of salary and space for professionals, which poses a trade-off for hospital managers. Time is also needed to implement an HTA unit in a hospital because of the learning curve, but having an experienced HTA professional leading the unit can shorten this.

Assessing the impact of local/hospital-based HTA recommendations on decision-making can be challenging, particularly for technologies whose value is only perceived after several years of utilization[21]. In such cases, it is difficult to predict with certainty whether the dissemination of the technology would have been the same if the HTA were not carried out. Moreover, the presence of some incentives or circumstances that promote the dissemination of the technology may impede assessment of the impact of some HTA recommendations[21]. Other sources of information such as scientific publications may have an impact on the introduction of new technologies, making identification of the specific impact of HTA recommendations more difficult[29].

Although the first studies on local/hospital HTA were published almost 25 years ago, most of these experiences are recent. The limited evidence currently available on local/hospital HTA makes it difficult to evaluate its effects and impacts at the different levels of health service delivery. Besides, many of these evaluations were conducted internally, by people who are involved in HTA, introducing a potential bias. Further research is necessary to explore the conditions under which local/hospital-based HTA results and recommendations can impact hospital policies, clinical decisions, and quality of patient care and optimize the use of scarce resources. It would be necessary to conduct more independent studies that use high-quality qualitative and quantitative methods for evaluating the impact of HTA in several dimensions. One of the dimensions deserving special attention is the impact of HTA on patients, as they should benefit from optimal resource allocation based on scientific evidence.

Another shortcoming of the studies reviewed is the absence of a theoretical model to evaluate the impact of local/hospital HTA. The only exception is the study by Shumacher and Zeichmester[23] that uses a multidimensional conceptual model with seven impact categories. A framework for evaluating local/hospital HTA should consider its impact at different levels, starting with the uptake of HTA recommendations by decision makers and its effect on funding decisions, its effect on health-care professional practices, and ultimately its impact on health-care outcomes both short and long term. Such evaluation would however require significant resources due to the need to control many confounding factors.

Study limitations

Although it provides a comprehensive synthesis of the effects and impact of local/hospital-based HTA conducted internationally, this review has some limitations. First, given that only published studies have been included in the review, some valuable studies concerning the impact of hospital-based HTA may have been overlooked. For example, we found an abstract about the potential impact of hospital-based HTA in Italy[30], but we were unable to obtain the full study report from the authors. Furthermore, many relevant abstracts are presented each year at the HTAi annual conference, such as the experience of the Hospital Clinic in Catalonia[31]. However, information reported in an abstract is often limited, making it difficult to include this information in our synthesis. However, we have consulted two other reviews that were published as reports and include gray literature such as conference abstracts and research reports[6, 32]. Although those reviews present a few international experiences of local/hospital HTA that are not reported in our review, they do not provide new evidence on the specific topic of our review.

Second, the use of the model proposed by the Hospital Based HTA Interest Sub-Group[1] to classify the different approaches to conducting hospital-based HTA has limitations because it includes distinct features: organizational structure (HTA committee and HTA unit), HTA tool (mini-HTA), and dissemination strategy (ambassador model). Also, the structure and functions of HTA committees varied greatly between studies, and the early HTA experiences reported might not conform to the actual standards in this field. This leads to another limitation of this review due to the great heterogeneity between included studies. In fact, this is not possible to make comparisons between the different local/hospital HTA experiences because they represent distinct interventions. Further work is needed in order to refine the conceptualization of local/hospital-based HTA and provide a common understanding of what it is and how it could be evaluated. A third limitation to this review is the fact that we have not considered study quality in the interpretation of the results. Given that studies published after 2000 generally have higher quality, an update of this review should focus on the most recent literature on local/hospital HTA.

Conclusions

This systematic review provides a basis for understanding how local/hospital-based HTA could impact decision-making regarding the introduction of new technologies in the health-care system. However, our capacity to evaluate the real impacts of local/hospital-based HTA is limited given the relatively small number of evaluations with quantitative data, the lack of clear comparators, and the fact that most evaluations are conducted internally. Further research, using rigorous methods and preferably conducted by external assessors, is necessary to understand the conditions under which local/hospital-based HTA results and recommendations can impact hospital policies, clinical decisions, and quality of care and optimize the use of scarce resources.

Abbreviations

HTA: 

Health technology assessment

HTAi: 

Health Technology Assessment International

MMAT: 

Mixed Methods Appraisal Tool

TAU: 

Technology Assessment Unit

CEDIT (French): 

Committee for the Assessment and Dissemination of Technological Innovations

MUHC: 

McGill University Health Center

CNCP: 

Chronic non-cancer pain.

Declarations

Acknowledgements

The realization of this systematic review was made possible with the support of a planning grant from the Canadian Institutes of Health Research (# 201306-KPE-309195). We also acknowledge the contribution of the former Canadian Health Services Foundation for Research for a synthesis grant that provided a basis for this work (# CHS-2419). MPG holds the Tier 2 Canada Research Chair in Technologies and Practices in Health. We are grateful to Ziggi Ivan Santini for the translation of the publication in Danish.

Authors’ Affiliations

(1)
Centre de recherche du CHU de Québec
(2)
Faculty of Nursing, Université Laval
(3)
UETMIS, Hôpital Hôtel-Dieu, CHUS
(4)
Centre de recherche du CHUS

References

  1. Cicchetti A, Marchetti M, Dibidino R, Corio M, on behalf of HTAi’s Hospital Based Sub Interest Group: Hospital based HTA. Hospital based health technology assessment. World-wide survey. [http://www.htai.org/fileadmin/HTAi_Files/ISG/HospitalBasedHTA/2008Files/HospitalBasedHTAISGSurveyReport.pdf]
  2. Granados A: Health technology assessment and clinical decision making: which is the best evidence?. Int J Technol Assess Health Care. 1999, 15: 585-592.PubMedGoogle Scholar
  3. Ehlers L, Vestergaard M, Kidholm K, Bonnevie B, Pedersen PH, Jorgensen T, Jensen MF, Kristensen FB, Kjolby M: Doing mini-health technology assessments in hospitals: a new concept of decision support in health care?. Int J Technol Assess Health Care. 2006, 22: 295-301.View ArticlePubMedGoogle Scholar
  4. Martelli N, Lelong AS, Prognon P, Pineau J: Hospital-based health technology assessment for innovative medical devices in university hospitals and the role of hospital pharmacists: learning from international experience. Int J Technol Assess Health Care. 2013, 29: 185-191. 10.1017/S0266462313000019.View ArticlePubMedGoogle Scholar
  5. Moher D, Liberati A, Tetzlaff J, Altman DG, Group P: Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009, 6: e1000097-10.1371/journal.pmed.1000097.View ArticlePubMedPubMed CentralGoogle Scholar
  6. Gagnon M-P, Abdeljelil AB, Desmartis M, Légaré F, Ouimet M, Gagnon J, St-Pierre M, Rhainds M, Coulombe M: Opportunities to Promote Efficiency in Hospital Decision-Making Through the Use of Health Technology Assessment. 2011, Canadian Health Services Research Foundation: OttawaGoogle Scholar
  7. Proposal: a mixed methods appraisal tool for systematic mixed studies reviews. [http://mixedmethodsappraisaltoolpublic.pbworks.com]. Retrieved 2014, July 2. Archived by WebCite® at http://www.webcitation.org/5tTRTc9yJ
  8. Centre for Reviews and Dissemination: Systematic Reviews. CRD’s Guidance for Undertaking Reviews in Health Care. 2009, York: CRD, University of YorkGoogle Scholar
  9. Menon D, Marshall D: Technology assessment in teaching hospitals. Dimens Health Serv. 1990, 67: 26-28.PubMedGoogle Scholar
  10. Cram N, Groves J, Foster L: Technology assessment–a survey of the clinical engineer’s role within the hospital. J Clin Eng. 1997, 22: 373-382. 10.1097/00004669-199711000-00012.View ArticlePubMedGoogle Scholar
  11. Rosenstein AH, O’Daniel M, Geoghan K: Assessing new technology: how are other hospitals facing the challenge?. Healthc Financ Manage. 2003, 57: 70-74.PubMedGoogle Scholar
  12. Luce BR, Brown RE: The use of technology assessment by hospitals, health maintenance organizations, and third-party payers in the United States. Int J Technol Assess Health Care. 1995, 11: 79-92. 10.1017/S0266462300005274.View ArticlePubMedGoogle Scholar
  13. Patail BM, Aranha AN: Role of the biomedical engineering department in William Beaumont Hospital’s technology assessment process. J Clin Eng. 1995, 20: 290-296. 10.1097/00004669-199507000-00013.View ArticlePubMedGoogle Scholar
  14. Weingart SN: Deciding to buy expensive technology. The case of biliary lithotripsy. Int J Technol Assess Health Care. 1995, 11: 301-315. 10.1017/S0266462300006917.View ArticlePubMedGoogle Scholar
  15. Mitchell MD, Williams K, Brennan PJ, Umscheid CA: Integrating local data into hospital-based healthcare technology assessment: two case studies. Int J Technol Assess Health Care. 2010, 26: 294-300. 10.1017/S0266462310000334.View ArticlePubMedGoogle Scholar
  16. Veluchamy S, Alder HC: Health care technology assessment and adoption: a case study. Hosp Technol Ser. 1989, 8: 1-12.PubMedGoogle Scholar
  17. Saaid H: The impact of health technology assessment on decision-making processes in public versus not-for-profit private hospitals. Am J Med. 2011, 2: 72-78. 10.3844/amjsp.2011.72.78.View ArticleGoogle Scholar
  18. Poulin P, Austen L, Kortbeek JB, Lafreniere R: New technologies and surgical innovation: five years of a local health technology assessment program in a surgical department. Surg Innov. 2012, 19: 187-199. 10.1177/1553350611421916.View ArticlePubMedGoogle Scholar
  19. McGregor M: The Impact of Reports of the Technology Assessment Unit of the McGill University Health Centre. 2012, Montreal: Technology Assessment Unit (TAU) of the McGill University Health Centre (MUHC)Google Scholar
  20. McGregor M, Brophy JM: End-user involvement in health technology assessment (HTA) development: a way to increase impact. Int J Technol Assess Health Care. 2005, 21: 263-267.PubMedGoogle Scholar
  21. Bodeau-Livinec F, Simon E, Montagnier-Petrissans C, Joel ME, Fery-Lemonnier E: Impact of CEDIT recommendations: an example of health technology assessment in a hospital network. Int J Technol Assess Health Care. 2006, 22: 161-168.View ArticlePubMedGoogle Scholar
  22. Lee RC, Marshall D, Waddell C, Hailey D, Juzwishin D: Health technology assessment, research, and implementation within a health region in Alberta, Canada. Int J Technol Assess Health Care. 2003, 19: 513-520.View ArticlePubMedGoogle Scholar
  23. Schumacher I, Zechmeister I: Assessing the impact of health technology assessment on the Austrian healthcare system. Int J Technol Assess Health Care. 2013, 29: 84-91. 10.1017/S0266462312000748.View ArticlePubMedGoogle Scholar
  24. Zechmeister I, Schumacher I: The impact of health technology assessment reports on decision making in Austria. Int J Technol Assess Health Care. 2012, 28: 77-84. 10.1017/S0266462311000729.View ArticlePubMedGoogle Scholar
  25. Folkersen J, Pedersen PH: [Attitudes to the use of a decision support method when introducing new medical technology at the University Hospital of Copenhagen]. Ugeskr Laeger. 2006, 168: 2069-2074.PubMedGoogle Scholar
  26. Rashiq S, Barton P, Harstall C, Schopflocher D, Taenzer P: The Alberta Ambassador Program: delivering health technology assessment results to rural practitioners. BMC Med Educ. 2006, 6: 21-10.1186/1472-6920-6-21.View ArticlePubMedPubMed CentralGoogle Scholar
  27. Jonsson E: History of health technology assessment in Sweden. Int J Technol Assess Health Care. 2009, 25 (Suppl 1): 42-52.View ArticlePubMedGoogle Scholar
  28. Umscheid CA, Williams K, Brennan PJ: Hospital-based comparative effectiveness centers: translating research into practice to improve the quality, safety and value of patient care. J Gen Intern Med. 2010, 25: 1352-1355. 10.1007/s11606-010-1476-9.View ArticlePubMedPubMed CentralGoogle Scholar
  29. Jacob R, McGregor M: Assessing the impact of health technology assessment. Int J Technol Assess Health Care. 1997, 13: 68-80. 10.1017/S0266462300010242.View ArticlePubMedGoogle Scholar
  30. Boscolo PR, Ciani O, Torbica A: Hospital-based HTA in Italy: diffusion and potential impact. Value Health. 2012, 15: A283-View ArticleGoogle Scholar
  31. Morilla-Bachs I, Gutierrez-Moreno S, Bigorra- Llosas J, Sampietro-Colom L: Impact of a hospital based HTA unit. The experience of the Hospital Clinic Barcelona (Spain). 9th HTAi Annual Meeting: 25-27. 2012, June ; BilbaoGoogle Scholar
  32. Bigorra Llosas J, Gomis R, Sampietro-Colom L, Huc M, Lurigados Delgado C, Zamora AG, Carné Cladellas X, Piqué Badia JM: Desarrollo de un sistema de conocimiento compartido para la evaluación en red de la innovación tecnológica en medicina. 2009, Madrid: Plan de Calidad para el Sistema Nacional de Salud. Ministerio de Ciencia e Innovación. Agència d’Avaluació de Tecnologia i Recerca Mèdiques de CataluñaGoogle Scholar

Copyright

© Gagnon et al.; licensee BioMed Central Ltd. 2014

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Advertisement