Skip to main content

Advertisement

Effects and repercussions of local/hospital-based health technology assessment (HTA): a systematic review

Abstract

Background

Health technology assessment (HTA) is increasingly performed at the local or hospital level where the costs, impacts, and benefits of health technologies can be directly assessed. Although local/hospital-based HTA has been implemented for more than two decades in some jurisdictions, little is known about its effects and impact on hospital budget, clinical practices, and patient outcomes. We conducted a mixed-methods systematic review that aimed to synthesize current evidence regarding the effects and impact of local/hospital-based HTA.

Methods

We identified articles through PubMed and Embase and by citation tracking of included studies. We selected qualitative, quantitative, or mixed-methods studies with empirical data about the effects or impact of local/hospital-based HTA on decision-making, budget, or perceptions of stakeholders. We extracted the following information from included studies: country, methodological approach, and use of conceptual framework; local/hospital HTA approach and activities described; reported effects and impacts of local/hospital-based HTA; factors facilitating/hampering the use of hospital-based HTA recommendations; and perceptions of stakeholders concerning local/hospital HTA. Due to the great heterogeneity among studies, we conducted a narrative synthesis of their results.

Results

A total of 18 studies met the inclusion criteria. We reported the results according to the four approaches for performing HTA proposed by the Hospital Based HTA Interest Sub-Group: ambassador model, mini-HTA, internal committee, and HTA unit. Results showed that each of these approaches for performing HTA corresponds to specific needs and structures and has its strengths and limitations. Overall, studies showed positive impacts related to local/hospital-based HTA on hospital decisions and budgets, as well as positive perceptions from managers and clinicians.

Conclusions

Local/hospital-based HTA could influence decision-making on several aspects. It is difficult to evaluate the real impacts of local HTA at the different levels of health care given the relatively small number of evaluations with quantitative data and the lack of clear comparators. Further research is necessary to explore the conditions under which local/hospital-based HTA results and recommendations can impact hospital policies, clinical decisions, and quality of care and optimize the use of scarce resources.

Background

While health technology assessment (HTA) is often done at a national or international level, many local health services and hospitals consider that it makes sense to move the assessment closer to the point of care, where the costs, impacts, and benefits of technologies can be directly assessed. This is justified by the fact that many decisions regarding health technologies (prioritization, investment, adoption, and disinvestment) are made at the local/hospital level[1]. This is also the result of an increasing awareness that specific organizational contexts should be taken into account when assessing health technologies[2].

With the emergence of HTA activities in hospitals, the Hospital Based HTA Interest Sub-Group was created within the HTAi—the international scientific and professional society for HTA—in 2006. In 2008, this sub-group elaborated a conceptual model to classify the different approaches for performing HTA within hospitals around the world[1]. Four different approaches were described: 1) ambassador model, 2) mini-HTA, 3) internal committee, and 4) HTA unit. The ambassador model seeks to promote changes in practice through a specific HTA dissemination approach. In this approach, interested clinicians who are recognized as opinion leaders play the role of ambassadors of the HTA message within health-care organizations at regional and local levels. Mini-HTA is a management and decision support tool that consists of questions about the technology, the patient, the organization, and the financial aspects[3]. The mini-HTA is usually done by a single professional who often participates in the assessment process, collecting data at the hospital level in order to inform decision makers. The internal committee consists in many cases of an ad hoc multidisciplinary group representing different perspectives, in charge of reviewing evidence and making recommendations to the health-care organization. The HTA unit represents the most structured model for hospital-based HTA. It is a formal organizational structure with specialized HTA personnel working on a full-time basis on the production of HTA material of high scientific quality.

According to a recent survey, the number of hospitals performing HTA is growing around the world[4]. While local/hospital-based HTA has been in place for a few decades in some jurisdictions, there is very limited knowledge of its effects on decisions regarding health technologies. Thus, it is important to review current evidence about local/hospital-based HTA in order to inform future initiatives.

This paper presents a systematic review of the effects and impact of local/hospital-based HTA reported in the literature on decision-making or management. The main questions that guided this review are the following: 1) Have HTA recommendations been accepted and implemented? 2) What expenses and savings are related to HTA activities and their recommendations? 3) What are the perceptions of various stakeholders towards local/hospital HTA? We also compiled information about strengths and weaknesses of different approaches of local/hospital-based HTA, as well as barriers and facilitators with respect to the implementation of their recommendations, as secondary outcomes.

Methods

This review adheres to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist[5] (see Additional file1).

Screening and selection

A review protocol was established, based on a previously published knowledge synthesis[6]. We searched for qualitative, quantitative, or mixed-methods studies with empirical data about the effects or impact of local/hospital-based HTA. We did not use date or language limits. Studies were excluded if they were not about local/hospital HTA activities; did not provide data concerning the impact of HTA activities on decision-making or management, budget, or perceptions of stakeholders as regards HTA activities; and were not based on an empirical analysis of the effects or impact of HTA activities or recommendations.

An information specialist (WW) developed the search strategy and performed literature search in two databases (PubMed and Embase; see Additional file2). Other literature were identified through citation tracking of the included studies. Two members of the team (WW, MD) independently reviewed titles and abstracts for relevance, with the intervention of a third reviewer (MPG) in case of discrepancy. Teams of two reviewers (MD, TP, WW, and MPG) then independently conducted full-text reviews for eligibility; any disagreement was resolved by discussion among reviewers.

Extraction of data

The same teams of two reviewers independently extracted the following information from the selected studies: general characteristics of the studies (country, methodological approach); local/hospital HTA approach and activities; use of conceptual framework or model; reported effects and impacts of local/hospital-based HTA; factors facilitating/hampering the use of hospital-based HTA recommendations; perceptions of stakeholders regarding local/hospital HTA activities (benefits and concerns); and resources needed for implementing hospital-based HTA. We also assessed study quality using the Mixed Methods Appraisal Tool (MMAT)[7]. Due to the great heterogeneity among studies, we conducted a narrative synthesis of their results[8].

Results

A total of 707 potentially relevant articles were identified from the main search strategy, of which 15 met the eligibility criteria and were included. Other search strategies identified four additional eligible publications, but one of them was an update of an included study. Consequently, a total of 18 studies were included, described in 19 papers. The study selection process is presented in Additional file3.

Overview of studies

More than half of the studies were from the USA (n = 7) and Canada (n = 5). Other studies were from Denmark (n = 2), Austria (n = 2), and Australia and France (n = 1 each). Over half of the articles (10/18) were published since 2005 and a third (6/18) since 2010. The six articles published prior to 2000 were all from the USA, but one study was conducted in Canada[9].

Purpose and methods of the studies

Four studies surveyed hospitals in a country or in part of a country to explore how HTA was used in decision-making for the introduction of technological innovations. They used questionnaires[911] or semi-structured interviews[12]. Five studies used a case study to report the experience of use of HTA in a particular hospital[1315] or in a few hospitals[16, 17]. In all of these studies, with the exception of two[15, 16], HTA was performed by committees of different types, often of an ad hoc nature.

Six studies reported an analysis of the outcomes of an HTA program or unit and the impact of their reports and recommendations. In Canada, Poulin et al.[18] conducted a retrospective analysis of the outcomes of an HTA program (over a 5-year period) and of the kinds of decisions that were made based on local committee recommendations. McGregor and Brophy[19, 20] evaluated the impact of 55 HTA reports that their Technology Assessment Unit (TAU) produced during 8 years of service. They used interviews with local administrative and clinical decision makers as well as document analysis to evaluate the impact of HTA reports and recommendations on hospital policy decision-making and hospital spending. Other impact studies focused on an HTA unit or committee that covered many hospitals. For instance, Bodeau-Livinec et al.[21] studied the Committee for the Assessment and Dissemination of Technological Innovations (CEDIT in French), an HTA unit that covers a network of 39 university hospitals located in the Paris region. Similarly, the study by Lee et al.[22] illustrated the impact of the activities of an HTA unit in a health region in Alberta (Canada). Finally, Schumacher and Zechmeister[23, 24] focused on the impact of an HTA research program that produces various types of reports for decision makers at different levels of the health-care system in Austria, including hospitals.

Using the survey approach, two studies assessed the use of mini-HTA in the introduction of new technologies in the Danish hospital sector[3] or in a major public hospital in Denmark[25]. The study by Rashiq et al.[26] evaluated the Alberta Ambassador Program put in place to inform clinicians about current research evidence on the management of chronic non-cancer pain, using a pre- and post-session questionnaire.

Quality of the studies

The assessment of the quality of studies can be found in Additional file4. Even if the assessment of the quality of studies was conducted using the MMAT, we decided not to take this into account in the interpretation of our results due to the exploratory aim of the review. Overall, studies conducted after 2000 were of better quality than earlier studies.

Reported effects and impacts of local/hospital-based HTA activities

Given the great heterogeneity among studies, notably in terms of approaches for performing HTA, we report their effects and impacts according to the four categories proposed by the Hospital Based HTA Interest Sub-Group[1] (ambassador model, mini-HTA, internal committee, and HTA unit). However, we found a wide range of “committees” in the studies reviewed. Their composition and structure varied across studies (from ad hoc committees to very structured ones). In this review, we distinguish committees from HTA units that are formal organizational structures with dedicated HTA personnel.

HTA committees

Reported effects and impacts of activities of different types of HTA committees on decision-making (or management) are presented in Table 1. As the only financial aspect reported in some of these studies[12, 13] is the minimum cost related to a technology for deciding whether to assess them, we will not create a specific section here for financial impact.

Table 1 Reported effects and impact—HTA committees
Impact on hospital policies and management

A survey conducted in 30 organizations (including hospitals, health maintenance organizations (HMOs), and third-party payers) in the USA in 1995[12] showed that decision makers used HTA to inform purchasing or coverage decisions regarding new and expensive technologies and drugs, as a means to better use resources. Decision makers usually followed the purchase or no purchase recommendation made by the assessment committee. However, with the exception of pharmacy committees, decisions were based on financial evaluation with little or no formal evaluation of changes in patient outcomes or medical practice patterns. A survey in teaching hospitals across Canada conducted in 1990[9] found that HTA was common in teaching hospitals (43/50), although it took various forms. Thirty-four of the 43 hospitals practicing HTA stated that information from HTA was used in decision-making about new technology acquisition. However, only 23 hospitals had a formal management structure for HTA. In the Cram et al. study[10] of clinical engineering departments throughout the USA conducted in 1997, 80% of the respondents (20/25) who used an HTA process felt that it was a useful tool and several of them had used it to cut costs and provide more standardization. The benefits of different HTA processes varied from hospital to hospital, but some respondents stressed that the HTA process informed purchase requests to be evaluated by a multifunctional team, which allows for broadening the input involved in decision-making processes[10].

In a case study from 1995 concerning the impact of the assessment of biliary lithotripsy by a task force of physicians in a major teaching hospital in the USA, Weingart[14] reported that the works of this committee have led to purchasing of the technology, which was later described as an engineering disaster. The author outlined many reasons for this failure: the exaggerated optimism of physicians in the task force, despite limited data in published reports; a mandate that was too narrow and did not include comparisons with alternative technologies; the lack of expertise of decision makers in assessing feasibility and profitability; and an assessment process that was not structured enough.

Patail and Aranha[13] reported the impact of the work of a multidisciplinary assessment team, including managers and biomedical engineers, in a US hospital in 1995. Their results showed that of 16 technologies formally approved by the assessment team between 1988 and 1993, a total of 13 had been implemented. According to these authors, HTA made it possible for decision makers to avoid taking the information provided by manufacturers and vendors for granted.

In 2003, a survey among 19 hospitals in the western part of the USA[11] revealed that although 90% of hospitals reported the existence of a specific staff responsible for providing formal reviews, only 42% had a dedicated technology assessment committee. Among these committees, 28% had direct responsibility for technology approval. Even if committees did not make the final decision, respondents reported that their recommendations were well integrated into the hospital’s mission and strategic plan.

Recently, the multiple case study by Saaid[17] examined the use of HTA in decision-making processes for acquiring new health technologies in four selected hospitals (three not-for-profit private hospitals and one public hospital) of southeast Queensland (Australia). The results showed that the impact of HTA as a support tool for decision makers was relatively minimal. Decisions in private hospitals were informal and driven by business strategy and the cost-effectiveness of the technologies and also significantly influenced by physicians. In the public hospital, HTA was a requirement in decision-making, and a formal committee was in place, but it was at an early stage of development.

For their part, Poulin et al.[18] analyzed the outcomes of a local HTA program in Alberta (Canada) over a 5-year period. They reported that decisions based on recommendations of this program were rarely “yes” or “no” but offered many approval options between full acceptance and rejection. Many technologies have received restricted approval, with full approval contingent on satisfying conditions such as clinical outcomes review, training protocol development, or funding.

HTA unit

Impact on hospital policies and management

Reported effects and impact of activities of HTA units are presented in Table 2. Four studies of this category analyzed the impact of HTA reports and recommendations on hospital policies and implementation. The study by McGregor and Brophy[19, 20] evaluated the impact of 55 reports issued by the TAU of the McGill University Health Center (MUHC) between 2004 and 2011. Of the 63 recommendations produced in these reports, 45 (71%) have been accepted and incorporated into hospital policy. The most frequent reason for recommendations not being accepted was failure to identify administrative responsibility to carry this out.

Table 2 Reported effects and impact—HTA units

In France, Bodeau-Livinec et al.[21] assessed the perceptions of various stakeholders regarding the use of HTA recommendations produced by the CEDIT. Decision makers found these recommendations very useful and reported a good match between the recommendations and their implementation. Of the 13 recommendations produced, ten had an impact on the introduction of the technology in health organizations and only one did not have an impact; the impact of the two remaining recommendations was impossible to assess.

In two studies[23, 24], Schumacher and Zechmeister analyzed the impact of the HTA research program of the Institute of Technology Assessment (ITA) and the Ludwig Boltzmann Institute for HTA (LBI-HTA) on the Austrian health-care system. In one study, they used a multidimensional framework based on seven impact categories: awareness, acceptance, policy process, policy decision, practice (clinical, reimbursement), final outcomes and economic impact, and, lastly, enlightenment[23]. Their results showed evidence of impact for all of the predefined categories, but particularly on hospitals where HTA is used for investment/reimbursement decisions, treatment guidelines, budget allocation, and the preparation of negotiation. For example, authors reported that the recommendation and decision were totally consistent for 48% of the reports produced for reimbursement/investment decisions. For 40% of the reports, technologies that had not been recommended were included based on certain conditions, while in 12% of the reports, the decision was more restrictive than the recommendation[24].

Veluchamy and Alder[16] described the many positive effects of an HTA unit in two hospitals of the Mount Carmel Health Region (USA). In fact, their case study reported integration of patient needs and medical staff interests and capabilities with the hospital’s resources, higher speed of delivery of newly developed treatment, better access for patients to these technologies, and reduced length of stay as consequences of the presence of the HTA unit. Finally, the study by Mitchell et al.[15] described two examples of how local data integrated into hospital-based HTA were used at the institutional level to inform decisions. In the first example, qualitative local data (staffing patterns and local preferences) had considerable bearing on technology choice (the selection of a new cardiac catheterization lab). In the second example, local outcomes data from administrative records were decisive in the decision on whether or not to continue telemedicine services in critical care units. In both case studies, important differences were found among the hospitals.

Financial impact

Among studies that reported financial impact of HTA activities, McGregor’s study[19] demonstrated that 19 accepted reports have resulted in conservation of hospital resources. To measure this impact, they considered that without a negative recommendation their hospital would have authorized the technology; hence, the recommendation translates into cost saving for the hospital. The extent of savings could only be estimated for 15 reports, which accounted for overall estimated savings of CAN$ 9,840,270. Over the course of 8 years in this HTA unit, the average annual quantifiable saving was CAN$ 1,140,958.

The financial impact of the Calgary Health Technology Implementation Unit was illustrated through an example of an evaluation that addressed the issue of arthroplasty operations in a health region of Alberta (Canada)[22]. The authors estimated that savings reached CAN$ 1 million annually through orthopedic supply standardization and a new contract with vendors.

The Schumacher and Zechmeister studies[23, 24] showed that using HTA in decision-making resulted in various economic outcomes. First, several hospital technologies that had been assessed as showing patterns of over-usage were used more restrictively after the HTA report had been published, leading to a decrease in expenditure[23]. Interviews and analysis of administrative data showed that the impact of HTA recommendations translated into a global cost saving of several million euros for single hospital associations[24].

Mini-HTA

Impact on hospital policies and management

Although mini-HTA is widely used in hospitals in Denmark as the principal basis for decision-making, the Ehlers et al. study[3] reported that no decision makers based their decisions exclusively on them. Mini-HTA could ease technology implementation to a considerable degree; through their local participation in the analysis, stakeholders may acquire a better understanding of the new technology and become more willing to implement it[3]. The study by Folkersen and Pedersen[25] showed similar positive effects of the use of mini-HTA in one major Danish hospital: a greater level of contact between doctors and administrative staff and improved relationships between health professionals and economists, which have often been problematic due to the perception of competing priorities (quality vs. budget). A satisfaction rate of 77% with the HTA method among respondents has also been found in this study (Table 3).

Table 3 Reported effects and impact—mini-HTA and ambassador model

Ambassador model

The Rashiq et al. study[26] evaluated the Alberta Ambassador Program, which was seeking to promote changes in practice in rural areas and among isolated practitioners in Alberta, on the topic of chronic non-cancer pain (CNCP) management. The results of a pre- and post-evaluation of 2-h interactive sessions provided by the HTA ambassador showed that it was successful in increasing awareness of the best evidence in CNCP management and positively influenced treatment decisions. The evaluation showed that 35% of participants reported practice changes as a consequence of the workshops, 70% indicated that an action plan has been developed as a result of the program, and 80% indicated that they shared the material with other practitioners[26].

Barriers and facilitators with respect to the success of local/hospital-based HTA and the uptake of recommendations

HTA committee

In a study of proto-HTA—that is the application of an HTA approach without using the term HTA—in the USA, internal politics was one of the most important hindrances in successfully applying the HTA process[10]. The large amount of time required to perform HTA was also seen as a problem by some respondents[10]. Furthermore, lack of understanding also led committees to make poor decisions on some occasions[10]. Another study from the USA[12] reported that hospitals used technology assessment to control expenditures, and there was little or no evaluation of patient outcomes or medical practice patterns. This study also mentioned the importance of producing a credible and useful assessment. Patail and Aranha[13], Luce and Brown[12], Menon and Marshall[9], and Weingart[14] underscored the importance of a multidisciplinary team with strong corporate leadership and the use of a structured process.

According to a study by Rosenstein et al.[11], organizations that demonstrated success in evaluating technology had the following two characteristics: a multidisciplinary composition of the committee that included physician representation and an organizational commitment to dedicating resources to support the technology assessment program. Technology assessment should also be an important component of the hospital’s strategic plan. According to Poulin et al.[18], the use of multicriteria decision tools and patient/public input were also important factors facilitating the use of recommendations. Saaid[17] suggested that the HTA committee or unit needed to be independent to prevent too close a connection between HTA actors and decision makers.

HTA unit

According to Mitchell et al.[15], clinicians should be consulted before decision-making criteria are finalized because they have crucial insight into how a device or technology can increase or decrease the efficiency of care, which ensures better “buy-in” of the finished product[15]. They also mentioned the importance of documenting costs and benefits and considering the attributes that are important for doctors and managers. Among factors explaining uptake of the recommendations produced by their HTA unit, McGregor[19] mentioned the role of a policy committee that includes representatives of nurses, physicians, allied health professionals, patients, and administrators and that reflects the values of institutional members. McGregor and Brophy[19, 20] also mentioned transparency, relevance of topics for hospital management, fairness, and timeliness of reports (ideally delivered within 6 months) as factors increasing the uptake of HTA recommendations. Clear identification of the authorities responsible for the initiation of the report and for its acceptance, as well as the individuals in charge of carrying out its recommendations, was also recommended[19]. Lastly, Lee et al.[22] underscored local focus, involvement in implementation of HTA recommendations, and collaboration with academia as factors related to the success of the Calgary Health Technology Implementation Unit in Alberta.

Among the barriers to the uptake of HTA recommendations, Bodeau-Livinec et al.[21] mentioned time taken to complete investigative procedures, poor knowledge of recommendations, and recommendations becoming obsolete as a result of developments in knowledge and technology. These authors also stressed that some respondents felt that the HTA unit (CEDIT) was too closely connected with decision-making departments[21]. For their part, Schumacher and Zechmeister[23] reported the lack of acceptance of HTA based on stakeholders’ perception that it was more a vehicle for cost containment and rationing rather than a tool supporting redistribution of resources based on scientific evidence for a more efficient use of resources.

Mini-HTA

According to Ehlers et al.[3], mini-HTA facilitated implementation of recommendations because key stakeholders’ participation in the assessment favored a higher degree of ownership and willingness to implement the new technologies. Advantages of using mini-HTAs included the form of the tool, be it a tabular form or a checklist, and the way which it could be used (flexibility, openness, and timing). Disadvantages of mini-HTA typically centered on insufficient evaluation of the evidence base and the lack of quality control, too much emphasis on financial factors, and difficulty answering financial questions for some hospital staff[3].

Ambassador model

The Rashiq et al. study[26] demonstrated that developing a teaching strategy by a multidisciplinary team is a condition for the success of the ambassador approach addressed to rural health practitioners and administrators. The perceived credibility of the ambassadors as content and methodology experts has been shown to play a major role in the uptake of HTA recommendations. To provide one-page summaries of HTA evidence and to deliver locally interactive sessions are other factors facilitating the success of the ambassador approach as a way to transfer HTA knowledge. Good communication skills are also a factor of success[26].

Discussion

Although HTA has been conducted at the local and hospital level for more than two decades, evidence from the scientific literature is very limited regarding its effects on decision-making as well as its impact on costs. This is mainly due to the fact that few evaluations have been conducted by those who are involved in local/hospital-based HTA (internal evaluation) and even fewer by people outside these organizations (external evaluation).

Nevertheless, most studies reviewed reported a positive impact of local/hospital-based HTA on decisions regarding the acquisition or withdrawal of health technologies in hospitals, as well as positive perceptions from managers and clinicians[3, 911, 13, 16, 1825].

Each of the four HTA approaches for performing HTA (ambassador model, mini-HTA, internal committee, and HTA unit) corresponds to specific needs and structures and has its strengths and limitations. The literature shows that the ambassador model can impact clinicians’ decisions[27], but it remains a strategy that relies upon individual clinicians whose influence, interest, and availability may vary. However, this model is associated with minimal costs, essentially limited to support for training and networking of the ambassadors.

The structure (form or checklist) of mini-HTAs and their features (flexibility, openness, and timing) are greatly appreciated by decision makers[3]. However, insufficient evaluations and lack of quality control could be important disadvantages. Consequently, there may be some concern about transparency and partiality[3]. The costs of performing mini-HTAs in Denmark and elsewhere have not been documented in the scientific literature.

With respect to internal HTA committees, some authors expressed concerns about the fact that these committees may not have the expertise to appraise or synthesize scientific evidence adequately[10, 14]. Moreover, risk for conflicts of interests may exist when evaluations are performed at the level of a clinical department rather than at the hospital level[28]. In this case, evaluations may be too narrow in scope and biased towards interventions performed by that department. Although the composition of internal committees varies from one hospital to another, members of such committees are already employed by the health-care organization, which limits the operating costs of these committees. According to several authors[914, 1820], the most efficient structure for an HTA committee would be a single, multidisciplinary committee that includes physician and nursing representation, members of the administration and finance sectors, and, eventually, patient representatives[18, 20].

Lastly, the formal HTA unit, which is the most complex organizational structure for hospital HTA, presents several advantages, such as depth, high quality, and scientific rigor of the HTA process[6, 15, 16]. The fact that the HTA unit works in partnership with all stakeholders interested in the technology, and its relative independence from clinicians or hospital management, is also highlighted as a benefit of this kind of structure[20]. Nevertheless, its main disadvantage is the fact that the HTA unit requires investments in terms of salary and space for professionals, which poses a trade-off for hospital managers. Time is also needed to implement an HTA unit in a hospital because of the learning curve, but having an experienced HTA professional leading the unit can shorten this.

Assessing the impact of local/hospital-based HTA recommendations on decision-making can be challenging, particularly for technologies whose value is only perceived after several years of utilization[21]. In such cases, it is difficult to predict with certainty whether the dissemination of the technology would have been the same if the HTA were not carried out. Moreover, the presence of some incentives or circumstances that promote the dissemination of the technology may impede assessment of the impact of some HTA recommendations[21]. Other sources of information such as scientific publications may have an impact on the introduction of new technologies, making identification of the specific impact of HTA recommendations more difficult[29].

Although the first studies on local/hospital HTA were published almost 25 years ago, most of these experiences are recent. The limited evidence currently available on local/hospital HTA makes it difficult to evaluate its effects and impacts at the different levels of health service delivery. Besides, many of these evaluations were conducted internally, by people who are involved in HTA, introducing a potential bias. Further research is necessary to explore the conditions under which local/hospital-based HTA results and recommendations can impact hospital policies, clinical decisions, and quality of patient care and optimize the use of scarce resources. It would be necessary to conduct more independent studies that use high-quality qualitative and quantitative methods for evaluating the impact of HTA in several dimensions. One of the dimensions deserving special attention is the impact of HTA on patients, as they should benefit from optimal resource allocation based on scientific evidence.

Another shortcoming of the studies reviewed is the absence of a theoretical model to evaluate the impact of local/hospital HTA. The only exception is the study by Shumacher and Zeichmester[23] that uses a multidimensional conceptual model with seven impact categories. A framework for evaluating local/hospital HTA should consider its impact at different levels, starting with the uptake of HTA recommendations by decision makers and its effect on funding decisions, its effect on health-care professional practices, and ultimately its impact on health-care outcomes both short and long term. Such evaluation would however require significant resources due to the need to control many confounding factors.

Study limitations

Although it provides a comprehensive synthesis of the effects and impact of local/hospital-based HTA conducted internationally, this review has some limitations. First, given that only published studies have been included in the review, some valuable studies concerning the impact of hospital-based HTA may have been overlooked. For example, we found an abstract about the potential impact of hospital-based HTA in Italy[30], but we were unable to obtain the full study report from the authors. Furthermore, many relevant abstracts are presented each year at the HTAi annual conference, such as the experience of the Hospital Clinic in Catalonia[31]. However, information reported in an abstract is often limited, making it difficult to include this information in our synthesis. However, we have consulted two other reviews that were published as reports and include gray literature such as conference abstracts and research reports[6, 32]. Although those reviews present a few international experiences of local/hospital HTA that are not reported in our review, they do not provide new evidence on the specific topic of our review.

Second, the use of the model proposed by the Hospital Based HTA Interest Sub-Group[1] to classify the different approaches to conducting hospital-based HTA has limitations because it includes distinct features: organizational structure (HTA committee and HTA unit), HTA tool (mini-HTA), and dissemination strategy (ambassador model). Also, the structure and functions of HTA committees varied greatly between studies, and the early HTA experiences reported might not conform to the actual standards in this field. This leads to another limitation of this review due to the great heterogeneity between included studies. In fact, this is not possible to make comparisons between the different local/hospital HTA experiences because they represent distinct interventions. Further work is needed in order to refine the conceptualization of local/hospital-based HTA and provide a common understanding of what it is and how it could be evaluated. A third limitation to this review is the fact that we have not considered study quality in the interpretation of the results. Given that studies published after 2000 generally have higher quality, an update of this review should focus on the most recent literature on local/hospital HTA.

Conclusions

This systematic review provides a basis for understanding how local/hospital-based HTA could impact decision-making regarding the introduction of new technologies in the health-care system. However, our capacity to evaluate the real impacts of local/hospital-based HTA is limited given the relatively small number of evaluations with quantitative data, the lack of clear comparators, and the fact that most evaluations are conducted internally. Further research, using rigorous methods and preferably conducted by external assessors, is necessary to understand the conditions under which local/hospital-based HTA results and recommendations can impact hospital policies, clinical decisions, and quality of care and optimize the use of scarce resources.

Abbreviations

HTA:

Health technology assessment

HTAi:

Health Technology Assessment International

MMAT:

Mixed Methods Appraisal Tool

TAU:

Technology Assessment Unit

CEDIT (French):

Committee for the Assessment and Dissemination of Technological Innovations

MUHC:

McGill University Health Center

CNCP:

Chronic non-cancer pain.

References

  1. 1.

    Cicchetti A, Marchetti M, Dibidino R, Corio M, on behalf of HTAi’s Hospital Based Sub Interest Group: Hospital based HTA. Hospital based health technology assessment. World-wide survey. [http://www.htai.org/fileadmin/HTAi_Files/ISG/HospitalBasedHTA/2008Files/HospitalBasedHTAISGSurveyReport.pdf]

  2. 2.

    Granados A: Health technology assessment and clinical decision making: which is the best evidence?. Int J Technol Assess Health Care. 1999, 15: 585-592.

  3. 3.

    Ehlers L, Vestergaard M, Kidholm K, Bonnevie B, Pedersen PH, Jorgensen T, Jensen MF, Kristensen FB, Kjolby M: Doing mini-health technology assessments in hospitals: a new concept of decision support in health care?. Int J Technol Assess Health Care. 2006, 22: 295-301.

  4. 4.

    Martelli N, Lelong AS, Prognon P, Pineau J: Hospital-based health technology assessment for innovative medical devices in university hospitals and the role of hospital pharmacists: learning from international experience. Int J Technol Assess Health Care. 2013, 29: 185-191. 10.1017/S0266462313000019.

  5. 5.

    Moher D, Liberati A, Tetzlaff J, Altman DG, Group P: Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009, 6: e1000097-10.1371/journal.pmed.1000097.

  6. 6.

    Gagnon M-P, Abdeljelil AB, Desmartis M, Légaré F, Ouimet M, Gagnon J, St-Pierre M, Rhainds M, Coulombe M: Opportunities to Promote Efficiency in Hospital Decision-Making Through the Use of Health Technology Assessment. 2011, Canadian Health Services Research Foundation: Ottawa

  7. 7.

    Proposal: a mixed methods appraisal tool for systematic mixed studies reviews. [http://mixedmethodsappraisaltoolpublic.pbworks.com]. Retrieved 2014, July 2. Archived by WebCite® at http://www.webcitation.org/5tTRTc9yJ

  8. 8.

    Centre for Reviews and Dissemination: Systematic Reviews. CRD’s Guidance for Undertaking Reviews in Health Care. 2009, York: CRD, University of York

  9. 9.

    Menon D, Marshall D: Technology assessment in teaching hospitals. Dimens Health Serv. 1990, 67: 26-28.

  10. 10.

    Cram N, Groves J, Foster L: Technology assessment–a survey of the clinical engineer’s role within the hospital. J Clin Eng. 1997, 22: 373-382. 10.1097/00004669-199711000-00012.

  11. 11.

    Rosenstein AH, O’Daniel M, Geoghan K: Assessing new technology: how are other hospitals facing the challenge?. Healthc Financ Manage. 2003, 57: 70-74.

  12. 12.

    Luce BR, Brown RE: The use of technology assessment by hospitals, health maintenance organizations, and third-party payers in the United States. Int J Technol Assess Health Care. 1995, 11: 79-92. 10.1017/S0266462300005274.

  13. 13.

    Patail BM, Aranha AN: Role of the biomedical engineering department in William Beaumont Hospital’s technology assessment process. J Clin Eng. 1995, 20: 290-296. 10.1097/00004669-199507000-00013.

  14. 14.

    Weingart SN: Deciding to buy expensive technology. The case of biliary lithotripsy. Int J Technol Assess Health Care. 1995, 11: 301-315. 10.1017/S0266462300006917.

  15. 15.

    Mitchell MD, Williams K, Brennan PJ, Umscheid CA: Integrating local data into hospital-based healthcare technology assessment: two case studies. Int J Technol Assess Health Care. 2010, 26: 294-300. 10.1017/S0266462310000334.

  16. 16.

    Veluchamy S, Alder HC: Health care technology assessment and adoption: a case study. Hosp Technol Ser. 1989, 8: 1-12.

  17. 17.

    Saaid H: The impact of health technology assessment on decision-making processes in public versus not-for-profit private hospitals. Am J Med. 2011, 2: 72-78. 10.3844/amjsp.2011.72.78.

  18. 18.

    Poulin P, Austen L, Kortbeek JB, Lafreniere R: New technologies and surgical innovation: five years of a local health technology assessment program in a surgical department. Surg Innov. 2012, 19: 187-199. 10.1177/1553350611421916.

  19. 19.

    McGregor M: The Impact of Reports of the Technology Assessment Unit of the McGill University Health Centre. 2012, Montreal: Technology Assessment Unit (TAU) of the McGill University Health Centre (MUHC)

  20. 20.

    McGregor M, Brophy JM: End-user involvement in health technology assessment (HTA) development: a way to increase impact. Int J Technol Assess Health Care. 2005, 21: 263-267.

  21. 21.

    Bodeau-Livinec F, Simon E, Montagnier-Petrissans C, Joel ME, Fery-Lemonnier E: Impact of CEDIT recommendations: an example of health technology assessment in a hospital network. Int J Technol Assess Health Care. 2006, 22: 161-168.

  22. 22.

    Lee RC, Marshall D, Waddell C, Hailey D, Juzwishin D: Health technology assessment, research, and implementation within a health region in Alberta, Canada. Int J Technol Assess Health Care. 2003, 19: 513-520.

  23. 23.

    Schumacher I, Zechmeister I: Assessing the impact of health technology assessment on the Austrian healthcare system. Int J Technol Assess Health Care. 2013, 29: 84-91. 10.1017/S0266462312000748.

  24. 24.

    Zechmeister I, Schumacher I: The impact of health technology assessment reports on decision making in Austria. Int J Technol Assess Health Care. 2012, 28: 77-84. 10.1017/S0266462311000729.

  25. 25.

    Folkersen J, Pedersen PH: [Attitudes to the use of a decision support method when introducing new medical technology at the University Hospital of Copenhagen]. Ugeskr Laeger. 2006, 168: 2069-2074.

  26. 26.

    Rashiq S, Barton P, Harstall C, Schopflocher D, Taenzer P: The Alberta Ambassador Program: delivering health technology assessment results to rural practitioners. BMC Med Educ. 2006, 6: 21-10.1186/1472-6920-6-21.

  27. 27.

    Jonsson E: History of health technology assessment in Sweden. Int J Technol Assess Health Care. 2009, 25 (Suppl 1): 42-52.

  28. 28.

    Umscheid CA, Williams K, Brennan PJ: Hospital-based comparative effectiveness centers: translating research into practice to improve the quality, safety and value of patient care. J Gen Intern Med. 2010, 25: 1352-1355. 10.1007/s11606-010-1476-9.

  29. 29.

    Jacob R, McGregor M: Assessing the impact of health technology assessment. Int J Technol Assess Health Care. 1997, 13: 68-80. 10.1017/S0266462300010242.

  30. 30.

    Boscolo PR, Ciani O, Torbica A: Hospital-based HTA in Italy: diffusion and potential impact. Value Health. 2012, 15: A283-

  31. 31.

    Morilla-Bachs I, Gutierrez-Moreno S, Bigorra- Llosas J, Sampietro-Colom L: Impact of a hospital based HTA unit. The experience of the Hospital Clinic Barcelona (Spain). 9th HTAi Annual Meeting: 25-27. 2012, June ; Bilbao

  32. 32.

    Bigorra Llosas J, Gomis R, Sampietro-Colom L, Huc M, Lurigados Delgado C, Zamora AG, Carné Cladellas X, Piqué Badia JM: Desarrollo de un sistema de conocimiento compartido para la evaluación en red de la innovación tecnológica en medicina. 2009, Madrid: Plan de Calidad para el Sistema Nacional de Salud. Ministerio de Ciencia e Innovación. Agència d’Avaluació de Tecnologia i Recerca Mèdiques de Cataluña

Download references

Acknowledgements

The realization of this systematic review was made possible with the support of a planning grant from the Canadian Institutes of Health Research (# 201306-KPE-309195). We also acknowledge the contribution of the former Canadian Health Services Foundation for Research for a synthesis grant that provided a basis for this work (# CHS-2419). MPG holds the Tier 2 Canada Research Chair in Technologies and Practices in Health. We are grateful to Ziggi Ivan Santini for the translation of the publication in Danish.

Author information

Correspondence to Marie-Pierre Gagnon.

Additional information

Competing interests

The authors declare that they have no competing interests.

Authors’ contributions

MPG and MD designed the review protocol, conducted full-text reviews for eligibility, extracted data from selected studies, conducted quality assessments, and drafted the manuscript. TP reviewed studies for eligibility, extracted data from selected studies, conducted quality assessments, and revised the manuscript critically. WW carried out the literature searches and reviewed studies for eligibility. All authors commented upon and approved the final version of the manuscript.

Electronic supplementary material

Additional file 1: PRISMA checklist. Description of the items included in the reporting of the systematic review. (DOC 64 KB)

Additional file 2: Electronic search strategy. Description of the search strategy used in PubMed and Embase. (DOCX 13 KB)

Additional file 3: PRISMA flow diagram. Diagram presenting the study selection process. (PDF 91 KB)

Additional file 4: Assessment of the quality of the studies. Table presenting the study quality assessment score. (DOCX 29 KB)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Keywords

  • Health technology assessment
  • Local/hospital-based HTA
  • Impact of HTA activities
  • Hospital budget
  • Perceptions of stakeholders
  • HTA units
  • Internal committee
  • Mini-HTA
  • Ambassador model

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate. Please note that comments may be removed without notice if they are flagged by another user or do not comply with our community guidelines.