The Health Information Technology for Economic and Clinical Health (HITECH) Act of 2009 incentivizes the adoption of health information technology by US hospitals. This Act, part of the American Reinvestment and Recovery Act, allocates up to $29 billion over 10 years for the implementation and 'meaningful use' of electronic health records by hospitals and healthcare providers [1]. Hospitals that satisfy meaningful use criteria can receive millions of dollars. Implementing computerized provider order entry (CPOE) with clinical decision support systems (CDSS) that check for allergies and drug-drug interactions is one of several basic (Stage 1) criteria for meaningful use by hospitals [2]. As of 2008, approximately 9% of general acute care hospitals had at least basic electronic health record (EHR) systems including CPOE for medications. By 2012, 44% had such systems, specifically, 38% of small, 47% of medium, and 62% of large hospitals [3]. Thus, despite the financial incentives, about half of small and medium hospitals and almost 40% of large hospitals had not adopted CPOE with CDSS in the most recent survey.

The primary potential benefit of adopting CPOE is reducing patient injuries caused by medication errors, called preventable adverse drug events (pADEs) [4–6]. Counterbalancing this is concern about unintended adverse consequences [7–9], including increases in medication errors and even mortality, which have been detected in some hospitals after implementation of CPOE [10, 11]. To date, no systematic review has examined net effects on pADEs, the primary outcome of interest for this intervention. Previous reviews have, instead, focused almost exclusively on an intermediate outcome, medication errors. However, not all medication errors pose an equal risk of causing injury. Errors in timing, for example, are generally less risky than giving a medication to the wrong patient. Many commonly used medications, such as anti-hypertensives and antibiotics, have sufficiently long half-lives that receiving a dose an hour or two late has little clinical effect. By contrast, receiving an anti-hypertensive or antibiotic intended for someone else poses risks of low blood pressure or an allergic reaction. In one study at six hospitals, only about 20% of medication errors led to pADEs [12]. Thus, the effect of CPOE on the patient outcome of pADEs is an important clinical and policy question that has remained unanswered, until now.

In addition to focusing on medication errors rather than pADEs, previous systematic reviews have reached conflicting conclusions about the effects of CPOE on medication errors in acute care settings. Some have concluded that CPOE reduces errors, whereas others argue that net effects remain uncertain [4, 5, 13–42]. This controversy stems, in part, from the fact that the association between CPOE implementation and medication errors has exhibited substantial heterogeneity across primary studies [37]. Three basic types of factors could explain such variability: intervention factors, such as differences in how the intervention is designed and implemented; contextual factors, such as differences in patient populations and settings; and methodological factors, such as differences in study design and execution [43].

Uncertainty about the effects of CPOE on patient outcomes and its variable effects on medication errors may contribute to the reluctance of some hospitals and physicians to adopt CPOE, despite the financial incentives available via HITECH. Consequently, our primary objective in this study was to quantitatively assess the effectiveness of CPOE at reducing pADEs in hospital-related acute care settings. Our secondary objective was to identify factors contributing to variability in effectiveness at reducing medication errors. This analysis is timely as several studies have been published recently and, therefore, were not included in previous reviews and meta-analyses [4, 13, 34, 37, 41], enabling us to examine effects on pADEs and reasons for heterogeneity.

We adhered to recommendations in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Statement [44, 45], including developing the protocol before undertaking the analysis.

Data sources and searches

First, we developed search strategies for eight databases: MEDLINE; Cochrane Library; Econlit; Campbell Collaboration; the Agency for Healthcare Research and Quality (AHRQ) Health Information Technology Library, Health Information Technology Bibliography, Health Information Technology Costs and Benefits Database Project, and PSNET; Information Service Center for Reviews and Dissemination at the University of York; Evidence for Policy and Practice Information and Coordinating Centre (EPPI-Centre), University of London; Oregon Health Sciences Searchable CPOE Bibliography; and Health Systems Evidence, McMaster University. A number of search terms, such as 'order entry' and 'electronic prescribing' (see Additional file 1), were chosen and strategies developed, in part based on a search strategy published by Eslami et al. [4].

We used this strategy to search the eight databases for systematic reviews of CPOE or CDSS that might contain potentially relevant primary studies (last updated September 23, 2013) (Figure 1). Next, we used the same strategy to search the eight databases for potentially relevant primary studies that were published after two large previous systematic reviews on CPOE (January 1, 2007 to September 23, 2013) [4, 13]. In addition, we hand-searched nine websites (AHRQ HIT Library, AHRQ PSNET, National Patient Safety Foundation, Joint Commission, Leapfrog Group, Micromedex, Institute for Healthcare Improvement), the Web of Science, and bibliographies of other publications known to us.

We screened 4,891 potentially eligible records, including the bibliographies of 32 systematic reviews on CPOE or CDSS [4, 5, 13–42], and then examined 93 full-text articles on CPOE. Of these 93 full-text articles, 74 were excluded: 32 did not test the effectiveness of CPOE, 3 addressed non-hospital settings, 6 addressed pediatric settings, 5 used incident reporting alone to detect events, 1 did not describe event detection methods, 16 addressed outcomes other than medication errors or pADEs (for example, workflow or cost), 5 addressed errors limited to specific conditions, 1 addressed specific types of errors, 2 (1 in French) addressed errors that were excluded because they posed a lower risks of harm, 1 (in Spanish) compared event rates in dissimilar clinical units (obstetric and oncology), and 2 were duplicate publications of articles meeting the selection criteria (Figure 1; see Additional file 1, Additional file 2).

The remaining 19 original articles met the selection criteria and addressed medication errors; 7 of these also addressed pADEs (Table 1) [11, 49, 50, 53–55, 57–68, 70]. Of these 19 studies, 3 omitted the data needed to estimate variance and, therefore, were excluded from pooled effect calculations, resulting in 16 eligible studies, including 6 that addressed pADEs [57, 62, 67].

Of the 16 studies, half were based in the US, including two in community hospitals [11, 55]. Thirteen studies used pre/post designs [11, 49, 50, 53–55, 58–60, 63, 65, 68, 70], two compared similar units within a hospital during the same time period [64, 66], and one compared changes over time between intervention and control units (differences in differences design) [61]. Definitions of medication errors and the methods used to detect them varied across studies (see Additional file 1). Seven studies identified events using data from routine pharmacist review of medication orders [49, 54, 55, 58, 61, 64, 68]. One study provided information on reviewer training [11], three on blinding of reviewers [11, 59, 64], and none on reliability. The baseline percentage of hospitalizations affected by medication errors ranged from 3.6% [49] to 99.9% [60].

Nine studies assessed commercially developed CPOE systems [11, 49, 50, 54, 55, 61, 64, 65, 70], six evaluated homegrown systems [53, 58, 59, 63, 66, 68], and one examined both [60]. No two studies assessed the same commercial system. CDSS was present in twelve studies [11, 55, 58–61, 63–66, 68, 70], and absent in four [49, 50, 53, 54]; we contacted and obtained responses from authors for three of the studies (Table 1).

For all but one study [58], most of the desired information on implementation was missing (see Additional file 1). Based on the information that was reported, ten studies described the use of CPOE as mandatory at one or more sites [49, 50, 53, 55, 58–61, 63, 64]. CPOE was implemented hospital-wide in four studies [11, 58, 65, 70], in the emergency department in two studies [49, 68], and in a limited number of inpatient units in the rest. Four studies were conducted in complex organizations with facilities in multiple communities [55, 59, 63, 65], another study was in a large hospital with affiliated clinics [49], and another was in community hospitals [11]. Past experience with information technology was reported in seven studies [49, 50, 55, 58, 59, 63, 65]. Three studies reported that organizational leadership influenced the adoption decision [55, 58, 65], and four stated that staff training and education facilitated implementation [53, 54, 58, 66]. One study mentioned the role of staff time to learn CPOE, a person to lead implementation, extensive project management, an implementation timeline, teamwork, and patient safety culture related to CPOE [58]. Another study described the effects of having a responsible person, local tailoring, and teamwork [65].

The three studies omitted from the pooled analysis due to lack of variance estimates were similar to the included studies. They were conducted in the US in medium to large hospitals, including one in a community hospital. One study evaluated a commercially developed system [67]; the other two did not report the developer. Two studies included CDSS [57, 67]. All three used pre/post designs, one detected events using pharmacist review of medication orders [67], and none reported reviewer training, blinding, or reliability. These studies also did not report implementation context or processes in detail [62, 67], except for one, which discussed financial considerations and leadership [57].

Primary outcome: preventable adverse drug events

Of the 19 studies, 7 assessed pADEs [11, 59, 60, 62–64, 70]. For the six studies in the pooled analysis, RRs ranged from 0.17 to 0.81. Overall, CPOE was associated with about half as many pADEs as paper-order entry (pooled RR = 0.47, 95% CI 0.31 to 0.71). Studies were heterogeneous (I ² = 69%) (Figure 2). Serial removal of each study did not substantially influence results (pooled RR range 0.40 to 0.58). There was no evidence of publication bias using a funnel plot, or Begg and Mazumdar’s test (see Additional file 1). For one study excluded from the pooled analysis due to lack of data on variance, we calculated an RR of 0.11 [62].

Secondary outcome: medication errors

All 19 studies meeting selection criteria assessed medication errors [11, 49, 50, 53–55, 57–68, 70]. Across the 16 studies eligible for the pooled analysis, RRs ranged from 0.16 to 2.08. The pooled estimate showed that medication errors were approximately half as common when providers used CPOE than when they used paper-order entry (pooled RR = 0.46, 95% CI 0.35 to 0.60). The studies were highly heterogeneous (I ² = 99%) (Figure 3). Results were robust to serial removal of each individual study (pooled RR range 0.42 to 0.49), and to selection of an alternative unit of exposure in the four studies where that was possible (pooled RR = 0.45, 95% CI 0.34 to 0.59). There was no evidence of publication bias using a funnel plot, or Begg and Mazumdar’s test (see Additional file 1).

Two studies included in the pooled analysis reported increases in medication errors after the introduction of CPOE, however, both also reported statistically significant decreases in preventable adverse drug events [11, 70]. A third study, excluded due to lack of data on variance, also showed an increase in errors and a decrease in pADEs, but statistical testing was not performed [62].

For two studies excluded from the pooled analysis due to lack of data on variance, we calculated RRs of 0.61 [67], and 1.73, respectively [62]. In the third such study, the authors reported a 50% decline in medication errors (see Additional file 1) [57].

The principal finding of this analysis is that CPOE is associated with a significant reduction in pADEs (hat is, the patient injuries it was designed to prevent) in adult hospital-related acute care settings. Specifically, compared with using paper orders, using CPOE was associated with about half as many pADEs. Medication errors, likewise, were also about half as common with CPOE as with paper-order entry, and the reduction was generally similar across studies with different intervention designs and different implementation, contextual, and methodological characteristics. There were no statistically significant differences in effect between commercial and homegrown systems, with or without CDSS of differing sophistication levels, and between hospital-wide or more limited implementations. The baseline rate of hospitalizations associated with medication errors was significantly associated with effectiveness, as increasing baseline rates of errors were associated with increasing effectiveness. This is expected, because, with few errors, there can be little to change.

Our pooled analysis is conclusive that CPOE is associated with a reduction in pADEs. Shamliyan et al. examined ADEs that might or might not have been related to medication errors, and, therefore, were not as likely to be affected by CPOE. These authors observed significant declines in only three of seven studies (including pediatric ones), and did not perform a pooled analysis [37].

With regards to the overall pooled result for medication errors, our findings are generally consistent with those of earlier, more limited systematic reviews and meta-analyses [34, 37, 41]. Radley and colleagues also found that medication error rates declined by about half with CPOE implementation (48%, 95% CI 41 to 55%), using a small set of early studies [34]. Van Rosse and colleagues observed greater effectiveness with CPOE than we did (RR of medication errors = 0.08, 95% CI 0.01 41 to 0.76), but examined only three diverse studies [41]. Shamliyan and colleagues found that CPOE was slightly more effective than we did (odds ratio for medication errors = 0.34, 95% CI 0.22 41 to 0.52), based on inpatient and outpatient studies from before 2006 [37]. In comparison to these previous studies, we were able to identify a greater number of relevant articles despite having more restrictive selection criteria (see Additional file 1), enabling us to explore reasons for study heterogeneity.

Also like previous reviews [37], we observed substantial variability across studies in the effectiveness of CPOE at reducing medication errors. It has long been suspected that variability in the effectiveness of a complex sociotechnical intervention such as CPOE may be related not only to intervention design but also to context and implementation factors [16, 77, 78]. However, across the intervention design and implementation as well as contextual variables that we assessed, we did not see any statistically significant differences in the associations between CPOE use and reductions in medication errors. Two studies of commercial CPOE systems in hospital-wide implementations reported increases in medication errors but reductions in pADEs [11, 70]. One potential explanation for these seemingly contradictory results is that the CPOE systems may have created new medication errors at lower risk for causing ADEs (such as concurrent submission of duplicate orders due to order sets) but reduced medication errors at higher risk of causing ADEs (such as serious drug-drug interactions). Alternatively, CPOE may have made errors easier to detect. The potential to create new types of low-risk medication errors calls attention to the importance of tailoring the CPOE system to the local environment because such errors place a time burden on providers.

This analysis has limitations. We relied on 32 previous systematic reviews to detect primary studies published before 2007. Because each review detected a slightly different set of publications (see Additional file 1), performing our own search of that period would have been unlikely to detect additional studies. We excluded pediatric studies instead of examining population age as a subgroup because these groups differ in their risk for experiencing medication errors and pADEs. Future investigators could evaluate the feasibility of conducting a similar meta-analysis for pediatric populations. We also excluded studies that relied upon incident reporting or did not describe event detection methods, considering these to be minimum criteria for study quality. The number of studies that examined pADEs was not large, but all studies detected declines. Most studies were conducted in academic centers, limiting generalizability to community hospitals. Finally, the included studies all used limited methods, including using pre/post designs and lacking robust data-collection methods.

Implementing CPOE is associated with a greater than 50% decline in pADE rates in hospital-related settings, although results vary. Medication errors decline to a similar degree. Changes in medication errors appear to be consistent across commercial and homegrown systems, with or without clinical decision support, and in individual units or hospital-wide implementations. Many context and implementation variables have, unfortunately, not been reported sufficiently to assess their association with effectiveness. Overall, these findings suggest that the CPOE requirements for meaningful use under the HITECH Act may benefit public health. Knowledge about how to make CPOE more effective would be greatly facilitated by greater reporting of context and implementation details.