- Open Access
- Open Peer Review
The effectiveness of computerized order entry at reducing preventable adverse drug events and medication errors in hospital settings: a systematic review and meta-analysis
Systematic Reviewsvolume 3, Article number: 56 (2014)
The Health Information Technology for Economic and Clinical Health (HITECH) Act subsidizes implementation by hospitals of electronic health records with computerized provider order entry (CPOE), which may reduce patient injuries caused by medication errors (preventable adverse drug events, pADEs). Effects on pADEs have not been rigorously quantified, and effects on medication errors have been variable. The objectives of this analysis were to assess the effectiveness of CPOE at reducing pADEs in hospital-related settings, and examine reasons for heterogeneous effects on medication errors.
Articles were identified using MEDLINE, Cochrane Library, Econlit, web-based databases, and bibliographies of previous systematic reviews (September 2013). Eligible studies compared CPOE with paper-order entry in acute care hospitals, and examined diverse pADEs or medication errors. Studies on children or with limited event-detection methods were excluded. Two investigators extracted data on events and factors potentially associated with effectiveness. We used random effects models to pool data.
Sixteen studies addressing medication errors met pooling criteria; six also addressed pADEs. Thirteen studies used pre-post designs. Compared with paper-order entry, CPOE was associated with half as many pADEs (pooled risk ratio (RR) = 0.47, 95% CI 0.31 to 0.71) and medication errors (RR = 0.46, 95% CI 0.35 to 0.60). Regarding reasons for heterogeneous effects on medication errors, five intervention factors and two contextual factors were sufficiently reported to support subgroup analyses or meta-regression. Differences between commercial versus homegrown systems, presence and sophistication of clinical decision support, hospital-wide versus limited implementation, and US versus non-US studies were not significant, nor was timing of publication. Higher baseline rates of medication errors predicted greater reductions (P < 0.001). Other context and implementation variables were seldom reported.
In hospital-related settings, implementing CPOE is associated with a greater than 50% decline in pADEs, although the studies used weak designs. Decreases in medication errors are similar and robust to variations in important aspects of intervention design and context. This suggests that CPOE implementation, as subsidized under the HITECH Act, may benefit public health. More detailed reporting of the context and process of implementation could shed light on factors associated with greater effectiveness.
The Health Information Technology for Economic and Clinical Health (HITECH) Act of 2009 incentivizes the adoption of health information technology by US hospitals. This Act, part of the American Reinvestment and Recovery Act, allocates up to $29 billion over 10 years for the implementation and 'meaningful use' of electronic health records by hospitals and healthcare providers . Hospitals that satisfy meaningful use criteria can receive millions of dollars. Implementing computerized provider order entry (CPOE) with clinical decision support systems (CDSS) that check for allergies and drug-drug interactions is one of several basic (Stage 1) criteria for meaningful use by hospitals . As of 2008, approximately 9% of general acute care hospitals had at least basic electronic health record (EHR) systems including CPOE for medications. By 2012, 44% had such systems, specifically, 38% of small, 47% of medium, and 62% of large hospitals . Thus, despite the financial incentives, about half of small and medium hospitals and almost 40% of large hospitals had not adopted CPOE with CDSS in the most recent survey.
The primary potential benefit of adopting CPOE is reducing patient injuries caused by medication errors, called preventable adverse drug events (pADEs) [4–6]. Counterbalancing this is concern about unintended adverse consequences [7–9], including increases in medication errors and even mortality, which have been detected in some hospitals after implementation of CPOE [10, 11]. To date, no systematic review has examined net effects on pADEs, the primary outcome of interest for this intervention. Previous reviews have, instead, focused almost exclusively on an intermediate outcome, medication errors. However, not all medication errors pose an equal risk of causing injury. Errors in timing, for example, are generally less risky than giving a medication to the wrong patient. Many commonly used medications, such as anti-hypertensives and antibiotics, have sufficiently long half-lives that receiving a dose an hour or two late has little clinical effect. By contrast, receiving an anti-hypertensive or antibiotic intended for someone else poses risks of low blood pressure or an allergic reaction. In one study at six hospitals, only about 20% of medication errors led to pADEs . Thus, the effect of CPOE on the patient outcome of pADEs is an important clinical and policy question that has remained unanswered, until now.
In addition to focusing on medication errors rather than pADEs, previous systematic reviews have reached conflicting conclusions about the effects of CPOE on medication errors in acute care settings. Some have concluded that CPOE reduces errors, whereas others argue that net effects remain uncertain [4, 5, 13–42]. This controversy stems, in part, from the fact that the association between CPOE implementation and medication errors has exhibited substantial heterogeneity across primary studies . Three basic types of factors could explain such variability: intervention factors, such as differences in how the intervention is designed and implemented; contextual factors, such as differences in patient populations and settings; and methodological factors, such as differences in study design and execution .
Uncertainty about the effects of CPOE on patient outcomes and its variable effects on medication errors may contribute to the reluctance of some hospitals and physicians to adopt CPOE, despite the financial incentives available via HITECH. Consequently, our primary objective in this study was to quantitatively assess the effectiveness of CPOE at reducing pADEs in hospital-related acute care settings. Our secondary objective was to identify factors contributing to variability in effectiveness at reducing medication errors. This analysis is timely as several studies have been published recently and, therefore, were not included in previous reviews and meta-analyses [4, 13, 34, 37, 41], enabling us to examine effects on pADEs and reasons for heterogeneity.
Data sources and searches
First, we developed search strategies for eight databases: MEDLINE; Cochrane Library; Econlit; Campbell Collaboration; the Agency for Healthcare Research and Quality (AHRQ) Health Information Technology Library, Health Information Technology Bibliography, Health Information Technology Costs and Benefits Database Project, and PSNET; Information Service Center for Reviews and Dissemination at the University of York; Evidence for Policy and Practice Information and Coordinating Centre (EPPI-Centre), University of London; Oregon Health Sciences Searchable CPOE Bibliography; and Health Systems Evidence, McMaster University. A number of search terms, such as 'order entry' and 'electronic prescribing' (see Additional file 1), were chosen and strategies developed, in part based on a search strategy published by Eslami et al. .
We used this strategy to search the eight databases for systematic reviews of CPOE or CDSS that might contain potentially relevant primary studies (last updated September 23, 2013) (Figure 1). Next, we used the same strategy to search the eight databases for potentially relevant primary studies that were published after two large previous systematic reviews on CPOE (January 1, 2007 to September 23, 2013) [4, 13]. In addition, we hand-searched nine websites (AHRQ HIT Library, AHRQ PSNET, National Patient Safety Foundation, Joint Commission, Leapfrog Group, Micromedex, Institute for Healthcare Improvement), the Web of Science, and bibliographies of other publications known to us.
We included peer-reviewed studies, regardless of language or design, if they compared CPOE with paper-order entry and examined either of our two primary outcomes, rates of pADEs or medication errors, across a variety of clinical conditions. Eligible settings included adult medical or surgical wards, adult medical or surgical intensive care units (ICUs), emergency departments, or the entire hospital. To reduce unwarranted variability due to contextual and methodological factors, we excluded studies that were from non-hospital settings; that addressed events limited to specific conditions (for example, infections) or types of errors (for example, allergy alerts); or that compared events in highly dissimilar patient care units. As minimum criteria for study quality, we excluded studies that did not describe methods for detecting medication events, or that used incident reporting alone, which detects 0.2–-6% of events . We also excluded pediatric studies because including them would increase heterogeneity: children comprise only 6% of hospitalized patients whereas ADEs disproportionately affect older adults [12, 47, 48].
Two investigators independently screened the article titles and then abstracts for eligibility. We obtained full-text articles when either investigator found the abstract (or title, if the abstract was unavailable) potentially eligible. Disagreements about the eligibility of full-text articles were resolved by consensus, with a third investigator participating for ties.
Data extraction and quality assessment
We defined pADEs as injuries to patients due to medication errors. Medication errors were defined as errors in the process of prescribing, transcribing, dispensing, or administration of a medication, which had the potential to or actually did cause harm. To focus on errors involving relatively higher risk, we excluded, when reported, 'errors' described as having no or almost no potential for harm as well as incomplete or illegible orders, disallowed abbreviations, disallowed drug names, and medications given at the wrong time (see Additional file 1).
Two investigators independently extracted data from each study using a standardized form (see Additional file 1). Disagreements were resolved by consensus, with a third investigator adjudicating ties. Extracted elements included numbers of pADEs and medication errors meeting study definitions, units of exposure to risk of pADEs or medication errors (for example, number of orders, dispensed doses, admissions, or patient days). When studies reported rates or proportions rather than these elements, variance could not be estimated, so the studies could not be included in pooled effect calculations and thus we qualitatively summarized their results instead.
From the studies included in the pooled analysis of medication errors, we extracted several elements related to intervention design and implementation, context, and study methods. Elements related to intervention design included: CPOE developer (homegrown versus commercial); and presence or absence of CDSS, CDSS sophistication (basic, moderate, or advanced; see Table 1 for definitions). When information about the system developer and CDSS were missing from the published article, we contacted the original authors.
Elements related to implementation were based on an AHRQ report addressing context-sensitive patient safety practices, including CPOE. These included: factors influencing the decision to adopt, factors facilitating implementation, and aspects of implementation described in the studies, as well as timing, extent of implementation (limited number of units versus hospital-wide), and whether use was mandatory (see Additional file 1 for details) .
Contextual elements included setting/population (type of clinical unit within the hospital, academic status, public versus private hospital, hospital size, country, primary language in country, payer mix), and baseline proportion of hospitalizations affected by medication errors.
Methodological elements included type of study design, event detection methods, items related to study quality (adapted from relevant reporting criteria in the Standards for Quality Improvement Reporting Excellence; SQUIRE) , and funding source.
Data synthesis and analysis
Using the DerSimonian–Laird random effects model , we conducted meta-analyses for two outcomes (pADEs and medication errors) for all eligible studies combined, and for different subgroups of studies as described below. For each eligible study and outcome measure, we calculated a risk ratio (RR) as the number of events per unit of exposure in the CPOE group divided by events per unit of exposure in the paper-order entry group. Units of exposure varied across studies. If a study provided more than one unit of exposure, we selected the unit most commonly used in the included studies.
Within each meta-analysis, we tested the heterogeneity of the log-transformed RRs using Q and I 2 statistics . Heterogeneity was present when the I 2 statistic was 50% or more and the P-value for the Q statistic was 0.05 or less.
We conducted two sensitivity analyses, removing one study at a time from each meta-analysis to assess the influence of each individual study, and testing whether the choice of units of exposure affected results. To assess publication bias, we examined funnel plots, Begg and Mazumdar’s rank correlation test, and Egger’s regression intercept test .
Intervention design and implementation, contextual, and methodological factors
A priori, we identified nine factors that might be associated with heterogeneity in medication errors across studies. Intervention design factors included type of CPOE developer (homegrown versus commercial), presence or absence of CDSS, and sophistication of CDSS (basic, moderate, or advanced). Intervention implementation factors included scope of implementation (hospital-wide versus limited) and timing of CPOE implementation (year CPOE was implemented or, if missing, the year the study was published). Contextual factors included country (US versus non-US) and baseline proportion of hospitalizations affected by medication errors. Methodological design factors included study design (pre-post versus other designs) and event detection methods (pharmacist order review versus more comprehensive methods). For each discrete factor, we conducted a subgroup analysis when there were at least three studies per subgroup, for example, pre/post design versus other design. For each continuous factor, we conducted a meta-regression using the factor as the sole predictor. In each meta-regression, we pooled log-transformed RRs, and presented the pooled results on the original RR scale.
Pooled meta-analyses were conducted using Comprehensive Meta-analysis, V2 (Biostat, Englewood, NJ, USA); meta-regression analyses were conducted in STATA (V13) (StataCorp LP, College Station, TX, USA).
We screened 4,891 potentially eligible records, including the bibliographies of 32 systematic reviews on CPOE or CDSS [4, 5, 13–42], and then examined 93 full-text articles on CPOE. Of these 93 full-text articles, 74 were excluded: 32 did not test the effectiveness of CPOE, 3 addressed non-hospital settings, 6 addressed pediatric settings, 5 used incident reporting alone to detect events, 1 did not describe event detection methods, 16 addressed outcomes other than medication errors or pADEs (for example, workflow or cost), 5 addressed errors limited to specific conditions, 1 addressed specific types of errors, 2 (1 in French) addressed errors that were excluded because they posed a lower risks of harm, 1 (in Spanish) compared event rates in dissimilar clinical units (obstetric and oncology), and 2 were duplicate publications of articles meeting the selection criteria (Figure 1; see Additional file 1, Additional file 2).
The remaining 19 original articles met the selection criteria and addressed medication errors; 7 of these also addressed pADEs (Table 1) [11, 49, 50, 53–55, 57–68, 70]. Of these 19 studies, 3 omitted the data needed to estimate variance and, therefore, were excluded from pooled effect calculations, resulting in 16 eligible studies, including 6 that addressed pADEs [57, 62, 67].
Of the 16 studies, half were based in the US, including two in community hospitals [11, 55]. Thirteen studies used pre/post designs [11, 49, 50, 53–55, 58–60, 63, 65, 68, 70], two compared similar units within a hospital during the same time period [64, 66], and one compared changes over time between intervention and control units (differences in differences design) . Definitions of medication errors and the methods used to detect them varied across studies (see Additional file 1). Seven studies identified events using data from routine pharmacist review of medication orders [49, 54, 55, 58, 61, 64, 68]. One study provided information on reviewer training , three on blinding of reviewers [11, 59, 64], and none on reliability. The baseline percentage of hospitalizations affected by medication errors ranged from 3.6%  to 99.9% .
Nine studies assessed commercially developed CPOE systems [11, 49, 50, 54, 55, 61, 64, 65, 70], six evaluated homegrown systems [53, 58, 59, 63, 66, 68], and one examined both . No two studies assessed the same commercial system. CDSS was present in twelve studies [11, 55, 58–61, 63–66, 68, 70], and absent in four [49, 50, 53, 54]; we contacted and obtained responses from authors for three of the studies (Table 1).
For all but one study , most of the desired information on implementation was missing (see Additional file 1). Based on the information that was reported, ten studies described the use of CPOE as mandatory at one or more sites [49, 50, 53, 55, 58–61, 63, 64]. CPOE was implemented hospital-wide in four studies [11, 58, 65, 70], in the emergency department in two studies [49, 68], and in a limited number of inpatient units in the rest. Four studies were conducted in complex organizations with facilities in multiple communities [55, 59, 63, 65], another study was in a large hospital with affiliated clinics , and another was in community hospitals . Past experience with information technology was reported in seven studies [49, 50, 55, 58, 59, 63, 65]. Three studies reported that organizational leadership influenced the adoption decision [55, 58, 65], and four stated that staff training and education facilitated implementation [53, 54, 58, 66]. One study mentioned the role of staff time to learn CPOE, a person to lead implementation, extensive project management, an implementation timeline, teamwork, and patient safety culture related to CPOE . Another study described the effects of having a responsible person, local tailoring, and teamwork .
The three studies omitted from the pooled analysis due to lack of variance estimates were similar to the included studies. They were conducted in the US in medium to large hospitals, including one in a community hospital. One study evaluated a commercially developed system ; the other two did not report the developer. Two studies included CDSS [57, 67]. All three used pre/post designs, one detected events using pharmacist review of medication orders , and none reported reviewer training, blinding, or reliability. These studies also did not report implementation context or processes in detail [62, 67], except for one, which discussed financial considerations and leadership .
Primary outcome: preventable adverse drug events
Of the 19 studies, 7 assessed pADEs [11, 59, 60, 62–64, 70]. For the six studies in the pooled analysis, RRs ranged from 0.17 to 0.81. Overall, CPOE was associated with about half as many pADEs as paper-order entry (pooled RR = 0.47, 95% CI 0.31 to 0.71). Studies were heterogeneous (I 2 = 69%) (Figure 2). Serial removal of each study did not substantially influence results (pooled RR range 0.40 to 0.58). There was no evidence of publication bias using a funnel plot, or Begg and Mazumdar’s test (see Additional file 1). For one study excluded from the pooled analysis due to lack of data on variance, we calculated an RR of 0.11 .
Secondary outcome: medication errors
All 19 studies meeting selection criteria assessed medication errors [11, 49, 50, 53–55, 57–68, 70]. Across the 16 studies eligible for the pooled analysis, RRs ranged from 0.16 to 2.08. The pooled estimate showed that medication errors were approximately half as common when providers used CPOE than when they used paper-order entry (pooled RR = 0.46, 95% CI 0.35 to 0.60). The studies were highly heterogeneous (I 2 = 99%) (Figure 3). Results were robust to serial removal of each individual study (pooled RR range 0.42 to 0.49), and to selection of an alternative unit of exposure in the four studies where that was possible (pooled RR = 0.45, 95% CI 0.34 to 0.59). There was no evidence of publication bias using a funnel plot, or Begg and Mazumdar’s test (see Additional file 1).
Two studies included in the pooled analysis reported increases in medication errors after the introduction of CPOE, however, both also reported statistically significant decreases in preventable adverse drug events [11, 70]. A third study, excluded due to lack of data on variance, also showed an increase in errors and a decrease in pADEs, but statistical testing was not performed .
For two studies excluded from the pooled analysis due to lack of data on variance, we calculated RRs of 0.61 , and 1.73, respectively . In the third such study, the authors reported a 50% decline in medication errors (see Additional file 1) .
Intervention design and implementation, contextual, and methodological factors
Six of the a priori subgroup analyses met the requirement to have at least three studies per subgroup and were, therefore, conducted (two were on one variable, CDSS) (Figure 3). Two univariate meta-regression analyses were able to examine whether baseline medication error rate or year of publication (a proxy for maturity of CPOE intervention; date of implementation was frequently missing) predicted effectiveness.
Of five intervention design and implementation factors examined, none reached the conventional level of statistical significance, including type of developer (commercial 0.56 (95% CI 0.36 to 0.85) versus homegrown 0.37 (0.29 to 0.47)), type of CDSS (present 0.44 (0.32 to 0.62) versus absent 0.51 (0.31 to 0.87), and basic 0.40 (0.38 to 0.87) versus moderate or advanced 0.51 (0.26 to 0.97)), and scope of implementation (hospital-wide 0.78 (0.36 to 1.70) versus limited 0.38 (0.32 to 0.46)). Year of publication was not associated with differential effectiveness.
Two contextual factors were evaluated. Studies performed in the US showed greater effectiveness than non-US studies, but this difference was not statistically significant. As the baseline percentage of hospitalizations associated with medication errors increased from 3.6% to 99.9% (data available for 12 studies), the predicted RR of medication errors with CPOE decreased from 1.90 to 0.08 (P < 0.001).
Regarding methodological factors, studies that used pharmacist order review reported greater effectiveness than studies using more comprehensive event detection methods, although this difference was not statistically significant. Almost all studies used pre/post designs so this subgroup analysis was not conducted.
The principal finding of this analysis is that CPOE is associated with a significant reduction in pADEs (hat is, the patient injuries it was designed to prevent) in adult hospital-related acute care settings. Specifically, compared with using paper orders, using CPOE was associated with about half as many pADEs. Medication errors, likewise, were also about half as common with CPOE as with paper-order entry, and the reduction was generally similar across studies with different intervention designs and different implementation, contextual, and methodological characteristics. There were no statistically significant differences in effect between commercial and homegrown systems, with or without CDSS of differing sophistication levels, and between hospital-wide or more limited implementations. The baseline rate of hospitalizations associated with medication errors was significantly associated with effectiveness, as increasing baseline rates of errors were associated with increasing effectiveness. This is expected, because, with few errors, there can be little to change.
Our pooled analysis is conclusive that CPOE is associated with a reduction in pADEs. Shamliyan et al. examined ADEs that might or might not have been related to medication errors, and, therefore, were not as likely to be affected by CPOE. These authors observed significant declines in only three of seven studies (including pediatric ones), and did not perform a pooled analysis .
With regards to the overall pooled result for medication errors, our findings are generally consistent with those of earlier, more limited systematic reviews and meta-analyses [34, 37, 41]. Radley and colleagues also found that medication error rates declined by about half with CPOE implementation (48%, 95% CI 41 to 55%), using a small set of early studies . Van Rosse and colleagues observed greater effectiveness with CPOE than we did (RR of medication errors = 0.08, 95% CI 0.01 41 to 0.76), but examined only three diverse studies . Shamliyan and colleagues found that CPOE was slightly more effective than we did (odds ratio for medication errors = 0.34, 95% CI 0.22 41 to 0.52), based on inpatient and outpatient studies from before 2006 . In comparison to these previous studies, we were able to identify a greater number of relevant articles despite having more restrictive selection criteria (see Additional file 1), enabling us to explore reasons for study heterogeneity.
Also like previous reviews , we observed substantial variability across studies in the effectiveness of CPOE at reducing medication errors. It has long been suspected that variability in the effectiveness of a complex sociotechnical intervention such as CPOE may be related not only to intervention design but also to context and implementation factors [16, 77, 78]. However, across the intervention design and implementation as well as contextual variables that we assessed, we did not see any statistically significant differences in the associations between CPOE use and reductions in medication errors. Two studies of commercial CPOE systems in hospital-wide implementations reported increases in medication errors but reductions in pADEs [11, 70]. One potential explanation for these seemingly contradictory results is that the CPOE systems may have created new medication errors at lower risk for causing ADEs (such as concurrent submission of duplicate orders due to order sets) but reduced medication errors at higher risk of causing ADEs (such as serious drug-drug interactions). Alternatively, CPOE may have made errors easier to detect. The potential to create new types of low-risk medication errors calls attention to the importance of tailoring the CPOE system to the local environment because such errors place a time burden on providers.
This analysis has limitations. We relied on 32 previous systematic reviews to detect primary studies published before 2007. Because each review detected a slightly different set of publications (see Additional file 1), performing our own search of that period would have been unlikely to detect additional studies. We excluded pediatric studies instead of examining population age as a subgroup because these groups differ in their risk for experiencing medication errors and pADEs. Future investigators could evaluate the feasibility of conducting a similar meta-analysis for pediatric populations. We also excluded studies that relied upon incident reporting or did not describe event detection methods, considering these to be minimum criteria for study quality. The number of studies that examined pADEs was not large, but all studies detected declines. Most studies were conducted in academic centers, limiting generalizability to community hospitals. Finally, the included studies all used limited methods, including using pre/post designs and lacking robust data-collection methods.
Implementing CPOE is associated with a greater than 50% decline in pADE rates in hospital-related settings, although results vary. Medication errors decline to a similar degree. Changes in medication errors appear to be consistent across commercial and homegrown systems, with or without clinical decision support, and in individual units or hospital-wide implementations. Many context and implementation variables have, unfortunately, not been reported sufficiently to assess their association with effectiveness. Overall, these findings suggest that the CPOE requirements for meaningful use under the HITECH Act may benefit public health. Knowledge about how to make CPOE more effective would be greatly facilitated by greater reporting of context and implementation details.
Adverse drug event
Preventable adverse drug event
Clinical decision support systems
Computerized provider order entry
Electronic health record
Health Information Technology for Economic and Clinical Health
intensive care unit.
Blumenthal D: Wiring the health system–origins and provisions of a new federal program. N Engl J Med. 2011, 365 (24): 2323-2329.
Classen DC, Bates DW: Finding the meaning in meaningful use. N Engl J Med. 2011, 365 (9): 855-8. doi:10.1056/NEJMsb1103659
Desroches CM, Charles D, Furukawa MF, Joshi MS, Kralovec P, Mostashari F, Worzala C, Jha AK: Adoption of electronic health records grows rapidly, but fewer than half of US hospitals had at least a basic system in 2012. Health Aff (Millwood). 2013, 32 (8): 1478-1485.
Eslami S, de Keizer NF, Abu-Hanna A: The impact of computerized physician medication order entry in hospitalized patients–a systematic review. Int J Med Inform. 2008, 77 (6): 365-376.
Weir CR, Staggers N, Phansalkar S: The state of the evidence for computerized provider order entry: a systematic review and analysis of the quality of the literature. Int J Med Inform. 2009, 78 (6): 365-374.
Bates D, Cullen D, Laird N, Petersen L, Small S, Servi D, Laffel G, Sweitzer B, Shea B, Hallisey R, Vandervliet M, Nemeskal R, Leape LL: Incidence of adverse drug events and potential adverse drug events: implications for prevention: ADE prevention study group. JAMA. 1995, 274 (1): 29-34.
Campbell EM, Sittig DF, Ash JS, Guappone KP, Dykstra RH: Types of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc. 2006, 13 (5): 547-556.
Koppel R, Metlay JP, Cohen A, Abaluck B, Localio AR, Kimmel SE, Strom BL: Role of computerized physician order entry systems in facilitating medication errors. JAMA. 2005, 293 (10): 1197-1203.
Magrabi F, Ong MS, Runciman W, Coiera E: Using FDA reports to inform a classification for health information technology safety problems. J Am Med Inform Assoc. 2012, 19 (1): 45-53.
Han YY, Carcillo JA, Venkataraman ST, Clark RS, Watson RS, Nguyen TC, Bayir H, Orr RA: Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system. Pediatrics. 2005, 116 (6): 1506-1512.
Leung AA, Keohane C, Amato M, Simon SR, Coffey M, Kaufman N, Cadet B, Schiff G, Zimlichman E, Seger DL, Yoon C, Song P, Bates DW: Impact of vendor computerized physician order entry in community hospitals. J Gen Intern Med. 2012, 7: 801-7. doi:10.1007/s11606-012-1987-7. Epub 2012 Jan 21
Hug B, Witkowski D, Sox C, Keohane C, Seger D, Yoon C, Matheny M, Bates D: Adverse drug event rates in six community hospitals and the potential impact of computerized physician order entry for prevention. J Gen Intern Med. 2010, 25 (1): 31-38.
Ammenwerth E, Schnell-Inderst P, Machan C, Siebert U: The effect of electronic prescribing on medication errors and adverse drug events: a systematic review. J Am Med Inform Assoc. 2008, 15 (5): 585-600.
Bright TJ, Wong A, Dhurjati R, Bristow E, Bastian L, Coeytaux RR, Samsa G, Hasselblad V, Williams JW, Musty MD, Wing L, Kendrick AS, Sanders GD, Lobach D: Effect of clinical decision-support systems: a systematic review. Ann Intern Med. 2012, 157 (1): 29-43.
Buntin MB, Burke MF, Hoaglin MC, Blumenthal D: The benefits of health information technology: a review of the recent literature shows predominantly positive results. Health Aff (Millwood). 2011, 30 (3): 464-471.
Chaudhry B, Wang J, Wu S, Maglione M, Mojica W, Roth E, Morton SC, Shekelle PG: Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med. 2006, 144 (10): 742-752.
Colpaert K, Decruyenaere J: Computerized physician order entry in critical care. Best Pract Res Clin Anaesthesiol. 2009, 23 (1): 27-38.
Durieux P, Trinquart L, Colombet I, Nies J, Walton R, Rajeswaran A, Rege Walther M, Harvey E, Burnand B: Computerized advice on drug dosage to improve prescribing practice. Cochrane Database Syst Rev. 2008, 3: CD002894-doi:10.1002/14651858.CD002894.pub2
Garg AX, Adhikari NK, McDonald H, Rosas-Arellano MP, Devereaux PJ, Beyene J, Sam J, Haynes RB: Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA. 2005, 293 (10): 1223-1238.
Goldzweig CL, Towfigh A, Maglione M, Shekelle PG: Costs and benefits of health information technology: new trends from the literature. Health Aff. 2009, 28 (2): w282-w293.
Hemens BJ, Holbrook A, Tonkin M, Mackay JA, Weise-Kelly L, Navarro T, Wilczynski NL, Haynes RB: Computerized clinical decision support systems for drug prescribing and management: a decision-maker-researcher partnership systematic review. Implement Sci. 2011, 6 (1): 89-
Jamal A, McKenzie K, Clark M: The impact of health information technology on the quality of medical and health care: a systematic review. HIM J. 2009, 38 (3): 26-37.
Jones SS, Rudin RS, Perry T, Shekelle PG: Health information technology: an updated systematic review with a focus on meaningful use. Ann Intern Med. In Press
Kaushal R, Shojania KG, Bates DW: Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med. 2003, 163 (12): 1409-1416.
Kawamoto K, Lobach DF: Clinical decision support provided within physician order entry systems: a systematic review of features effective for changing clinician behavior. AMIA Annu Symp Proc. 2003, 361-365.
Kuperman G, Bobb A, Payne T, Avery A, Gandhi T, Burns G, Classen D, Bates D: Medication-related clinical decision support in computerized provider order entry systems: a review. J Am Med Inform Assoc. 2007, 14 (1): 29-40.
Manias E, Williams A, Liew D: Interventions to reduce medication errors in adult intensive care: a systematic review. Br J Clin Pharmacol. 2012, 74 (3): 411-423.
Maslove D, Rizk N, Lowe H: Computerized physician order entry in the critical care environment: a review of current literature. J Am Med Inform Assoc. 2007, 14 (1): 29-40. Epub 2006 Oct 26
McKibbon K, Lokker C, Handler S, Dolovich L, Holbrook A, O’Reilly D, Tamblyn R, Hemens B, Basu R, Troyan S, Roshanov P: The effectiveness of integrated health information technologies across the phases of medication management: a systematic review of randomized controlled trials. J Am Med Inform Assoc. 2012, 19 (1): 22-30.
Mollon B, Chong JJ, Holbrook A, Sung M, Thabane L, Foster G: Features predicting the success of computerized decision support for prescribing: a systematic review of randomized controlled trials. BMC Med Inform Decis Mak. 2009, 9: 11-doi:10.1186/1472-6947-9-11
Niès J, Colombet I, Degoulet P, Durieux P: Determinants of success for computerized clinical decision support systems integrated in CPOE systems: a systematic review. AMIA Annu Symp Proc. 2006, 594-598.
Oren E, Shaffer E, Guglielmo B: Impact of emerging technologies on medication errors and adverse drug events. Am J Health Syst Pharm. 2003, 60 (14): 1447-1458.
Pearson S-A, Moxey A, Robertson J, Hains I, Williamson M, Reeve J, Newby D: Do computerised clinical decision support systems for prescribing change practice? A systematic review of the literature (1990–2007). BMC Health Serv Res. 2009, 9 (1): 154-
Radley DC, Wasserman MR, Olsho LE, Shoemaker SJ, Spranca MD, Bradshaw B: Reduction in medication errors in hospitals due to adoption of computerized provider order entry systems. J Am Med Inform Assoc. 2013, 20 (3): 470-6. doi:10.1136/amiajnl-2012-001241. Epub 2013 Feb 20
Reckmann M, Westbrook J, Koh Y, Lo C, Day R: Does computerized provider order entry reduce prescribing errors for hospital inpatients? A systematic review. J Am Med Inform Assoc. 2009, 16 (5): 613-623.
Schedlbauer A, Prasad V, Mulvaney C, Phansalkar S, Stanton W, Bates DW, Avery AJ: What evidence supports the use of computerized alerts and prompts to improve clinicians’ prescribing behavior?. J Am Med Inform Assoc. 2009, 16 (4): 531-538.
Shamliyan TA, Duval S, Du J, Kane RL: Just what the doctor ordered: review of the evidence of the impact of computerized physician order entry system on medication errors. Health Serv Res. 2008, 43 (1 Pt 1): 32-53.
Shekelle P, Morton SC, Keeler EB: Costs and Benefits of Health Information Technology. 2006, Rockville (MD): Agency for Healthcare Research and Quality (US), (Evidence Reports/Technology Assessments, No. 132.) Available from: http://www.ncbi.nlm.nih.gov/books/NBK37988/
Shojania KG, Jennings A, Mayhew A, Ramsay C, Eccles M, Grimshaw J: Effect of point-of-care computer reminders on physician behaviour: a systematic review. Can Med Assoc J. 2010, 182 (5): E216-E225.
Shojania KG, Jennings A, Mayhew A, Ramsay CR, Eccles MP, Grimshaw J: The effects of on-screen, point of care computer reminders on processes and outcomes of care. Cochrane Database Syst Rev. 2009, 3: CD001096-
van Rosse F, Maat B, Rademaker CMA, van Vught AJ, Egberts ACG, Bollen CW: The effect of computerized physician order entry on medication prescription errors and clinical outcome in pediatric and intensive care: a systematic review. Pediatrics. 2009, 123 (4): 1184-1190.
Wolfstadt JI, Gurwitz JH, Field TS, Lee M, Kalkar S, Wu W, Rochon PA: The effect of computerized physician order entry with clinical decision support on the rates of adverse drug events: a systematic review. J Gen Intern Med. 2008, 23 (4): 451-458.
Deeks JJ, Higgins JPT, Altman DG: Chapter 9: Analysing data and undertaking meta-analyses. Cochrane Handbook for Systematic Reviews of Interventions. Version 5.0.1 [updated. Edited by: Higgins JPT, Green S. 2008, Available from http://hiv.cochrane.org/sites/hiv.cochrane.org/files/uploads/Ch09_Analysing.pdf, last accessed June 1, 2014, September ]. The Cochrane Collaboration, 2008
Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group: Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009, 6 (7): e1000097-
Shea BJ, Hamel C, Wells GA, Bouter LM, Kristjansson E, Grimshaw J, Henry DA, Boers M: AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol. 2009, 62 (10): 1013-1020.
Flynn EA, Barker KN, Pepper GA, Bates DW, Mikeal RL: Comparison of methods for detecting medication errors in 36 hospitals and skilled-nursing facilities. Am J Health Syst Pharm. 2002, 59 (5): 436-446.
Centers for Disease Control and Prevention: National Hospital Discharge Survey: Centers for disease control and prevention: national hospital discharge survey. 2007, cited 2011 July 14]; Available from: http://www.cdc.gov/nchs/nhds/nhds_tables.htm
Kaushal R, Bates DW, Landrigan C, McKenna KJ, Clapp MD, Federico F, Goldmann DA: Medication errors and adverse drug events in pediatric inpatients. JAMA. 2001, 285 (16): 2114-2120.
Assessing the Evidence for Context-Sensitive Effectiveness and Safety: Developing Criteria. 2010, Rockville, MD: Agency for Healthcare Research and Quality,http://www.ahrq.gov/research/findings/final-reports/contextsensitive/index.html,
Franklin BD, Birch S, Savage I, Wong I, Woloshynowych M, Jacklin A, Barber N: Methodological variability in detecting prescribing errors and consequences for the evaluation of interventions. Pharmacoepidemiol Drug Saf. 2009, 18 (11): 992-999.
Donyai P, O’Grady K, Jacklin A, Barber N, Franklin BD: The effects of electronic prescribing on the quality of prescribing. Br J Clin Pharmacol. 2007, 65 (2): 230-237.
Franklin BD, O’Grady K, Donya P, Jacklin A, Barber N: The impact of a closed-loop electronic prescribing and administration system on prescribing errors, administration errors and staff time: a before-and-after study. Qual Saf Health Care. 2007, 16: 279-284.
Shawahna R, Rahman N-U, Ahmad M, Debray M, Yliperttula M, Decleves X: Electronic prescribing reduces prescribing error in public hospitals. J Clin Nurs. 2011, 20 (21–22): 3233-3245.
Shulman R, Singer M, Goldstone J, Bellingan G: Medication errors: a prospective cohort study of hand-written and computerised physician order entry in the intensive care unit. Crit Care. 2005, 9 (5): R516-R521.
Wess ML, Embi PJ, Besier JL, Lowry CH, Anderson PF, Besier JC, Thelen G, Hegner C: Effect of a computerized provider order entry (CPOE) system on medication orders at a community hospital and university hospital. AMIA Annu Symp Proc. 2007, 796-800.
ITN: Imaging technology news: health system gets the ‘LastWord’ with GE’s centricity October 9, 2006 [cited 2013 August 14]. 2006, Available from: http://www.itnonline.com/article/health-system-gets-%E2%80%9Clastword%E2%80%9D-ge%E2%80%99s-centricity, [cited 2013 August 14]
Taylor R, Manzo J, Sinnett M: Quantifying value for physician order-entry systems: a balance of cost and quality. Healthc Financ Manage. 2002, 56 (7): 44-48.
Barron W, Reed R, Forsythe S, Hecht D, Glen J, Murphy B, Lach R, Flores S, Tu J, Concklin M: Implementing computerized provider order entry with an existing clinical information system. Jt Comm J Qual Patient Saf. 2006, 32 (9): 506-516.
Bates DW, Leape LL, Cullen DJ, Laird N, Petersen LA, Teich JM, Burdick E, Hickey M, Kleefield S, Shea B, Vander Vliet M, Seger DL: Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. JAMA. 1998, 280 (15): 1311-1316.
van Doormaal JE, van den Bemt PMLA, Zaal RJ, Egberts ACG, Lenderink BW, Kosterink JGW, Haaijer-Ruskamp FM, Mol PGM: The influence that electronic prescribing has on medication errors and preventable adverse drug events: an interrupted time-series study. J Am Med Inform Assoc. 2009, 16 (6): 816-825.
Westbrook J, Reckmann M, Li L, Runciman W, Burke R, Lo C, Baysari M, Braithwaite J, Day R: Effects of two commercial electronic prescribing systems on prescribing error rates in hospital in-patients: a before and after study. PLoS Med. 2012, 9 (1): e1001164-
Weant KA, Cook AM, Armitstead JA: Medication-error reporting and pharmacy resident experience during implementation of computerized prescriber order entry. Am J Health Syst Pharm. 2007, 64 (5): 526-530.
Bates DW, Teich JM, Lee J, Seger D, Kuperman GJ, Ma’luf N, Boyle D, Leape L: The impact of computerized physician order entry on medication error prevention. JAMIA. 1999, 6 (4): 313-321.
Colpaert K, Claus B, Somers A, Vandewoude K, Robays H, Decruyenaere J: Impact of computerized physician order entry on medication prescription errors in the intensive care unit: a controlled cross-sectional trial. Crit Care. 2006, 10 (1): R21-
Mahoney CD, Berard-Collins CM, Coleman R, Amaral JF, Cotter CM: Effects of an integrated clinical information system on medication safety in a multi-hospital setting. Am J Health Syst Pharm. 2007, 64 (18): 1969-1977.
Oliven A, Michalake I, Zalman D, Dorman E, Yeshurun D, Odeh M: Prevention of prescription errors by computerized, on-line surveillance of drug order entry. Int J Med Inform. 2005, 74: 377-386.
Igboechi CA, Ng CL, Yang CS, Buckner AN: Impact of computerized prescriber order entry on medication errors at an acute tertiary care hospital. Hosp Pharm. 2003, 38 (3): 227-231.
Aronsky D, Johnston PE, Jenkins G, Waitman LR, Frelix D, Jones I, Patel NR: The effect of implementing computerized provider order entry on medication prescribing errors in an emergency department. AMIA Annu Symp Proc. 2007
Miller RA, Waitman LR, Chen S, Rosenbloom ST: The anatomy of decision support during inpatient care provider order entry (CPOE): empirical observations from a decade of CPOE experience at Vanderbilt. J Biomed Inform. 2005, 38 (6): 469-485.
Menendez MD, Alonso J, Rancaño I, Corte JJ, Herranz V, Vázquez F: Impact of computerized physician order entry on medication errors. 2012, Elsevier Editorial System(tm) for Revista de Calidad Asistencial, 16-
Wright A, Sittig DF, Ash JS, Sharma S, Pang JE, Middleton B: Clinical decision support capabilities of commercially-available clinical information systems. J Am Med Inform Assoc. 2009, 16 (5): 637-644.
Shekelle PG, Pronovost PJ, Wachter RM: Rockville, MD: Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services. 540 Gaither Road, Rockville, MD 208502010-Assessing the evidence for context-sensitive effectiveness and safety of patient safety practices: developing criteria, http://www.ahrq.gov contract no. HHSA-290-2009-10001C,
Ogrinc G, Mooney S, Estrada C, Foster T, Goldmann D, Hall L, Huizinga M, Liu S, Mills P, Neily J, Nelson W, Pronovos PJ, Provost L, Rubenstein L, Speroff T, Splaine M, Thomson R, Tomolo A, Watts B: The SQUIRE (Standards for QUality improvement reporting excellence) guidelines for quality improvement reporting: explanation and elaboration. Qual Saf Health Care. 2008, 17 (1): i13-32.
DerSimonian R, Laird N: Meta-analysis in clinical trials. Control Clin Trials. 1986, 7 (3): 177-188.
Higgins JP, Thompson SG, Deeks JJ, Altman DG: Measuring inconsistency in meta-analyses. BMJ. 2003, 327 (7414): 557-560.
Sutton AJ, Duval SJ, Tweedie RL, Abrams KR, Jones DR: Empirical assessment of effect of publication bias on meta-analyses. BMJ. 2000, 320 (7249): 1574-1577.
Shekelle PG, Pronovost PJ, Wachter RM, Taylor SL, Dy SM, Foy R, Hempel S, McDonald KM, Ovretveit J, Rubenstein LV, Adams AS, Angood PB, Bates DW, Bickman L, Carayon P, Donaldson L, Duan N, Farley DO, Greenhalgh T, Haughom J, Lake ET, Lilford R, Lohr KN, Meyer GS, Miller MR, Neuhauser DV, Ryan G, Saint S, Shojania KG, Shortell SM, Stevens DP, Walshe K: Advancing the science of patient safety. Ann Intern Med. 2011, 154 (10): 693-696.
Wears RL, Berg M: Computer technology and clinical work: still waiting for Godot. JAMA. 2005, 293 (10): 1261-1263.
This work was funded through a Mentored Clinical Scientist Career Development Award (K08) from the Agency for Healthcare Research and Quality (to TKN; grant number HS17954). The funder played no role in the design and conduct of the study; collection, management, analysis, or interpretation of the data; preparation, review, or approval of the manuscript; or decision to submit the manuscript for publication. There were no other funding sources for this work. The assistance of Lance Tan in preparing the manuscript is greatly appreciated.
The authors have no conflicts of interest with the work.
TKN, CSS, PGS: conception and design, data collection and analysis, manuscript writing; SCM data analysis and manuscript writing; SMA conception and design; VMP, LJA, and ELD: data collection and analysis. All authors read and approved the final manuscript.