The ability of comorbidity indices to predict mortality in an orthopedic setting: a systematic review
Systematic Reviews volume 10, Article number: 234 (2021)
Several comorbidity indices have been created to estimate and adjust for the burden of comorbidity. The objective of this systematic review was to evaluate and compare the ability of different comorbidity indices to predict mortality in an orthopedic setting.
A systematic search was conducted in Embase, MEDLINE, and Cochrane Library. The search were constructed around two primary focal points: a comorbidity index and orthopedics. The last search were performed on 13 June 2019. Eligibility criteria were participants with orthopedic conditions or who underwent an orthopedic procedure, a comparison between comorbidity indices that used administrative data, and reported mortality as outcome. Two independent reviewers screened the studies using Covidence. The area under the curve (AUC) was chosen as the primary effect estimate.
Of the 5338 studies identified, 16 met the eligibility criteria. The predictive ability of the different comorbidity indices ranged from poor (AUC < 0.70) to excellent (AUC ≥ 0.90). The majority of the included studies compared the Elixhauser Comorbidity Index (ECI) and the Charlson Comorbidity Index (CCI). In-hospital mortality was reported in eight studies reporting AUC values ranging from 0.70 to 0.92 for ECI and 0.68 to 0.89 for CCI. AUC values were generally lower for all other time points ranging from 0.67 to 0.78. For 1-year mortality the overall effect size ranging from 0.67 to 0.77 for ECI and 0.69 to 0.77 for CCI.
The results of this review indicate that the ECI and CCI can equally be used to adjust for comorbidities when analyzing mortality in an orthopedic setting.
The protocol for this systematic review was registered on PROSPERO, the International Prospective Register of Systematic Reviews on 13 June 2019 and can be accessed through record ID 133,871.
Research that uses data from registers and administrative databases often must deal with several potential sources of confounding factors, one of which is comorbidity. Comorbidity is all additional conditions, which have an effect on the patients’ conditions concomitant with the primary orthopedic condition. Therefore, taking into account and adjusting for comorbidity is often recommended when conducting research on mortality . In an orthopedic setting, comorbidity is often relevant as it is associated with an increased risk of complications, re-operations, and mortality.
Adjustment for all known comorbidity variables is often not methodologically possible. Consequently, several comorbidity indices have been created to measure and adjust for the estimated burden of comorbidity. The Charlson Comorbidity Index (CCI) and the Elixhauser Comorbidity Index (ECI) are the most commonly used risk predictor tools . CCI was developed and validated in 1987  to predict risk of death within 1 year of hospitalization and since then different versions that uses administrative data have been created . CCI was first adapted to ICD-codes by Deyo in 1992 and Romani in 1993 [5, 6]. Today, the most commonly used is the updated version by Quan et al.  that adapted CCI to be used with ICD-10 codes. The ECI was specifically developed using administrative data in 1998  and with the CCI, Quan et al. also made an adaption of the ECI to the ICD-10 code.
The contents of these indices differ and have had numerous adaptations. Consequently, it has become difficult to identify which index to select when performing a risk assessment or when adjusting for comorbidity.
Previous systematic reviews on this topic have found several factors to be essential when selecting a comorbidity index. Yourkovich et al.  and Sharabiani et al.  concluded that the predictive ability of a comorbidity index primarily depends on the outcome and study population. However, no systematic review has investigated the validity of comorbidity indices in an orthopedic setting. The aim of this paper was to conduct a systematic review comparing the ability of different comorbidity indices to predict mortality in an orthopedic setting.
This review is reported in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines (PRISMA) and the PRISMA checklist can be found in Additional file 1 . The protocol was registered in the International Prospective Register of Systematic Reviews (PROSPERO) on 13 June 2019 and can be accessed through record ID 133,871.
This systematic review included studies that compared the performance of different comorbidity indices in predicting mortality in an orthopedic setting.
Studies were included if they:
Included participants with orthopedic conditions or participants who underwent an orthopedic procedure.
Included a comparison of one comorbidity index with another comorbidity index.
Included comorbidity indices that were based on administrative data (registers and databases).
Reported mortality as an outcome.
Studies were excluded if they only included comorbidity indices that were validated only for specific diseases (e.g., the Cardiac-specific Comorbidity Index ). We did not have exclusion criteria in form of a minimum age limit nor for whether the study participants was in- or outpatients.
Primary outcome measures were defined as in-hospital, 30-day, 90-day, and 1-year mortality. Secondary outcome measures were not prespecified.
The language of included studies was restricted to articles written in English. No publication year, publication status, or study design limitations were imposed.
We searched the following databases: Ovid Embase Classic + Embase (1947–present), Ovid MEDLINE (R) all (1946–present), and the Cochrane Library. The last search was carried out on 2 May 2019. The reference list of included studies was screened by PHG and MJ for additional eligible studies. Grey literature was also searched through OpenGrey , the International Clinical Trial Registry Platform , and Orthopaedic Proceedings .
The search strategy was developed by the author team with assistance from librarians at the University of Southern Denmark and was primarily developed for Embase; thereafter, it was adapted for the other databases. The searches were constructed around two primary focal points: a comorbidity index and orthopedics. No language or publication year restrictions were used in the search. The following search terms were used to search Embase:
comorbidity ind*; comorbidity scor*; comorbidity scale*; exp Charlson Comorbidity Index; Charlson* adj2 (ind* or scale* or score*); Elixhauser*; Chronic Disease Score*; Rxrisk; Rx-risk; Rxrisk-v; Rx-risk-v; Medication-Based Disease Burden Index; exp orthopedics; orthop?edic*; exp orthopedic surgery; exp arthroplasty; arthroplast*; hemi-arthroplast*; hemiarthroplast*; exp osteoarthritis; osteoarthritis; exp arthroscopy; arthroscop*; exp fracture; fracture*; exp traumatology; traumatolog*; exp hip surgery; exp knee surgery; exp shoulder surgery; hip* or knee* or shoulder* adj3 (surg*).
A full description of the search strategies for Embase, MEDLINE, and the Cochrane Library can be found in the Additional file 2.
Eligibility assessment of the studies was performed independently in a blinded standardized manner by PHG and MJ using Covidence . The two authors assessed the 5338 articles by screening title and subsequent reading the abstract of the studies and classified the studies as “eligible,” “not eligible,” or “perhaps eligible.” Following the independent assessment, the two authors then compared result. Disagreements between PHG and MJ were resolved by consensus and by consulting with the other authors.
Data collection process
The data were extracted using a prespecified data collection form in Microsoft Excel.
Authors of three studies [16,17,18] were contacted for further information on the 95% confidence intervals (CIs). Bulöw et al.  and Kurichi et al.  responded and provided additional numerical data.
The following information was extracted from each included study: (1) characteristics of participants (orthopedic diagnosis or surgical procedure); (2) type of comorbidity index; (3) relevant outcome measures (mortality); (4) database/register used; (5) data period; (6) country; (7) number of patients; (8) study design; and (9) relevant summary effect estimate.
Risk of bias in individual studies
The risk of bias for each individual study was assessed by PHG and MJ using Risk Of Bias In Non-Randomized Studies of Interventions (ROBINS-I) .
No summary effect measure was prespecified in the protocol. All 16 included studies reported the area under the curve (AUC) of receiver operating characteristic curve. Therefore, AUC was chosen as the primary effect estimate in this review; the AUC measures the predictive performance of a comorbidity index. The AUC value ranges from 0.5 to 1.0, where 0.5 indicates no predictive ability and 1.0 indicates perfect predictive ability. In general, an inexact guide can be used: 0.50: no discriminating ability, 0.60–0.69: poor accuracy, 0.70–0.79: fair accuracy, 0.80–0.89: good accuracy, and 0.90–1.00: excellent accuracy . Confidence level of 95% is used when reporting AUC values.
We originally intended to do a meta-analysis on the studies that compared CCI with ECI. However, due to the variation in the different versions and models of the comorbidity indices, differences in reported outcomes, and validity issues concerning the use of meta-analysis of predictive models  we omitted the meta-analysis and instead did a more narrative report of the studies.
The searches of the MEDLINE, Embase, and Cochrane Library databases provided a total of 6333 citations (Fig. 1). After adjusting for duplicates using Endnote and Covidence, 5338 citations remained. Of these, 5300 studies were discarded after reviewing the titles and abstracts (Fig. 1).
The full text of the remaining 38 citations was assessed. Sixteen studies met the inclusion criteria and were included in the systematic review (Table 1). No further studies were identified by screening the references of the included articles or searching the grey literature.
All of the studies were cohort studies (Table 1). All but one of the included studies  used various types of registers to gather the administrative data, with the National Hospital Discharge Survey being the most commonly used database. The data periods varied from 1990 to 2014. Nine of the studies were carried out in the USA. The number of patients included in each study ranged from 506 to 33,194,141.
Eight of the included studies compared only the ECI with the CCI [16, 18, 23,24,25,26,27,28,29]. In addition to these two indices, six other indices were also assessed: a Multipurpose Australian Comorbidity Scoring System, the Cumulative Illness Rating Scale, the Preoperative Score to Predict Postoperative Mortality, the Iezzoni Index, and the Clinical Classifications Software and Modified Frailty Index.
Regarding the outcome measures, in-hospital mortality was the most commonly reported outcome, followed by 1-year and 30-day mortality.
Risk of bias within studies
Most studies were evaluated as having a moderate risk of bias. Two studies were evaluated to have serious risk of bias due to either confounding or bias in reported results (Table 2).
Results of individual studies
The 16 included studies varied markedly regarding the type and version of comorbidity indices assessed, and the follow-up time, which ranged from in-hospital to 14 years (Table 1). In eight studies (Bülow et al. 2019 , Bülow et al. 2017 , Menendez et al. 2014 , Menendez et al. 2015 , Menendez et al. 2015 , Ondeck et al. 2008 , Radley et al. 2008 , and Toson et al. 2016 [33), the comorbidity indices was compared to a base model, comprising of demographic variables (e.g., age and sex) and in two of those, the base model outperformed both CCI and ECI.
In nine studies, the comorbidity indices were combined with demographic variables (e.g., age and sex). In all nine studies, this combination increased the predictive power.
Comparison of the Elixhauser and Charlson Comorbidity Indices
The CCI and ECI were analyzed and compared in 12 studies (Table 3).
Eight of these studies investigated the in-hospital mortality [24, 25, 27,28,29,30,33]. The AUC values of the ECI ranged from 0.78 to 0.92 when focusing on adjusted values only, reflecting fair to excellent predictive ability. The CCI adjusted values were similar, ranging from 0.78 to 0.89.
In addition to the CCI and ECI, six other indices were identified. None of these six indices were assessed in more than one study. Most of the other indices were compared with the CCI and/or ECI, and in general they performed equally good or better, with the exception of the RxRisk-V and the Modified Frailty Index, which had AUC values that were slightly lower than those of the CCI and ECI [17, 31].
The Multipurpose Australian Comorbidity Scoring System demonstrated good discriminating ability in predicting in-hospital (AUC 0.81 (0.80–0.82)) and 30-day mortality (AUC 0.80 (CI 0.79–0.81)), but only fair prediction of 1-year mortality (AUC 0.77 (CI 0.76–0.77)) .
It is also worth recognizing that two studies assessed more inclusive comorbidity measure that was generated by combining comorbidities from the ECI and CCI (Menendez et al.  or the ECI, CCI, and RxRisk-V . Both studies showed that the combination of comorbidity indices outperformed the ECI and the CCI.
Summary of evidence
This systematic review summarizes the evidence on how different comorbidity indices compare to one another in predicting mortality in an orthopedic setting. Most studies compare the ECI and CCI and found no clear difference in the predictive ability of these two indices. Six studies compared other comorbidity indices to the CCI and ECI and four of those performed equally good or better. For all comorbidity indices, adjusting for demographic factors together with comorbidities generally provided better predictive ability.
We found no clear difference in the predictive ability of the ECI and CCI. Both have good to excellent predictive ability for in-hospital mortality, but only poor to fair performance for all other time points. As the ECI and CCI comprise of several of the same comorbidity conditions this result might not be surprising. However, these findings are inconsistent with other studies on this topic. Yurkovich et al.  conducted a systematic review on comorbidity indices using administrative data and concluded that the ECI consistently outperformed the CCI in predicting both short- and long-term mortality. Sharabiani et al.  conducted a systematic review on comorbidity indices and found that comorbidity adjustment methods showed better prediction for long-term (> 30 days) than short-term (< 30 days) mortality. However, neither of these reviews specified a particular patient setting. Therefore, one hypothesis to explain this discrepancy of results can be the differences in the patient settings. This corresponds well with the statement by Yurkovich et al.  that the predictive ability of the comorbidity indices primarily depends on the outcome and study population. Comorbidity indices might not perform as well in an orthopedic setting compare to other settings. Therefore, any difference between comorbidity indices might not be evident. Another difference to consider is that the current review included fewer studies than the two previously mentioned reviews.
The Multipurpose Australian Comorbidity Scoring System , the study by Menendez et al. , and the study Inacio et al.  were all characterized by including a large number of comorbidities and all three performed well. The systematic review by Sharabiani et al.  found the same association. However, the evidence is not sufficiently extensive at this point to make a clear recommendation to use these newly developed comorbidity indices. As this study shows that the predictive ability of comorbidity indices are very similar, the choice of which comorbidity index to use in adjusting of comorbidity must to a large extend depend on other variables. The most correct method of choosing which comorbidity index to use is to do a validation of the indices for the study population of interest. However, this is a time- and labor-consuming work. Other factors might also play a role, such as comparability to previous studies, which use a specific index, and which comorbidity variables that are available in the relevant register.
Since comorbidity is associated with mortality, one would presumably expect that adjustment for comorbidity would increase the predictive ability considerably. This was not always evident in this review. One challenge with using administrative data to gather information on comorbidities is the potential underreporting in administrative databases and coding practices. Several studies have shown that administrative data can be an unreliable source [34, 35]. This might explain the varying and sometimes poor predictive ability of the comorbidity indices. The choice of which administrative database to use is of importance, and research on this matter is limited.
There are several limitations of this work. This review was limited by heterogeneity in the included studies. For instance, the CCI was assessed as one index all together when in fact it includes all of its subtypes, with various amounts of comorbidities (ranging from 12 to 19) and different weighting algorithms. Initially, we wanted to do an exploratory subgroup analysis for the subtypes of CCI and ECI, but we were not able to define all of them due to insufficient descriptions in each study.
Additionally, although the patient population was more specific than in previous systematic reviews on this topic [8, 9], it could still be classified as broad. We can assume, for instance, that patients undergoing arthroplasty surgery for osteoarthritis often differ significantly from patients with hip fracture. Therefore, it is debatable how comparable these patient groups are.
Further, we were not able to define in-hospital mortality. Therefore, this time period could vary markedly between studies and might cause some misinterpretation of the results.
Finally, at the moment, the overall evidence is not sufficiently extensive to determine the appropriate index for every mortality time point.
The results of this review indicate that the ECI and CCI can equally be used to adjust for comorbidities when analyzing in-hospital mortality in an orthopedic setting where they have fair to excellent predictive ability. However, in general, both indices have poor to fair AUC values for 30-day, 90-day, and 1-year mortality.
Availability of data and materials
Area under the curve
Charlson Comorbidity Index
Elixhauser Comorbidity Index
Risk Of Bias In Non-Randomized Studies of Interventions
Austin SR, Wong YN, Uzzo RG, Beck JR, Egleston BL. Why summary comorbidity measures such as the Charlson Comorbidity Index and Elixhauser Score Work. Med Care. 2015;53(9):e65-72.
Molto A, Dougados M. Comorbidity indices. Clin Exp Rheumatol. 2014;32(5 Suppl 85):S-131-4.
Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis. 1987;40(5):373–83.
Quan H, Li B, Couris CM, Fushimi K, Graham P, Hider P, et al. Updating and validating the Charlson comorbidity index and score for risk adjustment in hospital discharge abstracts using data from 6 countries. Am J Epidemiol. 2011;173(6):676–82.
Deyo RA, Cherkin DC, Ciol MA. Adapting a clinical comorbidity index for use with ICD-9-CM administrative databases. J Clin Epidemiol. 1992;45(6):613–9.
Romano PS, Roos LL, Jollis JG. Adapting a clinical comorbidity index for use with ICD-9-CM administrative data: differing perspectives. J Clin Epidemiol. 1993;46(10):1075–9. discussion 81-90.
Elixhauser A, Steiner C, Harris DR, Coffey RM. Comorbidity measures for use with administrative data. Med Care. 1998;36(1):8–27.
Yurkovich M, Avina-Zubieta JA, Thomas J, Gorenchtein M, Lacaille D. A systematic review identifies valid comorbidity indices derived from administrative health data. J Clin Epidemiol. 2015;68(1):3–14.
Sharabiani MT, Aylin P, Bottle A. Systematic review of comorbidity indices for administrative data. Med Care. 2012;50(12):1109–18.
PRISMA Statement. Transparent reporting of systematic reviews and meta-analyses. 2015 Available from: http://www.prisma-statement.org/.
Azzalini L, Chabot-Blanchet M, Southern DA, Nozza A, Wilton SB, Graham MM, et al. A disease-specific comorbidity index for predicting mortality in patients admitted to hospital with a cardiac condition. CMAJ. 2019;191(11):E299-e307.
Open Grey. System for Information on Grey Literature in Europe 2019 Available from: http://www.opengrey.eu/.
World Health Organization. International Clinical Trials Registry Platform 2019 Available from: http://apps.who.int/trialsearch/.
The Bone & Joint Journal. Orthopaedic Proceedings 2019 Available from: https://online.boneandjoint.org.uk/journal/procs.
Covidence. Better systematic review management Melbourne, Australia 2019 Available from: https://www.covidence.org/home.
Bulow E, Cnudde P, Rogmark C, Rolfson O, Nemes S. Low predictive power of comorbidity indices identified for mortality after acute arthroplasty surgery undertaken for femoral neck fracture. Bone Joint J. 2019;101B(1):104–12.
Inacio MCS, Pratt NL, Roughead EE, Graves SE. Evaluation of three co-morbidity measures to predict mortality in patients undergoing total joint arthroplasty. Osteoarthritis Cartilage. 2016;24(10):1718–26.
Kurichi JE, Stineman MG, Kwong PL, Bates BE, Reker DM. Assessing and using comorbidity measures in elderly veterans with lower extremity amputations. Gerontology. 2007;53(5):255–9.
Sterne JA, Hernan MA, Reeves BC, Savovic J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355:i4919.
Radiopaedia. Receiver operating characteristic curve 2019 Available from: https://radiopaedia.org/articles/receiver-operating-characteristic-curve.
Steyerberg EW, Harrell FE Jr. Prediction models need appropriate internal, internal-external, and external validation. J Clin Epidemiol. 2016;69:245–7.
Boddaert J, Na N, Le Manach Y, Raux M, Cohen-Bittan J, Vallet H, et al. Prediction of postoperative mortality in elderly patients with hip fracture: Are specific and geriatric scores better than general scores? Br J Anaesth. 2017;118(6):952–4.
Bulow E, Rolfson O, Cnudde P, Rogmark C, Garellick G, Nemes S. Comorbidity does not predict long-term mortality after total hip arthroplasty. Acta Orthop. 2017;88(5):472–7.
Kim CY, Sivasundaram L, LaBelle MW, Trivedi NN, Liu RW, Gillespie RJ. Predicting adverse events, length of stay, and discharge disposition following shoulder arthroplasty: a comparison of the Elixhauser Comorbidity Measure and Charlson Comorbidity Index. J Shoulder Elbow Surg. 2018;27(10):1748–55.
Lopez-de-Andres A, Hernandez-Barrera V, Lopez R, Martin-Junco P, Jimenez-Trujillo I, Alvaro-Meca A, et al. Predictors of in-hospital mortality following major lower extremity amputations in type 2 diabetic patients using artificial neural networks. BMC Med Res Methodol. 2016;16(1):160.
Menendez ME, Neuhaus V, van Dijk CN, Ring D. The Elixhauser comorbidity method outperforms the Charlson Index in predicting inpatient death after orthopaedic surgery. Clin Orthop Relat Res. 2014;472(9):2878–86.
Menendez ME, Ring D. A comparison of the Charlson and Elixhauser comorbidity measures to predict inpatient mortality after proximal humerus fracture. J Orthop Trauma. 2015;29(11):488–93.
Menendez ME, Ring D, Harris MB, Cha TD. Predicting in-hospital mortality in elderly patients with cervical spine fractures: a comparison of the charlson and elixhauser comorbidity measures. Spine. 2015;40(11):809–15.
Menendez ME, Neuhaus V, Ring D. Inpatient mortality after orthopaedic surgery. International Orthopaedics. 2015;39(7):1307–14.
Neuhaus V, King J, Hageman MG, Ring DC. Charlson comorbidity indices and in-hospital deaths in patients with hip fractures. Clin Orthop Relat Res. 2013;471(5):1712–9.
Ondeck NT, Bohl DD, Bovonratwet P, McLynn RP, Cui JJ, Grauer JN. Discriminative ability of Elixhauser’s comorbidity measure is superior to other comorbidity scores for inpatient adverse outcomes after total hip arthroplasty. J Arthroplasty. 2018;33(1):250–7.
Radley DC, Gottlieb DJ, Fisher ES, Tosteson AN. Comorbidity risk-adjustment strategies are comparable among persons with hip fracture. J Clin Epidemiol. 2008;61:580–7.
Toson B, Harvey LA, Close JC. New ICD-10 version of the multipurpose Australian comorbidity scoring system outperformed Charlson and Elixhauser comorbidities in an older population. J Clin Epidemiol. 2016;79:62–9.
Hua-Gen Li M, Hutchinson A, Tacey M, Duke G. Reliability of comorbidity scores derived from administrative data in the tertiary hospital intensive care setting: a cross-sectional study. BMJ Healthc Inform. 2019;26(1):16.
Powell H, Lim LL, Heller RF. Accuracy of administrative data to assess comorbidity in patients with heart disease. an Australian perspective. J Clin Epidemiol. 2001;54(7):687–93.
No funding was received for this study.
Ethics approval and consent to participate
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Gundtoft, P.H., Jørstad, M., Erichsen, J.L. et al. The ability of comorbidity indices to predict mortality in an orthopedic setting: a systematic review. Syst Rev 10, 234 (2021). https://doi.org/10.1186/s13643-021-01785-4