The ability of comorbidity indices to predict mortality in an orthopedic setting: a systematic review

Background Several comorbidity indices have been created to estimate and adjust for the burden of comorbidity. The objective of this systematic review was to evaluate and compare the ability of different comorbidity indices to predict mortality in an orthopedic setting. Methods A systematic search was conducted in Embase, MEDLINE, and Cochrane Library. The search were constructed around two primary focal points: a comorbidity index and orthopedics. The last search were performed on 13 June 2019. Eligibility criteria were participants with orthopedic conditions or who underwent an orthopedic procedure, a comparison between comorbidity indices that used administrative data, and reported mortality as outcome. Two independent reviewers screened the studies using Covidence. The area under the curve (AUC) was chosen as the primary effect estimate. Results Of the 5338 studies identified, 16 met the eligibility criteria. The predictive ability of the different comorbidity indices ranged from poor (AUC < 0.70) to excellent (AUC ≥ 0.90). The majority of the included studies compared the Elixhauser Comorbidity Index (ECI) and the Charlson Comorbidity Index (CCI). In-hospital mortality was reported in eight studies reporting AUC values ranging from 0.70 to 0.92 for ECI and 0.68 to 0.89 for CCI. AUC values were generally lower for all other time points ranging from 0.67 to 0.78. For 1-year mortality the overall effect size ranging from 0.67 to 0.77 for ECI and 0.69 to 0.77 for CCI. Conclusion The results of this review indicate that the ECI and CCI can equally be used to adjust for comorbidities when analyzing mortality in an orthopedic setting. Trial registration The protocol for this systematic review was registered on PROSPERO, the International Prospective Register of Systematic Reviews on 13 June 2019 and can be accessed through record ID 133,871. Supplementary Information The online version contains supplementary material available at 10.1186/s13643-021-01785-4.


Background
Research that uses data from registers and administrative databases often must deal with several potential sources of confounding factors, one of which is comorbidity.
Comorbidity is all additional conditions, which have an effect on the patients' conditions concomitant with the primary orthopedic condition. Therefore, taking into account and adjusting for comorbidity is often recommended when conducting research on mortality [1]. In an orthopedic setting, comorbidity is often relevant as it is associated with an increased risk of complications, reoperations, and mortality.
Adjustment for all known comorbidity variables is often not methodologically possible. Consequently, several comorbidity indices have been created to measure and adjust for the estimated burden of comorbidity. The Charlson Comorbidity Index (CCI) and the Elixhauser Comorbidity Index (ECI) are the most commonly used risk predictor tools [2]. CCI was developed and validated in 1987 [3] to predict risk of death within 1 year of hospitalization and since then different versions that uses administrative data have been created [4]. CCI was first adapted to ICD-codes by Deyo in 1992 and Romani in 1993 [5,6]. Today, the most commonly used is the updated version by Quan et al. [4] that adapted CCI to be used with ICD-10 codes. The ECI was specifically developed using administrative data in 1998 [7] and with the CCI, Quan et al. also made an adaption of the ECI to the ICD-10 code.
The contents of these indices differ and have had numerous adaptations. Consequently, it has become difficult to identify which index to select when performing a risk assessment or when adjusting for comorbidity.
Previous systematic reviews on this topic have found several factors to be essential when selecting a comorbidity index. Yourkovich et al. [8] and Sharabiani et al. [9] concluded that the predictive ability of a comorbidity index primarily depends on the outcome and study population. However, no systematic review has investigated the validity of comorbidity indices in an orthopedic setting. The aim of this paper was to conduct a systematic review comparing the ability of different comorbidity indices to predict mortality in an orthopedic setting.

Methods
This review is reported in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines (PRISMA) and the PRISMA checklist can be found in Additional file 1 [10]. The protocol was registered in the International Prospective Register of Systematic Reviews (PROSPERO) on 13 June 2019 and can be accessed through record ID 133,871.

Eligibility criteria
This systematic review included studies that compared the performance of different comorbidity indices in predicting mortality in an orthopedic setting.
Studies were included if they: • Included participants with orthopedic conditions or participants who underwent an orthopedic procedure. • Included a comparison of one comorbidity index with another comorbidity index.
• Included comorbidity indices that were based on administrative data (registers and databases). • Reported mortality as an outcome.
Studies were excluded if they only included comorbidity indices that were validated only for specific diseases (e.g., the Cardiac-specific Comorbidity Index [11]). We did not have exclusion criteria in form of a minimum age limit nor for whether the study participants was in-or outpatients.
Primary outcome measures were defined as in-hospital, 30-day, 90-day, and 1-year mortality. Secondary outcome measures were not prespecified.
The language of included studies was restricted to articles written in English. No publication year, publication status, or study design limitations were imposed.

Information sources
We searched the following databases: Ovid Embase Classic + Embase (1947-present), Ovid MEDLINE (R) all (1946-present), and the Cochrane Library. The last search was carried out on 2 May 2019. The reference list of included studies was screened by PHG and MJ for additional eligible studies. Grey literature was also searched through OpenGrey [12], the International Clinical Trial Registry Platform [13], and Orthopaedic Proceedings [14].

Search
The search strategy was developed by the author team with assistance from librarians at the University of Southern Denmark and was primarily developed for Embase; thereafter, it was adapted for the other databases. The searches were constructed around two primary focal points: a comorbidity index and orthopedics. No language or publication year restrictions were used in the search. The following search terms were used to search Embase: comorbidity ind*; comorbidity scor*; comorbidity scale*; exp Charlson Comorbidity Index; Charlson* adj2 (ind* or scale* or score*); Elixhauser*; Chronic Disease Score*; Rxrisk; Rx-risk; Rxrisk-v; Rx-risk-v; Medication-Based Disease Burden Index; exp orthopedics; orthop?edic*; exp orthopedic surgery; exp arthroplasty; arthroplast*; hemi-arthroplast*; hemiarthroplast*; exp osteoarthritis; osteoarthritis; exp arthroscopy; arthroscop*; exp fracture; fracture*; exp traumatology; traumatolog*; exp hip surgery; exp knee surgery; exp shoulder surgery; hip* or knee* or shoulder* adj3 (surg*).
A full description of the search strategies for Embase, MEDLINE, and the Cochrane Library can be found in the Additional file 2.

Study selection
Eligibility assessment of the studies was performed independently in a blinded standardized manner by PHG and MJ using Covidence [15]. The two authors assessed the 5338 articles by screening title and subsequent reading the abstract of the studies and classified the studies as "eligible, " "not eligible, " or "perhaps eligible. " Following the independent assessment, the two authors then compared result. Disagreements between PHG and MJ were resolved by consensus and by consulting with the other authors.

Data collection process
The data were extracted using a prespecified data collection form in Microsoft Excel.

Risk of bias in individual studies
The risk of bias for each individual study was assessed by PHG and MJ using Risk Of Bias In Non-Randomized Studies of Interventions (ROBINS-I) [19].

Summary measures
No summary effect measure was prespecified in the protocol. All 16 included studies reported the area under the curve (AUC) of receiver operating characteristic curve. Therefore, AUC was chosen as the primary effect estimate in this review; the AUC measures the predictive performance of a comorbidity index. The AUC value ranges from 0.5 to 1.0, where 0.5 indicates no predictive ability and 1.0 indicates perfect predictive ability. In general, an inexact guide can be used: 0.50: no discriminating ability, 0.60-0.69: poor accuracy, 0.70-0.79: fair accuracy, 0.80-0.89: good accuracy, and 0.90-1.00: excellent accuracy [20]. Confidence level of 95% is used when reporting AUC values.

Data analysis
We originally intended to do a meta-analysis on the studies that compared CCI with ECI. However, due to the variation in the different versions and models of the comorbidity indices, differences in reported outcomes, and validity issues concerning the use of meta-analysis of predictive models [21] we omitted the meta-analysis and instead did a more narrative report of the studies.

Study selection
The searches of the MEDLINE, Embase, and Cochrane Library databases provided a total of 6333 citations (Fig. 1). After adjusting for duplicates using Endnote and Covidence, 5338 citations remained. Of these, 5300 studies were discarded after reviewing the titles and abstracts (Fig. 1).
The full text of the remaining 38 citations was assessed. Sixteen studies met the inclusion criteria and were included in the systematic review (Table 1). No further studies were identified by screening the references of the included articles or searching the grey literature.

Study characteristics
All of the studies were cohort studies ( Table 1). All but one of the included studies [22] used various types of registers to gather the administrative data, with the National Hospital Discharge Survey being the most commonly used database. The data periods varied from 1990 to 2014. Nine of the studies were carried out in the USA. The number of patients included in each study ranged from 506 to 33,194,141. Eight of the included studies compared only the ECI with the CCI [16,18,[23][24][25][26][27][28][29]. In addition to these two indices, six other indices were also assessed: a Multipurpose Australian Comorbidity Scoring System, the Cumulative Illness Rating Scale, the Preoperative Score to Predict Postoperative Mortality, the Iezzoni Index, and the Clinical Classifications Software and Modified Frailty Index.
Regarding the outcome measures, in-hospital mortality was the most commonly reported outcome, followed by 1-year and 30-day mortality.

Risk of bias within studies
Most studies were evaluated as having a moderate risk of bias. Two studies were evaluated to have serious risk of bias due to either confounding or bias in reported results ( Table 2).

Results of individual studies
The 16 included studies varied markedly regarding the type and version of comorbidity indices assessed, and the follow-up time, which ranged from in-hospital to 14 years (  [33), the comorbidity indices was compared to a base model, comprising of demographic variables (e.g., age and sex) and in two of those, the base model outperformed both CCI and ECI.
In nine studies, the comorbidity indices were combined with demographic variables (e.g., age and sex). In all nine studies, this combination increased the predictive power.

Comparison of the Elixhauser and Charlson Comorbidity Indices
The CCI and ECI were analyzed and compared in 12 studies ( Table 3).
Two studies reported on the 30-day mortality [16,33] and found that the predictive ability of both indices was quite low, ranging from poor to fair (0.67-0.767).
Regarding 1-year mortality, four studies reported on both CCI and ECI for this time point [16][17][18]33], and the two indices had similar AUC values, ranging from poor to fair (0.67-0.77).

Other indices
In addition to the CCI and ECI, six other indices were identified. None of these six indices were assessed in more than one study. Most of the other indices were compared with the CCI and/or ECI, and in general they performed equally good or better, with the exception of the RxRisk-V and the Modified Frailty Index, which had AUC values that were slightly lower than those of the CCI and ECI [17,31]. The Multipurpose Australian Comorbidity Scoring System demonstrated good discriminating ability in predicting in-hospital (AUC 0.81 (0.80-0.82)) and 30-day mortality (AUC 0.80 (CI 0.79-0.81)), but only fair prediction of 1-year mortality (AUC 0.77 (CI 0.76-0.77)) [33].
It is also worth recognizing that two studies assessed more inclusive comorbidity measure that was generated by combining comorbidities from the ECI and CCI (Menendez et al. [29] or the ECI, CCI, and RxRisk-V [17]. Both studies showed that the combination of comorbidity indices outperformed the ECI and the CCI.

Summary of evidence
This systematic review summarizes the evidence on how different comorbidity indices compare to one another in predicting mortality in an orthopedic setting. Most studies compare the ECI and CCI and found no clear difference in the predictive ability of these two indices. Six studies compared other comorbidity indices to the CCI and ECI and four of those performed equally good or better. For all comorbidity indices, adjusting for demographic factors together with comorbidities generally provided better predictive ability.
We found no clear difference in the predictive ability of the ECI and CCI. Both have good to excellent predictive ability for in-hospital mortality, but only poor to fair performance for all other time points. As the ECI and CCI comprise of several of the same comorbidity conditions this result might not be surprising. However, these findings are inconsistent with other studies on this topic. Yurkovich et al. [8] conducted a systematic review on comorbidity indices using administrative data and concluded that the ECI consistently outperformed the CCI in predicting both short-and long-term mortality. Sharabiani et al. [9] conducted a systematic review on comorbidity indices and found that comorbidity Table 2 Assessment of risk of bias using ROBINS-I assessment tool adjustment methods showed better prediction for longterm (> 30 days) than short-term (< 30 days) mortality. However, neither of these reviews specified a particular patient setting. Therefore, one hypothesis to explain this discrepancy of results can be the differences in the patient settings. This corresponds well with the statement by Yurkovich et al. [8] that the predictive ability of the comorbidity indices primarily depends on the outcome and study population. Comorbidity indices might not perform as well in an orthopedic setting compare to other settings. Therefore, any difference between comorbidity indices might not be evident. Another difference to consider is that the current review included fewer studies than the two previously mentioned reviews. The Multipurpose Australian Comorbidity Scoring System [33], the study by Menendez et al. [29], and the study Inacio et al. [17] were all characterized by including a large number of comorbidities and all three performed well. The systematic review by Sharabiani et al. [9] found the same association. However, the evidence is not sufficiently extensive at this point to make a clear recommendation to use these newly developed comorbidity indices. As this study shows that the predictive ability of comorbidity indices are very similar, the choice of which comorbidity index to use in adjusting of comorbidity must to a large extend depend on other variables. The most correct method of choosing which comorbidity index to use is to do a validation of the indices for the study population of interest. However, this is a time-and labor-consuming work. Other factors might also play a role, such as comparability to previous studies, which use a specific index, and which comorbidity variables that are available in the relevant register.

Study Confounding Selection Classification of intervention
Since comorbidity is associated with mortality, one would presumably expect that adjustment for comorbidity would increase the predictive ability considerably. This was not always evident in this review. One challenge with using administrative data to gather information on comorbidities is the potential underreporting in administrative databases and coding practices. Several studies have shown that administrative data can be an unreliable source [34,35]. This might explain the varying and sometimes poor predictive ability of the comorbidity indices. The choice of which administrative database to use is of importance, and research on this matter is limited.

Limitations
There are several limitations of this work. This review was limited by heterogeneity in the included studies. For instance, the CCI was assessed as one index all together when in fact it includes all of its subtypes, with various amounts of comorbidities (ranging from 12 to 19) and different weighting algorithms. Initially, we wanted to do an exploratory subgroup analysis for the subtypes of CCI and ECI, but we were not able to define all of them due to insufficient descriptions in each study. Additionally, although the patient population was more specific than in previous systematic reviews on this topic [8,9], it could still be classified as broad. We can assume, for instance, that patients undergoing arthroplasty surgery for osteoarthritis often differ significantly from patients with hip fracture. Therefore, it is debatable how comparable these patient groups are. Further, we were not able to define in-hospital mortality. Therefore, this time period could vary markedly between studies and might cause some misinterpretation of the results.
Finally, at the moment, the overall evidence is not sufficiently extensive to determine the appropriate index for every mortality time point.

Conclusions
The results of this review indicate that the ECI and CCI can equally be used to adjust for comorbidities when analyzing in-hospital mortality in an orthopedic setting where they have fair to excellent predictive ability. However, in general, both indices have poor to fair AUC values for 30-day, 90-day, and 1-year mortality.