Prognostic models of diabetic microvascular complications: a systematic review and meta-analysis

Background Many prognostic models of diabetic microvascular complications have been developed, but their performances still varies. Therefore, we conducted a systematic review and meta-analysis to summarise the performances of the existing models. Methods Prognostic models of diabetic microvascular complications were retrieved from PubMed and Scopus up to 31 December 2020. Studies were selected, if they developed or internally/externally validated models of any microvascular complication in type 2 diabetes (T2D). Results In total, 71 studies were eligible, of which 32, 30 and 18 studies initially developed prognostic model for diabetic retinopathy (DR), chronic kidney disease (CKD) and end stage renal disease (ESRD) with the number of derived equations of 84, 96 and 51, respectively. Most models were derived-phases, some were internal and external validations. Common predictors were age, sex, HbA1c, diabetic duration, SBP and BMI. Traditional statistical models (i.e. Cox and logit regression) were mostly applied, otherwise machine learning. In cohorts, the discriminative performance in derived-logit was pooled with C statistics of 0.82 (0.73‑0.92) for DR and 0.78 (0.74‑0.83) for CKD. Pooled Cox regression yielded 0.75 (0.74‑0.77), 0.78 (0.74‑0.82) and 0.87 (0.84‑0.89) for DR, CKD and ESRD, respectively. External validation performances were sufficiently pooled with 0.81 (0.78‑0.83), 0.75 (0.67‑0.84) and 0.87 (0.85‑0.88) for DR, CKD and ESRD, respectively. Conclusions Several prognostic models were developed, but less were externally validated. A few studies derived the models by using appropriate methods and were satisfactory reported. More external validations and impact analyses are required before applying these models in clinical practice. Systematic review registration PROSPERO CRD42018105287 Supplementary Information The online version contains supplementary material available at 10.1186/s13643-021-01841-z.


Background
Type 2 diabetes (T2D) has increased rapidly over the past 30 years becoming worldwide public health problem with prevalence in adults of 463 million (9.3%) in 2019. It is estimated to be 700 million (10.9%) by 2045 [1], in which currently about 79% of people have diabetes living in low-and middle-income countries [1,2]. Furthermore, diabetic progression due to its complications-increased disability, impaired quality of life and leading cause of premature death, which accounted for 11.3% of the global mortality [1,3].

Protocol registration
This study was conducted following Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [31] and in accordance with CHARMS checklist [32]. The review protocol was registered at PROS-PERO (CRD42018105287).

Search strategy
Studies were identified from PubMed and Scopus up to 31 December 2020. Search terms were constructed based on patients, interventions and outcomes, see details in Additional files 1 and 2.

Study selection
Studies, published in any language, were eligible if they studied in adult T2D, developed or validated any multivariable prognostic models of microvascular complications in T2D with applying any traditional statistical modelling (e.g. logit or Cox regression etcetera) or machine learning (ML), and reported model performance. We also included the studies from reference list of relevant publications.

Data extraction
Data extractions were performed by one reviewer (SAS) and checked by OP. Extracted data were characteristics of study and patients (i.e. country, study design, settings, data source, sample size and number of events, ethnicity, age, percent male and diabetic duration), study phase (i.e. derivation or validation), statistical methods, predictors, missing data and outcomes (i.e. DR, CKD and ESRD). In addition, two related properties of model performances (calibration and discrimination) were also extracted.

Risk of bias assessment
Risk of bias assessment was assessed by using Prediction Model Risk of Bias Assessment Tool (PROBAST) [32]. Each item was rated as low, high or unclear. The overall validity was low and high risk if all domains were low risk, and at least one domain was high risk, respectively. Discrepancies were solved by consensus between the team.

Statistical analysis
Characteristics of each prognostic model and predictive performances (including calibration and discrimination) were described. Discrimination was assessed according to original included studies, in which C-statistic was mostly used. If studies reported C-statistic without variance, it was estimated using equations in the previous guidelines [32][33][34]. Calibration was assessed [35] using calibration plot, goodness-of-fit testing (i.e. Hosmer and Lemeshow χ 2 test), calibration slopes or the observed/ expected (O/E) ratio.
A meta-analysis was applied for pooling C statistics across studies stratified by study's design/phase, statistical model and T2D complications. A random-effect model by DerSimonian-Laird [36,37] was used if heterogeneity was present (p value < 0.10 or I 2 > 25%); otherwise, a fixed-effect model was used. A heterogeneity was assessed by Cochrane Q test and I 2 statistic. Publication bias in external validation was assessed using funnel-plot [38] and Egger's test [39]. All statistical analyses were performed using STATA 16 [40]. A p value less than 0.05 was considered as statistically significant, except for heterogeneity which used 0.10.

Results
A total of 32/1009 and 44/3321 studies were eligible for DR and DN, respectively, see Table S1 (Figs. S1-S2). Amongst them, 205 prognostic equations were derivative, some of them performed internal and external validations. Most studies reported C statistics, but only a few-portions reported calibrations (Table S2).

Risk of bias assessment
Risk of bias assessment of all included studies was presented in Table S3. Amongst 71 studies, about 86 to 95% of studies were determined as low risk of bias for study participants, selection of predictors and outcome measurement. About 23% and 40% of studies were rated as high risk for sample size, participant flow and statistical analysis, respectively. As a result, 35% of studies were overall low risk of bias (Fig. S3).
The pooled C statistics in cohorts using Cox were 0.87  [26,29,44] cohorts (Table S5). Funnel plot and Egger's test (p > 0.513) showed no publication bias by the absence of small study effects in external validation studies for predicting ESRD (Fig. S7).

Discussion
This review summarised prognostic models that were developed and validated for predicting microvascular complications (i.e. DR, CKD and ESRD) in T2D patients. Model performances were described prognostic models separately by derived, internal and external validation.
Seven predictors were commonly used in predictive models of DR, DN and ESRD including age, sex, BMI, diabetic duration, HbA1c, SBP and eGFR. The DR models showed well discriminated with pooled C statistics of 0.82, 0.83 and 0.81 in D, I and E validations, respectively. Only a few prognostic models were externally validated with moderate to good discrimination performance, which are applicable in clinical practice. For instance, a few DR-models [11,12,41] had good discrimination and calibration in external validations. Three [12,20,68] DN models had good discrimination with fair calibration. Other three [12,26,29] ESRD models with very large size cohorts were generalisable with good discriminations and were even developed in different ethnicities. Calibration performance was less reported relative to discrimination, although both parameters should be reported for prognostic model development [95][96][97]. Particularly for observed to expected (O/E) ratio was reported in very few studies, which prevented meta-analysis of calibration.
Currently many prediction models are available by online calculators, or differently presented simplified risk scores or nomograms. Some online risk-calculators have been developed to simplify knowledge translation in clinical practice (i.e. DR [11,12,41,59], CKD [12,20] and ESRD [26]). However, very few of them have been applied due to the absence of some predictors and users' interpretations in routine health practice.
We found various clinical settings and developed equations, but only few of them were externally validated with insufficiently reported with a wide range of CKD definitions. Amongst them, there might be potentially overoptimistic as EPV was less than ten by the rule of thumb in a regression model. None of the studies performed impact assessments by applying prognostic models into clinical practice.
Missing data in clinical settings particularly for routine datasets are unavoidable. Frequently, the investigators only performed complete-case analysis. Handling missing data is vitally important to prevent biassed results and lost power in generalisations [98]. Additionally, categorisation of continuous predictors or dichotomisation may result in missing information, significant misleading [99], incorrect variable selection and may decrease prediction accuracy [100,101].
Cohort or RCT should be the most appropriated design for developing prognostic model, whereas a cross-sectional study could be used for external validation. Exceptionally, nested case-control and casecohort studies were still applicable [96]. The rule of thumb suggested that a number of 10-20 events should be available for one predictor in a multivariable logit/ Cox regression [96,102,103]. For instance, seven [14,17,24,52,56,58,61] studies in DR had EPV ratio of 1 [58] to 9 [24,52], which might cause overfitted model. In DN, eight [21,22,68,73,76,77,79,83] and seven [27, 28, 85-87, 91, 92] studies might be over-optimistic with the EPV ratio less than 10 for CKD and ESRD, respectively. Overfitting may result in poorer performance in external validation compared with derived-performance. As a result, performances of the traditional statistical models (i.e. logit, Cox) were quite varied across studies. However, ML may be better particularly when predictors themselves have collinearity and high-dimensional interaction amongst predictors. With the rapid era of big data, digitalisation and modern electronic medical records may increase used of ML techniques in derived and validation model.
As the backbone of big data analysis, ML provides the new insight and valuable algorithm in which traditional statistical models are often inadequate. Likewise, using image/signal [56,58] analysis for predicting DR, some investigators also applied classical ML (e.g. decision trees, random forest, Naïve Bayes and neural network) to predict DN [13,70,73]. Nonetheless, the results of ML are black boxes, which are often difficult to interpret due to its characteristics and algorithm complexities [104,105].
Few other factors may also influence on externalvalidation performance, e.g. availability of predictors, sources of data (i.e. primary data collection, surveydata or administrative/hospital-claims data), outcome rate and assessment and also population characteristics. However, only about 20 studies (25% of derived models) were externally validated. We therefore strongly suggest that those derived models should be externally validated or updated models where appropriate. Then, impact analysis should next be performed to be more confident in applying in clinical practice.

Conclusions
This study was conducted to systematically review prognostic models of diabetic microvascular complications. Weaknesses and strengths of those prognostic models for each complication were described and commented. Some prognostic models for microvascular complications were good in discrimination in external validations, but in practice none of them performed clinical impact. The existing prognostic models for DR and CKD still need further external validation or update where appropriate. In addition, the new prognostic models should be derived using ML techniques to improve prognostic performance where required.

Supplementary Information
The online version contains supplementary material available at https:// doi. org/ 10. 1186/ s13643-021-01841-z.  Table S1. Characteristics of studies included in systematic review. Table S2. Describe discrimination and calibration performances of prognostic models. Table S3. Risk of bias assessment using PROBAST. Table S4. Prognostic equations that were externally validated. Table S5. Summary of pooled C-statistic of prognostic model. Table S6. Describe variables that were included in the derivative equations.
Authors' contributions SAS, OP and AT conceptualised the study questions and developed protocol and search strategy for the systematic review. SAS conducted the review (screening, data extraction, analyse the data) and wrote the manuscript. SAS and SK did the risk of bias assessment and wrote interpretation. AT, OP and AP edited and revised the manuscript. All authors have read and approved the final version of the manuscript to be published.

Funding
This research did not receive any specific grant from funding agencies in the public, commercial or not-for-profit sectors.