Machine Learning–Driven Models to Predict Prognostic Outcomes in Patients Hospitalized With Heart Failure Using Electronic Health Records: Retrospective Study

Introduction

Heart failure (HF) syndrome is a life-threatening chronic disorder with a global prevalence that has been rising consistently over recent decades because of population aging, shifts in disease spectrums, and improved survival rates among patients with various cardiovascular diseases [1,2]. HF is characterized by complex therapeutic regimens, frequent hospitalizations, and a poor prognosis, resulting in a tremendous health care burden [3]. In certain instances, these issues are also fundamental targets of a strategy for HF prevention and treatment [4,5]. It is crucial to discriminate accurately among patients with HF to identify those who have a high risk of in-hospital mortality and readmission, as well as to guide the use of different therapies based on patients' features.

A prediction model was developed as an important risk assessment tool and used in various health care areas over the past decades. It has been recognized to facilitate early identification of patients at disease or event risk and enables effective interventions for those who might benefit most from identifying specific risk factors [6,7]. Previous studies in the field of cardiology, generally based on different populations with HF, have constructed models that are relevant to prognosis prediction, including the Seattle Heart Failure Model (SHFM) [8], Munich score [9], Enhanced Feedback for Effective Cardiac Treatment (EFFECT) scale [10], and Acute Decompensated Heart Failure National Registry (ADHERE) [11]. Moreover, several available parameters related to increased mortality have been identified as well, such as age [10,12,13], concentration of B-type natriuretic peptide (BNP) [14,15], urea nitrogen level [10,12], and systolic blood pressure (SBP) [9,13]. Although the model’s construction from such cohorts or databases provides a level of concrete evidence, it is typically limited to large volumes of clinical resources and unstructured data [16]. Given the growing popularity of big data use and mining, data derived from electronic health records (EHRs) are becoming more available and accessible for clinical research.

Besides this, predictive models have been constructed in various health care domains with a certain degree of success by automated mining of EHRs, combined with machine learning (ML) approaches [17], specifically for the prediction of HF outcomes [18-20]. In contrast to previous studies of predictive models for HF outcomes that apply traditional methods, recent research is adopting ML techniques for predicting HF mortality, readmission, and medication adherence, which might demonstrate better performance in their predictions because of their consideration of higher order and nonlinear relationships between multidimensional variables [21].

In our study, we explored the use of the traditional method—logistic regression (LR)—and the four novel ML approaches—support vector machine (SVM), artificial neural network (ANN), random forest (RF), and extreme gradient boosting (XGBoost)—to predict prognostic outcomes for subjects with HF in real-world settings. We demonstrated the development of EHR-based models to predict the 1-year in-hospital mortality, use of positive inotropic agents, and all-cause readmissions in a single year.

Methods

Patients and Data Source

We collected the EHR data from hospitalized patients diagnosed with HF at the First Affiliated Hospital of Dalian Medical University (Liaoning Province, China) over 7 years between December 2010 and August 2018. All patients with HF were diagnosed and treated according to institutional guidelines. Because the diagnostic terminologies of EHRs have been structured and normalized according to the International Statistical Classification of Diseases and Related Health Problems, 10th Revision (ICD-10 [22]), newly diagnosed patients with HF (aged >18 years) were screened by a dynamically updated big data intelligence platform developed by an artificial intelligence technology company in collaboration with hospitals (Yidu Cloud Technology Co., Ltd). Each patient’s data were extracted from various EHR systems integrated into a single platform, including the hospital information system, electronic medical record, radiation information system, laboratory information system, ultrasound system, and electrocardiogram system. The study was finally approved by the ethics committee of the First Affiliated Hospital of Dalian Medical University. Written informed consent was waived due to the retrospective design.

Selection of Variables

The platform incorporates comprehensive and detailed data on patients’ routine health care. Common and cardiovascular-specific variables were structured and normalized by natural language processing, ML techniques, and well-defined logical rules. Considering the large scale of EHR data and the advantages of ML techniques, candidate variables with known clinical significance and inaccessible through traditional medical records were collected in our study. We excluded variables with missing values greater than 20%. Finally, a total of 79 variables related to the first hospitalization were extracted from a big data intelligence platform. The features were as follows: demographics (age and sex), personal history (smoking and drinking), history (comorbidities and surgery), etiology, vital signs, routine laboratory examinations, interventions, and medication use on admission (Multimedia Appendix 1). Repeated measurements of vital signs and laboratory tests from patients with HF were taken over different periods to ensure data accuracy.

Outcomes

We established models using 5 algorithms separately to predict the primary outcome: all-cause in-hospital mortality within 1 year. The secondary outcomes were (1) use of positive inotropic agents for patients with HF and (2) all-cause readmission. Mortality was defined as a clear record of death of an inpatient within 1 year of hospitalization. The following commonly used positive inotropic agents in clinical practice were selected for our study: dopamine, dobutamine, milrinone, levosimendan, and deslanoside. Readmission was defined as any patient with an interval of more than 1 day from the last discharge until the next admission; no readmitted patients died within 1 year of hospitalization. Therefore, enrolled patients were labeled as “deceased” or “survivors.”

ML Model Development and Performance Evaluation

The synthetic minority oversampling technique was first employed to address the imbalance in the data set, with a ratio of 1:1 between deceased patients and survivors. It is common for a minority to be oversampled by deriving new, “synthetic” samples to alleviate imbalance [23].

The hold-out method was used to divide the data set into a training/validation data set (90%) and a hold-out test set (10%). Five approaches were explored with 10-fold cross-validation for building models and adjusting parameters: LR, RF, SVM, ANN, and XGBoost. The hold-out test set was used to evaluate the best-performing models created with the training set. The area under the curve (AUC) of the receiver operating characteristic (ROC) curve was chosen as the primary evaluation metric for our models, including accuracy, precision, and recall. The Brier score (range of 0 to 1)—the average squared error between the predicted and the actual value—was also commonly represented as a “calibration” for overall measurement. Shapley additive explanation (SHAP) values were used to evaluate feature importance.

LR Method

LR is the most basic dichotomous linear method of model selection that makes classification decisions. LR is superior in measuring the probability between “0” and “1” based on the relationships of binary classifications in continuous or categorical variables [24]. We used the sklearn.linear module to develop the LR models.

RF Method

RF is an algorithm that integrates multiple decision tree classifiers. Each node of the decision tree represents a predictive variable to separate the outcome classes by setting the optimal threshold. The importance of features can also be obtained with the sum of weights of the classifier’s nodes [25]. The RF method is generally used to deal with thousands of input variables without dimension reduction. We used the RF classifier from the sklearn.ensemble module to develop the models. The related parameters (n_estimators, max_features, max_depth, and min_samples_split) were adjusted to prevent poor results and overfitting.

SVM Method

As a dichotomous supervised algorithm, SVM can be used in high-dimensional feature space. The best hyperplane can be achieved using a kernel-based function for separating two classes at maximum intervals [26].

ANN Method

A multilayer perceptron (MLP) classifier was implemented using the sklearn.neural_network module to develop models [27]. MLP is a popular ANN method that generally consists of neurons from the input, hidden, and output layers. The data were processed through weighted connections and activation functions in the hidden layers [28]. In our study, the two hidden layers had 40 and 20 neurons. The rectified linear unit was chosen as their activation function.

XGBoost Method

XGBoost is one of the boosting methods; this algorithm aims to integrate weak classifiers into a single robust classifier in an iterative fashion [29]. This algorithm constructs a scalable classification and regression tree in a boosting ensemble manner on a gradient boosting decision tree basis, which can learn nonlinear relationships among variables and outcomes flexibly and accurately [30]. The small learning rate (0.1) indicates better generalization. The tree number and maximum depth were limited to 80 (estimators) and 3, respectively.

Statistical Analysis

All ML algorithms were performed using the scikit-learn (version 0.21.1) package in Python (version 3.6.5; Python Software Foundation), and statistical analysis was conducted using an open-source Scipy (version 1.3.0) database from Python (version 3.6.3). All categorical data were presented as percentages. All continuous data performing a normal distribution were presented as mean (SD); otherwise, they were expressed as median (IQR). Student t tests or chi-square tests were applied for group comparisons. P values <.05 were considered statistically significant.

Results

Baseline Characteristics

A total of 13,602 hospitalized patients with newly diagnosed HF were enrolled in this study, of whom 3.95% (n=537) died in hospital within 1 year. The usage rate of positive inotropic agents was 20.43% (n=2779). The overall all-cause readmission rates of 30 days, 60 days, and 1 year for patients with HF were 4.83%, 14.77%, and 21.16%, respectively (n=657, n=2009, and n=2878, respectively, of 13,602 cases). The eligible population’s baseline characteristics were compared between 2 groups according to survival status (Table 1). Patients in the deceased group were older than those in the survivor group (77 years, IQR 66.5-83.0 years versus 72 years, IQR 63.0-80.0 years, respectively). The proportion of male patients in both groups was 52.5% (282/537 and 6860/13,065 in the deceased group and survivor group, respectively). The number of patients with comorbid diagnoses of diabetes, hypertension, and tumors was remarkably different between the 2 groups (all P<.001). Concerning the etiology of HF, the number of patients with cardiomyopathy and cardiac arrhythmia was found to be different between the deceased and survivor groups (both P<.001). There was a significant difference in the vital signs (heart rate and blood pressure) of patients with HF between the 2 groups (all P<.001). In-hospital medication use (angiotensin-converting enzyme inhibitors [ACEIs] and aldosterone receptor antagonists [ARBs]) was higher among subjects who survived in the hospital (P<.001).

1-Year In-Hospital Mortality Models

Predictive models for 1-year in-hospital mortality risk assessment were conducted using 5 algorithms. Figure 1A shows the performances of models in the form of AUC. The AUC values for LR, RF, SVM, ANN, and XGBoost were 0.91, 1.00, 0.99, 0.99, and 0.99, respectively. RF had relatively higher AUC than the other algorithms. The calibration plots of our 5 methods are presented in Figure 2A. Four ML models had an accuracy of higher than 95%. Regarding precision, the RF and ANN algorithms emerged as the best, achieving the highest precision (0.96), followed by the SVM (0.93) and XGBoost (0.91) algorithms. The Brier score for RF and ANN was the lowest (0.03) (Table 2).

‎View this figure

‎View this figure

Furthermore, we explored the importance of the features that affect mortality prediction by applying RF and XGBoost approaches (Figure 3), with the weight assignment of each feature expressed as a SHAP value based on whether it favored judgment in survival. As shown in Figure 3A and 3B, blood urea, high-sensitivity cardiac troponin I (hs-cTnI), creatinine, aspartate aminotransferase (AST), and percentage of lymphocytes were the top 5 related variables in mortality prediction. In contrast, hs-cTnI was the most crucial marker identified using the XGBoost algorithm, followed by urea, respiration rate, percentage of basophils, and percentage of neutrophils.

‎View this figure

Positive Inotropic Agent Use Models

Figure 1B demonstrates the ROC of the predictive models in patients' use of positive inotropic agents. In comparing AUCs among the 5 models, the XGBoost and RF models had the highest AUC value (0.96), followed by the SVM model (0.91), ANN model (0.90), and LR model (0.85). In consideration of precision (0.85) and recall (0.91) values, RF was determined to be the best method for prediction (Table 3). As shown in calibration curves (Figure 2B), a nonparametric plot of the RF algorithm was close along the ideal diagonal line and had the lowest Brier score (0.10).

Interestingly, BNP, international normalized ratio (INR), pulse rate, hs-cTnI, and AST were the top 5 markers predicting use of positive inotropic agents by the RF approach. BNP was also identified as a critical marker for forecasting positive inotropic agent use in the XGBoost model, followed by basophil counts, pulse rate, INR, and hs-cTnI (Figure 3C and 3D).

1-Year All-Cause Readmission Models

The discrimination of different models for 1-year all-cause readmissions represented by AUCs is shown in Figure 1C. The SVM method achieved the best performance in terms of 1-year readmission prediction (AUC 0.96). XGBoost showed the lowest Brier score for calibration plots (0.12), followed by RF (0.13), SVM (0.16), ANN (0.18), and LR (0.24) (Figure 2C and Table 4).

Of the 79 variables analyzed, the presence of diabetes was identified as the most important marker for prediction of 1-year all-cause readmission using the SVM method, whereas basophil count was a significant predictor of readmission using the XGBoost algorithm. There appeared to be a certain discrepancy in the ranking of other features derived by the two methods (Figure 3E and 3F).

Mortality Risk Assessment Model

The patients were divided into subgroups of high-, intermediate-, and low-risk in-hospital mortality according to the cutoff value of various variables obtained using the XGBoost algorithm (Figure 4). Of the 79 variables, the hs-cTnI level (<0.068 μg/L) was identified as the first single predictor to discriminate between deceased and surviving patients. The next discriminator in the left node of hs-cTnI was the percentage of lymphocytes at a discrimination level of <14.688%; conversely, the next discriminator in the right node of hs-cTnI was neutrophil count at a cutoff value of less than 4.870×10⁹/L. Subsequently, the patients with HF were stratified by these branch points: high risk (hs-cTnI <0.068 μg/L, percentage of lymphocytes <14.688%, and cholinesterase <187.916 U/L); low risk (hs-cTnI <0.068 μg/L, percentage of lymphocytes ≥14.688%, and urea <10.113 mmol/L); intermediate risk 1 (hs-cTnI ≥0.068 μg/L, neutrophil count ≥4.870×10⁹/L, and AST <29.003 U/L); and intermediate risk 2 (hs-cTnI ≥0.068 μg/L, neutrophil count <4.870×10⁹/L, and respiratory rate <18.003 breaths/min).

‎View this figure

Discussion

Principal Findings

In our study, the EHR data–driven prognostic models based on 5 different ML algorithms were developed for predicting 1-year in-hospital mortality, use of positive inotropic agents, and all-cause readmissions within 1 year for patients with HF in a single Chinese class A tertiary comprehensive hospital. These ML models produced better predictive values for prognostic outcomes by growing interpretability than traditional linear methods. Besides, the novel ML techniques could take advantage of a large scale of complex, high-dimensional variables to widen the scope of HF predictive indicators concerning prognostic outcomes.

An integrated EHR system comprises various data resources, including patients’ demographic data, diagnostic information, laboratory test results, and prescriptions. However, many experts revealed that the availability of EHR-derived data is a prerequisite for promoting real-world studies [31,32]. In our study, the structured EHR data from over 10,000 patients in different years can be used directly for analysis without bias of data collection and clinical definitions. A total of 79 variables were applied from EHRs, including demographic information (n=4), current and previous disease history (n=8), HF diagnosis (n=3), vital signs (n=5), laboratory measurements (n=48), interventions (n=4), and medications (n=7). In general, the final target of using rich data derived from mature EHR systems is also patient outcome predictions. With the advancement of data science, different kinds of ML techniques have been broadly employed for model training because of their deep data processing and decision-making capabilities [33]. Compared with previous studies using the traditional LR method [11,20], our results provide better discrimination of risk for prognostic outcomes in a population of patients hospitalized for HF by using ML approaches. The strength of the algorithm could establish complex models and make accurate decisions in relevant big data sets. Besides, far more variables were allowed for modeling by using ML algorithms. Given the 5 methods in the present study, LR was considered the commonly used method in various fields of medicine [12,18]. The RF algorithm involves multiple decision tree creations that identify important predictive features with better accuracy in processing large numbers of highly nonlinear data [20,34]. ANN is a complex network method connected by a large number of simple neurons and simulates the human brain in parallel processing and nonlinear transformation [35]. The SVM approach is designed to build a good classifier using a nonlinear decision boundary between classes of variables that enables the labels from one or more feature vectors [36]. From the perspective of AUC values, it is important to mention that XGBoost performed better than the other 4 methods in predicting 1-year in-hospital mortality, use of positive inotropic agents, and 1-year all-cause readmission in patients with HF. The method’s dominant advantages are its ability to deal with missing values and integrate the power of weaker classifiers by creating combined and weighted variables. Relevant parameters were set to particular values.

Several traditional prognostic models have been established to estimate mortality in hospitalized patients with HF. Our results are consistent with the previous studies of 1-year mortality prediction while demonstrating a better predictive capability. Our study, which predicted outcomes for 13,602 patients with HF using 79 predictive variables, indicated that the ML approaches are superior to the EFFECT method (AUC 0.77) in predicting mortality, with an average AUC of 0.81 [10]. The classic SHFM study showed mortality prediction results within 1 year with AUCs of 0.729-0.76 [37], although the AUC value can be enhanced to 0.78 by combining the SHFM with the BNP level [38]. Furthermore, variables known to be associated with mortality in patients with HF were demonstrated in various previous models. Age, SBP, and level of blood urea nitrogen (BUN) were confirmed as the strongest independent predictors by the EFFECT model [10]. In the landmark Randomized Evaluation of Mechanical Assistance for the Treatment of Congestive Heart Failure (REMATCH) trial model [39], platelet counts, albumin, INR, and AST were associated with 1-year mortality in the HF population with implantable devices. The New York Heart Association class; use of beta-blockers, ACEIs, or ARBs; and presence of heart valve disease or atrial fibrillation remarkably influenced all-cause mortality in the Cardiac and Comorbid Conditions Heart Failure (3C-HF) score [40]. Our study obtained similar findings, demonstrating that in addition to specific biomarkers of HF or myocardial damage, renal function, coagulation indicators, and inflammation indicators are also critical factors regarding prognosis. Therefore, perhaps adopting a large scale of EHR-derived data could enhance discrimination and the predictive range of prognostic outcomes.

Readmission rate is a common index used to assess the quality of health care services for patient populations. Currently, most hospitals and institutions still implement traditional readmission risk models and certain variables to infer readmission probability [41]. Like previous prediction models, we found that the accuracy of mortality prediction was moderate, but the model was relatively poor at predicting readmission rates. Golas et al [42] suggested that the overall 30-day readmission rate was not improved by various ML algorithms, with AUC values around 0.66. Even compared with the established models for HF readmission (AUCs 0.6-0.7), our best ML model is still encouraging [43]. The variables that influenced mortality in the ML models were different from those of readmission among patients with HF. We believe that conventional diagnostic biomarkers (BNP) and vital signs (respiration rate and temperature) could dominate readmission prediction. However, diabetes, coronary heart disease, and gender were the top 3 significant features for predicting readmission using the SVM method. At an interpretable level, a high model AUC does not necessarily indicate that it best fits the scenario.

Positive inotropic agents are a kind of drug that can increase myocardial contractility and cardiac output and are often used to treat patients with HF. Among the drugs included in our study, dopamine and dobutamine have mainly inotropic effects, whereas milrinone, lanatoside C, and levosimendan have extra vasodilation functions [44]. However, they have been gradually limited because of poorer outcomes of patients after accepting inotropic therapy. Thus, controversy still exists regarding their reasonable use in clinical practice. Our study was found to have better performance than two other prognostic models in predicting the use of positive inotropic agents (AUCs in the range of 0.87 to 0.96). Furthermore, this study’s findings are consistent with Aljundi et al [45], who reported that the conventional cardiac biomarker BNP was predictive of inotropic agent use. Many studies have confirmed that comorbidities—such as dyslipidemia, chronic renal and liver impairment, and hyperglycemia—are associated with a higher likelihood of accepting inotropic treatment. Other critically associated variables (INR, AST, and basophil count) were obtained from our models and showed consistent results with common clinical thought. Our models provided good discrimination among patients at high risk for mortality, use of positive inotropic agents, and readmission. Incorporating more variables based on traditional models is of great significance to public health transformation for early identification and individualized intervention of people at high risk of HF outcomes.

The study’s strength was that we also developed a stratified risk assessment tool for the prediction of 1-year in-hospital mortality using the XGBoost algorithm. Far more features—including hs-cTnl, percentage of lymphocytes, percentage of neutrophils, cholinesterase, urea, respiratory rate, and AST—were identified for the first time using an ML approach for mortality prediction. Neutrophilia was reported to be associated with an increased incidence of acute decompensated HF (ADHF) in patients with acute myocardial infarction, and lymphopenia is related to poor prognosis in patients with HF [46,47]. Results from Seo et al [48] showed that cholinesterase was a simple marker for predicting adverse outcomes in patients with ADHF and tended to provide more accurate prognostic information than other objective nutritional features. These results can be attributed to the dimensionality and breadth of EHR-based data, facilitating the real-world study of HF in risk stratification, decision making, and disease management from multiple perspectives. Various HF risk models for predicting mortality have been developed abroad. The ADHERE risk tree by regression analysis [12] has demonstrated that patients with ADHF at low, intermediate, and high risk for in-hospital mortality can be easily identified using BUN, SBP, and creatinine obtained on hospital admission. The MUSIC risk model [49] showed that risk markers including atrial fibrillation, hyponatremia ≤138 mEq/L, N-terminal pro-brain natriuretic peptide >1.000 ng/L, and troponin positive were associated with cardiac mortality in a real-life setting. However, a limitation of the previous model is that it was based on a specific population of patients with HF (ie, symptomatic chronic HF). It should be noted that the etiology, clinical characteristics, and treatments of different phenotypes of HF are quite distinct. Therefore, we compiled all types of patients with HF in this study; it follows that the risk factors predicting mortality are likely to have been more comprehensive and provide new insights for further studies in specific subgroups. These markers possibly provide a more accurate risk evaluation of patients with HF, allowing early implementation of the appropriate intervention in daily public health practice, which leads to better outcomes in patients with HF.

There are several limitations to our study that are worth mentioning. First, this was an in-hospital outcome prediction study based on retrospective use of EHR-derived data. Although our models’ performances were considerable on their own, the predictive power could be further adjusted and compared with the established reference tools. Second, the number of patients who died was small compared with the number of surviving subjects; although rich in terms of clinical variables, the imbalance problem remained. Third, validation in an external cohort was not done in the present study but is planned for subsequent analysis. Fourth, different phenotypes of patients with HF should be taken into account in further model developments.

Conclusion

EHR-driven models, using novel ML algorithms, were developed to predict 1-year in-hospital mortality, use of positive inotropic agents, and 1-year all-cause readmission in patients hospitalized with HF. The discrimination and performance of our models also outperformed the existing tools constructed using traditional techniques. Besides, identifying a greater range of variables can further improve decisions regarding risk assessment for patients with HF.