An Artificial Intelligence Model for Predicting Trauma Mortality Among Emergency Department Patients in South Korea: Retrospective Cohort Study

Introduction

Trauma is the foremost cause of mortality worldwide, especially for those aged under 45 years [1]. Despite recent advances, trauma-related mortality remains a substantial challenge. In the trauma system, the emergency department (ED) is the hospital’s first contact and is vital for allocating medical resources [2]. Decision-making in the ED is crucial and determines further treatment or diagnosis [2,3]; thus, decisions should be made promptly and accurately to reduce the “golden hours” for treatment of a patient with severe trauma. Predicting ED mortality is critical for improving the trauma system and reducing individual, medical staff, and societal burdens. Injury severity, patient demographics, prehospital care, ED trauma care quality, and other complex and multifactorial elements that affect ED mortality pose challenges at gaining an in-depth understanding of the key factors contributing to ED mortality as a whole. Furthermore, ED mortality encompasses considerably more severe injuries than in-hospital mortality, and most patients that die in the ED may not receive additional diagnostic workups or treatments such as computed tomography (CT), angiography, or surgery. Due to hemodynamic instability, a complete workup is nearly impossible for unstable ED trauma patients. Therefore, ED mortality prediction depends on limited information, which could be facilitated by recent advancements in artificial intelligence (AI) technology.

Our research team previously developed two AI models for predicting in-hospital trauma patient mortality [4,5]; one utilizes Abbreviated Injury Scale (AIS) codes and the other incorporates the International Classification of Diseases, 10th revision (ICD-10) codes and other variables from the Korean National Emergency Department Information System (NEDIS) data set. Deceased ED patients were excluded from these models, as we assumed they may have received an insufficient workup or indicate an inaccurate diagnosis in previous studies. These AI models exhibited high accuracy for predicting in-hospital mortality but did not effectively learn from deceased ED patient data.

We deemed that ED patients who died before admission sustained severe injuries and that their information was insufficient compared to that available for patients who died after intensive care unit (ICU) or ward admission. We concluded that ED mortality should be predicted using a different patient data set than in-hospital mortality, and that the new model should also have alternative weights and input variables. Therefore, in this study, we developed AI models for predicting ED mortality in trauma patients using the NEDIS data set that was not incorporated into the previous AI models [4,5].

Methods

Patients and Data Set

The Korean National Emergency Medical Center has gathered NEDIS data from over 400 hospitals in Korea since 2016. This study employed the 2016 to 2019 NEDIS data set (data acquisition number N20212920825) to develop an AI model for predicting trauma mortality among all patients visiting the ED according to the Transparent Reporting of a Multivariable Model for Individual Prognosis or Diagnosis (TRIPOD) statement [6]. Data used for the AI model included those related to patients experiencing physical trauma (but not psychological) with an “S” or “T” diagnostic code from the ICD-10 (N=7,664,443); the S code represents trauma in a single body region, whereas T signifies trauma in multiple or unspecified regions. All ED patient data with an S or T diagnosis code were included, regardless of ward or ICU admittance or discharge.

Patient data with the following conditions were excluded: (1) patients who died before or upon hospital arrival (n=9506, 0.12%), who were regarded as dead-on-arrival and did not undergo any treatment or cardiopulmonary resuscitation; (2) ED patients who died due to severe conditions without further treatment, such as cardiac arrest without receiving cardiopulmonary resuscitation, despite being brought to the hospital alive (n=2800, 0.04%); (3) nontrauma patients (n=156, 0.002%); (4) patients transferred to another hospital from the ED (n=124,180, 1.62%); (5) patients with end-stage disease such as cancer who wished to receive care at home (n=2050, 0.03%); (6) patients who had to leave the hospital because there was no hope of recovery (n=270, 0.003%); (7) voluntarily discharged patients (n=162,499, 2.12%); (8) patients for whom it was difficult to record data due to being unidentifiably detained for criminal activity (n=29,242, 0.38%); (9) patients without S or T diagnostic codes, since all physical trauma receives an S or T diagnostic code; and (10) patients with frostbite (ICD-10 T33-T35.6), intoxication (ICD-10 T36-T65), or unspecified injury or complication diagnostic codes (ICD-10 T66-T78, T80-T88) (n=797,434, 10.4%). Our AI model was trained and tested using data from 6,536,306 patients. Figure 1 presents the patient selection process.

‎

AI Model Variables

This study followed Developing and Reporting Machine Learning Predictive Models in Biomedical Research guidelines [7]. We used the following 14 NEDIS data variables for AI model input: age; gender; intentionality; injury mechanism; emergent symptom; Alert, Verbal, Painful, Unresponsive (AVPU) scale; initial Korean Triage and Acuity Scale (KTAS); systolic blood pressure; diastolic blood pressure; pulse rate per minute; respiratory rate per minute; body temperature; oxygen saturation; and ICD-10 codes. Moreover, intentionality includes six categories: accidental/unintentional, self-harm/suicide, violence/assault, other specified, unspecified, and missing data. The injury mechanism comprises 16 categories: car accident, bike accident, motorcycle accident, other traffic accidents, unspecified traffic accident, fall, slipped, struck, firearm/cut/pierce, machine, fire/flames/heat, drowning, poisoning, choking/hanging, others, and unknown. Emergency and nonemergency are the two emergent symptom categories. The AVPU scale is a simplified version of the Glasgow Coma Scale (GCS) [8,9] and includes four categories: A, alert; V, verbal responsive (drowsy); P, painful response (stupor, semicoma); and U, unresponsive (coma).

The KTAS is a standardized triage tool that avoids complexity and ambiguity by employing five categories: Level 1, resuscitation; Level 2, emergent; Level 3, urgent; Level 4, less urgent; and Level 5, nonurgent. The KTAS was initially developed as an ED severity triage in 2012, based on the Canadian Triage and Acuity Scale [10]. According to NEDIS policy, a certified faculty member must conduct the initial KTAS within 2 minutes of ED admission. ICD-10 codes starting with S or T entail 865 categories. All AI model variables are summarized in Table S1 of Multimedia Appendix 1. The mortality of patients visiting the ED was defined as a patient with a dead result code or an indication of discharge with medical futility in NEDIS.

Data Split and Cross-Validation

Training and testing data used in this study are detailed in Table S2 of Multimedia Appendix 1. Data from 6,536,306 patients were divided into training (n=5,229,008) and testing (n=1,307,252) data sets with an 8:2 ratio in a stratified fashion. Only the testing set was used to test our developed AI model. We first performed 5-fold cross-validation in the training data set to assess how the prediction results generalize to an independent data set. The training data set was randomly shuffled and stratified into five equal groups, of which four were selected for training and the remaining group was used for internal validation. This process was repeated five times by shifting the internal validation group.

Handling Data Imbalance

The data were severely imbalanced as there were 6351 (0.1%) deceased patients recorded (Table S2 in Multimedia Appendix 1). To minimize the model bias toward a majority (ie, the survived patient group), we used the synthetic minority oversampling technique (SMOTE) [11] to upsample the deceased patient quantity and match the survived patient group. Next, we identified each group’s optimized weight values for the loss function during model training. We then iteratively learned demographic parity-based coefficients for the weight value search [12], providing a closed-form expression for the data weights. These two methods prevented bias toward survived patient data.

Machine Learning Models

We used three feature sets to develop our AI model: the first set included all 921 NEDIS variables and 878 features from ICD-10 codes, the second set only used the 878 features from ICD-10 codes, and the third set utilized all features except those from ICD-10 codes. We applied eight machine learning models from each feature set: adaptive boosting (AdaBoost) [13], extreme gradient boosting (XGBoost) [14], light gradient boosting machine (LightGBM) [15], gradient boosting machine (GBM) [16], extremely random trees (ERT) [17], logistic regression (LR) [18], random forest (RF) [19], and deep neural network (DNN). We chose the best three among the eight models and applied an ensemble approach by considering all possible combinations. Finally, we evaluated the feature importance, listing features in the order they contributed to mortality prediction.

In decision tree approaches such as AdaBoost, XGBoost, and LightGBM, calculating feature importance values hinges on assessing the decrease in node impurity while factoring in the probability of reaching each node. Node impurity is determined using a well-established metric called the Gini index, which quantifies the impurity degree at a given node by measuring the extent to which a specific variable would be incorrectly classified if selected randomly. The impurity is evaluated by considering the weighted sum of each class’s squared probabilities within the node. This comprehensive methodology incorporates node probabilities, impurity measures, and feature importance calculations, allowing for a nuanced understanding of how different variables impact prediction outcomes and can offer valuable insights for decision-making and analysis.

Performance evaluations were based on 5-fold cross-validation using the following metrics: sensitivity, specificity, accuracy, balanced accuracy, and area under the receiver operating characteristic (ROC) curve (AUROC). Due to the significant data imbalance, we used balanced accuracy as the primary model evaluation metric.

The models used Python (version 3.7.13), NumPy (version 1.21.6), Pandas (version 1.3.5), Matplotlib (version 3.5.1), and Scikit-learn (version 1.0.2). All statistical analyses were performed with R software version 4.1.2 (R Foundation for Statistical Computing). Continuous variables are presented using the mean and standard deviation, while categorical data are presented using proportions. Statistical continuous data comparisons were performed using the Student t-test or the Mann-Whitney U test as appropriate. Similarly, proportions were compared through χ² or Fisher exact tests as appropriate. A two-sided P value <.05 was considered statistically significant.

Conventional Metrics

To further evaluate the performance of our AI model, we implemented ICD-10–based conventional metrics for comparison: inclusive survival risk ratio (SRR), exclusive SRR, and KTAS. The ICD-based Injury Severity Score (ICISS) utilizes the SRR to calculate survival probability [20]. SRRs can be quantified as the number of survived patients with a specific injury code divided by all patients with the same code. Patient survival probability was determined by multiplying all patient injury code SRRs [20]. The traditional ICISS was calculated as the survival probability product for up to 10 injuries [21]. There are two different SRR calculation approaches: inclusive and exclusive SRR. Inclusive SRR can be calculated for each injury without considering the associated injury. In contrast, exclusive SRR divides the number of survivors with an isolated specific injury by the total number of patients with only that injury. Thus, patients with multiple injuries were excluded from exclusive SRR calculations [20]. We used the survival probability determined from our previous study [5] because other studies [20,21] did not use ED mortality.

Ethical Approval

The Institutional Review Board of Wonkwang University Hospital approved this study (WKUH 2019-11-004-001). The requirement for informed consent was waived due to the study’s observational nature and the deidentification of each patient.

Results

Patients

Table 1 shows a comparison of the main variables between the deceased and survived patients among all patients visiting the ED, whereas Table S3 of Multimedia Appendix 1 compares ICD-10 codes between deceased and survived patients (878 features).

K-Fold Cross-Validation

Table 2 summarizes the 5-fold cross-validation results. The AdaBoost model with all 921 features achieved the highest balanced accuracy (0.9801) and AUROC (0.9973) values among the seven models: XGBoost, LightGBM, AdaBoost with XGBoost, AdaBoost with Light GBM, XGBoost with LightGBM, and the three models combined. In addition, the 921-feature model provided higher accuracy metrics than the 878-feature model with the ICD-10 codes and the 43-feature model excluding the ICD-10 codes. Interestingly, the model with all features except for the ICD-10 codes reached higher accuracy metrics than the 878-feature model with the ICD-10 codes.

These results substantiate that patient information, symptoms, and trauma causes predict mortality better than ICD-10 codes. We also compared the performance of traditional methods. Inclusive SRR, exclusive SRR, and KTAS exhibited lower balanced accuracies (0.9069, 0.9175, and 0.9619, respectively) and AUROCs (0.9345, 0.9554, and 0.9372, respectively).

Ranked Feature Importance: Explainable AI

Next, we conducted a feature importance analysis to confirm each feature’s contribution. Figure 2 ranks the normalized feature importance from the AdaBoost model, which showed the best performance. Age and systolic blood pressure were the top two mortality predictors among visiting ED patients, followed by unresponsiveness (coma), pulse rate per minute, oxygen saturation, KTAS Level 5, S224 (multiple rib fractures), respiratory rate per minute and painful response (stupor and semicoma). Interestingly, only 49 among the 921 features had nonzero importance values, indicating that the other 872 features did not contribute to mortality prediction. Table S4 in Multimedia Appendix 1 summarizes the total ranked normalized feature importance values.

‎

Testing Data

Table 3 summarizes the results from the isolated testing data set results (n=1,307,252). The testing data results corroborate that the 921-feature AdaBoost model achieved the highest balanced accuracy (0.9813) and AUROC (0.9974) values among the seven models: XGBoost, LightGBM, AdaBoost with XGBoost, AdaBoost with Light GBM, XGBoost with LightGBM, and the three-model combination. Similar to the cross-validation results, the 921-feature model provided higher accuracy metrics than the 878-feature model with ICD-10 codes and the 43-feature model excluding ICD-10 codes. Furthermore, our selected model also performed better than traditional inclusive SRR, exclusive SRR, and KTAS methods (see Table S5 in Multimedia Appendix 1). The similarity between the cross-validation and testing data results denotes minimal overfitting or underfitting. Figure 3 depicts the ROC curve comparison, including comparison of the selected AdaBoost model and the three traditional models (left) and the selected AdaBoost model relative to the 921-, 878-, and 43-feature models (right).

Finally, we compared the performance of the cutting-edge AI model, as it was designed for in-hospital mortality predictions based on the NEDIS data set [5]. The in-hospital mortality model’s balanced accuracy and AUROC values were lower, regardless of the feature set. Notably, the in-hospital mortality model with all 921 features obtained 0.9614 balanced accuracy and 0.9929 AUROC values, and the model with 43 features excluding ICD-10 yielded a 0.9648 balanced accuracy and 0.9923 AUROC. Interestingly, when using the in-hospital mortality model with only 878 features and the ICD-10 codes, the values significantly dropped to a 0.6298 balanced accuracy and 0.7675 AUROC.

‎

AI-Driven Public Website Deployment

We launched our final AI model on a public website [22] to allow access to the mortality prediction results among visiting ED patients. Figure S1A in Multimedia Appendix 1 displays the web interface for entering information. A user inputs age, gender, intentionality, injury mechanism, emergent symptoms, AVPU scale, initial KTAS, systolic blood pressure, diastolic blood pressure, pulse rate per minute, respiratory rate per minute, body temperature, oxygen saturation, and ICD-10 codes. For ICD-10 codes, a user can input multiple codes with commas (eg, S072, S224, T083). After entering information into the web app, the user can obtain the mortality prediction results (see Figure S1B in Multimedia Appendix 1 for an example). The prediction results also include the mortality probability.

Towards Comprehensive Model

Our previous study [5] revealed that blood pressure, heart rate, body temperature, and other vital signs weakened the previous AI model’s performance, whereas incorporating vital signs strengthened our present AI model. This observation implies that ED and in-hospital mortality patients exhibit differing data distributions. Patients who died after ICU or ward admission and not in the ED were labeled survivors in the present AI model; therefore, the present AI model predicts a more severe mortality type. In future studies, we plan to design a more comprehensive model incorporating both in-hospital and ED mortality. We propose a new pipeline for predicting ED and in-hospital mortality based on our present AI model and the previous two models [4,5] (Figure 4).

‎

Discussion

Principal Findings

This study developed an AI model that accurately predicts mortality among visiting ED patients. Our final AI model achieved substantially high metrics of 0.9738 sensitivity, 0.9888 specificity, 0.9887 accuracy, 0.9813 balanced accuracy, and 0.9974 AUROC. In addition, the proposed AI model outperformed traditional inclusive SRR, exclusive SRR, and KTAS models and the previously developed in-hospital mortality prediction AI model [5]. We identified several significant mortality predictors through the feature importance analysis, including age, AVPU scale, multiple rib fractures, lumbar vertebra fractures, and KTAS Level 2. Furthermore, we devised a risk calculator leveraging our AI model, demonstrating its substantial clinical potential for triage and diagnosis.

We compared AdaBoost, XGBoost, LightGBM, GBM, ERT, LR, RF, and DNN models. After determining that AdaBoost, XGBoost, and LightGBM models were the top three, these models were combined as an ensemble approach. During the evaluation, we considered balanced accuracy as the primary metric due to the data imbalance. Using the best three single models included the boosting algorithm principle, another ensemble approach. Boosting incorporates the stagewise addition method, where multiple weak models are trained and combined into one stronger model. Specifically, the AdaBoost grows decision trees as weak models and adds penalties or weights to the incorrectly predicted samples. After each prediction stage, this process assigns higher weight values to the misclassified samples.

Although XGBoost and LightGBM also utilize boosting algorithms, they differ slightly from AdaBoost regarding the gradient boosting algorithm. The gradient boosting algorithm adjusts the new prediction model using the previous model’s residual errors. Thus, XGBoost computes the residuals and builds decision trees by selecting features, finding optimal splits, estimating leaf node values, and applying regularization. The predictions are updated and the residuals are recalculated at each iteration. LightGBM resembles XGBoost in many aspects, but it has a faster execution rate and maintains high accuracy levels with gradient-based one-side sampling (GOSS) and exclusive feature bundling. In contrast, XGBoost uses a presorted and histogram-based algorithm for computing the best split with GOSS in LightGBM.

We also assessed GBM during the initial single-model evaluation. Both LightGBM and GBM models are founded on gradient boosting frameworks but differ in tree construction, feature discretization, gradient computation, and memory usage. Notably, each machine learning model performed better under different data characteristics. For instance, LR may work well with a linear association between features and a target, whereas a decision tree is more effective for nonlinear relationships. Other models with similar GBM and LightGBM operations can also produce different prediction outcomes due to slightly different tree constructions and data characteristics. Therefore, we evaluated various machine learning models to identify the best-fit solution for a particular problem type: predicting trauma mortality for all patients visiting the ED.

Finally, we enhanced prediction performance by considering all possible combinations based on the three best models. In our final ensemble model, we found the optimal hyperparameters with a maximum depth of 6, 0.01 learning rate, and 400 tree estimators. AdaBoost’s optimal hyperparameters were 400 tree estimators, 0.1 learning rate, a maximum depth of 1, and balanced class weights. Comparatively, LightGBM’s optimal hyperparameters were a maximum depth of 3, 0.01 learning rate, and 400 tree estimators.

Our AI model has several clinical practice advantages. First, our proposed model best predicts severe trauma patients with wounds that may not survive admission or surgery, with most unable to undergo CT or magnetic resonance imaging. Moreover, our feature importance analysis indicates different prominent diagnostic codes than the previous AI model for predicting in-hospital mortality. This finding implies that ED mortality patients have distinct clinical data distributions from those with in-hospital mortality. We also used the present test data set to assess the previous in-hospital mortality AI model [5]. We discovered that the current model was superior, likely due to our current training data set’s ED mortality specificity. Second, we did not use AIS codes requiring an expert or exact diagnoses such as CT or surgery; thus, it is not a time-consuming process. All variables in our AI model are available promptly after a portable X-ray or point-of-care ultrasound. Finally, our AI model’s excellent performance would efficiently allocate medical resources. As our risk calculator tool is accessible through a mobile web app, clinicians can utilize it without time or location limitations.

We used 5,229,008 patient data sets for model training and cross-validation and 1,307,252 data sets for evaluation. To our knowledge, this is the most extensive study on developing an AI model for predicting mortality in trauma patients visiting the ED by incorporating ICD-10 codes and other clinical variables; such a large data set contributes to establishing generalization. In addition, we used SMOTE and optimized class weight search techniques to minimize an imbalanced data distribution. These techniques resolved the overfitting issue by reducing the cross-validation and testing data result difference.

Our team previously developed two AI models for predicting in-hospital mortality from various input features [4,5]. In our first study [4], we rearranged AIS codes relative to 46 anatomical regions, which was considerably more differentiated than the Injury Severity Score (ISS) system’s conventional six regions. We anticipated that the AI model would provide more appropriate weights for each anatomical organ such as the pancreas, rib, or liver. Our second study [5] used the NEDIS data set without any AIS information. Therefore, we used ICD-10 codes, procedure codes, KTAS, and other clinical variables.

We excluded ED mortality patients in both studies as they may have received insufficient diagnostic workup. Postmortem CT or autopsy is not popular in South Korea. Patients discharged from the ED and returning home may also undergo insufficient diagnosis compared to admitted patients. Additionally, some patients may die after ICU or ward admission. We defined these patients as not indicative of ED mortality. ED mortality patients may have more severe injuries that prohibit admittance. Thus, we postulated that ED mortality patients must be predicted using alternative features.

Two recent systematic reviews compared machine learning models for predicting mortality and decision support [23,24]. Zhang et al [24] discussed six studies using machine learning models based on the national database. However, their training sets ranged from 12,640 to 799,233 patients [25-30]. Our model incorporated data from over 6.5 million patients, establishing this as the largest cohort study. Moreover, our model achieved the highest performance (AUROC of 0.99) compared to previous studies with less accurate performance (AUROC of 0.89 to 0.95) [25-30]. Kim et al [26] introduced a neural network model using 408,316 patients from the National Trauma Database with a primarily ED mortality outcome. They reported an AUROC of 0.86 for the neural network using age, systolic blood pressure, respiration rate, heart rate, and GCS or simplified consciousness scores [26] and an AUROC of 0.93 when incorporating ISS. Our present outcome (AUROC of 0.99) dramatically outperforms that of the previous study. Furthermore, the ICD-10 code is more practical than the ISS regarding ED because ISS requires an expert such as a trauma coordinator, whereas any clinician can determine an ICD-10 code. In a validation study including 934,053 patients from the American College of Surgeons Trauma Quality Improvement Program database, Maure et al [29] reported an AUROC of 0.93 in penetrating injury and 0.88 in blunt injury for predicting mortality. However, they also excluded ED mortality, similar to our previous AI models [4,5].

Limitations and Future Work

Several limitations of this study are acknowledged. First, this is a retrospective study despite the substantial quantity of patient data. Therefore, a further prospective study is needed to avoid potential selection and survival bias. Second, we did not perform external validation. Our study only used training data derived from patients in South Korea; thus, it is unclear whether our model could be adapted to other countries. In future work, we plan to conduct an external validation study using data from another country to develop a global version of the model. Third, our primary outcome was ED mortality, not in-hospital or overall mortality. Since our previous AI model [5] excluded patients with ED mortality, our two AI models would help clinicians predict various mortality types. Fourth, some data were provided as categorical variables in NEDIS, such as age. However, a sufficiently large data set may enable us to overcome this issue. Fifth, NEDIS did not include AIS or ISS, as we could not locate this information. In future work, we plan to develop an ensemble model incorporating our previously developed AIS code–based AI model [4].

Conclusions

Our proposed AI model for predicting ED mortality achieved exceptionally high accuracy. This model is derived from a population-based data set in South Korea and provides better insight into trauma care and systems, complementing our previous AI models [4,5]. In future studies, we must consider data from various ethnic groups and integrate our previous AI models.