Enhancing Performance of the National Field Triage Guidelines Using Machine Learning: Development of a Prehospital Triage Model to Predict Severe Trauma

Introduction

Trauma is a universal health challenge that places a massive burden on national economies. It causes 4.4 million deaths annually, and an estimated 10% of all years lived with disability [1,2]. The American College of Surgeons Committee on Trauma (ACS-COT) recommends that severe trauma be treated at levels 1 and 2 trauma care facilities [3]. Patients with severe trauma have approximately 25% lower mortality rates when treated at levels 1 or 2 trauma centers than when treated at lower-level or nontrauma centers [4]. Prehospital estimation of injury severity is essential for prehospital triage. It is a critical step for emergency medical service (EMS) providers in making decisions regarding patient destination. Under- and overtriage are incorrect triages. A low-sensitivity triage tool results in many false-negative cases, indicating undertriage and a possible failure in trauma first aid. Conversely, low specificity is associated with a high rate of false-positive cases, indicating overtriage [5].

The national field triage guidelines were initially developed by ACS-COT in 1976 and revised in 2011 and 2021 [6,7]. The national field triage guidelines have been widely implemented in the United States and represent 1 of the few standardized national protocols for EMS. It was developed based on peer-reviewed research, resulting in biased estimates and reduced generalizability [3,8]. A prospective national triage guidelines validation study for identifying high-risk trauma patients found that the guidelines were relatively insensitive in identifying severely injured patients and those requiring early critical resource use [9]. In addition, other triage tools, such as the Glasgow Coma Score (GCS), Prehospital Index (PHI), and revised trauma score (RTS), have not shown ideal predictive performance [5,10-12]. Therefore, it emphasizes the need to improve the prehospital triage tool [13].

Machine learning (ML) development has advanced rapidly in the medical field, notably in trauma medicine, and has demonstrated that the ML model’s predictive ability is significantly better than that of the conventional trauma triage tools for mortality outcomes, hospitalization, and critical care admission [14,15]. Therefore, this study aimed to build a prehospital triage model to predict severe trauma (pTEST) and enhance the performance of the national field triage guideline.

Methods

Recruitment

This multisite prediction study was conducted to predict trauma severity during field triage. We developed, validated, and reported our triage model following the TRIPOD (Transparent Reporting of a Multivariable Model for Individual Prognosis or Diagnosis) statement [16], as shown in Multimedia Appendix 1.

Ethical Considerations

The Naval Medical University Ethics Committee approved the study protocol (reference number NMUEC2022-088). Our study consisted of secondary analyses using the National Trauma Data Bank (NTDB) with primary consent, and the data were anonymized.

Source of Data and Patients

Data from the NTDB, the largest aggregation of trauma registry data in the United States assembled by the ACS, were used in this study [17]. In 2017, the ACS Trauma Quality Program transitioned to a new technical vendor and redesigned the NTDB infrastructure. The 2017 and 2018 NTDB datasets comprising 2,041,706 patients were used for pTEST model development, hyperparameter tuning, and internal validation. The 2019 NTDB dataset comprising 1,097,190 patients was used for the external validation of the pTEST model.

Notably, all patients with injury, aged 16 years of age or older, and transported by ground or aerial ambulance from the injury scene to any trauma center were potentially eligible. Due to a lack of crucial information on outcomes and predictors, we excluded patients who died in the EMS, patients discharged from the emergency department (ED) to another hospital, and those without any EMS records or an Injury Severity Score (ISS). In total, 960,443 and 508,703 participants were included in the development and validation sets, respectively. Figure 1 shows the patient selection process.

‎

Outcome and Predictors

Defining “severe trauma” is a challenge in prehospital triage model development. It varies widely across studies. The reference standard (primary outcome) of “severe injury” was considered as an ISS ≥16 as the benchmark to evaluate triage accuracy recommended by the ACS [7,18,19]. The ISS calculated by anatomical criteria is assumed to be consistent with the patient status on the scene and is associated with high mortality [4,19]. However, it does not reflect resource use directly; therefore, we included a composite resource-based early critical resource use measure as the secondary outcome. According to similar studies [9,18], early critical resource use included intubation in the EMS or ED, discharge to the intensive care unit from the ED, surgery for hemorrhage control, interventional radiology procedures, cerebral monitoring, and in-hospital death, all within 24 hours. A detailed definition of severe trauma is provided in Multimedia Appendix 2.

According to a recent field triage protocol review in 2017 [10], the significant predictors of a severely injured patient were age, vital signs, injury patterns, and injury mechanism. In addition, in the US National Guidelines for the Field Triage of Injured Patients in 2011 and 2021 [6,7], age, vital signs, and injury pattern measurements were the field triage’s top priorities. In the 2011 national field triage guidelines, severely injured patients who should be transported preferentially to the highest-level trauma center were identified using the RED criteria, which included 3 vital signs (GCS, systolic blood pressure, and respiratory rate) and 8 injury patterns. During field triage, time is essential, and the number and complexity of hand-collected variables must be limited. Therefore, as the US National Guideline for the Field Triage recommended and recorded in the NTDB, we incorporated 16 candidate variables in EMS for model development, including age at the time of injury (AGEYEARS), GCS eye (EMSGCSEYE), GCS motor (EMSGCSMOTOR), GCS verbal (EMSGCSVERBAL), systolic blood pressure (EMSSBP), oxygen saturation (EMSPULSEOXIMETRY), respiratory rate (EMSRESPIRATORYRATE), pulse rate (EMSPULSERATE), penetrating injuries (TCCPEN), chest wall instability (TCCCHEST), long-bone fractures (TCCLONGBONE), crushed extremity (TCCCRUSHED), amputation (TCCAMPUTATION), pelvic fracture (TCCPELVIC), skull fracture (TCCSKULLFRACTURE), and paralysis (TCCPARALYSIS). The detailed definitions of candidate variables are listed in Multimedia Appendix 3.

Model Development

The pTEST model was developed using the extreme gradient boosting model (XGBoost) and Shapley additive explanation analysis (SHAP). XGBoost is a novel boosting tree-based ensemble algorithm through which new models are created to predict residuals or errors of prior models and then combined to make a final prediction [20]. Recently, XGBoost has been widely used in ML due to its outstanding prediction performance, ability to use continuous and categorical inputs, lack of data preprocessing, imbalanced data handling capacity, high internal optimization, and relatively modest computational costs [21].

Patients in the development set from the 2017 and 2018 NTDB datasets were randomly grouped into training (70%) and internal validation (30%) sets for model development. Using a grid search, a 10-fold cross-validation process was used on the training set for hyperparameter tuning. The goal of hyperparameter tuning is to find the values that lead to the best-predicted performance. The optimal values of hyperparameters were learning rate=0.04071151, minimum loss reduction required to make a further partition=20.36485, maximum tree depth=14, minimum sum of instance weights needed in a leaf node=39, maximum number of boosting iterations=1051, and subsample ratio of the training instance=0.7763707, with the other hyperparameters set to default. In addition to the training set, the model’s reproducibility, transportability, and generalizability were evaluated using internal and external validation sets.

Missing values are an essential concern in trauma triage because there may not always be time to measure critical variables. The absent proportions of the training, internal validation, and external validation sets are shown in Multimedia Appendix 4, with all variables, except pulse oximetry, missing below 6%. XGBoost supports branch directions for predictors with missing values, creating an advantage in real-world situations where XGBoost can still achieve individual prediction without complete prehospital data.

To gain insight into the risk prediction model, we investigated different predictors’ contributions based on Shapley values using SHAP, a game theory concept introduced in the 1950s [22,23]. A predictor’s SHAP value can be positive or negative, suggesting an increased or decreased probability of severe trauma, respectively. In our study, the SHAP values were visualized at global (dataset level) and local (patient-specific) levels to investigate the predictors’ impact. XGBoost and SHAP were implemented using the R (R Core Team) packages tidy models and shapviz.

Statistical Analysis Methods

For the sample size calculation, the prevalence of events was set at 17.8%, and the number of predictors was 16 based on the development set. The area under the curve (AUC) of the optimal prehospital triage model in a previous study was 0.68 [11], and our pTEST model was expected to achieve an AUC of 0.7. At least 1871 patients were required for model development or validation using the R package pmsampsize [24].

Continuous data are presented as mean and SD, and categorical data are presented as frequencies and percentages (%). The 2-tailed t test was used to evaluate the differences in continuous data, which followed a normal distribution and variance homogeneity; otherwise, the Wilcoxon rank-sum test was used. The differences in categorical data were evaluated using the Pearson chi-square test. The area under the receiver operating characteristic curve was calculated to assess model discrimination. The AUCs between models were compared using the DeLong test. The best thresholds of the models were determined by maximizing the Youden index, and performance metrics, including sensitivity, specificity, accuracy, positive predictive value (PPV), and negative predictive value (NPV), were calculated. Performance metrics 95% CIs were calculated using 500 bootstrap replicates. In addition, the pTEST model is intended to identify severe trauma that requires high sensitivity and NPV to rule it out. Sensitivity and specificity are inversely proportional and a tradeoff needs to be made between sensitivity and specificity. Therefore, the sensitivities and NPV of the different models were compared using a 0.5 specificity as in the previous studies [25,26].

Our study defined the over- and undertriage rates. Overtriage rate (1-PPV) is equal to the number of patients with negative outcomes (ISS <16 or no critical resource use) predicted as positive outcomes divided by the total number of predicted positive outcomes. The undertriage rate (1-NPV) is equal to the number of patients with positive outcomes predicted as negative outcomes divided by the total number of predicted negative outcomes.

In addition, a calibration plot and Brier score were generated to assess how closely the predicted probability approximated the actual probability. The clinical benefit of the models was evaluated using a decision curve analysis method. The discrimination, calibration, and clinical benefit of the pTEST were compared with the GCS, PHI, RTS, and RED criteria of the 2011 National Field Triage Guidelines. Statistical significance was set at P<.05. All statistical analyses were performed using the R software (version 4.3.1).

Results

Patient Characteristics

Table 1 shows the patients’ baseline characteristics in the training, internal validation, and external validation sets. In these 3 sets, severe trauma proportions were 17.80% (119,690/672,309), 17.80% (51,296/288,134), and 17.08% (86,902/508,703), respectively, and critical resource use proportions were 29.36% (177,570/604,806), 29.56% (76,604/259,148), and 28.17% (129,551/459,843), respectively. Notably, most variables showed statistically significant differences among the 3 sets for large sample sizes, but the differences were minimal. The demographic characteristics, vital signs, and injury patterns of nonsevere and severe trauma are listed in Multimedia Appendices 5-7, and the attributes of noncritical and critical resource users are listed in Multimedia Appendices 8-10. Severe trauma and critical resource users are usually male, air-transported, taken to higher-level trauma centers, and have extreme trauma patterns.

Model Performance

For predicting severe trauma, we compared the performance metrics of the other models at the same specificity fixed at a moderate number of 0.5. The pTEST model showed a higher sensitivity of 0.799 (95% CI 0.797-0.801), a lower undertriage rate of 0.080 (95% CI 0.079-0.081), and a lower overtriage rate of 0.743 (95% CI 0.742-0.743) in the training set (Table 2). In addition, for critical resource use prediction fixed at a specificity of 0.5, the pTEST model showed a higher sensitivity of 0.774 (95% CI 0.772-0.776), lower undertriage rate of 0.158 (95% CI 0.157-0.159), and lower overtriage rate of 0.609 (95% CI 0.608-0.609) than the other models in the training set (Multimedia Appendix 11). We validated the pTEST model performance using 2 validation sets and obtained consistent results (Table 2 and Multimedia Appendix 11). The model performance metrics for predicting severe trauma and critical resource use at the best thresholds with the maximum Youden index are listed in Multimedia Appendices 12 and 13, demonstrating a higher pTEST Youden index than other models.

In Figure 2, pTEST AUCs for severe trauma prediction were 0.755 (95% CI 0.753-0.757), 0.751 (95% CI 0.749-0.754), and 0.750 (95% CI 0.749-0.752) in training, internal validation, and external validation sets, respectively, and the AUCs for predicting critical resource use were 0.736 (95% CI 0.734-0.737), 0.732 (95% CI 0.730-0.734), and 0.733 (95% CI 0.732-0.735), respectively, demonstrating better discrimination ability than GCS, PHI, RTS, and RED criteria. Multimedia Appendix 14 depicts the pTEST model predicted outcome probability as a waterfall plot. The calibration curves in Multimedia Appendix 15 show that the severe trauma predicted probability and pTEST critical resource use agreed with the proportion observed using the smallest Brier score. In Multimedia Appendix 16, pTEST provides a consistently higher net benefit across a broad range of risk thresholds (10%-100%) than the 2 default strategies and other models.

‎

Model Interpretation

As shown in the SHAP summary plots (Figure 3A), the contributions of the variables to the pTEST model for severe trauma prediction were evaluated using the average absolute SHAP values; the top 5 important variables were EMSGCSVERBAL, EMSSBP, EMSRESPIRATORYRATE, EMSGCSMOTOR, and EMSPULSEOXIMETRY. Figure 3B lists the impact of the different variables illustrated by the SHAP values for severe trauma prediction. Figure 3C shows each variable’s SHAP values versus measured values. Figure 3B, C shows that the higher the EMSGCSVERBAL score, the lower the probability of severe trauma (“negative” impact). Similarly, AGEYEARS, EMSGCSEYE, EMSGCSMOTOR, and EMSPULSEOXIMETRY negatively contributed to the predicted probability. In contrast, 8 injury patterns contributed positively to the predicted probability. The SHAP summary and dependence plots of the pTEST model for critical resource use prediction are shown in Multimedia Appendix 17. The personalized feature attributes for 2 representative patients with and without severe trauma in the training set are provided in Multimedia Appendix 18.

‎

Subgroup Analysis

In Multimedia Appendix 19, subgroup analyses were performed according to age, sex, transport mode, trauma type, and prehospital time. The pTEST model AUCs for severe trauma prediction in patients 60 years of age or older were relatively low at 0.717 (95% CI 0.714-0.720), 0.712 (95% CI 0.708-0.717), and 0.710 (95% CI 0.707-0.714) in the 3 sets, respectively. The AUCs in patients with penetrating injuries were relatively high at 0.815 (95% CI 0.810-0.820), 0.810 (95% CI 0.802-0.817), and 0.810 (95% CI 0.805-0.816) in the 3 sets, respectively. The high proportion of severe trauma in helicopter-transported patients (approximately 41%) led to a high undertriage rate (>0.2) and a low overtriage rate (<0.5). Multimedia Appendix 20 illustrates the pTEST model performance in critical resource use prediction in different subgroups.

Discussion

Principal Findings

In this multisite, large-sample study, we present a prehospital trauma triage tool, pTEST, for severe trauma prediction in EMS. To our knowledge, this is the first study combining ML with national triage guidelines. The pTEST performed optimally in internal and external validations. In addition, its diagnostic accuracy was evaluated using anatomical and resource-based outcomes, and the resource-based outcome is a better alternative to determine the need for specialized trauma care [27]. Furthermore, the pTEST was developed based on national triage guidelines, a globally adopted standard in many organizations. Therefore, the pTEST model can be conveniently applied in EMS practice and may have global relevance.

Comparison With Previous Studies

Consistent with previous studies [8], we also demonstrated the poor performance of the RED criteria from the national triage guidelines. Globally, all triage guidelines are based on a criteria checklist, including vital signs, injury type, and mechanism of injury [7,28,29]. These guidelines are simplistic—patients meeting any 1 of the criteria should be transported to the highest-level trauma center. In reality, there was an interaction among variables and nonlinear effects of continuous variables (Figure 3C). XGBoost, as a nonlinear ensemble method, can train a more accurate classifier from several weak classifiers and has other benefits, such as dealing with missing values and interaction, avoiding overfitting, and accelerating the training speed by parallel calculation [30]. Our study incorporated XGBoost into national field triage guidelines and developed the pTEST model to enhance performance. The pTEST model included age, 7 vital signs, and 8 injury patterns. We did not perform further variable selection because these variables are the most important in the national field triage guidelines, and the overall number is moderate for field triage. The pTEST model did not achieve an undertriage rate of <5% or an overtriage rate of <35%, as the ACS Committee on Trauma targeted. However, 2 aspects must be noted. First, the definition of the undertriage rate by the ACS is different from that used in our study. For example, the undertriage rate in ACS is equal to the number of patients with ISS ≥16 transported to a low-level trauma center or nontrauma center divided by the total number of patients transported to a low-level trauma center or nontrauma center [19]. In contrast, the undertriage rate in our study is equal to the number of patients with ISS ≥16 predicted as ISS <16 divided by the total number of patients predicted ISS <16. In the national field triage guideline, the patients predicted as ISS <16 can be transported to a low-level or nontrauma center. If the patients with predicted ISS <16 and ≥16 are transferred to low and high-level trauma centers, respectively, then the undertriage rate in our study is the same as that in ACS, but this is unlikely in practice. Second, the under- and overtriage rates are affected by severe trauma proportion. Since most (>90%) of our study population came from level 1 and 2 trauma centers, the severe trauma proportion was approximately 18%, resulting in undertriage rate overestimation and overtriage rate underestimation. Based on previous studies with good sample representativeness [9], we assumed that the proportion of severe trauma was 3%. Keeping the sensitivity (0.794) and specificity (0.5) of the pTEST model and the sample size (n=508,703) in the external validation set unchanged, the under- and overtriage rates were 1.26% and 95.3%, respectively, in the external validation set (Multimedia Appendix 20), meeting the ACS undertriage rate target.

The pTEST model and the national field triage guidelines in subgroup analyses were particularly insensitive among older adults [9]. Possible explanations include different physiological responses to injury [31], medication use that potentially worsens injury [32], high prevalence of frailty, and comorbidities [33]. Previous studies have explored elderly-specific triage criteria [15,34,35]. The pTEST model performed better for penetrating traumas. Notably, several studies have found that penetration is a strong severe trauma predictor, and severely penetrated injured patients are more easily recognized [10]. The undertriage rate was high in patients transported using helicopters. Patients with a high proportion of severe trauma, such as those experiencing large-scale casualties, should be transferred to high-level trauma centers without field triage to reduce the undertriage rate.

Previous studies have reported controversial results regarding the use of ML in medical prediction issues [36]. Overall, in studies with a limited number of predictors, ML does not demonstrate advantages over traditional models, such as logistic regression [37], whereas, for studies with many predictors, advanced ML may have an advantage [38]. A recent review, including 14 studies, demonstrated that the predictive ability of ML-based models was significantly better than that of conventional trauma triage tools for outcomes of mortality, hospitalization, and critical care admission, and XGBoost was the most commonly used ML algorithm [14]. In this study, the relatively large number of predictors and sufficient amount of data tended to favor ML applications. We built the pTEST model using XGBoost but did not evaluate other ML methods. We believe an excellent model can be created using a large sample size, an advanced ML method, and robust hyperparameter tuning. In addition, we minimized the risk of chance findings and overfitting by avoiding exploring other modeling strategies.

Limitations

This study had some limitations. First, most of our study population were from level 1 and 2 trauma centers, and the proportion of patients with severe trauma (approximately 18%) was significantly higher than that of all prehospital trauma patients (approximately 3%) [9]. However, unlike PPV and NPV, the sensitivity, specificity, and AUC of the pTEST model were not affected by the proportion of severe trauma, and the high sensitivity, specificity, and AUC objectively reflect the pTEST model’s good performance. In addition, some emergency resources may be unavailable in low-level trauma and nontrauma centers [39], and samples from high-level trauma centers make it possible to evaluate the pTEST model with critical resource use as the end point. Second, the pTEST model was not developed into a software application, as in other studies [40], because software development requires adaptation to existing information systems in EMS, which is a complex project. However, in the future, EMS providers can develop software based on available data and programs. Third, the 2017-2019 NTDB data followed the 2011 national field triage guidelines, and the latest guidelines have been revised in 2021. An additional “active bleeding” has been added to high-risk trauma types [7]. The new guidelines will take several years to be implemented, and our model must be further validated and updated as necessary.

Conclusions

We constructed a prehospital triage model, pTEST, to predict severe trauma and achieved an undertriage rate of <10%. Moreover, our study demonstrated that ML is a promising method for enhancing field triage guidelines performance. In the future, we will validate our pTEST model using populations from different countries and casualty backgrounds. In addition, software must be developed to increase user convenience of the pTEST model in the EMS.