Identification of a Susceptible and High-Risk Population for Postoperative Systemic Inflammatory Response Syndrome in Older Adults: Machine Learning–Based Predictive Model

Introduction

Systemic inflammatory response syndrome (SIRS) is a nonhomeostatic, self-destructive, and uncontrollable inflammatory response of the whole body triggered by infection, trauma, or major operations [1]. Recognizing SIRS has been a prerequisite of suspecting potential sepsis and implementing decisions such as sample culturing for the source of infection, escalating antibiotic regimens, and the level of patient monitor and care [2,3]. Although there is a tendency to apply criteria including the Sequential Organ Failure Assessment (SOFA) score or quick SOFA score to identify the possibility of sepsis [4], SIRS criteria has demonstrated higher sensitivity than the quick SOFA score [3], and it has served as both useful inclusion criteria and therapeutic target of trials aiming to treat sepsis [5]. Early identification of patients who will develop postoperative SIRS may enable clinicians to provide timely interventions to prevent sepsis and improve outcomes. It has been reported that the incidence of postoperative SIRS could be as high as 89% [6,7] in patients undergoing abdominal surgery, and the condition frequently developed into sepsis and even multiple organ dysfunction syndrome [8]. A 13-fold increase in mortality was reported in patients with postoperative SIRS compared with those without SIRS [9]. Notably, the incidences of SIRS and sepsis steadily increase with age, and octogenarians are almost twice as likely to develop sepsis than those aged less than 50 years [10,11]. Although standardized preoperative antibiotic prophylaxis has been recommended and clinically applied for older adult populations with a high risk of postoperative infection and SIRS, nearly one-quarter of older adult patients still develop SIRS within 3 days after surgery [12]. Thus, it is important to identify the risk of postoperative SIRS for older adult patients at a sufficiently early stage, which would allow preemptive individualized enhanced therapy to be conducted to improve the prognosis of older adult patients.

Compared with traditional biostatistical methods, machine learning (ML) methods hold the advantages of flexibility, scalability, and the ability to analyze diverse data types, which can be deployed for many tasks, such as risk stratification, diagnosis and classification, and survival predictions [13-15]. In recent years, there have been many ML models used to predict sepsis [16-22], the majority of which were developed in similar populations such as patients in the intensive care unit [23]. However, only a few have focused on older adult surgical patients, and an ML model to predict postoperative SIRS has been rarely reported.

The goal of our study was to use ML methods to develop an individualized predictive model for older adult surgical patients to screen for a susceptible population at a high risk for SIRS in order to instruct appropriate early interventions. Meanwhile, the generalizability of the model was validated with the data sets from the other 2 medical centers.

Methods

Setting, Dates, and Population

The study was performed based on the electronic health record (EHR) systems of 3 independent medical centers including the Third Affiliated Hospital of Sun Yat-sen University (Guangzhou, China), Lingnan Hospital of Sun Yat-sen University, and Yuedong Hospital of Sun Yat-sen University using data from surgical patients aged ≥65 years from 2015 to 2020.

During the retrospective enrollment, the inclusion criteria included (1) age ≥65 years, (2) patients who underwent general anesthetic with endotracheal intubation or laryngeal mask, and (3) patients who had preoperative antimicrobial prophylaxis. The exclusion criteria included (1) patients whose total intraoperative infusion volumes, fluid loss, or American Society of Anesthesiologists (ASA) classifications were not recorded and (2) patients diagnosed with SIRS before surgery or undergoing infectious surgeries.

Data Sources

The data comes from the EHR system of the 3 hospitals, which were established by extracting medical records from the hospital information system, laboratory information system, picture archiving and communication system (PACS), and Docare Anesthesia System (2005-2020 Medical System Co, Ltd), which enabled access to a comprehensive data set collected during hospital admissions, inpatient stays, and post-hospital follow-up visits, including demographic characteristics, daily documentation, laboratory tests, imaging results, and anesthesia records.

Outcome

SIRS was diagnosed according to the American College of Chest Physicians [24]. It was defined when 2 or more of the following criteria were present: (1) temperature <36 °C or ≥38 °C, (2) heart rate ≥90 bpm, (3) respiratory rate ≥20 bpm or arterial carbon dioxide tension <32 mm Hg, and (4) white blood cell count <4×10⁹/L, ≥12×10⁹/L, or >10% immature forms. The incidence of SIRS within 3 postoperative days was recorded in the study.

Data Sets

Data sets from the 3 medical centers were created separately and included: (1) patient demographics such as age, gender, and smoking history; (2) preoperative complications, including diabetes, hypertension, and fever; (3) anesthesia records, including ASA, administration of ulinastatin, dexamethasone, methylprednisolone, and dexmedetomidine; (4) liquid management including volume of fluid loss, blood loss, and colloid input; (5) duration of surgery; (6) laboratory parameters, including alanine aminotransferase (ALT), white blood cell count, hemoglobin, creatinine, albumin, high-sensitivity C-reactive protein (hs-CRP), activated partial thromboplastin time, glucose, low-density lipoprotein, high-density lipoprotein, blood urea nitrogen (BUN), fibrinogen, thrombin time, lymphocyte, red blood cell count, indirect bilirubin. These variables were selected based on EHR availability and their relevance to SIRS risk according to the literature and clinical experience [25].

For the purpose of developing and validating the ML models for risk prediction, the cohort of eligible patients from the Third Affiliated Hospital of Sun Yat-sen University was randomly divided into an 80% training set (2882 patients) and a 20% internal validation set (720 patients). In addition, 2 validation data sets were created with the eligible patients from the Lingnan Hospital and Yuedong Hospital for external validation. Based on the events per variable (EPV)-10 principle [26], this study has a sufficient sample size.

Statistical Analysis and ML Model Training

This study was reported as per the Guidelines for Developing and Reporting Machine Learning Predictive Models in Biomedical Research [27]. Categorical variables are expressed as number (%), and continuous variables that follow a normal distribution are expressed as mean (SD) or otherwise as median (IQR). The missing rates of the variables are shown in Table S1 in Multimedia Appendix 1. Missing values were imputed on each of the 3 cohorts (hospitals) independently: Categorical variables were filled in using mode counts, and continuous variables were filled in using means. The outliers were eliminated based on clinical experience, and 0 variance variables were removed.

We trained 4 different classical ML algorithms trained on the training set, including random forest (RF), XGBoost, logistic regression (LR), and multilayer perceptron (MLP) models. RF represents a bagging parallel integration algorithm, XGBoost represents a boosting algorithm; LR represents the most classical linear model, and MLP represents a neural network algorithm. Grid search was applied based on 5-fold cross-validation to find the optimal hyperparameters for each algorithm based on the training set. The formulas for the evaluation indicators in the models are provided in Multimedia Appendix 1.

The performance of the 4 algorithms was compared, and the optimal model was selected for this study. A comparison between the optimal model and nomogram we established before [28] and was also conducted with the internal validation set. Note that the prepared data sets incurred a data imbalance problem. To address this problem, we used the synthetic minority oversampling technique (SMOTE) provided by the performanceEstimation package [29,30].

The statistical analyses were all done using Python 3.7 [31-34] and R-3.6.2 [35]. All results were considered statistically significant at P<.05.

Ethical Considerations

The study protocol followed the principles of the Declaration of Helsinki and was approved by the Institutional Ethics Committee of the Third Affiliated Hospital of Sun Yat-sen University on July 27, 2022 (number [2019]02-609-04). The requirements for informed consent and clinical trial registration were waived by the committee.

Results

Study Cohorts and Characteristics

Among the 16,141 patients aged ≥65 years accessed from the EHR system, only 3602 patients who met the inclusion criteria were included in the development cohort, with 876 (24.3%) postoperative SIRS events. The development cohort was then randomly separated into the training set and the internal validation set, which consisted of 2882 and 720 patients, respectively. Meanwhile, 844 and 307 patients were finally included in the external validation-1 set (Lingnan Hospital) and the external validation-2 set (Yuedong Hospital), respectively (Figure 1).

‎

The characteristics of the 3 cohorts from different medical centers are shown in Table 1. The incidence rates of postoperative SIRS in each of the 3 medical centers were 24.3% (876/3602), 29.6% (250/844), and 6.5% (20/307), respectively. The different incidences of SIRS across the 3 medical centers may result from factors such as different regional characteristics, economic development levels, patient characteristics, surgical procedures, and the medical and nursing team.

Among the 3602 patients in the development cohort with a mean age of 70.0 years, 2144 (59.5%) were women, 2152 (59.7%) had comorbid hypertension, 1140 (31.6%) had comorbid diabetes, and 437 (12.1%) were smokers. In addition, 420 (11.7%) patients had preoperative fever, and 1330 (36.9%) were assessed as ASA III/IV/V preoperatively.

The characteristics of patients with or without postoperative SIRS in the development cohort are shown in Table 2. Patients who ended up with SIRS (SIRS group vs the non-SIRS group) were mainly women (573/876, 65.4% vs 1571/2726, 57.6%, P<.001) and more likely to have been diagnosed with diabetes (334/876, 38.1% vs 806/2726, 29.6%, P<.001) and assessed as ASA III/IV/V preoperatively (472/876, 53.9% vs 858/2726, 31.5%, P<.001). With evident incidence of preoperative fever (182/876, 20.8% vs 238/2726, 8.7%, P<.001), the total volume of fluid loss, volume of blood loss, and duration of surgery were also significantly higher and longer in the SIRS group than in the non-SIRS group (700 [330-1200] mL vs 430 [200-800] mL, 100 [50.0-200] mL vs 50 [20.0-100] mL, 207 [133-310] min vs 150 [90-232] min, respectively; all P<.001). Additionally, laboratory indicators including hs-CRP, BUN, direct bilirubin (DBILI), and prothrombin time (PT) were higher in SIRS patients than in non-SIRS patients, whereas albumin was lower in SIRS patients (all P<.001; Table 2).

The prognoses of the non-SIRS and SIRS groups in the development cohort are shown in Table 3. Compared with the non-SIRS group, patients in the SIRS group were significantly more likely to develop postoperative complications that included hemorrhage (446/876, 50.9% vs 939/2726, 34.4%, P<.001), acute respiratory distress syndrome (14/876, 1.6% vs 2/2726, 0.1%, P<.001), cardiac arrest (10/876, 1.1% vs 5/2726, 0.2%, P<.001), agitation and delirium (65/876, 7.4% vs 32/2726, 1.2%, P<.001), coma (60/876, 6.8% vs 7/2726, 0.3%, P<.001), and acute kidney injury (81/876, 9.2% vs 55/2726, 2%, P<.001). Furthermore, SIRS patients had a longer median postoperative hospitalization (12.0 [8.0-19.0] days vs 7.0 [5.0-10.0] days, P<.001), higher median cost (¥91,846 [¥61,765-¥135.5] vs ¥56,542 [¥34,273-¥76,956], P<.001; a currency exchange rate of ¥1=US $0.14 is applicable), higher risk of postoperative intensive care unit admissions (273/876, 37.1% vs 79/2726, 3.3%, P<.001), and higher in-hospital mortality rate (21/876, 2.4% vs 10/2726, 0.4%, P<.001) than non-SIRS patients.

Variable Selection

Details of the missing rates in each center for the 36 variables selected based on the literature and our clinical experience are shown in Table S1 in Multimedia Appendix 1, and the highly correlated variables with a correlation coefficient higher than 0.7 were deleted to avoid collinearity (based on the principle of keeping the variable with the highest area under the receiver-operating curve [AUC] in the independent prediction of postoperative SIRS; Figure S1 in Multimedia Appendix 1).

Since partially relevant or less important features may negatively affect the performance of ML models, feature selection was performed on the development cohort using least absolute shrinkage and selection operator (LASSO) regression methods. As shown in Figure 2, the dimensionality was reduced to 15 features after LASSO, including preoperative fever, ASA, PT, hsCRP, BUN, diabetes, duration of surgery, ulinastatin, methylprednisolone, alanine aminotransferase, total volume of fluid loss, volume of blood loss, DBILI, albumin, and gender. Except albumin and gender, which were negatively associated with postoperative SIRS, all other features were positively associated with postoperative SIRS in older adult patients (Table 4).

‎

Model Construction, Internal Validation, and Horizontal Comparison

Finally, the 15 selected predictors were used in the 4 ML models, including RF, XGBoost, LR, and MLP, to predict postoperative SIRS. The performance of the different ML algorithms in the internal validation set are shown in Figure 3, and the calibration curves are presented in Figure 4. The AUC of the RF model was 0.751 (95% CI 0.709-0.793) with the highest sensitivity of 0.682 and specificity of 0.681. The F₁ score of the RF model (0.508) was the highest among the 4 ML models, and the Brier score (0.153) was also relatively higher. As a result, upon general consideration of the AUC, F₁ score, Brier score, calibration curve, and the ability of subsequent cross-center promotion and application, we thought that the RF model demonstrated the best performance. Additionally, the performance of RF model in the internal validation set was compared with the nomogram [23] we once established to predict postoperative SIRS in older patients (Figure 3). The results showed that the RF model had a significantly higher AUC (AUC=0.751) than the nomogram (AUC=0.671) in the internal validation set, which further proved the generalizability of the RF model (Table 5). The internal validation results after applying SMOTE are shown in Table S2 in Multimedia Appendix 1, where the AUC values increased for almost all ML algorithms except for LR and RF still exhibited the best performance with an AUC of 0.923.

‎

‎

External Validation Performance

External validation of the developed RF algorithm was conducted with the eligible data from patients from Lingnan Hospital (external validation-1) and Yuedong Hospital (external validation-2). As shown in Table 6, the RF model achieved relatively higher AUCs for the external validation-1 (0.759, 95% CI 0.723-0.795) and external validation-2 (0.804, 95% CI 0.746-0.863) sets. Moreover, the external validation-2 set demonstrated much higher sensitivity (lower false-negative rate) than the internal validation set (0.800 vs 0.682), and the specificity of the model was improved in both external validation sets. On the other hand, the negative predictive value was high, and the positive predictive value (PPV) was relatively low for all 3 validation sets since we chose a balanced cutoff between sensitivity and specificity to ensure as high a true positive (TP) as possible, which also results in a high false positive (FP) and low false negative. Notably, the PPV and F₁ score of external validation-2 was very low due to the low incidence rate of postoperative SIRS (20/307, 6.5%) at Yuedong Hospital.

After applying SMOTE, the performance of the RF model in the external validation set was basically the same as the model without SMOTE, with AUCs of 0.783 vs 0.759 in external validation set 1 and 0.784 vs 0.804 in external validation set 2, which further proves that the RF model has good generalizability to different validation populations and different data processing methods (Table S3 in Multimedia Appendix 1).

Feature Importance Weight

The feature importance permutation was used to rank the levels of feature importance, which is defined as the decrease in a model score when a single feature value is randomly shuffled (Figure 5). The results showed that the total volume of fluid loss, duration of surgery, ASA, and volume of blood loss had a significant impact on the outcome, with higher importance weight values of 0.121, 0.119, 0.118, and 0.101, respectively (Table 7).

‎

Online Application

Figure S2 in Multimedia Appendix 1 shows the online calculator for our RF model to predict the risk of postoperative SIRS in older adult patients, which can be found at [36]. Users can quickly obtain the predicted probability of the patient’s postoperative SIRS risk by entering the values of the 15 predictors.

Discussion

Principal Findings

In this study, we evaluated the ability of 4 ML algorithms including RF, XGBoost, LR, and MLP to predict postoperative SIRS in older adult patients based on eligible data from EHR systems and concluded that the RF model has moderately better performance than the other ML algorithms, with an AUC value of 0.751, highest sensitivity of 0.682, and specificity of 0.681. The RF model also exhibited better performance in the internal validation set than the nomogram we once established to predict postoperative SIRS in older patients. Furthermore, the applicability of the RF model was proved by external validation in the other 2 independent medical centers, with relatively higher AUC values of 0.759 and 0.804, indicating good reproducibility and generalizability of the model in older adult patients. Notably, the model exhibited a high negative predictive value and relatively low PPV because we chose a balanced cutoff between sensitivity and specificity to ensure as high a TP as possible, which inevitably leads to a high FP and low false negative. Especially for external validation-2 (Yuedong Hospital), which had a very low incidence rate of postoperative SIRS (20/307, 6.5%), 16 TP were successfully predicted with a high sensitivity of 0.800, which comes with 89 FP and results in a low PPV (0.152) and F₁ score (0.256).

The model indicators used in this study include preoperative fever, ASA, PT, hsCRP, BUN, diabetes, duration of surgery, ulinastatin, methylprednisolone, ALT, total volume of fluid loss, volume of blood loss, DBILI, albumin, and gender. These indicators are mainly related to the individual disease characteristics and are not affected by the level of different medical institutions or regional characteristics, which makes the model more generalizable. The results of our study further prove that the model has a good generalization ability for populations with different SIRS incidences. The high recall of our model is important for reminders to check patients early and prevent SIRS in advance; this demonstrates a more responsible attitude toward patients.

Currently, given the more severe mortality and adverse prognosis, most studies have adopted sepsis as a clinical endpoint [2]. In our study, we used SIRS as the primary outcome because it has been an acknowledged criterion that is easier to identify and can help physicians notice the possibility of sepsis and prescribe tests to examine whether infection truly exists.

As the population ages, the surgical population is also aging faster than the general population, with higher morbidity and mortality rates [37]. Notably, older adults are predisposed to postoperative infections, SIRS, and even sepsis due to preexisting comorbidities, repeated and prolonged hospitalizations, immune dysregulation, and functional limitations [38]. Although preoperative antimicrobial prophylaxis has been routinely administered for older adult patients in various specialized operations, the persistent high incidence of postoperative SIRS suggests that it cannot be effectively prevented in this way [12]. In fact, for older patients undergoing different types of surgery, there is still a lack of effective tools to identify high-risk patients with postoperative SIRS and assist with decision-making on the need for individualized interventions.

In this study, we used ML methods to analyze diverse data types [13] due to the advantages of flexibility and scalability. A predictive model based on an RF algorithm was developed to identify older adult patients with a high risk of postoperative SIRS. Internal validation and two external validations confirmed that the established model could predict postoperative SIRS with high accuracy and specificity. The results might be important derivatives and supplements to the current perioperative prevention and management programs, which enable surgeons and anesthesiologists to identify older adult patients who are suspected of having postoperative SIRS. In addition to screening high-risk groups, this model can also help prevent and treat postoperative SIRS in older adult patients more accurately and in a timelier manner by using various drug or nondrug means under the guideline of enhanced recovery after surgery, so as to promote the short-term and long-term prognoses of patients [39]. Finally, an online risk calculator was developed to improve clinical usability and make our model accessible to anesthesiologists and peers around the world.

Comparison With Prior Work

Distinguished from previous investigations that mainly focused on a single indicator or a single surgery, we fully used the perioperative data of patients and constructed an optimal combination of risk factors to predict postoperative SIRS in older adult patients. In this study, 15 variables were identified to be significantly associated with postoperative SIRS, including preoperative fever, ASA [40], PT, hsCRP, BUN, diabetes [41], duration of surgery [42], ulinastatin, methylprednisolone, ALT, total volume of fluid loss, volume of blood loss, DBILI [43], albumin, and gender [44]. These variables have also been reported to be associated with postoperative SIRS and sepsis in earlier studies, adding clinical credibility to our model. Meanwhile, all the variables were routinely recorded and widely used in clinical practice, which makes the model more feasible and can be widely used in different hospitals.

Methylprednisolone and ulinastatin have also been associated with postoperative SIRS, which may be attributed to the fact that both medicines are widely used among high-risk surgical patients with postoperative infection according to anesthesiologists’ clinical experience [45-47]. Additionally, intraoperative fluid loss ranks first in the feature importance weight of the RF model, which indicates that fluid management such as intraoperative intravenous infusion should be given higher priority during the perioperative period to prevent postoperative SIRS, and this was consistent with the consensus on early management of sepsis [48].

Limitations

Notably, several limitations must be listed here, including that (1) this is a retrospective cohort study with collection and entry bias, as well as possible residual confounding, which requires future prospective studies to validate the model. In addition, some indicators for SOFA scoring were missing, so we could not perform a comparative study between our model and SOFA scoring. (2) We selected 36 variables based on the availability in the EHR, correlation with SIRS risk, and predictive potential, but it should be noted that we may have missed other indicators with more predictive performance in postoperative SIRS due to the limitation of our data. (3) The model demonstrated a low PPV because we wanted the model to have a relatively high recall rate to predict more patients who were truly at a high risk of postoperative SIRS, which would inevitably lead to more FPs, but from a health economics perspective, the economic burden associated with enhanced perioperative management of high-risk patients is much less than that resulting from the progression of the patient to SIRS or even after sepsis, resulting in medical-economic stress. (4) The generalizability of this model in different ethnic groups and different regions also needs further validation. (5) It should be emphasized that estimates in our model are predictive and should not be interpreted as causal [49], such as the association between intraoperative use of methylprednisolone and ulinastatin and the higher SIRS incidence rate. Intraoperative use of such drugs is likely a marker of risk stratification with a lower risk of infection patients using these drugs than those at high risk.

Conclusions

We enrolled 3 independent cohorts to develop and validate a generalizable RF model for the prediction of postoperative SIRS in older adult patients that enables surgeons and anesthesiologists to screen susceptible and high-risk populations for SIRS in older adult surgical patients and to implement early individualized interventions based on existing prevention and management programs.