Original Paper
Abstract
Background: Septic shock (SS) is a syndrome with high mortality. Early forewarning and diagnosis of SS, which are critical in reducing mortality, are still challenging in clinical management.
Objective: We propose a simple and fast risk-stratified forewarning model for SS to help physicians recognize patients in time. Moreover, further insights can be gained from the application of the model to improve our understanding of SS.
Methods: A total of 5125 patients with sepsis from the Medical Information Mart for Intensive Care-IV (MIMIC-IV) database were divided into training, validation, and test sets. In addition, 2180 patients with sepsis from the eICU Collaborative Research Database (eICU) were used as an external validation set. We developed a simplified risk-stratified early forewarning model for SS based on the weight of evidence and logistic regression, which was compared with multi-feature complex models, and clinical characteristics among risk groups were evaluated.
Results: Using only vital signs and rapid arterial blood gas test features according to feature importance, we constructed the Septic Shock Risk Predictor (SORP), with an area under the curve (AUC) of 0.9458 in the test set, which is only slightly lower than that of the optimal multi-feature complex model (0.9651). A median forewarning time of 13 hours was calculated for SS patients. 4 distinct risk groups (high, medium, low, and ultralow) were identified by the SORP 6 hours before onset, and the incidence rates of SS in the 4 risk groups in the postonset interval were 88.6% (433/489), 34.5% (262/760), 2.5% (67/2707), and 0.3% (4/1301), respectively. The severity increased significantly with increasing risk in both clinical features and survival. Clustering analysis demonstrated a high similarity of pathophysiological characteristics between the high-risk patients without SS diagnosis (NS_HR) and the SS patients, while a significantly worse overall survival was shown in NS_HR patients. On further exploring the characteristics of the treatment and comorbidities of the NS_HR group, these patients demonstrated a significantly higher incidence of mean blood pressure <65 mmHg, significantly lower vasopressor use and infused volume, and more severe renal dysfunction. The above findings were further validated by multicenter eICU data.
Conclusions: The SORP demonstrated accurate forewarning and a reliable risk stratification capability. Among patients forewarned as high risk, similar pathophysiological phenotypes and high mortality were observed in both those subsequently diagnosed as having SS and those without such a diagnosis. NS_HR patients, overlooked by the Sepsis-3 definition, may provide further insights into the pathophysiological processes of SS onset and help to complement its diagnosis and precise management. The importance of precise fluid resuscitation management in SS patients with renal dysfunction is further highlighted. For convenience, an online service for the SORP has been provided.
doi:10.2196/58779
Keywords
Introduction
Sepsis is a syndrome or group of symptoms associated with infection rather than a single disease []. Reportedly, approximately 48.9 million cases of sepsis occurred globally in 2017, with approximately 11 million sepsis-related deaths, accounting for approximately 19.7% of all deaths worldwide []. According to the Third International Consensus on Sepsis and Septic Shock held in 2016 [], sepsis is defined as life-threatening organ dysfunction caused by a dysregulated host response to infection, and septic shock (SS) is defined as an advanced state of sepsis in which severe circulatory, cellular, and metabolic abnormalities lead to a greater risk of death than sepsis alone, with a mortality rate estimated to be as high as 45% [,]. Kumar et al [] found that patients with SS treated within the first hour of diagnosis had a 79.9% survival rate, but each hour of delay in treatment was associated with a 7.6% increase in mortality. Therefore, early forewarning of the onset of SS and timely treatment can help reduce the morbidity, mortality, and length of intensive care unit (ICU) stay. However, early forewarning is challenging due to the complexity of the disease in the clinical context [,] and the heterogeneity of the sepsis population [].
Recently, machine learning techniques have been applied to electronic health record data to develop models for risk prediction [,], clinical phenotyping [-], and early forewarning [-]. While several studies have focused on early forewarning of SS in the ICU, practical limitations remain [-]. Many models required extensive clinical features from multiple examinations [,], increasing the complexity and time for data collection, which can delay forewarnings. Efforts to reduce the feature number often relied on distributed examinations [] and left limited time for intervention []. Additionally, many studies prioritized predictive performance [-] while neglecting clinical interpretability. Specifically, limited attention has been given to the triggers of inconsistent predictions, a crucial factor for improving models and the understanding of disease diagnostic criteria. To enhance clinical applicability, models must offer straightforward actionable insights rather than complex algorithms that burden clinicians.
Therefore, we developed a real-time risk-stratified forewarning model for SS that minimized the need for extensive clinical examinations, enabling physicians to quickly and easily assess the patient’s risk and facilitate timely intervention. Furthermore, after stratifying patients based on the risk scores provided by the model, we further analyzed the clinical characteristics of patients in different risk groups, with a particular focus on high-risk (HR) patients. A public web application was deployed that enables easy access and use of our proposed model [].
Methods
Datasets and Participants
To improve the quality and reproducibility of reporting, we prepared a checklist () according to the TRIPOD (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis) guidelines [].
Data were collected from an open-source dataset released for sizeable critical care databases, including the Medical Information Mart for Intensive Care-IV (MIMIC-IV) [] and the eICU Collaborative Research Database (eICU) []. The MIMIC-IV contains comprehensive and high-quality data on deidentified patients admitted to ICUs at the Beth Israel Deaconess Medical Center (BIDMC), with more than 76,000 admissions between 2008 and 2019. The eICU is a multicenter publicly available database containing deidentified highly granular medical data for more than 200,000 patients in 335 ICUs from 208 hospitals across the United States between 2014 and 2015. Both databases include vital signs (VSs), laboratory results, diagnosis, treatment, care plan documentation, severity of illness measures, and other information.
Patients aged 18 years or older who were admitted for the first time and stayed in the ICU for more than 24 hours were included in the analysis. To reduce the impact of complicating factors in defining SS, patients designated as “do not resuscitate” or “encounter for palliative care” according to the International Classification of Diseases (ICD) were excluded. Moreover, patients in the coronary care unit were excluded to reduce the impact of cardiogenic shock. A final cohort of 5125 patients (including 766 SS patients) in the MIMIC-IV and 2180 patients (480 SS patients) in the eICU was obtained. The MIMIC-IV data were randomly partitioned into training (60%), validation (20%), and testing (20%) sets. The eICU data were used for external validation.
Ethical Considerations
The MIMIC-IV and eICU databases are publicly accessible, deidentified, and transformed, and have been approved for use by the institutional review boards at BIDMC and the Massachusetts Institute of Technology, in accordance with the Declaration of Helsinki. Both databases received waivers for informed consent due to the deidentification of all protected health information. Access to these databases was granted upon completion of human research ethics training and signing of a data use agreement with PhysioNet (certification number 38539898).
Definitions of Sepsis and SS
The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3) [] were applied to determine the onset time of sepsis and SS. Specifically, all episodes of suspected infection (tsuspicion) were identified by the timestamp of antibiotic administration or blood culture, whichever was earlier, within a specific period. If the antibiotic was administered first, the culture sampling was required to be obtained within 24 hours. If the culture sampling occurred first, the antibiotic was required to be ordered within 72 hours. The onset time of sepsis (tsepsis) was defined as an episode of suspected infection with a change in the Sequential Organ Failure Assessment (SOFA) score of 2 or more points from up to 48 hours before to up to 24 hours after tsuspicion. Patients with SS were identified by a clinical construct of sepsis with persisting hypotension requiring vasopressors to maintain mean arterial pressure ≥65 mmHg and with a serum lactate level >2 mmol/L (18 mg/dL) despite adequate volume resuscitation []. Specifically, adequate fluid resuscitation was defined as a total fluid volume greater than 30 mL/kg or 1000 mL within a 3-hour window before the initiation of vasoactive drugs or hypotension. The first lactate level needed to be greater than 2 mmol/L within 6 hours after adequate resuscitation. The onset time of SS (tseptic shock) was the earliest time when all criteria for SS were met. Due to a lack of information on the starting and ending times of fluid resuscitation, SS, according to Sepsis-3, could not be defined in the eICU data. Instead, we used the ICD code labels in the “diagnosis.csv” table to determine SS onset.
Data Processing
Given the characteristics of time-series data and the need for real-time monitoring for SS risk, this study defined a 6-hour retrospective window, whereby data collected during the 6 hours prior to the current state of the patient were used to provide a forewarning of the SS risk in the next 6 hours. We discretized the features of high-frequency detection, such as VSs, and the features of low-frequency testing, such as laboratory test data, to an hourly value. If multiple measurements were performed in the same hour, their mean value was calculated as the value for this hour of the feature. The forward-filling method was used to fill in missing measurements (with the early measurement), if there was no measurement in this hour.
Next, several time intervals were defined to assess and compare differences among risk groups (derived from the Septic Shock Risk Predictor [SORP] risk scores) in terms of patient pathophysiology (based on VSs and laboratory testing features), clinical characteristics, etc. The time of SS onset is defined as 0. Considering this, –1 indicates 1 hour before onset and +1 indicates 1 hour after onset, and the risk prediction time is –6. Three primary time intervals were delimited, namely, –12 to –6, –6 to 0, and 0 to +6, representing the look-back interval, the preonset interval, and the postonset interval, respectively, where the look-back interval was the time interval used for data collection for risk calculation by the model. For easier comparison among risk groups, the mean value of a clinical feature was used to represent its value within a specified time interval.
Notably, information on patient comorbidities (eg, chronic kidney disease [CKD]) was taken from ICD codes, and invasive procedures (eg, dialysis-continuous renal replacement therapy [CRRT]) were taken from the procedures documented (“procedureevents” table).
Risk Model
To facilitate clinical application, we filtered the features and then developed a scorecard [,] to evaluate SS onset risk by logistic regression (LogR). Next, to present the patient’s risk more visually, we stratified the risk scoring and assessed the characteristics of patients in each risk group. The workflow is shown in 
Figure 1.
The scorecard was constructed as follows. The scorecard allowed for transferring continuous variables into multiple bins, and each feature was divided into multiple bins. In this study, we used the weight of evidence (WOE) to measure the probability of illness in each bin and determine the most appropriate number of bins by detecting the change in information value (IV) under each bin. The WOE and IV were calculated according to equations 1 and 2, respectively. Specifically, first, features were divided into 10 bins, making sure that each bin contained both SS patients and nonshock (NS) patients. Subsequently, the chi-square test was performed on 2 neighboring bins, traversing and merging those bins with the largest P-value until the number of bins met the expectation. Then, the final number of bins was determined based on the IV. After the binning was completed, the WOE value of each bin was calculated, and then, the WOE was mapped back to the original dataset to build the LogR model.
where i represents each subgroup, DistributionNSi denotes the proportion of septic NS patients in group i to the NS patients in the entire feature, and Distribution Si denotes the proportion of SS patients in group i to the SS patients in the entire feature.
The scorecard points were calculated using the following equations:
where P0 and PDO (points to double odds) are the score ranges that were set manually, woej can be calculated with equation 1, βi denotes the coefficient of the ith feature in LogR, α is the intercept term from LogR, n is the number of features, and k is the number of bins in each feature.
The scorecard was adjusted to a range of scores between 0 and 100. The final scorecard consisted of a base score and a score for each bin in each feature ().
Statistical Methods
We employed the Shapley Additive Explanations (SHAP) [] method to calculate the importance of clinical features for SS in the look-back interval. SHAP assigns a value to each feature in the given prediction. These values are calculated for each prediction separately and do not provide general information about the entire model. Higher absolute SHAP values indicate higher importance, whereas values close to 0 indicate low significance of a feature.
To facilitate observation and evaluation and determine risk intervals, we employed the Kolmogorov-Smirnov (KS) curve to describe the overall score distribution. The KS curve illustrated the changes in the sample proportion of SS patients, the proportion of septic NS patients, and the trend of differences between SS and NS patients (blue line) with a score from 0 to 100. From the KS curve, it could be seen that the proportion of SS patients in the low score accumulated faster, while the proportion of NS patients in the high score accumulated faster. The blue curve showed the variation process of the difference between SS and NS patients at each score point. At the highest point of the blue curve, the ability to distinguish between SS and NS patients was the strongest. Therefore, to provide a more direct scoring effect, based on the distributions described above, the scores were divided into 4 risk groups: HR, medium risk (MR), low risk (LR), and ultralow risk (ULR), with score intervals of 0-40, 40-60, 60-80, and 80-100, respectively.
The Population Stability Index (PSI) was used to evaluate the stability of risk group distribution across different datasets in the postonset interval. A smaller PSI value indicates a minor difference between 2 distributions and more consistency. In addition, the area under the curve (AUC) was calculated to evaluate the performance of each model. Estimation and comparison of the survival curves among risk groups were performed using Kaplan-Meier survival analysis and log-rank tests, respectively. The Local Polynomial Regression Fitting (locally estimated scatterplot smoothing [LOESS]) method was used to fit the relationship between the risk score and SS onset rates at 6 hours. Hierarchical clustering was used to assess the similarity between risk groups. All statistical analyses and algorithm approaches in this study were performed using Python (Python Software Foundation) and R (R Project for Statistical Computing).
Results
Development of the SORP
Since a particular time window is required for early forewarning of SS, the longer it takes to collect the necessary data for the model, the less time is left for physicians to intervene. Therefore, we calculated the importance of each feature in the early forewarning of SS in the look-back interval and selected the top-ranked feature set that could be accessed in the same testing batch. In particular, we observed that 8 of the 10 most important clinical features for SS prediction were arterial blood gases (ABGs) and VSs (
Figure 2A). Notably, point-of-care ABG testing results could be obtained even within 5-10 minutes in the ICU, and VS features could be obtained at any time from the monitor. This result indicated that septic preshock core pathophysiological shifts could be mainly monitored by ABG and VS features, allowing us to quickly obtain essential features for early forewarning of SS and providing a prompt assessment.
Since features obtained simultaneously from an ABG test might have a certain degree of complementarity, we included 8 ABG and 7 VS features collected in the MIMIC-IV and eICU in the subsequent analysis. An SS risk predictor (SORP) based on a scorecard was constructed by combining LogR with WOE in the look-back interval (detailed in the Methods section). According to the scorecard, a patient’s SS risk score could be obtained by adding interval scores to the base score (). Lower total scores indicated a higher risk of SS onset. In the training, validation, and test datasets, the AUCs were 0.9557, 0.9537, and 0.9458, respectively (Figure 2B). Meanwhile, based on all available clinical features, multi-feature complex models were constructed, including deep learning and machine learning models (Figure 1A; ). These features are related to acid-base balance, coagulation, renal aspects, hepatic aspects, pulmonary aspects, hematologic aspects, the Glasgow Coma Scale, etc. The predictive performance of the SORP is only slightly lower than that of the optimal complex model. In the test datasets, the AUCs of the SORP and the optimal complex model were 0.9458 and 0.9651, respectively. It indicated that the construction of the SORP was reasonable and had strong early forewarning ability.
To facilitate observation and evaluation and determine risk intervals, according to the KS curve plot (Figure 2C), the scores were divided into 4 risk groups as detailed in the Statistical Methods subsection. The distribution of SS was highly consistent and stable among the training, validation, and test datasets (PSI <0.01; Figure 2D; ). In total, the incidences of SS for the HR, MR, LR, and ULR groups in the postonset interval were 88.6% (433/489), 34.5% (262/760), 2.5% (67/2707), and 0.3% (4/1301), respectively (Figure 2D; ). A total of 88.6% (433/489) of HR patients developed SS, and 90.7% (695/766) of SS patients were classified into the HR or MR group. Further, a curve was fitted for the relationship between the risk score and SS onset rate at 6 hours (Figure 2E; ), and the result showed a clear stratification and monotonicity, with an increase in the SS onset rate with increasing risk (decreasing risk score). These results suggest that the SORP exhibits a great early forewarning performance.
Characteristics of Different Risk Groups After SS Onset
To verify the rationality of risk stratification, we explored the differences in clinical features among risk groups in the postonset interval. Remarkably, increased abnormal values for standardized means of ranking continuous features, especially features related to the SS definition (such as lactate and blood pressure), kidneys, acid-base balance, and coagulation (
Figure 3A and 3B; ), were observed in HR patients when compared with those in other risk groups. The degree of abnormality of these features further increased as risk increased (Figure 3A). For all patients, Kaplan-Meier survival curves progressively declined as patient risk increased (Figure 3C). An even more rapid progressive decline was observed in NS patients in the HR group (Figure 3D), but there was no difference for SS patients (Figure 3E). Furthermore, the length of time from sepsis to SS was significantly reduced with increasing risk (Figure 3F). These results suggest that disease severity increased with increasing risk groups and that the SORP has a reliable risk stratification capability.
Are NS_HR Patients Truly Septic NS Patients?
Among the HR patients identified by the SORP, 11.5% (56/489) did not meet the SS definition according to the Sepsis-3 criteria [] and were labeled NS_HR. To further confirm the disease status of these patients, the distribution of features in the postonset interval for NS_HR patients was compared to that for SS and other NS (NS_O) patients. In clustering analysis and heatmap visualization, NS_HR patients clustered with the SS groups but not with the other 3 NS groups (
Figure 4A). This result indicates that the NS_HR group was more similar to SS patients in terms of clinical features, such as lactate, the SOFA score, and features related to renal function (creatinine and urine output), coagulation (prothrombin time, international normalized ratio, and platelet count), and acid-base balance (base excess, bicarbonate, total CO2, and anion gap) (Figure 4B). Compared to NS_O patients, both NS_HR and SS patients displayed a consistently higher degree of variation in clinical features (Figure 4C). The distribution and variation of features demonstrated that both HR_NS and SS patients had similar levels of disorder compared with NS_O patients. Moreover, both NS_HR and SS patients had significantly shorter overall survival than NS_O patients (Figure 4D). In total, NS_HR patients were highly similar to SS patients and distinct from NS_O patients. In addition, we queried the patients’ ICD codes, and 35% (19/55) of NS_HR patients were labeled with “septic shock,” which implied that a proportion of NS_HR patients might meet the previous criteria for SS.
Notably, the overall survival of NS_HR patients was significantly worse than that of SS patients (Figure 4D), and we explored the potential factors contributing to this survival difference. SS-associated treatments, such as vasopressors and fluid resuscitation, were evaluated. Compared to the SS group, the NS_HR group had a significantly higher incidence of mean blood pressure (MBP) <65 mmHg and a significantly lower rate of vasopressor use and infusion volume (all P values <.001; Figure 4E; ). Furthermore, NS_HR patients showed the highest percentage of CKD (P=.006) and the most abnormal kidney-related features (Figure 4E; ). According to this finding, one of the reasons for high mortality in NS_HR patients might be inadequate treatments, including both vasopressor use and infusion. One of the reasons NS_HR patients did not receive adequate fluid resuscitation might be severe kidney injury, as physicians may reduce fluid infusion in these patients to protect them from further severe kidney damage.
Application of the SORP in Real-Time Risk Monitoring for SS
The SORP showed great performance in sepsis stratification, and we further evaluated its feasibility as a real-time monitoring risk score for patients with sepsis. The hourly risk score for SS onset was calculated by the SORP, and the patient’s risk scores and trends throughout the ICU process were fully displayed. Two patients, one with NS and another with SS, are shown in . The NS patient had a progressively lower risk (increasing score) nearing discharge. In comparison, the SS patient had an increasingly higher risk until SS onset, with entry into the HR zone 6 hours before onset, and a gradual reduction in risk after onset, which might also reflect the role of treatment. Furthermore, based on continuous real-time scoring, we found that 50.0% (383/766) and 74.9% (574/766) of SS patients with risk scoring entered the HR zone 13 hours and 6 hours before onset (IQR 6-26), respectively. In addition, hourly risk scores were shown for each risk group during the 48-hour period before and after SS onset (
Figure 5A). The risk gradually increased (risk score gradually decreased) as septic patients progressed to shock and decreased with treatment intervention (after SS onset), demonstrating that the SORP can monitor the patient’s risk for SS in real-time (Figure 5A).
In contrast to the gradual increase in risk in the other 2 groups, SS_LR patients maintained a relatively stable risk during and before the look-back interval but demonstrated an extremely rapid increase in risk (risk scores decreased) in the preshock interval (Figure 5A). Further analysis demonstrated that SS_LR patients had a higher rate of invasive procedures than other SS patients (Figure 5B; ), which was significantly higher than that of other LR patients (Figure 5C; ). Therefore, we speculated that this might be one of the reasons for the rapid progression of SS_LR patients: invasive procedures can introduce additional microorganisms into the body, which can exacerbate the infectious response and lead to rapid deterioration. Analysis showed that 52% (35/67) of SS_LR patients were recognized by the SORP as their risk group increased from LR to MR 3 hours before SS onset. Therefore, we recommend that LR patients exposed to invasive procedures should be assessed more frequently for the timely recognition of SS.
Reliability and Robustness of the SORP
To further verify these findings, we additionally used multicenter eICU data for validation. Consistently, in the look-back interval, eICU patients were grouped into 4 risk groups. Although the proportion of SS patients was reduced in the HR group in the postonset interval (), which might be due to different definitions, the distribution of SS was consistent and stable in both the eICU dataset and the MIMIC-IV dataset (PSI <0.05; ). Similar to the MIMIC-IV, the patients’ features became increasingly abnormal as risk increased, especially features related to the SS definition (eg, lactate and blood pressure), kidneys, acid-base balance, and coagulation (
Figure 6A; ). For overall survival, the 4 risk groups also showed significant differences, even in NS patients and SS patients (Figure 6B-D). SS_LR patients, similar to MIMIC-IV patients, underwent significantly more invasive procedures than the other LR patients (), but there was no significant difference from the other SS patients (). NS_HR patients, similar to MIMIC-IV patients, significantly differed from NS_O patients in clinical features and survival, being more similar to SS patients but with a more severe status (Figure 6E-G). Regarding treatment, as the eICU did not provide start and end times for infusion, we could not obtain accurate infusion volumes for eICU patients within a certain interval, so we focused only on vasopressor use. Compared to SS patients, NS_HR patients showed significant hypotension and less vasopressor use, which were similar to the results of the MIMIC-IV (Figure 6H; ). Although this analysis was based on a different definition of SS, similar results were still observed, further confirming the robustness of the SORP and the reasonability of the risk stratification system.
In summary, the SORP has a reliable and robust ability to not only provide early forewarning but also stratify patients with potentially different risks of SS. For convenience, we provide an online service of the SORP []. The “Help” section in the navigation bar of the website contains detailed instructions on how to use the online tool.
Discussion
Principal Findings
Timely forewarning and early intervention are especially important for patients who are developing SS, and clinically applicable forewarning tools for SS are still lacking. In this study, we constructed the SORP, a simplified model, that not only provides real-time risk stratification for patients but can also easily trace the SS risk of patients during the entire ICU process and forewarn SS onset with a median of 13 hours in advance. The SORP can precisely assess the risk of SS 6 hours in advance by using only ABG and VS features, with an AUC of 0.9458 in the test set. Compared to the findings in previous studies, the SORP remarkably reduces both time and economic costs by using easily available ABG and VS features that can be accessed within minutes and can achieve competitive predictive performance (). In contrast, other studies require additional tests, such as biochemical tests, SOFA score calculations, and specific urine output measurements, which extend data collection by several hours. While some studies reduce the number of features through feature engineering methods, these features may still be distributed across test batches, which may not effectively shorten the overall time and costs needed for assessments. Moreover, in addition to being available through the web service [], the patient’s risk score can be simply and quickly calculated manually using a scorecard (). The ease of data collection and the simplicity of use will greatly enhance the usefulness of the SORP in clinical applications. Furthermore, we focused more on the clinical characteristics of HR patients, especially septic NS patients, and the assessment of newly discovered NS_HR patients, overlooked by the Sepsis-3 definition, may provide further insights into the pathophysiological processes of SS onset and help to complement its diagnosis and precise management.
In this study, 4 risk groups were identified by the SORP. For these risk groups, we observed distinct clinical features and a significantly stratified risk for SS onset rates and survival status. For SS patients in different risk groups, the time from sepsis onset to SS decreased as their risk increased. Significant survival differences were observed in NS patients across different risk groups, but these same differences were not observed in SS patients (Figure 3E). This may be because of the active guideline-directed resuscitation treatment of SS patients. These findings confirm the effectiveness of the current SS resuscitation treatment guidelines and further affirm the accuracy of risk stratification and the reliability of the SORP.
Interestingly, despite the high agreement between SS patients diagnosed by the Sepsis-3 definition and those identified by the SORP, we still identified 2 relatively distinct groups. One group was SS_LR, where patients were defined as having SS by Sepsis-3 but were identified as LR by the SORP. The overall survival of these patients was not significantly different from that of other SS groups, but rapid progression and an extremely increased risk were demonstrated within 6 hours before SS onset (Figure 5A). In addition, 52% (35/67) of SS_LR patients showed an increase from LR to MR 3 hours before SS onset. Thus, although these patients could not be identified 6 hours in advance, their risk changes were captured in real-time, which highlights the high sensitivity of the SORP. Further analysis showed that SS_LR patients underwent more invasive procedures than other SS patients and much more than other patients in the LR group, such as “invasive ventilation,” “dialysis catheter,” “arterial line,” and “dialysis-CRRT” (Figure 5B and 5C). Thus, we speculate that invasive procedures might be one of the causes of the rapidly exacerbating patient status. We recommend that LR patients undergoing invasive procedures should be evaluated more frequently to recognize rapid exacerbation in a timely manner.
Another notable group was NS_HR, where patients were not defined as having SS by Sepsis-3 but were identified as HR by the SORP. This prompted us to further analyze this particular patient group, and the results showed that these patients had the following characteristics: (1) NS_HR patients were quite similar to SS patients in terms of clinical phenotypes (Figure 4A-C); (2) NS_HR patients had the worst overall survival, which was significantly poorer than even that of SS_HR patients (Figure 3C and 3D; Figure 4D); (3) NS_HR patients did not receive sufficient fluid resuscitation or timely vasopressor administration (Figure 4E), which might be a key factor that resulted in their noncompliance with the SS diagnosis and their higher mortality; and (4) NS_HR patients had more significant abnormalities in renal-related features (Figure 4E), which reflect higher incidences of renal malfunction, such as CKD (Figure 4E). The literature has shown that excessive fluid resuscitation in patients with severe renal disease exacerbates renal injury and leads to poorer outcomes [,]. Therefore, cautious and conservative strategies might be adopted by physicians, resulting in a much higher likelihood of insufficient fluid resuscitation in these patients. Hence, precise fluid resuscitation management is crucial in SS patients with renal dysfunction. In total, NS_HR patients share a similar pathophysiologic process with SS patients but experience a more aggressive clinical outcome due to insufficient treatment.
Compared to previous versions, the Sepsis-3 shock definition is more focused on tissue-organ hypoperfusion and metabolic dysfunction and the corresponding therapeutic response, which largely ensures the authenticity and reliability of SS diagnosis, showing a significant increase in specificity but at the cost of sensitivity []. Studies have shown that a proportion of patients who met the previous version of the SS criteria, but not the Sepsis-3 shock criteria, had significant organ failure and high mortality [], with a value of up to 31.6% []. In this study, we identified a group of patients (NS_HR) with poor survival and a similar clinical phenotype to SS. In addition to low vasopressor use, the NS_HR group had inadequate fluid resuscitation due to severe renal disease, thus failing the Sepsis-3 shock definition. These patients are missed by the current SS definition, but they are at HR and are worthy of attention. Therefore, for more precise diagnosis and management of SS patients, “adequate fluid resuscitation” in the SS definition may have different criteria in different populations. The need for fluid personalization has emerged as an admirable goal, especially in SS [].
Limitations
In this study, the designed SS risk early forewarning and stratification model, SORP, can easily and quickly provide ICU physicians with the patient’s risk status and its changing trend, which can provide valuable support for the early identification of patient deterioration. However, it should be noted that there are limitations in this study. The current data sources are relatively limited in terms of geographic regions, which to some extent hinders the study of differences in different regions. Future studies should collect data from different regions to test the accuracy of the SORP for early forewarning, analyze the impact of regional differences on risk factors for SS, and establish a more accurate early forewarning system. Moreover, it is crucial to regularly monitor and assess the early forewarning capability and stability of the system and to make necessary adjustments as additional data becomes available. In addition, in the eICU dataset, we were unable to define SS by the Sepsis-3 criteria due to data limitations, which resulted in the incomparability of performance metrics such as AUC. However, based on a different SS definition, we were still able to observe results similar to those with the MIMIC-IV data, which further confirms the robustness of the SORP and the rationality of the risk stratification system.
Summary and Outlook
Early forewarning and diagnosis of SS are essential for reducing mortality. To support this, we developed the SORP early forewarning model, which offers several key benefits: (1) easy and rapid feature collection within minutes; (2) ease of use via online tools or a scorecard (); (3) excellent predictive performance (Figure 2B); (4) continuous status monitoring for risk deterioration or improvement; and (5) identification of HR patients, including those with similar symptoms who may not fully meet diagnostic criteria. In clinical practice, the SORP can be easily integrated into sepsis management as an auxiliary monitoring tool (). For MR patients whose SORP score is still decreasing, it is recommended to increase the frequency of blood gas testing for timely monitoring. For those identified as HR, the SS determination process can be initiated ahead of time, facilitating the early administration of appropriate treatments, such as fluid resuscitation.
Future research should further explore the applicability of the SORP model in diverse clinical settings, including validation with additional datasets. We have also planned a prospective trial to evaluate its impact on prognosis and survival, particularly in NS_HR patients. These efforts aim to advance early SS identification and management, ultimately improving patient outcomes.
Acknowledgments
This study was supported partly by the Natural Science Foundation of Fujian Province (grant number: 2022J05054, 2023J01321), Fujian Science and Technology Innovation Joint Fund Project (grant number: 2020Y9035), Fujian Key Laboratory of Medical Bioinformatics Open Fund (grant number: FKLMB-202002), and Youth Scientific Research Fund Program of School of Medical Technology and Engineering, Fujian Medical University (2021xy001).
Data Availability
The data generated and analyzed during this study are available from the corresponding author upon request. The electronic health record data used in this study are openly available and can be downloaded at the MIMIC-IV [] and eICU [].
Authors' Contributions
GL, JH, TC, SZ, and XS contributed to designing and performing the research, analyzing the data, and writing the manuscript. RW, ZZ, YY, HF, LH, HZ, HS, HY, and YX contributed to performing the research. XS also conceived and supervised the project and was responsible for its overall strategy and management. XS is the guarantor of the manuscript and takes responsibility for the content of the manuscript, including the data and analysis.
Conflicts of Interest
None declared.
Multimedia Appendix 1
Checklist according to the TRIPOD (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis) guidelines.
DOCX File , 53 KBMultimedia Appendix 2
Septic shock risk scorecard for the Septic Shock Risk Predictor.
DOCX File , 52 KBMultimedia Appendix 4
Distribution stability of septic shock in Medical Information Mart for Intensive Care-IV (MIMIC-IV) for different risk groups.
DOCX File , 435 KBMultimedia Appendix 5
Medical Information Mart for Intensive Care-IV (MIMIC-IV) data for the rate of patients with septic shock in each risk group.
DOCX File , 49 KBMultimedia Appendix 6
The onset rate of septic shock in each risk score according to the fitted curve.
DOCX File , 52 KBMultimedia Appendix 7
Boxplots showing change trends for each clinical feature by risk groups in Medical Information Mart for Intensive Care-IV (MIMIC-IV) data.
DOCX File , 883 KBMultimedia Appendix 8
Medical Information Mart for Intensive Care-IV (MIMIC-IV) data for the clinical event distribution and significance of risk groups.
DOCX File , 50 KBMultimedia Appendix 9
The changing trends in risk across the whole intensive care unit (ICU) process.
DOCX File , 312 KBMultimedia Appendix 10
Medical Information Mart for Intensive Care-IV (MIMIC-IV) data for the invasive operation distribution and the significance of septic shock risk groups.
DOCX File , 50 KBMultimedia Appendix 11
Medical Information Mart for Intensive Care-IV (MIMIC-IV) data for the invasive operation distribution and the significance of low-risk groups.
DOCX File , 50 KBMultimedia Appendix 12
eICU Collaborative Research Database (eICU) data for the rate of patients with septic shock in each risk group.
DOCX File , 49 KBMultimedia Appendix 13
Distribution stability of septic shock between the eICU Collaborative Research Database (eICU) and Medical Information Mart for Intensive Care-IV (MIMIC-IV) datasets.
DOCX File , 438 KBMultimedia Appendix 14
Boxplots showing change trends for each clinical feature by risk groups in the eICU Collaborative Research Database (eICU) data.
DOCX File , 584 KBMultimedia Appendix 15
eICU Collaborative Research Database (eICU) data for the invasive operation distribution and the significance of low-risk groups.
DOCX File , 50 KBMultimedia Appendix 16
eICU Collaborative Research Database (eICU) data for the invasive operation distribution and the significance of septic shock risk groups.
DOCX File , 49 KBMultimedia Appendix 17
eICU Collaborative Research Database (eICU) data for the clinical event distribution and the significance of risk groups.
DOCX File , 50 KBMultimedia Appendix 19
Schematic diagram for auxiliary monitoring of the Septic Shock Risk Predictor (SORP) in clinical practice.
DOCX File , 144 KBReferences
- Singer M, Deutschman CS, Seymour CW, Shankar-Hari M, Annane D, Bauer M, et al. The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). JAMA. Feb 23, 2016;315(8):801-810. [FREE Full text] [CrossRef] [Medline]
- Rudd KE, Johnson SC, Agesa KM, Shackelford KA, Tsoi D, Kievlan DR, et al. Global, regional, and national sepsis incidence and mortality, 1990-2017: analysis for the Global Burden of Disease Study. Lancet. Jan 18, 2020;395(10219):200-211. [FREE Full text] [CrossRef] [Medline]
- Kaukonen K, Bailey M, Suzuki S, Pilcher D, Bellomo R. Mortality related to severe sepsis and septic shock among critically ill patients in Australia and New Zealand, 2000-2012. JAMA. Apr 02, 2014;311(13):1308-1316. [CrossRef] [Medline]
- Daviaud F, Grimaldi D, Dechartres A, Charpentier J, Geri G, Marin N, et al. Timing and causes of death in septic shock. Ann Intensive Care. Dec 2015;5(1):16. [FREE Full text] [CrossRef] [Medline]
- Kumar A, Roberts D, Wood KE, Light B, Parrillo JE, Sharma S, et al. Duration of hypotension before initiation of effective antimicrobial therapy is the critical determinant of survival in human septic shock. Crit Care Med. Jun 2006;34(6):1589-1596. [CrossRef] [Medline]
- Vincent J. The clinical challenge of sepsis identification and monitoring. PLoS Med. May 17, 2016;13(5):e1002022. [FREE Full text] [CrossRef] [Medline]
- Talisa VB, Yende S, Seymour CW, Angus DC. Arguing for adaptive clinical trials in sepsis. Front Immunol. 2018;9:1502. [FREE Full text] [CrossRef] [Medline]
- de Grooth H, Postema J, Loer SA, Parienti J, Oudemans-van Straaten HM, Girbes AR. Unexplained mortality differences between septic shock trials: a systematic analysis of population characteristics and control-group mortality rates. Intensive Care Med. Mar 2018;44(3):311-322. [FREE Full text] [CrossRef] [Medline]
- Yang H, Luo Y, Ma C, Zhang T, Zhou T, Ren X, et al. A gender specific risk assessment of coronary heart disease based on physical examination data. NPJ Digit Med. Jul 31, 2023;6(1):136. [FREE Full text] [CrossRef] [Medline]
- Yang H, Luo Y, Ren X, Wu M, He X, Peng B, et al. Risk Prediction of Diabetes: Big data mining with fusion of multifarious physical examination indicators. Information Fusion. Nov 2021;75:140-149. [CrossRef]
- Zhang Z, Zhang G, Goyal H, Mo L, Hong Y. Identification of subclasses of sepsis that showed different clinical outcomes and responses to amount of fluid resuscitation: a latent profile analysis. Crit Care. Dec 18, 2018;22(1):347. [FREE Full text] [CrossRef] [Medline]
- Yang T, Chen L, Lin C, Lin H, Fang T, Ge H, et al. Identifying novel clusters of patients with prolonged mechanical ventilation using trajectories of rapid shallow breathing index. Front Med (Lausanne). Jul 4, 2022;9:880896. [FREE Full text] [CrossRef] [Medline]
- Seymour CW, Kennedy JN, Wang S, Chang CH, Elliott CF, Xu Z, et al. Derivation, validation, and potential treatment implications of novel clinical phenotypes for sepsis. JAMA. May 28, 2019;321(20):2003-2017. [FREE Full text] [CrossRef] [Medline]
- Delahanty RJ, Alvarez J, Flynn LM, Sherwin RL, Jones SS. Development and evaluation of a machine learning model for the early identification of patients at risk for sepsis. Ann Emerg Med. Apr 2019;73(4):334-344. [CrossRef] [Medline]
- Khojandi A, Tansakul V, Li X, Koszalinski RS, Paiva W. Prediction of sepsis and in-hospital mortality using electronic health records. Methods Inf Med. Sep 2018;57(4):185-193. [CrossRef] [Medline]
- Scherpf M, Gräßer F, Malberg H, Zaunseder S. Predicting sepsis with a recurrent neural network using the MIMIC III database. Comput Biol Med. Oct 2019;113:103395. [CrossRef] [Medline]
- Nemati S, Holder A, Razmi F, Stanley MD, Clifford GD, Buchman TG. An interpretable machine learning model for accurate prediction of sepsis in the ICU. Crit Care Med. Apr 2018;46(4):547-553. [FREE Full text] [CrossRef] [Medline]
- Kam HJ, Kim HY. Learning representations for the early detection of sepsis with deep neural networks. Comput Biol Med. Oct 01, 2017;89:248-255. [CrossRef] [Medline]
- Henry KE, Adams R, Parent C, Soleimani H, Sridharan A, Johnson L, et al. Factors driving provider adoption of the TREWS machine learning-based early warning system and its effects on sepsis treatment timing. Nat Med. Jul 2022;28(7):1447-1454. [CrossRef] [Medline]
- Lin C, Zhang Y, Ivy J, Capan M, Arnold R, Huddleston J. Early Diagnosis and Prediction of Sepsis Shock by Combining Static and Dynamic Information Using Convolutional-LSTM. 2018. Presented at: IEEE International Conference on Healthcare Informatics (ICHI); June 04-07, 2018; New York, NY. [CrossRef]
- Henry KE, Hager DN, Pronovost PJ, Saria S. A targeted real-time early warning score (TREWScore) for septic shock. Sci Transl Med. Aug 05, 2015;7(299):299ra122. [CrossRef] [Medline]
- Fagerström J, Bång M, Wilhelms D, Chew MS. LiSep LSTM: A machine learning algorithm for early detection of septic shock. Sci Rep. Oct 22, 2019;9(1):15132. [FREE Full text] [CrossRef] [Medline]
- Liu R, Greenstein JL, Granite SJ, Fackler JC, Bembea MM, Sarma SV, et al. Data-driven discovery of a novel sepsis pre-shock state predicts impending septic shock in the ICU. Sci Rep. Apr 16, 2019;9(1):6145. [FREE Full text] [CrossRef] [Medline]
- Mollura M, Romano S, Mantoan G, Lehman L, Barbieri R. Prediction of Septic Shock Onset in ICU by Instantaneous Monitoring of Vital Signs. 2020. Presented at: 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC); July 20-24, 2020; Montreal, QC, Canada. [CrossRef]
- Rangan ES, Pathinarupothi RK, Anand KJS, Snyder MP. Performance effectiveness of vital parameter combinations for early warning of sepsis-an exhaustive study using machine learning. JAMIA Open. Dec 2022;5(4):ooac080. [FREE Full text] [CrossRef] [Medline]
- Kim JK, Ahn W, Park S, Lee S, Kim L. Early prediction of sepsis onset using neural architecture search based on genetic algorithms. Int J Environ Res Public Health. Feb 18, 2022;19(4):2349. [FREE Full text] [CrossRef] [Medline]
- Goh KH, Wang L, Yeow AYK, Poh H, Li K, Yeow JJL, et al. Artificial intelligence in sepsis early prediction and diagnosis using unstructured data in healthcare. Nat Commun. Jan 29, 2021;12(1):711. [FREE Full text] [CrossRef] [Medline]
- Septic Shock Risk Predictor (SORP): an online service for early forewarning and risk stratification of septic shock. Department of Bioinformatics, Fujian Medical University. URL: http://www.informatics.icu/SORP/home [accessed 2025-01-09]
- Collins GS, Moons KGM, Dhiman P, Riley RD, Beam AL, Van Calster B, et al. TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ. Apr 16, 2024;385:e078378. [FREE Full text] [CrossRef] [Medline]
- Johnson A, Bulgarelli L, Pollard T, Horng S, Celi LA, Mark R. MIMIC-IV. PhysioNet. URL: https://physionet.org/content/mimiciv/2.0/ [accessed 2025-01-09]
- Pollard TJ, Johnson AEW, Raffa JD, Celi LA, Mark RG, Badawi O. The eICU Collaborative Research Database, a freely available multi-center database for critical care research. Sci Data. Sep 11, 2018;5:180178. [FREE Full text] [CrossRef] [Medline]
- Lundberg SM, Lee SI. A unified approach to interpreting model predictions. In: NIPS'17: Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017. Presented at: 31st International Conference on Neural Information Processing Systems; December 4-9, 2017; Long Beach, CA. [CrossRef]
- Dodd RL, Archambault ME. Preventing kidney injury by avoiding fluid overload in patients with sepsis. JAAPA. Dec 2019;32(12):40-45. [CrossRef] [Medline]
- Rice DM, Ratliff PD, Judd WR, Kseibi SA, Eberwein KA. Assessing the impact of CKD on outcomes in septic shock patients receiving standard Vs reduced initial fluid volume. Am J Emerg Med. Oct 2020;38(10):2147-2150. [CrossRef] [Medline]
- Henning DJ, Puskarich MA, Self WH, Howell MD, Donnino MW, Yealy DM, et al. An emergency department validation of the SEP-3 sepsis and septic shock definitions and comparison with 1992 consensus definitions. Ann Emerg Med. Oct 2017;70(4):544-552.e5. [FREE Full text] [CrossRef] [Medline]
- Sterling SA, Puskarich MA, Glass AF, Guirgis F, Jones AE. The impact of the Sepsis-3 septic shock definition on previously defined septic shock patients. Crit Care Med. Sep 2017;45(9):1436-1442. [FREE Full text] [CrossRef] [Medline]
- Vermassen J, Decruyenaere J, De Bus L, Depuydt P, Colpaert K. Characteristics of Sepsis-2 septic shock patients failing to satisfy the Sepsis-3 septic shock definition: an analysis of real-time collected data. Ann Intensive Care. Oct 30, 2021;11(1):154. [FREE Full text] [CrossRef] [Medline]
- Monnet X, Lai C, Teboul J. How I personalize fluid therapy in septic shock? Crit Care. Mar 24, 2023;27(1):123. [FREE Full text] [CrossRef] [Medline]
- Pollard T, Johnson A, Raffa J, Celi LA, Badawi O, Mark R. eICU Collaborative Research Database (version 2.0). PhysioNet. URL: https://physionet.org/content/eicu-crd/2.0/ [accessed 2025-01-09]
Abbreviations
| ABG: arterial blood gas |
| AUC: area under the curve |
| BIDMC: Beth Israel Deaconess Medical Center |
| CKD: chronic kidney disease |
| CRRT: dialysis-continuous renal replacement therapy |
| eICU: eICU Collaborative Research Database |
| HR: high risk |
| ICD: International Classification of Diseases |
| ICU: intensive care unit |
| IV: information value |
| KS: Kolmogorov-Smirnov |
| LogR: logistic regression |
| LR: low risk |
| MBP: mean blood pressure |
| MIMIC-IV: Medical Information Mart for Intensive Care-IV |
| MR: medium risk |
| NS: nonshock |
| NS_O: other nonshock patients, excluding NS_HR |
| PSI: Population Stability Index |
| SHAP: Shapley Additive Explanations |
| SOFA: Sequential Organ Failure Assessment |
| SORP: Septic Shock Risk Predictor |
| SS: septic shock |
| ULR: ultralow risk |
| VS: vital sign |
| WOE: weight of evidence |
Edited by A Schwartz; submitted 09.04.24; peer-reviewed by BN Beshey, S Liu; comments to author 26.10.24; revised version received 09.11.24; accepted 19.12.24; published 06.02.25.
Copyright©Guanghao Liu, Shixiang Zheng, Jun He, Zi-Mei Zhang, Ruoqiong Wu, Yingying Yu, Hao Fu, Li Han, Haibo Zhu, Yichang Xu, Huaguo Shao, Haidan Yan, Ting Chen, Xiaopei Shen. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 06.02.2025.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research (ISSN 1438-8871), is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.