Clinical Timing-Sequence Warning Models for Serious Bacterial Infections in Adults Based on Machine Learning: Retrospective Study

Introduction

The detection and management of infectious diseases continue to be a primary concern for individuals experiencing fever [1]. The prevalence of infectious diseases is notably elevated, especially in regions characterized by low and lower-middle income [2,3]. Severe infectious diseases have the potential to rapidly advance to conditions such as sepsis, septic shock, or, in extreme cases, fatalities. This exerts significant strain on health care services and results in a depletion of intensive care resources. Swiftly identifying and accurately diagnosing serious bacterial infections (SBIs) are imperative to promptly initiate suitable treatments. The established gold standard for diagnosing infections in clinical samples relies on a process involving incubation, isolation, and identification. However, this method has its drawbacks, including a notable incidence of false-positive and false-negative outcomes from bacterial cultures. Additionally, the procedure is time-consuming, typically requiring 2-3 days or even longer to yield a definitive result [4,5]. The management of SBIs demands prompt action due to the time-sensitive nature of the condition, which can rapidly deteriorate. Nonetheless, the inappropriate use of broad-spectrum antibiotics or combinations of antimicrobials carries various drawbacks, including heightened health care expenditures, adverse drug reactions, and an elevated risk of developing drug-resistant bacteria. Therefore, the timely detection of SBIs in febrile patients suspected of having an infectious origin is paramount. This approach holds the potential to save lives and encourage a swift response without resorting to antibiotic misuse.

In this digital era, electronic health records serve as an extensive repository of electronic data points encompassing a wide range of clinical information [6]. Machine learning (ML) techniques possess a distinctive capability to analyze vast data sets in a flexible and trainable manner, enabling them to comprehend the intricate relationships between variables [7]. Owing to their enhanced processing capabilities, a variety of ML and artificial intelligence (AI) techniques are extensively used for identifying risk factors for diseases in patients and providing assistance to clinicians. Nevertheless, a previous study [8] has indicated that diverse feature selection methods and classification techniques can yield differing performance outcomes. Therefore, determining the optimal ML methods is essential for ensuring stable and accurate predictions when applied in clinical settings.

Several clinical models utilizing ML have been developed and validated for SBIs in infants and young children [9-11]. Yiu et al [12] specifically created a multivariable model to forecast the risk of serious infection in patients with psoriasis. Nonetheless, the discriminative ability was found to be unsatisfactory, with a C-statistic of only 0.64. Rawson et al [13] developed and validated a model to predict the presence of bacterial infection within 72 hours. However, the prediction did not prove to be timely enough for SBIs, and no applicable tools for physicians are currently available for use in their clinical practice. Qu et al [14] conducted a study comparing the timing-sequence recovery effect on exercise-induced muscle damage at various time points, including 1, 24, 48, and 72 hours, shedding new light on the timely prediction of SBIs. Building on the insights from this research, our study aims to develop and validate clinical timing-sequence warning models for SBIs using multiple ML methods. The goal is to provide physicians with practical and clinically applicable tools for use in their busy clinical practice for patients suspected of having an infective fever.

Methods

Study Design and Data Preprocessing

This retrospective observational cohort study was conducted at a 2200-bed teaching hospital, spanning from January 2013 to December 2020. An independent physician, blinded to the study, was invited to collect data from January to December 2021 within the same hospital. Patients were included in the study if they met the following criteria: (1) diagnosed with “fever of unknown origin” or presented with a chief complaint of fever persisting for several days, and (2) had a temperature of ≧37.5°C, indicative of fever. Exclusion criteria encompassed patients who met any of the following: (1) age <18 years, (2) pregnancy, (3) presence of an SBI upon admission, (4) noninfectious fever, (5) missing data on the variables of interest, and (6) undiagnosed fever. More specific details about the patient enrollment process are illustrated in Figure 1. The confirmation of SBI diagnosis involved discussions with the clinical research team and a review of the read code classification [15] and the International Classification of Diseases, 10th revision [16].

The variables of interest were obtained from the hospitalization records, encompassing demographic information such as sex, age, BMI, past medical history (hypertension, diabetes mellitus, heart disease, nephropathy, hepatitis B, and malignant tumors), as well as laboratory test results collected within the initial 24 hours of admission to the hospital. Outlier values, defined as falling below the 0.5th percentile or exceeding the 99.5th percentile, were adjusted to the values at the 0.5th percentile if lower or the 99.5th percentile if higher. Subsequently, these outliers were removed from the data set. Any other missing laboratory values were imputed using the K-nearest neighbors approach, replacing them with the mean of the respective feature from the 10 most similar samples in the training data. The data underwent normalization for each laboratory feature through transformations involving subtraction of the mean and division by the SD [17]. The research adhered to the CONSORT (Consolidated Standards of Reporting Trials) checklist [18] (Multimedia Appendix 1) and followed the “Guidelines for Developing and Reporting Machine Learning Predictive Models in Biomedical Research” [19].

‎

Ethics Considerations

Data retrieval received approval from the ethics committee of the First Affiliated Hospital, College of Medicine, Zhejiang University (ITT20220327B; date of approval: July 19, 2022). No potentially identifiable human images or data were included in the data collection, and the requirement for informed consent was thus waived.

Data Partitioning and Statistical Analysis

The data were randomly classified into training and test sets at a ratio of 7:3, using equal proportion sampling. This was done to ensure that both the training and test sets encompassed approximately the same proportion of positive events (ie, SBIs). Descriptive statistics, including the IQR and frequency (proportion), were utilized to illustrate the distribution of the features of interest. The chi-square test (or the Fisher exact probability test) was applied for categorical variables, while the Wilcoxon rank sum test was used for continuous variables to assess the presence of differences in the distribution of clinical features between the 2 data sets. Differences between the SBI and non-SBI groups were assessed using either the Wilcoxon rank sum test or the chi-square test. Furthermore, the correlation between variables was gauged by calculating Pearson’s correlation coefficients and visualized in a heatmap. All statistical analyses and algorithms were executed using R, version 4.0.2 (R Foundation) statistical software.

Feature Selection

Three popular algorithms—Boruta, Lasso (least absolute shrinkage and selection operator), and RFE (recursive feature elimination)—were utilized to screen important features in this study. This approach was adopted to prevent a preference bias that may arise from relying solely on 1 algorithm. To provide further detail, Boruta uses the random forest (RF) model as a classifier package algorithm [20]. Lasso operates on a linear regression model by selecting and compressing variables, effectively addressing the issue of overfitting [21]. By contrast, RFE selects features by recursively reducing the size of the examined feature set [22]. Features deemed important were those marked as such by at least two ML algorithms, a measure taken to enhance the simplicity and stability of the models.

Model Construction

The chosen features were subsequently used to construct models predicting SBIs using logistic regression (LR), RF, and extreme gradient boosting (XGBoost) analyses, incorporating 5-fold cross-validation (CV). Considering the clinical timing of indicator acquisition (ie, basic characteristics and routine blood indices were assessed and obtained immediately after admission, while other serum biochemical indices were typically obtained 12-24 hours later) and the practical demands of clinical practice, 2 clinical timing-sequence warning models were developed. Specifically, these models were designed for 2 purposes: the prediction of SBI risk at early admission (model 1), incorporating basic characteristics plus routine blood indices, and a prediction within 24 hours of admission (model 2), utilizing basic characteristics plus biochemical indicators.

Performance metrics, including the receiver operating characteristic (ROC) curve and area under the ROC curve (AUC), accuracy, sensitivity, specificity, positive predictive value, negative predictive value, positive likelihood ratio (LR+), and negative likelihood ratio (LR–), were used. The optimal cutoff value was determined based on the Youden index.

Construction and Evaluation of the Predictive Nomogram

Nomograms predicting the risk of SBIs were generated based on the best model, providing a convenient tool for physicians to assess SBI risk for individual patients in clinical practice. Independent data were utilized for further validation of the nomogram’s performance and clinical utility. Additionally, to enhance practical usability for health care professionals globally and enable open validation by peers, we have implemented the 2 nomogram tools online using DynNom [23,24]. Through this platform, the SBI risk score can be automatically generated and visually displayed.

Results

Patient Characteristics

In this study, a total of 945 patients clinically diagnosed with infective fever and possessing complete clinical data were included. Table 1 presents the demographic characteristics of these patients. Among them, data from 661 patients were utilized for training, while data from 284 patients were used for validation. The median age of the patients was 54 years (IQR 38-66 years), and 47.4% (448/945) were female. The most prevalent comorbidity among the patients was hypertension, affecting 27.9% (264/945), while other comorbidities such as diabetes mellitus, cardiopathy, and nephropathy were observed in 12.1% (114/945), 6.3% (60/945), and 5.5% (52/945) of patients, respectively. In the training cohort, consisting of 661 patients, 69% (456/661) were diagnosed with SBIs. The median age of the patients was 59 years, and among them, 226 were female. Further clinical characteristics of the patients with and without SBIs are detailed in Multimedia Appendix 2.

Regarding the serious infectious episodes, the top 3 clinical sites of infections were the lung (pneumonia; 224/945, 23.7%), the urinary tract (83/945, 8.8%), and the abdomen (78/945, 8.3%). Based on the clinical course and microbiological data, bacterial infections and viral infections were present in 717 patients (717/945, 75.9%) and 185 patients (185/945, 19.6%), respectively (Multimedia Appendix 3).

Characteristics of the Selected Features

A total of 37 indicators, spanning demographic variables, clinical characteristics, vital signs, and blood biochemical indicators, were screened and compared. Notably, 14, 9, and 2 features were identified as important by the Boruta, Lasso, and RFE algorithms, respectively (Figure 2A-2D). A total of 18 independent features were recognized, with nearly all demonstrating significant differences (P<.05) between the SBI and non-SBI groups (Multimedia Appendix 2). However, notable collinearity was observed among these indicators, as depicted in Multimedia Appendix 4. To enhance the simplicity and stability of the models, features identified as important by at least two ML algorithms were chosen for model construction. These included age, neutrophil proportion (NP), hemoglobin, C-reactive protein (CRP), and fibrinogen levels (Figure 2E). Additionally, given that the neutrophil-to-lymphocyte ratio (NLR) simultaneously reflects lymphocyte proportions and NPs and is commonly used in clinical practice, it was used as a replacement for NP.

‎

Model Performance Result

Considering the sequence of collection for the chosen indicators and the clinical requirements, 2 timing-sequence models for predicting SBI risk were formulated. The early admission model (model 1) incorporated age, hemoglobin level, and NLR, while the model within 24 hours of admission (model 2) included age, hemoglobin, NLR, fibrinogen, and CRP levels. LR exhibited greater stability compared with RF and XGBoost in the 5-fold CV of both models. For model 1, LR achieved AUCs of 0.749 (5-fold CV training set) and 0.744 (5-fold CV validation set), while for model 2, LR showed AUCs of 0.806 (5-fold CV training set) and 0.807 (5-fold CV validation set; see panels A and B in Multimedia Appendix 5). Moreover, LR outperformed the other 2 algorithms in model 2, with an AUC, accuracy, and sensitivity of 0.780 (95% CI 0.720-841), 0.754 (95% CI 0.698-804), and 0.776 (95% CI 0.711-832), respectively, as presented in Tables 2 and 3. In model 1, XGBoost demonstrated an advantage over LR in terms of AUC (0.708, 95% CI 0.641-775 vs 0.686, 95% CI 0.617-754), while RF achieved superior accuracy (0.729, 95% CI 0.673-780) and sensitivity (0.790, 95% CI 0.728-844) compared with the other 2 algorithms (Tables 2 and 3).

Evaluation of the Predictive Nomogram

Two SBI-predicting nomograms based on LR, offering a succinct SBI risk score for each patient, were developed for models 1 and 2 (Figure 3A and 3C). To assess the clinical utility of the predictive nomograms, data on 164 patients diagnosed with infective fever and possessing complete prediction indicators (ie, age, hemoglobin level, lymphocyte count, neutrophil count, fibrinogen level, and CRP level) collected in 2021 were included. The early predictive nomogram exhibited good accuracy and sensitivity, with accuracy at 0.707 and sensitivity at 0.729. In the assessment of the model within 24 hours of admission, high sensitivity and good accuracy were observed, with sensitivity at 0.927 and accuracy at 0.750 (Figure 3B and 3D). These results demonstrate practical utility in clinical practice for distinguishing SBIs from non-SBIs. Two dynamic nomogram tools are deployed online [23,24].

‎

Discussion

Principal Findings

Given the advancement of AI and the growing need for extensive data in clinical medicine, the integration of AI techniques into clinical practice and research holds significant importance. This retrospective study focused on the development of 2 timing-sequence SBI-risk models for adults with infectious fever. Leveraging multiple ML algorithms, these models were designed with a minimal set of 3 and 5 features, aiming to achieve accurate and stable predictions. The nomograms developed and validated for physicians provide a straightforward and convenient way to assess the risk of SBI for each patient in clinical practice. This suggests their promising potential in clinical decision-making, aiming to minimize unnecessary hospitalizations and the empirical use of antibiotics. As far as our knowledge extends, this study is the first to utilize timing-sequence models for detecting SBI in adults with suspected infectious fever, and their validation in clinical applications adds a novel dimension to the existing body of research.

The utilization of various feature screening methods comes with its set of advantages and disadvantages. In this study, we opted for a comprehensive approach by combining multiple feature screening methods with clinician advice for the final confirmation of features. The features selected by multiple ML algorithms, namely, age, hemoglobin levels, CRP levels, fibrinogen levels, and the NLR, align with the consensus in the current management of SBIs. This alignment enhances the reliability and interpretability of the models we constructed.

Indeed, aging has been associated with a decrease in the efficiency of bacterial killing by neutrophils under ex vivo conditions [25]. The reduced ability of older adults to mount an adequate immune response makes them more susceptible to bacterial infections compared with their younger counterparts [26].

The SBI group exhibited lower hemoglobin levels and higher CRP levels, aligning with findings from prior studies [27,28]. The inflammatory response to an infection, characterized by the secretion of interleukin-6 (IL-6) in significant amounts, could be a primary factor contributing to anemia [29,30]. IL-6 plays a role in upregulating hepcidin transcription, inhibiting the transport of intracellularly stored iron from macrophages and hepatocytes into the plasma, and reducing intestinal iron uptake, resulting in low levels of iron in the plasma [31]. The mechanism, initiated within hours of infection, restricts the availability of plasma iron required for erythroblast proliferation, consequently inhibiting erythropoiesis [31]. CRP is an acute-phase reactive protein released in significant amounts from the liver upon stimulation with IL-6 and other cytokines [32]. In the presence of infection, CRP plays dual roles by exerting both proinflammatory and anti-inflammatory effects. Its functions are mediating the elimination of pathogens and inhibiting the interaction between white blood cells and endothelial cells. Consequently, CRP is frequently used for the early diagnosis of infectious diseases. However, the diagnostic value of CRP alone for bacterial infections is considered only moderate. Indeed, elevated CRP levels are observed not only in pediatric patients infected with adenovirus in the absence of a secondary bacterial infection but also in patients with bacterial infections. This suggests that both viruses and bacteria can trigger an inflammatory host response [33].

Fibrinogen possesses an intrinsic ability to combat invading bacterial pathogens [34]. In a recent study, wild-type mice demonstrated rapid and efficient clearance of Staphylococcus aureus, while mice lacking fibrinogen (Fbg^–) failed to clear the microbes. This disparity in the early clearance of bacteria resulted in all wild-type mice surviving the infection even after 2 weeks, whereas all Fbg^– mice succumbed within 24 hours [35]. Fibrinogen serves as an antimicrobial host defense protein through various mechanisms, including (1) establishing a physical barrier at the air–liquid interface, (2) entrapping bacteria in fibrinogen or fibrin networks, and (3) facilitating the recruitment and activation of host immune cells [34].

The sensitivity of the NLR in the diagnosis of bacteremia, infection, and sepsis has been validated in numerous studies [36-38]. Aird [39] reported that the hematologic system plays a primary role in sepsis pathogenesis. In systemic infection and sepsis, all blood cells undergo activation, leading to significant changes in their functions, counts, receptor expression, humoral substances, and the secretion of various signaling molecules. A complete blood count can offer a wealth of valuable information, including parameters such as the NLR.

The clinical presentation of SBIs is often variable and nonspecific [40]. Therefore, the application of ML to assess SBI risk is feasible, given its capability to identify associations not previously considered among a large number of variables. Moreover, the potential of ML tends to increase with larger data sets. These 3 ML algorithms are extensively used in constructing clinical prediction models and consistently demonstrate excellent performance. Given the variation in information, specifically blood indicators, obtained at admission among patients, we devised 2 sequential models to enhance clinical applicability. Both models demonstrated commendable performance in early and late evaluations, aiding in the early warning of SBIs. In the early model (model 1), the RF model yielded the best results, achieving the highest accuracy scores (0.731 for the training set and 0.729 for the test set). RF is an ensemble learning method known for its effectiveness in various classification and regression tasks [41]. It operates as a classifier that leverages multiple trees for training and predicting outcomes. Notably, RF exhibits high accuracy and is adept at balancing errors when analyzing data sets with unbalanced classifications [42]. In the later model (model 2), the LR model demonstrated an excellent accuracy score, achieving 0.777 for the training set and 0.754 for the test set. LR is a classic predictive analysis algorithm rooted in the concept of probability. It utilizes a sigmoid function to derive the predicted output for classification [10]. To visualize and quantify the LR results, a nomogram was constructed, with regression coefficients standardized and expressed as risk scores on a number line. The nomogram serves as a highly practical visual tool for constructing clinical models, offering great convenience for physicians in busy clinical practice. Our nomograms were rigorously tested to demonstrate their capability to sensitively and accurately identify patients at a high risk of SBIs. This aspect is critical in clinical practice, providing physicians with timely alerts to take proactive measures and avoid the development of severe conditions. Additionally, we deployed online dynamic nomogram tools, making model parameters publicly accessible on the same website. This setup enables health care professionals and peers to openly validate the clinical utility of the model, further enhancing its practical usage.

Strengths

The strength of this study lies in the meticulous process of feature capture, ensuring that the selected covariates are highly relevant to the risk of SBIs in a clinical setting. This approach facilitates the construction of concise and stable models with clinical interpretation. We anticipate that the findings of our study could serve as a valuable reference for the application of ML models in various health care systems. Second, our study introduced and developed clinical timing-sequence warning models, representing an innovative approach adaptable to complex clinical situations. Third, we constructed and validated nomograms to visualize and quantify the risk of SBI for each patient, providing a practical tool for clinical application.

Limitations

Our study has several limitations. The retrospective nature of the study may introduce bias in the analysis of the results. Additionally, the predictive model was generated using 1 data set and tested on another, which could impact the generalizability of the findings. While we believe that our model has undergone validation, it is crucial to apply it in other health institutions to assess its generalizability and test its applicability in clinical practice. Second, due to the retrospective design of the study, data were solely collected from past medical records. Recent laboratory tests that could be linked to SBIs, such as lactate levels, were not included as they were not tested.

Conclusions

Early prediction of SBIs in adults is a crucial clinical requirement. Our research has the potential to assist clinicians in identifying SBIs in adults soon after onset, facilitating early clinical monitoring and treatment while avoiding unnecessary antibiotic use. In the future, additional prospective and multicenter studies are essential to further confirm the clinical utility of these models.