Machine Learning–Based Prediction of Acute Kidney Injury Following Pediatric Cardiac Surgery: Model Development and Validation Study

Introduction

An increasing number of pediatric patients worldwide undergo cardiac surgery each year for various reasons, including congenital heart disease and acquired cardiac conditions [1]. Cardiac surgery–associated acute kidney injury (CSA-AKI), characterized by an abrupt decrease in renal function, is a major complication following pediatric cardiac surgery. The reported incidence of CSA-AKI among pediatric patients undergoing cardiac surgery ranges from 5% to 42% [2]. Importantly, CSA-AKI is associated with significantly increased morbidity and mortality, prolonged length of hospital stay, and an increased risk of chronic kidney disease [3-5].

The early prediction of CSA-AKI could significantly improve the implementation of preventive and therapeutic strategies during the perioperative periods. Specifically, preoperative prediction could facilitate surgery risk assessment and prevention of CSA-AKI, and early postoperative prediction could help with the early identification of CSA-AKI for proactive interventions [6]. Therefore, it is of great clinical interest to establish precise prediction models for CSA-AKI to identify high-risk patients and to optimize the perioperative management of pediatric patients undergoing cardiac surgery. Recently, several prediction models combining biomarkers and clinical variables have been established with the goal to predict the development of CSA-AKI in the pediatric population [7-9]. However, those models were limited by small sample sizes, lack of internal and external validation, and additional financial burdens due to the use of novel biomarkers.

The widespread use of machine learning to analyze clinical data derived from electronic health records offers considerable advantages for establishing prediction models. Machine learning is a scientific discipline that uses computer algorithms and learns from data with minimal human intervention [10]. Advanced machine learning algorithms can model nonlinear relationships, analyze complex high-order interactions, and robustly handle multicollinearity among the predictor variables. The application of machine learning has led to significant breakthroughs in various medical fields such as emergency department triage [11], prediction of postinduction hypotension [12], and risk stratification of postcontrast acute kidney injury [13]. Machine learning approaches have also shown promising performance in the early prediction of adult CSA-AKI [14-19], but their predictive performance for CSA-AKI in the pediatric population has not been tested. The primary diseases, underlying pathophysiology, and risk factors of CSA-AKI in pediatric patients are significantly different from those in adult patients [20-22]. Therefore, the existing prediction models for adult CSA-AKI are not applicable to pediatric patients. The objective of this study was to develop and validate machine learning models for predicting the development of CSA-AKI in pediatric patients undergoing cardiac surgery.

Methods

Study Design

This study includes patients from 3 distinct medical centers of Central South University in China. The derivation cohort comprised patients admitted at the Second Xiangya Hospital between January 2015 and March 2022. The external validation cohort consisted of patients admitted at Xiangya Hospital between January 2016 and December 2021 and patients admitted at the Third Xiangya Hospital between January 2015 and December 2021.

Ethics Approval

This study follows the Declaration of Helsinki and the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis statement [23]. This study protocol was approved by the medical ethics committee of the Second Xiangya Hospital of Central South University (2022-K031). The requirement for informed consent was waived due to the retrospective nature and minimal risk of this study.

Study Participants

This study includes all pediatric patients aged between 1 month and 18 years who underwent cardiac surgery with cardiopulmonary bypass. We included patients with at least one serum creatinine (SCr) measurement before surgery and another within 7 days after surgery. We excluded patients with congenital renal malformation, preoperative estimated glomerular filtration rate (eGFR) of 15 mL/min/1.73 m² or lower, or multiple surgeries within 7 days. We calculated eGFR using the modified Schwartz equation, or if body length was missing, using the Full Age Spectrum equation [24,25]. Details on patient selection in the derivation and external validation cohorts are provided in Figure 1.

‎

Predictor Variables

Potential predictors included those considered clinically relevant to the development of CSA-AKI and available in the electronic health records, with less than 30% of such observations missing. Potential predictors were divided into preoperative and intraoperative variables. Preoperative variables included patient demographics, preoperative conditions, laboratory tests, and medications. Preoperative conditions were determined according to the diagnosis on admission, preoperative diagnosis, and preoperative anesthesia interview records. The most recent preoperative measurements were used for laboratory variables. Medications were classified based on the Anatomic Therapeutic Chemical classification system and included if administered within 7 days before the surgery [26]. Intraoperative data were extracted from the records of anesthesia, cardiopulmonary bypass, and surgery. The complexity of the surgeries was determined by the Risk Adjustment for Congenital Heart Surgery-1 (RACHS-1) score, which is a consensus-based tool for short-term mortality risk based on the type of the procedure performed [27]. The volume of blood loss was calculated as the sum of blood loss in operative fields, residual volume in the pump, and any additional loss by other routes. The total fluid balance was calculated as the difference between the total fluid input and total fluid output and corrected for body weight.

Outcome Measures

The primary outcome was the development of CSA-AKI, which was determined according to the 2012 Kidney Disease: Improving Global Outcomes (KDIGO) clinical practice guideline [28]. CSA-AKI was defined as an increase in SCr level of at least 0.3 mg/dL within 48 hours or 50% within 7 days after surgery compared with the baseline. The most recent SCr value within 90 days prior to the surgery was used as the baseline. Secondary outcomes included CSA-AKI stages 2-3 (defined by the KDIGO criteria), in-hospital mortality, length of stay in the intensive care unit, and length of postoperative hospital stay.

Statistical Analysis

Descriptive statistics are presented as medians and interquartile ranges for continuous variables and as numbers and percentages for categorical variables. Data distributions were compared using the Mann-Whitney U test for continuous variables and chi-square tests for categorical variables. One-hot encoding was performed by preprocessing each categorical variable into binary variables. Missing data were imputed by the random forest method using the missForest package in R [29]. The description of the data types and the missing values for each variable can be found in Table S1 of Multimedia Appendix 1.

Feature Selection

Predictor variables with near-zero variance, identified as those with the percentages of unique values less than 5%, were removed from the analysis. Subsequently, 4 feature selection methods were used to obtain subsets of the predictor variables for further model development. The methods included Least Absolute Shrinkage and Selection Operator, Boruta algorithm, random forest-recursive feature elimination, and random forest-filtering. The results obtained by the 4 methods were comprehensively evaluated, and the predictor variables that appeared more than 3 times among the 4 methods were ultimately selected to build the model. Feature selection was conducted twice—first to include only the preoperative variables and then by combining the preoperative and intraoperative variables. The glmnet, Boruta, and caret packages in R were used for feature selection.

Model Development and Validation

For model development, the following machine learning algorithms were applied to the preoperative-only and the combined data sets of the derivation cohort: K-nearest neighbor, naive Bayes, support vector machines, random forest, extreme gradient boosting (XGBoost), and neural networks. We conducted 5 random shuffles of 5-fold cross-validation to ensure an unbiased assessment of model performance and to identify the optimal hyperparameters for each model. Model performance was assessed based on the mean area under the receiver operating characteristic curves (AUROCs) from 5×5 iterations. After that, the best performing machine learning model was chosen for each data set. The caret package in R was used for model development. The details on functions, packages, and tuning parameters used for each machine learning algorithm are provided in Table S2 of Multimedia Appendix 2.

The performance of the final prediction models was further evaluated in the external validation cohort. The metrics for model performance included AUROC, the area under the precision-recall curve (AUPRC), and the calibration plot. AUROC was used as the primary performance metric because it is independent of the thresholds in the setting of class imbalance. AUPRC is known to be more informative for class-imbalanced prediction tasks because it is sensitive to changes in the number of false-positive predictions [30]. In addition, the performance of the final machine learning models was compared with that of the traditional logistic regression models. The framework of model establishment is depicted in Figure S1 of Multimedia Appendix 3.

Model Interpretations

The Shapley additive explanations (SHAP) method was used to explore the interpretability of the final prediction models. SHAP is a unified approach to the interpretations of model predictions, which provides consistent and locally accurate attribution values for each feature within a prediction model, namely, the SHAP values [31]. Higher SHAP values indicate an increased probability of CSA-AKI. The contribution of the predictor variables to CSA-AKI can be explained as the cumulative effects of variable attributions to the entire output risk for each observation.

Sensitivity Analysis

Sensitivity analysis was performed to examine the predictive power of the models for CSA-AKI stages 2-3. Model performance was also evaluated in the subgroups, focusing on patients in different age groups (infancy: 1 month to 1 year; childhood: 2-10 years; adolescence: 11-18 years) [32] and patients with different surgical complexities (RACHS-1 score of 2 or lower; RACHS-1 score of 3 or higher). In addition, to translate the prediction models into clinical risk-stratification tools, we identified the low- and high-risk cutoff values of the predictive probabilities. The analyses included sensitivity, specificity, positive and negative predictive values, and positive and negative likelihood ratios. Finally, to evaluate the effect of the imbalanced outcomes on model performance, we applied upsampling or downsampling to generate a balanced derivation cohort for model training. All statistical analyses were performed using R 4.1.2 [33]. A 2-tailed P value of <.05 was considered statistically significant.

Results

Patient Characteristics

A total of 3863 participants were enrolled in this study, that is, 3278 in the derivation cohort and 585 in the external validation cohort. The baseline characteristics and outcomes of the patients in the derivation and external validation cohorts are shown in Table 1. In the derivation cohort, 564 (17.2%) patients developed CSA-AKI, comprising 356 (10.9%) with stage 1, 142 (4.3%) with stage 2, and 66 (2%) with stage 3. In the external validation cohort, 51 (8.7%) patients developed CSA-AKI, comprising 25 (4.3%) with stage 1, 14 (2.4%) with stage 2, and 12 (2.1%) with stage 3. The comparison of the baseline characteristics and outcomes according to the development of CSA-AKI are provided in Table S3 and Table S4 of Multimedia Appendix 4.

Predictor Variables

A total of 25 preoperative variables were selected as predictors of CSA-AKI by the 4 feature selection methods and included in the machine learning models (Table S5 of Multimedia Appendix 5). When preoperative and intraoperative variables were combined, a total of 27 variables were incorporated; of those, 20 were preoperative variables and 7 were intraoperative variables (Table S6 of Multimedia Appendix 5).

Model Performance

Among the considered machine learning models, the XGBoost model achieved the best performance on both the preoperative-only and the combined data sets, with a mean AUROC of 0.795 and 0.832 in cross-validation, respectively (Table S7 of Multimedia Appendix 6). The inclusion of the intraoperative variables improved the predictive power of all considered machine learning models. The AUROCs of the XGBoost and traditional logistic regression models are shown in Figure 2. The XGBoost model with only the preoperative variables exhibited an AUROC of 0.890 (95% CI 0.876-0.906) in the derivation cohort and 0.857 (95% CI 0.800-0.903) in the external validation cohort. When the intraoperative variables were included, the AUROC of the XGBoost model increased to 0.912 (95% CI 0.899-0.924) in the derivation cohort and 0.889 (95% CI 0.844-0.920) in the external validation cohort. The XGBoost algorithm achieved higher AUROCs than traditional logistic regression for both the derivation and external validation cohorts. Details on other performance metrics, including the AUPRCs and the calibration plots, are provided in Figures S2 and S3 of Multimedia Appendix 7. Due to the imbalance in the proportions of patients with and without CSA-AKI, the AUPRC of the XGBoost model using the combined data set was 0.747 in the derivation cohort and 0.476 in the external validation cohort. The Brier scores of the XGBoost model for the combined data set were 0.085 and 0.060 in the derivation and the external validation cohorts, respectively. The final XGBoost models can be accessed at LuoXiaoqin123/pediatric-CSA-AKI (GitHub) [34].

‎

Model Interpretations

The SHAP summary plots of the XGBoost models are shown in Figure 3, which illustrates how high and low values of each feature relate to SHAP values. The plots also identify the features that influenced the model predictions the most. Baseline SCr, perfusion time, body length, operation time, and intraoperative blood loss were the top 5 predictor variables associated with CSA-AKI in the XGBoost model using the combined data set. The SHAP dependence plots of the XGBoost model with the combined data set are shown in Figures S4 of Multimedia Appendix 8, which shows the nonlinear association between the predictors and the risk of CSA-AKI. The plots show how changes in a single feature can affect model output.

‎

Sensitivity Analysis

The XGBoost models showed good predictive performance for CSA-AKI stages 2-3, with AUROCs higher than 0.85 in both the derivation and the external validation cohorts, respectively (Figure S5 of Multimedia Appendix 9). When the models were applied to patients in different age groups or patients with different surgical complexities, the performance of the models remained stable (Figures S6-S10 of Multimedia Appendix 10). The low- and high-risk cutoff values were identified to facilitate clinical applications of the XGBoost models. The diagnostic test characteristics of the models at the low- and high-risk cutoff points are shown in Table 2. For example, the XGBoost model applied to the combined data set yields the following clinically relevant threshold values. The low-risk cutoff value is 0.099 with sensitivity of 95% and a negative predictive value of 98.3%; this value captures the vast majority of patients with CSA-AKI and leaves out only a small proportion of those that are falsely negative. The high-risk cutoff value of 0.374 has specificity of 95% and a positive likelihood ratio of 12.12. Therefore, the low-risk cutoff is appropriate for low-level interventions, while the high-risk cutoff value can be used to identify patients at high risk of CSA-AKI who require more intensive interventions. Additionally, when we used the balanced derivation cohort by upsampling or downsampling to train the models, the models still achieved promising predictive performance (Figures S11 and S12 of Multimedia Appendix 11).

Discussion

Principal Findings

In this multicenter retrospective study, we developed and externally validated prediction models for pediatric CSA-AKI by using machine learning approaches. Multiple machine learning algorithms were tested in the process of model development, with the XGBoost algorithm ultimately identified as offering the strongest discrimination. In addition, the XGBoost models showed promising predictive performance on both the preoperative-only and combined data sets, demonstrating their potential usefulness for predicting pediatric CSA-AKI. To the best of our knowledge, our study is the first to establish machine learning models for CSA-AKI in the pediatric population that are valuable for risk stratification and clinical decision-making.

Previous studies have shown the advantages of machine learning algorithms in predicting CSA-AKI in adults [14-19]. Lee et al [14] were the first to apply machine learning approaches to predict all stages of CSA-AKI in adults. Their study showed that XGBoost performed better in predicting CSA-AKI than either the traditional logistic regression or risk scores. Additional prediction models based on machine learning algorithms were developed to predict CSA-AKI in the Chinese adult population [18]. Another study established the machine learning models that incorporated the intraoperative time-series and other features to predict adult CSA-AKI. In that study, the ensemble model (random forest + XGBoost) showed the best predictive performance [19]. However, the primary disease, comorbid conditions, and renal physiology in the pediatric population differ significantly from those in adults, which makes the adult prediction models unsuitable for predicting CSA-AKI in pediatric patients [35]. To date, no prediction model for CSA-AKI with prospective applications of machine learning techniques has been established for infants and children. In this study, the models using the XGBoost algorithm had the strongest predictive power among the considered machine learning models. The XGBoost models were further validated using an external validation cohort, and they exhibited consistent predictive performance for pediatric CSA-AKI. Additionally, the sensitivity analyses showed that the XGBoost models displayed comparable predictive performance in most subsets of patients grouped by age or complexity of surgery. Overall, the results demonstrated the robustness and applicability of the XGBoost models in pediatric patients undergoing cardiac surgery.

Both preoperative and intraoperative factors proved to contribute to the prediction of postoperative AKI. Tseng et al [19] emphasized the value of intraoperative features that reflected rapid physiologic changes during surgery relevant to the prediction of CSA-AKI in adults. Another retrospective study found that, by integrating preoperative and intraoperative features, the model for postoperative AKI could reclassify 40% of the false-negative patients from the preoperative model [36]. Our study shows that the XGBoost model combining the preoperative and intraoperative variables achieved better predictive performance than that with the preoperative variables only. Nevertheless, both models exhibited adequate predictive power and potential utility for the preoperative or early postoperative prediction of pediatric CSA-AKI. The preoperative prediction could assist practitioners in evaluating the risk of surgery and support the implementation of preventive strategies. Early postoperative prediction is useful for optimizing postoperative management and care plans, such as continuous assessment of renal function, hemodynamic monitoring, avoidance of nephrotoxin, or renal replacement therapy.

The SHAP method was used to uncover the black box of the XGBoost models. This method is a model-agnostic explanation technique that has been widely used to interpret the contribution of predictors to the model output [37,38]. Consistent with an updated systemic review, we found that lower baseline SCr, longer perfusion time, longer operation time, higher baseline eGFR, younger age, and lower body weight are associated with the development of pediatric CSA-AKI [3,39]. We also identified body length, intraoperative blood loss, serum potassium, and serum calcium as important predictors of CSA-AKI. Notably, 3 of the top 5 important predictors were intraoperative variables, suggesting that the surgical procedure itself has a significant impact on the occurrence of CSA-AKI. During cardiac surgery, significant pathophysiological changes may occur, such as ischemia-reperfusion injury, inflammation response, or activation of coagulation pathways, which may lead to renal injury and dysfunction [40]. Additionally, the conventional logistic regression used in previous studies is limited to revealing the linear relationship between the predictors and CSA-AKI [41-43]. In contrast, the SHAP dependence plots in our study reflected complex nonlinear relationships, which can assist in understanding the association between the changes in predictor variables and the risk of CSA-AKI.

Our findings have significant clinical implications. First, the low- and high-risk cutoff values were identified to promote the clinical application of the XGBoost models. This should allow the care team to identify the patients at high risk of CSA-AKI and to develop optimal perioperative management strategies. Second, our models used the preoperative and intraoperative variables that are routinely collected in clinical practice, thus adding no extra laboratory tests or financial burdens to the standard clinical care procedures. Third, the discovery of modifiable predictors may promote early interventions to mitigate the risk of CSA-AKI.

Limitations

Our study has several limitations. First, data were retrospectively collected from electronic health records. Second, the study population was restricted to tertiary medical institutions, as pediatric cardiac surgery is typically not offered in primary health care institutions in China. Thus, the applicability of our prediction models needs further validation in diverse populations. Third, the urine output criteria were not used to define CSA-AKI because hourly urine output data were not available for most patients. However, given the routine use of diuretics in the intraoperative and postoperative periods to maintain urine output, few patients with CSA-AKI were missed in this study. Finally, the causality between the predictors and CSA-AKI needs further exploration. Randomized controlled trials would need to be performed to verify whether the modification of certain predictors can prevent the occurrence of CSA-AKI.

Conclusion

Our study demonstrates the applicability of machine learning approaches in predicting the development of CSA-AKI in the pediatric population. The XGBoost models had consistent and clinically applicable performance in the derivation and external validation cohorts, which indicated their robustness and expandability. Additionally, the predictive value of the preoperative and intraoperative factors was demonstrated by the improved performance of the model when these factors were combined. Ultimately, our models should prove useful in assisting practitioners with risk stratification and clinical decision-making in pediatric patients undergoing cardiac surgery.