A Web-Based Calculator to Predict Early Death Among Patients With Bone Metastasis Using Machine Learning Techniques: Development and Validation Study

Introduction

Accurately estimating the survival outcome of patients with bone metastasis is crucial for guiding appropriate therapeutic interventions [1-3]. Current therapeutic strategies for bone metastasis primarily involve radiation, chemotherapy, and surgery, often in combination [1]. However, it is generally advised that patients with a life expectancy of <3 months should not undergo extensive invasive surgeries, as the potential risks may outweigh the benefits [3]. Conversely, patients with a relatively longer life expectancy may benefit from surgical interventions for bone fracture or spinal instability, rather than relying solely on radiotherapy or best supportive care [4]. It is worth noting that the duration of radiotherapy in patients with bone metastasis is also dependent on survival prediction. Patients with a more favorable life expectancy may undergo a longer course of radiotherapy because shorter courses have been associated with higher rates of in-field recurrence [5].

In recent years, advancements in therapeutic modalities have led to an increasing number of surgical interventions for bone metastasis [6,7], such as osteosynthesis for extremity bone metastasis and excisional surgery for spinal metastasis. These procedures aim to maintain or improve the functional outcomes of patients and enhance their quality of life. However, striking a balance between the benefits and potential harms of surgery remains a challenge. Survival prediction plays a crucial role in guiding surgical strategies [3]. For patients with a relatively short life expectancy, simpler fixation techniques for extremity bone metastasis or posterior decompression and fixation for spinal metastasis are recommended because these approaches are associated with fewer complications but an increased risk of implant failure if patients have a better survival prognosis. In contrast, excisional surgery for spinal metastasis and prosthetic replacement for bone metastasis in the extremities are preferred for patients with a longer life expectancy [1,3].

Previous studies on prognostic factors for patients with bone metastasis have been limited by bias in terms of patient’s selection [8], data sample, and modeling methodology, which impacted their accuracy and generalization. However, the emergence of machine learning techniques has provided new opportunities to improve prognostic models in various aspects of cancer, including early diagnosis, treatment, and understanding biological processes [9-11]. Machine learning involves the application of algorithms that can explore nonlinear associations between variables and outcomes, allowing for the calculation of risk probabilities in different data sets. Machine learning–based models have demonstrated superior accuracy compared with standard eligibility criteria [11] and other nonmachine learning strategies [12,13].

Therefore, this study aimed to develop and validate a more accurate machine learning–based prediction model to assess the risk of early death among patients with bone metastasis. This study hypothesized that by identifying risk factors significantly associated with early death and using machine learning algorithms, an optimal prediction model could be developed. This model would enhance the accuracy and generalizability of prognostic assessments for patients with bone metastasis, ultimately improving clinical decision-making and patient outcomes.

Methods

Inclusion and Exclusion Criteria

This study analyzed 186,069 patients with bone metastasis between 2010 and 2019 from the Surveillance, Epidemiology, and End Results (SEER) database. The SEER database is a reliable and authoritative source of cancer statistics, covering approximately 28% of the population in the United States. It is supported by the Surveillance Research Program in the National Cancer Institute and encompasses data from various locations and sources across the country.

In this study, patients with bone metastasis were extracted from the database (2010-2019) using the SEER*Stat software (version 8.4.0.1; National Institutes of Health National Cancer Institute). The inclusion criteria were patients with bone metastasis, whereas the exclusion criteria were applied to remove patients who were aged ≤18 years, had unknown survival time, died owing to causes other than cancer, had unknown or missing cause of death, had missing data, or were alive with a follow-up time of only ≤3 months. We excluded patients with a follow-up period of <3 months from the study, and this decision was made because the primary outcome of the study was early death, and we could not ascertain whether a patient would die within 3 months after their diagnosis of bone metastases based on such a short follow-up period.

Figure 1 illustrates the flowchart of patients and the study design used in this research, which provided a visual representation of how patients were selected and included in the study as well as the overall design and methodology used. This study included 118,227 patients with bone metastasis based on the inclusion and exclusion criteria. We included a diverse range of patients with bone metastasis, encompassing both spinal and extremity cases, and we also considered various treatment modalities including surgery, radiotherapy, chemotherapy, and no therapy. This comprehensive inclusion of patients with different characteristics and treatment histories enables the prediction model to be applicable to a broader population rather than being limited to a specific subset of patients.

‎

The patients were randomly split 9:1 into a training group (n=106,492) and a validation group (n=11,735). Patients in the training group were used to train and optimize the prediction models, and patients in the validation group were used to assess and validate the prediction performance of the models. According to the inclusion and exclusion criteria, patients (n=332) from the Fifth Medical Center of Chinese PLA General Hospital were included in the external validation group, and external validation was also performed in this set.

Ethical Considerations

This study was approved by the Ethics Committee of the Fifth Medical Center of Chinese PLA General Hospital (KY-2019-12-59). Informed written consent was obtained from all patients, and strict measures were implemented to ensure the anonymity and confidentiality of the data during the analysis process. The study adhered to the principles outlined in the Declaration of Helsinki, which provides ethical guidelines for medical research involving human participants. Our findings were reported in compliance with the Standards for Developing and Reporting Machine Learning Predictive Models in Biomedical Research [14].

Potential Risk Variables

A total of 14 potential risk variables were included in the analysis, including patient’s demographics, tumor conditions, metastatic conditions, and clinical therapeutic interventions. The demographics included age, sex (female vs male), race (Black vs White vs others vs unknown), marital status (divorced vs married [including common law] vs separated vs single [never married] vs widowed vs unmarried or domestic partner vs unknown), and rural-urban continuum (metropolitan counties vs nonmetropolitan counties vs unknown); tumor conditions included primary site (slow vs moderate vs rapid growth), tumor stage (T stage; T0 vs T1 vs T2 vs T3 vs T4 vs TX [unknown T stage]), and node stage (N stage; N0 vs N1 vs N2 vs N3 vs NX [unknown N stage]); and metastatic conditions included brain metastasis (no vs unknown vs yes), liver metastasis (no vs unknown vs yes), and lung metastasis (no vs unknown vs yes). The clinical therapeutic interventions included cancer-directed surgery (no vs unknown vs yes), radiation (none or unknown vs yes), and chemotherapy (none or unknown vs yes).

All patients entered our database after their diagnosis of bone metastases owing to cancer. Early death was defined as patients who died at or within 3 months [15,16] after the diagnosis of bone metastases, and the death was attributed to causes related to their cancer. T and N stages were recorded based on the combination of the American Joint Committee on Cancer and Extent of Disease classification. The primary site of cancer was divided according to a previous study [8]. Table S1 in Multimedia Appendix 1 provides the detailed information regarding the classification of primary cancers.

Model Development and Validation

This study used multiple logistic regression analysis of clinical characteristics for predicting early death, supplemented by subgroup analysis of clinical characteristics stratified by early death, to conduct variable selection. Significant clinical characteristics in the multivariate analysis served as input features to train and optimize the models. As for machine learning approaches, in the training group, 6 algorithms including logistic regression, extreme gradient boosting (XGBoosting) machine, decision tree, random forest, neural network, and gradient boosting machine were used to train and optimize the models [17-21]. The introduction of the 6 machine learning algorithms is summarized in Table S2 in Multimedia Appendix 1. To ensure robustness and reproducibility, all the models were provided with identical input features. To optimize the performance of each model, an extensive exploration of the hyperparameter space was conducted through both grid and random searches. The area under the curve (AUC), a widely accepted metric for classification tasks, was used as the objective function for hyperparameter optimization. To strike a balance between model complexity and generalization ability, careful consideration was given to prevent both overfitting and underfitting. As such, the search space for the hyperparameters was deliberately defined with generous upper and lower bounds. For instance, the decision tree depth was constrained within the range of 2 to 100, accounting for a wide spectrum of potential tree structures. Optimal model parameters were obtained after a grid search or random hyperparameter search using 5-fold cross-validation after 100 iterations of bootstraps.

The prediction performance of the models was assessed using 11 measures in the validation group: Brier score, intercept-in-large, calibration slope, AUC, discrimination slope, specificity, negative predictive value, precision, recall, Youden index, and accuracy. In addition, the clinical usefulness of the models was evaluated using decision curve analysis after calculating the net benefits in a range of threshold probability. The metrics were scored by sorting them based on their prediction performance, as per the findings of a previous study [16]. On the basis of each evaluation metric, the highest score was assigned to the best-performing model, followed by the next best model. The overall score of the model was the sum of the scores of each metric. The total score of the scale was 66 points, with a score of >52 indicating excellent prediction performance. The heat map was conducted to show the data using pheatmap package in R language software (R Foundation for Statistical Computing). External validation was performed using the optimal machine learning model.

Model Explainability and Variable Importance

The Local Interpretable Model-Agnostic Explanations (LIME) technique was used to enhance the explainability of the best model, thereby promoting its clinical utility and transparency [22]. LIME allowed the calculation of the risk probability of early death and facilitated a deeper understanding of how the predicted probability changes with different observations by assigning individual weights to each variable. This approach significantly increased users’ trust in the prediction model through reasonable model explanations and enhanced model transparency.

In addition, the importance of predictors was assessed using the SHAP (Shapley Additive Explanations) method [19]. This evaluation enabled the ranking of variables based on their contributions to the outcome, providing insights into which factors had the most significant impact on the prediction. By using SHAP, a comprehensive understanding of the relative importance of different variables in determining the risk of early death among patients with bone metastasis was obtained.

Model Risk Stratification

To achieve risk stratification, this study used the average threshold derived from the 6 approaches as the optimal threshold. On the basis of this threshold value, patients were classified into 2 distinct groups: a low-risk group and a high-risk group, as defined by each approach. Specifically, patients with a predicted probability equal to or less than the threshold were assigned to the low-risk group, whereas patients with a predicted probability greater than the threshold were assigned to the high-risk group. This approach allowed for the differentiation of patients based on their predicted risk of early death, facilitating targeted interventions and personalized management strategies.

Development of the Web-Based Calculator

Using the algorithm of the optimal model from this study, we developed an web-based calculator to predict the probability of early death in patients with bone metastasis. This calculator is accessible through the Streamlit application and has been designed with 4 main sections. The first section of the calculator allows users to input the relevant parameters and select the model variables. This interactive panel enables users to customize the prediction based on specific patient characteristics and treatment options. The second section of the calculator displays the predicted probability of early mortality. Once the user inputs the necessary parameters, the calculator generates an estimate of the likelihood of early death for the given patient. This information can provide valuable insights into clinical decision-making. The third section of the calculator provides detailed information about the model itself. This section offers an overview of the algorithm used, the data set used, and the validation process. It aims to enhance transparency and provide users with a clear understanding of how the predictions are generated. The fourth section of the calculator offers recommendations for therapeutic strategies based on the predicted risk of early death. This section provides guidance on potential treatment options, taking into account the patient’s risk level and the goal of palliative pain relief.

Statistical Analysis

In this study, quantitative data were summarized as mean and SD, and qualitative data were presented as proportions. The difference comparison was performed using 2-tialed t test for the quantitative variables and using adjusted continuity chi-square test for the qualitative variables. Multivariate logistic regression analysis was conducted to identify the significant variables associated with early death. Kaplan-Meier survival curves were plotted according to risk stratification among all approaches using “survival” and “survminer” package. Survival difference between the low-risk and high-risk groups were compared using the log-rank test. The association between age and early death was investigated using an automatic machine learning method. This association was visualized according to the deciles of all patients. Machine learning and model explainability were conducted using Python (version 3.9.7; Python Software Foundation); visualization and statistical analysis were both performed using R programming language (version 4.1.2). All P values were 2-tailed, and a P value of <.05 was considered statistically significant.

Results

Patient’s Basic Demographics and Clinical Characteristics

A total of 186,069 patients were included in the analysis, with a mean age of 67.10 (SD 12.39) years. Most patients were male (68,296/118,227, 57.77%), White (94,061/118,227, 79.56%), and married (60,812/118,227, 51.44%). Rapid-growth primary cancer accounted for 62.49% (73,881/118,227) of patients, followed by slow growth (32,309/118,227, 27.33%) and moderate growth (12,037/118,227, 10.18%). A significant proportion of the patients were in the advanced stages, with 41.57% (49,167/118,227) in the T3 and T4 stages and 35.99% (42,553/118,227) in the N2 and N3 stages. In the entire cohort, early death occurred in 36.63% (43,305/118,227) of patients. Multimedia Appendix 2 illustrates the incidence of early death over the course of the study period, which revealed a consistent and stable pattern of occurrence per year. The therapeutic interventions and organ metastases are summarized in Table 1. It also showed the distribution of clinical characteristics between the training and validation groups, indicating that the 2 groups were comparable.

Model Development

In the training group, a comparison of variables between patients with and without early death was performed (Table 2). According to multivariate analysis, age, sex, race, marital status, rural-urban continuum, primary site, T stage, N stage, brain metastasis, liver metastasis, lung metastasis, cancer-directed surgery, radiation, and chemotherapy were all significantly associated with early death (Table 3) and were included as input features for model training and optimization. The full parameter weights of the machine learning algorithms are summarized in Table S3 in Multimedia Appendix 1.

Model Validation

The AUC of the gradient boosting machine and XGBoosting machine was 0.858 (95% CI 0.851-0.865; Multimedia Appendix 3), followed by random forest and neural network (both 0.856, 95% CI 0.849-0.863). The calibration slopes of the logistic regression, XGBoosting machine, decision tree, random forest, neural network, and gradient boosting machine were 1.05, 1.03, 0.97, 1.13, 0.97, and 1.02, respectively (Multimedia Appendix 4).

Figure 2 shows the probability curves generated for each algorithm. Notably, the gradient boosting machine, neural network, and XGBoosting machine demonstrated significant separation between positive and negative events, with minimal overlap in the probability curves. These models exhibited the greatest distinction between patients with and without early death, indicating their superior discriminatory ability. To further quantify this discrimination, Multimedia Appendix 5 shows the discrimination slopes for each algorithm. The gradient boosting machine, neural network, and XGBoosting machine exhibited the 3 largest discrimination slopes, further supporting their superior performance in distinguishing between patients with different outcomes. Furthermore, a decision curve analysis (Multimedia Appendix 6) was conducted to evaluate the clinical usefulness of the algorithms. All the algorithms, particularly the gradient boosting machine, neural network, and XGBoosting machine, demonstrated favorable clinical utility. This analysis underscores the practical value of these models in guiding clinical decision-making.

‎

All evaluation metrics are summarized in Table 4. Figure 3 shows a heat map for visualizing data on prediction performance among all models, and it demonstrated that the gradient boosting machine had the highest score of prediction performance (54 points), followed by the neural network (52 points) and XGBoosting machine (50 points). These results suggest that the gradient boosting machine approach performed best in developing a prediction model to estimate the risk of early death among patients with bone metastasis.

‎

The baseline clinical characteristics of the patients in the external validation set are summarized in Table S4 in Multimedia Appendix 1. External validation was performed using the optimal machine learning model, and the AUC was 0.847 (95% CI 0.798-0.895; Multimedia Appendix 7). Although the blue line of the calibration curve in the external validation set (Multimedia Appendix 8) deviated slightly upward from the diagonal around 0.30 in the predicted probability, it still remained close to the diagonal and showed a tendency to regress toward it. In terms of quantitative assessment, the calibration slope was 1.06 and the intercept-in-large value was 0.17, which suggests favorable calibration in the external validation.

Model Explainability and Feature Importance

Model explainability was conducted to rank the variables and visualize their contributions to early death based on the gradient boosting machine algorithm. Figure 4 shows 4 individual cases in the internal validation set using LIME. Multimedia Appendix 9 shows the machine learning curve of the H₂O method, and it depicted that error metric dependence on learning progress. The SHAP summary plot revealed that chemotherapy, primary site, and liver metastasis were the top 3 important variables associated with early death (Multimedia Appendix 10). Multimedia Appendix 11 also demonstrates the variable importance heat map, which depicted variable importance across multiple models. It demonstrated that chemotherapy, primary site, age, and liver metastasis were significantly important variables. Multimedia Appendix 12 investigates the relationship between early death and age, and it showed that early death gradually increased with age.

‎

Model Risk Stratification

This study used an average threshold of 37% derived from all 6 machine learning algorithms as the optimal threshold for risk stratification. On the basis of this threshold, patients were categorized into 2 distinct groups within each machine learning approach: a low-risk group and a high-risk group (Table 5). In the low-risk group, patients were identified as having a predicted risk probability ≤37%. In contrast, patients in the high-risk group had a predicted risk probability exceeding 37%. The actual probability of early death was compared between the 2 groups for each algorithm (all P<.001). On the basis of the risk stratification, Kaplan-Meier survival curves were plotted for each machine learning algorithm (Multimedia Appendix 13; all P<.001).

Web-Based Calculator

As the gradient boosting machine was the optimal model in this study, this study further deployed an web-based calculator to predict early death among patients with bone metastasis using this model (Multimedia Appendix 14). This web-based application is available [23]. The online application could enhance the accessibility and usability of the prediction model, and it allowed health care professionals to easily input patient data and obtain valuable predictions for early mortality risk among patients with bone metastases.

Discussion

Principal Findings

This study developed an accurate model to predict the risk of early death among patients with bone metastasis. On systematically evaluating the prediction performance of all algorithms in this study, the model developed by the gradient boosting machine scored the highest and performed the best, followed by the neural network and XGBoosting machine in predicting early death among patients with bone metastases. Furthermore, the web-based calculator developed in this study offers a valuable tool for clinicians to make more informed clinical decisions. This information can guide treatment planning, allowing clinicians to tailor interventions based on individual patient risk profiles.

Previous studies have proposed several scoring systems to predict the survival prognosis of patients with bone metastases. For instance, Katagiri et al [8] updated a scoring system for patients with skeletal metastases by introducing a new factor and analyzing 808 patients with symptomatic skeletal metastases. Among these patients, 749 were treated nonsurgically and 59 underwent surgery for skeletal metastasis. However, the accuracy and AUC value of the model were not presented in this scoring system. Similarly, Sawada et al [24] evaluated factors associated with 30-day mortality after surgery for spinal metastasis and developed a risk scoring system based on the analysis of 3524 patients. The clinical predictors of the scoring system included sex, emergency admission, rapid-growth tumors, and nonskeletal metastasis. Other scoring systems have been designed specifically for patients with either spinal metastasis [3] or specific primary cancer [25-27]. However, these scoring systems may be subject to patient selection bias, making them less applicable to the general population of patients with bone metastases or those diagnosed with other primary cancer types. Although scoring systems tailored to specific cancers may offer greater precision, clinicians may prefer a simple and general system that can be applied across different primary sites to avoid the inconvenience associated with multiple systems [8]. In addition, some scoring systems have been exclusively designed for patients treated with surgery [3,27] or radiotherapy [28] alone. Moreover, scoring systems developed for various primary cancer types tended to focus mainly on the common primary cancers, such as lung, breast, and prostate cancer, which account for approximately 50% of all primary cancer types [29]. Consequently, other malignancies are often categorized as “others” or receive little attention in these scoring systems. Notably, these studies relied on relatively small sample sizes, ranging from tens to hundreds of patients [2,8]. Hence, the accuracy and reliability of these scoring systems may be limited owing to the restricted size of the data samples.

In light of these limitations, the development of a more accurate and comprehensive machine learning–based prediction model, as demonstrated in this study, holds significant clinical value. By using a large cohort and machine learning algorithms, this model provides a more robust and accurate assessment of the likelihood of early death among patients with bone metastasis. It offers a standardized approach that can be applied across various primary cancer types, enhancing clinical utility and facilitating personalized treatment decision-making for a broader patient population. In the entire cohort, 36.63% (43,305/118,227) of patients experienced early death. Other studies reported that the 3-month mortality rate was approximately 30% [30-32], which was consistent with this study. According to variables’ importance, this study found that chemotherapy, primary site, and liver metastasis were the top 3 important variables associated with early death. Primary tumor type and visceral metastasis have already been widely proven to be prognostic factors associated with survival outcomes in patients with bone metastasis [33]. Chemotherapy is also a significant variable related to survival [33]. Although previous studies have reported inconsistent findings regarding the association between age and survival prognosis in patients with bone metastasis [8], this study observed a gradual increase in the risk of early death with advancing age. These contrasting results could potentially be attributed to differences in age classification and the sample sizes among the various studies. In light of the findings of this study, which suggested an association between age and early death, it is important to consider age as a potential prognostic factor in clinical decision-making among patients with bone metastases.

Regarding modeling methodology, this study used 6 machine learning algorithms including logistic regression, XGBoosting machine, decision tree, random forest, neural network, and gradient boosting machine to train and optimize models [34]. These sophisticated machine learning algorithms have the capability to leverage large amounts of data and provide improved performance compared with traditional logistic regression models [13]. Machine learning approaches offer flexibility and scalability, making them suitable for various tasks such as risk stratification and survival estimation, particularly when analyzing big data [35]. Previously, machine learning has been used to develop prediction models to predict the mortality among bone metastases from specific cancer types, such as hepatocellular carcinoma [15], lung cancer [16], cancer of unknown primary site [36], and breast cancer [37]. However, our study was specifically designed to develop a machine learning model to predict the risk of early death among patients with general bone metastases. Finally, model explainability was achieved based on the gradient boosting machine algorithm in this study to improve model transparency and user trust because it could not only show the risk probability of early death among individual patients but also provide reasonable explanations behind it [38]. The model aimed to assess the probability of early death among patients with bone metastasis, and it could be used at different time points depending on the clinical scenario and the patient’s treatment history. For instance, it can be used to assess the probability of early death in patients newly diagnosed with bone metastasis, aiding in treatment decision-making. Similarly, for patients who have undergone medical oncology therapies or are considering surgery, the model can offer additional information regarding the likelihood of early death, thus assisting in the selection of the most appropriate treatment approach.

Limitations

This study had some limitations. First, the number of some primary cancers, such as penis and eye and orbit, was very small, possibly owing to the low incidence rate of bone metastasis among those cancers. Thus, it might be difficult to draw a conclusion to guide clinical decision among patients with those primary cancers. Second, only a small fraction of patients (11,532/118,227, 9.75%) underwent cancer-directed surgery, possibly owing to the increasing use of antiosteolytic drugs, such as bisphosphonates and denosumab, which largely reduced skeletal-related events. Thus, validation of the model among a sufficient number of surgically treated patients was necessary. Third, some laboratory factors [39] such as albumin, hemoglobin, and lymphocytes or the Eastern Cooperative Oncology Group score [40] were not considered for analysis in the study because of the unavailability of these data in the SEER database. Therefore, although our model obtained favorable prediction in the internal and external validation, clinical decision should not rely on survival estimation alone; it also needs a complete evaluation of the patient’s pain, neurological impairment, and general performance status.

Conclusions

This study developed a machine learning–based prediction model to accurately assess the probability of early death among patients with bone metastasis. The gradient boosting machine demonstrated the highest prediction performance among the 6 approaches evaluated in this study. Age, sex, race, marital status, rural-urban continuum, primary site, T stage, N stage, brain metastasis, liver metastasis, lung metastasis, cancer-directed surgery, radiation, and chemotherapy were identified as significant factors associated with early death. The model’s explainability, using LIME, provides insights into the contributions of these variables to early death. External validation of the model demonstrated its robustness and reliability. This prediction model, which is presented in the format of an online application, has the potential to guide clinical decision-making and improve the care of patients with bone metastasis by identifying those at a higher risk of early death. However, further research and clinical judgment are necessary to determine the appropriate treatment options for individual patients.