Mortality Risk Prediction in Patients With Antimelanoma Differentiation–Associated, Gene 5 Antibody–Positive, Dermatomyositis–Associated Interstitial Lung Disease: Algorithm Development and Validation

Introduction

Idiopathic inflammatory myopathies (IIMs) encompass a heterogenous group of autoimmune diseases characterized by muscle weakness and inflammation. Dermatomyositis (DM), which predominantly affects the skin and muscles, is a prevalent subtype within the IIM spectrum. Interstitial lung disease (ILD) is a common comorbidity in DM, with an incidence varying widely from 5% to 65% [1]. The clinical presentations, treatment responses, and outcomes of IIMs are highly variable and correlate with the presence of specific myositis-specific antibodies (MSAs). One such antibody, the antimelanoma differentiation–associated gene 5 (MDA5) antibody, targets the MDA5 protein, a member of the retinoic acid inducible gene I (RIG-I) receptor family, which is crucial for the detection of viral RNA and the activation of the interferon pathway in response to viral infections [2]. Approximately 1%-30% of patients with DM test positive for anti-MDA5 antibodies [3]. A recent surge in MDA5+DM cases has been observed in England in the wake of the COVID-19 pandemic, with an associated mortality rate of 32% for antimelanoma differentiation–associated gene 5 antibody–positive dermatomyositis–associated interstitial lung disease (anti-MDA5+DM-ILD) [4]. Patients with anti-MDA5 autoantibody often exhibit minimal muscle involvement. Initially, anti-MDA5 autoantibody was identified in a cohort of East Asian patients with clinically amyopathic myositis (81%) and rapidly progressive ILD (74%) [5], with >90% of rapidly progressive interstitial lung disease (RP-ILD) cases and 84% mortality occurring within the first 6 months of disease onset [6]. Notably, the first 3 months following diagnosis are particularly critical, with 46% of the deaths occurring in this period [6]. However, recent studies have revealed that early and aggressive intervention can significantly improve the 6-month survival rate of patients with anti-MDA5+DM-ILD from 33% to 89% [7-9]. Thus, there is an urgent need to accurately identify these high-risk patients at an early stage of the disease.

In recent years, various mortality risk models incorporating independent predictors have been explored to predict outcomes in patients with anti-MDA5+DM-ILD. Lian et al [10] developed the FLAIR score, which includes 5 key indicators: ferritin levels, lactate dehydrogenase (LDH) levels, the anti-MDA5 antibody grade, the high-resolution computed tomography (HRCT) imaging score, and RP-ILD [10]. However, the FLAIR score was designed to predict mortality in patients with amyopathic DM. Some independent prognostic factors, such as Krebs von den Lungen-6 (KL-6), interleukin (IL)-6, IL-18, and soluble CD206, are challenging to access in clinical practice [11], and HRCT imaging scores are subject to considerable interobserver variability [10]. Most existing prediction models rely on logistic or Cox regression analysis, with limited external validation and only fair overall performance. Recent evidence suggests that machine learning (ML) algorithms, which can handle numerous multidimensional variables with nonlinear relationships, may offer superior predictive accuracy for clinical outcomes [12]. To date, ML approaches for predicting mortality risk and prognosis in patients with anti-MDA5+DM-ILD have been infrequent, and there remains a lack of a concise, practical tool for directly calculating mortality risk in these patients.

Considering the limited sample sizes and inadequate establishment and validation of practical quantitative methods in previous studies, we identified the 3-month mortality risk, established a prediction model, and validated the model using real-world data from multicenter studies in China involving patients with anti-MDA5+DM-ILD. The aim was to create a reliable and easily implemented tool for clinical use.

Methods

Ethical Considerations

The Medical Ethics Committee of the Drum Tower Hospital Affiliated to Nanjing University Medical School approved the study (2020-050-01). The patients included in this study were derived from a secondary analysis of the previously confirmed study. The ethics committee determined that a second round of approval was not required.

Patients and Study Design

From January 2017 to December 2022, our study screened patients with anti-MDA5+DM-ILD across participating institutions. Inclusion criteria were patients with a newly observed ILD, detectable serum anti-MDA5, and DM. The diagnostic criteria for DM included Bohan and Peter criteria and were retrospectively reconfirmed according to the 2017 European League Against Rheumatism (EULAR)/American College of Rheumatology IIM classification criteria or the 2018 European Neuromuscular Centre (ENMC) DM criteria by experienced rheumatologists and pulmonary clinicians [13-15]. Exclusion criteria were as follows: (1) DM-ILD with other positive MSAs; (2) other connective tissue diseases (CTDs); (3) evidence of pulmonary infections at initial admission, as diagnosed clinically and by identifying pathogens; and (4) incomplete data or lost to follow-up.

Following application of these criteria, 509 (83.6%) eligible cases from an initial 609 patients were identified. Patients from the Nanjing Drum Tower Hospital (NJDTH; n=254, 49.9%) were randomly allocated to a training data set (n=203, 79.9%) for model development and internal validation using k-fold cross-validation and a testing set (n=51, 20.1%) for further model validation. External validation was conducted using cohorts from 4 additional institutes, referred to as external cohort 1 (n=92, 18.1%) and external cohort 2 (n=163, 32%).

Clinical Data Collection and Reference Standards

All clinical and laboratory data were extracted from electronic medical records (EMRs) at the beginning of the patients’ admission. Demographic characteristics, including age and gender, and clinical signs, such as muscle weakness, the Raynaud phenomenon, mechanic’s hands, Gottron sign/papules, heliotrope rash, fever, and arthralgia/arthritis, were documented and compared. Serum levels of muscle enzymes, such as creatine kinase (CK), alanine aminotransferase (ALT), aspartate aminotransferase (AST), and lactate dehydrogenase (LDH), were recorded. Serum antibody abnormalities, including Ro/SSA-52kDa antibody (Ro52) and MDA5 (autoimmune inflammatory myopathies 16 Ag-(IgG); EUROLINE), were analyzed. Disease severity evaluations and inflammatory mediators, such as C-reactive protein (CRP), the erythrocyte sedimentation rate (ESR), the white blood cell (WBC) count, the lymphocyte count, the neutrophil-to-lymphocyte ratio (NLR), and ferritin levels, were recorded and compared. In this study, all these variables were meticulously recorded during hospitalization, with a low incidence of missing data of less than 2%. Missing data were processed with the k-nearest neighbor (KNN) algorithm.

ILD has been defined by the American Thoracic Society/European Respiratory Society [16]. RP-ILD is characterized by the acute and progressive deterioration of ILD, manifesting as worsening dyspnea and interstitial changes on chest HRCT, necessitating hospitalization due to disease-related acute respiratory failure within 3 months of the onset of respiratory symptoms, after exclusion of other potential etiologies (eg, pulmonary infection, heart failure, embolism) [17,18]. The duration of the disease is marked from the initial occurrence of pulmonary, muscle, joint, or skin symptoms/signs.

The primary outcome was the risk of mortality at 3 months. Secondary outcomes included the 1-month mortality risk and the 6-month mortality risk.

Screening of Important Features for Model Construction

We used the BorutaShap algorithm for feature selection, which integrates Boruta and Shapley Additive Explanations (SHAP) algorithms to pinpoint the most influential features within a data set [19]. Consequently, the BorutaShap algorithm was used to sift through 25 parameters, including demographics, clinical features, and laboratory parameters, from the EMR system and identify the most pertinent features for model construction. In this study, only features categorized as “accepted” by the BorutaShap algorithm were incorporated into model development. Features labeled “tentative” or “rejected” were omitted.

Establishment and Validation of Prediction Models

Six ML algorithms were deployed to build prediction models: Extreme Gradient Boosting (XGBoost), logistic regression (LR), Light Gradient Boosting Machine (LightGBM), random forest (RF), support vector machine (SVM), and KNN. To guarantee robustness and reproducibility, all models were provided with identical input features. A thorough search of the hyperparameter space was conducted using both grid and random searches to optimize each model’s performance. The reliability of these models was appraised using various evaluation metrics, including the area under the receiver operating characteristic (ROC) curve (AUC), sensitivity, specificity, the positive predictive value (PPV), the negative predictive value (NPV), accuracy, and the F₁-score. Moreover, 10-fold cross-validations were implemented in the derivation cohort to validate the models.

The optimal model was elucidated using the SHAP methodology, which delineates the importance and contribution of each feature to the model. The model results were interpreted by qualifying the impact of each feature on the predictions. Ultimately, the optimal prediction model was integrated into a user-friendly web-based tool.

Statistical Analysis

Descriptive statistics were applied to summarize continuous variables as mean (SD) values and categorical variables as frequencies and percentage. The Fisher exact test was performed for categorical variables, while the Mann-Whitney U test was used for continuous variables to assess differences between the survival and mortality groups within 3 months. Statistical significance was set at P<.05. All statistical analyses were performed using SPSS Statistics version 26.0 (IBM Corporation), R Studio version 3.6.3 (R Foundation for Statistical Computing) and Python version 3.7.

Results

Patients’ Basic Clinical Characteristics

The study flowchart is shown in Figure 1. The NJDTH cohort had 280 (55%) patients with anti-MDA5+DM-ILD, and after exclusion, the final count was 254 (49.9%) patients. These patients were followed for a median of 6 months, divided into a training cohort of 203 (79.9%) patients and an internal testing cohort of 51 (20.1%) patients. The external validation cohort was further refined to 92 (18.1%) patients in cohort 1 and 163 (32%) patients in external cohort 2 (Figure 1). The entire patient group was analyzed for all-cause mortality. A comparative baseline analysis of 31 clinical variables, encompassing demographics, clinical severity, inflammatory indicators, serum antibody levels, is presented in Table 1. The study evaluated statistically significant differences in these variables among the groups. Of 254 valid samples, 159 (62.6%) were in the 3-month survival group and 95 (37.4%) in the 3-month death group in the NJDTH cohort. In terms of mortality, 108 (42.5%) patients died during the investigation period, with 71 (27.9%) within 30 days, 95 (37.4%) within 90 days, and 101 (39.78%) within 180 days. The 3-month mortality was up to 37.4%, suggesting the majority of deaths occurred within the first 3 months. Patients who died within 3 months showed more severe injury, as indicated by higher levels of inflammatory markers, such as CRP, ESR, LDH, WBC, ferritin, and NLR, and lower levels of lymphocytes compared to patients who survived beyond 3 months. RP-ILD characteristics were observed in 78 (82.1%) of 95 patients who died within 3 months in contrast to only 30 (18.9%) of 159 survivors. The mean age of patients who died within 3 months was 56.68 (SD 9.97) years, which was higher than that of patients who survived for 3 months (mean 51.50, SD 10.68 years). There were no statistically significant differences in patients’ characteristics between the training and testing sets. Other severity indices were comparable between the training and testing sets (Table 2).

‎

To account for varying severities of with anti-MDA5+DM-ILD patients, we recruited external cohort 1 mainly from the pulmonary medicine department and external cohort 2 from the rheumatology department. Patients in external cohort 1 tended to have severe illness than those in external cohort 2, as patients with respiratory failure were more likely to be admitted to the pulmonary medicine department. Analysis on a diverse range of severities of patients enhanced our model’s practicality and robustness.

Screening of Characteristic Factors for Risk of 3-Month Mortality

In total, 9 parameters (ie, RP-ILD, age, LDH, albumin [ALB], CRP, ESR, AST, NLR, and duration of disease) were selected for analysis using the BorutaShap algorithm (Figure 2). However, due to the overlap between disease duration and RP-ILD, RP-ILD was left for model development to balance model complexity and generalizability. Hence, 8 parameters were selected for further analysis.

‎

Comprehensive Analysis of the Prediction Model and Selection of the Optimal Model

To identify the best prediction model for the 3-month mortality risk in patients with anti-MDA5+DM-ILD, all models were evaluated using identical input features and assessed via AUC values, decision curve analysis (DCA), calibration curves, and precision-recall (PR) curves. XGBoost, LightGBM, and RF algorithms showed the highest AUCs in the training data set, but LR demonstrated the highest AUC of 0.877 (95% CI 0.744-0.985) in the validation data set (Figure 3A,B). Tables S1 and S2 in Multimedia Appendix 1 provide more details, indicating potential overfitting in the XGBoost, LightGBM, and RF algorithms. Furthermore, the LR model achieved a significantly lower Brier score of 0.145 compared to other models. The DCA curves in Figure 3C and the calibration plots in Figure 3D demonstrate that all 6 models provided a net clinical benefit over the full treat-all or treat-none strategies, with the LR model outperforming the other 5 models in the validation data set. Additionally, the LR model obtained the highest average precision (AP) value of 0.813 on the validation data set (Figure 3E,F). Based on these findings, the LR model was selected as the final model due to its superior performance.

‎

Ultimately, we selected RP-ILD, age, LDH, AST, ALB, NLR, CRP, and ESR as input features for the LR model. As shown in Figure 4 and Table S3 in Multimedia Appendix 1, the LR model demonstrated optimal performance with an AUC, sensitivity, specificity, and accuracy of 0.86, 84.8%, 84.4%, and 79.3%, respectively, on the validation data set and 0.90, 85.0%, 83.9%, and 84.3%, respectively, on the training data set (Figure 4A-D). The calibration of the LR model indicated strong concordance between the predicted probabilities and the actual observed 3-month deaths using Brier scores (Figure 4E). DCA revealed that the model provided greater net benefits over both treat-all and treat-none strategies (Figure 4F).

‎

SHAP Model Interpretation and Individual Analysis

To provide a clear explanation of the selected variables, we used SHAP to illustrate how these variables predict the 3-month mortality within the model. Figure 5A displays the 8 most influential features (RP-ILD, ESR, ALB, age, CRP, AST, NLR, and LDH) in our model. Each feature’s importance line is represented by colored dots, where red dots indicate high-risk values and blue dots signify low-risk values. Figure 5B ranks these 8 risk factors based on their average absolute SHAP values, with the x axis representing SHAP values that denote the prediction model’s importance. Additionally, in this model, the probability threshold for the 3-month mortality risk was established at 0.40473. We then provided 2 exemplary cases to illustrate the model’s interpretability. In one instance, a patient presenting with RP-ILD exhibited a notably elevated SHAP-predicted mortality risk of 0.718 (Figure 5C), and unfortunately, this individual succumbed to the disease within a 3-month period postonset. In contrast, a 64-year-old participant demonstrated a substantially lower SHAP-predicted mortality risk of 0.105 (Figure 5D) and remained alive, underscoring the potential utility of the web-based prognostic tool in clinical practice.

‎

Model External Validation and Establishment of a Web-Based Calculator

The AUC of our proposed model on external cohort 1 from 4 hospitals in China was 0.836 (Figure 6). The sensitivity, specificity, and accuracy for this cohort were 0.87, 0.739, and 0.783, respectively. For external cohort 2, the AUC, sensitivity, specificity, and accuracy were 0.915, 0.892, 0.865, and 0.847, respectively, indicating the model’s robust extrapolation capabilities. In addition, secondary outcomes (1-month and 6-month mortality risks) were evaluated in the 2 external validation cohorts. The model’s superior discrimination performance, using the AUC, sensitivity, specificity, precision, accuracy, and the F₁-score, is detailed in Tables S4 and S5 in Multimedia Appendix 1, Figure S1 in Multimedia Appendix 2, and Figure S2 in Multimedia Appendix 3.

‎

To enhance generalizability and facilitate clinical application, the optimal ML model was adapted to create a web-based calculator [20]. The calculator serves as a practical clinical tool for physicians specializing in respiratory medicine, rheumatology, and critical care medicine. It is particularly useful in the assessment of patients presenting with RP-ILD attributed to anti-MDA5+DM-ILD. A score exceeding 40.47% indicates a heightened risk of mortality, warranting consideration for aggressive management strategies.

Discussion

Principal Findings

To the best of our knowledge, this is the largest retrospective multicenter study to investigate the risk factors for 3-month mortality and to explore 6 ML models for comprehensive predictive analyses on anti-MDA5+DM-ILD cohorts in China. We identified a set of predictive risk factors and constructed a prediction model for patients with anti-MDA5+DM-ILD using ML algorithms in conjunction with clinical and laboratory data easily extracted from the EMR system. In this study, LR, using routinely collected data from real-world clinical practice, demonstrated superior performance across 8 predictive measures. The optimal model exhibited favorable discrimination and calibration, while also emphasizing explainability to enhance trust and transparency. Furthermore, variable importance was investigated at both individual and population levels. Additionally, external validation confirmed the superior predictive performance of this optimal model. To facilitate clinical application, a web-based tool was developed for health care professionals to use the proposed prediction model.

Previous studies have consistently reported high mortality rates exceeding 60% among patients with anti-MDA5+DM-ILD, particularly in East Asian populations, such as China [21]. The majority of deaths occurred within the first 3 months [22]. It is widely recognized that an excessive inflammatory response is associated with a heightened risk of RP-ILD and all-cause mortality [23]. Hyperinflammation plays a crucial role in the development and progression of anti-MDA5+DM-ILD. Uncontrolled hyperinflammation and persistent immune activation contribute to severe lung injury and RP-ILD, which may resemble the phenomenon known as “cytokine storm” observed in COVID-19 [24]. Previous studies have indicated that poor outcomes in patients with DM are correlated with certain factors, including advanced age and elevated levels of serum ferritin, LDH, NLR, and CRP [21]. Consistent with these findings, we established a prediction model incorporating 8 key variables of significant predictive importance: RP-ILD, ESR, ALB, age, CRP, AST, NLR, and LDH. Most of these variables are inflammatory markers that can reflect the hyperinflammatory state and were easy to obtain and observe in previous studies on patients with DM in a clinical setting. Moreover, classical statistical analysis has identified risk factors in these patient populations, including RP-ILD and age [21,25,26]. In line with previous findings, RP-ILD was found to be the most important feature according to the SHAP value and contributed to this prediction model. Age was ranked fourth in terms of its significance as a parameter associated with worsened outcomes. Therefore, this prediction model demonstrated superior performance, yielding AUCs of 0.90 in the internal cohorts and 0.832-0.919 in external cohorts.

There are currently no existing ML-based prediction models for 3-month mortality in individuals with anti-MDA5+DM-ILD, making it impossible to make a direct comparison with prior work. Previous studies have used logistic or Cox proportional hazard models as alternatives [10,27-29]. For instance, a nomogram model incorporating the duration of first-time symptoms, the presence of fever, pleural effusion (PE), total CT scores, and AST has been recommended as a prognostic predictor for anti-MDA5+DM-ILD outcomes [29]. Another model known as the “FLAIR score,” which combines ferritin, LDH, the semiquantitative anti-MDA5 grade, the HRCT imaging score, and RP-ILD/non-RP-ILD classification, was proposed to predict mortality in amyopathic DM-related ILD based on a large-scale Chinese single-center cohort study [10]. In contrast to our study, which specifically targeted patients with anti-MDA5+DM-ILD, Lian et al’s [10] investigation encompassed a more expansive patient population, not restricting their analysis solely to anti-MDA5+DM-ILD cases. Furthermore, although their investigation incorporated HRCT findings for scoring purposes, our study deliberately excluded HRCT as a discriminating factor in the evaluation process, attributable to the recognized interobserver and intraobserver variability inherent in HRCT interpretation.

Ouyang et al [30] reported a novel matrix prediction model that includes baseline characteristics, such as fever, ferritin≥1250 µg/L, and positive carcinoma embryonic antigen, which can predict 6-month all-cause mortality in patients with anti-MDA5+DM-ILD. However, these prediction models lack thorough external validation due to their reliance on small and nonrepresentative populations (eg, individuals from a single hospital), primarily due to the rarity of the disease. Moreover, the HRCT imaging score and the radiomics-based prediction model [31,32] are overly complex and impractical for routine clinical use. In contrast, this study leveraged routinely collected data from real-world clinical practices in China to develop an optimal prediction model, which is now accessible as a web-based tool. This tool enables clinicians to calculate individualized risks of 3-month mortality based on easily obtainable clinical data, including RP-ILD, age, AST, CRP, NLR, LDH, ALB, and NLR. It is important to emphasize that in real-world clinical practice, such models should be used holistically, considering all key features, rather than relying on a single feature for risk prediction. Therefore, the models require the input of a comprehensive set of features to accurately assess the risk in patients with anti-MDA5+DM-ILD.

Strengths and Limitations

Our study has several notable strengths. First, we conducted external validation in independent cohorts from 4 hospitals in China, encompassing various regions and settings. This inclusion of external cohorts 1 and 2 in China allows our findings to be representative of real-life clinical practice in our country. Second, we extracted, analyzed, and used data from routine clinical practice and finally established a practicable web-based algorithm for predicting 3-month mortality for patients with anti-MDA5+DM-ILD. Additionally, our study included a substantial number of patients with anti-MDA5+DM-ILD, further enhancing the robustness of our results.

However, it is important to acknowledge some limitations of our study. First, it was a retrospective study, which inherently introduces certain biases. Our patient population was sourced from large tertiary referral hospitals and consisted only of hospitalized patients, potentially leading to the inclusion of individuals with more severe disease. Second, although the predicted forced vital capacity (FVC<50%) was shown to independently predict 6-month all-cause mortality in patients with anti-MDA5-positive DM [29], we were unable to incorporate pulmonary function parameters into our prediction model due to missing data, particularly in patients with severe disease who had no opportunity to finish the pulmonary function test. Lastly, although the model demonstrated good discrimination and calibration in our study, the external validation cohort was limited to a Chinese population. It is necessary to develop a multiregion and multirace worldwide analysis and generate a more comprehensive and classical system to predict 3-month mortality for patients with anti-MDA5+DM-ILD.

Conclusion

In summary, our use of ML techniques and routinely collected clinical features helped us successfully develop an LR model that reliably predicts the risk of 3-month mortality in patients with anti-MDA5+DM-ILD. To facilitate the translation of our findings into clinical practice, we created a web-based tool that allows clinicians to calculate individualized risks of 3-month mortality. This tool can guide risk-stratified approaches to patient care and treatment.