Development and Validation of a Machine Learning–Based Early Warning Model for Lichenoid Vulvar Disease: Prediction Model Development Study

Introduction

In different time periods, lichenoid vulvar disease (LVD) has been variously named such as white spot disease, or vulvar white lesions. The term LVD was coined by the International Society for the Study of Vulvovaginal Disease in 1975. LVD is classified into 3 pathological types, namely vulvar squamous cell hyperplasia type, lichen sclerosus type, and mixed type [1,2]. It primarily manifests as mucocutaneous hypopigmentation of vulva to gray-brown or whitening, roughening of skin with lichenification or reduction of elasticity with atrophic thinning and chapping, and introital stenosis. It is characterized by pruritus or pain, regardless of seasons or day-night variations, with exacerbation often occurring at night [3]. The lesion locations include the labia majora and minora, the clitoral region, perianal area, and the pubic hairline. LVD is often recurrent, and in the later stage, it can lead to vaginal stenosis, causing dyspareunia and even carrying the risk of carcinoma, which severely impacts the physical and psychological well-being of the patients [4-6]. It is evident that Chinese women are often reluctant to seek medical care from formal health care facilities due to economic, cultural and traditional concept, and other factors. Clinical practice, both domestically and internationally, has shown that early warning, timely diagnosis, prompt intervention, and immediate treatment are effective strategies in preventing the development of LVD [6,7]. The early identification of high-risk individuals and proactive prevention of LVD have garnered increasing attention. However, current research on the early identification of risk factors for LVD remains inadequate, lacking systematic and comprehensive approaches. For instance, Sideri et al [8] conducted a case-control study with 75 cases and 225 age-matched controls to assess risk factors for LVD. The study collected data on personal characteristics and habits, gynecologic and obstetric history, general sexual habits, and selected dietary habits. The findings indicated a higher risk of vulvar lichen sclerosis (VLS) in parous women compared with nulliparous women, with no significant increase in risk correlating with the number of births. Smith and Haefner [9] highlighted the association of VLS with autoimmune conditions, genetic factors, hormone levels, and infections. These studies were limited to identifying high-risk factors and did not develop early warning models.

Given the complexity and diversity of LVD risk factors, it is crucial to actively explore these factors and construct personalized warning models using relevant clinical variables to assess disease risk in patients. Yet, to date, there has been insufficient research by scholars, both domestically and internationally, on the risk factors and warning models for LVD. In light of these gaps, this study represents the first systematic exploration of the risk factors associated with LVD. Using an evidence-based medicine approach, a warning model has been constructed, offering a tool for the early identification of high-risk individuals. Furthermore, this model has the potential to enhance the diagnostic and treatment capabilities of medical personnel in primary community health service centers. This holds great significance in mitigating overall health care expenditures and the disease burden. More importantly, our research is the first to apply machine learning techniques to predict LVD, filling a novel research gap in the field.

Methods

Study Candidates

The study incorporated 2634 female patients who attended the West China Second Hospital of Sichuan University from January 2013 to December 2017. Based on the results of their medical examinations, these patients were divided into 2 groups, that is, the normal vulvovagina group (group 0) consisting of 1080 cases and the LVD group (group 1) comprising 1554 cases. Inclusion criteria for the study encompassed female patients who were definitively diagnosed with LVD, were fully informed, voluntarily participated in the study, and could undergo long-term follow-ups. Exclusion criteria were as follows: (1) patients with mental illness history, cerebral infarction, or other conditions that may lead to communication difficulties; (2) patients in the acute phase of the disease; and (3) patients diagnosed with malignant tumors.

Diagnosis of LVD

Per the 2021 European Guideline for the Management of Vulval Conditions, patients can be diagnosed with LVD according to the characteristic clinical manifestations and signs, including symptoms such as vulvar pain and itching (with severe cases leading to scratching), a burning sensation in the vulvovaginal area, abnormal vaginal discharge, dyspareunia, or even dysuria, along with signs such as vulvar skin keratinization hypertrophy, coarsening and thickening of the cortex or with part atrophy thinning, reduced or absent skin elasticity, and in severe cases, chapping and festering, hypopigmentation, or whitening of the skin [10]. In this study, however, the diagnostic criteria for LVD involve not only the characteristic clinical appearances but also require confirmation through biopsy. Conducting a biopsy before treatment is advantageous for establishing a definitive histopathological record for the diagnosis.

Collection of Clinical Information

Based on previous domestic and international research findings combined with clinical experience, we selected and collected clinical information related to female LVD [11-14]. This included factors such as age, educational attainment, vaginitis, urinary incontinence, humidity of the long-term residential environment, type of long-term residence (urban or rural), spicy dietary habits, regular intake of coffee or caffeinated beverages, good sleep habits, daily sleep duration (<8 hours or ≥8 hours), diabetes mellitus, smoking history (≥100 cigarettes smoked in lifetime) [15], alcohol consumption, presence of autoimmune diseases, human papilloma virus status, presence of allergic diseases, menopausal status, hypertension, frequent seafood consumption, and income level.

Statistical Analysis

In this study, the K-Nearest Neighbor imputer method was applied to fill in missing variables. To ensure the accuracy of the results after filling, variables with a missing rate greater than 50% (such as occupation, income level, educational attainment, age, and frequent consumption of seafood or high-fat foods) were excluded from modeling. Risk factors associated with female LVD were obtained after univariate analysis of all variables on the postfilled data. Variables that exhibited significance in the univariate statistical test were included in the recursive feature elimination for multivariate selection, and the area under the curve (AUC) value was highest when 10 variables were included [16]. Subsequently, these selected variables were used for machine learning modeling, resulting in the construction of logistic regression (LR), random forests (RF), gradient boosting machine (GBM), adaptive boosting (ADA), eXtreme gradient boosting (XGB), and categorical boosting (CatBoost) analysis models. The rationale for selecting these models lies in their exemplification of fundamental and archetypal algorithmic types within machine learning, LR for classification model, RF for bagging model, ADA for boosting model, and GBM for gradient boosting tree model. Widely applied across diverse domains and tasks, these algorithms consistently demonstrate robust performance. Furthermore, XGB and CatBoost represent advanced models that refine and enhance these classic algorithms, showcasing notable prowess, especially within the gradient boosting tree category (Multimedia Appendix 1) [17]. The predictive efficacy of these 6 models was assessed using receiver operating characteristic (ROC) curves, accuracy, sensitivity, and specificity. Based on the above best model, the correlation ranking between each variable and the target variables was investigated using SHapley Additive exPlanations (SHAP) plots. The details are shown in the study design workflow (Figure 1).

‎

Ethical Considerations

Ethical approval was granted by the Ethics Committee of West China Second Hospital of Sichuan University (2023179) on September 21, 2023. Informed consent was obtained from all participants. All procedures were performed in accordance with relevant guidelines.

Results

Univariate Analysis of Risk Factors for LVD

According to the results of case examinations, the study cohort was stratified into 2 groups, the normal vulvovagina group (group 0) with 1080 cases and the LVD group (group 1) with 1554 cases. The univariate analysis was performed, revealing statistically significant differences between the 2 groups in various factors, including vaginitis (P<.001), humidity of the long-term residential environment (P=.01), spicy dietary habits (P=.00), regular intake of coffee or caffeinated beverages (P=.00), daily sleep duration (P<.001), diabetes mellitus (P<.001), smoking history (P<.001), presence of autoimmune diseases (P<.001), menopausal status (P<.001), and hypertension (P<.001). Conversely, no statistically significant discrepancies were observed between the 2 groups in other baseline data, such as urinary incontinence (P=.07), type of long-term residence (P=.11), good sleep habits (P=.24), alcohol consumption (P=.42), and presence of allergic diseases (P=.52). Detailed findings are presented in Table 1.

A Risk Model to Predict LVD

The establishment of the warning model begins with the foundation of massive data provided by large-sample retrospective clinical profiles, as the basis for evaluating numerous variables and screening out truly meaningful risk factors. Subsequently, the construction of machine learning models serves as the cornerstone for developing the early warning model. Further validation of the model’s accuracy is performed in clinical follow-up cohorts and prospective studies to establish the genuinely significant warning model. Since 2013, the pelvic floor group in West China Second Hospital of Sichuan University, has set up a big data cloud platform encompassing data from tens of thousands of patients with LVD, which provides a substantially big data foundation for the development of early warning and diagnostic model for LVD.

The variables that showed significance in the univariate statistical tests were included in the recursive feature elimination multifactorial screening. As depicted in Figure 2, AUC was used as the evaluation metric, and the highest AUC score with 0.794 was achieved when 10 variables were included. In Table 2, the predictive models were constructed using various machine learning methods, including LR, RF, GBM, ADA, XGB, and CatBoost analysis models, based on the above selected 10 variables. Optimal parameters for each model were determined using grid search method combined with 5-fold cross validation. A comprehensive elucidation of the model principles and parameter selections is provided in the appendix. Results indicated superior performance of the GBM model, yielding an accuracy value of 0.747, sensitivity of 0.703, and specificity of 0.818. The ROC curves demonstrated that the AUC score of the GBM model was 0.814, as illustrated in Figure 3. In summary, the GBM warning model exhibited superior predictive efficacy, with higher values for the area under the ROC curve, accuracy, sensitivity, and specificity compared with the other 5 warning models.

‎

‎

Shapley Additive Explanations Correlation Analysis

To further analyze the impact of multiple factors on the outcome variable within the GBM model, SHAP plots were generated. These plots were constructed to interpret the correlations between various factors and the target variables. Notably, it was observed that menopausal status exhibited the strongest positive dependence with the target variables. Following this, the presence of autoimmune diseases also displayed a positive relevance. Daily sleep duration was positively correlated with LVD, while vaginitis showed a negative association. Further details of the correlation results are presented in Figure 4.

‎

External Validation of the LVD Warning Model

To verify the effectiveness of the model, it was first internally validated with the ROC curve, and the results indicated the AUC value of 0.814, as illustrated in Table 2. Furthermore, external validation was performed using baseline data from 356 patients in 2018, and the results showed that the corresponding GBM model achieved an AUC score of 0.778, an accuracy of 0.732, sensitivity of 0.713, and specificity of 0.767. In comparison with the internal validation, the AUC, accuracy, and specificity values only decreased by 0.036, 0.015, and 0.051, respectively, while the sensitivity value improved by 0.1. Therefore, it is evident that the constructed risk warning model exhibits favorable predictive efficacy, as depicted in Figure 5 and Table 3. Afterwards, the GBM warning model was applied to predict the training dataset and obtain predictive probabilities. Subsequently, k-means algorithm was used to cluster those probabilities, yielding 3 cluster centers, 0.45529467, 0.81832383, and 0.26538643. The thresholds for 3 risk zones were calculated as 0.3603 and 0.6368. In short, Figure 6 displays the GBM warning model’s classification of individuals into 3 groups based on predictions, those below 0.3603 as the low-risk group, those above 0.6368 as the high-risk group, and those within the range of 0.3603 to 0.6368 as the medium-risk group. The use of population-related variables as input for the alert model led to the generation of predicted values, depicting varying degrees of risk, as demonstrated in Figure 7.

‎

‎

‎

Discussion

Principal Findings

LVD is a common chronic gynecological condition, clinically characterized by vulvar hypopigmentation and pruritus. In middle and later stage, it may lead to dysuria, dyspareunia, introital stenosis, and vulvovaginal atrophy, causing considerable physical and psychological anguish to affected women [18,19]. Many patients experience delayed diagnosis and treatment due to the sensitive nature of the condition, which can cause embarrassment and feelings of shame, or due to misdiagnosis resulting from symptomatic variations. Early misdiagnosis of LVD, particularly VLS, can result in delayed treatment, leading to irreversible vulvar structural abnormalities, scarring, and, in severe cases, genital cancer [20]. Currently, there is no gold-standard treatment for LVD. Conventional first-line therapies are associated with adverse reactions and frequent disease recurrence. Emerging approaches, such as energy-based therapies and platelet-rich plasma injections, have yet to be fully integrated into clinical practice. Therefore, it is of paramount importance to identify factors potentially contributing to LVD in women and establish a predictive model to warn about its risk. To our knowledge, this study is the first systematic investigation of risk factors associated with LVD in women and the development of a predictive model rooted in the principles of evidence-based medicine. Univariate analysis revealed that vaginitis, urinary incontinence, humidity of the long-term residential environment, spicy dietary habits, regular intake of coffee or caffeinated beverages, daily sleep duration, diabetes mellitus, smoking history, presence of autoimmune diseases, menopausal status, and hypertension were all significant risk factors affecting female LVD. Furthermore, the GBM warning model outperforms the other 5 predictive models, demonstrating superior performance with higher ROC curve area, accuracy, sensitivity, and F₁-score. The GBM model reveals that menopausal status has the strongest positive correlation with female LVD, followed by the presence of autoimmune diseases, which also displayed a positive dependency.

Comparison With Previous Work

The GBM algorithm, a type of boosting algorithm, is capable of exploring the relationships between observed outcomes and various target variables, making it useful for clinical analysis of high-risk factors contributing to inducing the disease [21,22]. In this study, the GBM model was used to construct SHAP plots for analyzing the correlations between multiple factors and the target variables. The results revealed that menopausal status exerted the strongest positive correlation with the development of LVD in women, exhibiting a positive dependency. This finding aligns with the clinical observation that perimenopausal women have a higher incidence of LVD [23]. Günthert et al [24] have reported that the disease may be attributed to endocrine factors, the dearth of sex hormones, as the sex hormone levels in serum of affected individuals are lower than those of normal women of the same age, suggesting that sex hormone may be one of the contributing factors to the pathogenesis of LVD. Furthermore, the presence of autoimmune systemic diseases, daily sleep duration, diabetes, and humidity of the long-term residential environment are all positively correlated with LVD. The skin is a peripheral lymphoid organ with immunological functions that is closely connected to the systemic immune system. A substantial body of evidence has established a close association between LVD and autoimmune disorders. This relationship is particularly evident in cases of vulvar lichen sclerosus, a lymphocyte-mediated chronic localized skin disease, where both cellular and humoral immunity are involved in the pathogenesis. Therefore, patients with LVD often present with various autoimmune diseases, such as vitiligo and hypothyroidism, among others [11,25]. Furthermore, some researches have indicated that diabetes is a systemic metabolic disorder often associated with various skin lesions [26,27]. In particular, individuals with type 2 diabetes are more likely to develop a range of skin disorders. In addition, diabetes tends to exacerbate the pruritus severity in patients with LVD. The intensity of itching is positively correlated with blood glucose levels, indicating that higher blood glucose values result in more pronounced itching symptoms. Similarly, excessive daily sleep duration has been linked to an increased risk of developing type 2 diabetes, thereby further raising the risk of LVD [28,29]. On the other hand, vaginitis showed a negative association with LVD. This could be attributed to the fact that women with vaginitis typically receive prompt and effective diagnosis and treatment, which reduces the risk of overusing medications, for some cases of LVD are due to the overuse of drugs mainly containing aromatics.

Strengths and Limitations

To sum up, it is crucial for women to cultivate healthy lifestyles and dietary habits, as well as to be informed about the factors related to LVD in order to reduce the incidence of this condition. Furthermore, the study demonstrates that the warning model exhibits strong predictive capabilities and discriminatory effect. The characteristics and strengths of the risk warning model primarily manifest in the following aspects: (1) it is convenient and easy to implement in clinical practice and (2) it can promptly identify the high-risk individuals and provide preventive interventions. To enhance the model’s clinical use, we devised a web-based calculator and mobile app featuring easily accessible functionalities for patients. As a self-diagnostic tool, it offers early detection and timely intervention for LVD.

For female patients identified as low risk for predicted outcomes, it is advisable to undergo appropriate treatment for autoimmune conditions such as thyroid disorders, systemic lupus erythematosus, psoriasis, rheumatism, or arthritis. In addition, managing blood sugar levels, incorporating moderate physical activity, and ensuring adequate sleep duration are recommended. Furthermore, avoiding tight fitting garments, using mild soaps during bathing (avoiding its application on the vulva and instead cleansing the area with water only), gently patting the vulva dry post bathing, and using 100% cotton, unscented or fragrance-free menstrual pads are suggested. Emphasizing guidance on vulvovaginal medications, particularly avoiding products containing fragrances and preservatives, alongside health education, is paramount. Elevating awareness and self-management skills regarding LVD, especially recognizing the risk of neoplastic change associated with the absence of standardized treatment and follow-up, is crucial [7]. Low-risk individuals should maintain annual follow-up appointments.

For individuals categorized as medium-risk, extending lifestyle interventions akin to those recommended for low-risk individuals is advisable. Furthermore, incorporating general management strategies for perimenopause and hormone replacement therapy. In addition, applying preservative-free, unscented or fragrance-free emollient to hold moisture in the skin and improve barrier function is recommended [10]. Medium-risk individuals should maintain biannual follow-up appointments.

For high-risk patients, a higher frequency of follow-ups is recommended, ideally every 3 months. Depending on the situation, timely referral to a higher-level hospital should be considered to implement more precise and personalized treatment and monitoring. In cases where vulvar symptoms are significant, such as persistent itching and lack of self-relief, prompt medical attention is advised. Routine secretion tests should be conducted to rule out herpes simplex and candidal infections, with additional tests for autoimmune diseases, thyroid disease, and diabetes performed as necessary. Dermatoscopy can be used when feasible. After excluding conditions such as vaginitis, vulvar pathological examination may be warranted. In addition, it is advised to avoid any genital skin irritants and allergens, such as cleansing products, frequent water exposure, incontinence, and tight-fitting clothing. Some activities, such as cycling or horse riding, may exacerbate symptoms. Regular application of a barrier emollient to affected areas can protect against local irritants such as urine and menstrual blood [20]. During intercourse, it is recommended to use adequate lubrication, with unscented or fragrance-free silicone-based lubricants. Patients should be advised to closely monitor their vulvar condition and contact their doctor if they notice any changes in appearance or texture, such as lumps, ulceration, or skin hardening, or if there are significant changes in symptoms [10].

Limitations

This study has a few limitations. It is a single-center cohort study based on data from a single ethnic group, which may limit the generalizability of the findings. In addition, the relatively small sample size could affect the accuracy of the model’s predictions. Future research should aim to include larger sample sizes and to optimize and validate the LVD warning model through multi-center studies involving diverse ethnic and racial groups.

Conclusions

In conclusion, developing an early warning model for female LVD on the basis of evidence-based medicine, promptly identifying high-risk factors for the condition, and implementing risk-stratified population management are crucial measures to enhance the effectiveness of preventing and treating female LVD. By using the warning model to classify individuals into high-risk and low-risk groups, it is conductive to explore the tailored health intervention strategies, and afford personalized health management recommendations, encompassing adjustments to individual lifestyle habits and the implementation of improved conservative treatment protocols. This contributes to an effect evaluation of grading management mode, facilitating the realization of personalized and long-term pelvic health control patterns. Early detection and prompt treatment of LVD can thus be achieved, thereby ameliorating the quality of life and overall well-being for women.