This paper is in the following e-collection/theme issue:

Digital Health Reviews (1174) Clinical Information and Decision Making (1314) Decision Support for Health Professionals (1127) Machine Learning (1370) Artificial Intelligence (1186) Precision Medicine (63)

Published on in Vol 25 (2023)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/44895, first published .
The Value of Applying Machine Learning in Predicting the Time of Symptom Onset in Stroke Patients: Systematic Review and Meta-Analysis

The Value of Applying Machine Learning in Predicting the Time of Symptom Onset in Stroke Patients: Systematic Review and Meta-Analysis

The Value of Applying Machine Learning in Predicting the Time of Symptom Onset in Stroke Patients: Systematic Review and Meta-Analysis

Authors of this article:

Jing Feng1 Author Orcid Image ;   Qizhi Zhang1 Author Orcid Image ;   Feng Wu2 Author Orcid Image ;   Jinxiang Peng3 Author Orcid Image ;   Ziwei Li4 Author Orcid Image ;   Zhuang Chen5 Author Orcid Image
Article Authors Cited by Tweetations (4) Metrics

    Review

    1Department of Neurology, Fifth People’s Hospital of Jinan, Jinan, China

    2Department of Pulmonary Disease and Diabetes Mellitus, Central Hospital of Enshi Tujia and Miao Autonomous Prefecture, Enshi, China

    3Medical Department, Hubei Enshi College, Enshi, China

    4Experimental Center, Shandong University of Traditional Chinese Medicine, Jinan, China

    5Department of Cardiovascular Medicine, Fifth People’s Hospital of Jinan, Jinan, China

    *these authors contributed equally

    Corresponding Author:

    Zhuang Chen, MD

    Department of Cardiovascular Medicine

    Fifth People’s Hospital of Jinan

    24297 Jingshi Road

    Huaiyin District

    Jinan, 250000

    China

    Phone: 86 18764026019

    Email: humourzhuang@163.com


    Background: Machine learning is a potentially effective method for identifying and predicting the time of the onset of stroke. However, the value of applying machine learning in this field remains controversial and debatable.

    Objective: We aimed to assess the value of applying machine learning in predicting the time of stroke onset.

    Methods: PubMed, Web of Science, Embase, and Cochrane were comprehensively searched. The C index and sensitivity with 95% CI were used as effect sizes. The risk of bias was evaluated using PROBAST (Prediction Model Risk of Bias Assessment Tool), and meta-analysis was conducted using R (version 4.2.0; R Core Team).

    Results: Thirteen eligible studies were included in the meta-analysis involving 55 machine learning models with 41 models in the training set and 14 in the validation set. The overall C index was 0.800 (95% CI 0.773-0.826) in the training set and 0.781 (95% CI 0.709-0.852) in the validation set. The sensitivity and specificity were 0.76 (95% CI 0.73-0.80) and 0.79 (95% CI 0.74-0.82) in the training set and 0.81 (95% CI 0.68-0.90) and 0.83 (95% CI 0.73-0.89) in the validation set, respectively. Subgroup analysis revealed that the accuracy of machine learning in predicting the time of stroke onset within 4.5 hours was optimal (training: 0.80, 95% CI 0.77-0.83; validation: 0.79, 95% CI 0.71-0.86).

    Conclusions: Machine learning has ideal performance in identifying the time of stroke onset. More reasonable image segmentation and texture extraction methods in radiomics should be used to promote the value of applying machine learning in diverse ethnic backgrounds.

    Trial Registration: PROSPERO CRD42022358898; https://www.crd.york.ac.uk/Prospero/display_record.php?RecordID=358898

    J Med Internet Res 2023;25:e44895

    doi:10.2196/44895

    Keywords

    machine learning; ischemic stroke; onset time; stroke 


    Stroke is the second leading cause of disability and death; it makes the largest contribution to global neurological disability-adjusted life-years, with an estimated prevalence of 80.1 million cases in 2016 [1]. Current treatments for stroke are mainly thrombolytic therapy and intravascular thrombectomy [2-4]. The effect of thrombolytics varies with the time of treatment. Thrombolytic treatment within a specific time window can be effective, but the treatment is ineffective if it is outside the time window. Stroke treatment is urgent, with a recommended time window within 4.5 or 6 hours [5,6]. Unfortunately, patients with an unclear time of stroke onset are common in clinical practice [7]. Approximately a quarter of patients with acute stroke are not sure when their symptoms began, which causes enormous obstacles in selecting clinical medical strategies [8,9].

    Nowadays, the identification of the time of stroke onset mainly depends on magnetic resonance imaging (MRI). By screening brain tissue comprehensively, MRI can detect ischemia-induced hydrodynamic changes and ischemic tissue, which is helpful for clinicians to assess the pathological features of oversize, swelling, and bleeding [10]. MRI-based techniques have been adopted to stratify stroke patients with an unknown time of onset. However, image evaluation generally relies on clinical experience, which is inevitably subjective [11]. Existing evidence has shown that the performance of imaging in estimating stroke onset time needs to be improved [12]. Heterogeneity in image resolution and medical workers’ experience compromises the reliability of imaging in predicting the time of stroke onset [13,14]. Therefore, more objective approaches are needed to accurately identify the time of symptom onset in stroke patients.

    Machine learning has been introduced for the prediction of the onset time of acute ischemic stroke in recent years [15-18]. Numerous studies on the application of machine learning in identifying and predicting the time of stroke onset have been published [19-21].

    However, the heterogeneity between different studies cannot be ignored, given the diversity of machine learning methods and the differences in included modeling indicators. Whether machine learning can effectively predict stroke onset time remains hotly debated. Therefore, this systematic review and meta-analysis aimed to explore the value of applying machine learning in predicting the time of symptom onset in stroke patients.


    This systematic review and meta-analysis was reported in accordance with the PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) guidelines. The protocol for this study was registered on PROSPERO (CRD42022358898).

    Literature Search

    We searched the PubMed, Web of Science, Embase, and Cochrane databases comprehensively from their inception to July 24, 2022. A combination of keywords and subject headings was used for the search, and the language was restricted to English. The main search terms included stroke, cerebrovascular accident, apoplexy, machine learning, onset, and stroke time. The specific search strategy is presented in Multimedia Appendix 1. Two researchers independently searched the literature, and any disagreements were solved by a third researcher. We performed a further, manual search of the reference lists of the included articles to find potential eligible studies.

    Eligibility Criteria

    After removing duplicates using EndNote (Thomson ResearchSoft), we screened the literature according to the inclusion and exclusion criteria presented in Textbox 1.

    Textbox 1. Inclusion and exclusion criteria.

    Inclusion criteria

    • Population: ischemic stroke patients
    • Outcomes: (1) the machine learning prediction model for stroke onset time was completely constructed; (2) research was included on different machine learning models in the same data set
    • Research types: case-control studies, cohort studies, nested case-control studies, and case-cohort studies
    • Language: English
    • Other criteria: Studies without external validation

    Exclusion criteria

    • Outcomes: (1) Only the influencing factors were analyzed and a complete risk model was not built; (2) the following outcome indicators were missing: receiver operating characteristics curve, C statistic, C index, sensitivity, specificity, accuracy, recovery, confusion matrix, diagnostic quad table, F1-score, calibration curve
    • Research types: meta-analyses, reviews, guides, and expert opinions
    • Language: other than English
    • Other criteria: (1) studies with few samples (<50 cases), (2) validation studies of maturity scales, (3) studies on the accuracy of single-factor prediction

    Literature Screening and Data Extraction

    The retrieved documents were imported into EndNote X9 for management. The titles and abstracts were screened to exclude irrelevant studies. The full texts of the remaining articles were downloaded and read to select eligible studies. Before data extraction, we prepared a standard electronic information extraction form. The primary outcomes included in the study are C index, sensitivity, and specificity. The abovementioned literature screening and information extraction were independently carried out by 2 researchers, and any disagreement was resolved by a third researcher.

    Risk of Bias Assessment

    The PROBAST (Prediction Model Risk of Bias Assessment Tool) was used [22] to evaluate the risk of bias in the included studies. This tool contains several questions across 4 different domains: participants, predictors, outcome, and analysis. The overall applicability of the included studies was also assessed.

    The 4 domains comprise 2, 3, 6, and 9 specific questions, respectively, and each question was answered as “yes,” “probably yes,” “no,” “probably no,” or “no information.” If a domain included at least 1 response of “no” or “probably no,” it was considered at high risk. A domain with all the responses being “yes” or “probably yes” was judged low risk. When all domains were considered low risk, the overall risk of bias was rated low. When at least 1 domain was considered high risk, the overall risk of bias was high. Two researchers independently carried out the risk of bias assessment using PROBAST and cross-checked their results. If there was any dispute, a third researcher was consulted to assist in the final decision.

    Statistical Analysis

    We conducted a meta-analysis of the evaluation indicators (C index and accuracy) of the machine learning models. If the C index had no 95% CI or SE, we estimated the SE with the calculation method proposed by Debray et al [23]. Given different variables and inconsistent parameters in learning models, a random effects model was preferred in the meta-analysis of the C index. In addition, we used a bivariate mixed effects model to perform the meta-analysis of sensitivity and specificity. The meta-analysis of our study was conducted using R (version 4.2.0, R Development Core Team).


    Study Selection

    After removing the duplicates, a total of 843 papers were left. Based on the title and abstract screening, the full texts of 17 papers were downloaded and read. Finally, 13 eligible papers were included in this study. The details of the literature screening process are shown in Figure 1.

    ‎
    Figure 1. Flow chart of study selection process for studies included in and excluded from the meta-analysis.

    Study Characteristics

    There were 11 case-control studies and 2 cohort studies. Among the included studies [17,18,24-34], 5 were from China, 3 from Belgium, 4 from the United States, and 1 from South Korea (shown in Table 1) There were 55 models involved, with 41 models in the training set and 14 in the validation set. These models were mostly based on logistic regression (LR), with radiomics-based modeling variables.

    Table 1. Characteristics of included studies for the value of applying machine learning in predicting the time of symptom onset in stroke patients.
    Study, yearCountryStudy typePatient sourceN (total)Onset time (hours)N1a, participants, nN2b, participants, nValidation set generation methodN3c, participants, nModeling variablesType of model
    Wouters et al [24], 2016BelgiumCase-controlMulticenter141<4.5,<3,<650,15,92141Leave-one-outN/AdMRIeLRf
    Zhu et al [25], 2021ChinaCase-controlMulticenter268≤4.5,>4.5173,95N/ARandom sampling plus external verificationN/AMRISVMg, LR, RFh, GBDTi, DTj
    Zhang et al [17], 2022ChinaCase-controlSingle-center84≤4.5,>4.558,2651Random sampling33MRIDLk, XGBoostl, GBMm, GLMn, RF, DT
    Zhang et al [26], 2021USCase-controlSingle-center422N/AN/A340Random sampling82MRICNNo
    Yao et al [27], 2020ChinaCase-controlSingle-center316≤4.5,>4.5N/A156,81Random sampling52,27CTpLR
    Wouters et al [28], 2017BelgiumCohort studyRegister database181≤6 ,＞6124,5769,22External verification55,35MRILR
    Wouters et al [29], 2018BelgiumCohort studySingle-center3184.5N/AN/AN/AN/AMRILR
    Wen et al [30], 2021ChinaCase-controlMulticenter123≤4.5,>4.546,7785Random sampling plus external verification38,60CTLR
    Polson et al [31], 2021USCase-controlN/A772N/AN/AN/AExternal verificationN/AMRIDT
    Lee et al [18], 2020KoreaCase-controlSingle-center3554.5355299Random sampling56MRILR, RF, SVM
    Jiang et al [32], 2022ChinaCase-controlMulticenter5544.5N/AN/AN/AN/AMRISVM, NBq, KNNr, AdaBoost, ANNs
    Ho et al [33], 2019USCase-controlSingle-center131≤4.5,>4.585,46N/ACross-validationN/AMRILR, RF, XGBoost, SVM
    Ho et al [34], 2017USCase-controlRegister database105<4.5,≥4.583,22N/AN/AN/AMRISVM, RF, XGBoost

    aN1: people trained at the time of stroke onset. Values separated by commas refer to the number people with stroke onset at the respective times in the “onset time” column.

    bN2: training set. Values separated by commas refer to the number people with stroke onset at the respective times in the “onset time” column.

    cN3: validation set. Values separated by commas refer to the number people with stroke onset at the respective times in the “onset time” column.

    dN/A: not applicable.

    eMRI: magnetic resonance imaging.

    fLR: logistic regression.

    gSVM: support vector machine.

    hRF: random forest.

    iGBDT: gradient-boosted decision tree.

    jDT: decision tree.

    kDL: deep learning.

    lXGBoost: extreme gradient boosting.

    mGBM: gradient boosting machine.

    nGLM: generalized linear model.

    oCNN: convolutional neural network.

    pCT: computed tomography.

    qNB: naive Bayes.

    rKNN: k-nearest neighbor.

    sANN: artificial neural network.

    Quality of Evidence and Risk of Bias

    All 41 models in the training set were radiomics-based machine learning models. The inclusion and exclusion criteria for the cases were relatively reasonable. In addition, the onset time records were suitably reliable. In terms of data generation, only one study described the processing method for missing clinical values. Of note, no missing values in the radiomics data were mentioned. In the modeling process, the verification set was mostly generated through random sampling for internal verification, while 3 multicenter studies had independent external verification sets. There were fewer than 100 samples in the verification set. The risk of bias assessment of the included studies is presented in Figure 2.

    ‎
    Figure 2. Risk of bias assessment for studies included in the meta-analysis.

    Statistical Analysis

    C Index

    There were a total of 3770 participants and 55 machine learning models, with 41 models in the training set and 14 in the validation set. The meta-analysis of the C index in the training set and in the validation set included 13 studies and 6 studies, respectively. The overall C index was 0.800 (95% CI 0.773-0.826) in the training set and 0.781 (95% CI 0.709-0.852) in the validation set. The meta-analysis of the C index is shown in Table 2.

    Table 2. The C index results for the value of applying machine learning in predicting the time of symptom onset in stroke patients.
    SubgroupTraining setTesting set

    		Studies, nC index (95% CI)Studies, nC index (95% CI)
    Model

    		Logistic regression120.807 (0.741-0.872)70.807 (0.691-0.923)

    		Support vector machine80.782 (0.724-0.839)10.895 (0.854-0.936)

    		Artificial neural network20.798 (0.689-0.907)N/AaN/A

    		Boosting60.787 (0.697-0.877)20.714 (0.580-0.849)

    		Deep learning10.900 (0.816-0.984)10.754 (0.584-0.924)

    		Random forest60.795 (0.733-0.856)10.697 (0.502-0.893)

    		Decision tree30.822 (0.721-0.923)10.681 (0.488-0.874)

    		Generalized linear model10.798 (0.687-0.909)10.662 (0.459-0.865)

    		K-nearest neighbor10.797 (0.762-0.832)N/AN/A

    		Naive Bayes10.802 (0.767-0.837)N/AN/A
    Onset time (hours)

    		≤4.5 hours380.80 (0.77-0.83)130.79 (0.71-0.86)

    		≤6 hours20.76 (0.71-0.81)10.73 (0.62-0.84)

    		≤3 hours10.90 (0.81-0.99)N/AN/A
    Overall410.800 (0.773-0.826)140.781 (0.709-0.852)

    aN/A: not applicable.

    Sensitivity and Specificity

    The meta-analysis of sensitivity in the training and validation sets included 10 studies and 5 studies, respectively. The sensitivity was 0.76 (95% CI 0.73-0.80) in the training set and 0.81 (95% CI 0.68-0.90) in the validation set. The meta-analysis of specificity in the training and validation sets included 10 and 5 studies, respectively. The specificity was 0.79 (95% CI 0.74-0.82) in the training set and 0.83 (95% CI 0.73-0.89) in the validation set. The sensitivity and specificity in the training set are presented in Figure 3; the sensitivity and specificity in the validation set are shown in Figure 4.

    ‎
    Figure 3. The sensitivity and specificity of the training set for the value of applying machine learning in predicting the time of symptom onset in stroke patients. Some studies used multiple different types of machine learning models; therefore, some studies are presented multiple times in the figure [17, 18, 24-34].
    ‎
    Figure 4. The sensitivity and specificity of the validation set for the value of applying machine learning in predicting the time of symptom onset in stroke patients. Some studies used multiple different types of machine learning models; therefore, some studies are presented multiple times in the figure [17, 18, 24-34].

    Subgroup Analysis According to Onset Time

    Subgroup analyses of the C index were performed for onset times within 3 hours, 4.5 hours, and 6 hours. The analysis of onset time within 3 hours showed that the C index was 0.90 (95% CI 0.81-0.99) in the training set. For the onset time within 4.5 hours, there were 38 models in the training set and 13 models in the validation set. The C index for 4.5 hours was 0.80 (95% CI 0.77-0.83) in the training set and 0.79 (95% CI 0.71-0.86) in the validation set. Regarding the time within 6 hours, there were 2 models in the training set and 1 in the validation set. The C index for 6 hours was 0.76 (95% CI 0.71-0.81) in the training set and 0.73 (95% CI 0.62-0.84) in the validation set. The C index in the training set is depicted in Figure 5, and the C index of the validation set is presented in Figure 6.

    ‎
    Figure 5. The C index of the training set for the value of applying machine learning in predicting the time of symptom onset in stroke patients. Some studies used multiple different types of machine learning models; therefore, some studies are presented multiple times in the figure [17, 18, 24-34].
    ‎
    Figure 6. The C index of the verification set for the value of applying machine learning in predicting the time of symptom onset in stroke patients. Some studies used multiple different types of machine learning models; therefore, some studies are presented multiple times in the figure [17, 18, 24-34].

    Principal Findings

    Our analysis suggests that machine learning showed ideal performance in predicting and identifying the time of symptom onset in stroke patients. Accurate prediction of stroke onset time is extremely important for identifying the time window of thrombolytic therapy for stroke patients [18]. However, stroke patients usually do not know the specific time of symptom onset. Our research results can provide a reference for the clinical prediction of stroke onset time. We included 10 main types of models, including LR, support vector machine (SVM), artificial neural network (ANN), boosting, deep learning (DL), random forest (RF), decision tree (DT), generalized linear model (GLM), k-nearest neighbor (KNN), and naive Bayes (NB). Generally, the models showed ideal predictive performance, with an overall C index of 0.800 (95% CI 0.773-0.826) in the training set and 0.781 (95% CI 0.709-0.852) in the validation set. The overall C index of the LR model for predicting the stroke onset time was relatively high, at 0.807 (95% CI 0.741-0.872) in the training set and 0.807 (95% CI 0.691-0.923) in the validation set, despite high heterogeneity across the studies. These results indicate that the performance of machine learning in identifying stroke onset time is excellent, and it can be used as a potential tool to determine when a stroke begins. Furthermore, the sensitivity and specificity were 0.76 (95% CI 0.73-0.80) and 0.79 (95% CI 0.74-0.82) in the training set, and 0.81 (95% CI 0.68-0.90) and 0.83 (95% CI 0.73-0.89) in the validation set.

    Machine learning models have been widely used in the health care industry and show favorable performance in the stratification of patients. Numerous studies have been carried out on the use of machine learning models to predict the risk of stroke, the prognosis of acute stroke, and motor function recovery in stroke patients after treatment, as well as for the early identification of ischemic stroke patients at a high risk of recurrence [21,35-37]. Despite many publications on outcomes and treatment response in stroke patients, the use of MRI to determine the time of symptom onset is rarely researched. Therefore, this study explored the value of applying machine learning in predicting stroke onset time.

    In our study, the variable included in the model for predicting the time of stroke onset was MRI, specifically diffusion-weighted imaging (DWI) and fluid-attenuated inversion recovery (FLAIR). A mismatch between DWI and FLAIR has been adopted to identify stroke patients who are likely to benefit from thrombolysis therapy [38,39]. Previous studies have reported that DWI-FLAIR mismatch could achieve a specificity of 0.60 to 0.80 and a moderate sensitivity of 0.5 to 0.6 [40-42]. Machine learning is a branch of artificial intelligence that can use a variety of image features, including those invisible to humans, with a favorable predictive accuracy [16,18,43].

    In addition, our research shows that the LR model had the best predictive performance among the included models. Moreover, some institutions that specialize in detecting tumors have constructed LR-based machine learning models, which have achieved good results [44-46]. Based on existing research and our results, we vigorously recommend promoting the development of nomograms based on LR models for research on multiple diseases.

    This study is the first meta-analysis to investigate the value of applying machine learning in predicting the time of symptom onset in stroke patients, comprehensively promoting the development of accurate treatment of stroke. At the same time, it will be useful for identifying stroke onset time once its feasibility is confirmed. However, there are still some limitations to our review. First, the included studies are mainly single-center, and it is difficult to eliminate the impact of socioeconomic background. Second, although we have conducted a comprehensive search, the number of included studies is still small. Third, the diversity of the included models resulted in heterogeneity. Since it is difficult to avoid heterogeneity in machine learning models, more large-scale, multicenter studies are desired to reduce heterogeneity.

    Conclusion

    Machine learning has ideal performance in identifying stroke onset time, and it can be used as a potential tool to determine when a stroke begins. Nonetheless, its predictive accuracy still needs to be improved. As a result, minimally invasive or noninvasive, easily collected, and efficient predictors should be considered in future research to improve the predictive accuracy of machine learning. Moreover, it is necessary to explore reasonable image-segmentation and texture-extraction methods in terms of radiomics. In addition, comprehensive research on the popularization of machine learning and the value of its application is desired. Furthermore, the promotion and application of machine learning should take into account differences in ethnic backgrounds.

    Data Availability

    The data sets generated during and/or analyzed during this study are available from the corresponding author on reasonable request.

    Authors' Contributions

    All authors contributed to the study conception and design. JF, QZ, FW, JP and ZL contributed equally to this work. JF and QZ contributed to conceptualization, methodology, software, writing the original draft, data curation, and visualization. FW and JP contributed to investigation, writing the original draft, and reviewing and editing. ZL contributed to methodology, software, and writing the original draft. ZC contributed to conceptualization, supervision, and project administration. All authors read and approved the final manuscript.

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    Literature search strategy.

    DOCX File , 22 KB
    Multimedia Appendix 2

    PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) checklist.

    DOCX File , 30 KB
    1. GBD 2016 Neurology Collaborators. Global, regional, and national burden of neurological disorders, 1990-2016: a systematic analysis for the Global Burden of Disease Study 2016. Lancet Neurol. May 2019;18(5):459-480. [FREE Full text] [CrossRef] [Medline]
    2. Chamorro; Lo EH, Renú A, van Leyen K, Lyden PD. The future of neuroprotection in stroke. J Neurol Neurosurg Psychiatry. Feb 2021;92(2):129-135. [CrossRef] [Medline]
    3. Xiong Y, Wakhloo AK, Fisher M. Advances in acute ischemic stroke therapy. Circ Res. Apr 15, 2022;130(8):1230-1251. [CrossRef] [Medline]
    4. Saini V, Guada L, Yavagal DR. Global epidemiology of stroke and access to acute ischemic stroke interventions. Neurology. Nov 16, 2021;97(20 Suppl 2):S6-S16. [CrossRef] [Medline]
    5. Gariel F, Lapergue B, Bourcier R, Berge J, Barreau X, Mazighi M, et al. ASTER Trial Investigators. Mechanical thrombectomy outcomes with or without intravenous thrombolysis. Stroke. Oct 2018;49(10):2383-2390. [CrossRef] [Medline]
    6. Phipps MS, Cronin CA. Management of acute ischemic stroke. BMJ. Feb 13, 2020;368:l6983. [CrossRef] [Medline]
    7. Simonsen CZ, Leslie-Mazwi TM, Thomalla G. Which imaging approach should be used for stroke of unknown time of onset? Stroke. Jan 2021;52(1):373-380. [CrossRef] [Medline]
    8. Mistry EA, Mistry AM, Nakawah MO, Chitale RV, James RF, Volpi JJ, et al. Mechanical thrombectomy outcomes with and without intravenous thrombolysis in stroke patients: A meta-analysis. Stroke. Sep 2017;48(9):2450-2456. [CrossRef] [Medline]
    9. Kim Y, Kim BJ, Kwon SU, Kim JS, Kang D. Unclear-onset stroke: Daytime-unwitnessed stroke vs. wake-up stroke. Int J Stroke. Feb 2016;11(2):212-220. [CrossRef] [Medline]
    10. Campbell BCV, Ma H, Ringleb PA, Parsons MW, Churilov L, Bendszus M, et al. EXTEND‚ ECASS-4‚EPITHET Investigators. Extending thrombolysis to 4·5-9 h and wake-up stroke using perfusion imaging: a systematic review and meta-analysis of individual patient data. Lancet. Jul 13, 2019;394(10193):139-147. [CrossRef] [Medline]
    11. McGarry B, Kauppinen R. Timing the Ischemic Stroke by Multiparametric Quantitative Magnetic Resonance Imaging. Brisbane, Australia. Exon Publications; 2021.
    12. Rogers HJ, McGarry BL, Knight MJ, Jokivarsi KT, Gröhn OHJ, Kauppinen RA. Timing the ischaemic stroke by 1H-MRI: improved accuracy using absolute relaxation times over signal intensities. Neuroreport. Oct 22, 2014;25(15):1180-1185. [CrossRef] [Medline]
    13. Li H, Liu J, Dong Z, Chen X, Zhou C, Huang C, et al. Identification of high-risk intracranial plaques with 3D high-resolution magnetic resonance imaging-based radiomics and machine learning. J Neurol. Dec 2022;269(12):6494-6503. [CrossRef] [Medline]
    14. Hoyer UCI, Lennartz S, Abdullayev N, Fichter F, Jünger ST, Goertz L, et al. Imaging of the extracranial internal carotid artery in acute ischemic stroke: assessment of stenosis, plaques, and image quality using relaxation-enhanced angiography without contrast and triggering (REACT). Quant Imaging Med Surg. Jul 2022;12(7):3640-3654. [FREE Full text] [CrossRef] [Medline]
    15. Jordan MI, Mitchell TM. Machine learning: Trends, perspectives, and prospects. Science. Jul 17, 2015;349(6245):255-260. [CrossRef] [Medline]
    16. Zhang Y, Zhu D, Li T, Wang X, Zhao L, Yang X, et al. Detection of acute ischemic stroke and backtracking stroke onset time via machine learning analysis of metabolomics. Biomed Pharmacother. Nov 2022;155:113641. [FREE Full text] [CrossRef] [Medline]
    17. Zhang Y, Liu A, Man F, Zhang Y, Li C, Liu Y, et al. MRI radiomic features-based machine learning approach to classify ischemic stroke onset time. J Neurol. Jan 2022;269(1):350-360. [CrossRef] [Medline]
    18. Lee H, Lee E, Ham S, Lee H, Lee JS, Kwon SU, et al. Machine learning approach to identify stroke within 4.5 Hours. Stroke. Mar 2020;51(3):860-866. [CrossRef] [Medline]
    19. Patel UK, Anwar A, Saleem S, Malik P, Rasul B, Patel K, et al. Artificial intelligence as an emerging technology in the current care of neurological disorders. J Neurol. May 2021;268(5):1623-1642. [CrossRef] [Medline]
    20. Jiang F, Jiang Y, Zhi H, Dong Y, Li H, Ma S, et al. Artificial intelligence in healthcare: past, present and future. Stroke Vasc Neurol. Dec 2017;2(4):230-243. [FREE Full text] [CrossRef] [Medline]
    21. Heo J, Yoon JG, Park H, Kim YD, Nam HS, Heo JH. Machine learning-based model for prediction of outcomes in acute stroke. Stroke. May 2019;50(5):1263-1265. [CrossRef] [Medline]
    22. Moons KGM, Wolff RF, Riley RD, Whiting PF, Westwood M, Collins GS, et al. PROBAST: A tool to assess risk of bias and applicability of prediction model studies: explanation and elaboration. Ann Intern Med. Jan 01, 2019;170(1):W1-W33. [FREE Full text] [CrossRef] [Medline]
    23. Debray TP, Damen JA, Riley RD, Snell K, Reitsma JB, Hooft L, et al. A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes. Stat Methods Med Res. Sep 2019;28(9):2768-2786. [FREE Full text] [CrossRef] [Medline]
    24. Wouters A, Dupont P, Norrving B, Laage R, Thomalla G, Albers GW, et al. Prediction of stroke onset is improved by relative fluid-attenuated inversion recovery and perfusion imaging compared to the visual diffusion-weighted imaging/fluid-attenuated inversion recovery mismatch. Stroke. Oct 2016;47(10):2559-2564. [CrossRef] [Medline]
    25. Zhu H, Jiang L, Zhang H, Luo L, Chen Y, Chen Y. An automatic machine learning approach for ischemic stroke onset time identification based on DWI and FLAIR imaging. Neuroimage Clin. 2021;31:102744. [FREE Full text] [CrossRef] [Medline]
    26. Zhang H, Polson JS, Nael K, Salamon N, Yoo B, El-Saden S, et al. Intra-domain task-adaptive transfer learning to determine acute ischemic stroke onset time. Comput Med Imaging Graph. Jun 2021;90:101926. [CrossRef] [Medline]
    27. Yao X, Mao L, Lv S, Ren Z, Li W, Ren K. CT radiomics features as a diagnostic tool for classifying basal ganglia infarction onset time. J Neurol Sci. May 15, 2020;412:116730. [CrossRef] [Medline]
    28. Wouters A, Dupont P, Christensen S, Norrving B, Laage R, Thomalla G, et al. Multimodal magnetic resonance imaging to identify stroke onset within 6 h in patients with large vessel occlusions. Eur Stroke J. Jun 2018;3(2):185-192. [FREE Full text] [CrossRef] [Medline]
    29. Wouters A, Cheng B, Christensen S, Dupont P, Robben D, Norrving B, et al. Automated DWI analysis can identify patients within the thrombolysis time window of 4.5 hours. Neurology. May 01, 2018;90(18):e1570-e1577. [CrossRef] [Medline]
    30. Wen X, Shu Z, Li Y, Hu X, Gong X. Developing a model for estimating infarction onset time based on computed tomography radiomics in patients with acute middle cerebral artery occlusion. BMC Med Imaging. Oct 11, 2021;21(1):147. [FREE Full text] [CrossRef] [Medline]
    31. Polson J, Zhang H, Nael K, Salamon N, Yoo B, Kim N, et al. A semi-supervised learning framework to leverage proxy information for stroke MRI analysis. Annu Int Conf IEEE Eng Med Biol Soc. Nov 2021;2021:2258-2261. [CrossRef] [Medline]
    32. Jiang L, Wang S, Ai Z, Shen T, Zhang H, Duan S, et al. Development and external validation of a stability machine learning model to identify wake-up stroke onset time from MRI. Eur Radiol. Jun 2022;32(6):3661-3669. [CrossRef] [Medline]
    33. Ho KC, Speier W, Zhang H, Scalzo F, El-Saden S, Arnold CW. A machine learning approach for classifying ischemic stroke onset time from imaging. IEEE Trans Med Imaging. Jul 2019;38(7):1666-1676. [FREE Full text] [CrossRef] [Medline]
    34. Ho KC, Speier W, El-Saden S, Arnold CW. Classifying acute ischemic stroke onset time using deep imaging features. AMIA Annu Symp Proc. 2017;2017:892-901. [FREE Full text] [Medline]
    35. Campagnini S, Arienti C, Patrini M, Liuzzi P, Mannini A, Carrozza MC. Machine learning methods for functional recovery prediction and prognosis in post-stroke rehabilitation: a systematic review. J Neuroeng Rehabil. Jun 03, 2022;19(1):54. [FREE Full text] [CrossRef] [Medline]
    36. Vodencarevic A, Weingärtner M, Caro JJ, Ukalovic D, Zimmermann-Rittereiser M, Schwab S, et al. Prediction of recurrent ischemic stroke using registry data and machine learning methods: The Erlangen Stroke Registry. Stroke. Jul 2022;53(7):2299-2306. [CrossRef] [Medline]
    37. Chun M, Clarke R, Cairns BJ, Clifton D, Bennett D, Chen Y, et al. China Kadoorie Biobank Collaborative Group. Stroke risk prediction using machine learning: a prospective cohort study of 0.5 million Chinese adults. J Am Med Inform Assoc. Jul 30, 2021;28(8):1719-1727. [FREE Full text] [CrossRef] [Medline]
    38. Koga M, Yamamoto H, Inoue M, Asakura K, Aoki J, Hamasaki T, et al. THAWS Trial Investigators. Thrombolysis with alteplase at 0.6 Mg/kg for stroke with unknown time of onset: A randomized controlled trial. Stroke. May 2020;51(5):1530-1538. [FREE Full text] [CrossRef] [Medline]
    39. Scheldeman L, Wouters A, Boutitie F, Dupont P, Christensen S, Cheng B, et al. WAKE-UP Investigators. Different mismatch concepts for magnetic resonance imaging-guided thrombolysis in unknown onset stroke. Ann Neurol. Jun 2020;87(6):931-938. [CrossRef] [Medline]
    40. Ebinger M, Galinovic I, Rozanski M, Brunecker P, Endres M, Fiebach JB. Fluid-attenuated inversion recovery evolution within 12 hours from stroke onset: a reliable tissue clock? Stroke. Feb 2010;41(2):250-255. [CrossRef] [Medline]
    41. Thomalla G, Cheng B, Ebinger M, Hao Q, Tourdias T, Wu O, et al. STIRVISTA Imaging Investigators. DWI-FLAIR mismatch for the identification of patients with acute ischaemic stroke within 4·5 h of symptom onset (PRE-FLAIR): a multicentre observational study. Lancet Neurol. Nov 2011;10(11):978-986. [CrossRef] [Medline]
    42. Emeriau S, Serre I, Toubas O, Pombourcq F, Oppenheim C, Pierot L. Stroke. Jun 2013;44(6):1647-1651. [CrossRef] [Medline]
    43. Lee E, Kim Y, Kim N, Kang D. Deep into the brain: Artificial intelligence in stroke imaging. J Stroke. Sep 2017;19(3):277-285. [FREE Full text] [CrossRef] [Medline]
    44. Hiremath A, Shiradkar R, Fu P, Mahran A, Rastinehad AR, Tewari A, et al. An integrated nomogram combining deep learning, Prostate Imaging-Reporting and Data System (PI-RADS) scoring, and clinical variables for identification of clinically significant prostate cancer on biparametric MRI: a retrospective multicentre study. Lancet Digit Health. Jul 2021;3(7):e445-e454. [CrossRef] [Medline]
    45. Liu A, Wang Z, Yang Y, Wang J, Dai X, Wang L, et al. Preoperative diagnosis of malignant pulmonary nodules in lung cancer screening with a radiomics nomogram. Cancer Commun (Lond). Jan 2020;40(1):16-24. [FREE Full text] [CrossRef] [Medline]
    46. Huang J, Xie X, Wu H, Zhang X, Zheng Y, Xie X, et al. Development and validation of a combined nomogram model based on deep learning contrast-enhanced ultrasound and clinical factors to predict preoperative aggressiveness in pancreatic neuroendocrine neoplasms. Eur Radiol. Nov 2022;32(11):7965-7975. [CrossRef] [Medline]

    ‎
    ANN: artificial neural network
    CT: computed tomography
    DL: deep learning
    DT: decision tree
    DWI: fluid-attenuated inversion recovery
    FLAIR: diffusion-weighted imaging
    GLM: generalized linear model
    KNN: k-nearest neighbor
    LR: logistic regression
    MRI: magnetic resonance imaging
    NB: naive Bayes
    PRISMA: Preferred Reporting Items for Systematic reviews and Meta-Analyses
    PROBAST: Prediction Model Risk of Bias Assessment Tool
    RF: random forest
    SVM: support vector machine

    Edited by T Leung, A Mavragani; submitted 08.12.22; peer-reviewed by Y Zhang, S Masiero; comments to author 06.03.23; revised version received 02.04.23; accepted 14.09.23; published 12.10.23.

    Copyright

    ©Jing Feng, Qizhi Zhang, Feng Wu, Jinxiang Peng, Ziwei Li, Zhuang Chen. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 12.10.2023.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.