This paper is in the following e-collection/theme issue:

Extended Reality, Virtual Reality and Virtual Worlds (560) Dementia and Cognitive Decline (481) New Methods (944) Games for Cognitive Assessment (61) Games and Gamification for Health (650) Neurotech Innovations, Diagnostic Tools and Techniques (33) Digital Biomarkers and Digital Phenotyping (191) Clinical Integration of XR and Spatial Computing (32) XR for Assessment and Diagnosis (9) Machine Learning (1465) Artificial Intelligence (1371) Decision Support for Health Professionals (1185) Computer-Aided Diagnosis (128)

Published on in Vol 26 (2024)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/54538, first published .
Integrating Biomarkers From Virtual Reality and Magnetic Resonance Imaging for the Early Detection of Mild Cognitive Impairment Using a Multimodal Learning Approach: Validation Study

Integrating Biomarkers From Virtual Reality and Magnetic Resonance Imaging for the Early Detection of Mild Cognitive Impairment Using a Multimodal Learning Approach: Validation Study

Integrating Biomarkers From Virtual Reality and Magnetic Resonance Imaging for the Early Detection of Mild Cognitive Impairment Using a Multimodal Learning Approach: Validation Study

Authors of this article:

Bogyeom Park1 Author Orcid Image ;   Yuwon Kim1 Author Orcid Image ;   Jinseok Park2 Author Orcid Image ;   Hojin Choi2 Author Orcid Image ;   Seong-Eun Kim1 Author Orcid Image ;   Hokyoung Ryu3 Author Orcid Image ;   Kyoungwon Seo1 Author Orcid Image
Article Authors Cited by (4) Tweetations (6) Metrics

    Original Paper

    1Department of Applied Artificial Intelligence, Seoul National University of Science and Technology, Seoul, Republic of Korea

    2Department of Neurology, College of Medicine, Hanyang University, Seoul, Republic of Korea

    3Graduate School of Technology and Innovation Management, Hanyang University, Seoul, Republic of Korea

    *these authors contributed equally

    Corresponding Author:

    Kyoungwon Seo, PhD

    Department of Applied Artificial Intelligence

    Seoul National University of Science and Technology

    Sangsang Hall, 4th Fl

    Gongneung-ro, Gongneung-dong, Nowon-gu

    Seoul, 01811

    Republic of Korea

    Phone: 82 010 5668 8660

    Email: kwseo@seoultech.ac.kr


    Background: Early detection of mild cognitive impairment (MCI), a transitional stage between normal aging and Alzheimer disease, is crucial for preventing the progression of dementia. Virtual reality (VR) biomarkers have proven to be effective in capturing behaviors associated with subtle deficits in instrumental activities of daily living, such as challenges in using a food-ordering kiosk, for early detection of MCI. On the other hand, magnetic resonance imaging (MRI) biomarkers have demonstrated their efficacy in quantifying observable structural brain changes that can aid in early MCI detection. Nevertheless, the relationship between VR-derived and MRI biomarkers remains an open question. In this context, we explored the integration of VR-derived and MRI biomarkers to enhance early MCI detection through a multimodal learning approach.

    Objective: We aimed to evaluate and compare the efficacy of VR-derived and MRI biomarkers in the classification of MCI while also examining the strengths and weaknesses of each approach. Furthermore, we focused on improving early MCI detection by leveraging multimodal learning to integrate VR-derived and MRI biomarkers.

    Methods: The study encompassed a total of 54 participants, comprising 22 (41%) healthy controls and 32 (59%) patients with MCI. Participants completed a virtual kiosk test to collect 4 VR-derived biomarkers (hand movement speed, scanpath length, time to completion, and the number of errors), and T1-weighted MRI scans were performed to collect 22 MRI biomarkers from both hemispheres. Analyses of covariance were used to compare these biomarkers between healthy controls and patients with MCI, with age considered as a covariate. Subsequently, the biomarkers that exhibited significant differences between the 2 groups were used to train and validate a multimodal learning model aimed at early screening for patients with MCI among healthy controls.

    Results: The support vector machine (SVM) using only VR-derived biomarkers achieved a sensitivity of 87.5% and specificity of 90%, whereas the MRI biomarkers showed a sensitivity of 90.9% and specificity of 71.4%. Moreover, a correlation analysis revealed a significant association between MRI-observed brain atrophy and impaired performance in instrumental activities of daily living in the VR environment. Notably, the integration of both VR-derived and MRI biomarkers into a multimodal SVM model yielded superior results compared to unimodal SVM models, achieving higher accuracy (94.4%), sensitivity (100%), specificity (90.9%), precision (87.5%), and F1-score (93.3%).

    Conclusions: The results indicate that VR-derived biomarkers, characterized by their high specificity, can be valuable as a robust, early screening tool for MCI in a broader older adult population. On the other hand, MRI biomarkers, known for their high sensitivity, excel at confirming the presence of MCI. Moreover, the multimodal learning approach introduced in our study provides valuable insights into the improvement of early MCI detection by integrating a diverse set of biomarkers.

    J Med Internet Res 2024;26:e54538

    doi:10.2196/54538

    Keywords

    magnetic resonance imaging; MRI; virtual reality; VR; early detection; mild cognitive impairment; multimodal learning; hand movement; eye movement 


    Background

    Mild cognitive impairment (MCI) represents a transitional stage in cognitive decline, positioned between normal aging and Alzheimer disease (AD). This stage is marked by challenges such as memory loss and difficulties in executing complex daily activities [1-3]. Once MCI deteriorates into AD, cognitive function cannot be restored to normal levels [4,5], leading to a significant reduction in the ability of patients with AD to independently perform daily activities [6]. This makes MCI a critical intervention point to potentially slow down cognitive decline. Therefore, the early detection of MCI plays an essential role, not only in preventing its progression to AD but also in allowing for timely interventions aimed at restoring cognitive function to levels associated with normal aging [7].

    Conventionally, biomarkers such as neuropsychological tests and magnetic resonance imaging (MRI) have been deployed for early detection of MCI, facilitating the evaluation of cognitive functions and the tracking of brain changes [8]. For instance, neuropsychological tests aim to quantitatively assess multiple cognitive domains, including memory, to identify patients with MCI [9]. However, despite their widespread clinical use, these tests face challenges related to reproducibility, largely owing to elements such as response bias and the ceiling effect [10]. In contrast, MRI scans identify MCI by examining structural brain changes, particularly in memory-associated regions such as the hippocampus and entorhinal cortex [11,12]. However, the use of MRI scans is restricted due to their limited feasibility for regular and repeated assessments, given their lengthy procedures and high costs [13,14]. Thus, there is an increasing need for novel biomarkers that can effectively and economically detect MCI by leveraging reproducible behavioral characteristics observed in daily activities [15-18].

    In recent studies, virtual reality (VR) technology has been used to collect behavioral data related to instrumental activities of daily living (IADLs), which are then analyzed using machine learning techniques to enhance the early detection of MCI [17-20]. As an example, Kim et al [21] devised the virtual kiosk test, wherein participants interact within a virtual environment to order food using a kiosk. Throughout this test, behavioral data related to hand movements, eye movements, and performance were collected. Using these behavioral data, the machine learning model successfully distinguished patients with MCI from healthy controls with an accuracy rate of 80.2%. Although VR-derived biomarkers have shown promise in early MCI detection, there is still an ongoing question regarding the interpretation of these biomarkers in relation to measurable structural brain changes observed through MRI, which is of critical clinical significance.

    The integration of behavioral characteristics measured through VR-derived biomarkers with the brain’s structural characteristics obtained from MRI biomarkers is gaining increasing attention. For example, Castegnaro et al [22] examined object location memory using VR performance data and found that individuals with damage to the hippocampus and entorhinal cortex demonstrated poorer performance, suggesting a correlation between impaired performance and damage to these brain regions. Similarly, Howett et al [23] conducted a VR navigation task and observed that participants with entorhinal cortex damage exhibited inferior performance in the task. Although these studies established significant positive correlations between performance assessed using VR-derived biomarkers and atrophy in the hippocampus and entorhinal cortex identified using MRI biomarkers, the results were confined to correlation analysis. To address this limitation, Cavedoni et al [19] emphasized the importance of further exploring the multimodal integration of VR-derived and MRI biomarkers, as it would not only establish the clinical validity of VR but also provide valuable behavioral and structural information about MCI. Consistent with these objectives, this study aimed to leverage a multimodal learning approach by integrating VR-derived and MRI biomarkers to enhance clinical feasibility and enable early and accurate detection of MCI.

    Objectives

    This study had 2 objectives. First, the study compared the MCI classification performance of VR-derived and MRI biomarkers with that of neuropsychological tests, which are considered the gold standard for MCI classification. This comparative analysis aimed to provide a deeper understanding of the advantages and limitations of each approach, namely VR, MRI, and neuropsychological tests. Second, the study introduced and validated a multimodal learning model that effectively improves the early detection of MCI by integrating the unique attributes of VR-derived and MRI biomarkers. This integrated approach harnesses the strengths of both modalities for more accurate and reliable MCI detection. The findings of this study proposed a novel clinical application approach that incorporates VR-derived and MRI biomarkers sequentially, offering a promising framework for clinicians to enhance their diagnostic capabilities in evaluating MCI.


    Ethical Considerations

    This study received ethics approval from the institutional review board of Hanyang University Hospital, Republic of Korea, in accordance with the principles outlined in the Declaration of Helsinki (HYUH-2021-08-020-004). Before conducting the experiment, we ensured that all participants received a comprehensive explanation of the study, and we obtained written informed consent from each individual. In addition, all participants provided consent for the use of their de-identified data for scientific research purposes, and only such data were used in the subsequent analysis. As a token of appreciation for their participation, participants were offered compensation in the form of detailed reports summarizing the results from neuropsychological tests, VR tasks, and MRI.

    Participants

    This study involved a total of 54 participants recruited from Hanyang University Hospitals in Seoul and Guri between January 2022 and July 2023. The 54 participants included 27 (50%) individuals from Seoul (n=11, 41% healthy controls and n=16, 59% patients with MCI) and 27 (50%) individuals from Guri (n=11, 41% healthy controls and n=16, 59% patients with MCI). Participants were randomly recruited from both a volunteer pool and outpatient clinics at the hospitals. The 2 neurologists reached a consensus on the diagnosis of MCI based on the cutoff score of the Seoul Neuropsychological Screening Battery–Core (SNSB-C), a gold-standard neuropsychological assessment tool specifically standardized for the Korean population [24]. The study only included participants who were capable of interacting with the VR environment and who possessed normal sensory perception, specifically in response to visual and auditory stimuli. We excluded individuals with a history of dementia, neurodegenerative disorders, or psychiatric conditions or those who had undergone brain surgeries. It is important to note that one volunteer (1/27, 4%) from Guri was excluded from recruitment due to a history of brain surgery for hydrocephalus. None of the patients with MCI included in the study were diagnosed with dementia.

    Neuropsychological Tests

    This study used the SNSB-C, a neuropsychological test specifically designed and standardized for the Korean population. The SNSB-C serves as a reliable and clinically validated alternative to the Mini‐Mental State Examination for evaluating cognitive functions [24,25]. It assesses five distinct cognitive domains through separate assessments, including the following: (1) attention—measured using the Digit Span Test–Backward, (2) language function—evaluated using the Short Form of the Korean-Boston Naming Test, (3) visuospatial function—assessed using the Rey Complex Figure Test (RCFT), (4) memory—measured using the Seoul Verbal Learning Test–Elderly’s Version–Delayed Recall, and (5) frontal and executive function—evaluated using the Digit Symbol Coding. A professional psychological evaluator with 15 years of experience conducted all the SNSB-C evaluations in this study.

    VR-Derived Biomarkers

    This study used the virtual kiosk test, previously developed in our research [26], as a source of VR-derived biomarkers. VR-derived biomarkers consist of features used to assess cognitive impairment by analyzing behavioral data, including hand and eye movements, collected from VR environments [17]. The virtual kiosk test aims to detect early indications of MCI by assessing behavioral data collected while participants undertake a cognitively complex IADL task, specifically ordering menu items at a virtual kiosk. The experimental setup, as shown in Figure 1, was arranged in a room containing a laptop equipped with an Intel i7-12700H processor, 16 GB of RAM, and an NVIDIA GeForce RTX 3080 graphics card to execute Unity and VIVEPORT software, which is necessary to run the VR program. To facilitate a fully immersive VR experience, participants wore a head-mounted display with integrated eye-tracking capabilities (HTC VIVE Pro Eye) and used a hand controller in their dominant hand to select and order menu items from the virtual kiosk. Overall, 2 base stations tracked participants’ movements during the test. The behavioral data recorded during this task, encompassing hand and eye movements and performance data, can provide insightful VR-derived biomarkers. For safety, participants remained seated throughout the test.

    The virtual kiosk test, depicted in Figure 2, followed six steps, not including “Start” and “End” stages: (1) selecting a place to eat, (2) choosing a burger item, (3) selecting a side item, (4) choosing a drink item, (5) selecting a payment method, and (6) entering a 4-digit payment password. Before the test, participants received the following instructions verbally: “The place to eat is a restaurant. Please use the kiosk to order a shrimp burger, cheese sticks, and a Coca-Cola. Use a credit card as the payment method, and the card payment password is 6289.”

    Then, the behavioral data collected during the test were converted into 4 VR-derived biomarkers: hand movement speed, scanpath length, time to completion, and the number of errors (Figure 3). Hand movement speed, the first VR-derived biomarker, was determined by dividing the total distance of hand movements by the total test time, which correlates with cognitive abilities such as recognition and processing speed [21,27-29]. Scanpath length, the second VR-derived biomarker, reflects the distance traveled by participants’ gaze during the test, indicating overall cognitive ability [30], information processing efficiency [31], and comprehension level [32]. Time to completion [33], the third VR-derived biomarker, represents the duration required for participants to complete all 6 steps of the test. The number of errors [26], the final VR-derived biomarker, records the total number of incorrect actions during the test, such as incorrect choices for place to eat, burger, drink, side item, or payment method or incorrect password entry.

    ‎
    Figure 1. Experimental setup for the virtual kiosk test. The virtual kiosk test is operated via a laptop. Participants sit and wear a head-mounted display and interact with the virtual environment using a hand controller. Their hand movements, eye movements, and performance data are tracked via base stations.
    ‎
    Figure 2. The 6 sequential steps of the virtual kiosk test. In step 1, participants selected a place to eat. In step 2, they chose a burger item. Step 3 involved selecting a side item, and in step 4, participants chose a drink item. Step 5 required them to select a payment method, and finally, in step 6, they entered the payment password.
    ‎
    Figure 3. Extraction of 4 virtual reality (VR)–derived biomarkers from behavioral data in the virtual kiosk test. Hand movement speed is calculated using the hand movement data collected from the virtual kiosk test. Scanpath length is derived from the eye movement data. The time to completion and the number of errors are calculated based on the performance data.

    MRI Biomarkers

    MRI scans were performed at Hanyang University Hospital in both Guri and Seoul locations using a Philips Ingenia CX 3T scanner. The scans used the 3D T1-weighted magnetization-prepared rapid gradient echo technique, with specific protocols set for each hospital. The hospital at Guri adhered to the following parameters: echo time/repetition time=2.9 ms/6.3 ms, flip angle=9°, field of view=256×256 mm, with 211 slices and voxel size of 1×1×1 mm3. The hospital at Seoul followed a slightly different protocol: echo time/repetition time=4.1 ms/6.9 ms, flip angle=8°, field of view=300×299 mm, with 170 slices and voxel size of 0.8×0.8×1 mm3.

    We used AQUA 3.0 software (Neurophet Inc) to process the acquired T1-weighted MRIs, a method that aids in precise identification and outlining of the region of interest in the brain as depicted in Figure 4 [34,35]. The AQUA 3.0 software uses the Split-Attention U-Net deep learning architecture, which combines elements from ResNeSt and U-Net++. The architecture of Split-Attention U-Net demonstrates robustness to neuroanatomical variability through its encoder and decoder structure, skip pathways, and split-attention module. It also improves segmentation accuracy, particularly for small subcortical regions, by using EvoNorm-based convolution layers and 3D ResNeSt blocks. This methodological approach provides valuable information about brain volume from MRIs, which can be used as MRI biomarkers for neuroimaging analysis. Furthermore, the software accounts for cerebral hemispheric asymmetry by separately analyzing the left and right hemispheres, thus enabling a more holistic evaluation.

    In this study, we selected specific MRI biomarkers associated with brain regions known to exhibit early atrophy in AD and play crucial roles in various cognitive functions [12,36]. The chosen MRI biomarkers, as described in Table 1, encompass cerebral white matter [37]; cerebral gray matter [38]; ventricles [39]; amygdala [40]; hippocampus [41,42]; entorhinal cortex [41,43]; parahippocampal gyrus [43]; fusiform gyrus [44]; and the superior, middle, and inferior temporal gyri [45]. These MRI biomarkers were derived from both hemispheres, resulting in a total of 22 MRI biomarkers used in this study. Furthermore, we incorporated intracranial volume (ICV), a measure encapsulating total white matter, gray matter, and cerebrospinal fluid [46], to account for overall brain size. Then, we normalized all the MRI biomarkers to the ICV to mitigate differences in brain volume due to factors such as age, sex, and head circumference [14].

    ‎
    Figure 4. Extraction of 22 magnetic resonance imaging (MRI) biomarkers from both hemispheres of the brain using the Split-Attention U-Net architecture. Following multilabel segmentation of the region of interest in the brain, each brain region’s volume is quantified as an MRI biomarker. Each hemisphere has 11 biomarkers including the cerebral white matter; cerebral gray matter; ventricles; amygdala; hippocampus; entorhinal cortex; parahippocampal gyrus; fusiform gyrus; and superior, middle, and inferior temporal gyrus.
    Table 1. The 11 magnetic resonance imaging biomarkers for each hemisphere (ie, a total of 22 magnetic resonance imaging biomarkers from both hemispheres) and their descriptions.
    BiomarkersDescription
    Cerebral white matterCerebral white matter is associated with brain functions such as learning, memory, and transmitting neural information between brain regions.
    Cerebral gray matterCerebral gray matter is associated with cognitive functions, encompassing information processing, decision-making, and sensory perception.
    VentriclesVentricles are associated with cerebrospinal fluid circulation, maintaining optimal conditions for cognitive processes in the brain.
    AmygdalaThe amygdala is associated with emotional processing, and the modulation of memory consolidation.
    HippocampusThe hippocampus is associated with memory formation, spatial memory, and emotional processing.
    Entorhinal cortexThe entorhinal cortex is associated with functioning as a key network for spatial memory and the perception of time.
    Parahippocampal gyrusThe parahippocampal gyrus is associated with cognitive processes such as spatial and episodic memory.
    Fusiform gyrusThe fusiform gyrus is associated with high-level vision and multisensory perception such as object and word recognition.
    Superior temporal gyrusThe superior temporal gyrus is associated with the analysis of audio-visual social information such as verbal and nonverbal communication.
    Middle temporal gyrusThe middle temporal gyrus is associated with language-related tasks and integration of audio-visual emotional processing.
    Inferior temporal gyrusThe inferior temporal gyrus is associated with object recognition, such as recognizing objects based on prior experiences.

    Procedures

    Participants initially underwent an assessment using the SNSB-C, administered by a professional psychological evaluator with 15 years of experience. MCI diagnosis was subsequently performed by 2 experienced neurologists with 18 years and 22 years of clinical expertise, adhering to the criteria established by Albert et al [47], which considered the results of the SNSB-C. Then, participants completed both the virtual kiosk test and T1-weighted MRI scans in counterbalanced order. The virtual kiosk test was administered by the same neurologists responsible for the MCI diagnosis, whereas T1-weighted MRI scans were conducted by a radiologist with 16 years of experience. To ensure participants’ comfort and familiarity with the VR setup, they underwent 2 practice sessions before the virtual kiosk test, allowing them to get accustomed to the VR equipment and virtual environment. Throughout the experiment, measures were in place allowing participants to take breaks or halt the procedure if they experienced discomfort or dizziness due to either the VR environment or MRI scans. It is important to note that all participants completed the entire experiment successfully in an average time of 54.32 (SD 4.27) minutes, without requiring any breaks.

    Analysis

    We conducted statistical analysis using SPSS Statistics (version 27; IBM Corp). First, we compared of the study participants, encompassing aspects such as sex, age, and education level, were examined using a chi-square test and independent sample t tests to identify any statistically significant differences between the healthy control group and patients with MCI group, using a χ2 test for a categorical variable (ie, sex) and independent sample 2-tailed t tests for continuous variables. Second, we performed analyses of covariance (ANCOVA) to assess the differences in neuropsychological characteristics, VR-derived biomarkers, and MRI biomarkers between the healthy controls and patients with MCI, while considering age as a covariate in this analysis. This allowed us to discern distinct features within each biomarker. In addition, we examined the relationship between VR-derived and MRI biomarkers by conducting a Pearson correlation analysis. Notably, as our data exhibited a normal distribution and homoscedasticity, all analyses were conducted using parametric tests.

    Multimodal Learning

    This study used multimodal learning using Python 3, with the ultimate objective of amalgamating statistically significant VR-derived and MRI biomarkers to bolster early MCI detection. The support vector machine (SVM) algorithm, with a track record of extensive use and demonstrated effectiveness in analogous tasks [4,48-50], was chosen as the machine learning model for this study. Hyperparameters for the SVM algorithm were identified through grid search, resulting in the choice of the radial basis function kernel with the regularization parameter (cost) set at 1 and kernel coefficient (γ) set at 0.1. For external validation and to prevent overfitting, we used a train and test split with a ratio of 7:3, where 70% (38/54) of the participants were assigned to the train subcohort and the remaining 30% (16/54) of the participants were allocated to the test subcohort. During the biomarker integration process, the comparative performance of models using individual VR-derived and MRI biomarkers was examined, revealing the unique characteristics of each modality. The performance evaluation of these models was conducted using several metrics, including accuracy, sensitivity, specificity, precision, and F1-score. In addition, the area under the receiver operating characteristic curve (AUC) was used to evaluate the performance of the binary classifier (healthy controls vs patients with MCI) in our models.


    Demographic and Neuropsychological Characteristics

    The demographic characteristics of the study participants, encompassing aspects such as sex, age, and education level, were examined using a chi-square test and independent sample t tests to identify any statistically significant differences between the healthy control group and patients with MCI group. As per the results outlined in Table 2, no significant differences were found in terms of demographic characteristics between the 2 groups. Subsequently, neuropsychological characteristics were assessed using ANCOVA, with age as a covariate. The results showed a distinct contrast between the healthy control group and patients with MCI group. Patients with MCI exhibited significant impairment across all 5 evaluated cognitive functions—attention (F1,51=24.181; P<.001; ηp2=0.322), language function (F1,51=14.993; P<.001; ηp2=0.227), visuospatial function (F1,51=19.115; P<.001; ηp2=0.273), memory (F1,51=32.542; P<.001; ηp2=0.390), and frontal and executive function (F1,51=20.584; P<.001; ηp2=0.288)—when compared to the healthy control group, thus underscoring a marked cognitive decline in individuals with MCI.

    Table 2. Comparison of basic demographic characteristics and neuropsychological test results between healthy controls and patients with mild cognitive impairment (MCI).
    CharacteristicsGroupP value

    		Healthy controls (n=22)Patients with MCI (n=32)
    Basic demographic characteristics

    		Sex (female), n (%)14 (64)14 (44).15

    		Age (y), mean (SD)69.86 (6.72)73.47 (8.39).07

    		Educational level (years), mean (SD)12.09 (4.46)9.47 (5.12).06
    Neuropsychological tests, mean (SD)

    		DST–Ba (number of correct answers)4.27 (0.83)3.00 (0.88)<.001b

    		S-K–BNTc (number of correct answers)12.91 (1.44)10.78 (2)<.001b

    		RCFTd (score)33.34 (2.31)26.95 (5.93)<.001b

    		SVLT-E–DRe (number of correct answers)6.82 (2.63)2.50 (2.55)<.001b

    		DSCf (number of correct answers)60.95 (15.83)37.97 (17.24)<.001b

    aDST–B: Digit Span Test–Backward.

    bAnalyses of covariance, with age as a covariate.

    cS-K–BNT: Short Form of the Korean-Boston Naming Test.

    dRCFT: Rey Complex Figure Test.

    eSVLT-E–DR: Seoul Verbal Learning Test–Elderly’s version–Delayed Recall.

    fDSC: Digital Symbol Coding.

    Differences in VR-Derived Biomarkers Between Healthy Controls and Patients With MCI

    As illustrated in Table 3, significant differences were observed in VR-derived biomarkers between the healthy controls and patients with MCI when assessed through ANCOVA, with age as a covariate. Specifically, patients with MCI demonstrated slower hand movement speed (F1,51=13.426; P=.001; ηp2=0), longer scanpath length (F1,51=7.108; P=.01; ηp2=0.122), prolonged time to completion (F1,51=9.447; P=.003; ηp2=0.156), and a greater number of errors (F1,51=9.438; P=.003; ηp2=0.156) during the execution of the virtual kiosk test, as compared to healthy controls.

    Table 3. Comparison of virtual reality (VR)–derived biomarkers between healthy controls and patients with mild cognitive impairment (MCI).
    VR-derived biomarkersGroup, mean (SD)P valuea

    		Healthy controls (n=22)Patients with MCI (n=32)
    Hand movement features

    		Hand movement speed (m/s)0.23 (0.06)0.17 (0.06).001
    Eye movement feature

    		Scanpath length (m)23.66 (14.29)60.36 (54.58).01
    Performance features

    		Time to completion (s)39.48 (18.96)105.39 (86.35).003

    		Number of errors1.73 (1.61)4 (2.81).003

    aAnalyses of covariance, with age as a covariate.

    Differences in MRI Biomarkers Between Healthy Controls and Patients With MCI

    ANCOVA was performed with age as a covariate to scrutinize the discrepancies in ICV and the proportion of ICV between the healthy control group and the patients with MCI group. As presented in Table 4, although patients with MCI exhibited higher ICV compared to healthy controls, this difference was not statistically significant. While higher ICV is typically associated with age and sex effects [51-53], our statistical analysis did not identify significant variations in age and sex between healthy controls and patients with MCI. This suggests that factors other than age and sex may be contributing to the observed higher ICV among our patients with MCI. However, patients with MCI demonstrated notable atrophy in the proportion of ICV. Specifically, significant differences were discerned in the left entorhinal cortex (F1,51=7.821; P=.007; ηp2=0.133), right entorhinal cortex (F1,51=11.103; P=.002; ηp2=0.179), left hippocampus (F1,51=11.926; P=.001; ηp2=0.190), right hippocampus (F1,51=8.244; P=.006; ηp2=0.139), left amygdala (F1,51=7.979; P=.007; ηp2=0.135), and right amygdala (F1,51=6.618; P=.01; ηp2=0.115). Refer to Multimedia Appendix 1 for detailed raw volumes of MRI biomarkers comparing healthy controls and patients with MCI.

    Table 4. Comparison of magnetic resonance imaging (MRI) biomarkers between healthy controls and patients with mild cognitive impairment (MCI).
    MRI biomarkersGroup, mean (SD)P valuea

    		Healthy controls (n=22)Patients with MCI (n=32)
    Raw volume (cc)

    		ICVb1490.96 (127.40)1511.49 (128.89).56
    The proportion of ICV (%)

    		Left cerebral white matter14.40 (0.73)14.06 (0.91).62

    		Right cerebral white matter14.51 (0.70)14.13 (0.93).48

    		Left cerebral gray matter15.52 (1.01)14.98 (0.91).13

    		Right cerebral gray matter15.46 (1.10)14.77 (1.01).06

    		Left ventricles1.28 (0.70)1.57 (0.84).48

    		Right ventricles1.16 (0.80)1.41 (0.76).52

    		Left amygdala0.11 (0.01)0.10 (0.01).007

    		Right amygdala0.12 (0.01)0.11 (0.01).002

    		Left hippocampus0.24 (0.02)0.21 (0.03).001

    		Right hippocampus0.24 (0.02)0.22 (0.03).006

    		Left entorhinal cortex0.15 (0.02)0.13 (0.02).007

    		Right entorhinal cortex0.13 (0.01)0.11 (0.02).01

    		Left parahippocampal gyrus0.13 (0.02)0.12 (0.02).23

    		Right parahippocampal gyrus0.12 (0.02)0.12 (0.02).26

    		Left fusiform gyrus0.60 (0.06)0.58 (0.06).39

    		Right fusiform gyrus0.57 (0.07)0.57 (0.06).78

    		Left superior temporal gyrus0.72 (0.07)0.69 (0.06).17

    		Right superior temporal gyrus0.69 (0.06)0.67 (0.06).64

    		Left middle temporal gyrus0.72 (0.08)0.66 (0.08).11

    		Right middle temporal gyrus0.69 (0.08)0.65 (0.08).09

    		Left inferior temporal gyrus0.76 (0.06)0.74 (0.06).37

    		Right inferior temporal gyrus0.72 (0.08)0.68 (0.06).06

    aAnalyses of covariance, with age as a covariate.

    bICV: intracranial volume.

    Correlation Between VR-Derived and MRI Biomarkers

    We conducted a Pearson correlation analysis to explore the relationships among statistically significant biomarkers, including VR-derived biomarkers (hand movement speed, scanpath length, time to completion, and the number of errors) and MRI biomarkers from the amygdala, hippocampus, and entorhinal cortex. The aim was to uncover any significant associations among these biomarkers and elucidate their potential implications within the context of this study. The analysis revealed several significant correlations. Hand movement speed correlated with the right amygdala (r=0.31; P=.02), left hippocampus (r=0.40; P=.003), right hippocampus (r=0.43; P=.002), left entorhinal cortex (r=0.32; P=.02), and right entorhinal cortex (r=0.35; P=.009). Scanpath length correlated significantly with the left hippocampus (r=−0.28; P=.04). Meanwhile, time to completion showed significant correlations with the left amygdala (r=−0.31; P=.02), left hippocampus (r=−0.38; P=.005), right hippocampus (r=−0.28; P=.04), left entorhinal cortex (r=−0.29; P=.04), and right entorhinal cortex (r=−0.29; P=.04). Furthermore, the number of errors demonstrated significant correlations with the left amygdala (r=−0.34; P=.01), right amygdala (r=−0.38; P=.005), left hippocampus (r=−0.39; P=.003), right hippocampus (r=−0.38; P=.005), left entorhinal cortex (r=−0.29; P=.03), and right entorhinal cortex (r=−0.34; P=.01). Notably, the left hippocampus exhibited significant correlations with all 4 VR-derived biomarkers (Figure 5).

    ‎
    Figure 5. Significant correlations between 4 virtual reality–derived biomarkers (hand movement speed, scanpath length, time to completion, and the number of errors) and the magnetic resonance imaging biomarker, specifically, the left hippocampus. HC: healthy controls; MCI: mild cognitive impairment.

    Multimodal Learning Performance Using Both VR-Derived and MRI Biomarkers

    The comparative performance of multimodal learning is presented in Table 5 and Figure 6. Initially, the SVM model trained with SNSB-C data (RCFT and SVLT-E–DR), considered as a gold standard for MCI classification, exhibited strong performance, achieving an accuracy of 94.4%, a sensitivity of 100%, a specificity of 85.7%, a precision of 91.7%, an F1-score of 95.7%, and an AUC of 0.93. The results shown in Table 5 indicate that combining SNSB-C results with either VR-derived or MRI biomarkers improves MCI classification performance compared to using VR or MRI alone. The best-performing SVM model using only VR-derived biomarkers, including hand movement speed, scanpath length, and the number of errors, achieved an accuracy of 88.9%, a sensitivity of 87.5%, a specificity of 90%, a precision of 87.5%, an F1-score of 87.5%, and an AUC of 0.84 (refer to Multimedia Appendix 2 for more details). In the unimodal SVM model relying solely on MRI biomarkers, the combination of the left hippocampus and left entorhinal cortex led to the best performance, with an accuracy of 83.3%, a sensitivity of 90.9%, a specificity of 71.4%, a precision of 83.3%, an F1-score of 87%, and an AUC of 0.79 (refer to Multimedia Appendix 3 for more details). Remarkably, the highest performance was achieved when both VR-derived and MRI biomarkers—specifically, hand movement speed, scanpath length, the number of errors, left entorhinal cortex, and left hippocampus—were integrated. This multimodal approach yielded an accuracy of 94.4%, a sensitivity of 100%, a specificity of 90.9%, a precision of 87.5%, an F1-score of 93.3%, and an AUC of 0.89, with the corresponding code accessible on GitHub [54]. These results suggest that a combined approach using both VR-derived and MRI biomarkers offers the most promising outcomes, closely resembling the gold standard represented by SNSB-C. In addition, the use of VR-derived biomarkers alone showed promising results, whereas the performance while using MRI biomarkers alone was relatively lower.

    Table 5. Comparative performance of the multimodal learning approach using Seoul Neuropsychological Screening Battery–Core (SNSB-C), virtual reality (VR)–derived biomarkers, and magnetic resonance imaging (MRI) biomarkers used in the support vector machine model.
    Biomarker dataAccuracy, %Sensitivity, %Specificity, %Precision, %F1-score, %
    SNSB-C94.410085.791.795.7
    SNSB-C+VR94.492.310010096
    SNSB-C+MRI94.410083.392.396
    VR+MRI94.410090.987.593.3
    VR88.987.59087.587.5
    MRI83.390.971.483.387
    ‎
    Figure 6. Comparison of receiver operating characteristic curves and the area under the receiver operating characteristic curve (AUC). The best classification performance was obtained when the support vector machine was trained using a combination of virtual reality (VR)–derived biomarkers (hand movement speed, scanpath length, and the number of errors) and magnetic resonance imaging (MRI) biomarkers (left entorhinal cortex and left hippocampus). The gold-standard Seoul Neuropsychological Screening Battery–Core (SNSB-C; Rey Complex Figure Test and Seoul Verbal Learning Test–Elderly’s Version–Delayed Recall) was omitted from this comparison.

    Principal Findings

    The primary objective of this study was to probe the effectiveness of a multimodal learning approach, integrating both VR-derived and MRI biomarkers, in augmenting early MCI detection. The obtained results showed significant differences in both VR-derived and MRI biomarkers when comparing patients with MCI and healthy controls. Specifically, patients with MCI displayed considerably slower hand movement speed, lengthier scanpath length, prolonged time to completion, and a greater number of errors in the virtual kiosk test compared to their healthy counterparts. The MRI biomarkers indicated noteworthy cerebral atrophy in the bilateral amygdala, hippocampus, and entorhinal cortex among patients with MCI. A remarkable observation was the superior performance of the multimodal learning approach, which incorporated both VR-derived and MRI biomarkers, in the prediction of MCI. This integrated approach achieved an accuracy of 94.4%, a sensitivity of 100%, a specificity of 90.9%, a precision of 87.5%, an F1-score of 93.3%, and an AUC of 0.89. It outperformed models based solely on either VR-derived or MRI biomarkers and showed performance comparable to that of SNSB-C, the gold standard assessment tool for MCI diagnosis in our study. Importantly, the combination of VR-derived and MRI biomarkers allowed for faster MCI detection, significantly reducing the time required compared to neuropsychological tests such as SNSB-C, which typically take approximately 2 hours [55]. These findings provide substantial evidence highlighting the potential advantages of using a combination of VR-derived and MRI biomarkers for enhancing the detection of MCI.

    Our findings from the multimodal learning approach demonstrated the distinct advantages of VR-derived and MRI biomarkers in identifying patients with MCI. VR-derived biomarkers exhibited remarkable specificity of 90%, indicating their effectiveness in accurately classifying healthy controls. This finding aligns with previous studies demonstrating the high specificity of VR-derived biomarkers obtained through IADL tasks in VR, such as financial activities or public transportation tasks [16] and shopping in virtual supermarkets [56,57]. On the other hand, MRI biomarkers displayed superior sensitivity of 90.9%, showcasing their proficiency in correctly detecting patients with MCI. Interestingly, our results showed relatively higher sensitivity compared to recent MRI biomarker studies reviewed by Lombardi et al [58]. Recent MRI studies focused on optimizing a wide range of MRI features [59] and investigated subfields within brain regions, such as the hippocampal tail [60,61]. Meanwhile, our approach incorporated recent techniques for MCI screening. First, we separately measured the volumes of the left and right hemispheres to account for hemisphere asymmetry, as suggested by Mabrouk et al [62]. Subsequently, we used the hippocampus and entorhinal cortex as MRI biomarkers, which are well recognized for their sensitivity in MCI classification, based on the findings of Park et al [63]. Finally, to mitigate the potential influences of individual characteristics on brain volume, we applied the recently acclaimed ICV normalization technique [64,65]. We believe that these approaches may have contributed to the enhanced sensitivity in MCI detection compared to traditional methodologies. However, despite their strengths, each modality also has identifiable limitations. VR-derived biomarkers may sporadically misclassify patients with MCI as healthy controls, whereas MRI biomarkers might occasionally misidentify healthy controls as patients with MCI. By fusing the strengths of both VR-derived and MRI biomarkers, we managed to circumvent these restrictions, thereby achieving impressive rates of specificity (90.9%) and sensitivity (100%). These results underscore the potential benefits of multimodal learning, harnessing the complementarity of VR-derived and MRI biomarkers to significantly improve the early detection of MCI.

    The distinctive advantages of VR-derived and MRI biomarkers suggest their potential use as sequential diagnostic tools in a 2-stage diagnostic process, as proposed by Galvin et al [66]: the “detection” and “assessment and differentiation” phases. In the detection phase, a swift screening tool can be used to identify the risk of MCI among a wider population of older adults. VR-derived biomarkers align well with the requirements for a rapid screening tool in this phase, offering a brief testing duration (<5 min) with high specificity. Thus, they are effective in distinguishing healthy controls and identifying potential patients with MCI who may require further evaluation. For MCI care, these VR-derived biomarkers can be implemented in local dementia care centers. When VR-derived biomarkers are used to identify the risk of MCI among residents in these centers, individuals can be referred to local hospitals for more specific dementia assessments. Subsequently, during the assessment and differentiation phase, individuals who are flagged as potential MCI cases in the detection phase can undergo a more thorough diagnostic process to confirm MCI. In this study, MRI biomarkers prove to be an apt choice, given their high sensitivity in diagnosing patients with MCI and their ability to offer clinical evidence regarding changes in brain structure. For MCI care, the results from MRI biomarkers can be used to accurately diagnose patients with MCI and tailor treatment plans. Furthermore, the integration of both VR-derived and MRI biomarkers demonstrated the highest MCI classification performance. Overall, integrating VR-derived biomarkers as a rapid screening tool into the detection phase and MRI biomarkers as a diagnostic tool into the assessment and differentiation phase can significantly enhance the early detection of and care process for MCI. This approach not only reduces time and cost burdens on individuals but also provides invaluable support to clinicians in making accurate MCI diagnoses for the local older adult population.

    Our study’s findings highlight a notable correlation between behavioral characteristics derived from VR-derived biomarkers and observed alterations or damage in the brain, as identified through MRI biomarkers. The correlation analysis unveiled a positive relationship between VR-derived biomarkers and the left hippocampus. This implies that participants with a larger left hippocampus tended to exhibit faster hand movement speed, shorter scanpath length, reduced time to completion, and fewer errors during the virtual kiosk test. Considering the pivotal role of the hippocampus in memory formation and learning, this observation is congruent with previous studies [67,68] suggesting that hippocampal damage may lead to cognitive deficits and impact behaviors, including hand and eye movements during everyday activities. Consequently, individuals with hippocampal damage may display complex hand and eye movements when performing tasks such as those presented in the virtual kiosk test (as illustrated in Figure 7). An intriguing finding is that eye movement features exhibited significant correlations with MRI biomarkers (ie, the left hippocampus) and SNSB-C measures (ie, S-K–BNT, RCFT, and DSC), which aligns with recent studies emphasizing the potential importance of eye movements in identifying MCI during complex daily tasks [69,70]. Simultaneously, negative correlation was observed between the number of errors and the right amygdala, implying that participants with a smaller right amygdala tended to make more errors during the virtual kiosk test. The amygdala also plays a crucial role in memory, consistent with the findings of earlier studies [71,72] linking volume reduction to cognitive impairment. Consequently, individuals with amygdala damage may be more prone to errors during task performance. In this context, the use of VR-derived biomarkers obtained from augmented reality or VR tests involving daily activities represents a pioneering approach for the early detection of MCI [73,74], bridging the gap between the identification of behavioral abnormalities and the underlying structural changes in the brain [75,76].

    This study had certain limitations that should be acknowledged. First, the MCI classification model in this study did not encompass a diverse range of individuals with varying racial characteristics [77] or include those with neurodegenerative diseases such as Parkinson disease [78] and Lewy body diseases [79], which could potentially impact cognitive impairment in older adults. Future studies investigating these aspects may aid in the development of screening tools for neurodegenerative diseases among older adults, expanding the potential user pool to include different racial groups. Second, while this study used handcrafted features extracted from VR-derived biomarkers, training deep learning models directly on the input data could allow for a more comprehensive analysis of patterns and the exploration of novel features. Finally, although SVM was used as the multimodal learning approach based on previous studies [4,48-50], alternative multimodal learning approaches should be explored and compared to potentially improve classification performance. For instance, Contrastive Language-Image Pretraining [80] is one of the multimodal integration models using text and images. Similarly, future multimodal research with VR-derived and MRI biomarkers is necessary to delve deeper into their relationship.

    Despite the limitations acknowledged, our study makes a noteworthy contribution to the field compared to previous studies [21,23]. This is achieved by attaining superior MCI detection performance through the integration of VR-derived and MRI biomarkers via multimodal learning, surpassing the performance using VR-derived or MRI biomarkers individually. This consolidated approach garnered remarkable results, such as an accuracy of 94.4%, a sensitivity of 100%, a specificity of 90.9%, a precision of 87.5%, an F1-score of 93.3%, and an AUC of 0.89. These findings underscore the synergistic benefits of integrating VR-derived and MRI biomarkers. Moreover, our correlation analysis between VR-derived and MRI biomarkers illustrated how structural brain changes can translate into behavioral modifications in daily life. Finally, we proposed an innovative clinical application strategy, wherein VR-derived biomarkers are used in the preliminary detection stage, followed by MRI biomarkers in the assessment and differentiation stage. With a high specificity (90%) being ideal for initial screening, VR-derived biomarkers are perfect for the first stage, whereas MRI biomarkers, with their high sensitivity (90.9%), are optimal for the subsequent stage. This sequential approach could potentially reduce the time and cost burden for individuals and provide clinicians valuable assistance in making accurate diagnoses. To summarize, our study demonstrated the advantages of VR-derived and MRI biomarkers’ integration using multimodal learning to enhance early MCI detection.

    ‎
    Figure 7. Comparison of hand movements, eye movements, and T1-weighted magnetic resonance imaging (MRI) between healthy controls and patients with mild cognitive impairment (MCI). (A) 3D coordinates of hand movements (depicted in blue). (B) Participant focus points indicated by dots, with red, blue, and purple representing the start, middle, and end of gaze, respectively—dot size corresponds to fixation duration. (C) Patients with MCI exhibiting statistically significant atrophy in the amygdala, hippocampus, and entorhinal cortex compared to healthy controls.

    Conclusions

    This study determined the unique characteristics of VR-derived and MRI biomarkers while highlighting the significance of integrating both biomarkers for early detection of MCI. The results imply that selecting the appropriate biomarker at different stages is beneficial. Specifically, VR-derived biomarkers with high specificity can be used as an early screening tool for MCI, whereas MRI biomarkers with high sensitivity are more suitable for confirming MCI. However, the most valid approach is integrating both VR-derived and MRI biomarkers, as the SVM trained using both biomarkers outperformed other models that used a single biomarker. These compelling results provide strong evidence for the potential of multimodal learning to enhance overall diagnostic performance in the early detection of MCI. Furthermore, our study contributes to understanding how structural brain changes can manifest as behavior changes by showing the relationship between VR-derived and MRI biomarkers. Our study’s multimodal learning approach offers valuable insights into enhancing early MCI detection by integrating diverse biomarkers.

    Acknowledgments

    The support of patients was important in developing this paper. This study received support from the National Research Foundation of Korea, funded by the Korean government, Ministry of Science and Information and communication Technology (MSIT; NRF-2021R1C1C1005688 and NRF-2021R1A4A5033480). The funding sources had no role in the study conceptualization, methodology, investigation, curation, analysis of data, writing of the paper, or decision to submit the paper for publication.

    Data Availability

    The data sets produced and examined in this study are not publicly accessible due to concerns related to the privacy and consent of research participants. All data and materials related to the study are securely stored at the Department of Applied Artificial Intelligence, Seoul National University of Science and Technology, and can be accessed upon request.

    Authors' Contributions

    BP and YK contributed equally to the study conceptualization, methodology, investigation, and data curation and wrote the original draft. JP and HC contributed to the study methodology and investigation and reviewed and edited the paper. SEK reviewed and edited the paper. HR contributed to the study conceptualization, methodology, investigation, project administration, and funding acquisition and reviewed and edited the paper. KS contributed to the study conceptualization, methodology, investigation, visualization, supervision, project administration, and funding acquisition and wrote, reviewed, and edited the paper.

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    Comparison of raw volumes of magnetic resonance imaging (MRI) biomarkers between healthy controls and patients with mild cognitive impairment (MCI).

    DOCX File , 19 KB
    Multimedia Appendix 2

    Comparative performance of different virtual reality (VR)–derived feature combinations utilized in the Support Vector Machine (SVM) model.

    DOCX File , 16 KB
    Multimedia Appendix 3

    Comparative performance of different magnetic resonance imaging feature combinations used in the support vector machine model.

    DOCX File , 16 KB
    1. Opwonya J, Doan DN, Kim SG, Kim JI, Ku B, Kim S, et al. Saccadic eye movement in mild cognitive impairment and Alzheimer’s disease: a systematic review and meta-analysis. Neuropsychol Rev. Jun 06, 2022;32(2):193-227. [FREE Full text] [CrossRef] [Medline]
    2. Ryu H, Seo K. The illusion of having a large virtual body biases action-specific perception in patients with mild cognitive impairment. Sci Rep. Dec 15, 2021;11(1):24058. [FREE Full text] [CrossRef] [Medline]
    3. Scheltens P, De Strooper B, Kivipelto M, Holstege H, Chételat G, Teunissen CE, et al. Alzheimer's disease. Lancet. Apr 2021;397(10284):1577-1590. [CrossRef]
    4. Rawtaer I, Mahendran R, Kua EH, Tan HP, Tan HX, Lee TS, et al. Early detection of mild cognitive impairment with in-home sensors to monitor behavior patterns in community-dwelling senior citizens in Singapore: cross-sectional feasibility study. J Med Internet Res. May 05, 2020;22(5):e16854. [FREE Full text] [CrossRef] [Medline]
    5. Ma J, Zhang J, Wang Z. Multimodality Alzheimer's disease analysis in deep Riemannian manifold. Inf Process Manage. Jul 2022;59(4):102965. [CrossRef]
    6. Bringas S, Salomón S, Duque R, Lage C, Montaña JL. Alzheimer's Disease stage identification using deep learning models. J Biomed Inform. Sep 2020;109:103514. [FREE Full text] [CrossRef] [Medline]
    7. Porsteinsson AP, Isaacson RS, Knox S, Sabbagh MN, Rubino I. Diagnosis of early Alzheimer's disease: clinical practice in 2021. J Prev Alzheimers Dis. 2021;8(3):371-386. [CrossRef] [Medline]
    8. Sabbagh MN, Boada M, Borson S, Chilukuri M, Dubois B, Ingram J, et al. Early detection of mild cognitive impairment (MCI) in primary care. J Prev Alzheimers Dis. 2020;7(3):165-170. [CrossRef] [Medline]
    9. Harvey PD. Clinical applications of neuropsychological assessment. Dialogues Clin Neurosci. Apr 01, 2022;14(1):91-99. [CrossRef]
    10. Howieson D. Current limitations of neuropsychological tests and assessment procedures. Clin Neuropsychol. Feb 04, 2019;33(2):200-208. [CrossRef] [Medline]
    11. Huang H, Zheng S, Yang Z, Wu Y, Li Y, Qiu J, et al. Voxel-based morphometry and a deep learning model for the diagnosis of early Alzheimer's disease based on cerebral gray matter changes. Cereb Cortex. Jan 05, 2023;33(3):754-763. [FREE Full text] [CrossRef] [Medline]
    12. He P, Qu H, Cai M, Liu W, Gu X, Ma Q. Structural alteration of medial temporal lobe subfield in the amnestic mild cognitive impairment stage of Alzheimer’s disease. Neural Plast. Jan 24, 2022;2022:8461235-8461211. [FREE Full text] [CrossRef] [Medline]
    13. Carapelle E, Mundi C, Cassano T, Avolio C. Interaction between cognitive reserve and biomarkers in Alzheimer disease. Int J Mol Sci. Aug 30, 2020;21(17):6279. [FREE Full text] [CrossRef] [Medline]
    14. Mortimer JA, Borenstein AR, Gosche KM, Snowdon DA. Very early detection of Alzheimer neuropathology and the role of brain reserve in modifying its clinical expression. J Geriatr Psychiatry Neurol. Dec 29, 2005;18(4):218-223. [FREE Full text] [CrossRef] [Medline]
    15. Jekel K, Damian M, Wattmo C, Hausner L, Bullock R, Connelly PJ, et al. Mild cognitive impairment and deficits in instrumental activities of daily living: a systematic review. Alzheimers Res Ther. Mar 18, 2015;7(1):17. [FREE Full text] [CrossRef] [Medline]
    16. Seo K, Kim JK, Oh DH, Ryu H, Choi H. Virtual daily living test to screen for mild cognitive impairment using kinematic movement analysis. PLoS One. 2017;12(7):e0181883. [FREE Full text] [CrossRef] [Medline]
    17. Kim SY, Park J, Choi H, Loeser M, Ryu H, Seo K. Digital marker for early screening of mild cognitive impairment through hand and eye movement analysis in virtual reality using machine learning: first validation study. J Med Internet Res. Oct 20, 2023;25:e48093. [FREE Full text] [CrossRef] [Medline]
    18. Piau A, Wild K, Mattek N, Kaye J. Current state of digital biomarker technologies for real-life, home-based monitoring of cognitive function for mild cognitive impairment to mild Alzheimer disease and implications for clinical care: systematic review. J Med Internet Res. Aug 30, 2019;21(8):e12785. [CrossRef] [Medline]
    19. Cavedoni S, Chirico A, Pedroli E, Cipresso P, Riva G. Digital biomarkers for the early detection of mild cognitive impairment: artificial intelligence meets virtual reality. Front Hum Neurosci. Jul 24, 2020;14:245. [FREE Full text] [CrossRef] [Medline]
    20. Seo K, Lee A, Kim J, Ryu H, Choi H. Measuring the kinematics of daily living movements with motion capture systems in virtual reality. J Vis Exp. Apr 05, 2018;(134). [CrossRef]
    21. Kim SY, Park JS, Choi H, Ryu H, Seo K. Virtual kiosk test for early screening of mild cognitive impairment through eye tracking data analysis. In: Proceedings of the 2023 HCI Korea Academic Conference Presentation Paper Collection. 2023. Presented at: ACPPC '23; April 21-26, 2018;878-885; Virtual event. URL: https:/​/www.​dbpia.co.kr/​pdf/​pdfView.​do?nodeId=NODE11229766&googleIPSandBox=false&mark=0&ipRange=false&accessgl=Y&language=ko_KR&hasTopBanner=true
    22. Castegnaro A, Howett D, Li A, Harding E, Chan D, Burgess N, et al. Assessing mild cognitive impairment using object-location memory in immersive virtual environments. Hippocampus. Sep 02, 2022;32(9):660-678. [FREE Full text] [CrossRef] [Medline]
    23. Howett D, Castegnaro A, Krzywicka K, Hagman J, Marchment D, Henson R, et al. Differentiation of mild cognitive impairment using an entorhinal cortex-based test of virtual reality navigation. Brain. Jun 01, 2019;142(6):1751-1766. [FREE Full text] [CrossRef] [Medline]
    24. Jahng S, Na DL, Kang Y. Constructing a composite score for the Seoul neuropsychological screening battery-core. Dement Neurocong Disord. 2015;14(4):137. [CrossRef]
    25. Lee AY, Lee J, Oh E, Yoon SJ, Yoon B, Yu SD. Clinical utility of Seoul neuropsychological screening battery-core for dementia management project in the community. J Korean Neurol Assoc. Aug 01, 2019;37(3):277-283. [CrossRef]
    26. Kim SY, Park H, Kim H, Kim J, Seo K. Technostress causes cognitive overload in high-stress people: eye tracking analysis in a virtual kiosk test. Inf Process Manag. Nov 2022;59(6):103093. [FREE Full text] [CrossRef] [Medline]
    27. Chiu HM, Hsu MC, Ouyang WC. Effects of incorporating virtual reality training intervention into health care on cognitive function and wellbeing in older adults with cognitive impairment: a randomized controlled trial. Int J Hum Comput Stud. Feb 2023;170:102957. [CrossRef]
    28. Ilardi CR, Iavarone A, La Marra M, Iachini T, Chieffi S. Hand movements in mild cognitive impairment: clinical implications and insights for future research. J Integr Neurosci. Mar 23, 2022;21(2):67. [FREE Full text] [CrossRef] [Medline]
    29. Park J, Seo K, Kim SE, Ryu H, Choi H. Early screening of mild cognitive impairment through hand movement analysis in virtual reality based on machine learning: screening of MCI through hand movement in VR. J Cogn Interv Digit Health. 2022;1(1):1. [CrossRef]
    30. Oyama A, Takeda S, Ito Y, Nakajima T, Takami Y, Takeya Y, et al. Novel method for rapid assessment of cognitive impairment using high-performance eye-tracking technology. Sci Rep. Sep 10, 2019;9(1):12932. [FREE Full text] [CrossRef] [Medline]
    31. Tokushige SI, Matsumoto H, Matsuda SI, Inomata-Terada S, Kotsuki N, Hamada M, et al. Early detection of cognitive decline in Alzheimer's disease using eye tracking. Front Aging Neurosci. Mar 21, 2023;15:1123456. [FREE Full text] [CrossRef] [Medline]
    32. Gegenfurtner A, Lehtinen E, Säljö R. Expertise differences in the comprehension of visualizations: a meta-analysis of eye-tracking research in professional domains. Educ Psychol Rev. Jul 26, 2011;23(4):523-552. [CrossRef]
    33. Chua SI, Tan NC, Wong WT, Allen Jr JC, Quah JH, Malhotra R, et al. Virtual reality for screening of cognitive function in older persons: comparative study. J Med Internet Res. Aug 01, 2019;21(8):e14821. [FREE Full text] [CrossRef] [Medline]
    34. Kim RE, Lee M, Kang DW, Wang SM, Kim NY, Lee MK, et al. Deep learning-based segmentation to establish East Asian normative volumes using multisite structural MRI. Diagnostics (Basel). Dec 23, 2020;11(1):13. [FREE Full text] [CrossRef] [Medline]
    35. Lee M, Kim J, Ey Kim R, Kim HG, Oh SW, Lee MK, et al. Split-attention U-net: a fully convolutional network for robust multi-label segmentation from brain MRI. Brain Sci. Dec 11, 2020;10(12):974. [FREE Full text] [CrossRef] [Medline]
    36. de Flores R, Das SR, Xie L, Wisse LE, Lyu X, Shah P, et al. Medial temporal lobe networks in Alzheimer's disease: structural and molecular vulnerabilities. J Neurosci. Jan 27, 2022;42(10):2131-2141. [CrossRef]
    37. Buyanova IS, Arsalidou M. Cerebral white matter myelination and relations to age, gender, and cognition: a selective review. Front Hum Neurosci. Jul 6, 2021;15:662031. [FREE Full text] [CrossRef] [Medline]
    38. Yaxu Y, Ren Z, Ward J, Jiang Q. Atypical brain structures as a function of gray matter volume (GMV) and gray matter density (GMD) in young adults relating to autism spectrum traits. Front Psychol. Apr 8, 2020;11:523. [FREE Full text] [CrossRef] [Medline]
    39. Shao M, Han S, Carass A, Li X, Blitz AM, Shin J, et al. Brain ventricle parcellation using a deep neural network: application to patients with ventriculomegaly. Neuroimage Clin. 2019;23:101871. [FREE Full text] [CrossRef] [Medline]
    40. Galton CJ, Patterson K, Graham K, Lambon-Ralph MA, Williams G, Antoun N, et al. Differing patterns of temporal atrophy in Alzheimer's disease and semantic dementia. Neurology. Jul 24, 2001;57(2):216-225. [CrossRef] [Medline]
    41. Du AT, Schuff N, Amend D, Laakso MP, Hsu YY, Jagust WJ, et al. Magnetic resonance imaging of the entorhinal cortex and hippocampus in mild cognitive impairment and Alzheimer's disease. J Neurol Neurosurg Psychiatry. Oct 2001;71(4):441-447. [FREE Full text] [CrossRef] [Medline]
    42. Son SJ, Kim J, Park H. Structural and functional connectional fingerprints in mild cognitive impairment and Alzheimer's disease patients. PLoS One. Mar 23, 2017;12(3):e0173426. [FREE Full text] [CrossRef] [Medline]
    43. Tee B, Watson Pereira C, Lukic S, Bajorek LP, Allen IE, Miller ZA, et al. Neuroanatomical correlations of visuospatial processing in primary progressive aphasia. Brain Commun. 2022;4(2):fcac060. [FREE Full text] [CrossRef] [Medline]
    44. Weiner KS, Zilles K. The anatomical and functional specialization of the fusiform gyrus. Neuropsychologia. Mar 2016;83:48-62. [FREE Full text] [CrossRef] [Medline]
    45. Singh V, Chertkow H, Lerch JP, Evans AC, Dorr AE, Kabani NJ. Spatial patterns of cortical thinning in mild cognitive impairment and Alzheimer's disease. Brain. Nov 29, 2006;129(Pt 11):2885-2893. [CrossRef] [Medline]
    46. Sanfilipo MP, Benedict RH, Sharma J, Weinstock-Guttman B, Bakshi R. The relationship between whole brain volume and disability in multiple sclerosis: a comparison of normalized gray vs. white matter with misclassification correction. Neuroimage. Jul 15, 2005;26(4):1068-1077. [CrossRef] [Medline]
    47. Albert MS, DeKosky ST, Dickson D, Dubois B, Feldman HH, Fox NC, et al. The diagnosis of mild cognitive impairment due to Alzheimer's disease: recommendations from the National Institute on Aging-Alzheimer's Association workgroups on diagnostic guidelines for Alzheimer's disease. Alzheimers Dement. May 2011;7(3):270-279. [FREE Full text] [CrossRef] [Medline]
    48. Swati S, Kumar M, Namasudra S. Early prediction of cognitive impairments using physiological signal for enhanced socioeconomic status. Inf Process Manage. Mar 2022;59(2):102845. [CrossRef]
    49. Tuladhar A, Gill S, Ismail Z, Forkert ND, Alzheimer's Disease Neuroimaging Initiative. Building machine learning models without sharing patient data: a simulation-based analysis of distributed learning by ensembling. J Biomed Inform. Jun 2020;106:103424. [FREE Full text] [CrossRef] [Medline]
    50. Wei Q, Cao H, Shi Y, Xu X, Li T. Machine learning based on eye-tracking data to identify autism spectrum disorder: a systematic review and meta-analysis. J Biomed Inform. Jan 2023;137:104254. [FREE Full text] [CrossRef] [Medline]
    51. Kijonka M, Borys D, Psiuk-Maksymowicz K, Gorczewski K, Wojcieszek P, Kossowski B, et al. Whole brain and cranial size adjustments in volumetric brain analyses of sex- and age-related trends. Front Neurosci. Apr 3, 2020;14:278. [FREE Full text] [CrossRef] [Medline]
    52. Sanchis-Segura C, Ibañez-Gual MV, Adrián-Ventura J, Aguirre N, Gómez-Cruz ÁJ, Avila C, et al. Sex differences in gray matter volume: how many and how large are they really? Biol Sex Differ. Jul 01, 2019;10(1):32. [FREE Full text] [CrossRef] [Medline]
    53. Barnes J, Ridgway GR, Bartlett J, Henley SM, Lehmann M, Hobbs N, et al. Head size, age and gender adjustment in MRI studies: a necessary nuisance? Neuroimage. Dec 2010;53(4):1244-1255. [CrossRef] [Medline]
    54. seoultech-HAILAB/multimodal-learning-performance-using-both-VR-derived-and-MRI-biomarkers. GitHub. 2023. URL: https:/​/github.​com/​seoultech-HAILAB/​Multimodal-learning-performance-using-both-VR-derived-and-MRI-biomarkers [accessed 2023-12-29]
    55. Ryu HJ, Yang DW. The Seoul Neuropsychological Screening Battery (SNSB) for comprehensive neuropsychological assessment. Dement Neurocogn Disord. Jan 2023;22(1):1-15. [FREE Full text] [CrossRef] [Medline]
    56. Zygouris S, Giakoumis D, Votis K, Doumpoulakis S, Ntovas K, Segkouli S, et al. Can a virtual reality cognitive training application fulfill a dual role? Using the virtual supermarket cognitive training application as a screening tool for mild cognitive impairment. J Alzheimers Dis. 2015;44(4):1333-1347. [CrossRef] [Medline]
    57. Eraslan Boz H, Limoncu H, Zygouris S, Tsolaki M, Giakoumis D, Votis K, et al. A new tool to assess amnestic mild cognitive impairment in Turkish older adults: virtual supermarket (VSM). Neuropsychol Dev Cogn B Aging Neuropsychol Cogn. Sep 04, 2020;27(5):639-653. [CrossRef] [Medline]
    58. Lombardi G, Crescioli G, Cavedo E, Lucenteforte E, Casazza G, Bellatorre AG, et al. Structural magnetic resonance imaging for the early diagnosis of dementia due to Alzheimer's disease in people with mild cognitive impairment. Cochrane Database Syst Rev. Mar 02, 2020;3(3):CD009628. [FREE Full text] [CrossRef] [Medline]
    59. Zamani J, Sadr A, Javadi AH. Diagnosis of early mild cognitive impairment using a multiobjective optimization algorithm based on T1-MRI data. Sci Rep. Jan 19, 2022;12(1):1020. [FREE Full text] [CrossRef] [Medline]
    60. Qu H, Ge H, Wang L, Wang W, Hu C. Volume changes of hippocampal and amygdala subfields in patients with mild cognitive impairment and Alzheimer's disease. Acta Neurol Belg. Aug 12, 2023;123(4):1381-1393. [CrossRef] [Medline]
    61. Feng Q, Wang L, Tang X, Ge X, Hu H, Liao Z, et al. Machine learning classifiers and associations of cognitive performance with hippocampal subfields in amnestic mild cognitive impairment. Front Aging Neurosci. Nov 30, 2023;15:1273658. [FREE Full text] [CrossRef] [Medline]
    62. Mabrouk B, BenHamida A, Drissi N, Bouzidi N, Mhiri C. Contribution of brain regions asymmetry scores combined with random forest classifier in the diagnosis of Alzheimer’s disease in his earlier stage. J Med Biol Eng. Jan 28, 2023;43(1):74-82. [CrossRef]
    63. Park HY, Suh CH, Heo H, Shim WH, Kim SJ. Diagnostic performance of hippocampal volumetry in Alzheimer's disease or mild cognitive impairment: a meta-analysis. Eur Radiol. Oct 04, 2022;32(10):6979-6991. [CrossRef] [Medline]
    64. Wu Y, Wang X, Fang Y. Predicting mild cognitive impairment in older adults: a machine learning analysis of the Alzheimer's Disease Neuroimaging Initiative. Geriatr Gerontol Int. Mar 21, 2024;24 Suppl 1:96-101. [CrossRef] [Medline]
    65. Chawalparit O, Wontaneeporn N, Muangpaisan W, Thientunyakit T, Charnchaowanish P, Ngamsombat C. MRI hippocampal subfield volume analysis: comparison between Alzheimer’s disease, mild cognitive impairment, and normal aging subjects in an amyloid PET project. J Alzheimers Dis Parkinsonism. 2019;09(01). [CrossRef]
    66. Galvin JE, Aisen P, Langbaum JB, Rodriguez E, Sabbagh M, Stefanacci R, et al. Early stages of Alzheimer's disease: evolving the care team for optimal patient management. Front Neurol. Jan 22, 2020;11:592302. [FREE Full text] [CrossRef] [Medline]
    67. Horvath AA, Berente DB, Vertes B, Farkas D, Csukly G, Werber T, et al. Differentiation of patients with mild cognitive impairment and healthy controls based on computer assisted hand movement analysis: a proof-of-concept study. Sci Rep. Nov 09, 2022;12(1):19128. [FREE Full text] [CrossRef] [Medline]
    68. Whitlock JR, Sutherland RJ, Witter MP, Moser MB, Moser EI. Navigating from hippocampus to parietal cortex. Proc Natl Acad Sci U S A. Sep 30, 2008;105(39):14755-14762. [FREE Full text] [CrossRef] [Medline]
    69. Opwonya J, Ku B, Lee KH, Kim JI, Kim JU. Eye movement changes as an indicator of mild cognitive impairment. Front Neurosci. Jun 15, 2023;17:1171417. [FREE Full text] [CrossRef] [Medline]
    70. Wolf A, Tripanpitak K, Umeda S, Otake-Matsuura M. Eye-tracking paradigms for the assessment of mild cognitive impairment: a systematic review. Front Psychol. Jul 20, 2023;14:1197567. [FREE Full text] [CrossRef] [Medline]
    71. Cuénod CA, Denys A, Michot JL, Jehenson P, Forette F, Kaplan D, et al. Amygdala atrophy in Alzheimer's disease: an in vivo magnetic resonance imaging study. Arch Neurol. Sep 01, 1993;50(9):941-945. [CrossRef] [Medline]
    72. Poulin SP, Dautoff R, Morris JC, Barrett LF, Dickerson BC, Alzheimer's Disease Neuroimaging Initiative. Amygdala atrophy is prominent in early Alzheimer's disease and relates to symptom severity. Psychiatry Res. Oct 31, 2011;194(1):7-13. [FREE Full text] [CrossRef] [Medline]
    73. Muurling M, de Boer C, Vairavan S, Harms RL, Chadha AS, Tarnanas I, et al. Augmented reality versus standard tests to assess cognition and function in early Alzheimer's disease. NPJ Digit Med. Dec 18, 2023;6(1):234. [FREE Full text] [CrossRef] [Medline]
    74. Liu Q, Song H, Yan M, Ding Y, Wang Y, Chen L, et al. Virtual reality technology in the detection of mild cognitive impairment: a systematic review and meta-analysis. Ageing Res Rev. Jun 2023;87:101889. [CrossRef] [Medline]
    75. Apostolova LG, Cummings JL. Neuropsychiatric manifestations in mild cognitive impairment: a systematic review of the literature. Dement Geriatr Cogn Disord. Dec 14, 2008;25(2):115-126. [FREE Full text] [CrossRef] [Medline]
    76. Igarashi KM. Entorhinal cortex dysfunction in Alzheimer's disease. Trends Neurosci. Feb 2023;46(2):124-136. [FREE Full text] [CrossRef] [Medline]
    77. Wright CB, DeRosa JT, Moon MP, Strobino K, DeCarli C, Cheung YK, et al. Race/ethnic disparities in mild cognitive impairment and dementia: the Northern Manhattan study. J Alzheimers Dis. 2021;80(3):1129-1138. [FREE Full text] [CrossRef] [Medline]
    78. Saredakis D, Collins-Praino LE, Gutteridge DS, Stephan BC, Keage HA. Conversion to MCI and dementia in Parkinson's disease: a systematic review and meta-analysis. Parkinsonism Relat Disord. Aug 2019;65:20-31. [CrossRef] [Medline]
    79. Ciafone J, Little B, Thomas AJ, Gallagher P. The neuropsychological profile of mild cognitive impairment in Lewy body dementias. J Int Neuropsychol Soc. Dec 12, 2019;26(2):210-225. [CrossRef]
    80. Radford A, Kim JK, Hallacy C, Ramesh A, Goh G, Agarwal S, et al. Learning transferable visual models from natural language supervision. In: Proceedings of the 38 th International Conference on Machine Learning. 2021. Presented at: ICML '21; July 18-24, 2021;8748-8763; Virtual event. URL: https://proceedings.mlr.press/v139/radford21a/radford21a.pdf

    ‎
    AD: Alzheimer disease
    ANCOVA: analysis of covariance
    AUC: area under the receiver operating characteristic curve
    IADL: instrumental activities of daily living
    ICV: intracranial volume
    MCI: mild cognitive impairment
    MRI: magnetic resonance imaging
    RCFT: Rey Complex Figure Test
    SNSB-C: Seoul Neuropsychological Screening Battery–Core
    SVM: support vector machine
    VR: virtual reality

    Edited by G Eysenbach, T de Azevedo Cardoso; submitted 15.11.23; peer-reviewed by Y Wang, A Mastropietro; comments to author 11.12.23; revised version received 29.12.23; accepted 09.03.24; published 17.04.24.

    Copyright

    ©Bogyeom Park, Yuwon Kim, Jinseok Park, Hojin Choi, Seong-Eun Kim, Hokyoung Ryu, Kyoungwon Seo. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 17.04.2024.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.