Smartphone- and Tablet-Based Tools to Assess Cognition in Individuals With Preclinical Alzheimer Disease and Mild Cognitive Impairment: Scoping Review

Introduction

Background

Alzheimer disease (AD) is a neurodegenerative disease associated with gradual decline in cognition, where disease stages range from preclinical AD, in which cognitive decline is absent or only subtle, to mild cognitive impairment (MCI) and dementia [1,2]. Biologically, the disease is defined by pathological changes in amyloid accumulation and neurofibrillary tau protein tangles that appear years before the onset of cognitive symptoms [1-3]. However, it has been widely shown that, even in the preclinical stage, cognition does decline subtly [4-7]. With the increasing number of older adults, it is expected that AD will become a major societal problem, estimated to affect 131 million people in 2050 worldwide [8,9]. The emergence of AD biomarkers allows for the recognition of the disease in the preclinical stage, which has opened a window of opportunity for treatment studies, as interventions are likely to be most beneficial in early stages of the disease. Accordingly, disease-modifying treatments and nonpharmaceutical prevention trials have increasingly focused on preclinical stages of AD over the past years [10-12]. This shift toward the earliest AD stages highlights the need for new tools that provide outcome measures of cognition that are adequate for use in these early disease stages in the realm of early detection, diagnosis, disease monitoring, and evaluation of treatment effects [13-17]. In addition, to enable large-scale decentralized prevention, intervention, or disease-monitoring initiatives, tools are required that enable remote, time-efficient, and reliable assessment of cognition.

The current gold standard to assess cognition in AD is through neuropsychological testing, using paper-and-pencil tests that need to be supervised by a trained neuropsychologist. Importantly, such traditional cognitive tests are not suitable for the earliest AD stages as these tests often lack the sensitivity that is required to detect subtle cognitive changes [7,17,18]. Thus, novel, stage-specific testing paradigms are needed, and digital tools are promising to fill this gap [19,20]. A major advantage of digital cognitive tools is their suitability for unsupervised remote assessment, which enables highly frequent testing that may provide a more accurate reflection of cognition [21,22]. Given the intuitive person-device interaction of touch screen devices, smartphone- and tablet-based tools provide the optimal modality for unsupervised remote assessment [23,24]. Other advantages of such cognitive tools include reduced patient burden; time efficiency through automatic administration and scoring; high scalability; and the potential for rich data collection of precise measurements, including, for example, response times.

Over the last decade, numerous digital tools have been developed that have the potential for remote assessment of cognition in early AD stages [19]. However, the development of such tools has not yet led to large-scale implementation [25]. To date, the measurement quality of digital tools to assess cognition is largely unknown, although it has widely been acknowledged that information on psychometric properties is important for the use and implementation of these tools [19,20,26-28]. For the evaluation of cognitive tests in early AD stages, a recommendation framework has been proposed based on the Consensus-Based Standards for the Selection of Health Measurement Instruments methodology [16,29,30]. This framework highlights the importance of psychometric properties concerning structural and construct validity, responsiveness, and clinical meaningfulness. Hence, available information on psychometric properties is a prerequisite for the successful implementation of digital tools. Although previous reviews have described some validation aspects of digital cognitive assessments [19,31], a comprehensive overview of the reporting of crucial psychometric properties (ie, structural and construct validity, reliability, responsiveness, and clinical meaningfulness) of digital cognitive tools for use in preclinical AD or MCI is currently missing.

Objectives

This scoping review was conducted to identify the reporting on quality characteristics of smartphone- and tablet-based tools to assess cognition in individuals with preclinical AD or MCI. Our scope was limited to smartphone- and tablet-based cognitive tools with the potential for remote assessment. The primary focus was on the availability of information on psychometric properties, and the secondary focus on the reporting on practical tool characteristics. The findings of this review will be used to formulate recommendations about future steps that are needed to facilitate the implementation of remote smartphone- and tablet-based tools to assess cognition.

Methods

Overview

This scoping review is conducted in line with a methodological framework for conducting scoping studies [32]. The reporting complies with the guidelines of the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) checklist [33], as shown in Multimedia Appendix 1. The review was not registered, and no protocol was prepared.

Search Strategies

After several scoping searches, 4 bibliographic databases (PubMed, Embase [Elsevier], Web of Science Core Collection [Clarivate Analytics], and APA PsycINFO [EBSCO]) were searched for relevant literature from January 1, 2008 (ie, the year that the Google Play Store and Apple App Store, two of the biggest mobile app distribution platforms, were launched), to January 5, 2023. Searches were devised in collaboration with a medical information specialist (KAZ). Search terms including synonyms, closely related words, and keywords were used as free-text words (eg, “Alzheimer,” “digital,” and “cognition”). The search contained no methodological search filter that would limit results to specific study designs or languages. Duplicate studies were excluded using the R package ASYSD (R Foundation for Statistical Computing), an automated deduplication tool [34], which was followed by manual deduplication in EndNote (version X20.0.3; Clarivate Analytics). The full search strategy used for each database is detailed in Multimedia Appendix 2.

Eligibility Criteria

Published journal articles were included or excluded according to the criteria outlined in Textbox 1.

Screening Procedures

To minimize selection bias, 2 of the review authors (RLvdB and SMvdL) independently screened titles and abstracts based on the eligibility criteria (Textbox 1) using ASReview (version 1.1) [35] with default settings (naive Bayes, term frequency–inverse document frequency, and query strategy: max). Differences in judgment were resolved via a consensus procedure through discussion and, if necessary, by consulting a third reviewer (MJK). ASReview is an open-source screening tool using manual input in combination with an active learning algorithm to screen studies based on titles and abstracts. The algorithm was trained using 7 relevant and 7 irrelevant studies, where relevant studies were manually selected and irrelevant studies were randomly provided by the model and subsequently manually labeled as irrelevant. A data-driven stopping rule strategy was followed, which has been suggested previously, and proposes to end the screening process after n consecutive irrelevant records are provided by the active learning algorithm [36]. No consensus exists yet on the optimal n, and the number varies across studies (eg, 20-500) [36-41]. Importantly, it has previously been noted that the stopping rule should aim to find the trade-off between the costs of manually labeling additional records and the costs of erroneous exclusions of relevant records by the algorithm [42]. We stopped the screening process after 100 consecutive irrelevant studies were provided by the ASReview algorithm. This number was based on an initial pilot screening round where it was observed that, if the stopping rule was set to 50 consecutive irrelevant records, relevant records continued to appear after 45 to 49 records, so the cutoff was increased to 100 to include a higher certainty margin. After screening of titles and abstracts, 4 of the review authors (RLvdB, SMvdL, MJK, and AMvG) conducted full-text screening according to the eligibility criteria (Textbox 1) using the web application Rayyan (Qatar Computing Research Institute) [43]. Each study was screened by one of the reviewers independently. Uncertainties were discussed, and if this resulted in disagreement, this was resolved through discussion, or if necessary, a fifth reviewer was consulted.

Data Extraction and Data Synthesis

Overview

Two review authors (RLvdB and MJK) independently extracted data from the final set of included studies such that each study was assessed by 2 authors. Uncertainties were discussed, and if needed, a third author was consulted (SMvdL, AMvG, MvD, SAMS, and CdB). Data were extracted about practical tool characteristics, study characteristics, and psychometric properties. Practical tool characteristics and psychometric properties were extracted from the included studies and summarized at the tool level. Summarized information was calculated based on available data, and missing data were excluded. Study characteristics were extracted and reported at the study level. In case of missing information, this was reported as such. Examples of practical characteristics and psychometric properties were provided narratively.

Practical tool characteristics concerned the cognitive domains assessed (single or multiple), active or passive task completion (ie, evaluating performance during an assessment vs assessing performance in everyday life activities, respectively), administration time, testing design (ie, single or repeated assessment), and language in which the tool was examined in the included studies. Study characteristics included the study population examined, total sample size and investigative setting (on-site, eg, in the clinic, university, or day or community center, or remote, eg, at-home environment). In addition, we extracted information on the psychometric properties of the included digital tools, which we selected based on recommendations on how cognition should be measured as formulated in the “recommendation framework for evaluation of performance based cognitive tests” [16] related to construct validity and interpretability that were based on the Consensus-Based Standards for the Selection of Health Measurement Instruments methodology. The psychometric properties assessed in this review were (1) structural validity (factor analysis), (2) construct validity (clinical and biological associations and relevant clinical group differences), (3) test-retest reliability and responsiveness, and (4) interpretability (clinical meaningfulness), which are defined and explained in more detail in the following sections. Specifically, for each digital tool, we evaluated whether information on each of these 4 psychometric properties was reported (✓) or not available in the included studies. The ratings of ✓ and “not available” were assigned regardless of the method used or the quality of the reported psychometric property.

Structural Validity

Structural validity refers to the degree to which scores are an adequate reflection of the dimensionality that is measured (ie, underlying structure of one or multiple cognitive domains) [16,29]. As such, it is recommended to conducted a factor analysis to demonstrate adequate use of outcome scores [16]. The digital tool was given a rating of ✓ if a factor analysis was conducted.

Construct Validity

Construct validity refers to the extent to which outcome scores are in line with hypotheses regarding, for example, relationships to scores on other instruments or differences between relevant groups [29]. For clinical outcome assessments in early AD, it is recommended that test scores are validated against relevant clinical or biological measures [16]. We defined three subcategories for construct validity: (1) clinical association, (2) biological association, and (3) relevant group differences. The tool was assigned a rating of ✓ for clinical associations if correlations were evaluated between the digital outcome measures and traditional measures of global cognition or specific cognitive domains. The tool was given a rating of ✓ for biological associations if correlations between AD biomarkers (eg, β-amyloid and tau protein deposition), markers of neurodegeneration (eg, measures of cortical thickness), or AD biomarker group differences on the digital outcome measure were reported. The tool was given a rating of ✓ if differences between groups with different clinical status (eg, cognitively unimpaired [CU], MCI, and AD dementia) were reported.

Reliability and Responsiveness

Test-retest reliability refers to the extent to which cognitive performance measured using a digital tool is consistent for the same patient over time [29,44]. Given that a patient has not changed, no cognitive changes are expected for the same patient under the same conditions within a short time frame. The tool was assigned a rating of ✓ if test-retest reliability was reported.

Responsiveness refers to the sensitivity of a digital tool to detect change in cognition over time [16,29]. As such, a requirement for digital tools aiming at measuring cognitive change is that they are sensitive to it. Hence, responsiveness is an essential property of digital tools, and a validation study should be conducted on the ability to capture cognitive change in the target population [16]. Tools were assigned a rating of ✓ if responsiveness of the digital tool to cognitive change over time was reported.

Interpretability

Clinical meaningfulness is not considered a psychometric property but, rather, an important tool characteristic for interpretability of a (change in) score [29]. Clinical meaningfulness refers to a score (or its change) that can be interpreted as clinically relevant such that a qualitative meaning is assigned to the quantitative score [16,29]. For outcome measures in early AD, it is recommended that the scores that patients and caregivers perceive as clinically meaningful be examined [16]. We assigned the tool a rating of ✓ if it specified what (change in) score was considered clinically relevant. The tool was assigned a rating of “not available” if it did not specify what (change in) score was considered clinically relevant or if the term “clinical meaningfulness” was used but no defined meaningful score was provided.

Results

Search and Screening Results

The systematic literature search generated a total of 23,404 references: 5726 (24.47%) in PubMed, 8069 (34.48%; n=11,322, 48.38% with conference abstracts) in Embase, 7675 (32.79%) in Web of Science Core Collection, and 1934 (8.26%) in APA PsycINFO. After removing duplicates, of the 23,404 references, 11,300 (48.28%) remained. These references were screened based on title and abstract in ASReview. Of the 11,300 abstracts, the first reviewer (RLvdB) manually included 233 (2.06%) relevant ones and excluded 1465 (12.96%) irrelevant ones, and the second reviewer (SMvdL) manually included 234 (2.07%) relevant ones and excluded 1062 (9.4%) irrelevant ones. Within these 2 sets, there was agreement for 33.4% (156/467) of the included abstracts and 38.3% (969/2527) of the excluded abstracts, whereas 1.27% (144/11,300) of the abstracts were labeled differently (eg, included or excluded) such that, within the set of studies that were manually labeled by both reviewers, there was 88% agreement. After reaching consensus, 233 relevant studies were identified and included for further full-text screening. After full-text screening, of the 233 studies, 50 (21.5%) were included for data extraction. The PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flowchart [45] of the search and selection process is presented in Figure 1.

‎

Tool and Study Characteristics

Practical Tool Characteristics

In total, 50 studies were included that comprised 37 tools. Although it is complex to make a binary distinction between one-to-one digitalized versions of paper-and-pencil tests and tools that are based to some extent on existing traditional paper-and-pencil tests, it was observed that digital tools used a variety of approaches. One-to-one digitalized versions of widely used traditional paper-and-pencil tests included, for example, a digital trail making task (digital Trail Making Test–Black and White) [46], digitalized word list recall (ReVeRe word list recall) [47], a computerized associative learning task (paired associate learning [PAL]) [48], and an automated story recall task (ASRT) [49,50]. The digitalization of these existing paper-and-pencil tests allowed for new variables, such as movement and reaction time measures. Other tools comprised batteries of multiple digital tests that were based on existing neuropsychological tasks (eg, the Boston Remote Assessment for Neurocognitive Health [BRANCH] [51], cCOG [52], and Electronic Cognitive Screen [53]). Other approaches involved, for example, tapping as rapidly as possible or tapping following a presented sound rhythm (JustTouch [54]), identifying nonidentical stimuli between 2 presented blocks (the tablet-based cancellation test e-CT [55]), placing virtual objects and recalling their location (Altoida [56-58]), or remembering names and occupations associated with faces (FACEmemory [59,60]). One tool, Klondike Solitaire [61,62], incorporated an “off-the-shelf” game evaluating performance on the existing digitalized game Klondike solitaire. Other methods involved tasks simulating everyday life activities, such as dosing pills in a virtual pill box (Simulation-Based Assessment of Cognition [SIMBAC] [63,64]) or grocery shopping in a virtual supermarket (Virtual Supermarket Test [VST] [65-68]). Moreover, approaches included the measurement of everyday communication skills, such as typing (nQ Medical [69] and Type of Mood [70]) and speech production (ki:e speech biomarker for cognition [SB-C] [71], Winterlight Assessment [WLA] [72], and a “tablet-based automatic assessment using speech data” [73]).

Information about practical tool characteristics is shown in Table 1. Most tools (26/37, 70%) were reported to assess multiple cognitive domains. The cognitive domains assessed varied across tools and included, for example, attention, episodic memory, executive functioning, inhibition, orientation, language, implicit learning, processing speed, or working memory [52,74-76]. A total of 30% (11/37) of the tools assessed 1 specific domain. Of those 11 tools, 6 (55%) assessed memory (BRANCH [51], PAL [48], Episodix [77], FACEmemory [59,60], Learning Over Repeated Exposures [LORE] [78], and ReVeRe word list recall [47]). In total, 8% (3/37) of the tools assessed language (speech; WLA [72], ki:e SB-C [71], and the “tablet-based automatic assessment using speech data” [73]). A total of 3% (1/37) of the tools measured executive functioning (e-CT [55]), and 3% (1/37) measured motor function (JustTouch [54]).

All but 1 digital tool (36/37, 97%) were active cognitive tasks. The passive tool (Type of Mood [70]) measured everyday typing performance. However, it should be noted that the active or passive dichotomization should be placed on a continuum where, for instance, tasks that are not commonly performed in everyday life, such as the e-CT [55] (canceling nonmatching symbols) or JustTouch [54] (rapidly tapping), might be considered more on the active side. On the other hand, tools analyzing actively elicited speech (eg, WLA [72], ki:e SB-C [71], and the “tablet-based automatic assessment using speech data” [73]) or typing a copied text or simulated text conversation (nQ Medical app [69]) might be considered more passive testing because such activities may be argued to interfere with everyday activities to a lesser extent.

Administration time was, on average, 13.8 (SD 10.1; range 1-32) minutes. The shortest tasks involved finger tapping; symbol cancelation; story recall; and a short battery of a symbol-matching, color-shape-memory, and location-memory task (JustTouch [54], e-CT [55], ASRT [49,50], and Mobile Monitoring of Cognitive Change [86]). The tools with the longest administration time covered multiple tasks assessing different cognitive domains (Geras Solutions cognitive test [82] and Inbrain Cognitive Screening Test [CST] [83]) or learning face-name-occupation associations and memorizing these after a 20-minute delay (FACEmemory [59,60]). For 38% (14/37) of the tools, the administration time was unspecified, and for 24% (9/37) of the tools, the testing design (ie, single or repeated testing) was not reported. Most tools (21/37, 57%) had a single-testing design, whereas 16% (6/37) were incorporated into a multiday protocol, and for 3% (1/37) of the tools (Altoida [56-58]), the testing design differed across studies. In the repeated-testing designs, (1) participants completed short cognitive tasks over several days (Altoida [56,58], Ambulatory Research in Cognition [ARC] [22], ASRT [49,50], NeuroUX [88], and Mobile Monitoring of Cognitive Change [86]), (2) tasks were assessed monthly for 1 year (LORE [78]), or (3) passive monitoring comprised a longer period (ie, 6 months; Type of Mood [70]).

Most tools (31/37, 84%) were tested in a single language, which was primarily English. A total of 11% (4/37) of the tools were tested in 2 languages (ie, VST [65-68] in Greek and Turkish; University of California, San Francisco, Brain Health Assessment [UCSF-BHA] [90-93] in English and Spanish; ki:e SB-C [71] in English and Dutch; and Integrated Cognitive Assessment [84] in English and Farsi). In total, 3% (1/37) of the tools were assessed in 5 languages (ie, cCOG [52] in English, Finnish, Danish, Dutch, and Italian), and 3% (1/37) were assessed in 6 languages (ie, Altoida [56-58] in English, French, Spanish, Greek, German, and Italian). For 3% (1/37) of the tools (Natural and Artificial Intelligence Health Assistant Neuro Cognitive Test [76]), the language was not specified.

Study Characteristics

Table 2 provides a summary of the characteristics of the included studies that examined a smartphone- or tablet-based cognitive tool in individuals with MCI or preclinical AD. Most tools (29/37, 78%) were assessed in a single study. A total of 14% (5/37) of the tools were examined in 4% (2/50) of the studies (ASRT [49,50], Computerized Cognitive Composite [C3] [80,81], FACEmemory [59,60], Klondike Solitaire [61,62], and SIMBAC [63,64]). One tool (Altoida [56-58]) was assessed in 6% (3/50) of the included studies, and 5% (2/37) of the tools (VST [65-68] and UCSF-BHA [90-93]) were examined in 8% (4/50) of the studies. Most of the included studies (36/50, 72%) had a cross-sectional design, whereas 28% (14/50) had a longitudinal design. These longitudinal studies assessed 32% (12/37) of the digital tools.

The mean total sample size in the included studies was 295.4 (SD 621.6), with a range of 12 to 4486 individuals. Most studies (41/50, 82%) examined the digital tool in individuals with MCI but not in individuals with preclinical AD, 10% (5/50) of the studies included both individuals with MCI and individuals with preclinical AD, and 8% (4/50) of the studies included individuals with preclinical AD but not individuals with MCI. Summarizing these studies at an individual tool level, most tools (29/37, 78%) were examined in individuals with MCI but not in individuals with preclinical AD. In total, 8% (3/37) of the digital tools (ARC [22], C3 [80,81], and LORE [78]) were investigated in individuals with preclinical AD but not in individuals with MCI. A total of 14% (5/37) of the tools (ASRT [49,50], BRANCH [51], FACEmemory [59,60], PAL [48], and UCSF-BHA [90-93]) were examined in both individuals with preclinical AD and individuals with MCI. Thus, most tools were investigated in individuals with MCI (34/37, 92%), and fewer tools were examined in individuals with preclinical AD (8/37, 22%). A total of 54% (27/50) of the studies, covering 59% (22/37) of the tools, additionally included populations other than individuals with preclinical AD or MCI, such as individuals with dementia.

In 48% (24/50) of the studies the digital tools were examined on-site. In 22% (11/50) of the studies, the psychometric properties of the tool were investigated in a remote setting, and in 16% (8/50) of the studies, these properties were examined in both an on-site and remote setting. In 14% (7/50) of the studies, the investigative setting was not described.

Psychometric Properties

Overview

Table 3 provides a summary at the tool level on the reporting on psychometric properties in the included studies.

Structural Validity

For a minority of digital tools (3/37, 8%; Inbrain CST [83], National Institutes of Health Toolbox Cognition Battery [NIHTB-CB] [75], and the tablet-based cognitive test battery [94]), a factor analysis on outcome variables was conducted to explore the underlying structure of the construct. The factor structure was examined using a multiple-factor analysis, both an exploratory and confirmatory factor analysis, or a confirmatory factor analysis, respectively. For 5% (2/37) of the tools, (structural) validity was formally assessed (NIHTB-CB [75] and the tablet-based cognitive test battery [94]), whereas for other tools, limited details were provided with regard to methodology and model fit (Inbrain CST [83]).

Construct Validity

For most tools (36/37, 97%), construct validity was assessed with regard to at least one aspect (ie, clinical or biological associations or relevant group differences). For the tool that was not assessed regarding construct validity (NIHTB-CB [75]), the study focused rather on its structural validity. Specifically, a clinical association was examined for 68% (25/37) of the tools. Existing clinical tests that digital tools were correlated with were, for example, the Mini-Mental State Examination, Montreal Cognitive Assessment, Rey Auditory Verbal Learning Test, or Preclinical Alzheimer’s Cognitive Composite (PACC) [52,66,71,76,80,82,94]. Correlation coefficients differed largely between tools as well as between scores within tools. To illustrate, depending on the subscale and clinical test, correlations ranged from 0.174 to 0.735 for the Natural and Artificial Intelligence Health Assistant Neuro Cognitive Test [76], from −0.02 to −0.57 for ARC [22], and from −0.382 to 0.617 for BRANCH [51].

Biological validation was evaluated for a minority of the tools (10/37, 27%). Specifically, 11% (4/37) of the tools (ASRT [49], BRANCH [51], C3 [80,81], and LORE [78]) were validated against AD biomarkers (ie, amyloid and tau protein). In total, 3% (1/37) of the tools were validated against a neuroanatomical correlate of cortical thickness (Inbrain CST [83]), and 14% (5/37) of the tools (Altoida [58], ARC [22], FACEmemory [59,60], PAL [48], and UCSF-BHA [91-93]) were validated against both AD biomarkers and neuroanatomical correlates (eg, cortical, subcortical, cerebral, and hippocampal volumes). All 22% (8/37) of the tools that were assessed in a population of individuals with preclinical AD were validated against biological measures. The methodology used to assess biological associations was (1) calculating correlations with continuous biological measures, (2) using regression analyses to assess differences between biomarker groups, or (3) conducting receiver operating characteristic analyses to predict biomarker status. Reported correlation coefficients ranged between and within tools depending on the outcome measure and biological correlate, as illustrated, for example, by correlations ranging from −0.23 to 0.29 for the ARC [22] composite score or correlations ranging from −0.306 to 0.219 for BRANCH [51] subtasks. Similarly, in receiver operating characteristic analyses, the reported area under the curve (AUC) values for predicting amyloid positivity varied between and within tools, as illustrated by AUC values of 0.752 for UCSF-BHA [92] (combination of outcome scores) and AUC values ranging from 0.43 to 0.92 for ASRT [49] depending on the subtask, the sample (full sample, CU, or MCI), and whether data transcription was manual or automatic.

Relevant clinical group differences in digital outcome scores were examined for most tools (35/37, 95%). The tool’s ability to distinguish between clinical groups was reported for different clinical groups. For example, group differences were assessed for (1) large-contrast groups (CU vs dementia, eg, cCOG [52], e-CT [55], UCSF-BHA [90], and JustTouch [54]) and (2) smaller-contrast groups (eg, CompCog [74], digital Trail Making Test–Black and White [46], cCOG [52], e-CT [55], UCSF-BHA [90], and JustTouch [54] for CU vs MCI and JustTouch [54] for MCI vs dementia).

Reliability and Responsiveness

Test-retest reliability was examined for a minority (12/37, 32%) of the digital tools. The methods used to assess test-retest reliability were (1) intraclass correlation coefficients (ICCs; ARC [22]; Inbrain CST [83]; Miro Health platform [85]; the mobile screening test system for MCI [87]; SIMBAC [63]; and Survey for Memory, Attention, and Reaction Time [89]), (2) correlations (ie, Pearson correlations: BRANCH [51], cCOG [52], Computer Assessment of Mild Cognitive Impairment [79], SIMBAC [64], and WLA [72]; Spearman rank partial correlations: ki:e SB-C [71]), or (3) linear models to assess performance differences between test and retest assessments (C3 [80]). The settings in which test-retest reliabilities were assessed differed across the studies that were on-site (C3 [80], Computer Assessment of Mild Cognitive Impairment [79], and Inbrain CST [83]), remote (WLA [72]), or both on-site and remote (cCOG [52]). The test-retest reliability differed across tools and was largely dependent on the outcome score. It is inherent to digital tools that they generate large amounts of outcome scores, for each of which reliability features could be reported. This results in a range from low to high reliability coefficients within a tool, as reflected in ICCs that ranged from 0.49 to 0.91 for Inbrain CST [83] and Pearson correlations that ranged from 0.24 to 0.82 for cCOG [52] or from 0.38 to 0.83 for WLA [72]. For other tools, high reliability was consistently found for multiple scores, such as for the mobile screening test system for MCI [87], where ICCs ranged from 0.97 to 0.98.

For 5% (2/37) of the tools, both test-retest reliability and responsiveness were assessed (ki:e SB-C [71] and WLA [72]). Responsiveness was reported for 16% (6/37) of the tools (Altoida [56], C3 [81], ki:e SB-C [71], LORE [78], UCSF-BHA [91], and WLA [72]). The methodologies used to assess changes in digital scores over time were linear mixed-effects models (C3 [81], LORE [78], UCSF-BHA [91], and WLA [72]), a computation of a change score (ki:e SB-C [71]), or a longitudinal intraindividual variability metric that was compared between diagnostic groups (Altoida [56]). If a digital tool generated multiple outcome scores, the responsiveness could vary depending on the specific score. For example, for the WLA [72], rates of decline in 4 aggregate scores were assessed, where 1 of those scores was found to decline more rapidly for individuals with MCI or early AD than for individuals with Montreal Cognitive Assessment scores above the threshold for cognitive impairment (≥26). For another tool (C3 [81]), 8 subtask scores and 1 composite score were generated, with most of these scores showing responsiveness to change over time, and change in 4 of these scores was reported to be associated with amyloid or tau protein burden. For 2 tools (C3 [81] and LORE [78]), improvement over time was demonstrated, where individuals without cognitive impairment with AD pathology showed less steep learning curves than controls, indicating that diminished learning curves may also be a promising indicator of cognitive decline.

Interpretability

Clinically meaningful (changes in) scores were not defined for any of the digital tools (0%). However, for 11% (4/37) of the tools, information on clinical meaningfulness or related aspects was argued to have been demonstrated, whereas defined meaningful scores were not provided such that clinical meaningfulness was not assessed according to its definition as provided in the Methods section. C3 [80] was demonstrated to be associated with AD biomarkers and the PACC such that the authors concluded that this tool captures meaningful cognitive decrements. The ki:e SB-C [71] tool was associated with scores on the Clinical Dementia Rating scale and differed both between diagnostic groups and between those who cognitively declined versus those who did not, which the authors interpreted as support for the tool’s clinical relevance. For LORE [78], the authors argued that diminished learning curves on the digital measurement were meaningful, whereas further research is required to quantify its clinical meaningfulness. Hence, although the term “clinically meaningful” was used in these studies, clinical meaningfulness was not examined according to its definition, which presupposes a defined meaningful score [16,29].

Discussion

Principal Findings

In this scoping review, we identified 50 studies covering 37 different smartphone- and tablet-based tools to assess cognition in individuals with preclinical AD or MCI. These numbers indicate that multiple smartphone- and tablet-based cognitive tools are currently being developed, whereas only a limited number of validation studies in this target population have been conducted for individual tools. Specifically, except for construct validity, most psychometric properties (ie, structural validity, reliability, responsiveness, and clinical meaningfulness) have not been extensively assessed to date. Thus, considering the foundations of measurement, as reflected in the psychometric properties, these tools are still in the early stages of validation. Moreover, it was observed that practical characteristics such as administration time or the investigative setting were not consistently reported. Although the initial reporting on the psychometric properties of smartphone- and tablet-based tools to assess cognition in early AD support their potential, further and careful validation of their psychometric properties, as well as clear information on practical characteristics, is essential to facilitate implementation.

Practical tool characteristics may be argued to be important toward considering the tool’s use in clinical practice, supported by previous observations that, for health care professionals, time efficiency would facilitate the use of digital tools [26]. We highlight that, for a significant number of tools, information was not reported regarding administration time (14/37, 38%), the testing design (single or repeated; 9/37, 24%), or the investigative setting (on-site or remote; 7/37, 19%). This is concerning as information on the setting is crucial to interpret whether the psychometric properties assessed are applicable for on-site or remote use of a tool and for a single (eg, annual) or repeated (eg, daily or monthly) measurement. Especially given that the advantages of digital tools include their time-efficient administration and potential for remote and highly frequent administration, we recommend that such characteristics be clearly reported.

The generation of high-dimensional data is inherent to digital tools that assess cognition. While this may be regarded as an advantage, we observed that it also poses challenges to the overall evaluation of the quality of a tool and complicates one-to-one comparisons between tools. For instance, it remains challenging to decide which scores should be evaluated and compared, which raises questions such as whether the psychometric properties of single scores or aggregated scores should be considered when appraising the quality of the digital tool and whether the tool’s quality is determined by the outcome score that holds the lowest or highest psychometric quality. These challenges of high-dimensional datasets were specifically observed in the psychometric properties of construct validity and test-retest reliability, which widely ranged for the number of generated scores of a single tool. We recommend that the promising large amounts of generated data be reduced carefully by selecting those outcome scores that hold high psychometric quality, which we consider a crucial step to develop digital cognitive tools that provide clinically useful outcome scores. Another observation was that different methodologies were used between tools to determine their psychometric properties. This highlights the need for more uniformity regarding psychometric evaluation, which would facilitate the comparison of digital tools regarding their psychometric quality, which is needed to enable the end user to select the most appropriate tool for its context of use.

To standardize the process of clinical evaluation of digital tools, we propose a more widespread use of operating procedures, such as the FDA guidance on software as a medical device [95]. This guideline recognizes the importance of psychometric property evaluation and recommends 3 steps. The first step concerns clinical association, where the digital outcome should be demonstrated to be associated with the targeted clinical condition (ie, clinical decline as a result of AD). The second evaluation step regards analytical validation of the software, which is out of scope for this review. The third step involves clinical validation, where the digital outcome should be demonstrated to be reliable and clinically meaningful with regard to its intended purpose. The importance of provided information on psychometric properties is further emphasized by the previously reported notion that clear information on psychometric properties would facilitate the use of digital tools in clinical practice [26]. To this end, we will discuss the reporting on individual psychometric properties in the following paragraphs.

A first psychometric property that could be argued to be placed at the first step of the clinical evaluation process regards the conduct of factor analyses to confirm that outcome scores adequately reflect the dimensionality that is measured (ie, the underlying structure of one or multiple cognitive domains). In this review, we observed that a factor analysis on outcome scores was conducted for a limited number of digital tools (3/37, 8%) to demonstrate the underlying structure of the measured construct. This is concerning, and it is advisable for future validation studies to increasingly focus on conducting factor analyses to ensure the digital tool’s structural validity.

Another psychometric property regarding demonstrating clinical associations is construct validity, which was reported for most digital tools (36/37, 97%). Correlation levels between generated digital scores and scores on gold-standard measures of cognition ranged significantly both within and between tools, similarly to previous findings regarding self-administered computerized cognitive assessments [31]. It has been highlighted that, as currently used cognitive tests may not be suitable for early AD stages, validation against these “gold standard” tests may be particularly relevant in more progressed stages, whereas validation against biological measures of AD may be more relevant in preclinical stages [16,17,20]. Recently, proposed revisions to the diagnostic criteria for AD have been published, with the common denominator that preclinical AD concerns a stage characterized by the absence of symptoms [96,97]. Hence, to demonstrate an association with the disease in this stage, biological validation is needed instead of clinical validation, for which no gold-standard test exists at this stage. This importance of biological validation in the preclinical AD stage was reflected in our observation that validation against biological measures was reported for fewer tools (10/37, 27%) and all tools that were assessed in the preclinical stage were validated against biological measures. Still, most tools (36/37, 97%) were validated against clinical measures, including tools that were examined in preclinical AD. In the absence of a true gold standard for preclinical AD, it remains concerning that novel sensitive tests for preclinical populations are nonetheless validated against existing tests with limited sensitivity. As suggested previously, predictive validity might be considered as an alternative “copper standard” [16]. As we did not evaluate the reporting on predictive validity, further research should examine the ability of smartphone- and tablet-based tools for use in early AD stages to predict future cognitive decline or progression to MCI or dementia.

The third clinical evaluation step, clinical validation, involves demonstrating reliability. Clinical trials examining potential intervention effects inherently require repeated assessments over time, and to accurately interpret change in cognition over time, the test-retest reliability of an outcome measure should be ensured in the target population. The test-retest reliability was examined for a limited number of digital tools included in this review (12/37, 32%). As we selected studies that focused on the MCI and preclinical stages, it may be the case that test-retest reliability was assessed in studies focusing on other populations, including, for example, individuals with dementia. However, as test-retest reliability may differ across populations, it is important that this psychometric property is assessed in the target population. The importance of ensuring the tool’s reliability may be emphasized even more in the context of self-administered digital tools that may be more susceptible to noise than supervised paper-and-pencil tests. For the tools that had available information on the test-retest reliability, the reliability levels ranged substantially, largely depending on the outcome score of a tool (eg, total score or single scores), in line with previous observations [31]. This makes it highly challenging to determine the overall reliability of a specific tool and stresses the importance of careful selection of outcome scores. We recommend that the test-retest reliability should be thoroughly considered before selecting outcome scores for further validation steps. In addition, as the large absence of information on this psychometric property may hinder the implementation of digital tools, validation studies should increasingly focus on evaluating test-retest reliability. Alternatively, although we note that the independent validation studies are preferable, we recommend that the evaluation of test-retest reliability also be incorporated into clinical trial designs where, for example, the natural course of a digital outcome should be attested first before investigating its change after the intervention.

To determine whether a digital cognitive tool is sensitive to measure potential treatment effects or cognitive changes over time for monitoring purposes, information on the tool’s responsiveness is essential. Responsiveness was reported for a small number of tools (6/37, 16%) using different methodologies, and findings varied within the outcome scores of a single tool. Interestingly, of these 6 tools, 2 (33%) showed that less steep improvement was associated with preclinical AD [78,81]. Thus, although it may be expected that declining performance would be indicative of cognitive change, these new digital repeated testing approaches showed that diminished learning curves may also serve as an indicator of cognitive change in individuals with preclinical AD. Our finding that responsiveness was evaluated to a lesser extent may be explained by the fact that it requires a more effortful, longitudinal study design. In addition, this low number of tools for which responsiveness was reported may reflect the relatively young character of the research field involving digital tools to assess cognition in early AD stages, which may indicate that these tools are not ready for clinical implementation yet and warrant further validation. Accordingly, it is recommended that future validation studies focus on examining the responsiveness of digital outcome scores to change over time in the target population as an essential step toward clinical implementation.

Another aspect of the third clinical validation step regards clinical meaningfulness. To enable clinical interpretation of digital scores and clinically relevant change, the clinical meaningfulness of scores and their change should be determined. However, no consensus has been established yet on what score should be considered clinically meaningful [98]. The importance of clinically meaningful end points has also been stated by the US Food and Drug Administration draft guidance from March 2024 [99], where it has been recognized that, in early AD stages, it may be challenging to demonstrate clinical meaningfulness due to subtle cognitive changes and the absence of overt cognitive impairment. With the lack of a clear definition of clinical meaningfulness in early AD stages, this quality characteristic may be easily overlooked in the validation of digital cognitive tools, as reflected in our observation that clinically meaningful scores were not defined for any of the identified digital tools. However, it should be noted that, for some digital outcome scores, it was argued that their meaningfulness was supported by associations with AD biomarkers and clinical measures (PACC or Clinical Dementia Rating), as well as by scores or learning curves that differed between diagnostic groups [71,78,80]. Such definitions of clinical meaningfulness related to clinical, biological, and discriminative validity seem to refer to the relevance of the measure rather than the magnitude of the score or change in score that demonstrated clinical meaningfulness [100]. Hence, there is a pressing need for a clear definition of clinical meaningfulness, and it has been suggested that this definition should be developed by incorporating perspectives of various stakeholders (eg, patients, clinicians, and regulatory bodies) [98]. In addition, clinical meaningfulness may differ across patients, for which digital tools may offer an opportunity to determine personalized end points. Moreover, norm scores or cutoff scores may be considered toward establishing clinical meaningfulness. Thus, it is recommended that, within the field concerning cognitive assessment in early-stage AD in general, attention should be paid to reaching consensus on the definition of clinical meaningfulness of digital outcome scores to enable clinically useful interpretation of digital outcomes in future validation studies.

The major strength of this review was that we focused on the preclinical AD and MCI stages. These predementia stages offer the greatest potential for benefit from intervention strategies as well as having the highest need and offering the greatest feasibility in the context of novel digital remote assessments. Another strength was that we used a systematic search to identify validation studies of digital tools to assess cognition, which allowed for a large selection of studies. In addition, as our scope covered a range of psychometric properties that were selected based on previous recommendations, as well as practical tool characteristics that enabled the identification of an “information gap” in both areas, this allowed for the provision of a clear overview on what information is missing and, thus, what should be focused on to support the implementation of digital tools.

This review has a number of limitations. First, although we used a broad and systematic search strategy to minimize the chance of missing studies, the identified studies relied on specific search terms such that potentially relevant studies might have been excluded. For example, studies were not identified within the search if keywords related to the digital aspect of tools, such as “digital,” “computerized,” “smartphone,” or “tablet,” were not used in the title or abstract. Second, we did not conduct quality assessments of the included studies, which may have potentially biased our overall conclusions. Third, because this is a rapidly developing field, new digital cognitive assessments will be continuously emerging, highlighting the need for ongoing evaluation. Therefore, our aim was not to provide an elaborate overview of currently available tools but rather identify possible information gaps on reported psychometric properties in the landscape of digital cognitive tools. In addition, our scope was limited to tools that had the potential for remote assessment. The evaluation of this characteristic was not based on a strict criterion but rather on the contextual information provided indicating the suitability for self-administration that implies the potential for remote administration, which may, thus, be slightly subjective. We decided not to formulate a stricter eligibility criterion for this remote design as, for most tools, it was not explicitly stated whether the tool was intended for remote assessment. As this is in fact important information, it is recommended that future validation studies provide clear information on the intended context of the digital tool. Moreover, as we focused on validation studies, we did not include development studies, nor did we assess additional information on the tools’ websites on the tool characteristics, such as administration time, intended use, or available languages. Therefore, we emphasize that the provided overview of such characteristics may not adequately reflect all practical information or languages available to date but rather provide an overview of all available information as reported in validation studies targeting a population of individuals with preclinical AD or MCI. However, it is crucial to include such information in scientific publications for end users to decide which tools are most appropriate for their context of use and to evaluate the quality of digital cognitive tools in future systematic reviews [101]. Furthermore, we did not evaluate the quality of the methods used to assess the psychometric properties, and it is subject to discussion whether any method used was appropriate. For instance, to demonstrate structural validity, confirmatory factor analyses are argued to be the preferred method over exploratory factor analyses [102-104]. Similarly, to evaluate the quality of the tool’s construct validity, it is of relevance whether the clinical or biological correlations or group differences assessed are in the hypothesized direction, and for evaluating responsiveness, the time frame (eg, days, weeks, or months) should be considered. We provided a first step to identify the reporting on psychometric properties regardless of their quality, but future research should focus on the quality of the methodology used to determine psychometric properties.

Conclusions

Our results indicate that the landscape of smartphone- and tablet-based cognitive tools for early AD could currently be placed at the first step in the validation process, as described in the Food and Drug Administration guidance on software as a medical device [95]. Following this guidance, it is recommended that, after establishing a valid clinical association, digital tools should be further validated with regard to their context of use. For the development of novel tools, we recommend clearly stating the digital tool’s intended use to determine the prioritized validation steps with regard to the most relevant psychometric properties. For instance, if a tool would be used for monitoring purposes, demonstrating responsiveness is crucial. For digital screening tools, sensitivity should be demonstrated, whereas for digital diagnostic tools, the specificity for a clinical condition should be validated. As we identified multiple gaps in the available information on crucial psychometric properties, which halts the clinical validation of these tools, this stresses the need for more attention on psychometric quality, where the selection of outcome scores should be considered carefully and reported consistently. As a next step, future studies should determine the psychometric quality of currently developed digital tools for use in early AD stages. Ensuring psychometrically high-quality outcome scores must be a central theme in the development of smartphone- and tablet-based tools that assess cognition in individuals with preclinical AD and MCI. Further clinical validation tailored to the context of use will pave the way for the implementation of digital cognitive tools.