Background: With the advent of smart sensing technology, mobile and wearable devices can provide continuous and objective monitoring and assessment of motor function outcomes.
Objective: We aimed to describe the existing scientific literature on wearable and mobile technologies that are being used or tested for assessing motor functions in mobility-impaired and healthy adults and to evaluate the degree to which these devices provide clinically valid measures of motor function in these populations.
Methods: A systematic literature review was conducted by searching Embase, MEDLINE, CENTRAL (January 1, 2015, to June 24, 2020), the United States and European Union clinical trial registries, and the United States Food and Drug Administration website using predefined study selection criteria. Study selection, data extraction, and quality assessment were performed by 2 independent reviewers.
Results: A total of 91 publications representing 87 unique studies were included. The most represented clinical conditions were Parkinson disease (n=51 studies), followed by stroke (n=5), Huntington disease (n=5), and multiple sclerosis (n=2). A total of 42 motion-detecting devices were identified, and the majority (n=27, 64%) were created for the purpose of health care–related data collection, although approximately 25% were personal electronic devices (eg, smartphones and watches) and 11% were entertainment consoles (eg, Microsoft Kinect or Xbox and Nintendo Wii). The primary motion outcomes were related to gait (n=30), gross motor movements (n=25), and fine motor movements (n=23). As a group, sensor-derived motion data showed a mean sensitivity of 0.83 (SD 7.27), a mean specificity of 0.84 (SD 15.40), a mean accuracy of 0.90 (SD 5.87) in discriminating between diseased individuals and healthy controls, and a mean Pearson r validity coefficient of 0.52 (SD 0.22) relative to clinical measures. We did not find significant differences in the degree of validity between in-laboratory and at-home sensor-based assessments nor between device class (ie, health care–related device, personal electronic devices, and entertainment consoles).
Conclusions: Sensor-derived motion data can be leveraged to classify and quantify disease status for a variety of neurological conditions. However, most of the recent research on digital clinical measures is derived from proof-of-concept studies with considerable variation in methodological approaches, and much of the reviewed literature has focused on clinical validation, with less than one-quarter of the studies performing analytical validation. Overall, future research is crucially needed to further consolidate that sensor-derived motion data may lead to the development of robust and transformative digital measurements intended to predict, diagnose, and quantify neurological disease state and its longitudinal change.
Patient care is changing with the dawn of smart sensing technology. Mobile and wearable devices can provide continuous as well as objective monitoring and assessment of many health outcomes . Until recently, outcomes that represent various motor functions (ie, any movement of the entire body or part of the body that is controlled by motor neuron activity) have typically been measured by patient reports (eg, number of falls) or physician assessment (eg, gait abnormalities). Physician assessments are based on very brief observations in an office or clinic [ ], whereas self-reported outcomes are subjective and often not as sensitive nor as supervised as in-clinic measures [ ]. Finally, measurements may vary between assessors depending on the level of training, familiarity, and experience [ , ].
Wearable technologies have recently emerged as a potential supplemental source of data on motor function. Such technologies could increase the objectivity and ease of assessment for motor functions during clinical trials and care while also allowing for a richer dimension of data to be captured. Real-world and continuous monitoring of patient motor functions through wearable and mobile sensors is increasingly being investigated in areas such as disease progression through motor fluctuations in Parkinson disease , detection of amyotrophic lateral sclerosis [ ], and tremor activity in essential tremor [ ].
Data from digital measurement solutions can enhance the quality of clinical trials, as illustrated by the acceptance of wearable device–measured stride velocity (95th percentile) by the European Medicines Agency (EMA) as an end point in Duchenne muscular dystrophy . Given the implications these new data courses could have on the field, the current regulatory environment for mobile technologies is in flux [ ]. US and European regulatory bodies are responding to this emerging opportunity by adapting their regulatory processes to these technological advances [ ].
Previous reviews have described the characteristics of their patient samples and sensors involved in collecting motor function data [- ]. However, they do not evaluate the degree of validity produced by such sensors. This review follows the terminology used in previous reviews [ , ] and differentiates between analytical validation (ie, the same motion behavior is measured by an independent source and compared with the sensor-derived motion behavior) and clinical validation (ie, a clinical characteristic or measure of interest is measured and compared with the sensor-derived motion behavior). Gaining insight into the current clinical validity and utility of the data captured by mobile and wearable sensing technologies is of utmost importance. So, the aim of this study was to describe the existing scientific literature on digital measurement solutions that are being used or tested for assessing motor functions in mobility-impaired and healthy adults and to evaluate the degree to which these tools provide clinically valid measures of motor function in these populations. Specifically, we aimed to answer the following research questions: (1) What types of digital devices exist that capture motor function in mobility-impaired and healthy populations? (2) In what types of studies and in what populations have these devices been evaluated? (3) What outcomes do these digital devices measure? (4) What types of technologies and algorithms are used to capture and store the data? (5) To what degree have these technologies and their output been validated using established and recognized criteria?
This review was conducted in accordance with the Cochrane Handbook for Systematic Reviews of Interventions , and reporting is based on the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines [ ]. We included clinical trials (randomized and nonrandomized) as well as observational studies (case-control, retrospective cohort, prospective cohort, and cross-sectional) that provided validity estimates from wearable or mobile technologies to assess motor functions in adults (aged ≥18 years). Studies published in English after 2015 were included to focus on the most advanced technologies that are being used to assess motor function.
Study eligibility criteria were defined using an adapted PICO (Population, Intervention, Comparator, Outcomes) framework. We applied criteria based on the technology instead of the intervention or comparator, as the research question focused on the validity of measurement and not treatment efficacy (Table S1 in[ - ]).
A systematic literature search was conducted (January 1, 2015, to June 24, 2020) in the MEDLINE, Embase, and CENTRAL databases. Searches of relevant conferences for the last 3 years (2018-2020) were conducted via Embase. Search strings are available in Tables S2-S6 in. Gray literature searches were also conducted to capture studies from sources that were not included in the main literature databases, which included the US Food and Drug Administration website as well as the United States and European clinical trials registry databases for clinical trials which had reported results but were not published in peer-reviewed journals (for the years 2018-2020).
After duplicate removal, all titles and abstracts were screened for potential eligibility according to the prespecified PICO criteria, after which full-text articles were assessed using the same criteria. Study selection was performed by 2 independent reviewers, and disagreements were resolved through discussion. If no consensus could be achieved, a third researcher was consulted for arbitration.
A total of 2 independent reviewers extracted all relevant data from the final list of included studies. A reconciliation phase was again deployed to resolve any discrepancies between the reviewers, and a third reviewer intervened to resolve any remaining conflicts. The following data were extracted where available: (1) authors, year of publication, country, study setting, and follow-up period; (2) study design; (3) participant characteristics; (4) outcomes; (5) technology characteristics; and (6) validity outcomes. Motor function outcomes were manually sorted into categories by reviewers to facilitate summary where necessary.
A total of 2 independent reviewers assessed the quality of the included studies using the ROBINS-E (Risk Of Bias In Nonrandomized Studies of Exposures) tool . A third investigator intervened to reach consensus if there were any remaining unresolved discrepancies following reconciliation between the decisions of the 2 reviewers.
Effect size estimates were extracted from each study where reported, including standardized mean differences (ie, Cohen d), correlation coefficients (eg, Pearson r), sensitivity, specificity, accuracy, and area under the curve (AUC). In cases where studies provided none of these aforementioned effect size classes, effect sizes were calculated based on the information available in the manuscript using standard formulas [, ]. To facilitate comparison across the studies, extracted effect sizes were converted to Pearson r–based effect size estimates where possible. This extraction and conversion process allows for studies to be directly compared via r-based effect sizes, estimates of sensitivity and specificity, and estimates of accuracy. The average effect sizes were calculated across all studies as well as by specific study and sample characteristics of interest. As r is bound by −1 and +1, rs were transformed into Zr using the procedure described by Fisher for analyses [ , ] and then back-transformed for reporting. Differences across groups in the magnitude of obtained effect sizes were tested using restricted information maximum likelihood derived SEs [ ] using the inverse variance weight [ ]. A random effects approach was taken, which includes in the denominator an extra variance component representing true variation in the population from which the included studies can be considered a random sample. A significance threshold of .05 was used to determine if values significantly differed between groups.
A total of 9940 abstracts were identified from the electronic databases, and 2 articles [, ] were included from handsearching of a systematic review identified in our searches [ ]. After the removal of duplicates and exclusion based on title and abstract screening, 436 records remained for the full-text screening. A list of the records excluded during full-text screening and the reason for exclusion are provided in Table S7 in . A total of 91 publications describing 87 primary studies fulfilled all inclusion criteria ( ).
Study Characteristics and Data Collection
Across the 87 studies (n), the most common country settings reported were the United States (n=15) [- ], United Kingdom (n=10) [ - ], Italy (n=5) [ - ], Spain (n=4) [ - ], South Korea (n=4) [ - ], Germany (n=3) [ - ], and Japan (n=3) [ - ]. At least 1 study was conducted in each of the following countries: Canada (n=2) [ , ], the Netherlands (n=2) [ , ], Portugal (n=2) [ , ], Sweden (n=1) [ , ], Taiwan (n=2) [ , ], Australia (n=1) [ ], Brazil (n=1) [ ], Demark (n=1) [ ], France (n=1) [ ], Israel (n=1) [ ], Greece (n=1) [ , ], Lithuania (n=1) [ ], Norway (n=1) [ ], and United Arab Emirates (n=1) [ ]. Of the remaining reporting studies, 6 were multinational [ - ]. Sample size ranged from 8 [ ] to 1465 [ ] (median 40.5 participants). A total of 7995 participants were enrolled in the included studies. Table S8 in presents the list of included publications as well as key study characteristics.
All 87 studies were observational in nature. Most studies (n=50) did not report whether the study was conducted in a single-center or multicenter setting. However, among those that did report, 20 and 17 studies were single center and multicenter, respectively. Approximately half of the included studies were conducted in a laboratory setting (n=42), 11 studies were home based, and 15 were a combination of a laboratory-based and home-based setting. The remaining 19 studies did not specify the study setting. The included studies were categorized into 2 follow-up types: cross-sectional (n=62) with a follow-up period of ≤1 week and longitudinal (n=25) in which participants were followed up for ≥1 week. Follow-up length of longitudinal studies ranged from 7 days [, , , , ] to 8 years [ ]. A total of 30 studies reported the time allocated for data collection; in other words, the time needed to collect data in one session of data collection. In addition, 18 studies were able to capture their data in a session between 20 seconds [ , ] and 24 hours [ ]. Moreover, 13 studies required their participant to use the device for multiple days for their collection period, which ranged from 2 [ , ] to 14 consecutive days [ ]. This review follows the terminology used in previous reviews [ , ] for analytical validation (ie, the same motion behavior is measured by an independent source and compared with the sensor-derived motion behavior) and clinical validation (ie, a clinical characteristic or measure of interest is measured and compared with the sensor-derived motion behavior). Analytic validation was only performed in 21% (13/62) of cross-sectional studies and 4% (1/25) of longitudinal studies. Most of these studies performed clinical validation of sensor-based motion data. Studies applied a wide variety of technologies to capture motion outcomes. Motion data were captured by ≥30 different devices, including novel wearables (18/42, 43% devices), smartphone or smart watch (13/42, 31%), mass market digital technology (7/42, 17%), other digital technology (eg, PC; 3/42, 7%), and mass market wearables (1/42, 2%). Approximately 1 in 5 studies included a mass market device.
In terms of quality, studies were generally low to moderate risk of bias (; Table S9 in ). Less than 20% (14/42) of studies did not show that groups were balanced in terms of key baseline characteristics and were considered high risk for confounding. The risk of bias arising from measurement of the exposure was most often low because exposures were generally whether the patient had a disease or was healthy, and misclassifications were next to nonexistent. For the domain of selection of participants into the study, studies were often high risk of bias. Disease diagnosis (ie, the exposure) did not generally coincide with the start of follow-up, and the diseases being studied could fluctuate over time. Many of the studies relied on volunteers to participate in the study, and this may have led to participants entering the study if they were in a particularly good or bad disease state (eg, Parkinson disease has on and off states). Furthermore, no corrections that may have alleviated selection biases in the analysis were conducted. Studies were generally low risk with regard to the domain concerned with the risk of bias owing to postexposure interventions. By design, the included studies did not administer interventions to alleviate the effects of exposures, and therefore, bias was not a concern. Regarding missing data, this was not often accounted for, leading to high risk of bias in that domain. However, studies were generally low risk of bias for measurement of outcomes, as motor function outcomes were assessed objectively and similarly across groups. Finally, over half of the studies were rated low risk for selection of the reported result.
Concepts of Interest and Context of Use
Approximately half of the included studies compared the association between sensor-derived motion data and a standardized clinical assessment across diverse disease conditions (n=44). Other studies compared mobility-impaired diseased participants to a healthy control group of participants with no mobility impairment (n=43). The most represented disease condition was, by far, Parkinson disease (n=51); stroke (n=5); Huntington disease (n=5); and depression, cognitive impairment, cerebral palsy, and multiple sclerosis (n=2 for each). All other disease groups were only represented in a single study.
Among the 67 studies that reported the mean age of participants, values ranged from 23.6 years  to 77.2 years [ ] for mobility-impaired participants and from 19.5 years [ ] to 78.9 years [ ] for healthy participants. Control groups were generally well-matched by participant age and sex. Among the 71 studies that reported the proportion of males or females in their sample, the average percentage of the sample that were male ranged from 22.8% [ ] to 100% [ , ] in mobility-impaired participants and from 11% [ ] to 100% [ ] in healthy participants. Studies with the largest sex imbalances were those addressing the less frequently studied disease states (ie, represented in only 1 or 2 studies). In contrast, Parkinson disease, Huntington disease, and stroke reflected a more balanced representation of females and males.
The primary motion outcomes were gait (n=30), gross motor movements (n=25), fine motor movements (n=23), motor symptom severity (n=9), bradykinesia (n=7), motor fluctuations (n=6), dyskinesia (n=5), balance control (n=5), postural stability (n=4), voice or speech impairments (n=3), facial expression impairments (n=1), and nocturnal movements (n=1). A summary of commonly reported outcomes by disease that the outcome was measured in is provided in.
The most common motions that participants were required to enact for sensor data collection across these studies were based on diverse active motor tasks: multimovement tasks (16/87, 18%) including balancing and reaction time during tests such as the Timed Up and Go, the Cognitive Dual Task Timed Up and Go, and the Manual Dual Task Timed Up and Go, unscripted daily activities (17/87, 20%), walking (10/87, 11%), tapping (9/87, 10%), and scripted activities of daily living (7/87, 8%). Less commonly used motions (<5% of studies) included several real-world tasks such as reaching, sit-to-stand motion, seated tremors, wrist pronation-supination tracing or pointing, typing, seated conversation, standing, and sleeping movement. Together, these motions were used to extract ≥75 distinct motion outcomes across the included studies. Most of these outcomes only appeared in one study and were only measured at a single sensor location in each study (per our inclusion criteria). One exception was walking cadence, with different studies measuring it using sensors worn at wrists, ankles, lower back, and chest and in the pants pocket. Additional exceptions were tremor, dyskinesia, and bradykinesia (each measured using sensors placed on the wrists or ankles).
|Disease and motor function outcome category||Motor function outcome|
|Acquired brain injury|
|Gross motor impairment or performance and upper body|
|Fine motor impairment or performance and continuous motion|
|Fine motor impairment or performance and discrete motion|
|Fine motor impairment or performance and continuous motion|
|Fine motor impairment or performance and discrete motion|
|Gross motor impairment or performance and lower body|
|Gross motor impairment or performance and whole body|
|Motor symptom severity|
|Fine motor impairment or performance, discrete motion|
|Mild cognitive impairment|
|Fine motor impairment or performance and continuous motion|
|Fine motor impairment or performance and discrete motion|
|Fine motor impairment or performance and continuous motion|
|Fine motor impairment or performance and discrete motion|
|Gross motor impairment or performance and upper body|
|Gross motor impairment or performance and whole body|
|Motor symptom severity|
|Rapid eye movement (REM) sleep behavior disorder|
|Fine motor impairment or performance and discrete motion|
|Transthyretin familial amyloid polyneuropathy|
|Gross motor impairment or performance and upper body|
aIncluding Parkinson disease, Huntington disease, early dementia, cerebral palsy, and poststroke.
bIncluding Duchenne muscular dystrophy, limb-girdle muscular dystrophy, and spinal muscular atrophy.
Data Processing and Analysis
The process through which these researchers converted their raw data to validity coefficients is illustrated in. On collection of the raw data, 2 parallel processes were typically seen: outcome computation and algorithm or model development. Following the completion of these 2 processes, the model was subjected to either analytical or clinical validation.
In ≥90% of the studies, the raw data were first preprocessed before feature processing engineering and analyses. One preprocessing step frequently seen among these studies was the splitting of raw data into temporal epochs or slices. This was done because training an algorithm to detect movement features across long periods greatly reduced the algorithm’s validity. Data were trimmed by temporal position (eg, the beginning and ending of the motion recording) or based on extreme values (eg, outliers >4 SDs from the mean). Raw data were subjected to some form of standardization or transformation in ≥90% of the studies.
Although algorithm training (eg, feature selection and threshold determination) typically occurred using data across all participants, several studies took the approach of building the feature detection algorithm using data across all participants but then allowing each participant to vary in latter stages such as feature selection or determining thresholds [, , , ]. Validity estimates from this smaller group of studies were similar in magnitude to those studies that applied the same features and thresholds to the classification of all participants.
Researchers have to decide which of the hundreds of identified candidate features to treat as a signal (by retaining them in the model) and which to dismiss as mostly noise (by excluding them from the model). Relatively few studies clearly described whether they moved all detected features to the next analytic stage (feature selection), but some studies compared prediction based on all extracted features to prediction based on top-performing features [, ]. These studies reported that the inclusion of additional features did not guarantee a meaningful increase in algorithm performance or validity. One study using smartphones to assess Parkinson disease symptoms found AUC values >0.90 for 998 detected features, with a drop to 0.75 when based on the top 30 features [ ]. A second study of participants with Parkinson disease concluded, “Accuracies obtained using the 30 most salient features were broadly comparable with the corresponding sensitivity and specificity values obtained using all 998 features” [ ].
Algorithm or Model Development
The included studies showed no clear preference regarding algorithms for feature selection or classification, but the 2 most frequently applied approaches were support vector machines (12/87, 14%) and random forests (4/87, 5%). Authors of these studies were sensitive to the complications of trying to train a classification model with groups of different sizes, as most of the comparative studies included in this review include approximately equal sizes of participants with a disease or disorder and healthy controls.
No consistent pattern emerged from within-study comparisons of feature selection algorithms. A wrist-based sensor was able to detect upper limb movement among participants with pre-Parkinson disease best when using random forests relative to support vector machines and naïve Bayes . A smartphone app testing motor impairment found that both neural networks and boosting outperformed support vector machines and Fisher linear discriminant analysis [ ]. Not all motions required feature selection across studies (several needed only to define logic rules to estimate movement angles using geometry), and some studies used proprietary algorithms that were not described in detail. One study that studied freezing of gait among participants with Parkinson disease using a smartphone app found neural networks performed better than other bagging algorithms, including random forest, multilayer perception, decision tree, support vector machine, and naïve Bayes [ ]. Another study on motor symptoms among participants with Parkinson disease using ankle-worn sensors found that support vector machines performed better than logistic regression and decision trees [ ]. Using smartphone motion data to predict motor impairment among participants with Parkinson disease, another study found that random forests based on Ridge regression outperformed those based on Lasso, or Gini impurity, and that linear support vector machines outperformed logistic regression and boosting [ ]. The sole consistent pattern that emerged was that supervised machine learning techniques performed better than unsupervised techniques (eg, naïve Bayes).
Analytical and Clinical Validation
The most common validity criterion was clinical condition (37/87, 43%), which was used in many of these studies to establish known-group construct discriminant validity of sensor-derived motion data by comparing participants with a diseased condition to healthy controls (Table S10 in). The second most common validity criterion was the clinical validity established by assessing the convergence or concurrence with traditional standardized clinical assessments (30/87, 34%; eg, Wolf Motor Function Test and Unified Parkinson Disease Rating Scale). Other criteria were clinician ratings (7/87, 8%), research device (9/87, 10%), treatment status (3/87, 3%), and patient-reported outcome (1/87, 1%). Longitudinal studies were more likely to use nonsupervised assessments, whereas cross-sectional studies were more likely to use clinician-supervised assessments.
Across studies, motion data from the sensors identified showed an average Pearson r clinical validity coefficient of 0.52 ([ , , , - , , , , - , , , - , , , - , , , , - , , , , , - , , , ]). Among the studies that did not provide sufficient information to calculate a Pearson r, the average validity was 0.83 (sensitivity), 0.84 (specificity), and 0.90 (accuracy). These values could be interpreted as very good [ ]. The magnitude of validity coefficients did not vary (P=.10) between health care–related devices (mean r=.47), personal electronic devices (mean r=.44), and entertainment consoles (mean r=.63). Validity coefficients for motor function generated by healthy adults were higher than those generated by participants with a disease state or impairment (z score 3.19; P=.001). The only statistical decision that consistently predicted higher validity coefficients was the decision to trim observations during the preprocessing stage based on value (ie, outliers; z score 2.10; P=.04). There was no difference in validity coefficients across trimming observations based on temporal placement, transforming data, standardizing data, or which feature detection and validation analyses were used. The funnel plot from these studies was asymmetrical in a manner consistent with bias toward higher coefficients (Figure S1 in ). The magnitude of validity coefficients did not significantly vary across the different device types ( ).
Taken as a whole, no consistent pattern emerged from within-study comparisons of the relative analytic validity of any specific motion signal. One study using Kinect found high Pearson r validity coefficients (r>0.50) for more than 40 distinct motion outcomes but very low validity coefficients for a handful including deflection range roll (measured in degrees), mean sway velocity roll (measured in degrees per second), and up-down deviation (measured in centimeters) . A second study using Kinect found Pearson r validity coefficients above 0.50 for variables related to steps taken, distance, and speed but coefficients below 0.50 for variables related to angles (eg, trunk, hips, ankle, trunk, upper limb, and full body) [ ]. A third study using a triaxial accelerometer worn on the waist found Pearson r validity coefficients above 0.50 for gait, arising from chair, body bradykinesia, hypokinesia, and overall posture and validity coefficients below 0.50 for rigidity of lower and upper extremities axial rigidity, postural stability, legs agility, and tremors in lower or upper extremities [ ]. These numbers are in the same range as single items from widely established clinical tools [ - ]. As the validity coefficients for these single motions were moderate, it reinforces the need for future studies and clinical applications to include multiple validated motion signals for any screening or diagnostic tool to achieve adequate levels of composite test validity.
Regarding clinical validation, no clear within-study evidence emerged regarding the relative superiority or inferiority of motion data captured in laboratory settings versus data captured in home settings (). For example, 1 study comparing typing behavior of participants recently diagnosed with Parkinson disease to the typing behavior of healthy controls found AUC values of 0.76 (when administered at home) versus 0.83 (when administered in clinic) [ ]. A second study comparing participants with Parkinson disease to healthy adults on motor function during an activities of daily living task found slightly higher accuracy, sensitivity, and specificity when the task was completed at home [ ].
|Are there differences in the validity of sensor-derived measures of motor function as captured||Between-study (ie, meta-analytic) findings||Within-study findings|
|Using mass market devices vs medical sensors?||Insufficient data to evaluate|
|At specific sensor locations?||Insufficient data to evaluate|
|home vs in the laboratory?||No; 1 study found AUCa values of 0.76 (when administered at home) vs 0.83 (when administered in clinic) . A second study found slightly higher accuracy, sensitivity, and specificity when the task was completed at home [ ].|
|In longitudinal vs cross-sectional studies?||No; One study found high Pearson r validity coefficients (r>0.50) for over 40 distinct motion outcomes but very low validity coefficients for a handful, including deflection rage roll (measured in degrees), mean sway velocity roll (measured in degrees per second), and up-down deviation (measured in centimeters) . A second study found Pearson r validity coefficients above 0.50 for variables related to steps taken, distance, and speed, but coefficients below 0.50 for variables related to angles (eg, trunk, hips, ankle, trunk, upper limb, and full body) [ ]. A third study found Pearson r validity coefficients above 0.50 for gait, arising from chair, body bradykinesia, hypokinesia, and overall posture and validity coefficients below 0.50 for rigidity of lower and upper extremities axial rigidity, postural stability, legs agility, and tremors in lower or upper extremities [ ].|
|In healthy vs motor impaired patients?||Insufficient data to evaluate|
|Using different feature detection algorithms?||No; One study was able to detect movement best when using random forests relative to support vector machines and naïve Bayes . A second study found that both neural networks and boosting outperformed support vector machines and Fisher linear discriminant analysis [ ]. A third study found neural networks performed better than other bagging algorithms including random forest, multilayer perception, decision tree, support vector machine, and naïve Bayes [ ]. A fourth study found support vector machines performed better than logistic regression and decision trees [ ]. A fifth study found that random forests based on Ridge regression outperformed those based on Lasso, or Gini impurity, and that linear support vector machines outperformed logistic regression and boosting [ ]. The sole consistent pattern that emerged was that supervised machine learning techniques performed better than unsupervised techniques (eg, naïve Bayes).|
|Using particular motion sensor signal types?||Insufficient data to evaluate|
|Using all vs a subset of features?||No; One study found AUC values >0.90 for 998 detected features, with a drop to 0.75 when based on the top 30 features . A second study concluded “Accuracies obtained using the 30 most salient features were broadly comparable with the corresponding sensitivity and specificity values obtained using all 998 features” [ ].|
|With the thresholds held constant across patients vs patient-specific thresholds?||No; Although algorithm training typically occurred across a sample, several studies took the approach of starting the algorithm (feature detection) using data across all participants but then allowing each patient to vary in later stages such as feature selection or determining thresholds [, , , ]. Validity estimates from this smaller group of studies were similar in magnitude to those studies that applied the same features and thresholds to the classification of all participants.|
|Using clinically supervised vs nonsupervised assessments of patient clinical status?||Insufficient data to evaluate|
|With outliers trimmed vs retained in the feature detection stage?||Insufficient data to evaluate|
|With transformed data vs untransformed data?||Insufficient data to evaluate|
|With standardized data vs unstandardized data?||Insufficient data to evaluate|
aAUC: area under the curve.
To our knowledge, this is the first systematic literature review to evaluate the degree to which wearable and mobile technologies provide clinically valid measures of motor function in mobility-impaired and healthy adults. The identified literature generally consisted of proof-of-concept studies, which aimed to pilot a device and assess whether it could validly measure motor functions. Consequently, most studies used a short follow-up period (<1 week) and had a total sample size of <50 participants. Unsurprisingly, many of the longitudinal studies prioritized nonsupervised measures. Even so, taken together, these studies provide a respectable evidence base supporting the potential these movement sensors have to inform clinical practice.
As the eligibility criteria for our review were inclusive in terms of population, we identified a large range of disease types, which were all but one (chronic obstructive pulmonary disease) nervous system condition (); however, the most common disease was Parkinson disease, with stroke and Huntington disease coming in a very distant second and third place. The strong focus on Parkinson disease in this literature may be because of its prevalence or perhaps because motor function symptoms are a major characteristic of Parkinson disease for diagnosis and prognosis assessment purposes, making Parkinson disease an ideal model disease for testing the use of mobile technologies [ ]. However, it is most probably a mixture of these 2 hypotheses. Parkinson disease is also one of the few diseases with Food and Drug Administration–approved devices (eg, NexStride and Personal KinetiGraph), which assesses motor function to inform treatment decisions. The field would benefit from additional study of mobile technology–assessed motor function among other neurological diseases, including multiple sclerosis, spinal muscular atrophy, amyotrophic lateral sclerosis, and Alzheimer disease. In addition, future studies might consider the advantages of assessing digital devices per neurological impairment (such as difficulties in ambulation or upper limbs) rather than per disease.
Successful integration of wearable-based movement data into clinical tools requires both analytic validation and clinical validation. However, most of the reviewed literature compared wearable sensor-derived motion data to omnibus measures of functioning or disease progression (ie, clinical validation). More studies need to perform analytic validation by comparing wearable sensor-derived motion data to the same motions measured by another source (eg, observer assessment and motion-capture technology). Observed motions may be highly correlated with omnibus assessments of motor skills or disease status (ie, clinical validation), but the foundation of approval as a clinical end point can only be met if the motions identified using the sensor have been shown to be the exact motions that have been approved by the governing or regulatory body. Using as an example the EMA’s recent approval of 95% stride velocity as an approved secondary end point in Duchenne muscular dystrophy, appeal to the EMA’s approval of wearable sensor stride velocity data as an end point for a given study requires evidence that when the used algorithm claims to measure stride velocity (95th percentile), there be evidence that the algorithm has, in truth, measured stride velocity. Future research in this area should focus their attention on analytic validation.
There was considerable variation in methodological approaches. The review revealed one of the key reasons why this field may still show such inconsistency in analytic approach; it is still developing. Evidence of this is seen in which motion variables could be identified by the algorithms. Despite the hundreds of motion-derived outcome variables identified across these studies, not all theoretically meaningful motions could be recovered. One study of participants with Parkinson disease concluded, “Unfortunately, we failed to find parameters that reflected fatigue (decrement response) and hesitation (intertap irregularity), which are characteristics of motor dysfunction in Parkinson’s disease” . Those authors offered that more precise definitions of fatigue and hesitation may be needed to recover them in clinical settings with a smartphone-based tapping test similar to the one used in that study. In addition, the motor functions viewed by some authors as theoretically relevant were occasionally overshadowed by nonmotor signals. The tendency for studies to report diminishing returns after a certain point for additional motion signals is statistically analogous to other clinical efforts to identify causal markers from a multitude of candidates, which revealed many initially flagged markers as spurious [ ]. Future studies should include graphical displays to identify inflection points (similar to the scree plot in factor analysis or the elbow plot in latent class analysis) to help show where the statistical signal (or true score) from additional motions becomes outweighed by statistical noise.
The moderate to high validity coefficients reported in the identified literature may support the potential for sensor-derived motor function data from digital health technology tools to eventually contribute to screening, diagnosis, and monitoring of neurological diseases in particular. No significant differences in analytic or clinical validity estimates were found when comparing data generated by mass market devices (eg, smartphones, smartwatches, and Fitbits), game consoles (ie, Nintendo Wii and Microsoft Kinect or Xbox), and marketed motion sensors (eg, ActiGraph, ActivPAL, Axivity, Dynaport, KinetiSense, Opal devices, and PAMSys-X). Furthermore, the motion data provided by these technologies produced equivalent validity estimates in laboratory-based and home-based settings. This further supports the future potential for digital measurement solutions to provide clinically meaningful data and eventually become the gold standard for assessing motor behaviors. The degree and rate of application for motor function data from these devices to clinical practice will depend on how soon clear evidence bases are established for given sensor locations for given movements of interest.
Translation of these motor signals into clinical application is aided by demonstrating sufficient validity outside the scripted protocols of a controlled laboratory setting. The reviewed literature showed that scripted motion tasks were important when only a few minutes of motion data were to be captured. Furthermore, motion data from unscripted everyday living with longer data collection periods were also shown to be adequate and deemed complementary, as episodic scripted assessments of confined tasks might not capture the complex spectrum of potentially altered components of motor function in an unconstrained ecologic setting .
As a whole, the reviewed literature revealed several best practices as well as a few cautionary tales for mobile or wearable sensor-based movement data. Although cross-validation techniques all seek to counteract the inflation of validity coefficients that can occur during machine learning techniques, they can produce different results . Despite these best practices, there remained indirect evidence of model overfitting in the form of some abnormally high validity coefficients in the final models (ie, specificity of 1.0, which is perfect) [ , ].
The reviewed literature also highlights areas to consider during the development of any clinical application. One illustration from this review is the critical role of thresholds , which require researchers to decide between manual versus automatic thresholds [ ] and global versus person specific [ ]. Leveraging the strengths of these modeling approaches while keeping them robust and flexible will be important to consider as they are scaled up to create clinical applications [ ].
Comparison With Previous Reviews
We identified a number of similar literature reviews during our study selection [- ]. All identified reviews synthesized their evidence qualitatively, and none provided a quantitative synthesis of the validity of motion data generated from these sensors among patients with neurological conditions. Of the 9 identified reviews, 1 was narrative [ ], whereas the remaining were systematic reviews. None of the systematic reviews focused on neurological disorders. Overall, 2 reviews focused specifically on swimming motions [ , ], 2 were focused on older adult patients with no specific disease [ , ], and 2 reviews focused on only upper [ ] and lower limb movements [ ]. Of the remaining 2 systematic reviews with similar objectives and scope to that of our own, the paper by Díaz et al [ ] aimed to review the current literature on the use of wearable sensors in gait, balance, and range of motion analysis. Diseases of participants also varied across their 56 included studies and included a mix of neurological disorders (eg, Parkinson disease, Alzheimer disease, and multiple sclerosis), as well as stroke, amputees, and healthy participants. Similar to our own review, the authors found that most body-worn devices were complex to use and required strong experience in data analysis to interpret the collected information. In addition, the authors pointed out a need for further validation and improvements in sensor systems for them to be used as reliable and accurate clinical devices. A second systematic review conducted by Kristoffersson and Lindén [ ] provided a qualitative synthesis of 73 published articles on wearable body sensors used for health monitoring. Similar to our review, the authors found that included studies were generally observational in design and small in sample size. These methodological considerations should be taken into account for future studies testing clinical devices for assessing motor function.
Strengths and Limitations
One strength of this review is that it includes more studies than any other review of similar scope that we identified during our study selection process [- ]. This review is unique relative to other reviews on this same topic because it summarizes the validity estimates across the included studies instead of simply describing the characteristics of the samples and sensors involved [ - ]. This provides an evaluation of the degree of validity produced by such sensors. An additional strength was that we identified several meaningful patterns in this literature (eg, an absence of consistency in analytic approaches, equivalent validity of motion data collected at home or in a laboratory, and higher validity coefficients for healthy adults), which can help guide future research in this area. A final strength of this review is that it addresses statistical issues in this field. Although most reviews in this research area are silent as to statistical concerns, the findings of this review are consistent with the small group of previous reviews, which have also noted the statistical challenges present in this literature [ - ].
A limitation of this review is insufficient statistical power to address several questions of interest because of the methodological inconsistency and resulting sparseness across studies. A second limitation of this review is that the literature showed some signs of potential bias, which could limit the trustworthiness of the aggregate effect sizes. Examples of potential bias identified during the study quality assessment were that few studies provided a clear description of whether data were available for all participants throughout the study, and no studies corrected for potential selection biases in their analyses. In addition, it is unclear whether the patterns seen in the funnel plot and elsewhere are evidence of publication bias, selective outcomes, or an artifact of the dominant analytic approaches in this field. Much of the reviewed literature has focused on clinical validation, with less than one-fourth of the studies performing analytical validation. As important as clinical validation is for establishing the clinical and real-world utility of sensor-derived motion data, more studies are needed that focus on the fundamental step of analytic validation. An additional limitation may be the fact that some diseases are not as prevalent or well-studied than others, which may have impacted their representation in our analyses. Finally, our review was restricted to publications available in the English language. Therefore, some technologies being investigated for motor function assessment in non–English-speaking countries may have been missed.
Considerations for Future Research
Several questions we initially hoped to answer in this review could not be addressed because of lack of consistency across studies (eg, which technology or sensor is used, where the sensor is placed, which motions are required by participants, preprocessing steps, feature detection and selection algorithms, and number of motion features retained for the prediction algorithm). Even within studies examining the same disease state, there was limited consistency in these characteristics. As a result, we cannot say which movements and motion outcomes produce the most valid indicators of different neurological disease states, or what data preprocessing, feature processing engineering, and analysis should be considered best practices for converting raw sensor-derived motion data into meaningful digital measurements or biomarkers. It was notable that many of the most common movements from the larger clinical literature (eg, reaching, sit-to-stand, tracing, and pointing) appeared so infrequently in this literature. This lack of consistency in the literature could have affected the validity estimates [- ], and the lack of harmonization across studies limits any inference about methodological or analytic decisions [ ].
An earlier review described continuous monitoring using movement-detecting wearable sensors as a potential source of ground truth for motor function data, which were previously available only through participant self-reports . On the basis of the reviewed literature, the field cannot yet provide this type of objective truth. An existing algorithm needs to be applied to multiple samples without additional adjustments or enhancements and show an aggregate performance that approximates the estimates provided by the studies included in this review. No analytic technique will solve this issue; the only true solution is replication attempts in new samples. Researchers should report how many of the detected features were moved to feature selection to give readers a sense of how many features were excluded, a sense of the parsimony of the resultant model, and an awareness of how likely it is that the model may have been overfit. Care must be taken to design the classification algorithm in a way that maximizes the likelihood that it can perform equally well in future samples. This priority needs to be evaluated at each stage of the analysis: data set preparation, preprocessing, feature extraction, algorithm development, model development or validation, and analytical or clinical validation.
In conclusion, sensor-derived motion data can be leveraged to validly predict disease status for a variety of neurological conditions. Future research will elucidate to what extent sensor-derived motion data may yield robust and transformative digital measurements intended to quantify, diagnose, and predict neurological disease state and its longitudinal change.
The authors would like to thank Gráinne Donnellan, Sarah Hill, and Ana Howarth of Evidinno Outcomes Research Inc (Vancouver, British Columbia, Canada) for their contributions to conducting this systematic literature review.
Conflicts of Interest
AS, PAC, CM, and SB report employment with Biogen. During completion of the work related to this manuscript, CCG was an employee of Biogen. CCG’s current affiliation is ActiGraph, LLC, Pensacola, Florida, US, which was not involved in this work. TS, KH, and MSF report employment with Evidinno Outcomes Research Inc, which was contracted by Biogen to conduct this study.
Supplemental information on the methods and results of the systematic literature review.DOCX File , 288 KB
- Laguna MA, Finat J. Remote monitoring and fall detection: multiplatform Java based mobile applications. In: Proceedings of the 3rd International Workshop on Ambient Assisted Living. 2011 Presented at: IWAAL '11; June 8-10, 2011; Torremolinos-Málaga, Spain p. 1-8. [CrossRef]
- Reiman MP, Manske RC. The assessment of function: how is it measured? A clinical perspective. J Man Manip Ther 2011 May;19(2):91-99 [FREE Full text] [CrossRef] [Medline]
- Toosizadeh N, Mohler J, Lei H, Parvaneh S, Sherman S, Najafi B. Motor performance assessment in Parkinson's disease: association between objective in-clinic, objective in-home, and subjective/semi-objective measures. PLoS One 2015 Apr 24;10(4):e0124763 [FREE Full text] [CrossRef] [Medline]
- Duff SV, He J, Nelsen MA, Lane CJ, Rowe VT, Wolf SL, et al. Interrater reliability of the Wolf Motor Function Test-Functional Ability Scale: why it matters. Neurorehabil Neural Repair 2015 Jun;29(5):436-443 [FREE Full text] [CrossRef] [Medline]
- Ko J, Kim M. Reliability and responsiveness of the gross motor function measure-88 in children with cerebral palsy. Phys Ther 2013 Mar;93(3):393-400. [CrossRef] [Medline]
- Patel S, Lorincz K, Hughes R, Huggins N, Growdon J, Standaert D, et al. Monitoring motor fluctuations in patients with Parkinson's disease using wearable sensors. IEEE Trans Inf Technol Biomed 2009 Nov;13(6):864-873 [FREE Full text] [CrossRef] [Medline]
- Saadeh W, Altaf MA, Butt SA. A wearable neuro-degenerative diseases detection system based on gait dynamics. In: Proceedings of the 2017 IFIP/IEEE International Conference on Very Large Scale Integration. 2017 Presented at: VLSI-SoC '17; October 23-25, 2017; Abu Dhabi, United Arab Emirates p. 1-6. [CrossRef]
- Rigas G, Tzallas AT, Tsipouras MG, Bougia P, Tripoliti EE, Baga D, et al. Assessment of tremor activity in the Parkinson's disease using a set of wearable sensors. IEEE Trans Inf Technol Biomed 2012 May;16(3):478-487. [CrossRef] [Medline]
- Qualification opinion on stride velocity 95th centile as a secondary endpoint in Duchenne Muscular Dystrophy measured by a valid and suitable wearable device*. European Medicines Agency. 2019 Apr 26. URL: https://www.ema.europa.eu/en/documents/scientific-guideline/qualification-opinion-stride-velocity-95th-centile-secondary-endpoint-duchenne-muscular-dystrophy_en.pdf [accessed 2021-04-09]
- Aronson JK, Heneghan C, Ferner RE. Medical devices: definition, classification, and regulatory implications. Drug Saf 2020 Feb;43(2):83-93. [CrossRef] [Medline]
- H.R.34 - 21st Century Cures Act. US Congress. Washington, DC, USA: Library of Congress; 2016. URL: https://www.congress.gov/bill/114th-congress/house-bill/34 [accessed 2021-04-09]
- de Magalhaes FA, Vannozzi G, Gatta G, Fantozzi S. Wearable inertial sensors in swimming motion analysis: a systematic review. J Sports Sci 2015;33(7):732-745. [CrossRef] [Medline]
- Mooney R, Corley G, Godfrey A, Quinlan LR, ÓLaighin G. Inertial sensor technology for elite swimming performance analysis: a systematic review. Sensors (Basel) 2015 Dec 25;16(1):18 [FREE Full text] [CrossRef] [Medline]
- Walmsley CP, Williams SA, Grisbrook T, Elliott C, Imms C, Campbell A. Measurement of upper limb range of motion using wearable sensors: a systematic review. Sports Med Open 2018 Nov 29;4(1):53 [FREE Full text] [CrossRef] [Medline]
- Baig MM, Afifi S, GholamHosseini H, Mirza F. A systematic review of wearable sensors and IoT-based monitoring applications for older adults - a focus on ageing population and independent living. J Med Syst 2019 Jun 15;43(8):233. [CrossRef] [Medline]
- Chen M, Ma Y, Song J, Lai CF, Hu B. Smart clothing: connecting human with clouds and big data for sustainable health monitoring. Mobile Netw Appl 2016 Jul 7;21(5):825-845. [CrossRef]
- Díaz S, Stephenson JB, Labrador MA. Use of wearable sensor technology in gait, balance, and range of motion analysis. Appl Sci 2020;10(1):234. [CrossRef]
- Fong DT, Chan YY. The use of wearable inertial motion sensors in human lower limb biomechanics studies: a systematic review. Sensors (Basel) 2010;10(12):11556-11565 [FREE Full text] [CrossRef] [Medline]
- Kekade S, Hseieh CH, Islam MM, Atique S, Mohammed Khalfan A, Li YC, et al. The usefulness and actual use of wearable devices among the elderly population. Comput Methods Programs Biomed 2018 Jan;153:137-159. [CrossRef] [Medline]
- Kristoffersson A, Lindén M. A systematic review on the use of wearable body sensors for health monitoring: a qualitative synthesis. Sensors (Basel) 2020 Mar 09;20(5):1502 [FREE Full text] [CrossRef] [Medline]
- Goldsack JC, Coravos A, Bakker JP, Bent B, Dowling AV, Fitzer-Attas C, et al. Verification, analytical validation, and clinical validation (V3): the foundation of determining fit-for-purpose for Biometric Monitoring Technologies (BioMeTs). NPJ Digit Med 2020 Apr 14;3:55 [FREE Full text] [CrossRef] [Medline]
- Manta C, Mahadevan N, Bakker J, Ozen Irmak S, Izmailova E, Park S, et al. EVIDENCE publication checklist for studies evaluating connected sensor technologies: explanation and elaboration. Digit Biomark 2021 May 18;5(2):127-147 [FREE Full text] [CrossRef] [Medline]
- Higgins JP, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al. Cochrane Handbook for Systematic Reviews of Interventions. Version 6.1. London, UK: Cochrane Collaboration; 2020.
- Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021 Mar 29;372:n71 [FREE Full text] [CrossRef] [Medline]
- Boukhvalova AK, Kowalczyk E, Harris T, Kosa P, Wichman A, Sandford MA, et al. Identifying and quantifying neurological disability via smartphone. Front Neurol 2018 Sep 4;9:740 [FREE Full text] [CrossRef] [Medline]
- Montalban X, Mulero P, Midaglia L, Graves J, Hauser SL, Julian L, et al. FLOODLIGHT: smartphone-based self-monitoring is accepted by patients and provides meaningful, continuous digital outcomes augmenting conventional in-clinic multiple sclerosis measures (P3.2-024). Neurology 2019 May 7;92(15 Supplement):P3.2-P024.
- Abrami A, Heisig S, Ramos V, Thomas KC, Ho BK, Caggiano V. Using an unbiased symbolic movement representation to characterize Parkinson's disease states. Sci Rep 2020 Apr 30;10(1):7377 [FREE Full text] [CrossRef] [Medline]
- Adams RJ, Lichter MD, Krepkovich ET, Ellington A, White M, Diamond PT. Assessing upper extremity motor function in practice of virtual activities of daily living. IEEE Trans Neural Syst Rehabil Eng 2015 Mar;23(2):287-296 [FREE Full text] [CrossRef] [Medline]
- Alberts JL, Hirsch JR, Koop MM, Schindler DD, Kana DE, Linder SM, et al. Using accelerometer and gyroscopic measures to quantify postural stability. J Athl Train 2015 Jun;50(6):578-588 [FREE Full text] [CrossRef] [Medline]
- Brooks C, Shaafi Kabiri N, Mortazavi F, Auerbach S, Bonato P, Erb MK, et al. Variations in rest-activity rhythm are associated with clinically measured disease severity in Parkinson's disease. Chronobiol Int 2020 May;37(5):699-711. [CrossRef] [Medline]
- Christiansen C, Moore C, Schenkman M, Kluger B, Kohrt W, Delitto A, et al. Factors associated with ambulatory activity in de novo Parkinson disease. J Neurol Phys Ther 2017 Apr;41(2):93-100 [FREE Full text] [CrossRef] [Medline]
- Dehbandi B, Barachant A, Harary D, Long JD, Tsagaris KZ, Bumanlag SJ, et al. Using data from the Microsoft Kinect 2 to quantify upper limb behavior: a feasibility study. IEEE J Biomed Health Inform 2017 Sep;21(5):1386-1392. [CrossRef] [Medline]
- Dowling A, Brys M, Paskavitz J, Auclair V, McLaren D, Postuma R, et al. Quantitative assessment of appendicular bradykinesia in Parkinson's disease using wearable sensors. Mov Disord 2018;33(Supplement 2):S522.
- Hssayeni MD, Burack MA, Jimenez-Shahed J, Ghoraani B. Assessment of response to medication in individuals with Parkinson's disease. Med Eng Phys 2019 May;67:33-43. [CrossRef] [Medline]
- Hughes CM, Baye M, Gordon-Murer C, Louie A, Sun S, Belay GJ, et al. Quantitative assessment of upper limb motor function in Ethiopian acquired brain injured patients using a low-cost wearable sensor. Front Neurol 2019 Dec 12;10:1323 [FREE Full text] [CrossRef] [Medline]
- Levy IG, Yadav V, Abbas A, Koesmahargyo V, Kalali A. Digital markers of motor activity captured over smartphone is associated with negative symptoms of schizophrenia: results from a pilot observational study. Neuropsychopharmacology 2019 Dec 1;44(Suppl 1):348-349.
- Ma M, Proffitt R, Skubic M. Validation of a Kinect V2 based rehabilitation game. PLoS One 2018 Aug 24;13(8):e0202338 [FREE Full text] [CrossRef] [Medline]
- Mitsi G, Mendoza EU, Wissel BD, Barbopoulou E, Dwivedi AK, Tsoulos I, et al. Biometric digital health technology for measuring motor function in Parkinson's disease: results from a feasibility and patient satisfaction study. Front Neurol 2017 Jun 13;8:273 [FREE Full text] [CrossRef] [Medline]
- Ozinga SJ, Linder SM, Alberts JL. Use of mobile device accelerometry to enhance evaluation of postural instability in Parkinson disease. Arch Phys Med Rehabil 2017 Apr;98(4):649-658 [FREE Full text] [CrossRef] [Medline]
- Pradhan S, Kelly VE. Quantifying physical activity in early Parkinson disease using a commercial activity monitor. Parkinsonism Relat Disord 2019 Sep;66:171-175 [FREE Full text] [CrossRef] [Medline]
- Taylor-Piliae RE, Mohler MJ, Najafi B, Coull BM. Objective fall risk detection in stroke survivors using wearable sensor technology: a feasibility study. Top Stroke Rehabil 2016 Dec;23(6):393-399. [CrossRef] [Medline]
- Arora S, Baig F, Lo C, Barber TR, Lawton MA, Zhan A, et al. Smartphone motor testing to distinguish idiopathic REM sleep behavior disorder, controls, and PD. Neurology 2018 Oct 16;91(16):e1528-e1538 [FREE Full text] [CrossRef] [Medline]
- Bennasar M, Hicks YA, Clinch SP, Jones P, Holt C, Rosser A, et al. Automated assessment of movement impairment in Huntington's disease. IEEE Trans Neural Syst Rehabil Eng 2018 Oct;26(10):2062-2069 [FREE Full text] [CrossRef] [Medline]
- Coates L, Shi J, Rochester L, Del Din S, Pantall A. Entropy of real-world gait in Parkinson's disease determined from wearable sensors as a digital marker of altered ambulatory behavior. Sensors (Basel) 2020 May 05;20(9):2631 [FREE Full text] [CrossRef] [Medline]
- Del Din S, Barber T, Lo C, Rolinski M, Baig F, Ju M, et al. Free-living gait in REM sleep behaviour disorder: measures of prodromal Parkinson's disease? Mov Disord 2018;33(Supplement 2):S524-S525.
- Del Din S, Elshehabi M, Galna B, Hansen C, Hobert M, Suenkel U, et al. Instrumented gait analysis identifies potential predictors for Parkinson's disease converters. Mov Disord 2018;33(Supplement 2):S505.
- Hasan H, Burrows M, Athauda DS, Hellman B, James B, Warner T, et al. The BRadykinesia Akinesia INcoordination (BRAIN) tap test: capturing the sequence effect. Mov Disord Clin Pract 2019 Jun 25;6(6):462-469 [FREE Full text] [CrossRef] [Medline]
- Kassavetis P, Saifee TA, Roussos G, Drougkas L, Kojovic M, Rothwell JC, et al. Developing a tool for remote digital assessment of Parkinson's disease. Mov Disord Clin Pract 2015 Oct 20;3(1):59-64 [FREE Full text] [CrossRef] [Medline]
- Lo C, Arora S, Baig F, Lawton MA, El Mouden C, Barber TR, et al. Predicting motor, cognitive and functional impairment in Parkinson's. Ann Clin Transl Neurol 2019 Aug;6(8):1498-1509 [FREE Full text] [CrossRef] [Medline]
- Pantall A, Del Din S, Rochester L. Longitudinal changes over thirty-six months in postural control dynamics and cognitive function in people with Parkinson's disease. Gait Posture 2018 May;62:468-474. [CrossRef] [Medline]
- Pantall A, Suresparan P, Kapa L, Morris R, Yarnall A, Del Din S, et al. Postural dynamics are associated with cognitive decline in Parkinson's disease. Front Neurol 2018 Dec 5;9:1044 [FREE Full text] [CrossRef] [Medline]
- Prince J, Arora S, de Vos M. Big data in Parkinson's disease: using smartphones to remotely detect longitudinal disease phenotypes. Physiol Meas 2018 Apr 26;39(4):044005. [CrossRef] [Medline]
- van Wamelen DJ, Hota S, Podlewska A, Leta V, Trivedi D, Rizos A, et al. Non-motor correlates of wrist-worn wearable sensor use in Parkinson's disease: an exploratory analysis. NPJ Parkinsons Dis 2019 Oct 2;5:22 [FREE Full text] [CrossRef] [Medline]
- Capecci M, Pepa L, Verdini F, Ceravolo MG. A smartphone-based architecture to detect and quantify freezing of gait in Parkinson's disease. Gait Posture 2016 Oct;50:28-33. [CrossRef] [Medline]
- Cavallo F, Moschetti A, Esposito D, Maremmani C, Rovini E. Upper limb motor pre-clinical assessment in Parkinson's disease using machine learning. Parkinsonism Relat Disord 2019 Jun;63:111-116 [FREE Full text] [CrossRef] [Medline]
- Lopane G, Mellone S, Chiari L, Cortelli P, Calandra-Buonaura G, Contin M. Dyskinesia detection and monitoring by a single sensor in patients with Parkinson's disease. Mov Disord 2015 Aug;30(9):1267-1271. [CrossRef] [Medline]
- Scano A, Chiavenna A, Malosio M, Molinari Tosatti L, Molteni F. Kinect V2 implementation and testing of the reaching performance scale for motor evaluation of patients with neurological impairment. Med Eng Phys 2018 Jun;56:54-58. [CrossRef] [Medline]
- Vianello A, Chittaro L, Burigat S, Budai R. MotorBrain: a mobile app for the assessment of users' motor performance in neurology. Comput Methods Programs Biomed 2017 May;143:35-47. [CrossRef] [Medline]
- Arroyo-Gallego T, Ledesma-Carbayo MJ, Butterworth I, Matarazzo M, Montero-Escribano P, Puertas-Martín V, et al. Detecting motor impairment in early Parkinson's disease via natural typing interaction with keyboards: validation of the neuroQWERTY approach in an uncontrolled at-home setting. J Med Internet Res 2018 Mar 26;20(3):e89 [FREE Full text] [CrossRef] [Medline]
- Pérez-López C, Samà A, Rodríguez-Martín D, Català A, Cabestany J, Moreno-Arostegui J, et al. Assessing motor fluctuations in Parkinson's disease patients based on a single inertial sensor. Sensors (Basel) 2016 Dec 15;16(12):2132 [FREE Full text] [CrossRef] [Medline]
- Rodríguez-Martín D, Pérez-López C, Samà A, Català A, Moreno Arostegui JM, Cabestany J, et al. A waist-worn inertial measurement unit for long-term monitoring of Parkinson's disease patients. Sensors (Basel) 2017 Apr 11;17(4):827 [FREE Full text] [CrossRef] [Medline]
- Rodríguez-Molinero A, Samà A, Pérez-Martínez DA, Pérez López C, Romagosa J, Bayés À, et al. Validation of a portable device for mapping motor and gait disturbances in Parkinson's disease. JMIR Mhealth Uhealth 2015 Feb 02;3(1):e9 [FREE Full text] [CrossRef] [Medline]
- Chae SH, Kim Y, Lee KS, Park HS. Development and clinical evaluation of a web-based upper limb home rehabilitation system using a smartwatch and machine learning model for chronic stroke survivors: prospective comparative study. JMIR Mhealth Uhealth 2020 Jul 09;8(7):e17216 [FREE Full text] [CrossRef] [Medline]
- Kim HB, Lee HJ, Lee WW, Kim SK, Jeon HS, Park HY, et al. Validation of freezing-of-gait monitoring using smartphone. Telemed J E Health 2018 Nov;24(11):899-907. [CrossRef] [Medline]
- Lee J, Park S, Shin H. Detection of hemiplegic walking using a wearable inertia sensing device. Sensors (Basel) 2018 May 28;18(6):1736 [FREE Full text] [CrossRef] [Medline]
- Seok HY, Kim JW, Kim YH, Park MH, Kwon DY. Quantitative evaluation of hand motor function using a gyrosensor in mild and moderate carpal tunnel syndrome. Muscle Nerve 2019 Apr;59(4):465-469. [CrossRef] [Medline]
- Louter M, Maetzler W, Prinzen J, van Lummel RC, Hobert M, Arends JB, et al. Accelerometer-based quantitative analysis of axial nocturnal movements differentiates patients with Parkinson's disease, but not high-risk individuals, from controls. J Neurol Neurosurg Psychiatry 2015 Jan;86(1):32-37. [CrossRef] [Medline]
- Ossig C, Gandor F, Fauser M, Bosredon C, Churilov L, Reichmann H, et al. Correlation of quantitative motor state assessment using a kinetograph and patient diaries in advanced PD: data from an observational study. PLoS One 2016 Aug 24;11(8):e0161559 [FREE Full text] [CrossRef] [Medline]
- Otte K, Kayser B, Mansow-Model S, Verrel J, Paul F, Brandt AU, et al. Accuracy and reliability of the Kinect Version 2 for clinical measurement of motor function. PLoS One 2016 Nov 18;11(11):e0166532 [FREE Full text] [CrossRef] [Medline]
- Horigome T, Sumali B, Kitazawa M, Yoshimura M, Liang KC, Tazawa Y, et al. Evaluating the severity of depressive symptoms using upper body motion captured by RGB-depth sensors and machine learning in a clinical interview setting: a preliminary study. Compr Psychiatry 2020 Feb 20;98:152169 [FREE Full text] [CrossRef] [Medline]
- Terashi H, Taguchi T, Ueta Y, Okubo Y, Mitoma H, Aizawa H. Analysis of non-invasive gait recording under free-living conditions in patients with Parkinson's disease: relationship with global cognitive function and motor abnormalities. BMC Neurol 2020 Apr 29;20(1):161 [FREE Full text] [CrossRef] [Medline]
- Terui Y, Iwakura M, Suto E, Kawagoshi A, Sugawara K, Takahashi H, et al. New evaluation of trunk movement and balance during walking in COPD patients by a triaxial accelerometer. Int J Chron Obstruct Pulmon Dis 2018 Dec 7;13:3957-3962 [FREE Full text] [CrossRef] [Medline]
- Beange KH, Chan AD, Beaudette SM, Graham RB. Concurrent validity of a wearable IMU for objective assessments of functional movement quality and control of the lumbar spine. J Biomech 2019 Dec 03;97:109356. [CrossRef] [Medline]
- Campos C, DePaul VG, Knorr S, Wong JS, Mansfield A, Patterson KK. Validity of the ActiGraph activity monitor for individuals who walk slowly post-stroke. Top Stroke Rehabil 2018 May;25(4):295-304. [CrossRef] [Medline]
- Evers LJ, Raykov YP, Krijthe JH, Silva de Lima AL, Badawy R, Claes K, et al. Real-life gait performance as a digital biomarker for motor fluctuations: the Parkinson@home validation study. J Med Internet Res 2020 Oct 09;22(10):e19068 [FREE Full text] [CrossRef] [Medline]
- Zach H, Dirkx M, Pasman JW, Bloem BR, Helmich RC. The patient's perspective: the effect of levodopa on Parkinson symptoms. Parkinsonism Relat Disord 2017 Feb;35:48-54. [CrossRef] [Medline]
- do Carmo Vilas-Boas M, Choupina HM, Rocha AP, Fernandes JM, Cunha JP. Full-body motion assessment: concurrent validation of two body tracking depth sensors versus a gold standard system during gait. J Biomech 2019 Apr 18;87:189-196. [CrossRef] [Medline]
- do Carmo Vilas-Boas M, Rocha AP, Choupina HM, Cardoso MN, Fernandes JM, Coelho T, et al. Validation of a single RGB-D camera for gait assessment of polyneuropathy patients. Sensors (Basel) 2019 Nov 12;19(22):4929 [FREE Full text] [CrossRef] [Medline]
- Aghanavesi S, Westin J, Bergquist F, Nyholm D, Askmark H, Aquilonius SM, et al. A multiple motion sensors index for motor state quantification in Parkinson's disease. Comput Methods Programs Biomed 2020 Jun;189:105309. [CrossRef] [Medline]
- Aghanavesi S, Bergquist F, Nyholm D, Senek M, Memedi M. Motion sensor-based assessment of Parkinson's disease motor symptoms during leg agility tests: results from levodopa challenge. IEEE J Biomed Health Inform 2020 Jan;24(1):111-119. [CrossRef] [Medline]
- Hsu CY, Tsai YS, Yau CS, Shie HH, Wu CM. Gait and trunk movement characteristics of chronic ischemic stroke patients. Int J Gerontol 2019;13(2):144-148.
- Huang YP, Singh A, Chen S, Sun FJ, Huang CR, Liu SI. Validity of a novel touch screen tablet-based assessment for mild cognitive impairment and probable AD in older adults. Assessment 2019 Dec;26(8):1540-1553. [CrossRef] [Medline]
- Lee W, Evans A, Williams DR. Validation of a smartphone application measuring motor function in Parkinson's disease. J Parkinsons Dis 2016 Apr 02;6(2):371-382. [CrossRef] [Medline]
- de Paula JN, de Mello Monteiro CB, da Silva TD, Capelini CM, de Menezes LD, Massetti T, et al. Motor performance of individuals with cerebral palsy in a virtual game using a mobile phone. Disabil Rehabil Assist Technol 2018 Aug;13(6):609-613. [CrossRef] [Medline]
- Simonsen D, Nielsen IF, Spaich EG, Andersen OK. Design and test of an automated version of the modified Jebsen test of hand function using Microsoft Kinect. J Neuroeng Rehabil 2017 May 02;14(1):38 [FREE Full text] [CrossRef] [Medline]
- Lepetit K, Ben Mansour K, Boudaoud S, Kinugawa-Bourron K, Marin F. Evaluation of the kinetic energy of the torso by magneto-inertial measurement unit during the sit-to-stand movement. J Biomech 2018 Jan 23;67:172-176. [CrossRef] [Medline]
- Bernad-Elazari H, Herman T, Mirelman A, Gazit E, Giladi N, Hausdorff JM. Objective characterization of daily living transitions in patients with Parkinson's disease using a single body-fixed sensor. J Neurol 2016 Aug;263(8):1544-1551. [CrossRef] [Medline]
- Iakovakis D, Hadjidimitriou S, Charisis V, Bostantzopoulou S, Katsarou Z, Hadjileontiadis LJ. Touchscreen typing-pattern analysis for detecting fine motor skills decline in early-stage Parkinson's disease. Sci Rep 2018 May 16;8(1):7663 [FREE Full text] [CrossRef] [Medline]
- Iakovakis D, Diniz JA, Trivedi D, Chaudhuri RK, Hadjileontiadis LJ, Hadjidimitriou S, et al. Early Parkinson's disease detection via touchscreen typing analysis using convolutional neural networks. Annu Int Conf IEEE Eng Med Biol Soc 2019 Jul;2019:3535-3538. [CrossRef] [Medline]
- Lauraitis A, Maskeliūnas R, Damaševičius R, Krilavičius T. A mobile application for smart computer-aided self-administered testing of cognition, speech, and motor impairment. Sensors (Basel) 2020 Jun 06;20(11):3236 [FREE Full text] [CrossRef] [Medline]
- Hiorth YH, Larsen JP, Lode K, Tysnes OB, Godfrey A, Lord S, et al. Impact of falls on physical activity in people with Parkinson's disease. J Parkinsons Dis 2016;6(1):175-182. [CrossRef] [Medline]
- Mastoras RE, Iakovakis D, Hadjidimitriou S, Charisis V, Kassie S, Alsaadi T, et al. Touchscreen typing pattern analysis for remote detection of the depressive tendency. Sci Rep 2019 Sep 16;9(1):13414 [FREE Full text] [CrossRef] [Medline]
- Ahlrichs C, Samà A, Lawo M, Cabestany J, Rodríguez-Martín D, Pérez-López C, et al. Detecting freezing of gait with a tri-axial accelerometer in Parkinson's disease patients. Med Biol Eng Comput 2016 Jan;54(1):223-233. [CrossRef] [Medline]
- Klingelhoefer L, Bostanjopoulou S, Trivedi D, Hadjidimitriou S, Mayer S, Katsarou Z, et al. Medical evaluation as gold standard to control iPrognosis application derived data for early Parkinson’s disease detection. Mov Disord 2019;34(Supplement 2):S913-S914.
- Lalvay L, Lara M, Mora A, Alarcón F, Fraga M, Pancorbo J, et al. Quantitative measurement of akinesia in Parkinson's disease. Mov Disord Clin Pract 2016 Aug 3;4(3):316-322 [FREE Full text] [CrossRef] [Medline]
- Lipsmeier F, Simillion C, Bamdadian A, Smith A, Schobel S, Czech C, et al. Preliminary reliability and validity of a novel digital biomarker smartphone application to assess cognitive and motor symptoms in Huntington's disease (HD) (P1.8-042). Neurology 2019 Apr;92(15 Supplement):P1.8-P042.
- Lipsmeier F, Taylor KI, Postuma RB. Preliminary validation smartphone application to assess motor symptoms in recently diagnosed parkinson patients. Neurology 2019 May 8;92(15 Supplement):P4.7-005.
- Rodríguez-Molinero A, Samà A, Pérez-López C, Rodríguez-Martín D, Quinlan LR, Alcaine S, et al. Analysis of correlation between an accelerometer-based algorithm for detecting Parkinsonian gait and UPDRS subscales. Front Neurol 2017 Sep 1;8:431 [FREE Full text] [CrossRef] [Medline]
- Andrzejewski KL, Dowling AV, Stamler D, Felong TJ, Harris DA, Wong C, et al. Wearable sensors in Huntington disease: a pilot study. J Huntingtons Dis 2016 Jun 18;5(2):199-206. [CrossRef] [Medline]
- Horne MK, McGregor S, Bergquist F. An objective fluctuation score for Parkinson's disease. PLoS One 2015 Apr 30;10(4):e0124522 [FREE Full text] [CrossRef] [Medline]
- Wissel BD, Mitsi G, Dwivedi AK, Papapetropoulos S, Larkin S, López Castellanos JR, et al. Tablet-based application for objective measurement of motor fluctuations in Parkinson disease. Digit Biomark 2018 Jan 9;1(2):126-135 [FREE Full text] [CrossRef] [Medline]
- Shaafi Kabiri N, Best A, Johnson S, Ho B, Eden G, Dupee B, et al. Utility of a smartphone app in discretely assessing and monitoring symptoms of Parkinson’s disease. Mov Disord 2019;34(Supplement 2):S298.
- Arroyo-Gallego T, Ledesma-Carbayo MJ, Sanchez-Ferro A, Butterworth I, Mendoza CS, Matarazzo M, et al. Detection of motor impairment in Parkinson's disease via mobile touchscreen typing. IEEE Trans Biomed Eng 2017 Sep;64(9):1994-2002. [CrossRef] [Medline]
- Lowes L, Miller N, Iammarino M, Dugan M, Alfano L. Activity monitoring in neuromuscular disease: successes, challenges, and a path forward (P5.6-016). Neurology 2019 Apr;92(15 Supplement):P5.6-P016.
- Zhou H. Instrumented trail-making task to identify cognitive-motor impairment and assess cognitive frailty. J Am Geriatr Soc 2019 Apr 02;67(S1):S127-S128. [CrossRef]
- Ren P, Bosch Bayard JF, Dong L, Chen J, Mao L, Ma D, et al. Multivariate analysis of joint motion data by Kinect: application to Parkinson's disease. IEEE Trans Neural Syst Rehabil Eng 2020 Jan;28(1):181-190. [CrossRef] [Medline]
- Gordon MF, Grachev ID, Mazeh I, Dolan Y, Reilmann R, Loupe PS, et al. Quantification of motor function in Huntington disease patients using wearable sensor devices. Digit Biomark 2019 Sep 6;3(3):103-115 [FREE Full text] [CrossRef] [Medline]
- Shawen N, Knott J, Lee J, Simuni T, Daeschler D, Jayaraman A. Automatic scoring of Parkinson's disease motor symptoms using a smartwatch (P2.8-001). Neurology 2019 Apr;92(15 Supplement):P2.8-P001.
- Seiffert P, Kawa J, Derejczyk J, Czernek M, Stepien P, Marcisz C. Mobile test of manual dexterity in the diagnostics of frailty in older patients with mild parkinsonian signs. Eur Geriatr Med 2019;10(Supplement 1):S258. [CrossRef]
- Lee CY, Kang SJ, Hong SK, Ma HI, Lee U, Kim YJ. A validation study of a smartphone-based finger tapping application for quantitative assessment of bradykinesia in Parkinson's disease. PLoS One 2016 Jul 28;11(7):e0158852 [FREE Full text] [CrossRef] [Medline]
- Kim H, Lee HJ, Lee W, Kwon S, Kim SK, Jeon HS, et al. Unconstrained detection of freezing of Gait in Parkinson's disease patients using smartphone. Annu Int Conf IEEE Eng Med Biol Soc 2015 Aug;2015:3751-3754. [CrossRef] [Medline]
- Sigcha L, Costa N, Pavón I, Costa S, Arezes P, López JM, et al. Deep learning approaches for detecting freezing of gait in Parkinson's disease patients through on-body acceleration sensors. Sensors (Basel) 2020 Mar 29;20(7):1895 [FREE Full text] [CrossRef] [Medline]
- Bonnechère B, Jansen B, Haack I, Omelina L, Feipel V, Van Sint Jan S, et al. Automated functional upper limb evaluation of patients with Friedreich ataxia using serious games rehabilitation exercises. J Neuroeng Rehabil 2018 Oct 04;15(1):87 [FREE Full text] [CrossRef] [Medline]
- Hssayeni MD, Adams JL, Ghoraani B. Deep learning for medication assessment of individuals with Parkinson's disease using wearable sensors. Annu Int Conf IEEE Eng Med Biol Soc 2018 Jul;2018:1-4. [CrossRef] [Medline]
- Zhan A, Mohan S, Tarolli C, Schneider RB, Adams JL, Sharma S, et al. Using smartphones and machine learning to quantify Parkinson disease severity: the mobile Parkinson disease score. JAMA Neurol 2018 Jul 01;75(7):876-880 [FREE Full text] [CrossRef] [Medline]
- ROBINS-E Development Group, Higgins J, Morgan R, Rooney A, Taylor K, Thayer K, et al. Risk Of Bias In Non-randomized Studies - of Exposure (ROBINS-E). Launch Version. Risk Of Bias. 2022. URL: https://www.riskofbias.info/welcome/robins-e-tool [accessed 2022-06-01]
- Rosenthal R. Meta-Analytic Procedures for Social Research. Applied Social Research Methods Series 6. Newbury Park, CA, USA: Sage Publications; 1991.
- Cooper H, Hedges LV, Valentine JC. The Handbook of Research Synthesis and Meta-Analysis. 3rd edition. New York, NY, USA: Russell Sage Foundation; 2019.
- Fisher RA. On the 'probable error' of a coefficient of correlation deduced from a small sample. The University of Adelaide. 1921. URL: https://digital.library.adelaide.edu.au/dspace/handle/2440/15169 [accessed 2021-04-09]
- Fisher RA. Frequency distribution of the values of the correlation coefficient in samples from an indefinitely large population. Biometrika 1915 May;10(4):507-521. [CrossRef]
- Lipsey M, Wilson D. Practical Meta-Analysis. Thousand Oaks, CA, USA: Sage Publications; 2001.
- Alexander S, Peryer G, Gray E, Barkhof F, Chataway J. Wearable technologies to measure clinical outcomes in multiple sclerosis: a scoping review. Mult Scler 2021 Oct;27(11):1643-1656 [FREE Full text] [CrossRef] [Medline]
- Hosmer DW, Lemeshow S. Applied Logistic Regression. 2nd edition. Hoboken, NJ, USA: Wiley Online Library; 2000.
- Lee EH, Lee SJ, Hwang ST, Hong SH, Kim JH. Reliability and validity of the beck depression inventory-II among Korean adolescents. Psychiatry Investig 2017 Jan;14(1):30-36 [FREE Full text] [CrossRef] [Medline]
- Marrie RA, Goldman M. Validity of performance scales for disability assessment in multiple sclerosis. Mult Scler 2007 Nov;13(9):1176-1182. [CrossRef] [Medline]
- Martinez-Martin P, Rodriguez-Blazquez C, Alvarez-Sanchez M, Arakaki T, Bergareche-Yarza A, Chade A, et al. Expanded and independent validation of the Movement Disorder Society-Unified Parkinson's Disease Rating Scale (MDS-UPDRS). J Neurol 2013 Jan;260(1):228-236. [CrossRef] [Medline]
- Reich SG, Savitt JM. Parkinson's disease. Med Clin North Am 2019 Mar;103(2):337-350. [CrossRef] [Medline]
- Visscher PM, Brown MA, McCarthy MI, Yang J. Five years of GWAS discovery. Am J Hum Genet 2012 Jan 13;90(1):7-24 [FREE Full text] [CrossRef] [Medline]
- Bottou L, Peters J, Quiñonero-Candela J, Charles DX, Chickering M, Portugaly E, et al. Counterfactual reasoning and learning systems: the example of computational advertising. J Mach Learn Res 2013;14(101):3207-3260.
- Fiedler K. Voodoo correlations are everywhere-not only in neuroscience. Perspect Psychol Sci 2011 Mar;6(2):163-171. [CrossRef] [Medline]
- Vul E, Harris C, Winkielman P, Pashler H. Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition. Perspect Psychol Sci 2009 May;4(3):274-290. [CrossRef] [Medline]
- Sculley D, Holt G, Golovin D, Davydov E, Phillips T, Ebner D, et al. Machine learning: the high-interest credit card of technical debt. In: Proceedings of the 28th Conference on Neural Information Processing Systems. 2014 Presented at: NeurIPS '14; December 8-13, 2014; Montreal, Canada.
- Sculley D, Otey ME, Pohl M, Spitznagel B, Hainsworth J, Zhou Y. Detecting adversarial advertisements in the wild. In: Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2011 Presented at: KDD '11; August 21-24, 2011; San Diego, CA, USA p. 274-282. [CrossRef]
- Saeb S, Lonini L, Jayaraman A, Mohr DC, Kording KP. The need to approximate the use-case in clinical machine learning. Gigascience 2017 May 01;6(5):1-9 [FREE Full text] [CrossRef] [Medline]
- Vul E, Pashler H. Suspiciously high correlations in brain imaging research. In: Lilienfeld SO, Waldman ID, editors. Psychological Science Under Scrutiny: Recent Challenges and Proposed Solutions. Hoboken, NJ, USA: John Wiley & Sons; 2017:196-220.
- Head ML, Holman L, Lanfear R, Kahn AT, Jennions MD. The extent and consequences of p-hacking in science. PLoS Biol 2015 Mar;13(3):e1002106 [FREE Full text] [CrossRef] [Medline]
- Ioannidis JP. What have we (not) learnt from millions of scientific papers with P values? Am Stat 2019 Mar 20;73(sup1):20-25. [CrossRef]
- Mayo-Wilson E, Li T, Fusco N, Bertizzolo L, Canner JK, Cowley T, et al. Cherry-picking by trialists and meta-analysts can drive conclusions about intervention efficacy. J Clin Epidemiol 2017 Nov;91:95-110 [FREE Full text] [CrossRef] [Medline]
- Li T, Mayo-Wilson E, Fusco N, Hong H, Dickersin K. Caveat emptor: the combined effects of multiplicity and selective reporting. Trials 2018 Sep 17;19(1):497 [FREE Full text] [CrossRef] [Medline]
- Mayo-Wilson E, Fusco N, Li T, Hong H, Canner JK, Dickersin K, MUDS investigators. Multiple outcomes and analyses in clinical trials create challenges for interpretation and research synthesis. J Clin Epidemiol 2017 Jun;86:39-50 [FREE Full text] [CrossRef] [Medline]
- Dobkin BH. Wearable motion sensors to continuously measure real-world physical activities. Curr Opin Neurol 2013 Dec;26(6):602-608 [FREE Full text] [CrossRef] [Medline]
|AUC: area under the curve|
|EMA: European Medicines Agency|
|PICO: Population, Intervention, Comparator, Outcomes|
|PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses|
|ROBINS-E: Risk Of Bias In Nonrandomized Studies of Exposures|
Edited by R Kukafka; submitted 03.03.22; peer-reviewed by C Guinemer, K Radhakrishnan, M Bardus; comments to author 09.06.22; revised version received 18.07.22; accepted 11.10.22; published 21.11.22Copyright
©Christine C Guo, Patrizia Andrea Chiesa, Carl de Moor, Mir Sohail Fazeli, Thomas Schofield, Kimberly Hofer, Shibeshih Belachew, Alf Scotland. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 21.11.2022.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.