Background: Digital therapeutics (DTx) are software-based products that prevent, manage, or treat a medical condition and are delivered through a smartphone app, web application, or wearable device. Clinical trials assessing DTx pose challenges, foremost among which is designing appropriate digital shams (or digital placebos), which should ideally mimic DTx (in terms of design, components, and duration of treatment) while omitting the active principle or component.
Objective: The objective of our review was to understand how digital shams are being used in clinical research on DTx in neuroscience, which is the most common therapy area for DTx.
Methods: We conducted a systematic literature review of DTx in neuroscience (including neurodevelopmental, neurodegenerative, and psychiatric disorders) with a focus on controlled clinical trials involving digital shams. Studies were identified from trial registries (ClinicalTrials.gov, the European Union Clinical Trials Register, and Trial Trove) and through structured searches in MEDLINE and Embase (both via the Embase website) and were limited to articles in English published from 2010 onward. These were supplemented by keyword-based searches in PubMed, Google, and Google Scholar and bibliographic searches. Studies assessing DTx in neuroscience (including neurodevelopmental, neurodegenerative, and psychiatric disorders) were included. Details related to the publication, DTx, comparator, patient population, and outcomes were extracted and analyzed.
Results: Our search criteria identified 461 neuroscience studies involving 213 unique DTx. Most DTx were extended reality based (86/213, 40.4%) or mobile device based (56/213, 26.3%); 313 were comparative, of which 68 (21.7%) used shams. The most common therapeutic areas assessed in these studies were stroke (42/213, 19.7%), depression (32/213, 15%), and anxiety (24/213, 11.3%). The most common treatments were cognitive behavioral therapy or behavioral therapy (67/213, 32.4%), physical rehabilitation (60/213, 28.2%), and cognitive training (41/213, 19.2%). We identified the following important issues related to the use of digital shams in neuroscience: shams were not validated before use in studies, they varied widely in design (from being nearly identical to the DTx to using different software programs altogether), and the level of patient engagement or satisfaction with the sham and the impact of the sham on study outcomes were infrequently reported.
Conclusions: Digital shams are critical for the clinical development of DTx in neuroscience. Given the importance of sham controls in evaluating DTx efficacy, we provide recommendations on the key information that should be reported in a well-designed DTx trial and propose an algorithm to allow the correct interpretation of DTx study results. Sham-controlled studies should be routinely used in DTx trials—in early-phase studies—to help identify DTx active components and—in late-phase studies—to confirm the efficacy of DTx. The use of shams early in development will ensure that the appropriate sham control is used in later confirmatory trials.
Digital therapeutics (DTx), a new frontier in medicine, are software-based products that prevent, manage, or treat a medical condition [- ]. DTx are generally delivered through a smartphone app, web application, or wearable device and may use extended reality (XR) or resemble a computer game [ - ]. The explosion of interest in DTx has coincided with the popularity of social networks, the ubiquity of smartphones and wearables, and the increasing sophistication of cloud-based data platforms [ ]. The COVID-19 pandemic may have further driven interest in DTx. DTx may not only improve access to therapies with established clinical benefits (such as cognitive behavioral therapy [CBT]) but may also open new areas of intervention altogether. In neuroscience, DTx are being explored in motor rehabilitation and cognitive rehabilitation in patients with stroke, Parkinson disease, and multiple sclerosis (MS) as well as in depression, fatigue, and chronic pain [ , ]. Outside neuroscience, DTx are being developed for conditions such as gastrointestinal disorders, diabetes, heart failure, asthma, and chronic obstructive pulmonary disease [ ].
For DTx that do not fall under the United States Food and Drug Administration enforcement discretion, standard regulatory requirements for class I, II, and III devices apply [, ].
To appropriately study the effectiveness of DTx, it is crucial to establish clear control groups, including sham (or placebo) controls. There is no specific United States Food and Drug Administration guidance on the use of digital shams; however, several recently approved DTx have successfully used them in studies [, ]. Similarly, there is no specific regulatory guidance in Europe, although issues with designing a digital sham have been identified as a regulatory challenge [ , ].
The formulation of these control arms should consider several factors that contribute to therapeutic efficacy in DTx trials. These factors include the following:
- Nonspecific factors: These encompass elements such as the expectation of improvement and regression to the mean, which can influence trial outcomes.
- Engagement-related factors: Factors arising from engagement with the DTx, such as self-monitoring, can also impact trial results.
- Active component of the DTx: This is the specific therapeutic element of the DTx, such as CBT delivered through a digital platform.
In traditional clinical trials, a placebo control consists of an inert substance that may improve outcomes because of nonspecific effects such as regression to the mean, but it lacks an active ingredient, whereas an active control includes both the nonspecific factors and active ingredient.
DTx trials differ from traditional clinical trials owing to the presence of therapeutic factors arising from engagement with digital intervention. Therefore, in DTx trials, a placebo control or digital sham would need to include not only the same nonspecific factors but also the same engagement-related factors as the DTx itself, with only the active component excluded. Similarly, an active control for DTx would be a treatment that incorporates comparable nonspecific and engagement-related factors and includes an active component that differs from the active component of the DTx being studied. Properly formulating and implementing these control groups is essential for the reliable assessment of DTx effectiveness and for distinguishing the specific therapeutic effects of the digital intervention from nonspecific and engagement-related factors.
Given this scenario, designing and interpreting the results of DTx clinical trials poses several challenges. The first step involves identifying the fundamental active component of a DTx separately from the engagement-related factors in the intervention, which can be difficult. Second, designing a digital sham by removing the active component alone can be difficult. In the case of a DTx that involves CBT delivered via a video game, for example, a simple non–CBT-based psychoeducational sham control may serve the purpose of removing the active component; however, if it is significantly less engaging than the DTx and the variation in engagement levels could potentially introduce confounding into the study results. Such a study may demonstrate efficacy for the DTx even if the CBT component is ineffective (ie, it yields a false-positive result). Conversely, if the sham includes aspects of the active component, then such a sham-controlled study may fail despite the DTx being effective (ie, yielding a false-negative result) [- ]. The corollary of this is that interpreting a sham-controlled study is not straightforward. If a study fails to demonstrate the efficacy of the DTx, this may be because of a lack of efficacy of the DTx or because of an inappropriate sham control that included active components and thus masked the DTx effect. Conversely, a sham-controlled study may succeed in demonstrating DTx efficacy either because of the real efficacy of the DTx or because of an inappropriate sham control that was substantially less engaging than the DTx.
An ideal DTx confirmatory trial would involve an appropriately designed sham control that has been validated, that is, has been demonstrated to, or can reasonably be expected to, have a comparable impact on nonspecific and engagement-related factors, such as the DTx itself.
Given the complexities involved in DTx clinical trials, especially with reference to the use of appropriate sham controls, it is important to better understand how digital sham controls have been implemented in these trials and how their design and validation have been reported.
Thus, DTx are likely to shape the future of medicine. Therefore, it is important to understand how digital shams are being used in clinical research on DTx in neuroscience, which is the most common therapy area for DTx [, , ]. Recently, Lutz et al [ ] (nonsystematically) reviewed the control conditions in DTx trials. Although this review was limited to DTx listed in the Digital Therapeutics Alliance product list with findings based on only 14 randomized controlled trials, it also highlighted important challenges in defining sham controls for DTx [ ]. There remains a need for a more comprehensive review that explores all available clinical trials on DTx in neuroscience. Here, we describe a systematic literature review of DTx in neuroscience (including neurodevelopmental, neurodegenerative, and psychiatric disorders) with a focus on controlled clinical trials, particularly those involving digital shams. To the best of our knowledge, the use of digital shams in DTx clinical trials has not yet been systematically examined. We hope that the results will help establish a scientific road map for evidence-based DTx development and drive the formulation of best practices and guidelines for research, ensuring the correct evaluation of patient benefit and safety.
The research questions addressed in the literature review were as follows. In the field of neuroscience (including neurodevelopmental, neurodegenerative, and psychiatric disorders):
- How has digital sham been used in clinical research?
- What proportion of DTx studies involved digital sham?
- What was the objective of using digital sham in the study or in the larger developmental plan of the manufacturer?
- What was the nature of digital sham used in these studies? Was the digital sham validated before its use in clinical trials?
- How did digital sham compare with the active DTx interventions in these studies? Which aspects were common in both treatment arms, and which aspects differed, in terms of nature of intervention, number of hours of treatment, level or intensity of patient engagement, etc?
- What were the compliance or dropout rates with the use of digital sham? How did those compare with the compliance or dropout rates with active intervention?
- What were the results of these studies? What were the effect sizes (change from baseline) with the use of digital sham, across various outcomes? What were the differences in effect sizes between active DTx intervention and digital sham?
- What challenges were identified in the design, development, and use of digital sham? What modifications or improvements were proposed?
Studies were identified from trial registries (ClinicalTrials.gov , the European Union Clinical Trials Register [ ], and Trial Trove [ ]) and through structured searches in MEDLINE and Embase (both via the Embase website). The search strategy used a facet for DTx and digital shams and was limited to articles in English published from 2010 onward (refer to the Embase criteria in Table S1 in [ - ] and the final search strategy in Table S2 in ). Structured searches (run on May 10, 2021) were supplemented by additional searches conducted in November 2021. The additional supplementary searches included keyword-based searches in PubMed, Google, and Google Scholar to identify comparative studies involving DTx and digital shams in neuroscience and a review of bibliographies of relevant publications identified by the structured searches to identify further relevant citations.
Abstracts and titles were screened by a single reviewer who then examined the full text in detail. A quality check was conducted by a second reviewer who confirmed the accuracy of all the steps included in screening; this covered approximately 20% of all screening decisions.
Studies that assessed DTx in neuroscience (including neurodevelopmental, neurodegenerative, and psychiatric disorders) were included. Studies that assessed DTx in nonneuroscience indications (such as respiratory disorders, metabolic disorders, and cardiovascular disorders) were excluded. Study designs of interest were clinical trials. All observational studies were excluded. There were no restrictions on age, ethnicity, outcome, and geography. These selection criteria were applied to the structured and supplementary searches.
DTx, as per the Digital Therapeutics Alliance definition, is a product that is driven by high-quality software programs and that prevents, manages, or treats a medical disorder or disease.
In the absence of a universally accepted definition of digital sham, we operationally defined it as a comparator designed to mimic the DTx (eg, with a similar design, components, and duration of treatment as the DTx) but with the DTx active principle or component being removed or reduced in intensity .
DTx were categorized according to the treatment delivery techniques used. These include the following:
- Mobile devices: These include DTx that are delivered through software applications. Examples include Daylight, reSET-O, Quit genius, and EndeavorRx.
- Computer based: These include DTx that are training modules, psychoeducation, etc delivered through a computer. Examples include the Brain HQ, Cogmed, Cogniplus, and CogPack. These also include video games delivered through a computer. Examples include SPARX (Smart, Positive, Active, Realistic, X-factor) and Dr Kawashima’s Brain Training.
- XR based: These include DTx that use XR as the mode of delivery. Examples include Kinect-based virtual reality games, EaseVRx, and Reh@City.
- Wearables: These include wearable devices (smartwatch, wristband, headband, etc) that measure physiological data and provide stimulation, vibration, or a signal to treat the problem. For example, The NightWare program is an app that works on a smartwatch and is therefore classified as wearable. Other examples include Relivion and Rapael Smart Glove.
- Web based: These include web-based and internet-based DTx. Examples include SHUTi, Deprexis, and FitMindKit.
Details related to the publication, DTx, comparator, patient population, and outcomes were extracted and analyzed. Studies were categorized based on the efficacy results of DTx and sham, and this categorization was used to inform a future analytical framework. Groups 1 and 2 included studies in which there was no significant improvement from baseline in the primary outcome in the sham control arm. In group 1, both DTx and sham arms failed to show improvement from baseline (DTx− and Sh−), whereas group-2 studies successfully demonstrated the efficacy of DTx (DTx+ and Sh−). In group-2 studies, it is pertinent to examine whether the comparative efficacy of the DTx arm may have occurred because of inappropriate sham control. Groups 3 and 4 included studies in which there was significant improvement from baseline with sham. Group 3 included studies that failed despite the DTx showing improvement from baseline (DTx+, Sh+ [note that DTx was not significantly inferior to sham in any of the studies identified]); failure may have been because of the DTx being ineffective or because of inappropriately designed shams that produced strong therapeutic effects. Group 4 includes perhaps the ideal studies, in that the sham arm showed improvement from baseline, but the DTx arm showed significantly larger improvement (DTx++, Sh+), implying that the active components of the DTx were responsible for the greater efficacy.
As shown in the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow diagram (), database searches yielded 739 potentially relevant citations. Following title-abstract screening and a detailed full-text examination, 95 publications were identified (refer to Table S3 in for a list of studies excluded at this step). In addition, supplementary searches identified 186 publications and 289 registry records that met the eligibility criteria. With linking, this represented 461 unique studies (225 studies and 236 trial registry records).
Of the 225 studies (Table S4 in), 203 (90.2%) were full publications (journal articles), 196 (87.1%) were published since 2015, a total of 147 (65.3%) were comparative studies, and 47 (20.8%) used a digital sham. Of the 236 registry records, 166 (70.3%) were comparative studies and 21 (8.9%) used a digital sham. The 313 comparative studies involved 213 unique DTx (Table S5 in ).
Overview of Identified DTx
On the basis of the hardware and software components, we classified DTx as mobile device based (smartphone+software app), computer based (computer+software or video game), web based (link to access software accessible through computers or mobile devices), wearable (physical wearable devices), or XR based (headset+software or video game).
Of the 213 DTx, most were XR based (n=86, 40.4%) or mobile device based (n=56, 26.3%). The remaining DTx were computer based (33/213, 15.5%), web based (19/213, 8.9%), or wearables (19/213, 8.9%). Most DTx (126/213, 59.2%) were industry developed. DTx were most commonly developed for stroke (42/213, 19.7%), depression (32/213, 15%), anxiety (24/213, 11.3%), and substance use disorders (17/213, 8%). The DTx were most used for delivering CBT and behavioral therapy (69/213, 32.4%), physical rehabilitation (60/213, 28.2%), and cognitive training (41/213, 19.2%).
Overview of Comparative Studies
Of the 313 comparative studies, most (n=303, 96.8 %) were randomized controlled trials. The comparator was a digital sham in 21.7% (68/313) of studies, an active control in 38% (119/313) of studies, a waitlist control in 11.5% (36/313) of studies, and treatment as usual or no intervention in 32.9% (103/313) of studies. Quantitative results (efficacy or safety) were available for 125 studies, of which DTx efficacy was successfully demonstrated in 103.
Use of Digital Shams
Digital shams were comparators in 68 (21.7%) of the 313 studies. None of these was the sham validated for use as a placebo comparator; however, a few authors have cited prior research where the sham had demonstrated no clinical efficacy [- ].
Digital shams were identical to the DTx in terms of hardware and duration of treatment, except in 1 study  where the digital sham sessions were shorter than the DTx sessions (refer to Table S6 in for the nature of digital shams and Table S7 in for the duration of DTx and digital shams).
Different approaches were used for inactivating the active components of digital shams (refer to Table S6 infor additional details). The most common approach (48/68, 71%) was to replace the active component with an inactive or neutral component, for example, replacing positive images with neutral images (therapeutic evaluative conditioning) [ ]. In 9% (6/68) of the studies, the active component was simply removed; for example, the app showed only a timer instead of coping strategies (PEAR-004) [ ]. In other studies (7/68, 10%), the active component was replaced by a different active component unrelated to the study outcomes; for example, a video game using CBT for attention-deficit/hyperactivity disorder (ADHD) was replaced with a game targeting cognitive domains not primarily associated with ADHD [ , ]. Finally, some studies (7/68, 10%) used digital shams that were less intense versions of the active treatment, for example, using a lower game difficulty level (Brain HQ) [ ].
Excluding the studies and registry records for which only study protocols were available (25/68, 37%), 43 sham-controlled trials provided efficacy results. Of these 43, 32 (74%) provided efficacy data for digital shams, whereas the remaining 11 (26%) reported DTx efficacy without reporting the impact of digital shams on outcomes, that is, these studies reported information on the difference between DTx and sham over time but did not separately report quantitative information on changes in the sham arm alone (brief details in).
|Study||Sample size, n||Length of the study (wk)||Treatment delivered||Indication||Blinding status||Name of DTxa||Type of DTx||Name of digital sham||Nature of digital sham||Comparison between DTx and digital sham|
|Dropout rate||Patient engagement or satisfaction|
|Rosa et al , 2017||6||12||Cognitive training||ADHDb||Single blind||ACTIVATE||Computer based||Placebo cognitive training||Inactive or neutral||Higher in DTx||NRc|
|Ghaemi et al , 2022||112||12||CBTd or behavior therapy||Psychosis||Single blind||PEAR-004||Mobile devices based||PEAR-004 placebo app||Missing||Similar||NR|
|Cimenser et al , 2021||31||26||Others||Dementia||NR||GENUS (gamma entrainment using sensory stimuli) or synchronized 40-Hz gamma oscillation||Wearable||GENUS (gamma entrainment using sensory stimuli): sham settings||Inactive or neutral||Higher in digital sham||NR|
|Sosnowski et al , 2021||54||6||CBT or behavior therapy||ASDe||Double blind||Lookware||Computer based||Control video game||Inactive or neutral||Higher in digital sham||Similar|
|Kalmbach et al , 2020||91||7||CBT or behavior therapy||Insomnia||Single blind||Sleepio||Web based||Web-based sleep education or sleep hygiene education||Inactive or neutral||Similar||Similar|
|Kollins et al , 2020 (STARS-ADHDf)||348||4||Cognitive training||ADHD||Double blind||EndeavorRx (AKL-T01)||Mobile devices based||AKL-T09||Another active||Similar||NR|
|Yerys et al , 2019||19||4||Cognitive training||ASD||Single blind||Project-Evo||Mobile devices based||AKL-T09||Another active||Similar||Similar|
|Hancock et al , 2015||71||6||Cognitive training||MSg||Double blind||Brain HQ (InSight or BrainTwister visual n-back programs)||Computer based||Sham training (computerized cognitive training)||Lower intensity||Similar||NR|
|van der Molen et al , 2010||95||15||Cognitive training||Intellectual disability||Single blind||Odd Yellow training (Training A and B)||Computer based||Control training (with no demand on memory capacity)||Inactive or neutral||Similar||NR|
|Berman et al , 2020||89||6||CBT or behavior therapy||SUDh||Single blind||Telecoach||Mobile devices based||A web-based control app||Inactiveor neutral||Similar||NR|
|Teng et al , 2019||82||8.3||Others||Anxiety||Double blind||Home-delivered attentional bias modification||Mobile devices based||Placebo training||Inactive or neutral||Similar||NR|
|Batterham et al , 2018||194||15||CBT or behavior therapy||Anxiety||Single blind||FitMindKit||Web based||HealthWatch||Inactive or neutral||Similar||Higher with DTx|
|Davies et al , 2017||488||4||CBT or behavior therapy||SUD||Single blind||Drinks Meter||Mobile devices based||Imagination of information about alcohol misuse||Inactive or neutral||Similar||NR|
|Scholten et al , 2016||138||16||Others||Anxiety||Single blind||Dojo||Computer based||Rayman 2: The Great Escape||Inactive or neutral||Similar||NR|
|Schoneveld et al , 2016||134||13||CBT or behavior therapy||Anxiety||Single blind||MindLight||Computer based||Max and the Magic Marker||Inactive or neutral||Similar||Higher with digital sham|
|de Vries et al , 2015||121||12||Cognitive training||ASD||Double blind||Braingame Brian, an adaptive working memory training and cognitive flexibility||Computer based||Nonadaptive control training (mock-training)||Lower intensity||Similar||NR|
|Bikic et al , 2017||18||7||Cognitive training||ADHD||Double blind||Scientific Brain Training||Computer based||Placebo control arcade game (Tetris)||Another active||Higher in digital sham||NR|
|Chacko et al , 2014||85||8||Cognitive training||ADHD||Double blind||CWMTi||Computer based||CWMT placebo||Lower intensity||Similar||NR|
|Enock et al , 2014||429||12.6||Others||Anxiety||Double blind||Attention Bias Modification Training||Mobile devices based||Control training (no contingency training)||Inactive or neutral||Similar||NR|
|Bove et al , 2021 (DigCog)||44||14||Cognitive training||MS||Double blind||AKL-T03||Mobile devices based||AKL-T09||Another active||Higher in DTx||Similar|
|Garcia et al , 2021||188||8||CBT or behavior therapy||Pain||Double blind||EaseVRx headset with active intervention||Extended reality||EaseVRx headset without active intervention||Inactive or neutral||Similar||Similar|
|Pasula et al , 2020||252||NR||Others||Migraine||Double blind||NerivioMigra app (remote electrical neuromodulation device)||Wearable||NerivioMigra app (remote electrical neuromodulation device)—sham stimulation||Inactive or neutral||NR||NR|
|Wijnhoven et al , 2020||109||19||CBT or behavior therapy||Anxiety||NR||Mindlight||Computer based||Triple Town||Inactive or neutral||Similar||NR|
|Iacoviello et al , 2018||51||6||CBT or behavior therapy||Depression||Double blind||EFMTj||Computer based||EFMT—with neutral shapes||Inactive or neutral||Similar||NR|
|Parks et al , 2018||1051||8||CBT or behavior therapy||Anxiety||NR||Happify||Web based||Psychoeducation||Inactive or neutral||Higher in DTx||NR|
|Zwerenz et al , 2017||229||12||CBT or behavior therapy||Depression||Single blind||Deprexis||Web based||Web-based information about depression||Inactive or neutral||Similar||Higher with DTx|
|Deady et al , 2016||104||25.8||CBT or behavior therapy||Depression||Single blind||DEALk intervention||Web based||HealthWatch||Inactive or neutral||Similar||Higher with digital sham|
|Dovis et al , 2015||89||18||Cognitive training||ADHD||Double blind||Braingame Brian||Computer based||Braingame Brian on placebo mode||Lower intensity||Similar||NR|
|Espie et al , 2012||156||8||CBT or behavior therapy||Insomnia||Single blind||Sleepio||Web based||Imagery relief therapy||Inactive or neutral||Similar||NR|
|McEwen et al , 2014||59||7.3||Others||Stroke||Double blind||Virtual reality system (Interactive Rehabilitation Exercise software [IREX])||Extended reality||Control, virtual reality system (Interactive Rehabilitation Exercise software [IREX])||Inactive or neutral||Higher in digital sham||NR|
|Stasiak et al , 2014||34||10.3||CBT or behavior therapy||Depression||Double blind||The Journey||Computer based||Placebo program with psychoeducational content (computerized program with a brief psychoeducational content [CPE])||Inactive or neutral||Similar||NR|
|In et al , 2012||19||4||Others||Stroke||NR||Virtual reality reflection therapy program||Extended reality||Control (sham program)||Inactive or neutral||Similar||NR|
|Bentz et al , 2021||74||7||Others||Phobia||Single blind||Easy Heights||Extended reality||Google Street View app||Inactive or neutral||Higher in DTx||NR|
|Cheng et al , 2021 (Sleep to Prevent Evolving Affecting Disorders [SPREAD] trial)||1358||6||CBT or behavior therapy||Insomnia||Single blind||Sleepio||Web based||Web-based sleep education and sleep hygiene education||Inactive or neutral||Higher in DTx||NR|
|Espie et al , 2019 (Digital Insomnia Therapy to Assist Your Life as Well as Your Sleep [DIALS] study)||1448||24||CBT or behavior therapy||Insomnia||Single blind||Sleepio||Web based||Web-based sleep education and sleep hygiene education||Inactive or neutral||Higher in DTx||NR|
|Keefe et al , 2019 (STARS-MDDl)||74||6||Cognitive training||Depression||Double blind||AKL-T03||Mobile devices based||AKL-T09||Another active||NR||NR|
|Schenker et al [, ], 2019||55||NR||Others||Migraine||Double blind||Relivion||Wearable||Relivion Device—sham stimulation||Lower intensity||NR||NR|
|Bucci et al , 2018||36||22||CBT or behavior therapy||Psychosis||Single blind||Actissist||Mobile devices based||ClinTouch app||Missing||Higher in digital sham||NR|
|Azevedo et al , 2017||52||NR||Others||Anxiety||Single blind||Doppel device||Wearable||Doppel device with active component (Doppel) turned off condition||Inactive or neutral||NR||NR|
|Dennis-Tiwary et al , 2017||33||4||Others||Anxiety||Double blind||Personal Zen: attention bias modification training||Mobile devices based||Placebo—attention training application||Inactive or neutral||NR||NR|
|Perry et al , 2017 (TriPoD)||540||77.4||CBT or behavior therapy||Depression||Double blind||SPARX-R (Smart, Positive, Active, Realistic, X-factor thoughts)||Web based||lifeSTYLE||Inactive or neutral||Higher in DTx||NR|
|Christensen et al , 2016 (GoodNight study)||1149||25.8||CBT or behavior therapy||Insomnia||Single blind||SHUTi||Web based||HealthWatch||Inactive or neutral||Higher in DTx||NR|
|Franklin et al , 2016||Study 1: 114, study 2: 131, and study 3: 163||4||CBT or behavior therapy||Depression||NR||TECm||Mobile devices based||TEC with neutral (or blank) images||Inactive or neutral||Higher in digital sham||Similar|
aDTx: digital therapeutics.
bADHD: attention-deficit hyperactivity disorder.
cNR: not reported.
dCBT: cognitive behavioral therapy.
eASD: autism spectrum disorder.
fSTARS-ADHD: Software Treatment for Actively Reducing Severity of ADHD.
gMS: multiple sclerosis.
hSUD: substance use disorder.
iCWMT: Cogmed Working Memory Training.
jEFMT: Emotional Faces Memory Task training.
kDEAL: Depression-Alcohol Project.
lSTARS-MDD: Software Treatments for Actively Reducing Severity of Cognitive Deficits in Major Depressive Disorder.
mTEC: therapeutic evaluative conditioning.
DTx or digital sham were considered to demonstrate efficacy if they showed statistically significant improvement in primary outcomes from baseline to follow-up. DTx was considered better than digital sham if there were statistically significant differences between the 2 groups at follow-up. For studies in which patients were not blinded per se but were not informed about whether the treatment they were receiving was active or control or in situations where the expectations from the 2 arms were equalized, they were considered to be blinded. For dropout rate and patient engagement or satisfaction rates, we considered a difference of >5% when reporting the DTx-sham comparison.
On the basis of the efficacy results of DTx and sham, the 32 studies were categorized as follows:
Group 1: Studies Where Neither Sham Nor DTx Showed Improvement From Baseline (DTx–, Sh–; n=2)
In 2 studies, the DTx itself failed to show improvement from baseline. One study was underpowered, with data from only 6 patients, and the other was an unpublished study [, , ].
Group 2: Studies Where DTx Showed Improvement From Baseline But Sham Did Not (DT+, Sh–; n=7)
All these studies involved CBT or behavioral therapy or cognitive training. The sample size ranged from 19 to 348 patients, and the study duration ranged from 4 to 7 weeks (except for 2 studies with longer duration). Three studies were double blind and 3 were single blind. In most studies (4/7, 57%), the active component of the DTx was replaced by an inactive or neutral component. Indications included autism spectrum disorder (n=2), ADHD, intellectual disability, dementia, MS, and insomnia (1 each).
In these papers, the authors discussed the results in the active treatment arms only in relation to the digital sham and did not specifically explain the results of the digital sham.
Group 3: Studies Where Sham and DTx Showed Comparable Improvement From Baseline (DTx+, Sh+; n=10)
These included 4 studies involving CBT or behavioral therapy, 3 involving cognitive training, and 3 others involving attention bias modification training and emotion regulation training. The sample size ranged from 18 to 488 patients, and the study duration ranged from 4 to 16 weeks. Five studies were double blind and 5 were single blind. In most cases (7 studies), the active component of the DTx was replaced by an inactive or neutral component. The indications were autism spectrum disorder (1/10, 10%), ADHD (2/10, 20%), anxiety (5/10, 50%), and alcohol use disorder (2/10, 20%).
The key features of these studies and the reasons for their failure are summarized in. Only a few authors attributed study failure to an ineffective DTx ( ), noting deficiencies in control arms in previous studies that were remedied by an appropriately designed sham control in their study. In other cases, the authors felt that their DTx was probably effective but had not been implemented appropriately and needed to be made more engaging or intense. Most authors focused on the challenges with the sham control arm, speculating that the sham may have been too engaging or that the short-term distraction provided by the sham control played a large role. Some authors also proposed a more direct therapeutic effect of sham control on resilience, motivation, and cognition. Finally, some authors suggested that an adequately powered study may have succeeded in demonstrating the comparative efficacy of DTx.
|Study||Indication||DTx and sham||Potential reasons for study failure discussed by the study authors||Additional considerations|
|Berman et al , 2020||Problematic alcohol use|
|Teng et al , 2019||Anxiety|
|Batterham et al , 2018||Anxiety, depression, etc|
|Davies et al , 2017||Drinking issues|
|Scholten et al , 2016||Anxiety|
|Schoneveld et al , 2016||Anxiety|
|de Vries et al , 2015||ASDb|
|Enock et al , 2014||Anxiety|
|Bikic et al , 2017||ADHDd|
|Chacko et al , 2014||ADHD|
aABM: attentional bias modification.
bASD: autism spectrum disorder.
cEF: executive function.
dADHD: attention-deficit hyperactivity disorder.
eCWMT: Cogmed working memory training.
Group 4: Studies Where Both Sham and DTx Showed Improvement From Baseline and DTx Was Significantly Better Than Sham (DTx++, Sh+; n=13)
This included 8 studies involving CBT or behavioral therapy, 2 involving cognitive training, and 3 involving others (balance training, rehabilitation, and stimulation). The sample size ranged from 19 to 252 patients (except 1 study involving 1051 patients), and the study duration ranged from 4 to 26 weeks. Seven studies were double blind, 3 were single blind, and 3 did not report the blinding status. In most cases (11/13, 85% studies), the active component of the DTx was replaced by an inactive or neutral component.
The indications were depression (4/11, 36%), anxiety (2/11, 18%), stroke (2/11, 18%), pain (1/11, 9%), migraine (1/11, 9%), MS (1/11, 9%), ADHD (1/11, 9%), and insomnia (1/11, 9%).
Notably, in both stroke studies, the patients were also receiving conventional stroke rehabilitation treatment in both arms. In addition, in the study by Wijnhoven et al , the 2 treatment arms differentiated on parent-rated anxiety but not on child-rated anxiety, whereas in the study by Deady et al [ ], differences between DTx and sham arms did not persist at follow-up.
A summary of all studies based on the types of results discussed in, showing DTx type, blinding status, nature of sham, and other parameters, is presented in Figure S1 in ; high-level trends and their potential implications are covered in the Discussion section.
Seven studies had a sample size of <50 [, , , , , , ]. When the group-wise analysis was repeated without these small studies, the findings remained unchanged.
We identified >300 controlled trials of DTx in neuroscience, of which only approximately 20% used a digital sham control. Trials involving XR technology were the most common and were mostly available in recent trial registry records. Stroke and depression or anxiety were the most common indications, whereas physical rehabilitation and CBT or behavioral therapy were the most common active treatments.
Overview of Digital Shams
We categorized the studies into 4 groups (groups 1-4) based on the impact of DTx and sham on outcomes and used this framework to assess the impact of study design elements and evaluate author perspectives on the role of the digital sham.
We identified the following key issues:
- Sham controls were not validated before implementation.
- Sham treatments varied considerably, from being nearly identical to the DTx [ , , ] to using different software programs altogether [ , ]. This was further compounded by the fact that the shams were often not described adequately, making it difficult to evaluate DTx-sham differences consistently, and there was insufficient information on elements that might impact outcomes such as the expected level of engagement or difficulty relative to the DTx.
- In 26% (11/43) of the studies, the impact of the sham on outcomes (ie, change in scores from baseline to end point) was not reported, with the studies reporting only DTx versus sham differences over time. This limited our ability to fully understand and explore the study results.
- Only 26% (11/43) of the studies reported the actual level of engagement and satisfaction with the sham (compared with DTx). For the remaining studies, it was difficult to discern how much of the DTx-sham differences were driven by differences in engagement between the 2 arms. A recent meta-analysis showed that higher engagement features in apps targeting depression and anxiety symptoms were associated with larger effect sizes. The authors discussed the use of persuasive system design and gamification to improve user engagement and, thereby, clinical outcomes with these apps. This implies that a rigorous comparison of 2 treatment arms would need them to be matched on engagement levels to avoid confounding. At the very least, the level of engagement across the 2 arms would need to be reported for transparency [ ].
These issues limited the ability to compare the sham control arms across studies. For example, in studies that failed to differentiate DTx from sham, a few authors attributed study failure to DTx inefficacy [, , ], whereas most subjectively attributed it to the placebo effect or to the presence of active elements in the sham [ , , ]. Underlying these challenges was the lack of consensus on what is considered a digital sham. In some cases, the control was not called sham or placebo, although it met the characteristics of a digital sham. Currently, DTx are well understood, and there exists a definition of DTx that is accepted in the field and has been formalized by the Digital Therapeutics Alliance [ ]. However, there is no universal definition of digital sham (we had to build an operational definition for use in this review). We believe that this lack of consensus around what is a digital sham lies at the root of the subsequent issues seen in the reporting and interpretation of digital sham studies. Lutz et al [ ] also identified similar issues in their review of controls for DTx, where they highlighted the inconsistencies in nomenclature and the fact that multiple terms such as sham, placebo control, digital control, and attention-matched placebo control have been used in the literature; these different terms may correspond to different definitions making interstudy comparisons difficult. The authors also noted inconsistencies in the reporting of the control arm in DTx studies and highlighted the need to report data on engagement levels with the control arm [ ]. By analyzing the digital sham–related data systematically and across a larger data set, we were also able to quantify the extent of missing data in the literature: only 1 in 4 studies reported data on the level of engagement with sham, and only 1 in 4 studies reported the change from baseline in outcomes in the sham control arm.
Recommendations on Digital Sham Design and Reporting
and show parameters that we recommend should be included in the design and reporting of studies using digital shams. This information will help assess how well constructed the sham was, that is, whether it differed from the DTx in aspects other than the purported active components (design) and allow assessment of the sham versus the DTx in terms of engagement or motivation or adherence during the study, providing appropriate context for interpretation (reporting). These recommendations are DTx specific and may thus be combined with already existing generic frameworks to ensure adequate reporting of digital sham control arms [ ]. Although these requirements may appear quite detailed, particularly when compared with designing and reporting sham controls in traditional pharmaceutical interventions, we believe these are needed and that these aspects would (or should) already have been considered by study teams at the time of designing the studies and selecting appropriate digital sham controls, specifically because of the characteristics that differentiates digital intervention from traditional biological interventions described in the Introduction section. Despite some differing opinions to incorporating robust placebo control requirements in DTx studies (based on concerns about potential false negatives or the belief that inadequate controls are acceptable in pilot studies so long as authors acknowledge those limitations), it is widely acknowledged that maintaining methodological rigor in these studies is paramount, especially for such a novel class of medical interventions where long-term effectiveness and value still has to be fully established. Merely demonstrating the benefits of DTx only compared with no treatment or a waitlist control through a full clinical development is insufficient for establishing a causal relationship with the active component of DTx, as it does not provide clarity on whether the effects are because of the active component or patient engagement or expectations or motivation. Therefore, it is necessary to use a robust sham control group that effectively addresses various factors, including retest effects, regression to the mean, social contact, and differential motivation and expectations across the study groups [ , ].
For studies that have been adequately reported (using the parameters inand ), the study results can then be interpreted with more confidence using the algorithm shown in . For example, in a group 2 study (DTx+ Sh–), a true positive result (ie, the DTx is efficacious, and the DTx-sham difference can be fully attributed to the active components) would be more likely if the sham control arm was well constructed and showed equal or higher engagement or motivation or adherence compared with the DTx arm. In contrast, in a group 3 study (DTx+ Sh+), a true negative result (ie, the DTx is nonefficacious) would be more likely if the sham control arm was well constructed and showed equal or lower engagement or motivation or adherence compared with the DTx arm. Similarly, positive group 4 (DTx++ Sh+) studies would be considered true positives, unless there were considerable issues with the sham design. Note that this algorithm cannot be reliably applied across most of the current DTx literature because of inadequate information on the construction of the sham control arm, including the engagement levels seen in that arm during the studies.
Study protocol: the following information about the sham arm should be determined a priori and reported as part of clinical trial protocols.
- What is the scope of the study with respect to digital sham? Is it to identify and characterize active components of the digital therapeutics (DTx) or is it as a control arm to conclusively demonstrate DTx efficacy, that is, late phase validation of DTx?
- Whether the sham had been validated as a placebo control for this DTx in an earlier study
Nature of difference between sham and DTx
- What are the active components in the DTx?
- Whether the underlying software or technology in sham control was fundamentally different from that in the DTx
- Whether there were elements of the DTx that were switched off in the sham
- Whether the active component of the DTx was removed from the sham
- Whether the active component of the DTx was replaced with an inactive or neutral component
- Whether the sham was the same as the DTx but of lower intensity or difficulty compared with the DTx
- Whether the sham had adaptive levels (levels that changed based on users’ performance) that were comparable with those in the DTx
Sham design for engagement and motivation
- Whether the sham was designed to be comparable with the DTx in terms of engagement
- Whether the sham was designed to be as motivating and satisfying to users as the DTx
Blinding and expectation matching
- Whether the outcome assessments would be done in a blinded manner
- Whether the patients would be blinded. If not, whether the patients would be informed about which arm was active and which was control
- Whether expectations would be matched between the active and control arms, that is, whether the researchers would prime participants in both groups in an equal manner to expect improvement with their allocated intervention
Session and treatment details
- Number of sessions
- Duration of each session
- Duration of treatment
Study results: following the conduct of the study, the following information should be reported in the study publications.
- Success of blinding (for blinded studies)
Patient-reported experience with sham
- Patient-reported difficulty levels with digital therapeutics (DTx) and sham
- Patient-reported satisfaction with DTx and sham
- Patient engagement levels with DTx and sham, based on, for example, time spent on the intervention, number of completed sessions, and level of difficulty reached
- Dropout rates with DTx and sham
Impact on outcomes
- Magnitude of improvement from baseline, reported separately for DTx and sham arms
|In the study, how did sham compare with the DTxa in terms of engagement or motivation or adherence?||Was sham well constructed, that is, was it identical to DTx in all aspects except the purported active components?|
|Negative results (DTx+ sham+)|
|Sham>DTx (eg, sham more engaging)||May be false negative but complete interpretation is difficult|
|Sham≤DTx (eg, sham equally or less engaging)||May be true negative but complete interpretation is difficult|
|Positive results (DTx+ sham−)|
|Sham≥DTx (eg, sham equally or more engaging)||May be true positive but complete interpretation is difficult|
|Sham<DTx (eg, sham less engaging)||May be false positive but complete interpretation is difficult|
aDTx: digital therapeutics.
Given the heterogeneity of digital shams, all aspects listed inwill not always apply; therefore, we recommend reporting as much information as possible. In the future, we hope that a simple scale can be created to capture the required minimum information and perhaps derive a score or category classifying the sham in terms of comparability with the respective DTx.
Recommendations on Study Design
In proof-of-concept studies, the sham should mimic the DTx as closely as possible, differing only in the putative active components. In this early phase of development, researchers may need to experiment with different sham control arms to identify and confirm the actual active components of the DTx. In addition, as the sham arm can reasonably be expected to have a substantial impact on efficacy on its own, the risk of a false-negative result will need to be addressed by a priori statistical planning, anticipating a reduced DTx versus sham difference, and ensuring that the sample size is large enough for adequate study power.
In confirmatory trials, although the sham control arm may be omitted in favor of a waitlist control or treatment-as-usual comparator, we recommend using validated shams to conclusively demonstrate the efficacy of DTx. Thus, a 3-arm study would be ideal, where the DTx arm would be compared with a digital sham as well as with a waitlist control or treatment-as-usual comparator.
On the basis of the trends in our analysis, we recommend that more attention be paid to sham controls in psychiatric indications such as anxiety and substance use disorder because of the high risk of the impact of shams on outcomes. In such studies, patients appeared to respond to aspects of games that were not considered to be an active component or to the fact that they were being interviewed. More attention also needs to be paid to studies involving cognitive training, which risks failure because of the high impact of shams on some cognitive outcomes. This was seen not only in studies that used a nonadaptive sham or a sham with a lower intensity of training but also with shams consisting of a simple game such as Tetris; in these studies, even the control interventions that were assumed to be inactive resulted in a significant effect on outcomes. In contrast, studies involving CBT or behavioral therapy were more consistently successful.
In addition, the sample size and study duration should be adequate, and studies should preferably be double blind. In our analysis, larger and longer studies and industry-sponsored trials (which were generally larger than academic trials and more likely to be double blind) were more likely to show positive results. Finally, the analysis could be stratified by engagement levels to estimate the dose-response relationship, and novel study approaches (eg, a sham run-in period) were used to identify placebo responders. Blinding in DTx studies can be challenging, particularly for DTx involving psychological and behavioral therapies. A digital sham with no active component may have low face validity because of which some participants may become unblind to the treatment assignment . Challenges with blinding notwithstanding, there are sufficient advantages to blinding for us to recommend that this be implemented in DTx trials and that the success of the blinding be reported ( ).
In summary, there is no consensus definition of digital sham, and sham controls were used in only a small proportion (68/313, 21.7%) of the DTx studies. However, the role of digital shams remains underappreciated. For example, a recent review of evidence generation approaches in DTx noted the lack of high-quality evidence and discussed innovative strategies to meet the evidence requirements of the stakeholders; however, the challenges associated with digital sham were not discussed .
Our analysis shows that the use of sham control is, however, very important in the development of new DTx that are assessed using clinical trials, that is, DTx involving behavioral or training interventions that directly impact patient outcomes. Currently, there is a limited understanding of how to design an adequate sham control arm, and some sham-controlled trials may have failed for this reason. A lack of clarity regarding the active components of DTx can also result in inadequate sham controls; in many studies, key active components appeared to be shared between the DTx and sham [, , , , ]. We strongly recommend the use of sham-controlled studies in early research to identify DTx active ingredients, as this is a critical step in the optimization of the development of the intervention as well as in its validation. Given the central role of digital sham in generating robust controlled evidence, juxtaposed with our findings that demonstrate inadequacies in the use of sham controls in the literature, it is evident that implementing an adequately designed sham control is necessary to accurately evaluate the efficacy of DTx. Without appropriate sham-controlled data, the possibility of false-positive results cannot be ruled out. Finally, the sham control arm may need to adapt to changing DTx as the DTx evolves during clinical development; this could be an area of future investigation.
Although this review did not focus on the real-world applications of DTx, we noted that limited long-term and real-world data have been published on DTx efficacy. The study duration was ≤6 months in 82% (56/68) of the sham-controlled studies, and the transfer of acquired skills to long-term real-world outcomes was rarely assessed. Furthermore, assessing and separating the data into groups for analysis was not straightforward. In some studies, the identification of primary outcomes was not consistently reported, and the efficacy of DTx and sham interventions varied across different outcomes within these studies. Consequently, a nuanced evaluation is necessary to determine the overall effectiveness of the DTx and sham interventions. As noted earlier, another key challenge was the absence of a consensus definition for digital sham.
In addition to our focus on the definition and reporting of digital sham, there remains a need to appropriately define and implement other control strategies in DTx trials . For instance, controls that vary from the DTx solely in terms of the intensity of the active component or in terms of attention and engagement also play a role in DTx development. Different control strategies and solutions are more or less appropriate at different stages of the clinical development of a DTx and based on the objectives of the clinical study. We advocate additional research in this area to establish standardized definitions for control arms in DTx trials more comprehensively.
Gaps and Limitations
Screening, data extraction, data analysis, and categorization of the studies into groups were performed by a single reviewer. However, a quality check of all the steps was conducted to ensure the quality of this review.
In addition, despite our systematic approach, it is important to note that there might be DTx that we did not identify. This is because the field of DTx is vast and rapidly evolving. Furthermore, the line of demarcation between DTx and wellness apps is not always clearly defined. We distinguished between DTx and wellness apps based on the presence of well-developed software programs or algorithms. However, for a few therapeutic products, the underlying mechanism of action has not been adequately described in the available literature, and because of this, a definitive assessment could not be made. Finally, as the searches were completed in late 2021, the literature published from 2022 onward was not represented in this review.
We believe that digital shams are critical to the clinical development of DTx. Sham-controlled studies should be routinely used in trials to help identify DTx active components (ie, early-phase studies) as well as to confirm the efficacy of DTx (ie, late-phase studies). The use of shams early in development will ensure that the appropriate sham control is used in later confirmatory trials. Overall, the inclusion of principles for sham design and deployment into best practice guidelines for clinical investigations of DTx will ensure the correct evaluation of benefits for patients and the health care system at large.
The study was funded by F Hoffmann-La Roche. The authors wish to thank Anand Jacob, PhD, and Chaynika Sharma, MPharm, Bridge Medical Consulting Limited, for editorial assistance (funded by F Hoffmann-La Roche). The authors would also like to thank Brigitta Monz (F Hoffmann-La Roche Ltd) for their early comments on the study ideation and design.
The full data set will be made available upon request, after publication of this paper. Requests should be sent to DM.
DM, EB, KHP, and SK designed the study. GK and PM performed the literature search and data collection. SK, GK, and PM verified the underlying data. All authors interpreted the data. SK, GK, and PM drafted the manuscript. All authors were involved in critically revising the manuscript. All authors approved the final version of the manuscript. All authors had access to all the data in the study and had final responsibility for the decision to submit for publication.
Conflicts of Interest
SK, GK, and PM are employees of Bridge Medical Consulting Ltd. DM and EB are employees of F. Hoffmann-La Roche Ltd, which has digital health solutions that are under development and on the market. KHP was an employee of F. Hoffmann-La Roche Ltd at the time the study was conducted and is currently an employee of Boehringer-Ingelheim GmbH & Co. KG.
Additional tables and figures.DOCX File , 223 KB
PRISMA Checklist.PDF File (Adobe PDF File), 69 KB
- Sverdlov O, van Dam J, Hannesdottir K, Thornton-Wells T. Digital therapeutics: an integral component of digital innovation in drug development. Clin Pharmacol Ther 2018 Jul 23;104(1):72-80 [CrossRef] [Medline]
- AMCP partnership forum: digital therapeutics-what are they and where do they fit in pharmacy and medical benefits? J Manag Care Spec Pharm 2020 May;26(5):674-681 [https://europepmc.org/abstract/MED/32175784] [CrossRef] [Medline]
- Dang A, Arora D, Rane P. Role of digital therapeutics and the changing future of healthcare. J Family Med Prim Care 2020 May;9(5):2207-2213 [https://europepmc.org/abstract/MED/32754475] [CrossRef] [Medline]
- Abbadessa G, Brigo F, Clerico M, De Mercanti S, Trojsi F, Tedeschi G, et al. Digital therapeutics in neurology. J Neurol 2022 Mar 20;269(3):1209-1224 [http://hdl.handle.net/2318/1844671] [CrossRef] [Medline]
- Ferreira-Brito F, Fialho M, Virgolino A, Neves I, Miranda AC, Sousa-Santos N, et al. Game-based interventions for neuropsychological assessment, training and rehabilitation: which game-elements to use? A systematic review. J Biomed Inform 2019 Oct;98:103287 [https://linkinghub.elsevier.com/retrieve/pii/S1532-0464(19)30206-0] [CrossRef] [Medline]
- Shah A, Kraemer KR, Won CR, Black S, Hasenbein W. Developing digital intervention games for mental disorders: a review. Games Health J 2018 Aug;7(4):213-224 [CrossRef] [Medline]
- Cañado ML, Padilla JR. Digital Competence Development in Higher Education. Berlin, Germany: Peter Lang Verlag; 2021.
- Patel NA, Butte AJ. Characteristics and challenges of the clinical pipeline of digital therapeutics. NPJ Digit Med 2020 Dec 11;3(1):159 [https://doi.org/10.1038/s41746-020-00370-8] [CrossRef] [Medline]
- Joint HMA/EMA workshop on artificial intelligence in medicines regulation. European Medicines Agency. 2022 Oct 25. URL: https://www.ema.europa.eu/en/documents/report/report-joint-hma/ema-workshop-artificial-intelligence-medicines-regulation_en.pdf [accessed 2022-01-07]
- Wijnhoven LA, Creemers DH, Vermulst AA, Lindauer RJ, Otten R, Engels RC, et al. Effects of the video game 'Mindlight' on anxiety of children with an autism spectrum disorder: a randomized controlled trial. J Behav Ther Exp Psychiatry 2020 Sep;68:101548 [CrossRef] [Medline]
- Teng MH, Hou YM, Chang SH, Cheng HJ. Home-delivered attention bias modification training via smartphone to improve attention control in sub-clinical generalized anxiety disorder: a randomized, controlled multi-session experiment. J Affect Disord 2019 Mar 01;246:444-451 [CrossRef] [Medline]
- Enock PM, Hofmann SG, McNally RJ. Attention bias modification training via smartphone to reduce social anxiety: a randomized, controlled multi-session experiment. Cogn Ther Res 2014 Mar 4;38(2):200-216 [https://link.springer.com/article/10.1007/s10608-014-9606-z] [CrossRef]
- Lutz J, Offidani E, Taraboanta L, Lakhan SE, Campellone TR. Appropriate controls for digital therapeutic clinical trials: a narrative review of control conditions in clinical trials of digital therapeutics (DTx) deploying psychosocial, cognitive, or behavioral content. Front Digit Health 2022 Aug 18;4:823977 [https://europepmc.org/abstract/MED/36060538] [CrossRef] [Medline]
- Home page. National Library of Medicine. URL: https://clinicaltrials.gov/ [accessed 2023-06-22]
- Home page. European Union Clinical Trials Register. 2016. URL: https://www.clinicaltrialsregister.eu/ [accessed 2023-06-22]
- Home page. Pharma Intelligence UK Limited (Citeline). URL: https://citeline.informa.com/trials [accessed 2023-06-22]
- Cimenser A, Hempel E, Travers T, Strozewski N, Martin K, Malchano Z, et al. Sensory-evoked 40-Hz gamma oscillation improves sleep and daily living activities in Alzheimer’s disease patients. Front Syst Neurosci 2021 Sep 24;15:746859 [https://europepmc.org/abstract/MED/34630050] [CrossRef] [Medline]
- Keefe R. A randomized, double-blind, controlled, 6-week trial to assess a novel digital intervention designed to improve cognitive dysfunction as adjunct therapy to antidepressant medication in adults with major depressive disorder. Neuropsychopharmacol 2019
- Schenker E, Tepper SJ, Daniel O. First non-invasive combined occipital and trigeminal nerve stimulation digital therapeutics system for treatment of migraine: a randomized, sham-controlled, double-blind clinical trial. Headache 2019:1-22 [https://assets.researchsquare.com/files/rs-1385396/v1/476d126c-7751-4003-823f-80d1035eee52.pdf?c=1645732522] [CrossRef]
- Schenker E. Digital therapeutics treatment of migraine. Headache 2019;59:67-68
- Bucci S, Barrowclough C, Ainsworth J, Machin M, Morris R, Berry K, et al. Actissist: proof-of-concept trial of a theory-driven digital intervention for psychosis. Schizophr Bull 2018 Aug 20;44(5):1070-1080 [https://europepmc.org/abstract/MED/29566206] [CrossRef] [Medline]
- Clinical trial record (NCT03751280). US National Library of Medicine. URL: https://clinicaltrials.gov/ct2/show/NCT03751280 [accessed 2022-01-05]
- Clinical trial record (NCT02828644). US National Library of Medicine. URL: https://clinicaltrials.gov/ct2/show/NCT02828644 [accessed 2022-01-05]
- Clinical trial record (NCT04779372). US National Library of Medicine. URL: https://clinicaltrials.gov/ct2/show/NCT04779372 [accessed 2022-01-05]
- Clinical trial record (NCT03322774). US National Library of Medicine. URL: https://clinicaltrials.gov/ct2/show/NCT03322774 [accessed 2022-01-05]
- Clinical trial record (NCT03613519). US National Library of Medicine. URL: https://clinicaltrials.gov/ct2/show/NCT03613519 [accessed 2022-01-05]
- Clinical trial record (NCT03724305). US National Library of Medicine. URL: https://clinicaltrials.gov/ct2/show/NCT03724305 [accessed 2022-01-05]
- Clinical trial record (NCT04112225). US National Library of Medicine. URL: https://clinicaltrials.gov/ct2/show/NCT04112225 [accessed 2022-01-05]
- Clinical trial record (NCT03135418). US National Library of Medicine. URL: https://clinicaltrials.gov/ct2/show/NCT03135418 [accessed 2022-01-05]
- Clinical trial record (NCT03458611). US National Library of Medicine. URL: https://clinicaltrials.gov/ct2/show/NCT03458611 [accessed 2022-01-05]
- Clinical trial record (NCT04225884). US National Library of Medicine. URL: https://clinicaltrials.gov/ct2/show/NCT04225884 [accessed 2022-01-05]
- Clinical trial record (NCT04443127). US National Library of Medicine. URL: https://clinicaltrials.gov/ct2/show/NCT04443127 [accessed 2022-01-05]
- Clinical trial record (NCT02066116). US National Library of Medicine. URL: https://clinicaltrials.gov/ct2/show/NCT02066116 [accessed 2022-01-05]
- Clinical trial record (NCT01677429). US National Library of Medicine. URL: https://clinicaltrials.gov/ct2/show/NCT01677429 [accessed 2022-01-05]
- Clinical trial record (NCT04139564). US National Library of Medicine. URL: https://clinicaltrials.gov/ct2/show/NCT04139564 [accessed 2022-01-05]
- Clinical trial record (NCT03185559). US National Library of Medicine. URL: https://clinicaltrials.gov/ct2/show/NCT03185559 [accessed 2022-01-05]
- Clinical trial record (NCT03631550). US National Library of Medicine. URL: https://clinicaltrials.gov/ct2/show/NCT03631550 [accessed 2022-01-05]
- Clinical trial record (NCT03934658). US National Library of Medicine. URL: https://clinicaltrials.gov/ct2/show/NCT03934658 [accessed 2022-01-05]
- Clinical trial record (NCT04040387). US National Library of Medicine. URL: https://clinicaltrials.gov/ct2/show/NCT04040387 [accessed 2022-01-05]
- Clinical trial record (NCT00850447). US National Library of Medicine. URL: https://clinicaltrials.gov/ct2/show/NCT00850447 [accessed 2022-01-05]
- Clinical trial record: (NCT04574921). US National Library of Medicine. URL: https://clinicaltrials.gov/ct2/show/NCT04574921 [accessed 2022-01-05]
- Deady M, Johnston DA, Glozier N, Milne D, Choi I, Mackinnon A, et al. A smartphone application for treating depressive symptoms: study protocol for a randomised controlled trial. BMC Psychiatry 2018 Jun 01;18(1):166 [https://bmcpsychiatry.biomedcentral.com/articles/10.1186/s12888-018-1752-5] [CrossRef] [Medline]
- Denis D, Eley TC, Rijsdijk F, Zavos HM, Keers R, Espie CA, et al. Sleep Treatment Outcome Predictors (STOP) pilot study: a protocol for a randomised controlled trial examining predictors of change of insomnia symptoms and associated traits following cognitive-behavioural therapy for insomnia in an unselected sample. BMJ Open 2017 Dec 01;7(11):e017177 [https://bmjopen.bmj.com/lookup/pmidlookup?view=long&pmid=29196479] [CrossRef] [Medline]
- Giosan C, Cobeanu O, Mogoaşe C, Szentagotai A, Mureşan V, Boian R. Reducing depressive symptomatology with a smartphone app: study protocol for a randomized, placebo-controlled trial. Trials 2017 May 12;18(1):215 [https://trialsjournal.biomedcentral.com/articles/10.1186/s13063-017-1960-1] [CrossRef] [Medline]
- Smeijers D, Koole SL. Testing the effects of a virtual reality game for aggressive impulse management (VR-GAIME): study protocol. Front Psychiatry 2019 Feb 26;10:83 [https://europepmc.org/abstract/MED/30863328] [CrossRef] [Medline]
- Transforming global healthcare by advancing digital therapeutics. Digital Therapeutics Alliance™. URL: https://dtxalliance.org/ [accessed 2022-06-30]
- Kalmbach DA, Cheng P, O'Brien LM, Swanson LM, Sangha R, Sen S, et al. A randomized controlled trial of digital cognitive behavioral therapy for insomnia in pregnant women. Sleep Med 2020 Aug;72:82-92 [https://europepmc.org/abstract/MED/32559716] [CrossRef] [Medline]
- Batterham PJ, Calear AL, Farrer L, McCallum SM, Cheng VW. FitMindKit: randomised controlled trial of an automatically tailored online program for mood, anxiety, substance use and suicidality. Internet Interv 2017 Aug 23;12:91-99 [https://linkinghub.elsevier.com/retrieve/pii/S2214-7829(17)30064-7] [CrossRef] [Medline]
- Pinheiro MB, Ho KK, Ferreira ML, Refshauge KM, Grunstein R, Hopper JL, et al. Efficacy of a sleep quality intervention in people with low back pain: protocol for a feasibility randomized co-twin controlled trial. Twin Res Hum Genet 2016 Oct;19(5):492-501 [CrossRef] [Medline]
- Cognitive remediation in posttraumatic stress disorder (NCT00850447). US National Library of Medicine. URL: https://classic.clinicaltrials.gov/ct2/show/NCT00850447 [accessed 2022-01-02]
- Franklin JC, Fox KR, Franklin CR, Kleiman EM, Ribeiro JD, Jaroszewski AC, et al. A brief mobile app reduces nonsuicidal and suicidal self-injury: evidence from three randomized controlled trials. J Consult Clin Psychol 2016 Jun;84(6):544-557 [CrossRef] [Medline]
- Study of efficacy of PEAR-004 in schizophrenia (NCT03751280). US National Library of Medicine. URL: https://classic.clinicaltrials.gov/ct2/show/NCT03751280 [accessed 2022-01-02]
- Kollins SH, DeLoss DJ, Cañadas E, Lutz J, Findling RL, Keefe RS, et al. A novel digital intervention for actively reducing severity of paediatric ADHD (STARS-ADHD): a randomised controlled trial. The Lancet Digital Health 2020 Apr;2(4):e168-e178 [CrossRef]
- Software treatment for actively reducing severity of ADHD - follow up (STARS-ADHD2) (NCT02828644). US National Library of Medicine. URL: https://classic.clinicaltrials.gov/ct2/show/NCT02828644 [accessed 2022-01-02]
- Hancock LM, Bruce JM, Bruce AS, Lynch SG. Processing speed and working memory training in multiple sclerosis: a double-blind randomized controlled pilot study. J Clin Exp Neuropsychol 2015;37(2):113-127 [CrossRef] [Medline]
- Rosa VO, Schmitz M, Moreira-Maia CR, Wagner F, Londero I, Bassotto CF, et al. Computerized cognitive training in children and adolescents with attention deficit/hyperactivity disorder as add-on treatment to stimulants: feasibility study and protocol description. Trends Psychiatry Psychother 2017 Apr;39(2):65-76 [https://doi.org/10.1590/2237-6089-2016-0039] [CrossRef] [Medline]
- Sosnowski DW, Stough CO, Weiss MJ, Cessna T, Casale A, Foran A, et al. Brief report: a novel digital therapeutic that combines applied behavior analysis with gaze-contingent eye tracking to improve emotion recognition in children with autism spectrum disorder. J Autism Dev Disord 2022 May;52(5):2357-2366 [CrossRef] [Medline]
- Yerys BE, Bertollo JR, Kenworthy L, Dawson G, Marco EJ, Schultz RT, et al. Brief report: pilot study of a novel interactive digital treatment to improve cognitive control in children with autism spectrum disorder and co-occurring ADHD symptoms. J Autism Dev Disord 2019 Apr 19;49(4):1727-1737 [CrossRef] [Medline]
- van der Molen MJ, van Luit JE, van der Molen MW, Klugkist I, Jongmans MJ. Effectiveness of a computerised working memory training in adolescents with mild to borderline intellectual disabilities. J Intellect Disabil Res 2010 May;54(5):433-447 [CrossRef] [Medline]
- Berman AH, Molander O, Tahir M, Törnblom P, Gajecki M, Sinadinovic K, et al. Reducing risky alcohol use via smartphone app skills training among adult internet help-seekers: a randomized pilot trial. Front Psychiatry 2020 May 27;11:434 [https://europepmc.org/abstract/MED/32536880] [CrossRef] [Medline]
- Davies EL, Lonsdale AJ, Hennelly SE, Winstock AR, Foxcroft DR. Personalized digital interventions showed no impact on risky drinking in young adults: a pilot randomized controlled trial. Alcohol Alcohol 2017 Nov 01;52(6):671-676 [CrossRef] [Medline]
- Scholten H, Malmberg M, Lobel A, Engels RC, Granic I. A randomized controlled trial to test the effectiveness of an immersive 3D video game for anxiety prevention among adolescents. PLoS One 2016 Jan 27;11(1):e0147763 [https://dx.plos.org/10.1371/journal.pone.0147763] [CrossRef] [Medline]
- Schoneveld EA, Malmberg M, Lichtwarck-Aschoff A, Verheijen GP, Engels RC, Granic I. A neurofeedback video game (MindLight) to prevent anxiety in children: a randomized controlled trial. Comput Human Behav 2016 Oct;63:321-333 [https://www.sciencedirect.com/science/article/abs/pii/S0747563216303296] [CrossRef]
- de Vries M, Prins PJ, Schmand BA, Geurts HM. Working memory and cognitive flexibility-training for children with an autism spectrum disorder: a randomized controlled trial. J Child Psychol Psychiatry 2015 May;56(5):566-576 [CrossRef] [Medline]
- Bikic A, Christensen T, Leckman JF, Bilenberg N, Dalsgaard S. A double-blind randomized pilot trial comparing computerized cognitive exercises to Tetris in adolescents with attention-deficit/hyperactivity disorder. Nord J Psychiatry 2017 Aug;71(6):455-464 [CrossRef] [Medline]
- Chacko A, Bedard AC, Marks DJ, Feirsen N, Uderman JZ, Chimiklis A, et al. A randomized clinical trial of Cogmed working memory training in school-age children with ADHD: a replication in a diverse sample using a control condition. J Child Psychol Psychiatry 2014 Mar;55(3):247-255 [https://europepmc.org/abstract/MED/24117656] [CrossRef] [Medline]
- Bove R, Rowles W, Zhao C, Anderson A, Friedman S, Langdon D, et al. A novel in-home digital treatment to improve processing speed in people with multiple sclerosis: a pilot study. Mult Scler 2021 Apr;27(5):778-789 [CrossRef] [Medline]
- Garcia LM, Birckhead BJ, Krishnamurthy P, Sackman J, Mackey IG, Louis RG, et al. An 8-week self-administered at-home behavioral skills-based virtual reality program for chronic low back pain: double-blind, randomized, placebo-controlled trial conducted during COVID-19. J Med Internet Res 2021 Feb 22;23(2):e26292 [https://www.jmir.org/2021/2/e26292/] [CrossRef] [Medline]
- Pasula S, Thankachan L, Rao AS, Prasad BD. A comprehensive study on Nerivio Migra. World J Curr Med Pharm Res 2020 Feb;2(1):30-32 [https://wjcmpr.com/index.php/journal/article/view/81] [CrossRef]
- Iacoviello BM, Murrough JW, Hoch MM, Huryk KM, Collins KA, Cutter GR, et al. A randomized, controlled pilot trial of the emotional faces memory task: a digital therapeutic for depression. NPJ Digit Med 2018 Jun 06;1(1):21 [https://doi.org/10.1038/s41746-018-0025-5] [CrossRef] [Medline]
- Parks AC, Williams AL, Tugade MM, Hokes KE, Honomichl RD, Zilca RD. Testing a scalable web and smartphone based intervention to improve depression, anxiety, and resilience: a randomized controlled trial. Int J Wellbeing 2018 Dec 08;8(2):22-67 [https://internationaljournalofwellbeing.org/index.php/ijow/article/view/745] [CrossRef]
- Zwerenz R, Becker J, Knickenberg RJ, Siepmann M, Hagen K, Beutel ME. Online self-help as an add-on to inpatient psychotherapy: efficacy of a new blended treatment approach. Psychother Psychosom 2017;86(6):341-350 [CrossRef] [Medline]
- Deady M, Mills KL, Teesson M, Kay-Lambkin F. An online intervention for co-occurring depression and problematic alcohol use in young people: primary outcomes from a randomized controlled trial. J Med Internet Res 2016 Mar 23;18(3):e71 [https://www.jmir.org/2016/3/e71/] [CrossRef] [Medline]
- Dovis S, van der Oord S, Wiers RW, Prins PJ. Improving executive functioning in children with ADHD: training multiple executive functions within the context of a computer game. a randomized double-blind placebo controlled trial. PLoS One 2015 Apr 6;10(4):e0121651 [https://dx.plos.org/10.1371/journal.pone.0121651] [CrossRef] [Medline]
- Espie CA, Kyle SD, Williams C, Ong JC, Douglas NJ, Hames P, et al. A randomized, placebo-controlled trial of online cognitive behavioral therapy for chronic insomnia disorder delivered via an automated media-rich web application. Sleep 2012 Jun 01;35(6):769-781 [https://europepmc.org/abstract/MED/22654196] [CrossRef] [Medline]
- McEwen D, Taillon-Hobson A, Bilodeau M, Sveistrup H, Finestone H. Virtual reality exercise improves mobility after stroke: an inpatient randomized controlled trial. Stroke 2014 Jun;45(6):1853-1855 [CrossRef] [Medline]
- Stasiak K, Hatcher S, Frampton C, Merry SN. A pilot double blind randomized placebo controlled trial of a prototype computer-based cognitive behavioural therapy program for adolescents with symptoms of depression. Behav Cogn Psychother 2014 Jul;42(4):385-401 [CrossRef] [Medline]
- In TS, Jung KS, Lee SW, Song CH. Virtual reality reflection therapy improves motor recovery and motor function in the upper extremities of people with chronic stroke. J Phys Ther Sci 2012;24(4):339-343 [https://www.jstage.jst.go.jp/article/jpts/24/4/24_339/_article] [CrossRef]
- Bentz D, Wang N, Ibach MK, Schicktanz NS, Zimmer A, Papassotiropoulos A, et al. Effectiveness of a stand-alone, smartphone-based virtual reality exposure app to reduce fear of heights in real-life: a randomized trial. NPJ Digit Med 2021 Feb 08;4(1):16 [https://doi.org/10.1038/s41746-021-00387-7] [CrossRef] [Medline]
- Cheng P, Casement MD, Kalmbach DA, Castelan AC, Drake CL. Digital cognitive behavioral therapy for insomnia promotes later health resilience during the coronavirus disease 19 (COVID-19) pandemic. Sleep 2021 Apr 01;44(4):zsaa258 [https://europepmc.org/abstract/MED/33249492] [CrossRef] [Medline]
- Espie CA, Emsley R, Kyle SD, Gordon C, Drake CL, Siriwardena AN, et al. Effect of digital cognitive behavioral therapy for insomnia on health, psychological well-being, and sleep-related quality of life: a randomized clinical trial. JAMA Psychiatry 2019 Jan 01;76(1):21-30 [https://europepmc.org/abstract/MED/30264137] [CrossRef] [Medline]
- Azevedo RT, Bennett N, Bilicki A, Hooper J, Markopoulou F, Tsakiris M. The calming effect of a new wearable device during the anticipation of public speech. Sci Rep 2017 May 23;7(1):2285 [https://doi.org/10.1038/s41598-017-02274-2] [CrossRef] [Medline]
- Dennis-Tiwary TA, Denefrio S, Gelber S. Salutary effects of an attention bias modification mobile application on biobehavioral measures of stress and anxiety during pregnancy. Biol Psychol 2017 Jul;127:148-156 [https://europepmc.org/abstract/MED/28478138] [CrossRef] [Medline]
- Perry Y, Werner-Seidler A, Calear A, Mackinnon A, King C, Scott J, et al. Preventing depression in final year secondary students: school-based randomized controlled trial. J Med Internet Res 2017 Nov 02;19(11):e369 [https://www.jmir.org/2017/11/e369/] [CrossRef] [Medline]
- Christensen H, Batterham PJ, Gosling JA, Ritterband LM, Griffiths KM, Thorndike FP, et al. Effectiveness of an online insomnia program (SHUTi) for prevention of depressive episodes (the GoodNight Study): a randomised controlled trial. Lancet Psychiatry 2016 Apr;3(4):333-341 [CrossRef] [Medline]
- Clinical trial results for protocol CPEA001A12201. Novartis. 2020. URL: https://www.novctrd.com/ctrdweb/trialresult/trialresults/pdf?trialResultId=17712
- Comstock J. Novartis trial shows no benefits from Pear's schizophrenia app as CEO cites trial irregularities. Mobihealthnews. 2021 Jan 13. URL: https://www.mobihealthnews.com/news/novartis-trial-shows-no-benefits-pears-schizophrenia-app-ceo-cites-trial-irregularities#:~:text=The%20single%2Dblinded%20112%2Dperson,significant%20adverse%20effects%20were%20reported [accessed 2022-04-27]
- Wu A, Scult MA, Barnes ED, Betancourt JA, Falk A, Gunning FM. Smartphone apps for depression and anxiety: a systematic review and meta-analysis of techniques to increase engagement. NPJ Digit Med 2021 Feb 11;4(1):20 [https://doi.org/10.1038/s41746-021-00386-8] [CrossRef] [Medline]
- Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ 2014 Mar 07;348(mar07 3):g1687 [https://core.ac.uk/reader/19913708?utm_source=linkout] [CrossRef] [Medline]
- Boot WR, Simons DJ, Stothart C, Stutts C. The pervasive problem with placebos in psychology: why active control groups are not sufficient to rule out placebo effects. Perspect Psychol Sci 2013 Jul;8(4):445-454 [CrossRef] [Medline]
- Guo C, Ashrafian H, Ghafur S, Fontana G, Gardner C, Prime M. Challenges for the evaluation of digital health solutions-a call for innovative evidence generation approaches. NPJ Digit Med 2020 Aug 27;3(1):110 [https://doi.org/10.1038/s41746-020-00314-2] [CrossRef] [Medline]
|ADHD: attention-deficit/hyperactivity disorder|
|CBT: cognitive behavioral therapy|
|DTx: digital therapeutics|
|MS: multiple sclerosis|
|PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses|
|SPARX: Smart, Positive, Active, Realistic, X-factor|
|XR: extended reality|
Edited by A Mavragani; submitted 16.12.22; peer-reviewed by B Birckhead, H Luderer; comments to author 30.03.23; revised version received 26.06.23; accepted 20.07.23; published 24.10.23Copyright
©Ernst Bos, Katrin H Preller, Gavneet Kaur, Pooja Malhotra, Saifuddin Kharawala, Dario Motti. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 24.10.2023.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.