Beyond the Trial: Systematic Review of Real-World Uptake and Engagement With Digital Self-Help Interventions for Depression, Low Mood, or Anxiety

Introduction

Background

Digital interventions (including online or computerized programs or apps) have been shown to be efficacious for depression and anxiety [1-4] and they provide the opportunity to extend psychological therapy to people who might otherwise not receive it [3-5]. Adherence to digital interventions is important for therapeutic gains [3,6] and is generally satisfactory relative to face-to-face interventions [7]. Self-help interventions, provided without guidance or personal support, might reach users who are unable or unwilling to seek help and may be scaled up at lower cost than interventions involving assistance [4,5]. Adherence to self-help is generally lower than that to guided interventions [8], although approaches such as persuasive design and telepresence may enhance retention [5,9] and advantages such as scalability mean that self-help remains worthy of attention.

Interventions may have poorer outcomes when implemented in community or clinical settings than they do in research trials [10-12]. Proven interventions can fail in the real world because translation from research trials may involve changes in the conditions under which the original results were obtained [10,11]. For instance, trials may exclude participants with complex issues, and trial participants may have additional motivations to complete interventions, such as to please researchers or to help others. Trial participants might also benefit from assessment effects or face-to-face contacts that are independent of the intervention [10]. Digital technology is evolving rapidly; hence, digital interventions that are not updated or refined following trials are at risk of becoming dated and, consequently, less appealing by the time they are available outside of research settings [13]. For these reasons, it is important to examine the use of digital interventions in real-world settings. Examining self-help interventions in isolation provides the opportunity to highlight differences between individual interventions of this type.

Objectives

In this study, we aimed to systematically review peer-reviewed articles reporting user uptake (eg, number of users, registrations, or downloads) and/or ongoing use, adherence, retention, or completion data (hereafter usage data or, for brevity, engagement) from implemented digital self-help interventions for the prevention or treatment of anxiety or depression, or for the enhancement of mood. We note that aspects of engagement other than usage data (eg, emotional involvement) are important [14-16]. However usage data are widely reported, are important for efficacy [6,7], and are the focus here. We identified no prior systematic reviews on this topic.

Methods

Search Strategy

Electronic searches were conducted of the Scopus, Embase, MEDLINE, and PsychINFO databases. The following search terms were used in Scopus, and the equivalent search was repeated on the Ovid Embase, MEDLINE, and PsychINFO databases:

( TITLE-ABS-KEY ( implementation OR “real world” OR real-world OR naturalistic OR observational OR “open access” OR public OR “publicly available” OR “publically available” OR deployment OR community OR nationwide OR national OR regist* OR dissemination ) AND TITLE-ABS-KEY ( computerized PRE/5 therap* ) OR etherap* OR e-therap* OR ( online PRE/5 intervention* ) OR ( online PRE/5 treatment* ) OR ( internet PRE/5 intervention* ) OR ( website PRE/5 intervention* ) OR ( web-based PRE/5 intervention* ) OR ( web-based PRE/5 treatment* ) OR “smartphone app*” OR “mobile app*” OR “smartphone intervention*” OR “smartphone program*” OR “mobile program*” OR “mobile intervention*” OR mhealth OR mtherapy ) AND TITLE-ABS-KEY ( depression OR anxiety OR mood OR “mental health” OR “psychological wellbeing” ) AND TITLE-ABS-KEY ( uptake OR adopt* OR regist* OR enrol* OR recruit* OR logon OR “logged on” OR usage OR adherence OR compliance OR complet* OR attrition OR “drop out” OR dropout OR drop-out ) ).

The search strategy was developed in partnership with a specialist research librarian. The search of the Ovid databases included “mp” (“multi-purpose”), thus incorporated all subject headings in which one or more word(s) matched the search term.

The following journals were hand-searched from their inception (all post 2002) up to and including their February 2017 issue: Internet Interventions, Journal of Medical Internet Research (JMIR), and JMIR Mental Health. Finally, a hand search was conducted of the included studies’ reference lists, and the titles of articles that had cited the included papers.

Inclusion and Exclusion Criteria

Articles were included in the review if they:

• were digital (computerized or online programs or apps) self-help/unguided interventions explicitly described as being for the prevention or treatment of depression or anxiety, or for the enhancement or improvement of mood;
• reported data on user uptake (eg, number of users, registrations, or downloads) and/or usage, adherence, or attrition (eg, number or percentage of users beginning, completing, or partially completing the intervention, or using the intervention for a specified period of time);
• reported implementation (dissemination/observational) data; and
• were published in the peer-reviewed literature between January 1, 2002, and March 8, 2017.

Articles were excluded if they:

• were pilot, exploratory, or feasibility studies; randomized controlled trials (RCTs); or protocol papers;
• were studies in which users were subject to assessments for research purposes, over and above what would normally be embedded in the intervention (ie, interventions could be included if routinely administered assessments were embedded as part of the self-help tool, but were excluded if users were subject to face-to-face or additional assessments for research purposes);
• reported findings from supported digital interventions (ie, supported by a therapist or where other human support was provided) or interventions that utilized a moderator or that were blended (eg, an adjunct to face-to-face therapy); and/or
• were not available in English.

Study Selection

Two authors (TF and LB) independently screened all retrieved titles, and then read the abstracts of all potentially relevant articles. Articles identified by one or both screening author(s) as potentially relevant were reviewed in full text. For each article excluded at the full-text review, the main reason for exclusion was recorded.

Data Extraction

The characteristics of all the included articles were coded by 2 of the 3 authors (LB, KS, and SH) and checked by 1 author (ML). The authors utilized a data extraction template that was developed for this systematic review and piloted on 2 of the full-text articles. Any discrepancies were resolved by referring to the original article and via discussion. The following characteristics and data were extracted:

• article reference details and data collection period;
• intervention characteristics: name of the intervention, intervention type (eg, online program, computerized program such as CD-ROM, smartphone app), condition treated, therapeutic modality, intervention length, features of gamification and navigation, and whether previously trialled and reported in the peer-reviewed literature;
• number of persons registering for or downloading the intervention;
• registration rate (the percentage of visitors to the intervention’s Website who then registered for the intervention), where data allowed;
• indicators of at least minimal use, such as number or percentage of users who began or used the intervention at least once or, where those data were not reported, number or percentage of users who completed at least one module or one assessment;
• indicators of moderate use (ie, more than at least minimal use but less than completion or sustained use), such as number or percentage of users completing a specified number or proportion of modules, or number of logins, or use for a specific period of time; and
• indicators of completion or sustained use, the number or percentage of users completing the intervention or, where no end-point was specified, the number or percentage using it for at least 6 weeks. Where neither of these were specified, the number or percentage of users who completed a final assessment or assessment at 6 weeks or more was recorded.

Results

Study Selection

The database search yielded 1701 records, of which 970 remained after the removal of duplicates as shown in Figure 1. The initial title screening excluded 771 records, and the abstract screening excluded a further 158 records. The full texts of the remaining 41 articles were reviewed, of which 10 passed the inclusion and exclusion criteria. Several papers required detailed consideration. A paper by Al-Asadi and colleagues [17] was excluded because it was not possible to separate data for those persons who selected therapist-assisted self-help from those who selected pure self-help. Another paper by Al-Asadi and colleagues [18] was included because results for those receiving pure self-help were provided. A paper by Menzies and colleagues [19] was included, despite being described as a trial, because the intervention was available online without referral, no researcher contact was involved, no assessments beyond those routinely included in the intervention were used, and there was no randomization. An intervention for posttraumatic stress disorder (PTSD) [20] was included as PTSD was classified as an anxiety disorder in the Diagnostic and Statistical Manual, Fourth Edition, Text Revision (DSM-IV-TR) [21], during most of our review period (ie, 2002-2017), although it was reclassified in the fifth edition of the DSM, issued in 2013 [22]. The hand-searching process identified one additional article [23] (Happify) for inclusion in the systematic review, bringing the total of eligible articles to 11.

Study and Program Characteristics

The 11 articles reported implementation data from 7 different interventions. Five articles reported on the original (ie, Mark I) or updated (ie, Mark II) version of the MoodGYM program [24-28], and the remaining 6 each described a unique intervention [18-20,23,29,30].

Study and program characteristics are summarized in Table 1. All of the included interventions were available without referral and were free to the user, apart from one that offered purchases or subscription for some content [23]. All were described as based on cognitive behavioral therapy (CBT) or utilizing CBT among other therapeutic modalities (eg, positive psychology), apart from one that did not specify the modality used [18]. Four interventions were online programs of fixed length, using sequential navigation (where content is provided in a specific order), or a choice of sequential or open navigation. The remaining 3 interventions were available via smartphone as an app [20], a suite of apps [30], or could be accessed as an app or online [23]. These app-based interventions (hereafter apps) had no fixed length and used open navigation. Notably, the articles reporting on apps were all relatively recent (ie, from 2015 to 2016).

‎View this figure

The efficacy of one program (MoodGYM) has been empirically supported through RCTs [25,31,32]. For 3 interventions there was some evidence: a small, exploratory pre-post trial found beneficial effects for CBTPsych [33]; a single-arm trial, which included coaching, showed significant therapeutic improvements for users of Intellicare apps [30]; and, for PTSD Coach, there was both a pilot RCT reporting modest, nonsignificant effects [34] and a subsequent RCT reporting significant therapeutic effects at post-treatment [35]. At the time of writing, we found no published studies examining the efficacy of 3 interventions: Anxiety Online, Happify, and HDep.

Uptake and Usage Data

User uptake (registrations or downloads) and usage data for each intervention are summarized in Table 2.

The number of registrations or downloads were reported in 8 papers. The remaining 3 studies reported related indicators as shown in Table 2. Registrations or downloads varied markedly from an average of 8 to at least 40,053 per month. Across the 7 interventions (and using the Mark II community user registration rate for the MoodGYM program), the median number of registrations or downloads was 401 per month, and the mean when excluding the lower and upper outliers was 2098 per month. There were 3 interventions with thousands of downloads or registrants per month, including 2 of the 3 apps, PTSD Coach, and Happify [20,23], and 1 of the 4 online programs, MoodGYM Mark II [27].

In Table 2, for MoodGYM, the number of assessments completed is likely to be similar to the number of modules completed, but not interchangeable. Assessments are completed at the beginning of a module and some users may not complete modules that they commence. Furthermore, for the Mark I version, assessments were not compulsory (a module could be completed without doing the assessment associated with that module); therefore, users may have skipped assessments.

Just 1 study, which was of MoodGYM Mark II [27], reported both the number of website visitors and the number of registrations, allowing the calculation of a registration rate, which was 42.2%.

Available measures indicating at least minimal use were limited and varied widely as shown in Table 2, making direct comparisons challenging. Within these limitations, CBTPsych [19] had the highest percentage of registrants engaging in at least minimal use, although given the low number of registrations per month, this was very few individuals.

Next, we reviewed indicators of moderate use. Again, there was little consistency in available data. Of the online programs: up to 16% of MoodGYM users completed 2 or more modules or assessments (embedded within the modules) out of 5 [24-28]; 10% of HDep users completed module 4 of this 7-module program (although they could miss individual modules) [29]; and 39% of CBTPsych users completed at least 4 of the 7 modules [19]. The apps or blended interventions reported quite different measures. Over 40% of PTSD Coach users continued to use it a month after installation [20]. For Happify, 20.6% of those who had completed an initial assessment also completed a noncompulsory assessment 2 weeks later, and 7.2% completed an assessment at 4 weeks [23]. The study of Intellicare apps described those who used each app 10 or more times as “active users.” This group comprised 4.7% to 35.7% of users for each app [30].

Only 2 studies directly reported intervention completion rates. In these cases, 2.8% of MoodGYM users completed all 5 modules [28] and 19.5% of CBTPsych users completed the program [19]. Looking at other indicators of sustained use for the online programs, 0.5% of MoodGYM Mark I users completed a noncompulsory assessment in the final module [25]. In total, 3.7% of Anxiety Online users who had completed an initial assessment and were offered a self-help online intervention also completed a post-treatment assessment [18]; however, these users may or may not have been utilizing the treatment. There were no specific completion data reported for the apps. However, 19.4% of PTSD Coach users continued to use the app after 6 months [20] and 3.5% of Happify users completed a 6-week assessment [23], although again, these users may have completed assessments without engaging in other content.

Combining completion or sustained use data with the number of people beginning each intervention (uptake), as reported here, suggests that fewer than 40 persons per month completed final assessments or final modules for any of the online modular programs during the study periods. In contrast, over 800 users completed the Happify app assessments at 8 weeks [23], and over 1000 persons per month demonstrated continued use of PTSD Coach after 3 months [20].

Discussion

Summary of Evidence

For digital mental health interventions to have a population-level impact, significant numbers of people must receive beneficial doses. This requires both sufficient uptake and ongoing use of effective interventions. Despite over 10,000 digital mental health interventions being publicly available in 2017 [36], we identified only 11 peer-reviewed publications reporting uptake and/or usage data from publicly available digital self-help interventions for depression, anxiety, or low mood. This is disappointing, given 3 considerations. First, people may use interventions differently in the real world, as compared with trial conditions. Second, digital interventions allow relative ease of data collection through automation. Third, comparisons of uptake and usage across interventions could inform improvements in the field. Where data have been reported, diverse measures were used, making direct comparisons challenging. Nevertheless, large differences are apparent. The widest-reaching intervention in our review had tens of thousands of new users per month, whereas the least used one had fewer than ten. Moreover, ongoing use ranged from less than 1% to over 28% of users completing interventions or demonstrating sustained use.

The findings suggest that people may use digital mental health interventions differently in real-world settings, as compared with trial conditions. Although this may be true for many interventions, the phenomenon is easily quantifiable in digital interventions through embedded routine data collection. Completion rates, as reported in included studies, are lower than the completion rates of 43% to 99% in a systematic review of adherence in controlled trials of online interventions for depression and anxiety [7]. Direct comparisons between research trials and implementation usage data for the same intervention also suggest reduced adherence in real-world settings. For example, only 0.5% of community users (ie, users of MoodGYM as publicly implemented and freely available online) of MoodGYM completed a noncompulsory final assessment, compared with 22.5% of participants in a trial evaluating the same program [31]. Similarly, in the community, adolescent users of MoodGYM completed an average of 3.1 exercises, compared with an average of 9.4 exercises among adolescents in a school-based trial [28].

Previous studies have highlighted differences in adherence between guided and unguided self-help interventions [8]. Given the limited, heterogeneous data, we did not conduct meta-analyses or test for differences between interventions. However, on the face of it, a long established program (MoodGYM released in 2001) [24] and 2 of the recently tested app-based interventions (PTSD Coach [20] and Happify [23]) appear particularly promising in terms of uptake. Two interventions reported high sustained use. The first was CBTPsych [19], the 7-module online CBT program for addressing social anxiety among stutterers. Very few people used this, but retention was high. Second, a very different intervention, PTSD coach (available as an app, with no fixed length and an open navigation structure), had high sustained use [20]. Over 10 times as many people engaged in sustained use of apps as completed any of the online modular programs.

An issue for consideration is that of what is a beneficial dose at a population level. The dose or amount of exposure to digital mental health interventions for clinically significant effects has been considered in previous research, with greater adherence generally associated with greater clinical gains [6,7,9]. Included studies show that large numbers of people accessed some mental health interventions for brief periods. Relatively brief use might have a significant population impact if this exposes large numbers of people to ideas such as depression being common, there being a range of ways to address it, and help being available. Future research should consider this.

Perhaps, the strongest implication from this study is that future research should report intervention uptake, ongoing use, and impact in real-world settings. Transparent reporting of key data, such as those as shown in Textbox 1, would facilitate comparisons. Alongside these data, reporting of intervention characteristics, modes of delivery, and features of implementation such as marketing and methods of dissemination would provide opportunities for understanding which interventions, applied in which ways, engage and retain users.

Strengths and Limitations

This review examined data from peer-reviewed articles. We did not examine grey literature or request data from providers of interventions or Internet service providers due to resource constraints. Further research should explore this. There is a risk of publication bias, given that interventions with poor results may not be reported. A meta-analysis was not conducted due to the small number of published studies and the heterogeneity of data. However, with increased data and more consistent reporting, a meta-analysis would be a valuable future addition to the literature. We set inclusion and exclusion criteria to focus on interventions addressing the very common issues of depression, anxiety, and low mood. We included an intervention targeting PTSD. PTSD was included as an anxiety disorder in the edition of the Diagnostic and Statistical Manual/DSM-IV-TR [21], which was in use at the start of our review period (2002-2017), but not in the fifth edition, issued in 2013 [22]. Others might have made different decisions; however, we have endeavored to be transparent in this. These limitations notwithstanding, this is the first systematic review of implementation data in this area, and it highlights valuable opportunities for development.

Conclusion

Digital self-help interventions targeting depression, anxiety, or the enhancement of mood have the potential to improve population-level mental health in a highly scalable manner. However, for these interventions to achieve meaningful impact, they need to have adequate uptake and adherence in real-world settings. Only a handful of interventions have reported this information in the peer-reviewed literature to date, and these utilized diverse measures. Nevertheless, the published studies of unguided self-help interventions for anxiety, depression, and mood demonstrate large differences in uptake and engagement between interventions. Organizations delivering these interventions should take advantage of the opportunity to gather and publish data. Much of the data collection on intervention usage can be automated, making such collection and subsequent reporting generally easy and low-cost. We have proposed key metrics that should be considered. Transparent, comparable, and timely publication of real-world data would allow between-program comparisons and hence facilitate improvements in user uptake and engagement, arguably 2 of the major challenges in the digital health world.