This paper is in the following e-collection/theme issue:

Digital Health Reviews (1289) Digital Mental Health Interventions, e-Mental Health and Cyberpsychology (1514) Anxiety and Stress Disorders (1053) Depression and Mood Disorders; Suicide Prevention (1363) Mobile Health in Psychiatry (426) Web-based and Mobile Health Interventions (3297) Usability of Apps and User Perceptions of mHealth (974)

Published on in Vol 26 (2024)

This is a member publication of University of Toronto

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/52609, first published .
Acceptability of Digital Mental Health Interventions for Depression and Anxiety: Systematic Review

Acceptability of Digital Mental Health Interventions for Depression and Anxiety: Systematic Review

Acceptability of Digital Mental Health Interventions for Depression and Anxiety: Systematic Review

Authors of this article:

Carrie K Y Lau1, 2 Author Orcid Image ;   Anthony Saad1 Author Orcid Image ;   Bettina Camara3 Author Orcid Image ;   Dia Rahman4 Author Orcid Image ;   Blanca Bolea-Alamanac1, 5 Author Orcid Image
Article Authors Cited by Tweetations Metrics

    Review

    1Department of Psychiatry, Women's College Hospital, Toronto, ON, Canada

    2Factor-Inwentash Faculty of Social Work, University of Toronto, Toronto, ON, Canada

    3Department of Immunology, Faculty of Medicine, University of Toronto, Toronto, ON, Canada

    4Department of Psychiatry, University of Toronto, Toronto, ON, Canada

    Corresponding Author:

    Blanca Bolea-Alamanac, MSc, MD, PhD

    Department of Psychiatry

    Women's College Hospital

    76 Grenville St

    Toronto, ON, M5S 1B2

    Canada

    Phone: 1 647 331 2707

    Email: Blanca.BoleaAlamanac@wchospital.ca


    Background: Depression and anxiety disorders are common, and treatment often includes psychological interventions. Digital health interventions, delivered through technologies such as web-based programs and mobile apps, are increasingly used in mental health treatment. Acceptability, the extent to which an intervention is viewed positively, has been identified as contributing to patient adherence and engagement with digital health interventions. Acceptability, therefore, impacts the benefit derived from using digital health interventions in treatment. Understanding the acceptability of digital mental health interventions among patients with depression or anxiety disorders is essential to maximize the effectiveness of their treatment.

    Objective: This review investigated the acceptability of technology-based interventions among patients with depression or anxiety disorders.

    Methods: A systematic review was performed based on PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) and PROSPERO (International Prospective Register of Systematic Reviews) guidelines. We searched PubMed, Web of Science, and Ovid in May 2022. Studies were included if they evaluated digital interventions for the treatment of depression or anxiety disorders and investigated their acceptability among adult patients. Studies were excluded if they targeted only specific populations (eg, those with specific physical health conditions), investigated acceptability in healthy individuals or patients under the age of 18 years, involved no direct interaction between patients and technologies, used technology only as a platform for traditional care (eg, videoconferencing), had patients using technologies only in clinical or laboratory settings, or involved virtual reality technologies. Acceptability outcome data were narratively synthesized by the direction of acceptability using vote counting. Included studies were evaluated using levels of evidence from the Oxford Centre for Evidence-Based Medicine. The risk of bias was assessed using a tool designed for this review and GRADE (Grading of Recommendations, Assessment, Development, and Evaluation).

    Results: A total of 143 articles met the inclusion criteria, comprising 67 (47%) articles on interventions for depression, 65 (45%) articles on interventions for anxiety disorders, and 11 (8%) articles on interventions for both. Overall, 90 (63%) were randomized controlled trials, 50 (35%) were other quantitative studies, and 3 (2%) were qualitative studies. Interventions used web-based programs, mobile apps, and computer programs. Cognitive behavioral therapy was the basis of 71% (102/143) of the interventions. Digital mental health interventions were generally acceptable among patients with depression or anxiety disorders, with 88% (126/143) indicating positive acceptability, 8% (11/143) mixed results, and 4% (6/143) insufficient information to categorize the direction of acceptability. The available research evidence was of moderate quality.

    Conclusions: Digital mental health interventions seem to be acceptable to patients with depression or anxiety disorders. Consistent use of validated measures for acceptability would enhance the quality of evidence. Careful design of acceptability as an evaluation outcome can further improve the quality of evidence and reduce the risk of bias.

    Trial Registration: Open Science Framework Y7MJ4; https://doi.org/10.17605/OSF.IO/SPR8M

    J Med Internet Res 2024;26:e52609

    doi:10.2196/52609

    Keywords

    acceptability; mental health; depression; anxiety; mobile app; internet; mobile phone; mobile health; mHealth; digital health 


    Background

    Depression and anxiety disorders are common [1] and have high and increasing societal costs [2,3]. Patients with depression or anxiety disorders can benefit from psychological treatments but face a variety of barriers to accessing traditional mental health services.

    Digital health interventions include those delivered with technologies such as web and mobile apps. They have garnered optimism for their potential to lower barriers to accessing mental health services [4,5]. Digital mental health interventions can facilitate access to services by increasing convenience for patients and reducing demands on their time, eliminating geographic limitations, reducing costs, and increasing the capacity of health systems [6].

    The use of digital mental health interventions increased during the COVID-19 pandemic [7,8]. Their expanded reach occurred amid public health measures that limited in-person mental health treatment and predictions of a long tail of mental health repercussions [9]. Digital mental health interventions are now commonly available to patients alongside other mental health services, as an adjunct to in-person or synchronous video-based mental health treatment, or as a stopgap for long waitlists [10,11].

    However, uncertainty exists around patient adherence to digital mental health interventions. Significant variations in adherence levels, ranging from low to high, have been noted among digital interventions [12]. Issues with nonadherence cast some doubt on whether the potential benefits of digital mental health interventions can be realized. Related issues have been found with user engagement (initiation and continuation of use of a digital health intervention) [13] among digital mental health interventions.

    Acceptability, the degree to which individuals view a treatment or potential treatment positively, has been identified as predicting adherence and user engagement [14], and ultimately the intervention’s effectiveness [15]. Acceptability among patients is therefore critical to realizing the potential of a digital mental health intervention.

    Research into acceptability appears hampered by its varying conceptualizations. Sekhon and colleagues [16] propose that acceptability is the patient’s perception of the treatment as appropriate, arising from their cognitive and affective reactions. They also provide a theoretical framework for acceptability with 7 components: affective attitude, burden, perceived effectiveness, ethicality, intervention coherence, opportunity costs, and self-efficacy [16]. Furthermore, Sekhon and colleagues [16] also recommend measuring acceptability at different time points during the intervention.

    By contrast, Proctor and colleagues [17] define acceptability as the patient’s perception of the intervention as satisfactory or tolerable. Zauszniewski [18] describes acceptability as patients’ belief that the intervention is logical and suitable to them. They also describe 3 main components of acceptability, such as the acceptability of an intervention’s delivery method, acceptability of the content of an intervention, and acceptability of the provider of the intervention, although they acknowledge that acceptability is often assessed only as a whole [18]. In contrast, the technology acceptance model proposed by Davis [19], regarding attitudes toward technology and its use more generally, presents acceptability as hinging upon users’ perceptions of ease in using the technology and the technology’s usefulness.

    Conflict between the constructs of acceptability and satisfaction has also been identified, with recommendations that acceptability be understood as distinct from satisfaction [16,17]. However, studies continue to operationalize acceptability as a patient’s satisfaction with a treatment, indicating continued resonance with the construction of satisfaction as a representation of acceptability, or at least a lack of consensus.

    These examples demonstrate the variation with which acceptability has been understood and investigated in evaluations of digital mental health interventions thus far. Despite past recommendations for improvement (eg, Sekhon and colleagues [16]), evaluations have not yet converged on a particular approach to patient acceptability.

    Acceptability has implications for user engagement and adherence [14]. Acceptability can impact the benefit that a patient can derive from an intervention and, in turn, compromise or enhance the individual’s investment into digital mental health interventions. It is of particular importance to investigate the acceptability of digital interventions for patients with depression and anxiety disorders, given the prevalence of these diagnoses and the frequency with which clinicians are required to identify appropriate psychosocial interventions [1].

    Aims and Objectives

    This review summarizes the acceptability of digital mental health interventions for depression and anxiety disorders among adult patients. The research question we investigated was the following: “How acceptable to adult patients with depression or anxiety are digital mental health interventions (such as mobile apps) used in psychiatric care, and what is the evidence for their acceptability?” We accomplished this by identifying evaluation studies of digital mental health interventions where acceptability is one of the outcome measures, reviewing the quality of the available evidence for acceptability, and summarizing the acceptability of digital mental health interventions to patients with depression and anxiety disorders.


    Overview

    We started with Sekhon and colleagues’ [16] theoretical framework for acceptability, which identified in acceptability the component constructs of affective attitude, burden, perceived effectiveness, ethicality, intervention coherence, opportunity costs, and self-efficacy. We selected this theoretical framework because these component constructs can readily be identified in research studies having acceptability as an outcome measure. Where a study had an outcome measure that assessed one or more of these component constructs, whether or not the same terminology was used, the measure was considered relevant to our study.

    In addition, the abovementioned theoretical framework for acceptability recognizes different temporal assessments of acceptability, such as prospective acceptability (before the intervention), concurrent acceptability (during the intervention), and retrospective acceptability (following the intervention) [16]. Each temporal assessment of acceptability is understood to achieve a different purpose and to provide different information [16]. For the purposes of this review, we included studies that assessed acceptability at any of these multiple time points but took note of the time points selected by the researchers for further analysis and discussion.

    Sekhon and colleagues [16] suggest that satisfaction and acceptability are constructs that are often confounded but should be differentiated, referencing the fact that satisfaction can only be retrospectively assessed, while acceptability can be assessed at all time points. For this study, satisfaction is considered an aspect of retrospective acceptability. We therefore took into account measures of satisfaction as relevant to our review, despite satisfaction not being a part of Sekhon and colleagues’ [16] theoretical framework.

    Other constructs not named within Sekhon and colleagues’ [16] theoretical framework of acceptability and included in this review are credibility, expectancy, and usability. For this review, studies evaluating credibility, expectancy, and usability were eligible for potential inclusion as long as the measures assess one or more component constructs of acceptability. Studies identified by their authors as collecting feedback, experiences, attitudes, subjective appraisal, and subjective benefit were similarly eligible for inclusion if they provided information on one or more component constructs of acceptability.

    Inclusion and Exclusion Criteria

    Studies were included if (1) they investigated digital mental health interventions that monitor or treat depression or anxiety disorders; (2) they evaluated internet-based, mobile, or computer-based mental health interventions, and could involve the use of such technologies as mobile apps or web-based treatment modules; (3) their participants were 18 years of age or older; and (4) they collected data on the acceptability of the intervention to their participants.

    Excluded were studies that (1) focused on behavioral or lifestyle changes in healthy people; (2) focused on a physical health condition, a neurodevelopmental condition, or a mental health diagnosis other than depression or an anxiety disorder; (3) focused on a specific population such as military personnel, health care workers, students, or a target group representing only one component of a larger population; (4) focused on family members or caregivers; (5) evaluated technologies that act as a conduit to traditional care but add no potential therapeutic value, such as videoconferencing platforms used for remote psychotherapy; (6) evaluated interventions that used virtual reality technology; (7) did not have participants interacting directly with the technology for therapeutic purposes (eg, sensors used by clinicians to collect data without offering therapeutic feedback to participants); (8) had participants <18 years of age; (9) had participants using the technology exclusively in clinical or laboratory settings; or (10) were not completed at the time of searching (eg, study protocols captured in database searches).

    Database Review

    The review was conducted in 3 locations, PubMed, Web of Science (all databases in their Core Collection), and Ovid (all databases in their Health Sciences, Life Sciences, and Social Sciences categories). Search terms comprising both free text or natural language and Medical Subject Headings (MeSH) were applied in these searches (Multimedia Appendix 1). The searches were performed in May 2022 with no filters used to restrict the time period of the articles, and as a result, the oldest article in our search dated back to 1987. Reverse searches were also conducted by reviewing the references of relevant systematic reviews and meta-analyses, and by reviewing the references of included articles. Only articles with English abstracts were considered, but main texts could be in English, French, German, Italian, Spanish, or Chinese. Searches for gray literature using the same terms identified relevant published dissertations. PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) and PROSPERO (International Prospective Register of Systematic Reviews) [20] guided this review. The review was conducted using conventions for systematic reviews, screening first by title and abstract, followed by reviewing the complete article text. Furthermore, 2 reviewers (CKYL and one of AS, BC, or DR) independently assessed each title and abstract, and each complete text, to determine inclusion or exclusion. Disagreements in these assessments were resolved with further discussion (involving the reviewers and BB-A). The database software Covidence (Veritas Health Innovation Ltd) was used to organize and review the studies [21].

    Data Extraction and Synthesis

    Data were captured from the included articles by the first author (CKYL) in a table format, checked by the last author (BB-A), and approved by all the authors. These data comprised the following: (1) depression or anxiety disorders needed for study eligibility, (2) details of the intervention (ie, mode of technology used, program name if one exists, and treatment paradigm), (3) details of the participant sample (ie, sample size, breakdown by group allocation, and gender), (4) details of acceptability-related outcomes (ie, outcome name, instrument used, and time point of data collection), and (5) results of acceptability-related outcomes. From these data, we noted considerable variation in instruments used to measure acceptability and infrequent use of validated instruments. Acceptability outcome data were therefore narratively synthesized by the direction of acceptability and not by its effect size. The narrative synthesis applied vote counting to the tabulated results, was conducted by the first author (CKYL), and was checked by the last author (BB-A).

    Levels of Evidence

    Included studies were evaluated using the Oxford Centre for Evidence-Based Medicine–levels of evidence [22]. In this system, each study receives a designation based on a grading classification where 1 is the highest level (randomized controlled trials [RCTs] or systematic reviews of RCTs) and 5 is the lowest (mechanism-based reasoning).

    Reported evidence for patient acceptability was further evaluated using an assessment tool designed for this review (Multimedia Appendix 2). The purpose of the assessment tool was to evaluate the quality of acceptability measurement and reporting. An assessment tool was developed because, to our knowledge, one did not already exist. Given the frequency (and variation) with which acceptability is reported as an evaluation outcome, the assessment tool allows summarization of our concerns with the current evidence. The tool emphasizes acquiring and providing sufficient information to evaluate, operationalize, and measure acceptability. Sekhon and colleagues’ [16] recommendation that acceptability be measured at multiple time points informed item 4 of the tool. The assessment tool produces a score ranging from 0 to 15, where scores from 0 to 5 indicate low quality of evidence for patient acceptability, scores 6-10 indicate moderate quality, and scores 11-15 indicate high quality. Assessments using this tool were performed by one author (CKYL) and reviewed by another (BB-A), with discrepancies identified and consensus achieved through discussion.

    Risk of Bias

    Included studies were individually assessed on aspects of risk of bias using the assessment tool designed for this review (Multimedia Appendix 2). Specifically, the completeness of outcome data and the appropriateness of measurements (ie, selection of instruments) were determined using this assessment tool. These criteria were selected to assess each included study for their direct relevance to the single implementation outcome (ie, acceptability) that is the focus of this review. Preliminary assessments on these criteria indicated a significant risk of bias across the included studies, in terms of incomplete outcome data and low use of validated instruments.

    Given these preliminary results suggesting a significant risk of bias, and this review’s focus on a single implementation outcome, risk of bias was assessed using GRADE (Grading of Recommendations, Assessment, Development and Evaluation) [23]. GRADE assesses the quality of evidence for risk of bias across a group of studies in relation to a particular outcome [23]. Across a body of evidence, the risk of bias is assessed as low, unclear, or high [23]. This evaluation was done independently by 2 authors (CKYL and BB-A), with discrepancy resolved with further discussion.


    Overview

    From the database review process, a total of 2409 articles were identified for screening. Of these articles, 2266 were excluded as they did not meet the inclusion criteria. Included in the review were 143 articles, investigating digital mental health interventions for depression (n=67, 47%), anxiety disorders (n=65, 45%), and a combination of depression and anxiety disorders (n=11, 8%; Figure 1 provides a flow diagram of the review).

    ‎
    Figure 1. PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow diagram for each step of the screening process.

    In these articles, researchers most often reported on satisfaction as their acceptability-related construct of choice, with satisfaction investigated in 105 (73%) of the 143 included articles. Out of 143 articles, acceptability or acceptance was reported on in 78 (55%) articles, credibility in 26 (18%) articles, expectancy in 20 (14%) articles, and usability in 19 (13%) articles. There were also 21 (15%) articles using such terminology as feedback, experiences, attitudes, subjective appraisal, and subjective benefit.

    For the evidence provided for acceptability and its component constructs, the vast majority, at 137 (96%) out of 143 articles, evaluated acceptability “globally” for each digital mental health intervention (ie, across features or components), and 20 (14%) articles assessed acceptability for specific features or components of each digital mental health intervention.

    Referring to the component constructs of acceptability identified in Sekhon and colleagues’ [16] theoretical framework, each article investigated a subset of those component constructs. Specifically, out of 143 articles, 121 (85%) investigated affective attitude, 73 (51%) investigated perceived effectiveness, 37 (26%) investigated burden, and 35 (24%) investigated intervention coherence. The remaining component constructs were each investigated in 10% (15/143) or less of the articles (ie, opportunity cost: n=15, 10%; and self-efficacy: n=7, 5%). We did not find any articles that investigated ethicality.

    Other constructs that were not explicit components of Sekhon and colleagues’ [16] theoretical framework for acceptability but were investigated in the reviewed articles included perceived quality (17/143, 12%) and acceptability of program tempo or length (10/143, 7%).

    In terms of temporal assessments of acceptability or its component constructs, each article captured one or more of prospective acceptability (before the intervention began), concurrent acceptability (during the intervention), and retrospective acceptability (following the intervention), also set out in Sekhon and colleagues’ [16] theoretical framework. Prospective acceptability appeared in 42 (29%) of the 143 articles and concurrent acceptability in 9 (6%) articles. Retrospective acceptability was separated for this review into posttest (125/143, 87%) and follow-up (15/143, 10%). Of the total 143, only 43 (30%) articles contained acceptability assessments for more than 1 time point.

    Assessment of Quality

    Using our assessment tool for quality of acceptability (Multimedia Appendix 2), articles investigating digital mental health interventions for depression had scores ranging from 2 (low quality of evidence) to 14 (high quality of evidence), out of a total possible 15. The mean score for the quality of evidence for acceptability was 7.3 (SD 2.0). From the information captured with this assessment tool, out of 143 articles, 74 (53%) used one or more validated instruments for acceptability or its component constructs, while 66 (47%) did not use any validated instruments for acceptability or its component constructs.

    Risk of Bias

    The overall risk of bias was assessed as between high and unclear, considering the predominance of studies that lacked blinding (for RCTs), failed to adequately control confounding, had incomplete accounting of patients and outcome events (particularly losses to follow-up), and other limitations (eg, used measures that are not validated).

    Interventions for Depression

    From the 67 articles addressing digital mental health interventions for depression, 40 (60%) articles involved web-based programs, 21 (31%) involved mobile apps, and 6 (9%) involved computer-based programs. In terms of treatment modality, 36 (54%) articles explored digital mental health interventions that drew from cognitive behavioral therapy (CBT). These apps or programs were often structured as multimedia modules, with varying levels of clinician or technician support and patient feedback.

    The 67 articles investigating digital mental health interventions for depression were comprised of 41 (61%) RCTs, 23 (34%) non-RCT quantitative studies, and 3 (5%) qualitative studies. These studies are reviewed in Table S1 [24-65], Table S2 [66-89], and Table S3 [43,90-92], respectively, in Multimedia Appendix 3. The 64 quantitative studies (RCT and non-RCT) indicated overall positive ratings of acceptability among digital mental health interventions for depression. Among these 64 quantitative studies, we categorized 55 (86%) studies as indicating positive acceptability overall, 6 (9%) studies mixed acceptability, and 3 (5%) studies as having insufficient information to categorize the direction of acceptability. We considered results to be “mixed” if a study had more than 1 acceptability outcome measure and at least 1 positive and 1 negative result. These categorizations are presented in the “Summarized acceptability result” column within each table in Multimedia Appendix 3 [24-92].

    The 3 qualitative studies (completed secondarily or subsequently to RCTs) had more mixed results. In 2014, Schneider and colleagues [90] reported that open-ended participant responses on satisfaction levels were more negative than positive, though satisfaction or dissatisfaction were reported as arising from matters related and unrelated to the interventions, such as individual time management choices made during the study. Comments specific to the web-based program were more positive than negative. Knowles and colleagues [91] reported in 2015 that they conducted interviews with participants using digital mental health interventions for depression (MoodGYM [e-hub Health] and Beating the Blues [Manage My Health], both web-based programs in CBT). Responses were coded as positive, negative, or ambivalent toward the digital mental health intervention, with the largest group being ambivalent (17/36 participants, 47%), then negative (10/36 participants, 28%), and then positive (9/36 participants, 25%). Also in 2015, Ly and colleagues [92] held semistructured interviews with participants using a mobile app informed by CBT and mindfulness techniques. Participants’ responses were divided, with 5 (42%) of 12 participants indicating an overall positive experience, 4 (33%) indicating a neutral experience, and 3 (25%) indicating a negative experience.

    Interventions for Anxiety Disorders

    From the 65 articles addressing digital mental health interventions for anxiety, 52 (80%) articles involved web-based programs, 8 (12%) involved mobile apps, 3 (5%) involved computer-based programs, and 2 (3%) involved others. In terms of treatment modality, 58 (89%) articles explored digital mental health interventions that drew from CBT.

    The 65 articles investigating digital mental health interventions for anxiety disorders were comprised of 43 (66%) RCTs and 22 (34%) non-RCT quantitative studies. None were qualitative studies. The studies are reviewed in Table S4 [93-135] and Table S5 [136-157] in Multimedia Appendix 3. In general, the results indicated positive ratings of acceptability. We categorized 59 (91%) studies as indicating positive acceptability overall, 3 (5%) studies mixed acceptability, and 3 (5%) studies as having insufficient information to categorize the direction of acceptability. We considered results to be “mixed” if a study had more than 1 acceptability outcome measure and at least 1 positive and 1 negative result. These categorizations are presented in the “Summarized acceptability result” column within each table in Multimedia Appendix 3 [93-157].

    Interventions for Depression and Anxiety Disorders Together

    Among the 11 articles addressing digital mental health interventions for both depression and anxiety disorders, 8 (73%) articles involved web-based programs, 2 (18%) involved mobile apps, and 1 (9%) involved computer-based programs. In terms of treatment modality, 8 (73%) articles explored digital mental health interventions that drew from CBT.

    The 11 articles investigating digital mental health interventions for both depression and anxiety disorders were comprised of 6 (55%) RCTs and 5 (45%) non-RCT quantitative studies (reviewed in Table S6 [158-163] and Table S7 [164-168] in Multimedia Appendix 3). None were qualitative studies. In general, the results indicated positive ratings of acceptability. We categorized all 11 studies as indicating positive acceptability overall, with none that indicated mixed results or insufficient information to categorize the direction of acceptability. These categorizations are presented in the “Summarized acceptability result” column within each table in Multimedia Appendix 3 [158-168].


    Principal Findings

    Digital mental health interventions for depression and anxiety disorders were generally found to be acceptable to patients. Of the 143 included articles, 125 (87%) indicated positive acceptability, 12 (9%) had mixed results, and 6 had insufficient information (4%). These results arose out of evidence generated primarily from RCTs (90/143, 63%) and other quantitative studies (50/143, 35%), and largely investigations of web-based apps (100/143, 70%) and CBT-based content (102/143, 71%). These findings came primarily from views on a digital mental health intervention as a whole (137/143, 96%) rather than specific features, and centered on patient satisfaction (105/143, 73%) or Sekhon and colleagues’ [16] “affective attitude” (121/143, 85%) at posttest (125/143, 87%). We noted no substantial differences in the acceptability noted for digital mental health interventions for depression, anxiety, or depression and anxiety together.

    Our findings suggest some receptiveness among those with depression or anxiety disorders to digital mental health interventions. If the overall positive experience of digital mental health interventions translates into patient adherence to and engagement with treatment, digital mental health interventions can improve patient access to care and expand the capacity of health systems. Since some patients appear open to digital mental health interventions, clinicians can explore patients’ views on these technologies in the course of treatment planning and respond accordingly. Researchers can further our understanding by investigating patient and intervention characteristics that predict or improve acceptance and deepen our understanding of acceptability beyond satisfaction or affective attitudes. Developers can recognize the potential in continuing to invest in digital mental health interventions and take steps to improve user experiences and strengthen acceptability.

    The findings of positive acceptability toward digital mental health interventions must nevertheless be qualified by gaps in the existing research.

    First, the assessment tool revealed that studies had a wide range in quality of acceptability-related evidence and were generally only of moderate quality. This finding suggests a need to increase standardization of acceptability assessments. We propose that our assessment tool be used as a checklist when planning evaluation research. The results outlined above indicate that there is an opportunity to use validated measures for acceptability, measure acceptability at different time points, and consider measures that address multiple acceptability components.

    The second gap is the continued absence of a consensus definition for acceptability. As an example, Sekhon and colleagues [16] make clear that satisfaction ought to be treated as a construct separate from acceptability, but many studies continue to use satisfaction as an indicator of acceptability, or otherwise consider satisfaction alone when perhaps the construct of interest is acceptability. Similarly, Sekhon and colleagues [16] identified 7 component constructs of acceptability, but studies are generally designed to address only some of these components and vary by the components chosen. These findings suggest that there may not only be variation in how acceptability is investigated, but also variation in how acceptability is understood. It is necessary to establish a consensus on how acceptability should be defined, to increase the consistency and relevance of future acceptability research.

    Third, some aspects of acceptability such as affective attitude and perceived effectiveness were more frequently investigated, while other aspects were far less likely to be explored, particularly ethicality, self-efficacy, and opportunity cost. Likely, this relates to some component constructs being considered more important than others, or the feasibility of investigating some component constructs over others. Investigating all components of acceptability is important to better understand how users relate to digital mental health interventions.

    Finally, we noted a paucity of research on how acceptability changes over time, with most studies examining only retrospective acceptability. Evaluating acceptability as a variable that is continuous and changeable over time would enhance development processes for digital mental health interventions. It would allow us to identify the modifications that can be made to a technology to strengthen acceptability at different stages, to increase willingness to initiate use of a digital mental health intervention, to continue using it, and to use it as frequently or often as is needed to reap therapeutic benefits.

    Limitations

    An important limitation of research on acceptability, as identified in the risk of bias assessment, lies with evidence generally being provided by patients who have agreed to use the digital mental health intervention and continued to engage with the study or the digital mental health intervention. Often missing from these findings is the acceptability of digital mental health interventions among individuals who had the opportunity to participate in a study involving a digital mental health intervention but chose not to. Nonparticipation and early termination in digital health have long prompted calls for more research attention on attrition, for example, Eysenbach [169] in 2005, along with related constructs such as adherence and engagement. There remains a need to establish consensus approaches to analyzing attrition, adherence, and engagement [170], and their relationship with acceptability. It is recommended that appropriate methods for such analyses be developed, to be used consistently in future evaluations of digital mental health interventions.

    A second limitation comes from potential conflict in the research evidence between the acceptability of the intervention technology and the acceptability of the intervention as a whole. Future research should differentiate acceptability of the technology from the acceptability of other aspects, such as the timing or pacing of the intervention, or the treatment paradigm applied.

    A further limitation is that the terminology used to identify acceptability is varied and changeable between studies. Data on acceptability not labelled as such may be difficult to integrate into acceptability research at large.

    Finally, literature on this topic has increased exponentially following the COVID-19 pandemic [171]. Any review can only represent a snapshot of the available research at a point in time. At the same time, despite this proliferation of research, diverse populations (in terms of ethnicity, income and education levels, sexual and gender minority status, etc) remain underrepresented [172]. More evaluation studies are needed that ensure their inclusion and provide demographic data as appropriate, to further our understanding of their acceptability of digital mental health interventions.

    Conclusions

    There is evidence for the acceptability of digital mental health interventions among patients with depression or anxiety disorders, with most studies in this review indicating positive acceptability. However, the available research evidence for acceptability is, as a whole, of only moderate quality. It is suggested that future intervention studies be planned from the outset to produce higher quality evidence for acceptability. Enhanced research evidence for acceptability will provide better insights into the appropriateness of digital mental health interventions for patients with depression or anxiety disorders.

    Acknowledgments

    Support was provided by Emily Stowe Society funding at Women’s College Hospital Foundation (a nonprofit foundation).

    Authors' Contributions

    Each author contributed substantially to this work. BB-A conceived of the review and all other authors provided feedback on the design. CKYL, AS, BC, and DR were involved in data collection. All authors participated in data analysis and interpretation. BB-A and CKYL wrote the initial draft of the paper. All authors contributed to revisions of the manuscript. All authors read and approved the final submitted manuscript.

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    Search strategies.

    DOC File , 65 KB
    Multimedia Appendix 2

    Acceptability assessment tool.

    DOC File , 61 KB
    Multimedia Appendix 3

    Description of acceptability results in the included studies.

    DOCX File , 86 KB
    Multimedia Appendix 4

    PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) 2020 checklist.

    PDF File (Adobe PDF File), 282 KB
    Multimedia Appendix 5

    PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) 2020 checklist for abstracts.

    PDF File (Adobe PDF File), 267 KB
    1. Report from the Canadian Chronic Disease Surveillance System: Mood and Anxiety Disorders in Canada, 2016. Public Health Agency of Canada. 2016. URL: https:/​/www.​canada.ca/​content/​dam/​canada/​health-canada/​migration/​healthy-canadians/​publications/​diseases-conditions-maladies-affections/​mood-anxiety-disorders-2016-troubles-anxieux-humeur/​alt/​mood-anxiety-disorders-2016-troubles-anxieux-humeur-eng.​pdf [accessed 2023-07-08]
    2. Greenberg PE, Fournier AA, Sisitsky T, Simes M, Berman R, Koenigsberg SH, et al. The economic burden of adults with major depressive disorder in the United States (2010 and 2018). Pharmacoeconomics. Jun 2021;39(6):653-665. [FREE Full text] [CrossRef] [Medline]
    3. Konnopka A, Leichsenring F, Leibing E, König H-H. Cost-of-illness studies and cost-effectiveness analyses in anxiety disorders: a systematic review. J Affect Disord. Apr 2009;114(1-3):14-31. [CrossRef] [Medline]
    4. Lal S. E-mental health: Promising advancements in policy, research, and practice. Healthc Manage Forum. Mar 2019;32(2):56-62. [FREE Full text] [CrossRef] [Medline]
    5. Moroz N, Moroz I, D'Angelo MS. Mental health services in Canada: barriers and cost-effective solutions to increase access. Healthc Manage Forum. Nov 2020;33(6):282-287. [CrossRef] [Medline]
    6. Lal S, Adair CE. E-mental health: a rapid review of the literature. Psychiatr Serv. Jan 01, 2014;65(1):24-32. [CrossRef] [Medline]
    7. Gratzer D, Torous J, Lam RW, Patten SB, Kutcher S, Chan S, et al. Our digital moment: innovations and opportunities in digital mental health care. Can J Psychiatry. Jan 2021;66(1):5-8. [FREE Full text] [CrossRef] [Medline]
    8. Sorkin DH, Janio EA, Eikey EV, Schneider M, Davis K, Schueller SM, et al. Rise in use of digital mental health tools and technologies in the United States during the COVID-19 pandemic: survey study. J Med Internet Res. Apr 16, 2021;23(4):e26994. [FREE Full text] [CrossRef] [Medline]
    9. Vadivel R, Shoib S, El Halabi S, El Hayek S, Essam L, Gashi Bytyçi D, et al. Mental health in the post-COVID-19 era: challenges and the way forward. Gen Psychiatr. 2021;34(1):e100424. [FREE Full text] [CrossRef] [Medline]
    10. Strudwick G, Sockalingam S, Kassam I, Sequeira L, Bonato S, Youssef A, et al. Digital interventions to support population mental health in Canada during the COVID-19 pandemic: rapid review. JMIR Ment Health. Mar 02, 2021;8(3):e26550. [FREE Full text] [CrossRef] [Medline]
    11. Wellness Together Canada. URL: https://www.wellnesstogether.ca/ [accessed 2023-07-22]
    12. Kelders SM, Kok RN, Ossebaard HC, van Gemert-Pijnen JEWC. Persuasive system design does matter: a systematic review of adherence to web-based interventions. J Med Internet Res. Nov 14, 2012;14(6):e152. [FREE Full text] [CrossRef] [Medline]
    13. Borghouts J, Eikey E, Mark G, de Leon C, Schueller SM, Schneider M, et al. Barriers to and facilitators of user engagement with digital mental health interventions: systematic review. J Med Internet Res. Mar 24, 2021;23(3):e24387. [FREE Full text] [CrossRef] [Medline]
    14. Gulliver A, Calear AL, Sunderland M, Kay-Lambkin F, Farrer LM, Batterham PJ. Predictors of acceptability and engagement in a self-guided online program for depression and anxiety. Internet Interv. Sep 2021;25:100400. [FREE Full text] [CrossRef] [Medline]
    15. Perski O, Short CE. Acceptability of digital health interventions: embracing the complexity. Transl Behav Med. Jul 29, 2021;11(7):1473-1480. [FREE Full text] [CrossRef] [Medline]
    16. Sekhon M, Cartwright M, Francis JJ. Acceptability of healthcare interventions: an overview of reviews and development of a theoretical framework. BMC Health Serv Res. Jan 26, 2017;17(1):88. [FREE Full text] [CrossRef] [Medline]
    17. Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. Mar 2011;38(2):65-76. [FREE Full text] [CrossRef] [Medline]
    18. Zauszniewski JA. Intervention development: assessing critical parameters from the intervention recipient's perspective. Appl Nurs Res. Feb 2012;25(1):31-39. [CrossRef] [Medline]
    19. Davis FD. A technology acceptance model for empirically testing new end-user information systems : theory and results. Dissertation Massachusetts Institute of Technology. 1986. URL: http://hdl.handle.net/1721.1/15192 [accessed 2023-07-08]
    20. Welcome to the new preferred reporting items for systematic reviews and meta-analyses (PRISMA) website. PRISMA. URL: http://www.prisma-statement.org/ [accessed 2023-07-04]
    21. Covidence. URL: https://www.covidence.org/ [accessed 2023-07-04]
    22. Oxford Centre for Evidence-Based Medicine: levels of evidence (March 2009). Oxford Centre for Evidence-Based Medicine. URL: https:/​/www.​cebm.ox.ac.uk/​resources/​levels-of-evidence/​oxford-centre-for-evidence-based-medicine-levels-of-evidence-march-2009 [accessed 2023-07-22]
    23. Schünemann H, Brożek J, Guyatt G, Oxman A. GRADE handbook. GRADEpro. URL: https://gdt.gradepro.org/app/handbook/handbook.html [accessed 2024-02-18]
    24. Berger T, Hämmerli K, Gubser N, Andersson G, Caspar F. Internet-based treatment of depression: a randomized controlled trial comparing guided with unguided self-help. Cogn Behav Ther. 2011;40(4):251-266. [CrossRef] [Medline]
    25. Birney AJ, Gunn R, Russell JK, Ary DV. MoodHacker mobile web app with email for adults to self-manage mild-to-moderate depression: randomized controlled trial. JMIR Mhealth Uhealth. Jan 26, 2016;4(1):e8. [FREE Full text] [CrossRef] [Medline]
    26. Blackwell SE, Browning M, Mathews A, Pictet A, Welch J, Davies J, et al. Positive imagery-based cognitive bias modification as a web-based treatment tool for depressed adults: a randomized controlled trial. Clin Psychol Sci. Jan 2015;3(1):91-111. [FREE Full text] [CrossRef] [Medline]
    27. Bücker L, Schnakenberg P, Karyotaki E, Moritz S, Westermann S. Diminishing effects after recurrent use of self-guided internet-based interventions in depression: randomized controlled trial. J Med Internet Res. Oct 02, 2019;21(10):e14240. [FREE Full text] [CrossRef] [Medline]
    28. Buntrock C, Ebert D, Lehr D, Riper H, Smit F, Cuijpers P, et al. Effectiveness of a web-based cognitive behavioural intervention for subthreshold depression: pragmatic randomised controlled trial. Psychother Psychosom. 2015;84(6):348-358. [CrossRef] [Medline]
    29. Burton C, Szentagotai Tatar A, McKinstry B, Matheson C, Matu S, Moldovan R, et al. Pilot randomised controlled trial of Help4Mood, an embodied virtual agent-based system to support treatment of depression. J Telemed Telecare. Sep 2016;22(6):348-355. [CrossRef] [Medline]
    30. de Graaf LE, Huibers MJH, Riper H, Gerhards SAH, Arntz A. Use and acceptability of unsupported online computerized cognitive behavioral therapy for depression and associations with clinical outcome. J Affect Disord. Aug 2009;116(3):227-231. [CrossRef] [Medline]
    31. Geraedts AS, Kleiboer AM, Wiezer NM, Cuijpers P, van Mechelen W, Anema JR. Feasibility of a worker-directed web-based intervention for employees with depressive symptoms. Internet Interventions. Jul 2014;1(3):132-140. [CrossRef]
    32. Gómez Penedo JM, Babl AM, Grosse Holtforth M, Hohagen F, Krieger T, Lutz W, et al. The association of therapeutic alliance with long-term outcome in a guided internet intervention for depression: secondary analysis from a randomized control trial. J Med Internet Res. Mar 24, 2020;22(3):e15824. [FREE Full text] [CrossRef] [Medline]
    33. Høifødt RS, Lillevoll KR, Griffiths KM, Wilsgaard T, Eisemann M, Waterloo K, et al. The clinical effectiveness of web-based cognitive behavioral therapy with face-to-face therapist support for depressed primary care patients: randomized controlled trial. J Med Internet Res. Aug 05, 2013;15(8):e153. [FREE Full text] [CrossRef] [Medline]
    34. Johansson O, Bjärehed J, Andersson G, Carlbring P, Lundh LG. Effectiveness of guided internet-delivered cognitive behavior therapy for depression in routine psychiatry: a randomized controlled trial. Internet Interv. Sep 2019;17:100247. [FREE Full text] [CrossRef] [Medline]
    35. Kelders SM, Bohlmeijer ET, Pots WTM, van Gemert-Pijnen JEWC. Comparing human and automated support for depression: Fractional factorial randomized controlled trial. Behav Res Ther. Sep 2015;72:72-80. [CrossRef] [Medline]
    36. Kenter RMF, Cuijpers P, Beekman A, van Straten A. Effectiveness of a web-based guided self-help intervention for outpatients with a depressive disorder: short-term results from a randomized controlled trial. J Med Internet Res. Mar 31, 2016;18(3):e80. [FREE Full text] [CrossRef] [Medline]
    37. Kok G, Bockting C, Burger H, Smit F, Riper H. Mobile cognitive therapy: adherence and acceptability of an online intervention in remitted recurrently depressed patients. Internet Interventions. Apr 2014;1(2):65-73. [CrossRef]
    38. Lappalainen P, Langrial S, Oinas-Kukkonen H, Tolvanen A, Lappalainen R. Web-based acceptance and commitment therapy for depressive symptoms with minimal support: a randomized controlled trial. Behav Modif. Nov 2015;39(6):805-834. [CrossRef] [Medline]
    39. Levin W, Campbell DR, McGovern KB, Gau JM, Kosty DB, Seeley JR, et al. A computer-assisted depression intervention in primary care. Psychol Med. Jul 2011;41(7):1373-1383. [CrossRef] [Medline]
    40. Löbner M, Pabst A, Stein J, Dorow M, Matschinger H, Luppa M, et al. Computerized cognitive behavior therapy for patients with mild to moderately severe depression in primary care: a pragmatic cluster randomized controlled trial (@ktiv). J Affect Disord. Oct 01, 2018;238:317-326. [CrossRef] [Medline]
    41. Lüdtke T, Pult LK, Schröder J, Moritz S, Bücker L. A randomized controlled trial on a smartphone self-help application (Be Good to Yourself) to reduce depressive symptoms. Psychiatry Res. Nov 2018;269:753-762. [CrossRef] [Medline]
    42. Lukas CA, Berking M. Blending group-based psychoeducation with a smartphone intervention for the reduction of depressive symptoms: results of a randomized controlled pilot study. Pilot Feasibility Stud. Feb 24, 2021;7(1):57. [FREE Full text] [CrossRef] [Medline]
    43. Lukas CA, Eskofier B, Berking M. A gamified smartphone-based intervention for depression: randomized controlled pilot trial. JMIR Ment Health. Jul 20, 2021;8(7):e16643. [FREE Full text] [CrossRef] [Medline]
    44. Ly KH, Trüschel A, Jarl L, Magnusson S, Windahl T, Johansson R, et al. Behavioural activation versus mindfulness-based guided self-help treatment administered through a smartphone application: a randomised controlled trial. BMJ Open. Jan 09, 2014;4(1):e003440. [FREE Full text] [CrossRef] [Medline]
    45. Ly KH, Topooco N, Cederlund H, Wallin A, Bergström J, Molander O, et al. Smartphone-supported versus full behavioural activation for depression: a randomised controlled trial. PLoS One. 2015;10(5):e0126559. [FREE Full text] [CrossRef] [Medline]
    46. Meyer B, Berger T, Caspar F, Beevers CG, Andersson G, Weiss M. Effectiveness of a novel integrative online treatment for depression (Deprexis): randomized controlled trial. J Med Internet Res. May 11, 2009;11(2):e15. [FREE Full text] [CrossRef] [Medline]
    47. Meyer B, Bierbrodt J, Schröder J, Berger T, Beevers CG, Weiss M, et al. Effects of an internet intervention (Deprexis) on severe depression symptoms: randomized controlled trial. Internet Interventions. Mar 2015;2(1):48-59. [CrossRef]
    48. Moritz S, Schilling L, Hauschildt M, Schröder J, Treszl A. A randomized controlled trial of internet-based therapy in depression. Behav Res Ther. Aug 2012;50(7-8):513-521. [CrossRef] [Medline]
    49. Motter JN, Grinberg A, Lieberman DH, Iqnaibi WB, Sneed JR. Computerized cognitive training in young adults with depressive symptoms: effects on mood, cognition, and everyday functioning. J Affect Disord. Feb 15, 2019;245:28-37. [CrossRef] [Medline]
    50. Oehler C, Görges F, Rogalla M, Rummel-Kluge C, Hegerl U. Efficacy of a guided web-based self-management intervention for depression or dysthymia: randomized controlled trial with a 12-month follow-up using an active control condition. J Med Internet Res. Jul 14, 2020;22(7):e15361. [FREE Full text] [CrossRef] [Medline]
    51. Perini S, Titov N, Andrews G. Clinician-assisted internet-based treatment is effective for depression: randomized controlled trial. Aust N Z J Psychiatry. Jun 2009;43(6):571-578. [CrossRef] [Medline]
    52. Pinto MD, Greenblatt AM, Hickman RL, Rice HM, Thomas TL, Clochesy JM. Assessing the critical parameters of eSMART-MH: a promising avatar-based digital therapeutic intervention to reduce depressive symptoms. Perspect Psychiatr Care. Jul 2016;52(3):157-168. [CrossRef] [Medline]
    53. Reins JA, Boß L, Lehr D, Berking M, Ebert DD. The more I got, the less I need? Efficacy of Internet-based guided self-help compared to online psychoeducation for major depressive disorder. J Affect Disord. Mar 01, 2019;246:695-705. [CrossRef] [Medline]
    54. Ruwaard J, Schrieken B, Schrijver M, Broeksteeg J, Dekker J, Vermeulen H, et al. Standardized web-based cognitive behavioural therapy of mild to moderate depression: a randomized controlled trial with a long-term follow-up. Cogn Behav Ther. 2009;38(4):206-221. [CrossRef] [Medline]
    55. Smith J, Newby JM, Burston N, Murphy MJ, Michael S, Mackenzie A, et al. Help from home for depression: a randomised controlled trial comparing internet-delivered cognitive behaviour therapy with bibliotherapy for depression. Internet Interv. Sep 2017;9:25-37. [FREE Full text] [CrossRef] [Medline]
    56. Stephens VC. Effects on depressive symptoms of a web-based cognitive bias modification-interpretation (CBM-I) program for emotion recognition: a randomised controlled trial. University of Exeter. URL: http://hdl.handle.net/10871/16430 [accessed 2024-09-17]
    57. Adams S, Penton-Voak IS, Harmer CJ, Holmes EA, Munafò MR. Effects of emotion recognition training on mood among individuals with high levels of depressive symptoms: study protocol for a randomised controlled trial. Trials. Jun 01, 2013;14:161. [FREE Full text] [CrossRef] [Medline]
    58. Titov N, Dear BF, Staples LG, Terides MD, Karin E, Sheehan J, et al. Disorder-specific versus transdiagnostic and clinician-guided versus self-guided treatment for major depressive disorder and comorbid anxiety disorders: a randomized controlled trial. J Anxiety Disord. Oct 2015;35:88-102. [FREE Full text] [CrossRef] [Medline]
    59. Titov N, Andrews G, Davies M, McIntyre K, Robinson E, Solley K. Internet treatment for depression: a randomized controlled trial comparing clinician vs. technician assistance. PLoS One. Jun 08, 2010;5(6):e10939. [FREE Full text] [CrossRef] [Medline]
    60. Tønning ML, Faurholt-Jepsen M, Frost M, Martiny K, Tuxen N, Rosenberg N, et al. The effect of smartphone-based monitoring and treatment on the rate and duration of psychiatric readmission in patients with unipolar depressive disorder: The RADMIS randomized controlled trial. J Affect Disord. Mar 01, 2021;282:354-363. [CrossRef] [Medline]
    61. Wahbeh H. Internet mindfulness meditation intervention (IMMI) improves depression symptoms in older adults. Medicines (Basel). Nov 02, 2018;5(4):119. [FREE Full text] [CrossRef] [Medline]
    62. Watts S, Mackenzie A, Thomas C, Griskaitis A, Mewton L, Williams A, et al. CBT for depression: a pilot RCT comparing mobile phone vs. computer. BMC Psychiatry. Feb 07, 2013;13:49. [FREE Full text] [CrossRef] [Medline]
    63. Williams AD, Blackwell SE, Mackenzie A, Holmes EA, Andrews G. Combining imagination and reason in the treatment of depression: a randomized controlled trial of internet-based cognitive-bias modification and internet-CBT for depression. J Consult Clin Psychol. Oct 2013;81(5):793-799. [FREE Full text] [CrossRef] [Medline]
    64. Williams AD, O'Moore K, Blackwell SE, Smith J, Holmes EA, Andrews G. Positive imagery cognitive bias modification (CBM) and internet-based cognitive behavioral therapy (iCBT): a randomized controlled trial. J Affect Disord. Jun 01, 2015;178:131-141. [FREE Full text] [CrossRef] [Medline]
    65. Wong VWH, Ho FYY, Shi NK, Tong JTY, Chung KF, Yeung WF, et al. Smartphone-delivered multicomponent lifestyle medicine intervention for depressive symptoms: a randomized controlled trial. J Consult Clin Psychol. Dec 2021;89(12):970-984. [CrossRef] [Medline]
    66. Addington EL, Cheung EO, Bassett SM, Kwok I, Schuette SA, Shiu E, et al. The MARIGOLD study: feasibility and enhancement of an online intervention to improve emotion regulation in people with elevated depressive symptoms. J Affect Disord. Oct 01, 2019;257:352-364. [FREE Full text] [CrossRef] [Medline]
    67. Berman MI, Buckey JC, Hull JG, Linardatos E, Song SL, McLellan RK, et al. Feasibility study of an interactive multimedia electronic problem solving treatment program for depression: a preliminary uncontrolled trial. Behav Ther. May 2014;45(3):358-375. [FREE Full text] [CrossRef] [Medline]
    68. Burns MN, Begale M, Duffecy J, Gergle D, Karr CJ, Giangrande E, et al. Harnessing context sensing to develop a mobile intervention for depression. J Med Internet Res. Aug 12, 2011;13(3):e55. [FREE Full text] [CrossRef] [Medline]
    69. Callan JA, Dunbar Jacob J, Siegle GJ, Dey A, Thase ME, DeVito Dabbs A, et al. CBT MobileWork©: user-centered development and testing of a mobile mental health application for depression. Cogn Ther Res. Sep 25, 2020;45(2):287-302. [CrossRef]
    70. Caplan S, Sosa Lovera A, Reyna Liberato P. A feasibility study of a mental health mobile app in the Dominican Republic: the untold story. International Journal of Mental Health. Apr 04, 2019;47(4):311-345. [CrossRef]
    71. Cartreine JA, Locke SE, Buckey JC, Sandoval L, Hegel MT. Electronic problem-solving treatment: description and pilot study of an interactive media treatment for depression. JMIR Res Protoc. Sep 25, 2012;1(2):e11. [FREE Full text] [CrossRef] [Medline]
    72. Coutinho E, Alshukri A, de Berardinis J, Dowrick C. POLYHYMNIA mood – empowering people to cope with depression through music listening. 2021. Presented at: Adjunct Proceedings of the 2021 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2021 ACM International Symposium on Wearable Computers; September 24, 2021:188-193; Virtual event. [CrossRef]
    73. Dahne J, Kustanowitz J, Lejuez CW. Development and preliminary feasibility study of a brief behavioral activation mobile application (Behavioral Apptivation) to be used in conjunction with ongoing therapy. Cogn Behav Pract. Feb 2018;25(1):44-56. [FREE Full text] [CrossRef] [Medline]
    74. Dear BF, Zou J, Titov N, Lorian C, Johnston L, Spence J, et al. Internet-delivered cognitive behavioural therapy for depression: a feasibility open trial for older adults. Aust N Z J Psychiatry. Feb 2013;47(2):169-176. [CrossRef] [Medline]
    75. Drake G, Csipke E, Wykes T. Assessing your mood online: acceptability and use of Moodscope. Psychol Med. Jul 2013;43(7):1455-1464. [FREE Full text] [CrossRef] [Medline]
    76. Fuller-Tyszkiewicz M, Richardson B, Klein B, Skouteris H, Christensen H, Austin D, et al. A mobile app-based intervention for depression: end-user and expert usability testing study. JMIR Ment Health. Aug 23, 2018;5(3):e54. [FREE Full text] [CrossRef] [Medline]
    77. Gould CE, Carlson C, Ma F, Forman-Hoffman V, Ranta K, Kuhn E. Effects of mobile app-based intervention for depression in middle-aged and older adults: mixed methods feasibility study. JMIR Form Res. Jun 29, 2021;5(6):e25808. [FREE Full text] [CrossRef] [Medline]
    78. Harper Shehadeh MJ, Abi Ramia J, Cuijpers P, El Chammay R, Heim E, Kheir W, et al. Step-by-step, an e-mental health intervention for depression: a mixed methods pilot study from Lebanon. Front Psychiatry. 2019;10:986. [FREE Full text] [CrossRef] [Medline]
    79. Jacmon J, Malouff J, Taylor N. Treatment of major depression: effectiveness of cognitive behavior therapy with an internet course as a central component. E-J Appl Psychol. 2010;5(2):1-8.
    80. Kooistra LC, Ruwaard J, Wiersma JE, van Oppen P, van der Vaart R, van Gemert-Pijnen JE, et al. Development and initial evaluation of blended cognitive behavioural treatment for major depression in routine specialized mental health care. Internet Interv. May 2016;4:61-71. [FREE Full text] [CrossRef] [Medline]
    81. Mayer G, Hummel S, Oetjen N, Gronewold N, Bubolz S, Blankenhagel K, et al. User experience and acceptance of patients and healthy adults testing a personalized self-management app for depression: a non-randomized mixed-methods feasibility study. Digit Health. 2022;8:20552076221091353. [FREE Full text] [CrossRef] [Medline]
    82. Schlosser DA, Campellone TR, Truong B, Anguera JA, Vergani S, Vinogradov S, et al. The feasibility, acceptability, and outcomes of PRIME-D: a novel mobile intervention treatment for depression. Depress Anxiety. Jun 2017;34(6):546-554. [FREE Full text] [CrossRef] [Medline]
    83. Schuster R, Kalthoff I, Walther A, Köhldorfer L, Partinger E, Berger T, et al. Effects, adherence, and therapists' perceptions of web- and mobile-supported group therapy for depression: mixed-methods study. J Med Internet Res. Apr 28, 2019;21(5):e11860. [FREE Full text] [CrossRef] [Medline]
    84. Stiles-Shields C, Montague E, Kwasny MJ, Mohr DC. Behavioral and cognitive intervention strategies delivered via coached apps for depression: Pilot trial. Psychol Serv. May 2019;16(2):233-238. [FREE Full text] [CrossRef] [Medline]
    85. Strauss C, Dunkeld C, Cavanagh K. Is clinician-supported use of a mindfulness smartphone app a feasible treatment for depression? A mixed-methods feasibility study. Internet Interv. Sep 2021;25:100413. [FREE Full text] [CrossRef] [Medline]
    86. Elliott R, Slatick E, Urman M. Qualitative change process research on psychotherapy: alternative strategies. Psychologische Beitrage. 2001;43(3):69-111.
    87. Tomasino KN, Lattie EG, Ho J, Palac HL, Kaiser SM, Mohr DC. Harnessing peer support in an online intervention for older adults with depression. Am J Geriatr Psychiatry. Oct 2017;25(10):1109-1119. [FREE Full text] [CrossRef] [Medline]
    88. Walsh SM. Development of an online intervention using positive psychology for depression. Queen Mary University of London. UK.; 2018. URL: http://qmro.qmul.ac.uk/xmlui/handle/123456789/31871 [accessed 2024-09-17]
    89. Yeung A, Wang F, Feng F, Zhang J, Cooper A, Hong L, et al. Outcomes of an online computerized cognitive behavioral treatment program for treating Chinese patients with depression: a pilot study. Asian J Psychiatr. Dec 2018;38:102-107. [CrossRef] [Medline]
    90. Schneider J, Sarrami Foroushani P, Grime P, Thornicroft G. Acceptability of online self-help to people with depression: users' views of MoodGYM versus informational websites. J Med Internet Res. Mar 28, 2014;16(3):e90. [FREE Full text] [CrossRef] [Medline]
    91. Knowles SE, Lovell K, Bower P, Gilbody S, Littlewood E, Lester H. Patient experience of computerised therapy for depression in primary care. BMJ Open. Nov 30, 2015;5(11):e008581. [FREE Full text] [CrossRef] [Medline]
    92. Ly KH, Janni E, Wrede R, Sedem M, Donker T, Carlbring P, et al. Experiences of a guided smartphone-based behavioral activation therapy for depression: a qualitative study. Internet Interventions. Mar 2015;2(1):60-68. [CrossRef]
    93. Allen AR, Newby JM, Mackenzie A, Smith J, Boulton M, Loughnan SA, et al. Internet cognitive-behavioural treatment for panic disorder: randomised controlled trial and evidence of effectiveness in primary care. BJPsych Open. Mar 2016;2(2):154-162. [FREE Full text] [CrossRef] [Medline]
    94. Bell CJ, Colhoun HC, Carter FA, Frampton CM. Effectiveness of computerised cognitive behaviour therapy for anxiety disorders in secondary care. Aust N Z J Psychiatry. Jul 2012;46(7):630-640. [CrossRef] [Medline]
    95. Berger T, Hohl E, Caspar F. Internet-based treatment for social phobia: a randomized controlled trial. J Clin Psychol. Oct 2009;65(10):1021-1035. [CrossRef] [Medline]
    96. Berger T, Caspar F, Richardson R, Kneubühler B, Sutter D, Andersson G. Internet-based treatment of social phobia: a randomized controlled trial comparing unguided with two types of guided self-help. Behav Res Ther. Mar 2011;49(3):158-169. [CrossRef] [Medline]
    97. Berger T, Boettcher J, Caspar F. Internet-based guided self-help for several anxiety disorders: a randomized controlled trial comparing a tailored with a standardized disorder-specific approach. Psychotherapy (Chic). Jun 2014;51(2):207-219. [CrossRef] [Medline]
    98. Boettcher J, Leek L, Matson L, Holmes EA, Browning M, MacLeod C, et al. Internet-based attention bias modification for social anxiety: a randomised controlled comparison of training towards negative and training towards positive cues. PLoS One. 2013;8(9):e71760. [FREE Full text] [CrossRef] [Medline]
    99. Boettcher J, Aström V, Påhlsson D, Schenström O, Andersson G, Carlbring P. Internet-based mindfulness treatment for anxiety disorders: a randomized controlled trial. Behav Ther. Mar 2014;45(2):241-253. [FREE Full text] [CrossRef] [Medline]
    100. Boettcher J, Hasselrot J, Sund E, Andersson G, Carlbring P. Combining attention training with internet-based cognitive-behavioural self-help for social anxiety: a randomised controlled trial. Cogn Behav Ther. 2014;43(1):34-48. [FREE Full text] [CrossRef] [Medline]
    101. Carlbring P, Westling BE, Ljungstrand P, Ekselius L, Andersson G. Treatment of panic disorder via the internet: a randomized trial of a self-help program. Behavior Therapy. 2001;32(4):751-764. [CrossRef]
    102. Carlbring P, Ekselius L, Andersson G. Treatment of panic disorder via the Internet: a randomized trial of CBT vs. applied relaxation. J Behav Ther Exp Psychiatry. Jun 2003;34(2):129-140. [CrossRef] [Medline]
    103. Carlbring P, Nilsson-Ihrfelt E, Waara J, Kollenstam C, Buhrman M, Kaldo V, et al. Treatment of panic disorder: live therapy vs. self-help via the internet. Behav Res Ther. Oct 2005;43(10):1321-1333. [CrossRef] [Medline]
    104. Carlbring P, Bohman S, Brunt S, Buhrman M, Westling BE, Ekselius L, et al. Remote treatment of panic disorder: a randomized trial of internet-based cognitive behavior therapy supplemented with telephone calls. Am J Psychiatry. Dec 2006;163(12):2119-2125. [CrossRef] [Medline]
    105. Carter FA, Bell CJ, Colhoun HC. Suitability and acceptability of computerised cognitive behaviour therapy for anxiety disorders in secondary care. Aust N Z J Psychiatry. Feb 2013;47(2):142-152. [CrossRef] [Medline]
    106. Dear BF, Staples LG, Terides MD, Karin E, Zou J, Johnston L, et al. Transdiagnostic versus disorder-specific and clinician-guided versus self-guided internet-delivered treatment for generalized anxiety disorder and comorbid disorders: a randomized controlled trial. J Anxiety Disord. Dec 2015;36:63-77. [FREE Full text] [CrossRef] [Medline]
    107. Dear BF, Staples LG, Terides MD, Fogliati VJ, Sheehan J, Johnston L, et al. Transdiagnostic versus disorder-specific and clinician-guided versus self-guided internet-delivered treatment for Social Anxiety Disorder and comorbid disorders: a randomized controlled trial. J Anxiety Disord. Aug 2016;42:30-44. [FREE Full text] [CrossRef] [Medline]
    108. Ebenfeld L, Lehr D, Ebert DD, Kleine Stegemann S, Riper H, Funk B, et al. Evaluating a hybrid web-based training program for panic disorder and agoraphobia: randomized controlled trial. J Med Internet Res. Mar 04, 2021;23(3):e20829. [FREE Full text] [CrossRef] [Medline]
    109. Fogliati VJ, Dear BF, Staples LG, Terides MD, Sheehan J, Johnston L, et al. Disorder-specific versus transdiagnostic and clinician-guided versus self-guided internet-delivered treatment for panic disorder and comorbid disorders: a randomized controlled trial. J Anxiety Disord. Apr 2016;39:88-102. [FREE Full text] [CrossRef] [Medline]
    110. Furmark T, Carlbring P, Hedman E, Sonnenstein A, Clevberger P, Bohman B, et al. Guided and unguided self-help for social anxiety disorder: randomised controlled trial. Br J Psychiatry. Nov 2009;195(5):440-447. [CrossRef] [Medline]
    111. Gallego MJ, Emmelkamp PMG, van der Kooij M, Mees H. The effects of a Dutch version of an Internet- based treatment program for fear of public speaking: a controlled study. Int J Clin Health Psychol. 2011;11(3):459-472.
    112. Hedman E, Andersson G, Ljótsson B, Andersson E, Rück C, Mörtberg E, et al. Internet-based cognitive behavior therapy vs. cognitive behavioral group therapy for social anxiety disorder: a randomized controlled non-inferiority trial. PLoS One. Mar 25, 2011;6(3):e18001. [FREE Full text] [CrossRef] [Medline]
    113. Johnston L, Titov N, Andrews G, Spence J, Dear BF. A RCT of a transdiagnostic internet-delivered treatment for three anxiety disorders: examination of support roles and disorder-specific outcomes. PLoS One. 2011;6(11):e28079. [FREE Full text] [CrossRef] [Medline]
    114. Johnston L. Development and evaluation of a transdiagnostic internet-delivered cognitive behavioural therapy program for three anxiety disorders [Thesis]. UNSW Sydney. 2012. URL: http://hdl.handle.net/1959.4/52139 [accessed 2024-10-14]
    115. Jones SL, Hadjistavropoulos HD, Soucy JN. A randomized controlled trial of guided internet-delivered cognitive behaviour therapy for older adults with generalized anxiety. J Anxiety Disord. Jan 2016;37:1-9. [CrossRef] [Medline]
    116. Kiropoulos LA, Klein B, Austin DW, Gilson K, Pier C, Mitchell J, et al. Is internet-based CBT for panic disorder and agoraphobia as effective as face-to-face CBT? J Anxiety Disord. Dec 2008;22(8):1273-1284. [CrossRef] [Medline]
    117. Klein B, Richards JC, Austin DW. Efficacy of internet therapy for panic disorder. J Behav Ther Exp Psychiatry. Sep 2006;37(3):213-238. [CrossRef] [Medline]
    118. Klein B, Austin D, Pier C, Kiropoulos L, Shandley K, Mitchell J, et al. Internet-based treatment for panic disorder: does frequency of therapist contact make a difference? Cogn Behav Ther. 2009;38(2):100-113. [CrossRef] [Medline]
    119. Kok RN, van Straten A, Beekman ATF, Cuijpers P. Short-term effectiveness of web-based guided self-help for phobic outpatients: randomized controlled trial. J Med Internet Res. Sep 29, 2014;16(9):e226. [FREE Full text] [CrossRef] [Medline]
    120. Pham Q, Khatib Y, Stansfeld S, Fox S, Green T. Feasibility and efficacy of an mHealth game for managing anxiety: "Flowy" randomized controlled pilot trial and design evaluation. Games Health J. Feb 2016;5(1):50-67. [CrossRef] [Medline]
    121. Robinson E, Titov N, Andrews G, McIntyre K, Schwencke G, Solley K. Internet treatment for generalized anxiety disorder: a randomized controlled trial comparing clinician vs. technician assistance. PLoS One. Jun 03, 2010;5(6):e10942. [FREE Full text] [CrossRef] [Medline]
    122. Ruwaard J, Broeksteeg J, Schrieken B, Emmelkamp P, Lange A. Web-based therapist-assisted cognitive behavioral treatment of panic symptoms: a randomized controlled trial with a three-year follow-up. J Anxiety Disord. May 2010;24(4):387-396. [CrossRef] [Medline]
    123. Schneider AJ, Mataix-Cols D, Marks IM, Bachofen M. Internet-guided self-help with or without exposure therapy for phobic and panic disorders. Psychother Psychosom. 2005;74(3):154-164. [CrossRef] [Medline]
    124. Schröder J, Jelinek L, Moritz S. A randomized controlled trial of a transdiagnostic Internet intervention for individuals with panic and phobias - one size fits all. J Behav Ther Exp Psychiatry. Mar 2017;54:17-24. [CrossRef] [Medline]
    125. Stech EP, Chen AZ, Sharrock MJ, Grierson AB, Upton EL, Mahoney AEJ, et al. Internet-delivered exposure therapy versus internet-delivered cognitive behavioral therapy for panic disorder: A pilot randomized controlled trial. J Anxiety Disord. Apr 2021;79:102382. [CrossRef] [Medline]
    126. Titov N, Andrews G, Schwencke G, Drobny J, Einstein D. Shyness 1: distance treatment of social phobia over the Internet. Aust N Z J Psychiatry. Jul 2008;42(7):585-594. [CrossRef] [Medline]
    127. Titov N, Andrews G, Schwencke G. Shyness 2: treating social phobia online: replication and extension. Aust N Z J Psychiatry. Jul 2008;42(7):595-605. [CrossRef] [Medline]
    128. Titov N, Andrews G, Choi I, Schwencke G, Mahoney A. Shyness 3: randomized controlled trial of guided versus unguided Internet-based CBT for social phobia. Aust N Z J Psychiatry. Dec 2008;42(12):1030-1040. [CrossRef] [Medline]
    129. Titov N, Andrews G, Johnston L, Schwencke G, Choi I. Shyness programme: longer term benefits, cost-effectiveness, and acceptability. Aust N Z J Psychiatry. Jan 2009;43(1):36-44. [CrossRef] [Medline]
    130. Titov N, Andrews G, Choi I, Schwencke G, Johnston L. Randomized controlled trial of web-based treatment of social phobia without clinician guidance. Aust N Z J Psychiatry. Jan 01, 2009;43(10):913-919. [CrossRef]
    131. Titov N, Andrews G, Schwencke G, Solley K, Johnston L, Robinson E. An RCT comparing effect of two types of support on severity of symptoms for people completing internet-based cognitive behaviour therapy for social phobia. Aust N Z J Psychiatry. Jan 01, 2009;43(10):920-926. [CrossRef]
    132. Titov N, Andrews G, Robinson E, Schwencke G, Johnston L, Solley K, et al. Clinician-assisted internet-based treatment is effective for generalized anxiety disorder: randomized controlled trial. Aust N Z J Psychiatry. Jan 01, 2009;43(10):905-912. [CrossRef]
    133. Titov N, Andrews G, Johnston L, Robinson E, Spence J. Transdiagnostic Internet treatment for anxiety disorders: a randomized controlled trial. Behav Res Ther. Sep 2010;48(9):890-899. [CrossRef] [Medline]
    134. Titov N, Andrews G, Schwencke G, Robinson E, Peters L, Spence J. Randomized controlled trial of Internet cognitive behavioural treatment for social phobia with and without motivational enhancement strategies. Aust N Z J Psychiatry. Oct 2010;44(10):938-945. [CrossRef] [Medline]
    135. Tulbure BT, Szentagotai A, David O, Ștefan S, Månsson KNT, David D, et al. Internet-delivered cognitive-behavioral therapy for social anxiety disorder in Romania: a randomized controlled trial. PLoS One. 2015;10(5):e0123997. [FREE Full text] [CrossRef] [Medline]
    136. Aydos L, Titov N, Andrews G. Shyness 5: the clinical effectiveness of Internet-based clinician-assisted treatment of social phobia. Australas Psychiatry. Dec 2009;17(6):488-492. [CrossRef] [Medline]
    137. Botella C, Gallego M, Garcia-Palacios A, Baños R, Quero S, Alcañiz M. The acceptability of an Internet-based self-help treatment for fear of public speaking. British Journal of Guidance & Counselling. Aug 2009;37(3):297-311. [CrossRef]
    138. Botella C, Guillen V, Banos RM, García-Palacios A, Gallego MJ, Alcaniz M. Telepsychology and self-help: the treatment of fear of public speaking. Cognitive and Behavioral Practice. Feb 2007;14(1):46-57. [CrossRef]
    139. Botella C, Hofmann SG, Moscovitch DA. A self-applied, internet-based intervention for fear of public speaking. J Clin Psychol. Aug 2004;60(8):821-830. [CrossRef] [Medline]
    140. Carlbring P, Furmark T, Steczkó J, Ekselius L, Andersson G. An open study of Internet-based bibliotherapy with minimal therapist contact via email for social phobia. Clinical Psychologist. Feb 03, 2007;10(1):30-38. [CrossRef]
    141. Draper M, Rees CS, Nathan PR. Internet-based self-management of generalised anxiety disorder: a preliminary study. Behav. change. Feb 22, 2012;25(4):229-244. [CrossRef]
    142. Ebenfeld L, Kleine Stegemann S, Lehr D, Ebert DD, Funk B, Riper H, et al. A mobile application for panic disorder and agoraphobia: insights from a multi-methods feasibility study. Internet Interv. Mar 2020;19:100296. [FREE Full text] [CrossRef] [Medline]
    143. Gruber K, Moran PJ, Roth WT, Taylor CB. Computer-assisted cognitive behavioral group therapy for social phobia. Behavior Therapy. 2001;32(1):155-165. [CrossRef]
    144. Kenardy JA, Dow MGT, Johnston DW, Newman MG, Thomson A, Taylor CB. A comparison of delivery methods of cognitive-behavioral therapy for panic disorder: an international multicenter trial. J Consult Clin Psychol. Dec 2003;71(6):1068-1075. [CrossRef] [Medline]
    145. Kenwright M, Marks IM, Gega L, Mataix-Cols D. Computer-aided self-help for phobia/panic via internet at home: a pilot study. Br J Psychiatry. May 2004;184:448-449. [CrossRef] [Medline]
    146. Klein B, Shandley K, Austin D, Nordin S. A pilot trial of ‘Panic Online’ as a self-guided treatment for panic disorder. EJ Appl Psychol. 2008;4(2):25-30. [CrossRef]
    147. Krafft J, Ong CW, Davis CH, Petersen JM, Levin ME, Twohig MP. An open trial of group acceptance and commitment therapy with an adjunctive mobile app for generalized anxiety disorder. Cognitive and Behavioral Practice. Nov 2022;29(4):846-859. [CrossRef]
    148. Lim MH, Rodebaugh TL, Eres R, Long KM, Penn DL, Gleeson JFM. A pilot digital intervention targeting loneliness in youth mental health. Front Psychiatry. 2019;10:604. [FREE Full text] [CrossRef] [Medline]
    149. Loo Gee B, Batterham PJ, Gulliver A, Reynolds J, Griffiths KM. An ecological momentary intervention for people with social anxiety: a descriptive case study. Inform Health Soc Care. Dec 02, 2021;46(4):370-398. [CrossRef] [Medline]
    150. Miller CB, Gu J, Henry AL, Davis ML, Espie CA, Stott R, et al. Feasibility and efficacy of a digital CBT intervention for symptoms of generalized anxiety disorder: a randomized multiple-baseline study. J Behav Ther Exp Psychiatry. Mar 2021;70:101609. [CrossRef] [Medline]
    151. Miralles I, Granell C, García-Palacios A, Castilla D, González-Pérez A, Casteleyn S, et al. Enhancing in vivo exposure in the treatment of panic disorder and agoraphobia using location-based technologies: a case study. Clinical Case Studies. Dec 20, 2019;19(2):145-159. [CrossRef]
    152. Newman MG, Kenardy J, Herman S, Taylor CB. Comparison of palmtop-computer-assisted brief cognitive-behavioral treatment to cognitive-behavioral treatment for panic disorder. J Consult Clin Psychol. Feb 1997;65(1):178-183. [CrossRef] [Medline]
    153. Oser M, Wallace ML, Solano F, Szigethy EM. Guided digital cognitive behavioral program for anxiety in primary care: propensity-matched controlled trial. JMIR Ment Health. Apr 04, 2019;6(4):e11981. [FREE Full text] [CrossRef] [Medline]
    154. Pier C, Austin DW, Klein B, Mitchell J, Schattner P, Ciechomski L, et al. A controlled trial of internet-based cognitive-behavioural therapy for panic disorder with face-to-face support from a general practitioner or email support from a psychologist. Ment Health Fam Med. Mar 2008;5(1):29-39. [FREE Full text] [Medline]
    155. Shandley K, Austin DW, Klein B, Pier C, Schattner P, Pierce D, et al. Therapist-assisted, Internet-based treatment for panic disorder: can general practitioners achieve comparable patient outcomes to psychologists? J Med Internet Res. May 19, 2008;10(2):e14. [FREE Full text] [CrossRef] [Medline]
    156. Tolin DF, McGrath PB, Hale LR, Weiner DN, Gueorguieva R. A multisite benchmarking trial of capnometry guided respiratory intervention for panic disorder in naturalistic treatment settings. Appl Psychophysiol Biofeedback. Mar 2017;42(1):51-58. [FREE Full text] [CrossRef] [Medline]
    157. Williams AD, O'Moore K, Mason E, Andrews G. The effectiveness of internet cognitive behaviour therapy (iCBT) for social anxiety disorder across two routine practice pathways. Internet Interventions. Oct 2014;1(4):225-229. [CrossRef]
    158. Batterham PJ, Calear AL, Sunderland M, Kay-Lambkin F, Farrer LM, Christensen H, et al. A brief intervention to increase uptake and adherence of an internet-based program for depression and anxiety (enhancing engagement with psychosocial interventions): randomized controlled trial. J Med Internet Res. Jul 27, 2021;23(7):e23029. [FREE Full text] [CrossRef] [Medline]
    159. Kleiboer A, Donker T, Seekles W, van Straten A, Riper H, Cuijpers P. A randomized controlled trial on the role of support in Internet-based problem solving therapy for depression and anxiety. Behaviour Research and Therapy. Sep 2015;72:63-71. [CrossRef] [Medline]
    160. Newby JM, Mackenzie A, Williams AD, McIntyre K, Watts S, Wong N, et al. Internet cognitive behavioural therapy for mixed anxiety and depression: a randomized controlled trial and evidence of effectiveness in primary care. Psychol Med. Dec 2013;43(12):2635-2648. [CrossRef] [Medline]
    161. Proudfoot J, Clarke J, Birch M, Whitton AE, Parker G, Manicavasagar V, et al. Impact of a mobile phone and web program on symptom and functional outcomes for people with mild-to-moderate depression, anxiety and stress: a randomised controlled trial. BMC Psychiatry. Nov 18, 2013;13:312. [FREE Full text] [CrossRef] [Medline]
    162. Titov N, Dear BF, Johnston L, Lorian C, Zou J, Wootton B, et al. Improving adherence and clinical outcomes in self-guided internet treatment for anxiety and depression: randomised controlled trial. PLoS One. 2013;8(7):e62873. [FREE Full text] [CrossRef] [Medline]
    163. Titov N, Dear BF, Schwencke G, Andrews G, Johnston L, Craske MG, et al. Transdiagnostic internet treatment for anxiety and depression: a randomised controlled trial. Behav Res Ther. Aug 2011;49(8):441-452. [CrossRef] [Medline]
    164. Adam A, Jain A, Pletnikova A, Bagga R, Vita A, N Richey L, et al. Use of a mobile app to augment psychotherapy in a community psychiatric clinic: feasibility and fidelity trial. JMIR Form Res. Jul 03, 2020;4(7):e17722. [FREE Full text] [CrossRef] [Medline]
    165. Johnston L, Dear BF, Gandy M, Fogliati VJ, Kayrouz R, Sheehan J, et al. Exploring the efficacy and acceptability of internet-delivered cognitive behavioural therapy for young adults with anxiety and depression: an open trial. Aust N Z J Psychiatry. Sep 2014;48(9):819-827. [CrossRef] [Medline]
    166. Kladnitski N, Smith J, Allen A, Andrews G, Newby JM. Online mindfulness-enhanced cognitive behavioural therapy for anxiety and depression: outcomes of a pilot trial. Internet Interv. Sep 2018;13:41-50. [FREE Full text] [CrossRef] [Medline]
    167. Staples LG, Fogliati VJ, Dear BF, Nielssen O, Titov N. Internet-delivered treatment for older adults with anxiety and depression: implementation of the Wellbeing Plus Course in routine clinical care and comparison with research trial outcomes. BJPsych Open. Sep 2016;2(5):307-313. [FREE Full text] [CrossRef] [Medline]
    168. Wright JH, Wright AS, Salmon P, Beck AT, Kuykendall J, Goldsmith LJ, et al. Development and initial testing of a multimedia program for computer-assisted cognitive therapy. Am J Psychother. 2002;56(1):76-86. [CrossRef] [Medline]
    169. Eysenbach G. The law of attrition. J Med Internet Res. Mar 31, 2005;7(1):e11. [FREE Full text] [CrossRef] [Medline]
    170. Forbes A, Keleher MR, Venditto M, DiBiasi F. Assessing patient adherence to and engagement with digital interventions for depression in clinical trials: systematic literature review. J Med Internet Res. Aug 11, 2023;25:e43727. [FREE Full text] [CrossRef] [Medline]
    171. Lee P, Abernethy A, Shaywitz D, Gundlapalli AV, Weinstein J, Doraiswamy PM, et al. Digital health COVID-19 impact assessment: lessons learned and compelling needs. NAM Perspect. 2022;2022. [FREE Full text] [CrossRef] [Medline]
    172. Adu-Brimpong J, Pugh J, Darko DA, Shieh L. Examining diversity in digital therapeutics clinical trials: descriptive analysis. J Med Internet Res. Aug 02, 2023;25:e37447. [FREE Full text] [CrossRef] [Medline]

    ‎
    CBT: cognitive behavioral therapy
    GRADE: Grading of Recommendations, Assessment, Development and Evaluation
    MeSH: Medical Subject Headings
    PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses
    PROSPERO: International Prospective Register of Systematic Reviews
    RCT: randomized controlled trial

    Edited by T de Azevedo Cardoso; submitted 09.09.23; peer-reviewed by A Parks, M Boeker; comments to author 06.02.24; revised version received 27.02.24; accepted 30.04.24; published 28.10.24.

    Copyright

    ©Carrie K Y Lau, Anthony Saad, Bettina Camara, Dia Rahman, Blanca Bolea-Alamanac. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 28.10.2024.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research (ISSN 1438-8871), is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.