This paper is in the following e-collection/theme issue:

Digital Health Reviews (1520) Engagement with and Adherence to Digital Health Interventions, Law of Attrition (538) Mobile Health in Psychiatry (478) Digital Mental Health Interventions, e-Mental Health and Cyberpsychology (1803) Clinical Mental Health Informatics (50) Theory and Frameworks in Mental Health (29)

Published on in Vol 27 (2025)

This is a member publication of Centre for Addiction and Mental Health

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/67820, first published .
Application of a Sociotechnical Framework to Uncover Factors That Influence Effective User Engagement With Digital Mental Health Tools in Clinical Care Contexts: Scoping Review

Application of a Sociotechnical Framework to Uncover Factors That Influence Effective User Engagement With Digital Mental Health Tools in Clinical Care Contexts: Scoping Review

Application of a Sociotechnical Framework to Uncover Factors That Influence Effective User Engagement With Digital Mental Health Tools in Clinical Care Contexts: Scoping Review

Authors of this article:

Brian Lo1, 2 Author Orcid Image ;   Keri Durocher1, 3 Author Orcid Image ;   Rebecca Charow2, 4 Author Orcid Image ;   Sarah Kimball1 Author Orcid Image ;   Quynh Pham2, 4 Author Orcid Image ;   Sanjeev Sockalingam1, 4, 5 Author Orcid Image ;   David Wiljer2, 4, 5 Author Orcid Image ;   Gillian Strudwick1, 2 Author Orcid Image
Article Authors Cited by Tweetations (2) Metrics

    Review

    1Centre for Addiction and Mental Health, Toronto, ON, Canada

    2Institute for Health Policy, Management and Evaluation, University of Toronto, Toronto, ON, Canada

    3Arthur Labatt Family School of Nursing, Western University, London, ON, Canada

    4University Health Network, Toronto, ON, Canada

    5Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada

    Corresponding Author:

    Brian Lo, MHI

    Centre for Addiction and Mental Health

    100 Stokes Street

    Toronto, ON, M6J 1H4

    Canada

    Phone: 1 4165358501 ext 30635

    Email: brian.lo@camh.ca


    Background: Digital health tools such as mobile apps and patient portals continue to be embedded in clinical care pathways to enhance mental health care delivery and achieve the quintuple aim of improving patient experience, population health, care team well-being, health care costs, and equity. However, a key issue that has greatly hindered the value of these tools is the suboptimal user engagement by patients and families. With only a small fraction of users staying engaged over time, there is a great need to better understand the factors that influence user engagement with digital mental health tools in clinical care settings.

    Objective: This review aims to identify the factors relevant to user engagement with digital mental health tools in clinical care settings using a sociotechnical approach.

    Methods: A scoping review methodology was used to identify the relevant factors from the literature. Five academic databases (MEDLINE, Embase, CINAHL, Web of Science, and PsycINFO) were searched to identify pertinent articles using key terms related to user engagement, mental health, and digital health tools. The abstracts were screened independently by 2 reviewers, and data were extracted using a standardized data extraction form. Articles were included if the digital mental health tool had at least 1 patient-facing component and 1 clinician-facing component, and at least one of the objectives of the article was to examine user engagement with the tool. An established sociotechnical framework developed by Sittig and Singh was used to inform the mapping and analysis of the factors.

    Results: The database search identified 136 articles for inclusion in the analysis. Of these 136 articles, 84 (61.8%) were published in the last 5 years, 47 (34.6%) were from the United States, and 23 (16.9%) were from the United Kingdom. With regard to examining user engagement, the majority of the articles (95/136, 69.9%) used a qualitative approach to understand engagement. From these articles, 26 factors were identified across 7 categories of the established sociotechnical framework. These ranged from technology-focused factors (eg, the modality of the tool) and the clinical environment (eg, alignment with clinical workflows) to system-level issues (eg, reimbursement for physician use of the digital tool with patients).

    Conclusions: On the basis of the factors identified in this review, we have uncovered how the tool, individuals, the clinical environment, and the health system may influence user engagement with digital mental health tools for clinical care. Future work should focus on validating and identifying a core set of essential factors for user engagement with digital mental health tools in clinical care environments. Moreover, exploring strategies for improving user engagement through these factors would be useful for health care leaders and clinicians interested in using digital health tools in care.

    J Med Internet Res 2025;27:e67820

    doi:10.2196/67820

    Keywords

    user engagement; nursing informatics; clinical informatics; mental health; clinical care settings 


    Background

    Digital mental health tools continue to have a critical role in advancing the delivery of mental health care in clinical care settings [1-4]. In 2020, the World Health Organization released a global strategy on digital health innovation [5], which advocated for the opportunities and need to advance the implementation of digital health strategies and people-centric health systems fueled by digital health tools. In the United States, the American Medical Association [6] found that physician adoption of telehealth during the COVID-19 pandemic grew 5-fold, and the use of remote monitoring tools doubled due to improvements in patient outcomes and work efficiency.

    As such, many organizations have advocated for the use of digital health tools in mental health care delivery [7-9]; for example, the American Psychiatric Association has released a set of tool kits designed to support telepsychiatry and clinician recommendation of digital tools in the clinical setting [10], while the Mental Health Commission of Canada has released a number of tool kits to support adoption within Canada, including one for implementing digital mental health innovations [4]. There are also some frameworks for characterizing various digital mental health tools and use cases where digital tools can be helpful to support care [4]. Finally, the National Health Service in England has released a mental health digital playbook that focuses on outlining clinical care pathways that embed the use of digital mental health tools, as well as the associated governance, policies and change management [11]. This playbook, among others, has led to the uptake of many digital mental health tools across health systems, including Big White Wall (now known as Togetherall) in Canada and the United Kingdom [12-14] and the Digital Opportunities for Outcomes in Recovery Services program in the United States [15-19].

    However, there continues to be limited evidence on the outcomes and impact of digital mental health tools within clinical care settings in real-world environments [20]. A recent systematic review synthesized findings from 19 trials on digital mental health apps and determined that there is inconclusive evidence to suggest that these tools can be recommended as stand-alone interventions [21]. Many challenges have hindered the effective adoption and utility of the tools. While some are related to user-centered design and the adaptation of evidence-based principles and content within the platform, an emerging area of concern is suboptimal engagement with the tool by patients and families [22]. Several studies have found that continued use of the tool rapidly decreased after initial use [22-24]. The results of a systematic review that looked at user engagement for 7 apps for depression and anxiety showed that <42% of users stayed engaged and continued to use the tool beyond 4 weeks [25]. To ensure that the expected benefits are realized for the end user, it is essential that there is sufficient user engagement where appropriate and necessary.

    User Engagement

    User engagement has been a growing area of interest over the last few years, given the proliferation of tools and the recognition of this gap in the 2010s [22]. As a result, there have been numerous conceptualizations and characterizations of user engagement; for example, in 2008, O’Brien and Toms [26] characterized user engagement as a process that involves engagement and disengagement, and Perski et al [27] developed an integrative definition: “(1) the extent (e.g. amount, frequency, duration, depth) of usage and (2) a subjective experience characterised by attention, interest and affect.” Some work has also been conducted to characterize user engagement and to help visualize user engagement through use log data, Pham et al [28] developed a framework of metrics, while Pham et al [29] built an analytics platform. Likewise, Yaeger et al [30] looked at factors related to user engagement with a trauma recovery eHealth intervention using the Health Action Process Approach, while MacPhail et al [31] used the Health Action Process Approach to explore health behavior engagement in individuals with type 2 diabetes mellitus. Several researchers have since attempted to uncover factors related to user engagement in the context of facilitators and barriers [32], as well as neuropsychological [33] and persuasive design [34] frameworks. This process has resulted in a myriad of factors, from gamification and technical issues to personalization [32-34], alongside studies evaluating the impact of these interventions on boosting engagement [35].

    However, to date, there have been very limited discussions on the factors that influence user engagement with digital tools used specifically in clinical care models and settings. While the majority of tools being developed are focused on self-help in the community [21], there is growing demand and interest from the clinical community in implementing digital tools in clinical care; for example, the Stepped Care Model 2.0 offered by the Mental Health Commission of Canada highlights how tools can be used to augment care being delivered for individuals across various care levels [36]. Given that the integration of digital tools requires careful consideration of the environment and the broader health system, there is a timely need to look at the factors relevant to user engagement with digital mental health tools used specifically in clinical care pathways and delivery.

    Sociotechnical Frameworks

    One approach to addressing this issue is the application of a sociotechnical framework, which allows for the characterization and examination of complex environments to support the adoption and use of digital tools in clinical care settings [37]. In particular, it encourages the researcher to look at the interactions across components at the microlevel (eg, individual), mesolevel (eg, organizational), and macrolevel (eg, health system). The majority of digital health research has focused on factors related to end users or the tool itself [24]; as a result, it has yielded limited value in terms of understanding user engagement within the complex clinical environment. Applying a sociotechnical framework can help highlight the processes and workflows of the clinical environment as well as the broader policies of the organization and health system [38,39].

    Frameworks that have been used to look at innovations in health care include the nonadoption, abandonment, scale-up, spread, and sustainability framework [40] and the sociotechnical framework developed by Singh and Sittig [41]. Greenhalgh et al [40] developed the nonadoption, abandonment, scale-up, spread, and sustainability framework, which focuses on examining features across the condition, technology, adopters, and organization, among others [40]. Similarly, the sociotechnical framework developed by Singh and Sittig [41] outlines 8 components that are focused across the micro-, meso-, and macrolevel factors. These eight components include (1) hardware and software computing infrastructure; (2) clinical content; (3) human-computer interface; (4) people; (5) workflow and communication; (6) internal organizational policies, procedures, and culture; (7) external rules, regulations, and pressures; and (8) system measurement and monitoring. Both frameworks have been used to examine various clinical innovations such as virtual care [38] and issues related to ransomware attacks [42]. In particular, they have been used to look at factors influencing the adoption of artificial intelligence in Canadian health care [43] and clinical handoff tools [44].

    In this regard, this study aims to develop a comprehensive understanding of the factors related to user engagement with digital mental health tools in clinical care settings through the use of the sociotechnical framework developed by Singh and Sittig [41]. Obtaining a snapshot of the current evidence can help identify the current gaps in literature and inform the development of a comprehensive framework for assessing user engagement with digital mental health tools in clinical care contexts.


    Overview

    To identify the factors that influence user engagement with digital mental health tools in clinical care settings, we used a scoping review approach [45]. Given the exploratory nature and understanding of the factors related to user engagement, this approach was considered appropriate to identify a preliminary set of factors for further exploration [42]. The approach outlined by Arksey and O’Malley [45] and later refined by Levac et al [46] and Peters et al [47] was used. One of the authors (SK), a patient partner, was engaged in the development, implementation, and analysis of this scoping review.

    Step 1: Identify the Research Questions and Objectives

    The research questions (RQs) of the scoping review are as follows:

    • RQ1: Of the digital mental health tools being used in clinical care settings, what are the types of technologies (eg, mobile app and wearable) and functionalities used to deliver digital mental health care?
    • RQ2: What are the characteristics of the populations that are using digital mental health tools as part of clinical care?
    • RQ3: What are the sociotechnical factors that influence user engagement with digital mental health tools in clinical care environments over time?
    • RQ4: What are the characteristics of clinical programs that embed digital mental health tools as a component of care?

    We used the sociotechnical framework developed by Sittig and Singh [41] to guide the synthesis of factors that influence user engagement with digital mental health tools in clinical care environments. This ensured a comprehensive overview of the factors across individual, organizational, and health system levels.

    Step 2: Search Strategy Creation

    To identify articles on user engagement with digital mental health tools in clinical care contexts, a systematic search strategy was developed based on previous search strategies [32,48]. Relevant Medical Subject Headings (MeSH) terms and keywords related to user engagement, mental health, and digital health tools were applied across databases (Multimedia Appendix 1). The search was conducted on 5 databases—MEDLINE, Embase, PsycINFO, CINAHL, and Web of Science—in January 2022 without restrictions on date or study type. These databases are popular among health sciences researchers and were expected to index most of the literature published in this field. The search strategy was first developed in MEDLINE and was adapted to other databases. A research librarian was also consulted in the refinement of the search strategy. The search strategy was validated by confirming whether previously identified relevant articles (eg, the study by Hoffman et al [17]) were included in the search. The search was updated in October 2023 using the same strategy and approach.

    Step 3: Selection of Studies

    The inclusion and exclusion criteria for the scoping review are outlined in Textbox 1. Eligible studies must examine a digital health tool (eg, mobile app, patient portal, or wearable) that primarily addresses a mental health or addiction issue. To ensure relevance to clinical care, the tool must include at least 1 patient-facing component (eg, app) and 1 clinician-facing component (eg, dashboard) as specified in the Mental Health Commission of Canada Toolkit for E-Mental Health Implementation [4]. Studies must explore the concept of user engagement as an objective, following the aforementioned definition by Perski et al [27]. Articles published in languages other than English were excluded for feasibility. Systematic and literature reviews were excluded, but their reference lists were examined. Non–peer-reviewed article types, such as theses and conference presentations, were also excluded.

    Textbox 1. Inclusion and exclusion criteria for the scoping review.

    Inclusion criteria

    • Digital health tool must primarily address a mental health or addiction issue
    • Digital health tool must include at least 1 patient-facing component and 1 clinician-facing component [5]
    • Study must examine user engagement as per the definition by Perski et al [27]

    Exclusion criteria

    • Digital health tool is used by the patients themselves (eg, a self-help tool) and does not contain a clinician-facing component
    • Article is not in English
    • Study does not examine user engagement as an objective

    Study selection was conducted by the first author (BL) in duplicate with 2 doctoral students (KD and RC) in health informatics using Covidence (Veritas Health Innovation Ltd). Deduplication was performed by identifying records with identical titles and publication years, with verification by a member of the project team (BL). Screening was carried out in 2 stages: an initial title and abstract review, followed by full-text screening of studies meeting the inclusion criteria. To ensure consistency, a pilot screening (n=100) of titles and abstracts was conducted with each of the two doctoral students (KD and RC). Any discrepancies were discussed and resolved by the 3 reviewers, and Cohen κ was used to assess interrater reliability [49]. A Cohen κ value of >0.70 was achieved in the pilot, after which we proceeded with the screening process. A similar process was followed for full-text screening, in that small pilot rounds were conducted by 2 doctoral students, and any discrepancies were discussed and resolved by the 3 reviewers.

    Step 4: Extracting and Charting the Data

    The elements extracted for each RQ are outlined in Textbox 2. The article type, year of publication, and study objective were collected to understand the characteristics of the included articles. For RQ1, relevant information about the digital mental health tool, such as technology type, objective, and main features, were extracted. For RQ2—identifying the characteristics of the population—we collected information about the demographic characteristics of the users and access requirements for the tool. For RQ3, factors relevant to user engagement with digital mental health tools in clinical care contexts were extracted. Finally, for RQ4, information about the objective and the digital health delivery model was collected. KD and RC conducted a pilot extraction of 5 to 10 articles, and the extracted data were compared. On the basis of the feedback, the extraction table was iteratively refined (eg, by adding relevant elements).

    Textbox 2. Data elements extracted (including for each research question [RQ]).

    Characteristics of included articles

    • Year of publication
    • Article type
    • Country of publication
    • Type of user engagement examined (subjective vs objective)
    • Main objective of paper
    • Methodology (eg, study design and type of measurements and instruments used)

    RQ1: Type of digital mental health tool used

    • Name of digital health tool
    • Technology used for digital mental health tool
    • Main objective of digital health tool
    • Patient- and clinician-facing functionalities of the tool as per the Mental Health Commission of Canada Toolkit for E-Mental Health Implementation [4]

    RQ2: Population using digital mental health tool

    • Participant population
    • Demographics of the participants in the study (eg, mental health condition)
    • Duration of participation

    RQ3: Factors that influence user engagement with digital mental health tools

    • Factors relevant to user engagement with digital mental health tools in clinical care contexts

    RQ4: Clinical digital health delivery model

    • Objective of the treatment provided as part of the clinical care model
    • Delivery model of the program

    Step 5: Collating, Summarizing, and Reporting the Data

    Both quantitative and qualitative approaches were used to summarize and analyze the extracted data. Descriptive statistics (eg, mean and median) were used to analyze the characteristics of included studies. Descriptive statistics were also used to analyze article and population characteristics for RQ1 and RQ2. A thematic analysis [50] was conducted to categorize the factors from the literature that influence user engagement with digital mental health tools (RQ3) using the sociotechnical framework developed by Sittig and Singh [41]. A thematic analysis was also conducted to characterize the clinical digital health programs that were identified (RQ4). The findings were reported following the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) checklist [51] (Multimedia Appendix 2).


    Overview

    Study selection results were reported using the PRISMA-ScR diagram [51] (Figure 1). After deduplication, the search strategy yielded 11,503 records for title and abstract screening. Of the 417 articles reviewed for eligibility, 136 (32.6%) were included for analysis. Table 1 summarizes the characteristics of the included articles, and Multimedia Appendix 3 [52-177] provides details of the included articles.

    ‎
    Figure 1. PRISMA-ScR flow diagram.
    Table 1. Characteristics of included articles (n=136).
    CharacteristicsArticles, n (%)References
    Year of publication

    		20061 (0.7)[52]

    		20081 (0.7)[53]

    		20101 (0.7)[54]

    		20113 (2.2)[55-57]

    		20122 (1.5)[58,59]

    		20135 (3.7)[60-64]

    		20145 (3.7)[65-69]

    		20156 (4.4)[70-75]

    		20167 (5.1)[76-82]

    		201712 (8.8)[83-93,177]

    		201812 (8.8)[94-105]

    		201913 (9.6)[106-118]

    		202017 (12)[119,121-136]

    		202121 (15.4)[137,138,144-157]

    		202216 (11.8)[120,158-162,178-187]

    		202314 (10.3)[163-176]
    Country of publication

    		Australia11 (8.1)[54,59,69,110,118,133,165,168,177,178,188]

    		Canada11 (8.1)[3,73,85,93,116,129,132,148,171,172,174]

    		Chile2 (1.5)[81,123]

    		China1 (0.7)[128]

    		Denmark9 (6.6)[86,101-103,124,141,146,161,175]

    		Finland2 (1.5)[72,104]

    		Germany7 (5.1)[122,131,136,154,187,189,190]

    		Hungary1 (0.7)[56]

    		Ireland2 (1.5)[121,155]

    		Italy1 (0.7)[162]

    		Netherlands4 (2.9)[67,109,111,124]

    		Norway5 (3.7)[53,63,65,83,173]

    		New Zealand1 (0.7)[64]

    		Spain5 (3.7)[90,100,119,163,169]

    		Sweden4 (2.9)[74,79,80,184]

    		Switzerland1 (0.7)[70]

    		United Kingdom23 (16.9)[52,62,66,75,77,89,92,126,127,135,138,142,144,145,147,151,152,160,166,167,176,180,182]

    		United States47 (34.6)[55,57,58,60,61,68,71,76,78,82,84,87,88,91,94-96,98,105-108,112-114,117, 120,125,130,137,139,140,143,150,153,156-159,164,170,179,181,183,185,186]
    Study design

    		Case study6 (4.4)[102,118,135,150,186,191]

    		Cohort study35 (25.7)[55,57,59,61,62,68,69,73,78,81,95,96,100,105,112,114,116,117,120,123, 128,130,136,149,152,154,158,159,162,164,169-171,174,177]

    		Cross-sectional study1 (0.7)[94]

    		Feasibility or pilot study14 (10.3)[52,56,66,77,84,87,98,99,119,122,131,139,140,157]

    		Mixed methods evaluation11 (8.1)[75,82,83,115,124,142,143,166,167,175,185]

    		Observational study2 (1.5)[86,179]

    		Qualitative study43 (31.6)[53,54,63,65,85,88-90,92,93,101,103,104,107,110,111,113,132-134,137,138, 141,144-146,151,153,155,156,161,163,165,168,172,173,176,178,180,182-184,187]

    		Randomized controlled trial19 (14)[64,67,70-72,76,79,80,91,97,106,109,121,126,129,147,148,160,181]

    		Retrospective study2 (1.5)[74,125]

    		Usability study3 (2.2)[60,108,127]
    Perspective on user engagement (multiselect)

    		Qualitative95 (69.9)[53,54,57,58,60,62,63,65,66,72,73,75,77,79,81-83,85,88-90,92, 93,95,96,100-103,105,107,108,110,111,113,115,116,120-124,126,127,129-134,137, 138,141-149,151-158,160-163,165-173,175-178,180-187]

    		Quantitative55 (40.4) [52,54,55,59,61,64,66-69,71,72,74-76,80,83,84,91,94-98,106,109, 112,114,116-119,121,122,124-126,128,129,135-137,143,150,159,164,166,167,170,171, 174,175,179,185,186]

    In terms of the year of publication, 59.6% (81) of the 136 articles were published within the last 5 years. Publications on this topic originated from 18 countries. The top 4 countries were the United States (n=47, 34.6%), the United Kingdom (n=23, 16.9%), Australia (n=11, 8.1%), and Canada (n=11, 8.1%). A wide variety of study designs were used by the included articles, including cohort studies (n=35, 25.7%), feasibility or pilot studies (n=14, 10.3%), qualitative studies (n=43, 31.6%), and randomized controlled trials (19/136, 14%). With regard to how user engagement was examined, the majority of articles focused on user experience and interest (qualitative; 95/136, 69.9%) as opposed to the degree of use (quantitative; 55/136, 40.4%).

    RQ1: Digital Tools Used in Clinical Care Settings

    Of the 136 included articles, 113 (83.1%) disclosed the product or name the tool that was studied (Table 2). Commonly discussed tools in the included articles were FOCUS [60,68,96,113,117] and Apps4Intelligence [116,185], mobile apps for people with schizophrenia; Intellicare, a platform that focuses on navigation and recommendation of digital tools [84,106]; and Horyzons, a social therapy platform for first-episode psychosis [133,134]. In terms of tool type, the majority were websites (62/136, 45.6%) and mobile apps (49/136, 36%). Notably, 6 (10%) of the 62 websites were patient portals from health care organizations [132,158,159,171,173]. From a client-facing perspective, these digital tools offered a wide range of functionalities (Table 2). Specifically, among computerized interventions, resources, and applications (111/136, 81.6%), some examples include tools that were focused on delivering cognitive behavioral therapy [57,59,63-65,74,79,80,89,91,98,102,111,128,184] and educational workbooks or resources [54,56,66,123,125] for patients to complete. Regarding clinician involvement in the support and use of the tool, most provided support in a coaching (49/136, 36%) or comprehensive (53/136, 39%) manner [4]; for example, for the FOCUS app and Intellicare platform, clinicians would guide users on the use of the tool and develop a plan around how the tool should be used throughout the care journey [84,106,117]. Other tools, such as the BRAVE platform, required clinicians to use it in a more comprehensive manner to review results and to discuss and plan next steps in care through and with the platform [118].

    Table 2. Characteristics of digital tools examined in the included articles (n=136).
    CharacteristicsArticles, n (%)References
    Type of digital health tool (multiselect)

    		Chat group5 (3.7)[56,78,91,139,148]

    		Computer software2 (1.5)[75,187]

    		Mobile app49 (36)[60,62,68,84,88,92,93,95,96,101,103,105-107,109,113,116,117,119,120,126, 131,135,137,138,140,142,145,146,152,153,156,157,160-164,166,167,169,170,175,176, 179-182,185,186]

    		SMS text messaging or texting12 (8.8)[52,53,57,67,69,70,72,77,87,108,147,150]

    		Website62 (45.6)[54,56,58,59,61,63-66,71,73,74,79-83,86,89,90,94,97-100,102,104, 110-112,114,115,118,121-125,127-130,132-134,136,139,141,143,144,149,151,154,155, 158,159,168,171,173,177,183,184]

    		Telephone software2 (1.5)[76,172]

    		Wearable and virtual reality11 (8.1)[85,162,165-167,169,170,174,176,178,181]
    Functionality of digital health tool (multiselect)

    		Big data2 (1.5)[75,161]

    		Computerized interventions, resources, and applications111 (81.6)[54,56,57,59-70,74,75,77,79-84,86-90,92-94,96-118,121-136,138-146, 148-160,162-164,166,168-171,173,175,176,178,179,181-186]

    		Peer support11 (8.1)[56,58,83,93,107,122,133,139,148,149,185]

    		Telehealth and telemedicine54 (39.7)[52,53,55,57,64,67,68,70-73,75,76,78,81,83,88,91,94-96,98,105, 107,113,116,117,119-123,132,134,137,146,147,150,153,156-159,162,166,167,169,170, 172,176,177,180,183,185,186]

    		Virtual reality3 (2.2)[141,165,187]

    		Wearable computing and monitoring9 (6.6)[85,92,116,119,139,166,174,178,181]
    Level of clinician involvement

    		Promotion7 (5.1)[52,59,72,86,87,125,139]

    		Case management5 (3.7)[73,75,107,134,148]

    		Coaching49 (36)[57,61,64,65,67-71,74,79-82,84,89,90,97,99,100,104-106,109,110,112, 113,115,116,126,128-130,138,140-146,149-151,153,155-157,163]

    		Symptom focused22 (16.1)[54,63,66,91,92,94,95,101-103,108,121,122,124,131,133,136,137,147,152,159, 164,177,182,184]

    		Comprehensive53 (39)[53,55,56,58,60,62,76-78,83,85,88,93,96,98,111,114,117-120,123,127, 132,135,138,154,158,160-162,165-176,178-181,183,185-187,192]

    RQ2: Characteristics of Populations Using Digital Mental Health Tools in Clinical Care Settings

    With regard to the population of end users (Table 3), the majority of articles (101/136, 74.3%) examined tools that were developed for the adult population. Only a few articles (12/136, 8.8%) had a focus on younger adults (aged <18 years), while the study by Sheeran et al [55] evaluated a depression management program for individuals aged >65 years receiving home care services. Moreover, the target conditions of the tools varied significantly, with a little more than one-third of the articles (47/136, 34.6%) examining tools that supported >1 type of mental health condition. Regarding condition-specific tools, the most frequent conditions examined were depression (26/136, 19.1%) and schizophrenia spectrum and related psychotic disorders (20/136, 14.7%). Finally, regarding the duration of participation, 52.9% (72/136) of the studies reported how long participants were asked to use the tool. Of these 72 studies, 42 (58%) asked participants to use the tool for <3 months.

    Table 3. Characteristics of populations (patients) examined in the included articles (n=136).
    CharacteristicsArticles, n (%)References
    Age range (years)

    		<1812 (8.8)[77,99,104,107,118,120,133,139,144,149,151]

    		≥18101 (74.3)[53-55,57,58,60-64,66-69,72-76,78-87,90-98,100,105,106,108-117,119, 121-123,125-128,130,134-138,140-143,146,147,150,152,153,155,157-166,169,171-176, 178-187,192]

    		Mixed12 (8.8)[52,56,59,101,103,129,138,145,148,167,168,170,177]

    		Not reported11 (8.1)[65,70,71,88,89,102,124,131,132,154,156,158]
    Mental health condition

    		Anxiety disorders11 (8.1)[64,74,98-100,118,128,130,141,144,161]

    		Bipolar and related disorders7 (5.1)[54,76,78,92,121,157,174]

    		Depressive disorders26 (19.1)[55,57,61,63,65-67,81,85,86,97,109,111,112,115,125,127,131,138,142,149, 153,154,163,176,179,183]

    		Feeding and eating disorders7 (5.1)[52,56,58,80,82,91,101]

    		General and undifferentiated mental illness47 (34.6)[59,60,69,73,75,77,79,83,84,90,93,94,96,102,104-106,114,119,120,122, 124,129,132,134,135,137,140,147,148,150,155,158,159,164,166,168,170,171,173,177, 178,180,181,184,186,187]

    		Neurodevelopmental disorders3 (2.2)[71,139,151]

    		Obsessive-compulsive and related disorders2 (1.5)[89,165]

    		Personality disorders1 (0.7)[182]

    		Schizophrenia spectrum and related disorders20 (14.7)[60,62,68,72,87,96,103,108,116,117,126,133,138,143,145,146,152,162,169,185]

    		Substance-related and addictive disorders6 (4.4)[70,95,107,110,172,175]

    		Trauma and stress-related disorders4 (2.9)[88,123,136,160]
    Duration of participation

    		<6 wk12 (8.8)[22,55,62,86,87,120,142,158,162,164,185,186]

    		6 wk to 3 mo30 (22.6)[22,57,78,82,83,85,97,99,100,112,113,119,121,123,128,131,138,140,144,148-150, 152,160,161,163,167,169,170,184]

    		4 mo to 1 y27 (19.9)[52-54,56,61,64,66,69,70,73,76,77,80,92,95,98,101,117,126,143,146, 147,166,171,174,181,182]

    		>1 y3 (2.2)[74,133,176]

    		Not applicable or not reported64 (47.1)[58-60,63,65,67,68,71,72,75,79,81,84,88-94,96,102-112,114-116,118, 122,124,125,127-132,134-139,141,145,151,153,154,156-159,165,168,172,173,175, 177-180,183,187]

    RQ3: Factors That Influence User Engagement With Digital Mental Health Tools

    Overview

    Among the included articles, 26 factors were identified across 7 (88%) of the 8 categories outlined in the sociotechnical framework developed by Sittig and Singh [41]. The categories with the most number of factors were human-computer interface (5/26, 19%) and people (5/26, 19%). No factors were coded in the system measurement and monitoring component of the framework because this was considered an outcome of user engagement (eg, user analytics) [41]. Table 4 presents an overview of the factors and their descriptions; further details are provided in the subsections that follow.

    Table 4. Overview of the factors related to user engagement with digital mental health tools in clinical care settings.
    FactorsDescriptionsReferences
    Hardware and software computing infrastructure

    		Modality of the toolAbility to effectively use the tool across multiplatform systems to ensure compatibility with end-user devices[53,60,63,67,70,76,85,88,90,98,110,120,133,137,138,141-143,147,148, 168,169,176,180]

    		Access to the toolEase of access to the technology outside of the clinical setting[58,71,88,111,132,134,143,150,153,167,168,172,180]

    		Technical challengesTechnical glitches and issues affecting the full use of the tool[55,56,75,81,122,129,131,138,144,169,176]
    Clinical content

    		Personalization of the contentTailored delivery of questions and feedback based on the preferences and current status of the individual[63,64,78,81-83,92,93,103,105,108,113,123,127,131,135,140,142,144,145, 147,149,155,157,158,163,177,178,184]

    		Delivery of the contentDelivery of content using the right language, length, medium, and tone[60,63,82,100,111,115,124,127,130,138,142,146,158,168,183,184]

    		Spans across the patient journeyTools that adapt to the evolving needs of the individual across the care journey[54,110,133,142,147,152,158,177]

    		Appropriate follow-up and user interactionResponse and feedback are actionable, trauma informed, and informed by lived experiences[82,92,103,108,113,129,130,132,134,138,145,152,153,156]
    Human-computer interface

    		Preferred modality of content deliveryAvailability and alignment of content delivery to end-user preferences (eg, audiobook or video)[53,55,57,60,62,71,75,82,90,92,96,105,108,132,138,140,141,144,146,155,184,187]

    		Desired interaction durationExpected duration and resources needed to use the tool[64,75,90,108,110,115,118,119,121,150,162,167,168,170,182]

    		UsabilityHow easy or difficult it is to complete a task[82,89,92,103,107,110,111,116,131,132,134,138-140,143,152,162,163,167,175-178,181,183]

    		Feedback and incentivizationProviding immediate feedback and rewards for responses and routine use of the tool[52,57,58,70,85,92,101,103,127,135,142,145,147,157,158,161,162,178,180,186]

    		Interoperability with other platformsBidirectional connectivity to other platforms and tools (eg, a calendar app)[73,154]
    People

    		AffordabilityCosts to use the tool[56,110,111,132,142,145,153,167,168]

    		Presence of clinician supportActive support from clinicians to encourage use of the tool[56,66,75,88,111,115,123,126,135,145,150,151,153,158,160,161,163,165, 168,171,175,176,180-182,184,187]

    		Presence of support from family and friendsActive support from family and friends to encourage use of the tool[56,66,82,83,85,104,110,121,139,140,150,155,156,181]

    		Clinical condition and critical eventsThe impact of the mental health condition and related events (eg, suicide attempt or loss of home)[53,54,59,63,65,69,71,73-75,80-82,85-87,92,93,100,101, 103-106,108,114,116,119,120,122,123,127,129-133,135,136,145,146,148, 149,153,155,156,158,162,165,167,168,170,174,176,179,182-187]

    		Sociodemographic characteristicsImpact related to the characteristics of the population such as location, ethnicity, and age[54,57,76,84,91,94,97,112,117,128,136,148,153,154,159,164,166,168,174,179]
    Workflow and communication

    		Incorporation into care workflowsEmbeddedness of the tool into the daily practice and processes of clinical care[53,54,63,78,82,88,89,92,93,96,99,101-105,107,108,111,115,120, 122,124,127,131,133,134,137,138,157,158,161,162,167,168,170,177,181,184]

    		Expectation settingExpectations for tool use as agreed upon or imposed by patients, families, and clinicians[53,63,65,67,68,74-77,79,82,83,89,103,105,108,110,111,113,120, 124,129,135,137,138,144,145,147,149,153,156,158,160,165,173,175,178,180]

    		Delivery of educationGuidance on use and interpreting data on the platform[83,139,143]

    		Support for useProviding peer and technical support to help end users identify opportunities for meaningful use of the tool[54,63,71,97,100,109,110,116,118,128,129,131,132,142,151,152,154-156,171,176,183]
    Internal organizational policies, procedures, and culture

    		Privacy and securitySafeguards, regulations, and policies in place to support privacy and security requirements from end users[82,96,105,126,134,135,145,152,172]

    		Management buy-inEndorsement and support to build capacity and change management for digital health from clinical and health system leaders[77,104,176]

    		Administrative burden on cliniciansThe administrative burden on clinicians when working with patients and families on the platform[75,83,95,111,115,121,127,134,137,142,158,165,168,175]
    External rules, regulations, and pressures

    		Guidelines for useEstablishment of expectations for tool use by patients, families, and clinicians (eg, liability)[83,89,95,101,102,111,158,161,165,172]

    		Health system infrastructure and reimbursementEndorsement and capacity building for patients, families, and clinicians to use the tool (eg, billing)[66,82,83,89,102,137,151,158,161,172,173,176]
    Hardware and Software Computing Infrastructure

    Within the hardware and software computing infrastructure category, three factors were identified: (1) modality of the tool, (2) access to the tool, and (3) technical challenges. Of the 136 included articles, 25 (18.4%) highlighted how the tool’s modality (eg, mobile app or website) influenced engagement; for example, participants in Big White Wall (now Togetherall), a peer support network for mental health service users, found that the lack of a mobile app was problematic for easy access to the platform when needed [148]. Others also experienced issues when trying to run the platform on different or older models of mobile devices (eg, Android or iPhone) [137,144]. Moreover, 13 (9.6%) of the 136 articles outlined the importance of considering how patients would be able to access the tool. Medalia et al [143] examined the feasibility and acceptability of a cognitive remediation platform, which allows patients with schizophrenia to complete some of their homework remotely in public settings or at home. While the intent of the platform was to reduce the need for attending a clinic multiple times per week, the authors found that system and bandwidth requirements made it difficult for individuals to use the platform effectively in public settings (eg, libraries). Finally, 11 (8.1%) of the 136 articles reported challenges related to bugs and limitations, which influenced the ability of patients to stay engaged, particularly in platforms still in early development [81,146].

    Clinical Content

    There were four factors related to how the clinical content on the platform influenced user engagement: (1) personalization of the content, (2) delivery of the content, (3) spans across the patient journey, and (4) appropriate follow-up and user interaction. For content personalization, 29 (21.3%) of the 136 articles highlighted the importance of tailoring the content to users’ needs and preferences to enhance engagement; for example, in remote mood tracking tools, it was identified that repeatedly using the same wording for questions often felt impersonal and repetitive, regardless of whether a user’s condition improved or deteriorated [81,145]. This issue also extended to the modules [138], educational materials and treatment [131,188], and messages delivered within the therapeutic components [144,188]. In addition, 16 (11.7%) of the 136 articles highlighted the need to consider how the content is delivered; for example, in the case of clinical notes within patient portals, there is a need to ensure patient-centric wording in condition descriptions to ensure that the documentation supports patient care [108,158] and aligns with a patient-centric approach [111,131,146].

    Furthermore, a few articles (8/136, 5.9%) looked at the importance of considering each patient’s treatment stage. As needs and acuity levels differ from admission to discharge, use cases and needs related to engagement will likely vary [193]; for example, participants using the Mindframe platform highlighted how it enabled them to understand how their care has evolved since they began treatment [103]. Valentine et al [133] also reported that patients at the beginning of treatment are less likely to transparently report their mood due to concerns about increasing the intensity of their care. Thus, adjusting the content and its delivery may be influenced by the patient journey. This also extends to the need for appropriate follow-up and user interaction such that the tool can generate bidirectional conversations between the tool and the patient (15/136, 11%). While providing feedback and reminders could be useful in promoting engagement over time, some participants reported feeling scrutinized and judged based on the responses they received when they were unable to complete an assigned task [138].

    Human-Computer Interface

    Five factors related to the human-computer interface of the platform were identified: (1) preferred modality of content delivery, (2) desired interaction duration, (3) usability, (4) feedback and incentivization, and (5) interoperability with other platforms. Building on the need to have various modalities for content delivery, 22 (16.2%) of the 136 articles outlined the need to allow patients to choose their preferred modality for receiving clinical content [110]; for example, a number of therapists in the study by Rodda et al [110] mentioned that while patient preferences can vary significantly, there is typically a strong preference for a specific modality. This also extends to the expected duration of patient interaction with the platform (15/136, 11%). In the same study, therapists found that overly long content and modules can be overwhelming, making it difficult for patients to complete them and stay engaged over time [110].

    Other key areas identified in the included articles were usability and interoperability with other tools. In particular, ease of use, navigation, and accessibility within the tool’s features were reported to be important contributors to user engagement (26/136, 19.1%). The lack of an easy-to-use interface can be detrimental to the utility and use of the tool; for example, in the addiction comprehensive health enhancement support system examined by Hussey and Flynn [107], the emergency call feature was placed in a prominent location for users to access when needed. However, because of its placement, it would be clicked accidentally even when help was not required.

    Finally, with the growth of gamification, 20 (14.7%) of the 136 articles looked at the potential benefits and risks of providing immediate feedback and incentives to enhance engagement. While Lindgreen et al [101] found that the delivery of feedback alongside reminders and nudges was considered useful in enhancing engagement, it was important to be considerate and trauma informed; for example, in their study on an eating disorder app, it was observed that simply sending reminders to eat could be perceived as condescending and hence detrimental if not delivered appropriately. Some of these reminders and enhancements should also be interoperable with other platforms that individuals currently use on their device (eg, a calendar app; 2/136, 1.5%).

    People

    Several factors related to patients as end users were also identified as influencing user engagement. These included (1) affordability, (2) sociodemographic characteristics, (3) presence of clinician support, (4) presence of support from family and friends, and (5) clinical condition and critical events.

    With regard to affordability and sociodemographic characteristics, a few articles (9/136, 6.7%) discussed the challenges related to having devices for use with the digital mental health tool. While studies frequently provided a digital device for use with the digital mental health tool, many individuals highlighted that once device and data access were no longer provided, engagement decreased drastically [95]. Other studies using quantitative data highlighted how sociodemographic characteristics such as age and ethnicity (21/136, 15.4%) have also been found to be related to the extent of use [128,189]. Thus, understanding how sociodemographic characteristics and affordability intersect with engagement for a particular population is critical.

    The presence of clinician support as well as support from family and friends was another key factor influencing engagement. Of the 136 articles, 27 (19.9%) spoke about the importance of having a clinician support and be engaged throughout the use of the digital mental health tool; for instance, participants in the study on 3 stepped care tools by March et al [118] highlighted the role of clinicians in the customization and tailoring of strategies and implementation approaches to ensure that the tool aligns with the needs of end users. In the event that nonadherence was observed, there was an opportunity to intervene and provide adequate support. Moreover, the presence of support from family members and friends was reported to enhance engagement by offering encouragement, time, and space for patients to use the tool (14/136, 10.2%). Nitsch et al [82] also shared that participants found it meaningful to do it (engage in the tool) for their family and friends. Finally, 61 (44.9%) of the 136 studies discussed the role of patients’ clinical condition and critical events; for example, some individuals receiving care for general anxiety disorder reported feeling too anxious to use a digital tool [64], whereas other studies found that those with anxiety had higher levels of engagement than individuals with depression [74]. In addition, some clinicians highlighted that individuals with psychosis and active suicidal ideations may not be suitable candidates with regard to relying on a digital system for support [153].

    Workflow and Communication

    Several factors related to workflow and communication were identified. One key aspect outlined by several articles (40/136, 29.4%) was the tool’s ability to be incorporated into clinical care delivery. This can take various forms, such as enabling working together on worksheets through the platform [127] or providing advance information before a clinical visit [158]. In the case of a monitoring app for eating disorder, the lack of discussion regarding the information provided by clients during each session led to discouragement and a loss of trust in the clinician [101]. Thus, there is a need to set clear expectations (39/136, 28.7%) with clients about the appropriate and effective use of the tool as part of clinical care settings. Some individuals used the platform as a means to seek support after hours [113], while others held perceptions that the tool would take over the therapist’s role [151].

    Other factors within this category included the need to deliver education and support to patients (3/136, 2.2%), as well as provide adequate support throughout the duration of tool use to encourage engagement (22/136, 16.2%). The study by Morrison et al [66] highlighted the importance of having a care manager to provide guidance on the appropriate and meaningful use of the tool in the overall care.

    Internal Organizational Policies, Procedures, and Culture

    An emerging number of studies have highlighted the impact of internal organizational policies, procedures, and culture on user engagement. Foremost, privacy and security are considered pinnacle issues for patients (9/136, 6.7%). As these tools often collect, use, and analyze personal health information, patients have expressed concerns about how their data are collected and whether the data are safe and secure [82]. Moreover, some clients have queried whether the data would be anonymized or whether access would be limited to their care team [105]. Thus, clearly communicating and outlining privacy and security considerations with regard to the tool would be important for fostering engagement and encouraging users to enter private, sensitive information over time.

    In addition, buy-in and expectations from organizational administration and leadership influenced user engagement (3/136, 2.2%). Kurki et al [104] discussed how the expectations and accountability placed on clinicians affected their ability to encourage clients to engage with the tool as part of clinical care. This is particularly important when the app includes content related to suicide and self-harm because it can impact how clinicians should be responding to these incidents in a timely manner [95,107]. Administrative burden was raised as another issue for clinicians (15/136, 11.0%). While these studies outlined the administrative burden related to clients completing repetitive surveys [119], the workload for clinicians was cited as a critical barrier to reviewing and making meaningful use of the data for care [121]. Without protected time and adequate alignment and expectations regarding the time and effort required to use the tool, the workload often became a barrier to continued engagement over time.

    External Rules, Regulations, and Pressures

    Two factors related to the external rules, regulations, and pressures of the health system were identified. The first involves guidelines for tool use as prescribed by regulatory bodies and professional colleges (10/136, 7.3%). Folker et al [102] discussed the need to push system-level policies that support the uptake and adoption of digital mental health tools such that there is overall encouragement and support for the use of these emerging approaches in care. The other factor, highlighted by 12 (8.8%) of the 136 articles, is the financial reimbursement of clinicians for using these tools [115]. Given the administrative burden associated with the use of these tools, there is a strong need for clinicians to be reimbursed for the time spent in using these tools for care. It may also be useful to account for this workload within clinicians’ daily responsibilities.

    RQ4: Clinical Programs That Embed Digital Mental Health Tools

    Some of the studies (20/136, 14.8%) provided a brief description of how digital mental health tools can be embedded within clinical programs [53,55,66,75,104,109,113,120,121,129, 143,146,154,155,160,163,168,177,178,182]; for example, Kemmeren et al [109] outlined a clinical workflow specifying when these tools should be introduced and used with patients, as well as the intended duration and use of the various modules within the blended cognitive behavioral therapy tool. In addition, these articles described training for clinicians on using the tool with patients and provided guidance on how to navigate and support patients in its use [55,104]. For the FOCUS app [113], information was provided on the frequency and approach for checking in with clients regarding the use of the digital tool.


    Principal Findings

    To our knowledge, this is one of the first reviews to focus on identifying a preliminary, comprehensive framework of factors that influence user engagement with digital mental health tools in clinical care settings [24]. In this review, 26 factors were identified from 136 articles that spanned across the components of the sociotechnical framework developed by Sittig and Singh [41]. These factors illustrate how technology, relevant stakeholders, and environment can influence user engagement with digital mental health tools. Moreover, based on the studies that examined the clinical programs that embed digital mental health tools, there is increasing discussion on how these tools can be delivered in real-world environments.

    As there are a growing number of studies looking at ways to address the ongoing challenges of user engagement [194-196], this work has contributed to a better understanding of the dynamics of, and contributors to, effective user engagement in a clinical environment across individual, organizational, and health system levels [37]. Given that the focus of this scoping review was on digital mental health tools in clinical care environments, this work will help inform the development of implementation approaches and strategies for ensuring effective engagement and integration of digital mental health tools in clinical care settings and models of care. In addition, as digital tools continue to proliferate in the clinical environment, engagement with these tools has become an extension of how patients engage with clinicians, receive care, and navigate the clinical environment [24,33,197]. Hence, engagement with digital tools can be a key factor in the overall success of treatment. The specific impact of this work on several key areas of user engagement is discussed in the following paragraphs.

    First, as outlined in the Introduction section, a number of existing frameworks have explored the factors that influence user engagement with digital tools [27,32]. In this work, while many of the identified factors are consistent with the findings of previous reviews, the application of the sociotechnical framework developed by Sittig and Singh [41] was useful in 2 ways. Given the overarching goal of developing interventions to enhance effective user engagement, the sociotechnical framework helped to further break down the factors related to user engagement by focusing on digital mental health tools in clinical care settings; for example, technology-related factors were further characterized into factors that pertain to the content, the technology itself, and the human-computer interface [41]. Second, rather than providing a “laundry list” of factors within the categories, the sociotechnical framework helped to clarify and conceptualize the interplay of factors across its components. Thus, this work extends the broad scope by providing more specificity on how some of the factors would interact with each other through a sociotechnical lens. However, there were also challenges in applying the sociotechnical framework. Given that many of the factors are closely related, it was difficult to articulate the interconnectedness and interactions across the various components of the framework. Future work should further explore these nuances, including how these factors are connected and how factors specific to different end users (eg, patients and clinicians) interact.

    The findings from this work also provide more insights into factors specific to the discipline and the implementation setting of these tools. Borghouts et al [32] conducted a similar scoping review but focused more broadly on digital mental health tools and identified 16 factors that span across the user, program, technology, and environment. While many of these factors align with those in this review, the relatively narrower focus of our review uncovered 2 additional facets specific to tools integrated within the clinical environment. First, the importance of support from clinicians was identified as a critical component. While many studies have looked at how clinician-guided approaches can help improve engagement with digital mental health tools [198], there remains a lack of guidance and best practices on how best to support patients throughout their care journey [138]. As such, it may be useful to develop best practices that is informed by a customer experience approach through user experience perspectives [199].

    Moreover, as the tool will be embedded in the health care organization, it is critical to consider the workflows and internal processes. Establishing clear guidelines on the accountability and role of each member of the circle of care in the use of digital tools will ensure that end users understand what constitutes appropriate, safe, and effective use [20]; for example, for tools that are not actively monitored after hours, it is critical to reiterate through policy that patients and clinicians understand that the tool is not to be used during emergency crises [108]. Another important aspect of workflow and organizational policies is ensuring protected time for clinicians to use digital tools with patients. In the Canadian context, where physicians rely on a fee-for-service model, there is a need to establish remuneration policies so that clinicians can be compensated for the time and administrative burden associated with the use of these tools [115]. With clinician burnout related to digital tool use becoming a tremendous challenge in the last few years [200-204], there is a need to consider the impact of these tools on documentation and administrative burden. Thus, the findings from this review highlight the need to address engagement not only at the patient level but also at the clinician and system levels. In particular, investing in factors that support clinicians and the broader health system is equally important to ensure that all roles and key players can engage with the tool in a meaningful way, which can potentially be more cost-effective than focusing solely on patient-level factors [198,205,206]. Future work should capture how various organizations address the factors at these levels.

    Finally, the findings from this work have emerging implications for mental health clinicians, administrators, developers, and researchers. It is also important to note that while these factors were developed from mental health literature, it is likely that many of these factors are also relevant for digital health in other clinical areas [197,198]. For clinicians, it would be useful to consider these factors in formulating a plan and encouraging patients in the use of digital mental health tools in clinical care delivery [207]. Developers can consider these factors throughout the software development lifecycle and consider how these factors should be integrated into the features and development (eg, push reminders) of digital mental health tools. Educators and administrators may consider leveraging these factors to develop appropriate guidelines and curricula to enhance the capacity and readiness of current and future health care professionals to use these tools in their practice [208]. Similarly, administrators should consider these findings in developing a conducive environment for enhancing engagement with digital health tools in clinical practice. Finally, researchers can build on these findings to develop guidance (eg, tool kits) for designing tools keeping user engagement in mind. In addition, interventional studies can examine the impact of these factors on user engagement with digital mental health tools in clinical care settings.

    Limitations

    Several limitations should be considered when interpreting and applying the preliminary framework of factors from this literature review. First, given the nature of scoping reviews, the quality of the included articles was not assessed, and the literature search is not assumed to be exhaustive [46,51]. Second, this work primarily focused broadly on the patient population and, given the diverse characteristics of the populations included in this review, it was difficult to identify nuances and differences in impact across various sociodemographic and clinical populations.

    In addition, no studies were identified that looked at user engagement with digital mental health tools from the perspectives of caregivers and family members, despite their importance in mental health care [209]. As studies such as that by Simões de Almeida [210] have shown, exploring the prominence of various factors among different populations may be of interest. Finally, the preliminary framework of factors was developed based on the academic literature and were not validated with subject matter experts. This will be addressed in a subsequent study.

    Future Directions

    Several future directions have been identified to further strengthen and validate the framework of factors for digital mental health tools in clinical care settings. First, given the large number of factors identified from the included articles, there is a strong need to validate and identify the impact of these factors on user engagement. A Delphi study will be conducted to validate and identify any factors not found in this literature review [211]. Second, as the majority of the included studies (49/136, 36.1%) focused on the use of mobile apps, there was limited insight into the use of emerging tools such as patient portals and wearables. Finally, it would be imperative to explore how these factors influence user engagement in different clinical contexts.

    Conclusions

    A total of 26 factors influencing user engagement with digital mental health tools within clinical care settings were identified from the academic literature. These factors spanned across the categories of the sociotechnical framework, highlighting the complexity and interconnectedness of the factors in influencing user engagement. Future work should focus on identifying the factors considered essential for influencing user engagement and conducting exploratory studies to examine the differential impact of these factors on user engagement.

    Acknowledgments

    The authors would like to acknowledge the Centre for Addiction and Mental Health and the University of Toronto for their support in this work. The authors would also like to thank Sarah Bonato for her expert guidance in the development of the search strategy.

    This work is supported by the Centre for Addiction and Mental Health, the University of Toronto, a Frederick Banting and Charles Best Canada Graduate Scholarships–Doctoral Program Award (FBD-177877) from the Canadian Institutes for Health Research, and a Health Systems Impact Doctoral Fellowship (HI9-177457) from the Canadian Institutes for Health Research.

    Data Availability

    All data generated or analyzed during this study are included in this published paper and Multimedia Appendix 3.

    Authors' Contributions

    BL led the conceptualization and design of the study with input from GS, DW, SS, and QP. BL completed the article screening and data extraction with KD and RC. Analysis and reporting of the data was completed by BL and reviewed by GS, DW, SS, QP, and SK. BL wrote the manuscript, and all authors reviewed and made edits to it.

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    Search strategy.

    PDF File (Adobe PDF File), 241 KB
    Multimedia Appendix 2

    PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) checklist.

    PDF File (Adobe PDF File), 243 KB
    Multimedia Appendix 3

    Details of the included articles.

    XLSX File (Microsoft Excel File), 191 KB
    1. Smith KA, Blease C, Faurholt-Jepsen M, Firth J, Van Daele T, Moreno C, et al. Digital mental health: challenges and next steps. BMJ Ment Health. Feb 2023;26(1):e300670. [FREE Full text] [CrossRef] [Medline]
    2. Stern E, Micoulaud Franchi JA, Dumas G, Moreira J, Mouchabac S, Maruani J, et al. How can digital mental health enhance psychiatry? Neuroscientist. Dec 04, 2023;29(6):681-693. [CrossRef] [Medline]
    3. Lal S. E-mental health: Promising advancements in policy, research, and practice. Healthc Manage Forum. Mar 10, 2019;32(2):56-62. [FREE Full text] [CrossRef] [Medline]
    4. E-mental health in Canada. Mental Health Commission of Canada. URL: https://mentalhealthcommission.ca/what-we-do/e-mental-health/ [accessed 2024-04-29]
    5. Global strategy on digital health 2020-2025. World Health Organization. URL: https://www.who.int/docs/default-source/documents/gs4dhdaa2a9f352b0445bafbc79ca799dce4d.pdf [accessed 2024-04-29]
    6. Digital health care 2022 study findings. American Medical Association. URL: https://www.ama-assn.org/about/research/ama-digital-health-care-2022-study-findings [accessed 2024-04-29]
    7. Mental health apps. American Psychiatric Association. URL: https://www.psychiatry.org/psychiatrists/practice/mental-health-apps [accessed 2024-04-29]
    8. Gratzer D, Torous J, Lam RW, Patten SB, Kutcher S, Chan S, et al. Our digital moment: innovations and opportunities in digital mental health care. Can J Psychiatry. Jan 30, 2021;66(1):5-8. [FREE Full text] [CrossRef] [Medline]
    9. A plan for digital health and social care. Government of UK. URL: https:/​/www.​gov.uk/​government/​publications/​a-plan-for-digital-health-and-social-care/​a-plan-for-digital-health-and-social-care [accessed 2024-04-29]
    10. Telepsychiatry toolkit. American Psychiatric Association. URL: https://www.psychiatry.org/psychiatrists/practice/telepsychiatry/toolkit [accessed 2024-04-29]
    11. Mental health digital playbook. National Health Service England. URL: https:/​/transform.​england.nhs.uk/​key-tools-and-info/​digital-playbooks/​mental-health-digital-playbook/​ [accessed 2024-04-29]
    12. Hyatt J. Big white wall: expanding mental health access through the digital sphere. Health Affairs Forefront. 2015. URL: https:/​/www.​healthaffairs.org/​content/​forefront/​big-white-wall-expanding-mental-health-access-through-digital-sphere [accessed 2024-04-29]
    13. Marinova N, Rogers T, MacBeth A. Predictors of adolescent engagement and outcomes - a cross-sectional study using the togetherall (formerly Big White Wall) digital mental health platform. J Affect Disord. Aug 15, 2022;311:284-293. [FREE Full text] [CrossRef] [Medline]
    14. Hensel JM, Shaw J, Ivers NM, Desveaux L, Vigod SN, Cohen A, et al. A web-based mental health platform for individuals seeking specialized mental health care services: multicenter pragmatic randomized controlled trial. J Med Internet Res. Jun 04, 2019;21(6):e10838. [FREE Full text] [CrossRef] [Medline]
    15. Rodriguez-Villa E, Rauseo-Ricupero N, Camacho E, Wisniewski H, Keshavan M, Torous J. The digital clinic: implementing technology and augmenting care for mental health. Gen Hosp Psychiatry. 2020;66:59-66. [FREE Full text] [CrossRef] [Medline]
    16. Camacho E, Torous J. Impact of digital literacy training on outcomes for people with serious mental illness in community and inpatient settings. Psychiatr Serv. May 01, 2023;74(5):534-538. [FREE Full text] [CrossRef] [Medline]
    17. Hoffman L, Wisniewski H, Hays R, Henson P, Vaidyam A, Hendel V, et al. Digital opportunities for outcomes in recovery services (DOORS): a pragmatic hands-on group approach toward increasing digital health and smartphone competencies, autonomy, relatedness, and alliance for those with serious mental illness. J Psychiatr Pract. Mar 2020;26(2):80-88. [FREE Full text] [CrossRef] [Medline]
    18. Alon N, Perret S, Torous J. Working towards a ready to implement digital literacy program. Mhealth. Oct 2023;9:32. [FREE Full text] [CrossRef] [Medline]
    19. Perret S, Alon N, Torous J. A digital literacy program for adults with mental health conditions. Adult Lit Educ. Feb 01, 2023;5(1):49-54. [FREE Full text] [CrossRef]
    20. Marwaha JS, Landman AB, Brat GA, Dunn T, Gordon WJ. Deploying digital health tools within large, complex health systems: key considerations for adoption and implementation. NPJ Digit Med. Jan 27, 2022;5(1):13. [FREE Full text] [CrossRef] [Medline]
    21. Weisel KK, Fuhrmann LM, Berking M, Baumeister H, Cuijpers P, Ebert DD. Standalone smartphone apps for mental health-a systematic review and meta-analysis. NPJ Digit Med. 2019;2:118. [CrossRef] [Medline]
    22. Torous J, Nicholas J, Larsen ME, Firth J, Christensen H. Clinical review of user engagement with mental health smartphone apps: evidence, theory and improvements. Evid Based Ment Health. Aug 05, 2018;21(3):116-119. [FREE Full text] [CrossRef] [Medline]
    23. Ng MM, Firth J, Minen M, Torous J. User engagement in mental health apps: a review of measurement, reporting, and validity. Psychiatr Serv. Jul 01, 2019;70(7):538-544. [FREE Full text] [CrossRef] [Medline]
    24. Torous J, Michalak EE, O'Brien HL. Digital health and engagement-looking behind the measures and methods. JAMA Netw Open. Jul 01, 2020;3(7):e2010918. [FREE Full text] [CrossRef] [Medline]
    25. Fleming T, Bavin L, Lucassen M, Stasiak K, Hopkins S, Merry S. Beyond the trial: systematic review of real-world uptake and engagement with digital self-help interventions for depression, low mood, or anxiety. J Med Internet Res. Jun 06, 2018;20(6):e199. [FREE Full text] [CrossRef] [Medline]
    26. O'Brien HL, Toms EG. What is user engagement? A conceptual framework for defining user engagement with technology. J Am Soc Inf Sci. Feb 28, 2008;59(6):938-955. [CrossRef]
    27. Perski O, Blandford A, West R, Michie S. Conceptualising engagement with digital behaviour change interventions: a systematic review using principles from critical interpretive synthesis. Transl Behav Med. Jun 2017;7(2):254-267. [FREE Full text] [CrossRef] [Medline]
    28. Pham Q, Graham G, Carrion C, Morita PP, Seto E, Stinson JN, et al. A library of analytic indicators to evaluate effective engagement with consumer mHealth apps for chronic conditions: scoping review. JMIR Mhealth Uhealth. Jan 18, 2019;7(1):e11941. [FREE Full text] [CrossRef] [Medline]
    29. Pham Q, Graham G, Lalloo C, Morita PP, Seto E, Stinson JN, et al. An analytics platform to evaluate effective engagement with pediatric mobile health apps: design, development, and formative evaluation. JMIR Mhealth Uhealth. Dec 21, 2018;6(12):e11447. [FREE Full text] [CrossRef] [Medline]
    30. Yeager CM, Shoji K, Luszczynska A, Benight CC. Engagement with a trauma recovery internet intervention explained with the health action process approach (HAPA): longitudinal study. JMIR Ment Health. Apr 10, 2018;5(2):e29. [FREE Full text] [CrossRef] [Medline]
    31. MacPhail M, Mullan B, Sharpe L, MacCann C, Todd J. Using the health action process approach to predict and improve health outcomes in individuals with type 2 diabetes mellitus. Diabetes Metab Syndr Obes. 2014;7:469-479. [FREE Full text] [CrossRef] [Medline]
    32. Borghouts J, Eikey E, Mark G, De Leon C, Schueller SM, Schneider M, et al. Barriers to and facilitators of user engagement with digital mental health interventions: systematic review. J Med Internet Res. Mar 24, 2021;23(3):e24387. [FREE Full text] [CrossRef] [Medline]
    33. Nahum-Shani I, Shaw SD, Carpenter SM, Murphy SA, Yoon C. Engagement in digital interventions. Am Psychol. Oct 2022;77(7):836-852. [FREE Full text] [CrossRef] [Medline]
    34. Wu A, Scult MA, Barnes ED, Betancourt JA, Falk A, Gunning FM. Smartphone apps for depression and anxiety: a systematic review and meta-analysis of techniques to increase engagement. NPJ Digit Med. Feb 11, 2021;4(1):20. [FREE Full text] [CrossRef] [Medline]
    35. Gan DZ, McGillivray L, Larsen ME, Christensen H, Torok M. Technology-supported strategies for promoting user engagement with digital mental health interventions: a systematic review. Digit Health. Jun 01, 2022;8:20552076221098268. [FREE Full text] [CrossRef] [Medline]
    36. Stepped Care 2.0. Mental Health Commission of Canada. URL: https://mentalhealthcommission.ca/what-we-do/access/stepped-care-2-0/ [accessed 2024-04-29]
    37. Novak LL, Holden RJ, Anders SH, Hong JY, Karsh B. Using a sociotechnical framework to understand adaptations in health IT implementation. Int J Med Inform. Dec 2013;82(12):e331-e344. [FREE Full text] [CrossRef] [Medline]
    38. James HM, Papoutsi C, Wherton J, Greenhalgh T, Shaw SE. Spread, scale-up, and sustainability of video consulting in health care: systematic review and synthesis guided by the NASSS framework. J Med Internet Res. Jan 26, 2021;23(1):e23775. [FREE Full text] [CrossRef] [Medline]
    39. Papoutsi C, Wherton J, Shaw S, Morrison C, Greenhalgh T. Putting the social back into sociotechnical: case studies of co-design in digital health. J Am Med Inform Assoc. Feb 15, 2021;28(2):284-293. [FREE Full text] [CrossRef] [Medline]
    40. Greenhalgh T, Wherton J, Papoutsi C, Lynch J, Hughes G, A'Court C, et al. Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies. J Med Internet Res. Nov 01, 2017;19(11):e367. [FREE Full text] [CrossRef] [Medline]
    41. Sittig DF, Singh H. A new sociotechnical model for studying health information technology in complex adaptive healthcare systems. Qual Saf Health Care. Oct 2010;19 Suppl 3(Suppl 3):i68-i74. [FREE Full text] [CrossRef] [Medline]
    42. Sittig D, Singh H. A socio-technical approach to preventing, mitigating, and recovering from ransomware attacks. Appl Clin Inform. 2016;7(2):624-632. [FREE Full text] [CrossRef] [Medline]
    43. Darcel K, Upshaw T, Craig-Neil A, Macklin J, Steele Gray C, Chan TC, et al. Implementing artificial intelligence in Canadian primary care: barriers and strategies identified through a national deliberative dialogue. PLoS One. Feb 27, 2023;18(2):e0281733. [FREE Full text] [CrossRef] [Medline]
    44. Cornell EG, Harris E, McCune E, Fukui E, Lyons PG, Rojas JC, et al. Scaling up a diagnostic pause at the ICU-to-ward transition: an exploration of barriers and facilitators to implementation of the ICU-PAUSE handoff tool. Diagnosis (Berl). Nov 01, 2023;10(4):417-423. [CrossRef] [Medline]
    45. Arksey H, O'Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. Feb 2005;8(1):19-32. [CrossRef]
    46. Levac D, Colquhoun H, O'Brien KK. Scoping studies: advancing the methodology. Implement Sci. Sep 20, 2010;5(1):69. [FREE Full text] [CrossRef] [Medline]
    47. Peters MD, Marnie C, Tricco A, Pollock D, Munn Z, Alexander L, et al. Updated methodological guidance for the conduct of scoping reviews. JBI Evid Synth. Oct 2020;18(10):2119-2126. [CrossRef] [Medline]
    48. Search strategies for the identification of studies. Cochrane Common Mental Disorders. 2014. URL: https://cmd.cochrane.org/search-strategies-identification-studies [accessed 2024-04-29]
    49. McHugh ML. Interrater reliability: the Kappa statistic. Biochem Med (Zagreb). 2012;22(3):276-282. [FREE Full text] [Medline]
    50. Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. Jan 2006;3(2):77-101. [CrossRef]
    51. Tricco AC, Lillie E, Zarin W, O'Brien KK, Colquhoun H, Levac D, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. Oct 02, 2018;169(7):467-473. [FREE Full text] [CrossRef] [Medline]
    52. Robinson S, Perkins S, Bauer S, Hammond N, Treasure J, Schmidt U. Aftercare intervention through text messaging in the treatment of bulimia nervosa--feasibility pilot. Int J Eat Disord. Dec 2006;39(8):633-638. [CrossRef] [Medline]
    53. Bjerke TN, Kummervold PE, Christiansen EK, Hjortdahl P. “It made me feel connected”—an exploratory study on the use of mobile SMS in follow-up care for substance abusers. J Addict Nurs. 2008;19(4):195-200. [CrossRef]
    54. Nicholas J, Proudfoot J, Parker G, Gillis I, Burckhardt R, Manicavasagar V, et al. The ins and outs of an online bipolar education program: a study of program attrition. J Med Internet Res. Dec 19, 2010;12(5):e57. [FREE Full text] [CrossRef] [Medline]
    55. Sheeran T, Rabinowitz T, Lotterman J, Reilly CF, Brown S, Donehower P, et al. Feasibility and impact of telemonitor-based depression care management for geriatric homecare patients. Telemed J E Health. Oct 2011;17(8):620-626. [FREE Full text] [CrossRef] [Medline]
    56. Gulec H, Moessner M, Mezei A, Kohls E, Túry F, Bauer S. Internet-based maintenance treatment for patients with eating disorders. Prof Psychol Res Pr. Dec 2011;42(6):479-486. [CrossRef]
    57. Aguilera A, Muñoz RF. Text messaging as an adjunct to CBT in low-income populations: a usability and feasibility pilot study. Prof Psychol Res Pr. Dec 01, 2011;42(6):472-478. [FREE Full text] [CrossRef] [Medline]
    58. Binford Hopf RB, Le Grange D, Moessner M, Bauer S. Internet-based chat support groups for parents in family-based treatment for adolescent eating disorders: a pilot study. Eur Eat Disord Rev. May 05, 2013;21(3):215-223. [FREE Full text] [CrossRef] [Medline]
    59. Hilvert-Bruce Z, Rossouw PJ, Wong N, Sunderland M, Andrews G. Adherence as a determinant of effectiveness of internet cognitive behavioural therapy for anxiety and depressive disorders. Behav Res Ther. Aug 2012;50(7-8):463-468. [CrossRef] [Medline]
    60. Ben-Zeev D, Kaiser SM, Brenner CJ, Begale M, Duffecy J, Mohr DC. Development and usability testing of FOCUS: a smartphone system for self-management of schizophrenia. Psychiatr Rehabil J. Dec 1, 2013;36(4):289-296. [FREE Full text] [CrossRef] [Medline]
    61. Danaher BG, Milgrom J, Seeley JR, Stuart S, Schembri C, Tyler MS, et al. MomMoodBooster web-based intervention for postpartum depression: feasibility trial results. J Med Internet Res. Nov 04, 2013;15(11):e242-e247. [FREE Full text] [CrossRef] [Medline]
    62. Ainsworth J, Palmier-Claus JE, Machin M, Barrowclough C, Dunn G, Rogers A, et al. A comparison of two delivery modalities of a mobile phone-based assessment for serious mental illness: native smartphone application vs text-messaging only implementations. J Med Internet Res. Apr 05, 2013;15(4):e60. [FREE Full text] [CrossRef] [Medline]
    63. Wilhelmsen M, Lillevoll K, Risør MB, Høifødt R, Johansen M, Waterloo K, et al. Motivation to persist with internet-based cognitive behavioural treatment using blended care: a qualitative study. BMC Psychiatry. Nov 07, 2013;13:296. [FREE Full text] [CrossRef] [Medline]
    64. Carter FA, Bell CJ, Colhoun HC. Suitability and acceptability of computerised cognitive behaviour therapy for anxiety disorders in secondary care. Aust N Z J Psychiatry. Feb 1, 2013;47(2):142-152. [CrossRef] [Medline]
    65. Wilhelmsen M, Høifødt RS, Kolstrup N, Waterloo K, Eisemann M, Chenhall R, et al. Norwegian general practitioners' perspectives on implementation of a guided web-based cognitive behavioral therapy for depression: a qualitative study. J Med Internet Res. Sep 10, 2014;16(9):e208. [FREE Full text] [CrossRef] [Medline]
    66. Morrison C, Walker G, Ruggeri K, Hacker Hughes J. An implementation pilot of the MindBalance web-based intervention for depression in three IAPT services. Cogn Behav Therap. Nov 14, 2014;7(3):570-581. [CrossRef]
    67. Kok G, Bockting C, Burger H, Smit F, Riper H. Mobile cognitive therapy: adherence and acceptability of an online intervention in remitted recurrently depressed patients. Internet Interv. Apr 2014;1(2):65-73. [CrossRef]
    68. Ben-Zeev D, Brenner CJ, Begale M, Duffecy J, Mohr DC, Mueser KT. Feasibility, acceptability, and preliminary efficacy of a smartphone intervention for schizophrenia. Schizophr Bull. Nov 2014;40(6):1244-1253. [FREE Full text] [CrossRef] [Medline]
    69. Furber G, Jones GM, Healey D, Bidargaddi N. A comparison between phone-based psychotherapy with and without text messaging support in between sessions for crisis patients. J Med Internet Res. Oct 08, 2014;16(10):e219. [FREE Full text] [CrossRef] [Medline]
    70. Haug S, Lucht MJ, John U, Meyer C, Schaub MP. A pilot study on the feasibility and acceptability of a text message-based aftercare treatment programme among alcohol outpatients. Alcohol Alcohol. Mar 2015;50(2):188-194. [FREE Full text] [CrossRef] [Medline]
    71. Ingersoll B, Berger NI. Parent engagement with a telehealth-based parent-mediated intervention program for children with autism spectrum disorders: predictors of program use and parent outcomes. J Med Internet Res. Oct 06, 2015;17(10):e227-e253. [FREE Full text] [CrossRef] [Medline]
    72. Kauppi K, Kannisto KA, Hätönen H, Anttila M, Löyttyniemi E, Adams CE, et al. Mobile phone text message reminders: measuring preferences of people with antipsychotic medication. Schizophr Res. Oct 2015;168(1-2):514-522. [CrossRef] [Medline]
    73. Forchuk C, Donelle L, Ethridge P, Warner L. Client perceptions of the mental health engagement network: a secondary analysis of an intervention using smartphones and desktop devices for individuals experiencing mood or psychotic disorders in Canada. JMIR Ment Health. 2015;2(1):e1. [FREE Full text] [CrossRef] [Medline]
    74. El Alaoui S, Ljótsson B, Hedman E, Kaldo V, Andersson E, Rück C, et al. Predictors of symptomatic change and adherence in internet-based cognitive behaviour therapy for social anxiety disorder in routine psychiatric care. PLoS One. 2015;10(4):e0124258. [FREE Full text] [CrossRef] [Medline]
    75. Lucock M, Halstead J, Leach C, Barkham M, Tucker S, Randal C, et al. A mixed-method investigation of patient monitoring and enhanced feedback in routine practice: barriers and facilitators. Psychother Res. 2015;25(6):633-646. [FREE Full text] [CrossRef] [Medline]
    76. Ramirez M, Wu S, Jin H, Ell K, Gross-Schulman S, Myerchin Sklaroff L, et al. Automated remote monitoring of depression: acceptance among low-income patients in diabetes disease management. JMIR Ment Health. Jan 25, 2016;3(1):e6. [FREE Full text] [CrossRef] [Medline]
    77. Owens C, Charles N. Implementation of a text-messaging intervention for adolescents who self-harm (TeenTEXT): a feasibility study using normalisation process theory. Child Adolesc Psychiatry Ment Health. 2016;10:14. [FREE Full text] [CrossRef] [Medline]
    78. Wenze SJ, Armey MF, Weinstock L, Gaudiano B, Miller I. An open trial of a smartphone-assisted, adjunctive intervention to improve treatment adherence in bipolar disorder. J Psychiatr Pract. Nov 2016;22(6):492-504. [FREE Full text] [CrossRef] [Medline]
    79. Holländare F, Gustafsson SA, Berglind M, Grape F, Carlbring P, Andersson G, et al. Therapist behaviours in internet-based cognitive behaviour therapy (ICBT) for depressive symptoms. Internet Interv. Mar 2016;3:1-7. [FREE Full text] [CrossRef] [Medline]
    80. Högdahl L, Levallius J, Björck C, Norring C, Birgegård A. Personality predicts drop-out from therapist-guided internet-based cognitive behavioural therapy for eating disorders. Results from a randomized controlled trial. Internet Interv. Sep 2016;5:44-50. [FREE Full text] [CrossRef] [Medline]
    81. Espinosa HD, Carrasco Á, Moessner M, Cáceres C, Gloger S, Rojas G, et al. Acceptability study of "Ascenso": an online program for monitoring and supporting patients with depression in Chile. Telemed J E Health. Jul 2016;22(7):577-583. [CrossRef] [Medline]
    82. Nitsch M, Dimopoulos CN, Flaschberger E, Saffran K, Kruger JF, Garlock L, et al. A guided online and mobile self-help program for individuals with eating disorders: an iterative engagement and usability study. J Med Internet Res. Jan 11, 2016;18(1):e7. [FREE Full text] [CrossRef] [Medline]
    83. Gammon D, Strand M, Eng LS, Børøsund E, Varsi C, Ruland C. Shifting practices toward recovery-oriented care through an e-recovery portal in community mental health care: a mixed-methods exploratory study. J Med Internet Res. May 02, 2017;19(5):e145. [FREE Full text] [CrossRef] [Medline]
    84. Mohr DC, Tomasino KN, Lattie EG, Palac HL, Kwasny MJ, Weingardt K, et al. IntelliCare: an eclectic, skills-based app suite for the treatment of depression and anxiety. J Med Internet Res. Jan 05, 2017;19(1):e10. [FREE Full text] [CrossRef] [Medline]
    85. Chum J, Kim MS, Zielinski L, Bhatt M, Chung D, Yeung S, et al. Acceptability of the Fitbit in behavioural activation therapy for depression: a qualitative study. Evid Based Ment Health. Nov 2017;20(4):128-133. [FREE Full text] [CrossRef] [Medline]
    86. Lauritsen L, Andersen L, Olsson E, Søndergaard SR, Nørregaard LB, Løventoft PK, et al. Usability, acceptability, and adherence to an electronic self-monitoring system in patients with major depression discharged from inpatient wards. J Med Internet Res. Apr 21, 2017;19(4):e123-e160. [FREE Full text] [CrossRef] [Medline]
    87. Moitra E, Gaudiano BA, Davis CH, Ben-Zeev D. Feasibility and acceptability of post-hospitalization ecological momentary assessment in patients with psychotic-spectrum disorders. Compr Psychiatry. Apr 2017;74(3):204-213. [FREE Full text] [CrossRef] [Medline]
    88. Reger GM, Browne KC, Campellone TR, Simons C, Kuhn E, Fortney J, et al. Barriers and facilitators to mobile application use during PTSD treatment: clinician adoption of PE coach. Prof Psychol Res Pract. Dec 2017;48(6):510-517. [CrossRef]
    89. Gellatly J, Pedley R, Molloy C, Butler J, Lovell K, Bee P. Low intensity interventions for Obsessive-Compulsive Disorder (OCD): a qualitative study of mental health practitioner experiences. BMC Psychiatry. Feb 22, 2017;17(1):77. [FREE Full text] [CrossRef] [Medline]
    90. Fernández-Álvarez J, Díaz-García A, González-Robles A, Baños R, García-Palacios A, Botella C. Dropping out of a transdiagnostic online intervention: a qualitative analysis of client's experiences. Internet Interv. Dec 2017;10:29-38. [FREE Full text] [CrossRef] [Medline]
    91. Watson HJ, Levine MD, Zerwas SC, Hamer RM, Crosby RD, Sprecher CS, et al. Predictors of dropout in face-to-face and internet-based cognitive-behavioral therapy for bulimia nervosa in a randomized controlled trial. Int J Eat Disord. May 2017;50(5):569-577. [FREE Full text] [CrossRef] [Medline]
    92. Saunders KE, Bilderbeck AC, Panchal P, Atkinson LZ, Geddes JR, Goodwin GM. Experiences of remote mood and activity monitoring in bipolar disorder: a qualitative study. Eur Psychiatry. Mar 2017;41:115-121. [FREE Full text] [CrossRef] [Medline]
    93. Mackie C, Dunn N, MacLean S, Testa V, Heisel M, Hatcher S. A qualitative study of a blended therapy using problem solving therapy with a customised smartphone app in men who present to hospital with intentional self-harm. Evid Based Ment Health. Nov 2017;20(4):118-122. [FREE Full text] [CrossRef] [Medline]
    94. Abel EA, Shimada SL, Wang K, Ramsey C, Skanderson M, Erdos J, et al. Dual use of a patient portal and clinical video telehealth by veterans with mental health diagnoses: retrospective, cross-sectional analysis. J Med Internet Res. Nov 07, 2018;20(11):e11350. [FREE Full text] [CrossRef] [Medline]
    95. Quanbeck A, Gustafson DH, Marsch LA, Chih M, Kornfield R, McTavish F, et al. Implementing a mobile health system to integrate the treatment of addiction into primary care: a hybrid implementation-effectiveness study. J Med Internet Res. Jan 30, 2018;20(1):e37. [FREE Full text] [CrossRef] [Medline]
    96. Ben-Zeev D, Brian RM, Aschbrenner KA, Jonathan G, Steingard S. Video-based mobile health interventions for people with schizophrenia: bringing the "pocket therapist" to life. Psychiatr Rehabil J. Mar 2018;41(1):39-45. [CrossRef] [Medline]
    97. Fuhr K, Schröder J, Berger T, Moritz S, Meyer B, Lutz W, et al. The association between adherence and outcome in an internet intervention for depression. J Affect Disord. Mar 15, 2018;229(6):443-449. [CrossRef] [Medline]
    98. Muroff J, Steketee G. Pilot trial of cognitive and behavioral treatment for hoarding disorder delivered via webcam: feasibility and preliminary outcomes. J Obsessive Compuls Relat Disord. Jul 2018;18(6):18-24. [CrossRef]
    99. Stjerneklar S, Hougaard E, Nielsen AD, Gaardsvig MM, Thastum M. Internet-based cognitive behavioral therapy for adolescents with anxiety disorders: a feasibility study. Internet Interv. Mar 2018;11:30-40. [FREE Full text] [CrossRef] [Medline]
    100. Campos D, Mira A, Bretón-López J, Castilla D, Botella C, Baños RM, et al. The acceptability of an internet-based exposure treatment for flying phobia with and without therapist guidance: patients' expectations, satisfaction, treatment preferences, and usability. Neuropsychiatr Dis Treat. 2018;14:879-892. [FREE Full text] [CrossRef] [Medline]
    101. Lindgreen P, Lomborg K, Clausen L. Patient experiences using a self-monitoring app in eating disorder treatment: qualitative study. JMIR Mhealth Uhealth. Jun 22, 2018;6(6):e10253. [FREE Full text] [CrossRef] [Medline]
    102. Folker AP, Mathiasen K, Lauridsen SM, Stenderup E, Dozeman E, Folker MP. Implementing internet-delivered cognitive behavior therapy for common mental health disorders: a comparative case study of implementation challenges perceived by therapists and managers in five European internet services. Internet Interv. Mar 2018;11:60-70. [FREE Full text] [CrossRef] [Medline]
    103. Terp M, Jørgensen R, Laursen BS, Mainz J, Bjørnes CD. A smartphone app to foster power in the everyday management of living with schizophrenia: qualitative analysis of young adults' perspectives. JMIR Ment Health. Oct 01, 2018;5(4):e10157. [FREE Full text] [CrossRef] [Medline]
    104. Kurki M, Anttila M, Koivunen M, Marttunen M, Välimäki M. Nurses' experiences of the use of an internet-based support system for adolescents with depressive disorders. Inform Health Soc Care. Sep 2018;43(3):234-247. [CrossRef] [Medline]
    105. Bauer AM, Iles-Shih M, Ghomi RH, Rue T, Grover T, Kincler N, et al. Acceptability of mHealth augmentation of collaborative care: a mixed methods pilot study. Gen Hosp Psychiatry. 2018;51:22-29. [FREE Full text] [CrossRef] [Medline]
    106. Mohr DC, Schueller SM, Tomasino KN, Kaiser SM, Alam N, Karr C, et al. Comparison of the effects of coaching and receipt of app recommendations on depression, anxiety, and engagement in the IntelliCare platform: factorial randomized controlled trial. J Med Internet Res. Aug 28, 2019;21(8):e13609. [FREE Full text] [CrossRef] [Medline]
    107. Hussey D, Flynn KC. The utility and impact of the addiction comprehensive health enhancement support system (ACHESS) on substance abuse treatment adherence among youth in an intensive outpatient program. Psychiatry Res. Nov 2019;281:112580. [CrossRef]
    108. Ybarra ML, Rodriguez K, Madison H, Mojtabai R, Cullen BA. Developing texting for relapse prevention: a scalable mHealth program for people with schizophrenia and schizoaffective disorder. J Nerv Ment Dis. Oct 2019;207(10):854-862. [FREE Full text] [CrossRef] [Medline]
    109. Kemmeren LL, van Schaik A, Smit JH, Ruwaard J, Rocha A, Henriques M, et al. Unraveling the black box: exploring usage patterns of a blended treatment for depression in a multicenter study. JMIR Ment Health. Jul 25, 2019;6(7):e12707. [FREE Full text] [CrossRef] [Medline]
    110. Rodda SN, Merkouris S, Lavis T, Smith D, Lubman DI, Austin D, et al. The therapist experience of internet delivered CBT for problem gambling: service integration considerations. Internet Interv. Dec 2019;18:100264. [FREE Full text] [CrossRef] [Medline]
    111. Mol M, van Genugten C, Dozeman E, van Schaik DJ, Draisma S, Riper H, et al. Why uptake of blended internet-based interventions for depression is challenging: a qualitative study on therapists' perspectives. J Clin Med. Dec 30, 2019;9(1):91. [FREE Full text] [CrossRef] [Medline]
    112. Schmidt ID, Forand NR, Strunk DR. Predictors of dropout in internet-based cognitive behavioral therapy for depression. Cognit Ther Res. Jun 2019;43(3):620-630. [FREE Full text] [CrossRef] [Medline]
    113. Jonathan G, Carpenter-Song EA, Brian RM, Ben-Zeev D. Life with FOCUS: a qualitative evaluation of the impact of a smartphone intervention on people with serious mental illness. Psychiatr Rehabil J. Jun 2019;42(2):182-189. [FREE Full text] [CrossRef] [Medline]
    114. Etingen B, Hogan TP, Martinez RN, Shimada S, Stroupe K, Nazi K, et al. How do patients with mental health diagnoses use online patient portals? An observational analysis from the veterans health administration. Adm Policy Ment Health. Sep 2019;46(5):596-608. [CrossRef] [Medline]
    115. Schuster R, Kalthoff I, Walther A, Köhldorfer L, Partinger E, Berger T, et al. Effects, adherence, and therapists' perceptions of web- and mobile-supported group therapy for depression: mixed-methods study. J Med Internet Res. Apr 28, 2019;21(5):e11860. [FREE Full text] [CrossRef] [Medline]
    116. Kidd SA, Feldcamp L, Adler A, Kaleis L, Wang W, Vichnevetski K, et al. Feasibility and outcomes of a multi-function mobile health approach for the schizophrenia spectrum: App4Independence (A4i). PLoS One. 2019;14(7):e0219491. [FREE Full text] [CrossRef] [Medline]
    117. Achtyes ED, Ben-Zeev D, Luo Z, Mayle H, Burke B, Rotondi AJ, et al. Off-hours use of a smartphone intervention to extend support for individuals with schizophrenia spectrum disorders recently discharged from a psychiatric hospital. Schizophr Res. Apr 2019;206:200-208. [CrossRef] [Medline]
    118. March S, Donovan CL, Baldwin S, Ford M, Spence SH. Using stepped-care approaches within internet-based interventions for youth anxiety: three case studies. Internet Interv. Dec 2019;18:100281. [FREE Full text] [CrossRef] [Medline]
    119. Porras-Segovia A, Molina-Madueño RM, Berrouiguet S, López-Castroman J, Barrigón ML, Pérez-Rodríguez MS, et al. Smartphone-based ecological momentary assessment (EMA) in psychiatric patients and student controls: a real-world feasibility study. J Affect Disord. Sep 01, 2020;274:733-741. [CrossRef] [Medline]
    120. Glenn CR, Kleiman EM, Kearns JC, Santee AC, Esposito EC, Conwell Y, et al. Feasibility and acceptability of ecological momentary assessment with high-risk suicidal adolescents following acute psychiatric care. J Clin Child Adolesc Psychol. Apr 02, 2022;51(1):32-48. [CrossRef] [Medline]
    121. Enrique A, Duffy D, Lawler K, Richards D, Jones S. An internet-delivered self-management programme for bipolar disorder in mental health services in Ireland: results and learnings from a feasibility trial. Clin Psychol Psychother. Nov 2020;27(6):925-939. [FREE Full text] [CrossRef] [Medline]
    122. Bilić SS, Moessner M, Wirtz G, Lang S, Weisbrod M, Bauer S. Internet-based aftercare for patients with personality disorders and trauma-related disorders: a pilot study. Psychiatry Res. Jan 11, 2020;285:112771. [CrossRef] [Medline]
    123. Carrasco AE, Moessner M, Carbonell CG, Rodríguez C, Martini N, Pérez JC, et al. SIN-E-STRES: an adjunct internet-based intervention for the treatment of patients with posttraumatic stress disorder in Chile. CES Psico. Nov 24, 2020;13(3):239-258. [CrossRef]
    124. Kip H, Sieverink F, van Gemert-Pijnen LJ, Bouman YH, Kelders SM. Integrating people, context, and technology in the implementation of a web-based intervention in forensic mental health care: mixed-methods study. J Med Internet Res. May 26, 2020;22(5):e16906. [FREE Full text] [CrossRef] [Medline]
    125. Connolly SL, Etingen B, Shimada SL, Hogan TP, Nazi K, Stroupe K, et al. Patient portal use among veterans with depression: associations with symptom severity and demographic characteristics. J Affect Disord. Oct 01, 2020;275(6):255-259. [FREE Full text] [CrossRef] [Medline]
    126. Steare T, O'Hanlon P, Eskinazi M, Osborn D, Lloyd-Evans B, Jones R, et al. Smartphone-delivered self-management for first-episode psychosis: the ARIES feasibility randomised controlled trial. BMJ Open. Aug 26, 2020;10(8):e034927. [CrossRef] [Medline]
    127. Stawarz K, Preist C, Tallon D, Wiles N, Kessler D, Turner K, et al. Design considerations for the integrated delivery of cognitive behavioral therapy for depression: user-centered design study. JMIR Ment Health. Sep 03, 2020;7(9):e15972. [FREE Full text] [CrossRef] [Medline]
    128. Chen H, Rodriguez MA, Qian M, Kishimoto T, Lin M, Berger T. Predictors of treatment outcomes and adherence in internet-based cognitive behavioral therapy for social anxiety in China. Behav Cogn Psychother. May 2020;48(3):291-303. [CrossRef] [Medline]
    129. MacLean S, Corsi DJ, Litchfield S, Kucharski J, Genise K, Selaman Z, et al. Coach-facilitated web-based therapy compared with information about web-based resources in patients referred to secondary mental health care for depression: randomized controlled trial. J Med Internet Res. Jun 09, 2020;22(6):e15001. [FREE Full text] [CrossRef] [Medline]
    130. Alvarez-Jimenez M, Rice S, D'Alfonso S, Leicester S, Bendall S, Pryor I, et al. A novel multimodal digital service (moderated online social therapy+) for help-seeking young people experiencing mental ill-health: pilot evaluation within a national youth e-mental health service. J Med Internet Res. Aug 13, 2020;22(8):e17155. [FREE Full text] [CrossRef] [Medline]
    131. Weisel KK, Zarski A, Berger T, Krieger T, Moser CT, Schaub MP, et al. User experience and effects of an individually tailored transdiagnostic internet-based and mobile-supported intervention for anxiety disorders: mixed-methods study. J Med Internet Res. Sep 16, 2020;22(9):e16450. [FREE Full text] [CrossRef] [Medline]
    132. Strudwick G, Booth RG, McLean D, Leung K, Rossetti S, McCann M, et al. Identifying indicators of meaningful patient portal use by psychiatric populations. Inform Health Soc Care. Oct 01, 2020;45(4):396-409. [CrossRef] [Medline]
    133. Valentine L, McEnery C, Bell I, O'Sullivan S, Pryor I, Gleeson J, et al. Blended digital and face-to-face care for first-episode psychosis treatment in young people: qualitative study. JMIR Ment Health. Jul 28, 2020;7(7):e18990. [FREE Full text] [CrossRef] [Medline]
    134. Lal S, Gleeson J, Rivard L, D'Alfonso S, Joober R, Malla A, et al. Adaptation of a digital health innovation to prevent relapse and support recovery in youth receiving services for first-episode psychosis: results from the Horyzons-Canada phase 1 study. JMIR Form Res. Oct 29, 2020;4(10):e19887. [FREE Full text] [CrossRef] [Medline]
    135. Kellett S, Easton K, Cooper M, Millings A, Simmonds-Buckley M, Parry G. Evaluation of a mobile app to enhance relational awareness and change during cognitive analytic therapy: mixed methods case series. JMIR Ment Health. Dec 18, 2020;7(12):e19888. [FREE Full text] [CrossRef] [Medline]
    136. Vöhringer M, Knaevelsrud C, Wagner B, Slotta M, Schmidt A, Stammel N, et al. Should I stay or must I go? Predictors of dropout in an internet-based psychotherapy programme for posttraumatic stress disorder in Arabic. Eur J Psychotraumatol. 2020;11(1):1706297. [FREE Full text] [CrossRef] [Medline]
    137. Bechtel JM, Lepoire E, Bauer AM, Bowen DJ, Fortney JC. Care manager perspectives on integrating an mHealth app system into clinical workflows: a mixed methods study. Gen Hosp Psychiatry. Jan 2021;68:38-45. [FREE Full text] [CrossRef] [Medline]
    138. Matanov A, McNamee P, Akther S, Barber N, Bird V. Acceptability of a technology-supported and solution-focused intervention (DIALOG+) for chronic depression: views of service users and clinicians. BMC Psychiatry. May 20, 2021;21(1):263. [FREE Full text] [CrossRef] [Medline]
    139. Ola C, Gonzalez E, Tran N, Sasser T, Kuhn M, LaCount PA, et al. Evaluating the feasibility and acceptability of the lifestyle enhancement for ADHD program. J Pediatr Psychol. Jul 20, 2021;46(6):662-672. [FREE Full text] [CrossRef] [Medline]
    140. Primack JM, Bozzay M, Barredo J, Armey M, Miller IW, Fisher JB, et al. Feasibility and acceptability of the mobile application for the prevention of suicide (MAPS). Mil Psychol. Jan 2022;34(3):315-325. [FREE Full text] [CrossRef] [Medline]
    141. Arnfred BT, Bang P, Winther Davy J, Larsen LQ, Hjorthøj C, Christensen AB. Virtual reality exposure in cognitive behavioral group therapy for social anxiety disorder: A qualitative evaluation based on patients’ and therapists’ experiences. Transl Issues Psychol Sci. Sep 2021;7(3):229-247. [CrossRef]
    142. Strauss C, Dunkeld C, Cavanagh K. Is clinician-supported use of a mindfulness smartphone app a feasible treatment for depression? A mixed-methods feasibility study. Internet Interv. Sep 2021;25:100413. [FREE Full text] [CrossRef] [Medline]
    143. Medalia A, Saperstein AM, Stefancic A, Meyler S, Styke S, Qian M, et al. Feasibility and acceptability of remotely accessed cognitive remediation for schizophrenia in public health settings. Psychiatry Res. Jul 2021;301:113956. [FREE Full text] [CrossRef] [Medline]
    144. Smart K, Smith L, Harvey K, Waite P. The acceptability of a therapist-assisted internet-delivered cognitive behaviour therapy program for the treatment of anxiety disorders in adolescents: a qualitative study. Eur Child Adolesc Psychiatry. Apr 08, 2023;32(4):661-673. [FREE Full text] [CrossRef] [Medline]
    145. Steare T, Giorgalli M, Free K, Harju-Seppänen J, Akther S, Eskinazi M, et al. A qualitative study of stakeholder views on the use of a digital app for supported self-management in early intervention services for psychosis. BMC Psychiatry. Jun 19, 2021;21(1):311. [FREE Full text] [CrossRef] [Medline]
    146. Austin SF, Frøsig A, Buus N, Lincoln T, von Malachowski A, Schlier B, et al. Service user experiences of integrating a mobile solution (IMPACHS) into clinical treatment for psychosis. Qual Health Res. Apr 2021;31(5):942-954. [CrossRef] [Medline]
    147. Röhricht F, Padmanabhan R, Binfield P, Mavji D, Barlow S. Simple mobile technology health management tool for people with severe mental illness: a randomised controlled feasibility trial. BMC Psychiatry. Jul 16, 2021;21(1):357. [FREE Full text] [CrossRef] [Medline]
    148. Gordon D, Hensel J, Bouck Z, Desveaux L, Soobiah C, Saragosa M, et al. Developing an explanatory theoretical model for engagement with a web-based mental health platform: results of a mixed methods study. BMC Psychiatry. Aug 21, 2021;21(1):417. [FREE Full text] [CrossRef] [Medline]
    149. Bailey DP, Edwardson CL, Pappas Y, Dong F, Hewson DJ, Biddle SJ, et al. A randomised-controlled feasibility study of the REgulate your SItting Time (RESIT) intervention for reducing sitting time in individuals with type 2 diabetes: study protocol. Pilot Feasibility Stud. Mar 19, 2021;7(1):76. [FREE Full text] [CrossRef] [Medline]
    150. Knutson D, Kertz S, Chambers-Baltz S, Christie MB, Harris E, Perinchery R. A pilot test of a text message-based transgender and nonbinary affirmative cognitive-behavioral intervention for anxiety and depression. Psychol Sex Orientat Gend Divers. Dec 2021;8(4):440-450. [CrossRef]
    151. Khan K, Hollis C, Hall CL, Murray E, Davies EB, Andrén P, et al. Fidelity of delivery and contextual factors influencing children's level of engagement: process evaluation of the online remote behavioral intervention for tics trial. J Med Internet Res. Jun 21, 2021;23(6):e25470. [FREE Full text] [CrossRef] [Medline]
    152. Gire N, Caton N, McKeown M, Mohmed N, Duxbury J, Kelly J, et al. Care co-ordinator in my pocket': a feasibility study of mobile assessment and therapy for psychosis (TechCare). BMJ Open. Nov 16, 2021;11(11):e046755. [FREE Full text] [CrossRef] [Medline]
    153. Silfee V, Williams K, Leber B, Kogan J, Nikolajski C, Szigethy E, et al. Health care provider perspectives on the use of a digital behavioral health app to support patients: qualitative study. JMIR Form Res. Sep 28, 2021;5(9):e28538. [FREE Full text] [CrossRef] [Medline]
    154. Oehler C, Scholze K, Driessen P, Rummel-Kluge C, Görges F, Hegerl U. How are guide profession and routine care setting related to adherence and symptom change in iCBT for depression? - an explorative log-data analysis. Internet Interv. Dec 2021;26:100476. [FREE Full text] [CrossRef] [Medline]
    155. Lawler K, Earley C, Timulak L, Enrique A, Richards D. Dropout from an internet-delivered cognitive behavioral therapy intervention for adults with depression and anxiety: qualitative study. JMIR Form Res. Nov 12, 2021;5(11):e26221. [FREE Full text] [CrossRef] [Medline]
    156. Jonathan GK, Dopke CA, Michaels T, Bank A, Martin CR, Adhikari K, et al. A Smartphone-based self-management intervention for bipolar disorder (LiveWell): user-centered development approach. JMIR Ment Health. Apr 12, 2021;8(4):e20424. [FREE Full text] [CrossRef] [Medline]
    157. Jonathan GK, Dopke CA, Michaels T, Martin CR, Ryan C, McBride A, et al. A smartphone-based self-management intervention for individuals with bipolar disorder (LiveWell): qualitative study on user experiences of the behavior change process. JMIR Ment Health. Nov 22, 2021;8(11):e32306. [FREE Full text] [CrossRef] [Medline]
    158. Turvey CL, Fuhrmeister LA, Klein DM, Moeckli J, Howren MB, Chasco EE. Patient and provider experience of electronic patient portals and secure messaging in mental health treatment. Telemed J E Health. Feb 2022;28(2):189-198. [FREE Full text] [CrossRef] [Medline]
    159. Yue H, Mail V, DiSalvo M, Borba C, Piechniczek-Buczek J, Yule AM. Patient preferences for patient portal-based telepsychiatry in a safety net hospital setting during COVID-19: cross-sectional study. JMIR Form Res. Jan 26, 2022;6(1):e33697. [FREE Full text] [CrossRef] [Medline]
    160. Bisson JI, Ariti C, Cullen K, Kitchiner N, Lewis C, Roberts NP, et al. Guided, internet based, cognitive behavioural therapy for post-traumatic stress disorder: pragmatic, multicentre, randomised controlled non-inferiority trial (RAPID). BMJ. Jun 16, 2022;377:e069405. [FREE Full text] [CrossRef] [Medline]
    161. Tarp K, Holmberg TT, Moeller AM, Lichtenstein MB. Patient and therapist experiences of using a smartphone application monitoring anxiety symptoms. Int J Qual Stud Health Well-being. Dec 2022;17(1):2044981-2044986. [FREE Full text] [CrossRef] [Medline]
    162. Zarbo C, Agosta S, Casiraghi L, De Novellis A, Leuci E, Paulillo G, et al. Assessing adherence to and usability of Experience Sampling Method (ESM) and actigraph in patients with schizophrenia spectrum disorder: a mixed-method study. Psychiatry Res. Aug 2022;314:114675. [CrossRef] [Medline]
    163. Barceló-Soler A, García-Campayo J, Araya R, Doukani A, Gili M, García-Palacios A, et al. Working alliance in low-intensity internet-based cognitive behavioral therapy for depression in primary care in Spain: a qualitative study. Front Psychol. 2023;14:1024966. [FREE Full text] [CrossRef] [Medline]
    164. Chang S, Gray L, Torous J. Smartphone app engagement and clinical outcomes in a hybrid clinic. Psychiatry Res. Jan 2023;319:115015. [CrossRef] [Medline]
    165. Chung OS, Dowling NL, Brown C, Robinson T, Johnson AM, Ng CH, et al. Using the theoretical domains framework to inform the implementation of therapeutic virtual reality into mental healthcare. Adm Policy Ment Health. Mar 2023;50(2):237-268. [CrossRef] [Medline]
    166. de Angel V, Adeleye F, Zhang Y, Cummins N, Munir S, Lewis S, et al. The feasibility of implementing remote measurement technologies in psychological treatment for depression: mixed methods study on engagement. JMIR Ment Health. Jan 24, 2023;10:e42866. [FREE Full text] [CrossRef] [Medline]
    167. Denyer H, Deng Q, Adanijo A, Asherson P, Bilbow A, Folarin A, et al. Barriers to and facilitators of using remote measurement technology in the long-term monitoring of individuals with ADHD: interview study. JMIR Form Res. Jun 30, 2023;7:e44126. [FREE Full text] [CrossRef] [Medline]
    168. Fisher A, Corrigan E, Cross S, Ryan K, Staples L, Tan R, et al. Decision-making about uptake and engagement among digital mental health service users: a qualitative exploration of therapist perspectives. Clin Psychol. Mar 06, 2023;27(2):171-185. [CrossRef]
    169. Grasa E, Seppälä J, Alonso-Solis A, Haapea M, Isohanni M, Miettunen J, m-RESIST group, et al. m-RESIST, a mobile therapeutic intervention for treatment-resistant schizophrenia: feasibility, acceptability, and usability study. JMIR Form Res. Jun 30, 2023;7:e46179. [FREE Full text] [CrossRef] [Medline]
    170. Jiang A, Al-Dajani N, King C, Hong V, Koo HJ, Czyz E. Acceptability and feasibility of ecological momentary assessment with augmentation of passive sensor data in young adults at high risk for suicide. Psychiatry Res. Aug 2023;326:115347. [CrossRef] [Medline]
    171. Lo B, Shin HD, Kemp J, Munnery M, Chen S, Ma C, et al. Shifting mindsets: the impact of a patient portal on functioning and recovery in a mental health setting. Can J Psychiatry. Mar 2024;69(3):217-227. [CrossRef] [Medline]
    172. Marshall T, Viste D, Jones S, Kim J, Lee A, Jafri F, et al. Beliefs, attitudes and experiences of virtual overdose monitoring services from the perspectives of people who use substances in Canada: a qualitative study. Harm Reduct J. Jun 24, 2023;20(1):80. [FREE Full text] [CrossRef] [Medline]
    173. Nielsen MS, Steinsbekk A, Nøst TH. Views on patient portal use for adolescents in mental health care - a qualitative study. BMC Health Serv Res. Feb 09, 2023;23(1):132. [FREE Full text] [CrossRef] [Medline]
    174. Ortiz A, Park Y, Gonzalez-Torres C, Alda M, Blumberger DM, Burnett R, et al. Predictors of adherence to electronic self-monitoring in patients with bipolar disorder: a contactless study using Growth Mixture Models. Int J Bipolar Disord. May 17, 2023;11(1):18. [CrossRef] [Medline]
    175. Stenbro AW, Moldt S, Eriksen JW, Frostholm L. "I was treated by the program, the therapist, and myself": feasibility of an internet-based treatment program for gambling disorder. J Gambl Stud. Dec 2023;39(4):1885-1907. [FREE Full text] [CrossRef] [Medline]
    176. White KM, Dawe-Lane E, Siddi S, Lamers F, Simblett S, Riquelme Alacid G, et al. Understanding the subjective experience of long-term remote measurement technology use for symptom tracking in people with depression: multisite longitudinal qualitative analysis. JMIR Hum Factors. Jan 26, 2023;10:e39479. [FREE Full text] [CrossRef] [Medline]
    177. Hetrick SE, Goodall J, Yuen HP, Davey CG, Parker AG, Robinson J, et al. Comprehensive online self-monitoring to support clinicians manage risk of suicide in youth depression. Crisis. May 2017;38(3):147-157. [CrossRef] [Medline]
    178. Byrne S, Tohamy A, Kotze B, Ramos F, Starling J, Karageorge A, et al. Using a mobile health device to monitor physiological stress for serious mental illness: a qualitative analysis of patient and clinician-related acceptability. Psychiatr Rehabil J. Sep 2022;45(3):219-225. [CrossRef] [Medline]
    179. Darnell D, Pullmann MD, Hull TD, Chen S, Areán P. Predictors of disengagement and symptom improvement among adults with depression enrolled in talkspace, a technology-mediated psychotherapy platform: naturalistic observational study. JMIR Form Res. Jun 22, 2022;6(6):e36521. [FREE Full text] [CrossRef] [Medline]
    180. de Angel V, Lewis S, White KM, Matcham F, Hotopf M. Clinical targets and attitudes toward implementing digital health tools for remote measurement in treatment for depression: focus groups with patients and clinicians. JMIR Ment Health. Aug 15, 2022;9(8):e38934. [FREE Full text] [CrossRef] [Medline]
    181. Gonzales S, Okusaga OO, Reuteman-Fowler JC, Oakes MM, Brown JN, Moore S, et al. Digital medicine system in veterans with severe mental illness: feasibility and acceptability study. JMIR Form Res. Dec 22, 2022;6(12):e34893. [FREE Full text] [CrossRef] [Medline]
    182. Greenwood KE, Gurnani M, Ward T, Vogel E, Vella C, McGourty A, et al. SlowMo Patient‚ Public Involvement (PPI) team. The service user experience of SlowMo therapy: a co-produced thematic analysis of service users' subjective experience. Psychol Psychother. Sep 2022;95(3):680-700. [FREE Full text] [CrossRef] [Medline]
    183. Matthews EB, Savoy M, Paranjape A, Washington D, Hackney T, Galis D, et al. Acceptability of health information exchange and patient portal use in depression care among underrepresented patients. J Gen Intern Med. Nov 2022;37(15):3947-3955. [FREE Full text] [CrossRef] [Medline]
    184. Nicolaisen Sidén F, Spak F. Secondary psychiatric care patients' experiences of internet CBT for insomnia - a qualitative study. BMC Psychol. Oct 27, 2022;10(1):237. [FREE Full text] [CrossRef] [Medline]
    185. Price GD, Heinz MV, Nemesure MD, McFadden J, Jacobson NC. Predicting symptom response and engagement in a digital intervention among individuals with schizophrenia and related psychoses. Front Psychiatry. Aug 11, 2022;13:807116. [FREE Full text] [CrossRef] [Medline]
    186. Ramadurai R, Beckham E, McHugh RK, Björgvinsson T, Beard C. Operationalizing engagement with an interpretation bias smartphone app intervention: case series. JMIR Ment Health. Aug 17, 2022;9(8):e33545. [FREE Full text] [CrossRef] [Medline]
    187. Rus-Calafell M, Ehrbar N, Ward T, Edwards C, Huckvale M, Walke J, et al. Participants' experiences of AVATAR therapy for distressing voices: a thematic qualitative evaluation. BMC Psychiatry. May 24, 2022;22(1):356. [FREE Full text] [CrossRef] [Medline]
    188. Bailey E, Robinson J, Alvarez-Jimenez M, Nedeljkovic M, Valentine L, Bendall S, et al. Moderated online social therapy for young people with active suicidal ideation: qualitative study. J Med Internet Res. Apr 05, 2021;23(4):e24260. [FREE Full text] [CrossRef] [Medline]
    189. Fuhr K, Fahse B, Hautzinger M, Gulewitsch MD. [Implementation of an internet-based self-help for patients waiting for outpatient psychotherapy - first results]. Psychother Psychosom Med Psychol. Jun 2018;68(6):234-241. [CrossRef] [Medline]
    190. Schuster R, Fichtenbauer I, Sparr VM, Berger T, Laireiter A. Feasibility of a blended group treatment (bGT) for major depression: uncontrolled interventional study in a university setting. BMJ Open. Mar 12, 2018;8(3):e018412. [FREE Full text] [CrossRef] [Medline]
    191. Tregarthen JP, Lock J, Darcy AM. Development of a smartphone application for eating disorder self-monitoring. Int J Eat Disord. Nov 27, 2015;48(7):972-982. [CrossRef] [Medline]
    192. Yue H, Mail V, DiSalvo M, Borba C, Piechniczek-Buczek J, Yule AM. Patient preferences for patient portal-based telepsychiatry in a safety net hospital setting during COVID-19: cross-sectional study. JMIR Form Res. Jan 26, 2022;6(1):e33697. [FREE Full text] [CrossRef] [Medline]
    193. Ly S, Runacres F, Poon P. Journey mapping as a novel approach to healthcare: a qualitative mixed methods study in palliative care. BMC Health Serv Res. Sep 04, 2021;21(1):915. [FREE Full text] [CrossRef] [Medline]
    194. Baumel A, Yom-Tov E. Predicting user adherence to behavioral eHealth interventions in the real world: examining which aspects of intervention design matter most. Transl Behav Med. Sep 08, 2018;8(5):793-798. [CrossRef] [Medline]
    195. Baumel A, Kane JM. Examining predictors of real-world user engagement with self-guided eHealth interventions: analysis of mobile apps and websites using a novel dataset. J Med Internet Res. Dec 14, 2018;20(12):e11491. [CrossRef]
    196. Baumel A, Muench F, Edan S, Kane JM. Objective user engagement with mental health apps: systematic search and panel-based usage analysis. J Med Internet Res. Sep 25, 2019;21(9):e14567. [FREE Full text] [CrossRef] [Medline]
    197. Boucher EM, Raiker JS. Engagement and retention in digital mental health interventions: a narrative review. BMC Digit Health. Aug 08, 2024;2(1):e625. [CrossRef]
    198. Fujioka JK, Bickford J, Gritke J, Stamenova V, Jamieson T, Bhatia RS, et al. Implementation strategies to improve engagement with a multi-institutional patient portal: multimethod study. J Med Internet Res. Oct 28, 2021;23(10):e28924. [FREE Full text] [CrossRef] [Medline]
    199. Santos-Vijande ML, Gómez-Rico M, Molina-Collado A, Davison RM. Building user engagement to mhealth apps from a learning perspective: relationships among functional, emotional and social drivers of user value. J Retail Consum Serv. May 2022;66:102956. [CrossRef]
    200. Ferguson C, Low G, Shiau G. Resident physician burnout: insights from a Canadian multispecialty survey. Postgrad Med J. Jun 2020;96(1136):331-338. [CrossRef] [Medline]
    201. Lim R, Aarsen KV, Gray S, Rang L, Fitzpatrick J, Fischer L. Emergency medicine physician burnout and wellness in Canada before COVID19: a national survey. CJEM. Sep 2020;22(5):603-607. [FREE Full text] [CrossRef] [Medline]
    202. Healing the Healers: system-level solutions to physician burnout. Ontario Medical Association. 2021. URL: https://www.oma.org/siteassets/oma/media/pagetree/advocacy/issues/burnout/burnout-paper.pdf [accessed 2025-03-31]
    203. Position statement on physician burnout in Canada. The College of Family Physicians of Canada. Oct 24, 2022. URL: https:/​/www.​cfpc.ca/​en/​policy-innovation/​health-policy-goverment-relations/​cfpc-policy-papers-position-statements/​position-statement-on-physician-burnout-in-canada [accessed 2025-03-28]
    204. Cao DJ, Hurrell C, Patlas MN. Current status of burnout in Canadian radiology. Can Assoc Radiol J. Feb 06, 2023;74(1):37-43. [FREE Full text] [CrossRef] [Medline]
    205. Camacho E, Chang SM, Currey D, Torous J. The impact of guided versus supportive coaching on mental health app engagement and clinical outcomes. Health Informatics J. 2023;29(4):14604582231215872. [FREE Full text] [CrossRef] [Medline]
    206. Lipschitz JM, Pike CK, Hogan TP, Murphy SA, Burdick KE. The engagement problem: a review of engagement with digital mental health interventions and recommendations for a path forward. Curr Treat Options Psychiatry. Sep 25, 2023;10(3):119-135. [FREE Full text] [CrossRef] [Medline]
    207. van de Vijver S, Tensen P, Asiki G, Requena-Méndez A, Heidenrijk M, Stronks K, et al. Digital health for all: how digital health could reduce inequality and increase universal health coverage. Digit Health. Jul 07, 2023;9:628. [CrossRef]
    208. Strudwick G. People taking control of their own health information. Clin Integr Care. Apr 2023;17:100140. [CrossRef]
    209. Stepped care 2.0© e-mental health demonstration project. Mental Health Commission of Canada. URL: https://www.mentalhealthcommission.ca/sites/default/files/2019-09/emental_health_report_eng_0.pdf [accessed 2024-04-29]
    210. Simões de Almeida R, Marques A. User engagement in mobile apps for people with schizophrenia: a scoping review. Front Digit Health. 2022;4:1023592. [CrossRef] [Medline]
    211. Lo B, Pham Q, Sockalingam S, Wiljer D, Strudwick G. Identifying essential factors that influence user engagement with digital mental health tools in clinical care settings: protocol for a Delphi study. Digit Health. 2022;8:20552076221129059. [FREE Full text] [CrossRef] [Medline]

    ‎
    MeSH: Medical Subject Headings
    PRISMA-ScR: Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews
    RQ: research question

    Edited by T de Azevedo Cardoso; submitted 21.10.24; peer-reviewed by C Recsky, M Pulier; comments to author 06.12.24; revised version received 14.01.25; accepted 07.02.25; published 28.04.25.

    Copyright

    ©Brian Lo, Keri Durocher, Rebecca Charow, Sarah Kimball, Quynh Pham, Sanjeev Sockalingam, David Wiljer, Gillian Strudwick. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 28.04.2025.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research (ISSN 1438-8871), is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.