Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/24387, first published .
Barriers to and Facilitators of User Engagement With Digital Mental Health Interventions: Systematic Review

Barriers to and Facilitators of User Engagement With Digital Mental Health Interventions: Systematic Review

Barriers to and Facilitators of User Engagement With Digital Mental Health Interventions: Systematic Review

Review

Corresponding Author:

Judith Borghouts, PhD

University of California Irvine

6210 Donald Bren Hall

Irvine, CA, 92697-3425

United States

Phone: 1 9498240246

Email: jborghou@uci.edu


Background: Digital mental health interventions (DMHIs), which deliver mental health support via technologies such as mobile apps, can increase access to mental health support, and many studies have demonstrated their effectiveness in improving symptoms. However, user engagement varies, with regard to a user’s uptake and sustained interactions with these interventions.

Objective: This systematic review aims to identify common barriers and facilitators that influence user engagement with DMHIs.

Methods: A systematic search was conducted in the SCOPUS, PubMed, PsycINFO, Web of Science, and Cochrane Library databases. Empirical studies that report qualitative and/or quantitative data were included.

Results: A total of 208 articles met the inclusion criteria. The included articles used a variety of methodologies, including interviews, surveys, focus groups, workshops, field studies, and analysis of user reviews. Factors extracted for coding were related to the end user, the program or content offered by the intervention, and the technology and implementation environment. Common barriers included severe mental health issues that hampered engagement, technical issues, and a lack of personalization. Common facilitators were social connectedness facilitated by the intervention, increased insight into health, and a feeling of being in control of one’s own health.

Conclusions: Although previous research suggests that DMHIs can be useful in supporting mental health, contextual factors are important determinants of whether users actually engage with these interventions. The factors identified in this review can provide guidance when evaluating DMHIs to help explain and understand user engagement and can inform the design and development of new digital interventions.

J Med Internet Res 2021;23(3):e24387

doi:10.2196/24387

Keywords



Background

Nearly 1 in 5 adults in the United States experience a mental illness at some moment in their life [1]. Yet, accessing treatment for mental health problems can be difficult. Common barriers to mental health care include stigma, lack of available and evidence-based services, and inability to afford services [2,3]. In addition, people not diagnosed with a mental illness can experience periods of poor mental health and may benefit from support, although they have not sought professional treatment with a mental health provider. For instance, 73% of people surveyed in the United States experience stress related to money, work, and family responsibilities at a level that affects their mental health [4]. The translation of psychosocial interventions into digital formats, deemed digital mental health interventions (DMHIs), has the potential to overcome some existing barriers to traditional care and increase access to mental health support and resources.

DMHIs can be delivered via smartphone apps, internet websites, wearable devices, virtual reality, or video games [5] and range from self-guided DMHIs to those integrated with human support or traditional therapy [6]. Although some DMHIs have been shown to be as effective as traditional mental health services (eg, psychotherapy and pharmacotherapy) in improving mental health conditions such as depression [7] and can lead to greater reductions in anxiety compared with usual care [8], engagement with these technologies remains to be an ongoing issue, varies from study to study, and is typically lower in real-world use than research studies [9]. For example, a review in 2018 found that participant adherence to internet-delivered cognitive behavioral therapy (CBT) can range from 6% to 100% [10]. Similarly, systematic comparisons in 2018 and 2019 on self-help DMHIs found that real-world uptake varies widely [9,11], and acceptability can be lower than traditional treatment [12].

This paper aims to systematically review the literature on DMHIs to identify common barriers and facilitators that may influence user engagement with these interventions. There are different ways to define user engagement. For example, engagement can be referred to as the time a user spends on an intervention. However, the time spent on an intervention varies between different types of interventions, and little time spent using a DMHI does not have to be a negative feature per se. To get a comprehensive understanding of people’s use of DMHIs, we use a broader definition of user engagement. In this review, user engagement refers to a user’s uptake and sustained interactions with a digital intervention, which includes interest in adopting an intervention as demonstrated by signing up for the digital intervention, initial uptake as demonstrated by engaging with features of the digital intervention as part of the study, at a minimum during a demonstration as part of the study, and continued use of an intervention.

Understanding User Engagement With DMHIs

A range of factors can influence engagement with DMHIs, such as the relevance of information to the user provided by a digital intervention [13], a lack of user motivation to persist with a self-guided intervention [14], and poor user experience with the technology [15]. Although previous studies have each reported on some factors that can influence engagement, given a particular technology or context, a review is lacking that brings all these findings together. It is important to investigate the multitude of factors to fully understand the reasons for high versus low engagement. Previous reviews have highlighted the variability in engagement and uptake, analyzing both DMHIs published in the academic literature [9-11] and publicly available mental health apps in app stores [16]. However, these analyses did not report on factors related to this variability in engagement. This review seeks to address this gap by identifying the most common overarching factors that affect engagement.

Although analyzing engagement metrics of commercial apps can be used to examine variability in engagement, user studies are valuable to understand the underlying reasons why people may engage with some interventions more than others. For the purpose of this review, we focus on reviewing the academic literature.

Researchers and developers of DMHIs can use this knowledge to inform evaluations of engagement and the development of new digital interventions. In addition, it may provide insights into what services and facilitating conditions need to surround DMHIs to promote technology-enabled services and may help mental health service providers in selecting suitable interventions for their clients.

This review focuses on common mental health issues, such as depression, anxiety, psychological well-being and distress, and stress. There may be different barriers or facilitators for user engagement with other specific, serious mental illness interventions (eg, psychosis intervention) that are beyond the scope of this paper.


Inclusion Criteria

The inclusion and exclusion criteria of articles for this review are presented in Textboxes 1 and 2, respectively.

Inclusion criteria.
  • Report on an intervention aimed to improve mental health, psychological well-being, anxiety, depression, stress, and/or mood
  • Report on an intervention delivered in a digital format, such as a smartphone app or website
  • Report on some aspects of user experience (eg, usability, user satisfaction, and user feedback)
  • Report on factors that affected user experience
  • Include participants aged ≥16 years (eg, child and adolescent samples were excluded)
  • Report on an empirical study (eg, literature reviews that synthesized findings from other articles, columns, opinion pieces, comments or replies, and editorials were excluded)
  • Be a peer-reviewed article (eg, dissertations were excluded)
  • Be written in English
Textbox 1. Inclusion criteria.
Exclusion criteria.
  • Report on interventions that have a mental health component but do not have mental health as a primary intervention target (eg, an app that is primarily focused on physical pain symptoms, with a mental health component)
  • Report on interventions that only serve as an appointment booking system for in-person therapy
  • Report on interventions that are used as a component during an in-person session but cannot be used remotely outside of these sessions
  • Articles published before January 1, 2010
Textbox 2. Exclusion criteria.

The first exclusion criterion was added to identify barriers and facilitators that would be applicable to DMHIs. For example, a study that tests an app primarily focused on physical pain symptoms, with a mental health component, may find physical pain issues as a barrier to engaging with the app. It may not be clear from the study whether this is a common barrier related to DMHIs or interventions addressing physical pain.

The second and third exclusion criteria were added, as these types of interventions were designed to be a part of in-person sessions. It may not be clear from these studies whether users would be willing or able to engage with DMHIs apart from existing and traditional in-person sessions.

Finally, digital health interventions evolve rapidly [17,18], and the review was focused on the current state of DMHIs. Therefore, to avoid discussing on interventions or technologies that are now potentially out of date, the review was limited to contemporary studies published within the last 10 years (January 2010 to December 2019), a time frame that has been applied previously for systematic reviews on digital health technologies for mental illness [18].

Search Strategy

A literature search was conducted in multiple databases, including SCOPUS, PubMed, PsycINFO, Web of Science, and the Cochrane Library. On the basis of the inclusion criteria, a search query was developed to include an article if its title or abstract contained at least one keyword related to mental health, at least one keyword related to digital interventions, and at least one keyword related to user experience (Textbox 3; PRE/5 means that keywords were separated by a maximum of 5 words, for example, online PRE/5 intervention means there were 5 or less words between online and intervention).

The search query was built on keywords used in previous reviews on the uptake of mental health technologies [11,19], and additional keywords were added for the specific focus of this review (ie, the third part of the query with keywords related to user experience). The search terms for each database are included in Multimedia Appendix 1. Searches were not limited to the study design.

Search query.

TITLE-ABS-KEY ( depress* OR anxiet* OR anxious OR mood OR “mental health” 

OR “psychological wellbeing” OR “mental wellbeing” OR “behavioral health” OR 

“mental illness” ) 

AND TITL E-ABS-KEY ( ( online PRE/5 intervention* ) OR ( online PRE/5 

treatment ) OR ( digital PRE/5 intervention* ) OR ( digital PRE/5 treatment )

OR ( mobile PRE/5 intervention* ) OR ( mobile PRE/5 treatment ) OR ( smartphone PRE/5 intervention* ) OR ( smartphone PRE/5 treatment ) OR ( web-based 

PRE/5 intervention* ) OR ( web-based PRE/5 treatment ) OR ( internet PRE/5 

intervention* ) OR ( internet PRE/5 treatment ) OR ( computer PRE/5 intervention* ) OR ( computer PRE/5 treatment ) OR ( cyber PRE/5 intervention* ) OR ( cyber PRE/5 treatment ) OR ( electronic PRE/5 intervention* ) OR ( electronic PRE/5 treatment ) OR ( mobile AND program* ) OR mhealth OR ehealth OR mtherap* OR etherap* OR telehealth OR telemedicine OR “mobile app*” )

AND TITLE-ABS-KEY ( usability OR “user experience” OR evaluation* OR

engagement OR interface OR satisfaction OR usage OR adoption OR acceptability OR qualitative OR user perspective* OR barrier* OR interview* OR focus group* )

Textbox 3. Search query.

Study Selection

The search results were uploaded to Rayyan [20], a web-based software program for facilitating systematic reviews. Titles and abstracts were screened against the inclusion criteria, and excluded articles were labeled with reasons for exclusion.

The first author reviewed all titles and abstracts. Explicit inclusion criteria were determined between the first 3 authors a priori article selection to reduce coder bias. The coder (JB) was a PhD researcher with years of research expertise in user experience and thematic analysis.

A total of 6146 papers were extracted for the review. After the removal of 77 duplicates, 6069 article titles and abstracts were screened by the first author and discussed with the second and third authors. Uncertainties about inclusion were resolved by discussion among the first 3 authors, and reasons for exclusion or inclusion of these articles were discussed.

Furthermore, 480 full-text articles were reviewed, of which 208 met the inclusion criteria. Figure 1 shows a flow diagram of the screening papers. The same inclusion and exclusion criteria were used for reviewing the titles and abstracts in the screening phase and reviewing the full-text articles in the eligibility phase.

Articles that were not available were either not available on the web or were behind a paid firewall. Article types that were out of scope did not report on an empirical study.

Although there is a risk of bias in studies, the review considered all studies that met the inclusion criteria and included a large variety of different study methodologies, including qualitative studies with no reported quantitative outcomes. The primary focus of this review was to establish themes across the literature rather than extract the outcomes of quantitative studies. Therefore, the risk of publication bias with significant results is small compared with a meta-analysis of outcomes [21].

Figure 1. Flow diagram of article screening and inclusion.
View this figure

Data Extraction

A data extraction template (Multimedia Appendix 2) was developed for this review and piloted on 5 full-text papers. The main data elements extracted included reported factors, barriers, and facilitators to use and usage, such as retention and/or completion rate of the research study. The data were used to address the objective of the review to identify barriers and facilitators that influence user engagement.

Other extracted data were intended to document study and intervention characteristics, such as the type of technology and whether the intervention was publicly available, the target population, and the length of time that participants were able to engage with the intervention during the study.

Quality Assessment

To account for the methodological variety of studies, the quality of reporting tool by Carroll et al [22] was used to assess quality. This tool has been used earlier in systematic reviews that include qualitative and quantitative studies [23]. Using this tool, articles were assessed on 4 criteria: (1) was the study design explained, (2) was the recruitment and selection of participants explained (eg, random sampling and convenience sampling), (3) were details of the data collection method provided (eg, topic guides for interviews, number of items in a survey, use of open or closed items), and (4) were details of the analysis method provided (ie, form of analysis rather than merely reporting data were analyzed). Following the tool’s guidelines, studies were considered to be adequately reported if a “yes” was assigned to 2 or more of these criteria.

Analysis

An inductive thematic analysis [24] was used to identify common themes among these factors. This means that no preexisting coding scheme was used; rather, codes were created based on what emerged from the data.

We used a single coder approach, in which the first author iteratively identified codes from the data and refined themes throughout the analysis. Single coder approaches are methodologically sound when they include checks on validity and reliability [25]. For our analysis, validity and reliability were assessed by reviewing a selection of codes and their corresponding text with the second and third authors and by refining the codes. This process is common in qualitative research [26]. As we used emergent coding and there was no a priori codebook, a single coder approach also allowed for consistency of coding and interpretation of codes, and this approach has been used in systematic reviews [21].

The first author began the analysis by systematically reviewing each paper. For each paper, the following sections were analyzed: abstract, results or findings, and discussion. Individual codes were created each time a factor was described that affected engagement with DMHIs.

Factors were considered a barrier or facilitator if it was explicitly defined as a facilitator or barrier by the authors of the paper and/or the description in the paper pointed to it being a barrier or facilitator. For example, “participants reported they did not use mental health apps because they had privacy concerns on what would happen with their information.” In this instance, privacy concerns are identified as barriers.

A spreadsheet was used to keep track of the emerging codes. Each spreadsheet row corresponds to a single paper. The row contains the raw text of the paper that includes the identified factors and the initial codes. These codes were iteratively reviewed and compared with the raw text they were extracted from. Codes that referred to similar concepts, such as the ability to personalize an intervention and customize an intervention, were grouped together and given more descriptive names. As an understanding of the data was developed, earlier data were revisited to refine and combine codes, revalidating the previously coded material. Finally, the final codes were grouped into broader themes (eg, the roles of age, gender, and employment status were grouped into the theme demographic variables).


Study Characteristics

As seen in Multimedia Appendix 2, the 208 articles included in this review [27-237] reported on 2 types of user studies: (1) 69 studies were needs assessments that aimed to understand user needs and attitudes toward DMHIs without or before engaging with a specific intervention as part of a study and (2) 135 studies were evaluation studies that assessed users’ experience with a specific intervention over the course of the study. In total, 4 articles included both needs assessment and evaluation. Overall, 35 articles explored general user attitudes about DMHIs without focusing on a specific technology, whereas 173 studies focused on a specific technology (Table 1). Although all studies involved interventions for mental health, some studies focused on a particular area: 45 studies focused on depression, 22 studies focused on stress, 9 studies focused on anxiety, 6 focused on eating disorders such as bulimia nervosa, 4 studies focused on mood, and 2 studies focused on loneliness.

Table 1. Type of technology studied in included articles.
Type of technologyValues, n (%)a
Web based80 (38.5)
Smartphone based57 (27.4)
Computer based, but not web based9 (4.3)
Mobile phone (but not a smartphone)5 (2.4)
Wearable technology2 (1.0)
Tablet based2 (1.0)
Combination of technologies18 (8.7)

aNot all studies mentioned a particular treatment; hence, the percentages do not add up to 100%.

Measures related to user engagement included time spent using an intervention, number of log-ins, usability, acceptability, and feasibility. The usability and acceptability of the technology were assessed using qualitative methods and standard measures, such as the survey based on the Unified Theory of Acceptance and Use of Technology [27], the Mobile Application Rating Scale [28], and the System Usability and After-Scenario Questionnaire [29]. Feasibility was defined in these studies as either completion of a program offered through the intervention or retention rate, which is the number of people who completed the research study as a proportion of the people who started the study. In total, 42 studies employed qualitative interviews to understand people’s user engagement.

Factors that influenced user engagement were assessed through surveys (72/208, 34.6%), interviews (42/208, 20.2%), focus groups (34/208, 16.3%), randomized controlled trials (23/208, 11.1%), field studies (8/208, 3.8%), workshops (3/208, 1.4%), analysis of app usage data (7/208, 3.4%), and analysis of user reviews (2/208, 1.0%), using both qualitative and quantitative methods. For example, qualitative methods gathered subjective user perceptions of what formed barriers and facilitators for them to engage with interventions. A quantitative approach explored associations between variables, such as sociodemographic factors and intervention usage data, user satisfaction, and/or interest in using DMHIs.

The number of participants involved in these studies ranged from 6 to more than 2 million. In total, 6 studies conducted a secondary analysis of the usage data of an existing intervention or health database. For these 6 studies, the sample size was relatively large, ranging from 3158 to 2,171,325 users. Among the remaining 202 studies, the sample size ranged between 6 and 1558 users. For instance, 25% (52/208) of the studies had <18 participants, 49.5% (103/208) had <40 participants, and 75% (156/208) had <177 participants. The extent to which participants were exposed to an intervention ranged from a short demonstration before a focus group or survey to up to 1 year of usage.

Quality Assessment

All studies were assessed as adequately reported (Multimedia Appendix 3 [143,172]). Each study reported on the research question, study design, and method of data collection. Overall, 11 studies did not report the recruitment and/or selection process of study participants [30-40]. In addition, 11 studies did not specify the analysis method used to analyze the data [33,41-50]. One study reported on the analysis method for the quantitative data that were collected but not qualitative data [51].

Intervention Characteristics

Table 2 shows the types of technologies studied in the articles, and Table 3 shows the types of treatments and/or resources offered by the technology. Web- and smartphone-based interventions were the most common, reported in 38.5% (80/208) and 27.4% (57/208) of the papers, respectively. The most common type of treatment is internet-based CBT. Other treatments and features included acceptance and commitment therapy, psychotherapy, positive psychological interventions, meditation, peer support, resources, monitoring of symptoms, and journaling.

The target population included students, transitional age youth (aged 16-24 years), refugees, people who were homeless, veterans diagnosed with post-traumatic stress disorder, mothers with postpartum depression, patients being treated for a mental illness or another health concern, older adults, and caregivers and workers experiencing stress. Not all interventions specified the target population.

Table 2. Type of technology studied in included articles.
Type of technologyValues, n (%)a
Web-based80 (38.5)
Smartphone-based57 (27.4)
Computer-based, but not web-based9 (4.3)
Mobile phone (but not a smartphone)5 (2.4)
Wearable technology2 (1.0)
Tablet-based2 (1.0)
Combination of technologies18 (8.7)

aNot all studies mentioned a particular treatment; hence, the percentages do not add up to 100%.

Table 3. Type of treatment and resources offered.
Type of treatment or resourcesValues, n (%)a
Cognitive behavioral therapy30 (14.4)
Informational or educational resources23 (11.1)
Counseling17 (8.2)
Self-tracking tools (eg, journaling, monitoring symptoms)12 (5.8)
Mindfulness9 (4.3)
Acceptance and commitment therapy8 (2.9)
Peer support (eg, peer chat)7 (3.4)
Text messaging (eg, reminders)4 (1.9)
Positive psychology interventions3 (1.4)
Prolonged exposure therapy1 (0.5)
Passive data collection1 (0.5)
Combination of treatments and/or resources40 (19.2)

aNot all studies mentioned a particular treatment; hence, the percentages do not add up to 100%.

Constructs Associated With User Engagement

Textbox 4 shows the high-level constructs derived from the thematic analysis influencing user engagement with DMHIs, where the numbers in parentheses show the number of articles in which the constructs were identified. We caution that the most frequently occurring constructs are not necessarily the most important but rather indicate that more studies have reported on this topic. Table 4 summarizes the main findings associated with each construct. After several iterations of grouping and coding, 16 larger groups remained: demographic variables, personal traits, mental health status, beliefs, mental health and technology experience and skills, integration into life, type of content, perceived fit, perceived usefulness, level of guidance, social connectedness, impact of intervention, technology factors, privacy and confidentiality, social influence, and implementation. These themes fit into 3 categories: user-related factors, program-related factors, and factors related to the technology and implementation environment. The next section provides more detailed explanations. The full list of factors belonging to each construct is included in Multimedia Appendix 4.

The constructs influencing user engagement, grouped as constructs related to the user, the program offered by the intervention, and the technology and (implementation) environment. The numbers in parentheses indicate the number of articles in which the constructs occurred.

User

  • Demographic variables (31)
  • Personal traits (5)
  • Mental health status (59)
  • Beliefs (55)
  • Mental Health and Technology Experience and Skills (33)
  • Integration into life (42)

Program

  • Type of content (54)
  • Perceived fit (61)
  • Perceived usefulness (35)
  • Level of guidance (40)
  • Social connectedness (53)
  • Impact of intervention (62)

Technology and environment

  • Technology factors (100)
  • Privacy and confidentiality (47)
  • Social influence (16)
  • Implementation (39)
Textbox 4. The constructs influencing user engagement, grouped as constructs related to the user, the program offered by the intervention, and the technology and (implementation) environment. The numbers in parentheses indicate the number of articles in which the constructs occurred.
Table 4. Summary of findings for each construct.
ConstructSummary of main findings
User-related constructs

Demographic variables (sociodemographic factors, such as age, gender, and education)Overall, women were more likely to engage with DMHIsa than men

Personal traits (factors related to personality traits, such as neuroticism and extraversion)The personality traits neuroticism, agreeableness, openness, and resistance to change were associated with higher engagement, whereas extraversion was associated with lower engagement

Mental health status (factors related to the current mental health status of the user, such as the type and severity of symptoms)Severity of mental health symptoms increased the interest in DMHIs, but symptoms related to depression, mood, and fatigue were a barrier to actual engagement

Beliefs (beliefs held by the user with regard to technology, mental health, and mental health services)People’s positive beliefs about mental health help-seeking and technology-facilitated engagement

Mental health and technology experience and skills (previous experience the user has had with technology, mental health technology, and mental health services and skills related to their digital or mental health or digital health literacy)Digital health literacy and positive experiences with mental health services and technology were facilitators to engagement

Integration into life (the extent to which the user is able to find time and space to use the intervention and make the intervention part of their routine or life)Engagement was facilitated if people were able to integrate DMHI use into their daily lives
Program-related constructs

Type of content (the type of content and features offered by the intervention)Engagement was facilitated if content was credible and if activities offered by the DMHI were of an appropriate length (ie, not too short or too long)

Perceived fit (factors related to how well the intervention is appropriate to the user’s culture and values and is adaptable to the user’s needs rather than a one-size-fits-all solution)Engagement was facilitated if information offered by a DMHI was customizable and relevant to the user

Perceived usefulness (factors related to expected benefits of using the digital intervention over existing resources)Participants were more likely to engage with DMHIs if they understood the data and knew how to use it

Level of guidance (the level of guidance offered by the intervention on how [eg, when, how often] to use it, for example, through notifications or a coach)Guided interventions, either through a human therapist or automated reminders to use a DMHI, had higher engagement than unguided interventions

Social connectedness (the extent to which the intervention connects or isolates the user with or from others)Being able to connect with other people through a DMHI facilitated engagement

Impact of intervention (the impact that intervention usage had on the user, such as an improvement or exacerbation of mental health symptoms [as measured by a validated survey scale])DMHI engagement was facilitated if participants experienced a positive impact as a result of using a DMHI, such as the improvement of symptoms
Technology- and environment-related constructs

Technology-related factors (factors related to the technology through which the intervention is offered, such as the resources and costs required to use it, usability, and technical issues experienced by the user)Technical issues were a common barrier to engagement

Privacy and confidentiality (factors related to data security, storage, confidentiality, and privacy of the digital intervention)Engagement was facilitated if participants had a sense that the digital platform was private and anonymous, and they could safely disclose information

Social influence (factors from the users’ social environment, such as perceptions held by their peers, family, and health care provider, that influence their intention to use an intervention)Participants were more likely to use DMHIs if people close to them, such as family and friends, thought they should use DMHIs

Implementation (factors related to the implementation of the intervention that affects use, such as the availability of user training, the phase of the user’s mental health care–seeking process during which the intervention is introduced or accessed and characteristics of the health care organization supporting the DMHI)DMHI engagement was facilitated if people were trained on how to use it

aDMHI: digital mental health intervention.

User-Related Constructs

User-related factors refer to factors related to the user, such as personal beliefs, skills, and experiences.

Demographic Variables

Some demographic variables were found to be associated with DMHI engagement. Studies that found an effect of gender showed that women were more likely to adopt and engage with interventions [44,52-68]. Overall, 8 studies saw an effect of age: 2 studies found that people aged ≤50 years engaged more with interventions than older adults [66,67]. These 2 studies used a relatively large sample size (1,139 and 2,171,325), and participants were exposed to the intervention for up to 1 year. A total of 6 studies found that higher engagement was seen with adults aged ≥30 years [54,57,64,65,69-71]. These studies had a smaller sample size (samples ranged from 74 to 577 people), and participants engaged with the intervention for shorter periods (up to 12 weeks). Age was also found to influence interest and expectations: users’ interest in using digital therapy interventions increased with age [72], and Krause et al [58] found that older people have higher expectations of interventions.

Chudy-Onwugaje et al [73] found that age has an interacting effect with people’s depression symptoms. For people aged ≤40 years, adherence increased with depressive symptoms, but there was no association between depressive symptoms and adherence in people aged >40 years. Although the reasons for this interaction were unclear from the study, the authors of the article theorize that the effect of symptoms may interact with familiarity with technology, with younger people being more comfortable using technology.

Other demographic variables associated with user engagement were as follows: (1) employment status, with people who worked full time more likely to use the intervention than people who were retired [66] or unemployed [54,68]; (2) education, with participants with higher education reporting more acceptance of interventions than people with lower education (a high school diploma or lower) [74-76]; and (3) housing situation, with people who were experiencing homelessness responding less to messages sent by a phone intervention compared with individuals with stable housing [55].

Personal Traits

Certain personality traits were associated with willingness and interest in using DMHIs. People who scored high on neuroticism and agreeableness of the Big 5 personality traits were more interested in using smartphone apps to reduce stress [77]. In a different survey reported in the same article, neuroticism was strongly linked to self-reported stress. The cooperative nature of agreeable people made it easier to accept new technology.

In addition, extraversion was a predictor of lower likelihood to prefer web-based mental health services over in-person services [72]. People who scored high on extraversion preferred to meet and connect with a doctor in person. Other personality characteristics associated with user engagement were resistance to change and openness to experience [56]. Higher openness predicted higher engagement with mindfulness and relaxation interventions. Contrary to the hypothesis made by the authors of the article that higher resistance to change would lead to resistance to adopting a new health behavior, higher resistance instead predicted higher adherence. Once people started using the intervention, a higher resistance to change facilitated commitment to continue using the intervention.

Mental Health Status

A total of 59 studies reported that people’s mental health status plays a role in participants’ interest in and use of a digital intervention. First, certain mental health symptoms appeared to inhibit people’s motivation and/or ability to interact with an intervention. Depressive symptoms [78] and low mood [79], as measured by validated scales, have been reported as barriers for people to access and use web-based resources. Study participants reported that feeling tired also negatively affected their motivation and ability to use an intervention [44,80]. Second, the severity of these symptoms was related to engagement with digital interventions. In needs assessment studies, participants were more willing to use DMHIs if their symptoms were more severe [38,53,62,71,81,82]. However, evaluation studies have shown that more severe symptoms hamper actual engagement with digital interventions [51,56,83-101]. Depending on the type and severity of a person’s mental health symptoms, studies that involved health care providers supporting digital intervention use reported that there was sometimes a need for face-to-face contact, as issues could be difficult to address remotely via a digital platform [102-104].

Beliefs

Beliefs refer to preexisting beliefs the user has about mental health help-seeking [88], their need for help [51,105-107], the acknowledgment of having mental health needs [88], and using technology for mental health treatment [38,93,108,109]. For example, preexisting beliefs of needing help for mental health needs and having a positive perception about mental health help-seeking facilitated participants’ engagement with an intervention. However, even if people acknowledged a perceived need for help and were willing to seek help, engagement with a particular intervention was then affected by a person’s preconceived belief about whether a digital intervention would be effective [79,104,110-112]. In 2 studies, participants did not want to use a digital intervention because technology was seen as a stimulant and distracting [113,114].

Mental Health and Technology Experience and Skills

A positive prior experience with technology [115,116], mental health services [54,74,104,117-119], and mental health technology [72,120,121] facilitated people’s intention to use interventions, as well as actual engagement. A negative prior experience formed a barrier to engaging with a DMHI [74,117,122], whereas a positive experience increased participants’ engagement [54,72,104,115,116,118-121].

Mental health literacy refers to knowledge about mental health symptoms and appropriate treatment options [238]. Digital literacy refers to the skills required to use technology [239]. Digital health literacy refers to the ability to use technology to find and use health resources [240]. Participants’ mental health literacy [123], digital literacy [88,103,104,122,124-127], and digital health literacy [103,127] influenced the extent to which they were able to adapt and engage with DMHIs: for each type of literacy; higher literacy was associated with higher engagement.

Integration Into Life

Users reported that their engagement was affected by the extent to which they were able to integrate an intervention into their daily lives. Barriers that limited use included that participants felt they lacked time [44,128-131] or constantly forgot to use an intervention [93,95,129], participants felt the intervention took too much time to use [132-134], and participants experienced difficulties establishing a routine of use that worked for them [130,135].

Access to a private space to access mental health resources also affected the extent to which participants could integrate an intervention into their lives. In 3 studies, participants mentioned that as opposed to going to a health care provider office, it was challenging to find a private space at home or work to use an intervention, which formed a barrier to engaging with it [136-138].

Studies have also found difficulties among users in integrating the information and tips offered by the intervention into their lives. For example, Jonathan et al [139] evaluated a smartphone app for people with serious mental illness. Participants who spent most of their day indoors without leaving their house had a hard time trying to use the tips in actual real-life scenarios.

Summary of User-Related Constructs

In summary, user engagement with DMHIs is partly influenced by factors related to the users themselves. Demographic variables such as age, gender, employment, education, and housing situation can affect user engagement. The personality traits neuroticism, agreeableness, openness, and resistance to change facilitated engagement, whereas extraversion was a barrier.

If mental health symptoms were more severe, participants were more interested in using DMHIs, but symptoms related to depression, low mood, and tiredness prevented engagement. People’s beliefs about and past experiences with mental health services and technology were facilitators if these beliefs and experiences were positive, and they formed a barrier if these beliefs and experiences were negative. Participants’ literacy in understanding mental health and using technology facilitated their ability to use DMHIs, and any further engagement depended on the extent to which people were able to integrate it into their daily lives.

Program-Related Constructs

The second group of constructs is related to the type of therapy or content offered through the DMHI.

Type of Content

Higher satisfaction with the type of content and features offered increased user engagement. Uncertainty about the credibility of the information, which related to the evidence base of the intervention and the source of information, was a barrier [74,88,127,140-143]. Other factors related to the modality through which content was delivered, with some participants preferring to have audio or video options in addition to text-only information [144] and whether the content was considered by users to have a supportive, nonjudgmental tone [51,145,146].

Some interventions offered programs of a fixed length or time commitment, such as a CBT program consisting of 8 weekly sessions. The length of the program as well as the length of individual sessions played a role in participants’ satisfaction and their motivation to continue with the program [88,138,147-151]. In 2 studies evaluating a self-guided CBT program for 8 weeks, the length and pace of modules negatively impacted user motivation [88], and participants reported preference for more concise modules [148], although the articles did not attempt to identify an ideal module length. In other studies evaluating a CBT program that included in-person sessions with a therapist, some participants reported preference for both longer individual therapy sessions (greater than the standard 50-70 minutes) [151] and duration of treatment to maximize benefit [150].

Perceived Fit

Perceived fit refers to the extent to which users felt the intervention was appropriate and relevant to their culture and values and/or targeted to people similar to them, rather than a one-size-fits-all solution. This fit was, for example, facilitated by relevance of information to their current situation [14,46,150,152-157] and the ability to customize or personalize the intervention [30,46,83,84,122,134,135,138,158-165]. A facilitating factor was whether users were able to identify with the people presented in the intervention [166], which could be coaches or instructors, or examples of people with similar experiences. Factors that make the information relevant and in a language suitable to the user included culturally appropriate content [133,167,168], reading level suitable to the user [168], and content presented with limited jargon or technical language [169].

Perceived Usefulness

Perceived usefulness refers to the user’s experience with an intervention and their perceptions of whether the intervention would be useful to them. This perception was facilitated by whether users were able to understand the data presented to them [104,117,170], whether it was clear what action they should take [129,133,154,155,166], and whether the intervention provided a clear advantage over past or current care received [103,117,121,155,171]. Identified facilitators were easier access to services that users would otherwise not have access to [103,173] and the eliminated need to travel a long distance to a health center [121].

Level of Guidance

The level of guidance refers to the extent to which users were guided to use an intervention, for example, through reminders or a web-based supporter, holding them accountable to regularly engage with the content. A facilitating factor in using DMHIs was whether the use of the intervention increased locus of control, meaning that users felt more ownership over their own health [14,84,95,124,174,175]. However, for interventions that were completely self-guided, participants experienced difficulty engaging with them and at times neglected to use the intervention [44,95]. Participants expressed a need to receive more structured use, for example, through app reminders or a human coach checking on them on a regular basis [49,50,113,122,133,137,139,148,150,163,175-182]. In 6 studies, users stated that they would prefer if an intervention served as a complement to existing, in-person therapy rather than a replacement for in-person therapy [30,122,134,139,166,183].

Social Connectedness

The effect that an intervention had on participants’ sense of social connectedness was found to facilitate user engagement. For example, being able to connect to peers or have regular contact with a personal therapist through DMHIs facilitated engagement in 18 studies [14,32,33,104,114,122,125, 133,156,184-194]. In 6 studies, a noted barrier among both users and service providers was a concern about social avoidance, that is, a concern that people might use self-guided interventions in lieu of coming into a clinic in person and engaging in therapy or group sessions [78,104,122,133,138,195]. For therapy interventions, where study participants were introduced to therapists who they did not know before using the intervention, the extent to which participants could connect emotionally (therapeutic alliance) with the therapist-influenced engagement. Participants’ ratings of the quality of the emotional connection were positively related to the number of log-ins, frequency of self-monitoring mood, and completion of therapy [196].

Impact of Intervention

Participants reported that the perceived changes they experienced in their mental health as a result of using an intervention affected their further engagement. Perceived symptom improvement facilitated further engagement [89,95,103,145,146,189,197-199], whereas exacerbation of symptoms negatively impacted engagement [49,93,104,200]. Other negative impacts of the intervention were also observed as barriers to ongoing user engagement. For example, in 2 studies, some information that was shared within the digital intervention was found to trigger difficult memories or emotions [150], participants were uncomfortable with exercises or information [80], or participants were exposed to negative comments by other users [150].

Another facilitating factor was whether the intervention normalized people’s experiences [79,139,157,190,201], for instance, by providing examples of other people with similar experiences. Additional positive effects that could facilitate engagement were an increased insight into users’ health [79,84,114,124,134,138-140,158,176,202-205], a feeling of empowerment over being in control of their health [14,32,84,93,95,104,117,122,124,170,174,175,196,206,207], improved skills [89,189,198,208-211], such as managing negative emotions, and an improvement of participants’ existing relationships with others [182,205].

Summary of Program-Related Constructs

In summary, the content offered by a DMHI had to be credible and ideally offered in more than one modality. Participants engaged with DMHIs if they felt the intervention was a good fit, which could be facilitated if content was relevant, and the DMHI was customizable, culturally appropriate, and used a language that was understandable to the participant. Engagement was facilitated by participants’ perception of whether a DMHI was useful, which included whether they were able to understand the data and how to use it, and whether a DMHI provided a clear advantage over resources they already had access to.

Guided DMHIs had higher engagement than unguided interventions, and participants liked being able to connect with other people, although some studies identified concerns that DMHIs could be used to avoid in-person contact. The negative and positive impacts of DMHI use could form barriers and facilitators, respectively, to further engagement.

Technology- and Environment-Related Constructs

The third group of constructs refers to factors related to the technology itself or the implementation of the technology.

Technology-Related Factors

Technology-related factors refer to factors related to the technology through which the intervention was offered. The primary barrier to engagement noted in 25 studies was users’ experience of technical issues [44,50,80,92,100,103,118, 129,138,155,172,179,185,195,205,208,212-220], such as a mobile app crashing and shutting down unexpectedly; in 3 studies, participants did not have the resources required to use an intervention [171,221,222]. In 7 studies, participants expressed concerns over the eventual costs associated with using an intervention [85,93,104,123,165,223,224]. Costs could be related to the need for a smartphone, having internet access, or making purchases through the app. Usability issues formed a barrier to engaging with an intervention [46,50, 78,84,148-150,157,159,170,224-228]. Examples of usability issues were difficulty finding information in an intervention [78], a time-consuming process to log in to an intervention [159], and difficulty navigating within an intervention [150,157].

In addition to technical issues that formed barriers to engagement, there were also factors related to technology that facilitated the use of mental health resources and support. Facilitating factors made possible by the technology used were the flexibility of being able to access resources at any location [47,127] at any time [41,93,97,124,129,134,167,229-233] and having a temporal record of health data, such as symptoms, that users were able to track and access over time [176,180].

Privacy and Confidentiality

Privacy and confidentiality relate to how data were stored and shared and whether users felt safe and comfortable to disclose confidential information through an intervention. In 2 studies, participants were uncomfortable about their physical location being recorded [180,234], and in a study by Nicholas et al [234], participants were more comfortable with health information being recorded such as sleep and mood than personal data being recorded such as social activity and communication logs.

Accessing mental health resources via a digital platform raised concerns regarding privacy. Facilitators of user engagement and feeling safe to disclose information included assurance that the digital platform was private and participants’ information could not be easily accessed by third parties [129,158,205,235].

Participants in 5 studies expressed that concerns about confidentiality formed a barrier to engagement [51,104,127,236,237]. A facilitator to create a safe environment was moderation of the intervention [140], which means that a person was monitoring and moderating the content shared by and between users within an intervention.

Anonymity was found to be both a facilitator and a barrier to engagement. Overall, 7 studies listed anonymity, meaning that users could share and receive information anonymously, as a facilitating factor to engage and encourage disclosure of information [41,88,129,137,141,148,232]. However, anonymity could also make it more difficult for participants to trust a coach who they did not know [85,127]. In these studies, participants interacted with the coach through text, and there was the option to disclose names, but neither side could see each other. Other study participants were concerned about whether an intervention was truly anonymous if it was used in a small setting, with a limited number of known users [137]. Anonymity was also more important for people who were older and who had previous experience with medical treatment [127].

Social Influence

Users’ engagement was facilitated by whether the intervention was endorsed by other users [43] or peers [211,241], their friends and family [97], or their current health care provider [152,161,242]. However, if participants felt forced by others to use an intervention, it deterred them from using it [103]. If an intervention was used as part of ongoing in-person therapy, the way that therapists used or were willing to use an intervention influenced participants’ engagement with an intervention [84,103,127,150,169,212,243]. The adoption of an intervention as part of therapy depended on the therapist’s digital literacy skills [244,245], their past experience with mental health technology [120], and the ability to easily integrate its use into their practice as a provider [132,195,205,211,224,226,246].

Implementation

Although most studies in this review (93%, 194/208) primarily focused on factors related to the user and the intervention itself, 14 studies also described factors related to the implementation of the intervention. Examples included whether users received training on how to use the intervention [115,247] and if it was introduced early on or at a later stage in ongoing therapy. Participants in a study by Graham et al [68] used an intervention to support their mental health while in treatment for substance use. These participants found the intervention more useful at a later stage, as they felt the user was likely more familiar with their health and better able to make sense of the information provided by the intervention. Two other studies found that participants engaged more with an intervention if they were just starting treatment [104,178]. Two studies found that the way in which the intervention was labeled and introduced to users also mattered. For example, the term mental health was disliked by participants [248], and participants reported that they would be more likely to use an app if it was meant for well-being and mental fitness rather than mental health [37]. Other implementation factors were administrative barriers [42,118,129,211] and barriers related to the organization in which the intervention was or would be implemented [118,122,135]. Examples of administrative barriers were inadequate staffing and poor communication among staff members. An example of organizational barriers was a lack of support for DMHIs among managers.

Summary of Technology- and Environment-Related Constructs

In summary, although DMHIs introduced technical and usability issues that could form a barrier for participants to engage, the digital format also provided flexibility to access resources anywhere at any time and to have a record of health data. It was important that information was private and that participants could safely disclose information anonymously, although complete anonymity also made it more difficult to trust other people on the platform. Negative and positive opinions held by other people about DMHIs could form a barrier and facilitator, respectively, to engagement, and if DMHIs were to be used as part of ongoing therapy, the therapists’ past experience with DMHIs and the ability to integrate it into their practice played a role in user engagement. Finally, successful implementation facilitated user engagement. Providing training on how to use DMHIs and labeling an intervention for well-being or mental fitness (as opposed to mental health) can help users engage with DMHIs more. Participants may be more engaged with DMHIs if they are just starting treatment, but the identified benefit of introducing DMHIs at a later stage is that users may be more knowledgeable about their health and better able to make sense of their health information.


Principal Findings

This study aims to synthesize the literature on DMHIs and summarize the identified factors affecting user engagement with DMHIs. This review identifies 3 key areas that all contribute to DMHI engagement: (1) user characteristics, such as severe mental health symptoms, can form a barrier to engagement; (2) users’ experience of the program or content, with participants more likely to engage if they perceive the program to be useful and a good fit to them; and (3) the technology and implementation environment, such as technical issues being a common barrier to engaging with DMHIs. Providing content that is relevant and customizable according to personal preferences and offering technical assistance and/or training are important to achieve engagement. However, although these considerations may increase interest and uptake of DMHIs, it is important to understand whether characteristics specific to the user, such as their symptoms, will affect motivation to engage with these interventions. We first discuss the 3 key areas in more detail in the following three subsections; compare our constructs with other models on user engagement; and then discuss implications for researchers, developers, and health service providers.

User Constructs

Individual differences among users can affect engagement, including demographic variables such as age and gender, personality traits, mental health status, beliefs about mental health and DMHIs, experience with technology and mental health, and people’s ability to integrate DMHI use into their lives. Although the severity of symptoms may increase interest in engaging with health interventions [249], symptoms related to depression, mood, and tiredness were found to hamper actual engagement. This contrast may point to the unique implications that mental health symptoms can have on engagement with DMHIs.

The contrasting role of symptom severity between studies highlights the importance of understanding how people who would be more interested in DMHIs and may benefit more from its use are not limited by their symptoms to actually engage with these interventions. The contrast also illustrates the importance of including users at various stages of the design process, as people may be interested in the concept of a DMHI but may not be able to actually engage with it because of the nature of their symptoms.

Although studies looking at DMHI usage over 1 year found that younger people were more engaged with DMHIs, shorter research studies (ie, up to 12 weeks) found that older people were more engaged. Potentially, older adults perform better on study adherence, and younger people continue to engage more with an intervention long term, although the different interventions and settings make it difficult to make a direct comparison between these studies.

Program Constructs

Engagement with DMHIs was facilitated if participants liked the type of content; they perceived a DMHI to be a good fit for them and perceived it to be useful; there was a level of guidance on how to use it, it facilitated social connectedness, and it had a positive impact, such as improvement of symptoms.

Level of Guidance

Guided interventions typically have higher engagement than unguided interventions. However, human guidance can be resource intensive, and it may not always be possible or feasible to provide the desired level of guidance. Although human support enhances engagement more than automated means such as email reminders [250], several studies included in our review found that such automated reminders not only facilitated engagement but were also experienced positively by users. Automated reminders to use an intervention may therefore be a low-cost alternative to human support. The benefits of automated reminders may depend on the type of support and the type of barriers they are designed to address. Short text-based reminders may be suitable for in-the-moment interventions [251] and may be useful to address barriers to forgetting to use an intervention. On the other hand, human support may be more suitable for addressing the lack of motivation and facilitating social connectedness.

Furthermore, appropriate time commitments differ for self-guided exercises versus guided sessions. Participants across studies preferred shorter self-guided modules but longer guided therapy sessions. Finally, personalization may also meet different preferences. People who find videos or text-based material time consuming may be more engaged with shorter actionable exercises, whereas people with a preference for synchronous communication may engage more when they get dedicated time on one-on-one sessions. It would be worthwhile to further explore how engagement can be encouraged in self-guided interventions.

Social Connectedness

An important facilitator was whether a DMHI facilitated social connectedness and enabled the user to interact with other people. Previous work has shown that social support through social networks not only increases engagement but may also have a positive effect on depression symptoms [221,222]. However, in some studies in this review, mental health service users and providers were concerned that technology would facilitate social avoidance if people were to use a digital intervention in lieu of engaging in face-to-face individual or group therapy. It appears that it is important that an intervention allows users to connect with other people with whom they may have otherwise not connected, rather than replacing any existing face-to-face contact. For example, people can access a mental health app if they are not able to speak to someone in person about their concerns [175].

Technology and Environment Constructs

Offering mental health resources through technology offers both barriers and facilitators. Technical issues and concerns about privacy were common barriers, but technology also offered flexibility and could facilitate anonymity. Furthermore, the environmental context in which DMHIs are to be used are important to consider. Participants were more likely to use DMHIs if people close to them thought they should use it and if they received training on how to use it.

Anonymity

Anonymity was a prominent topic among studies but engaging with an intervention anonymously was seen as both a barrier to and facilitator of engagement, sometimes within the same study. This difference can be explained by factors related to the user, the type of implementation setting, and the type of intervention features that were anonymous, as outlined in the following paragraph.

First, a facilitating aspect of an anonymous intervention was that study participants found it less stigmatizing than seeing a live or in-person therapist. Anonymity may be an important facilitator for people who have experienced stigma and embarrassment, which is known to be a barrier to help-seeking for mental health concerns [2,3]. Similarly, prior work on mental health discourse on the web found that anonymity does not hinder the social support that people receive on their posts, which can facilitate open conversations, and that social media may be particularly useful for stigmatic illnesses such as mental health [252]. Second, the study setting matters. Interventions that are used in a relatively small setting may give a false sense of anonymity if it is possible for users to find out who else is using the intervention, for example, through content shared within the intervention or by seeing someone use it [108], which is important to consider for intimate settings, such as schools, workplaces, or small communities. Third, on community forums, where users could share their experiences and comment on other users’ posts, overall anonymity was seen as a facilitator to safely disclose information. In one-on-one sessions, however, where the user interacted with a coach or therapist but neither side could see each other, anonymity made it more difficult to establish a relationship and trust compared with a face-to-face session.

These differing perceptions shed light on an important trade-off. Should an intervention strive to be anonymous to address stigma and potential embarrassment or focus on allowing people to establish a trusted relationship with someone? This decision may depend on the objective of the intervention and whether anonymity is possible in the context in which it is to be used. Alternatively, a hybrid form or multiple options can be considered and offered. For example, forums with a larger number of users can be anonymous, whereas a private one-on-one session with a therapist can include telehealth options to allow for therapeutic alliance building between the user and therapist. The Supportive Accountability Model [250] also proposes that engagement is enhanced if human coaches are seen as trustworthy and that users may disclose more in computer-mediated than face-to-face communication. Although Mohr et al [250] argue that providing additional information about individuals, such as photographs, may reduce these types of benefits of this type of mediated communication, it may be important to establish initial trust with a coach or therapist. Additional research is needed to understand how best to support trust in DMHIs.

Privacy

A previous review of user engagement with mental health apps theorized that one reason for low engagement is that these apps do not consider user privacy [15]. In our review, privacy was discussed in terms of data storage and sharing but also with respect to the physical environment in which these interventions were to be used. Delivering mental health support through a digital platform was found to increase a sense of privacy in some studies, but in other studies, it was associated with a decreased sense of privacy. The study participants stated that they could access care more privately, without anyone knowing about it. In line with previous work [2], this again indicates that privacy can be important for people who experience stigma for or reluctance to help-seeking. Although participants’ living situation was not explicitly discussed in these studies, when compared with other studies, it is likely that participants were able to engage with these interventions in physically private settings. In other studies where people did not have access to a private space, a lack of privacy was a barrier to engagement. For example, study participants evaluating an app that delivered remote web-based therapy felt that they could disclose more in a closed therapist office than through a web-based intervention at home where other people in their household could see or disrupt them [136]. Study participants who used a mental health intervention in the workplace [137] said privacy was not possible, as colleagues could see what someone was doing at their desk and when they were interacting with the intervention.

These differing experiences highlight that technology can overcome existing privacy barriers of seeking mental health care but can also introduce other privacy issues, and users’ situational context (ie, where they are physically accessing the digital intervention) should be taken into account.

Comparison With Other Models of Technology and Digital Health Intervention Engagement

Some of the themes identified in this review overlap with previous models conceptualizing engagement with digital health interventions, as well as general technology acceptance and health behavior, such as the Efficiency Model of Support [251], Technology Acceptance Model [253], and Health Belief Model [249].

For instance, the Efficiency Model of Support [251] states that human support increases engagement in the context of the use of digital health interventions when it addresses 1 of 5 failure points: usability, implementation, fit, engagement, and knowledge. These broadly map to our constructs of technology factors, integration into life, perceived fit, beliefs, and experience, and skills. The implementation failure point in the efficiency model pertained to whether the user can apply knowledge gained from an intervention in their lives. Our review extends this concept, in that we found that an important issue is whether users can integrate the actual use of the intervention into their everyday routines.

Our findings are in line with the Technology Acceptance Model, which explains that users’ decisions to accept and use a technology are influenced by perceived usefulness, ease of use, and social influence of others. The Health Belief Model explains that adoption of health interventions is, among things, influenced by a person’s belief in the severity of their illness or health symptoms and the perceived benefits of seeking treatment for these symptoms, which map onto our constructs of beliefs and impact of the intervention. Themes revealed in this review, which have not been highlighted in these previous models, are the level of guidance, integration into life, and social connectedness. This gap may be explained by the way in which mental health interventions were intended to be used. To be effective, most DMHIs were intended to be used regularly by users on their own. This characteristic introduces the challenge for people to integrate it into their routine and have the discipline to use it regularly; therefore, the level of guidance provided within the intervention may have a particularly salient effect on engagement. Social connectedness may be especially important for mental health interventions, as it can improve mood [254] and help combat depression [255].

Implications

In this review, we have synthesized the literature on DMHIs to identify common factors influencing user engagement. This synthesis can be described as follows.

  • Researchers can use these factors to develop constructs that are important to measure when evaluating DMHIs. More concretely, it is important to capture user characteristics, users’ experience of the program and content, and details regarding the implementation setting. These constructs may help explain why someone would use one DMHI over another and may help evaluate how engaging a DMHI will be.
  • Developers can use these factors to facilitate engagement with DMHIs. Specifically, when developing a DMHI, it is important to understand the specific characteristics of the target audience, for example, if the severity of the audience’s symptoms can form a barrier to engagement; to tailor the program to the audience, such as offering the option to customize content; and to address issues related to the technology and environment, for example, by mitigating technical issues and providing technical assistance.
  • Mental health service providers, such as clinicians, can use this overview as guidance to select interventions that are appropriate for their clients or help guide their clients in selecting suitable interventions. For example, it is important to consider whether an intervention can be easily integrated into clients' lives and routines. In addition,Multimedia Appendix 2, which shows the full data set, can be used to filter the study setting, target population, and symptoms to see which barriers and facilitators have been observed for similar settings and populations.

The themes highlighted in this review identify factors that can facilitate engagement and barriers that should be considered to facilitate the successful implementation of a digitally mediated mental health intervention.

Limitations

We did not limit this review to particular study designs. As such, this review takes a much broader look at what factors influence engagement with digital mental health technologies rather than focusing on a single research method or technology. However, because of the heterogeneity of the included studies, we were unable to conduct a meta-analysis. In addition, there was inconsistency across studies in measures used to assess user engagement, such as the number of log-ins to an intervention, the length of continuing to engage with it, the total time spent using an intervention, or a self-reported measure of engagement by participants. This inconsistency has been found to be an issue in previous reviews on the user engagement of DMHIs [11,256]. This review was limited to peer-reviewed empirical articles. Although the review included articles that evaluated people’s experience with both research interventions and commercially available DMHIs, it is possible that some interventions may have been missed.

Finally, this review was conducted before the global COVID-19 pandemic. There may be unique factors that are pandemic related that make DMHI engagement more or less likely. For example, stay-at-home orders may exacerbate feelings of social isolation and make people more likely to engage with apps that increase social connectedness. On the other hand, it may also introduce additional barriers to finding a private space to use DMHIs if sheltering in place with others. The results presented in this review should be interpreted and used to understand DMHI engagement before and after the pandemic. A future review could be conducted solely during the pandemic period, and it could be compared with this review to understand DMHI use outside versus during a pandemic.

Conclusions

Previous studies have shown the potential of DMHIs to improve mental health. However, for these interventions to be clinically effective, they require engagement by users in real-world settings. Across the studies reviewed, we identified 16 common factors that affect user engagement. Further research on DMHIs can use these factors as guidelines when evaluating interventions with users, and future interventions can be developed with these factors in mind. By understanding the factors that affect engagement, targeted strategies can be developed to overcome addressable barriers and work toward the successful implementation of these interventions.

Acknowledgments

The authors thank Vicky Yu who helped with data entry for this manuscript.

Conflicts of Interest

SMS has received consulting payments from Otsuka Pharmaceuticals. All other authors declared no conflicts of interest.

Multimedia Appendix 1

Overview of search queries or terms used for each database.

DOC File , 17 KB

Multimedia Appendix 2

Data extraction template with metadata of the reviewed articles.

XLS File (Microsoft Excel File), 84 KB

Multimedia Appendix 3

Quality assessment of included studies.

DOCX File , 41 KB

Multimedia Appendix 4

Overview of barriers and facilitators for each theme.

XLS File (Microsoft Excel File), 12 KB

References

  1. Lipari, RN, Park-Lee E. Key Substance Use and Mental Health Indicators in the United States: Results from the 2018 National Survey on Drug Use and Health Internet. Substance Abuse and Mental Health Services Administration. 2019.   URL: https://www.samhsa.gov/data/ [accessed 2020-07-01]
  2. Gulliver A, Griffiths KM, Christensen H. Perceived barriers and facilitators to mental health help-seeking in young people: a systematic review. BMC Psychiatry 2010 Dec 30;10(113):1-9. [CrossRef]
  3. America's Mental Health 2018. Cohen Veterans Network. 2018.   URL: https://www.cohenveteransnetwork.org/wp-content/uploads/2018/10/Research-Summary-10-10-2018.pdf [accessed 2020-07-01]
  4. Anderson, NB, Belar CD, Breckler SJ, Nordal KC, Ballard DW, Bufka LF, et al. Stress in America: Paying With Our Health. American Psychological Association. 2014.   URL: https://www.apa.org/news/press/releases/stress/2014/stress-report.pdf [accessed 2020-07-01]
  5. Mohr DC, Burns MN, Schueller SM, Clarke G, Klinkman M. Behavioral intervention technologies: evidence review and recommendations for future research in mental health. Gen Hosp Psychiatry 2013 Aug;35(4):332-338 [FREE Full text] [CrossRef] [Medline]
  6. Muñoz RF. The Efficiency Model of Support and the Creation of Digital Apothecaries. Clin Psychol Sci Pract 2016 Nov 17;24(1):46-49. [CrossRef]
  7. Karyotaki E, Riper H, Twisk J, Hoogendoorn A, Kleiboer A, Mira A, et al. Efficacy of Self-guided Internet-Based Cognitive Behavioral Therapy in the Treatment of Depressive Symptoms: A Meta-analysis of Individual Participant Data. JAMA Psychiatry 2017 Apr 01;74(4):351-359. [CrossRef] [Medline]
  8. Graham AK, Greene CJ, Kwasny MJ, Kaiser SM, Lieponis P, Powell T, et al. Coached Mobile App Platform for the Treatment of Depression and Anxiety Among Primary Care Patients: A Randomized Clinical Trial. JAMA Psychiatry 2020 Sep 01;77(9):906-914. [CrossRef] [Medline]
  9. Baumel A, Edan S, Kane J. Is there a trial bias impacting user engagement with unguided e-mental health interventions? A systematic comparison of published reports and real-world usage of the same programs. Transl Behav Med 2019 Nov 25;9(6):1020-1033 [FREE Full text] [CrossRef] [Medline]
  10. Andrews G, Basu A, Cuijpers P, Craske MG, McEvoy P, English CL, et al. Computer therapy for the anxiety and depression disorders is effective, acceptable and practical health care: An updated meta-analysis. J Anxiety Disord 2018 Apr;55:70-78 [FREE Full text] [CrossRef] [Medline]
  11. Fleming T, Bavin L, Lucassen M, Stasiak K, Hopkins S, Merry S. Beyond the Trial: Systematic Review of Real-World Uptake and Engagement With Digital Self-Help Interventions for Depression, Low Mood, or Anxiety. J Med Internet Res 2018 Jun 06;20(6):e199 [FREE Full text] [CrossRef] [Medline]
  12. Kaltenthaler E, Sutcliffe P, Parry G, Beverley C, Rees A, Ferriter M. The acceptability to patients of computerized cognitive behaviour therapy for depression: a systematic review. Psychol Med 2008 Nov;38(11):1521-1530. [CrossRef] [Medline]
  13. Feng XL, Campbell A. Understanding E-Mental Health Resources: Personality, Awareness, Utilization, and Effectiveness of E-Mental Health Resources Amongst Youth. Journal of Technology in Human Services 2011 Apr;29(2):101-119. [CrossRef]
  14. Wilhelmsen M, Lillevoll K, Risør M, Høifødt R, Johansen M, Waterloo K, et al. Motivation to persist with internet-based cognitive behavioural treatment using blended care: a qualitative study. BMC Psychiatry 2013;13(1):296. [CrossRef]
  15. Torous J, Nicholas J, Larsen ME, Firth J, Christensen H. Clinical review of user engagement with mental health smartphone apps: evidence, theory and improvements. Evid Based Mental Health 2018 Jun 05;21(3):116-119. [CrossRef]
  16. Wasil AR, Gillespie S, Shingleton R, Wilks CR, Weisz JR. Examining the Reach of Smartphone Apps for Depression and Anxiety. AJP 2020 May 01;177(5):464-465. [CrossRef]
  17. Steinhubl SR, Muse ED, Topol EJ. The emerging field of mobile health. Sci. Transl. Med 2015 Apr 15;7(283):283rv3-283rv3. [CrossRef]
  18. Batra S, Baker R, Wang T, Forma F, DiBiasi F, Peters-Strickland T. Digital health technology for use in patients with serious mental illness: a systematic review of the literature. MDER 2017 Oct;10:237-251. [CrossRef]
  19. Lattie EG, Adkins EC, Winquist N, Stiles-Shields C, Wafford QE, Graham AK. Digital Mental Health Interventions for Depression, Anxiety, and Enhancement of Psychological Well-Being Among College Students: Systematic Review. J Med Internet Res 2019 Jul 22;21(7):e12869. [CrossRef]
  20. Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A. Rayyan—a web and mobile app for systematic reviews. Syst Rev 2016 Dec 5;5(210):1-10. [CrossRef]
  21. Lienemann BA, Unger JB, Cruz TB, Chu K. Methods for Coding Tobacco-Related Twitter Data: A Systematic Review. J Med Internet Res 2017 Mar 31;19(3):e91 [FREE Full text] [CrossRef] [Medline]
  22. Carroll C, Booth A, Lloyd-Jones M. Should We Exclude Inadequately Reported Studies From Qualitative Systematic Reviews? An Evaluation of Sensitivity Analyses in Two Case Study Reviews. Qual Health Res Internet 2012;22(10):1425-1434 [FREE Full text] [CrossRef]
  23. Golder S, Ahmed S, Norman G, Booth A. Attitudes Toward the Ethics of Research Using Social Media: A Systematic Review. J Med Internet Res 2017 Jun 06;19(6):e195. [CrossRef]
  24. Braun A, Clarke V. Using thematic analysis in psychology. Qual Res Psychol 2006 Aug;3(2):77-101. [CrossRef] [Medline]
  25. Ng A, Reddy M, Zalta A, Schueller SM. Veterans' perspectives on fitbit use in treatment for post-traumatic stress disorder: An interview study. J Med Internet Res 2018;20(6):e10415. [CrossRef]
  26. Guest G, MacQueen K, Namey E. Applied Thematic Analysis. Thousand Oaks, CA: SAGE Publications; 2011.
  27. Venkatesh V, Morris M, Davis G, Davis F. User Acceptance of Information Technology: Toward a Unified View. MIS Q Internet 2003;27(3):425-478 [FREE Full text] [CrossRef]
  28. Stoyanov SR, Hides L, Kavanagh DJ, Wilson H. Development and Validation of the User Version of the Mobile Application Rating Scale (uMARS). JMIR Mhealth Uhealth 2016;4(2):e72 [FREE Full text] [CrossRef] [Medline]
  29. Lewis JR. IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use. International Journal of Human-Computer Interaction 1995 Jan;7(1):57-78. [CrossRef]
  30. Pagliari C, Burton C, McKinstry B, Szentatotai A, David D, Serrano Blanco A, et al. Psychosocial implications of avatar use in supporting therapy for depression. Stud Health Technol Inform 2012;181:329-333. [Medline]
  31. Moessner M, Minarik C, Özer F, Bauer S. Can an internet-based program for the prevention and early intervention in eating disorders facilitate access to conventional professional healthcare? Journal of Mental Health 2016 Feb 05;25(5):441-447. [CrossRef]
  32. Kenny R, Dooley B, Fitzgerald A. Developing mental health mobile apps: Exploring adolescents' perspectives. Health Informatics J 2014 Nov 10;22(2):265-275. [CrossRef] [Medline]
  33. Wentzel J, van der Vaart R, Bohlmeijer ET, van Gemert-Pijnen JEWC. Mixing Online and Face-to-Face Therapy: How to Benefit From Blended Care in Mental Health Care. JMIR Ment Health 2016;3(1):e9 [FREE Full text] [CrossRef] [Medline]
  34. Schlosser D, Campellone T, Kim D, Truong B, Vergani S, Ward C, et al. Feasibility of PRIME: A Cognitive Neuroscience-Informed Mobile App Intervention to Enhance Motivated Behavior and Improve Quality of Life in Recent Onset Schizophrenia. JMIR Res Protoc 2016 Apr 28;5(2):e77. [CrossRef]
  35. Puszka S, Dingwall KM, Sweet M, Nagel T. E-Mental Health Innovations for Aboriginal and Torres Strait Islander Australians: A Qualitative Study of Implementation Needs in Health Services. JMIR Ment Health 2016 Sep 19;3(3):e43 [FREE Full text] [CrossRef] [Medline]
  36. Bennett-Levy J, Singer J, DuBois S, Hyde K. Translating E-Mental Health Into Practice: What Are the Barriers and Enablers to E-Mental Health Implementation by Aboriginal and Torres Strait Islander Health Professionals? J Med Internet Res 2017 Jan 11;19(1):e1 [FREE Full text] [CrossRef] [Medline]
  37. Deady M, Peters D, Lang H, Calvo R, Glozier N, Christensen H, et al. Designing smartphone mental health applications for emergency service workers. Occup Med (Lond) 2017 Aug 01;67(6):425-428. [CrossRef] [Medline]
  38. Arjadi R, Nauta MH, Bockting CLH. Acceptability of internet-based interventions for depression in Indonesia. Internet Interv 2018 Sep 01;13(2):8-15 [FREE Full text] [CrossRef] [Medline]
  39. Stallman HM, Kavanagh DJ. Development of an Internet Intervention to Promote Wellbeing in College Students. Aust Psychol 2016 Aug 28;53(1):60-67. [CrossRef]
  40. Clough BA, Zarean M, Ruane I, Mateo NJ, Aliyeva TA, Casey LM. Going global: do consumer preferences, attitudes, and barriers to using e-mental health services differ across countries? Journal of Mental Health 2017 Aug 31;28(1):17-25. [CrossRef]
  41. Gulec H, Moessner M, Mezei A, Kohls E, Túry F, Bauer S. Internet-based maintenance treatment for patients with eating disorders. Professional Psychology: Research and Practice 2011;42(6):479-486. [CrossRef]
  42. Adler G, Pritchett LR, Kauth MR, Nadorff D. A pilot project to improve access to telepsychotherapy at rural clinics. Telemed J E Health 2014 Jan;20(1):83-85 [FREE Full text] [CrossRef] [Medline]
  43. Batterham P, Calear A, O'Dea B, Larsen M, Titov N, March S, et al. Stakeholder perspectives on evidence for digital mental health interventions: Implications for accreditation systems. Digit Health 2019;5:1-7. [CrossRef]
  44. Harjumaa M, Halttu K, Koistinen K, Oinas-Kukkonen H. User Experience of Mobile Coaching for Stress-Management to Tackle Prevalent Health Complaints. : Springer International Publishing; 2015 Presented at: Scandinavian Conference on Information Systems; August 9-12; Oulu, Finland p. 152-164. [CrossRef]
  45. Rickard N, Arjmand H, Bakker D, Seabrook E. Development of a Mobile Phone App to Support Self-Monitoring of Emotional Well-Being: A Mental Health Digital Innovation. JMIR Ment Health 2016 Nov 23;3(4):e49 [FREE Full text] [CrossRef] [Medline]
  46. Rodriguez-Paras C, Sasangohar F. Usability Assessment of a Post-Traumatic Stress Disorder (PTSD) mHealth App. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting. 2017 Oct 20 Presented at: Proc Hum Factors Ergon Soc; 2017; Austin, Texas p. 1824-1828. [CrossRef]
  47. Melton L, Brewer B, Kolva E, Joshi T, Bunch M. Increasing access to care for young adults with cancer: Results of a quality-improvement project using a novel telemedicine approach to supportive group psychotherapy. Palliat Support Care 2017 Apr;15(2):176-180. [CrossRef] [Medline]
  48. Simões de Almeida R, Sousa T, Marques A, Queirós C. Patients’ perspectives about the design of a mobile application for psychotic disorders. Psychol. Community Health 2018 Aug 03;7(1):16-28. [CrossRef]
  49. Simões de Almeida R, Sousa T, Couto A, Marques A, Queirós C, Martins C. Development of weCope, a mobile app for illness self-management in schizophrenia. Clinical Psychiatry 2019;46(1):1-4. [CrossRef]
  50. Thach K. A Qualitative Analysis of User Reviews on Mental Health Apps: Who Used it? for What? and Why? A qualitative analysis of user reviews on mental health apps: Who used it? For what? And why? Tra Vinh University, School of Engineering and Technology, Tra Vinh, Viet Nam: Institute of Electrical and Electronics Engineers Inc; 2019 Presented at: 2019 IEEE-RIVF International Conference on Computing and Communication Technologies (RIVF); March 20-22; Danang, Vietnam p. 1-4. [CrossRef]
  51. Doherty G, Coyle D, Sharry J. Engagement with online mental health interventions: An exploratory clinical study of a treatment for depression. 2012 Presented at: CHI'12; May 5-10; Austin, Texas p. 1421-1430. [CrossRef]
  52. Proudfoot J, Parker G, Manicavasagar V, Hadzi-Pavlovic D, Whitton A, Nicholas J, et al. Effects of adjunctive peer support on perceptions of illness control and understanding in an online psychoeducation program for bipolar disorder: A randomised controlled trial. Journal of Affective Disorders 2012 Dec;142(1-3):98-105. [CrossRef]
  53. Crisp DA, Griffiths KM. Participating in Online Mental Health Interventions: Who Is Most Likely to Sign Up and Why? Depression Research and Treatment 2014;2014:1-11. [CrossRef]
  54. Gunn J, Cameron J, Densley K, Davidson S, Fletcher S, Palmer V, et al. Uptake of mental health websites in primary care: Insights from an Australian longitudinal cohort study of depression. Patient Educ Couns 2018;101(1):105-112 [FREE Full text] [CrossRef]
  55. Mackesy-Amiti ME, Boodram B. Feasibility of ecological momentary assessment to study mood and risk behavior among young people who inject drugs. Drug and Alcohol Dependence 2018 Jun;187:227-235. [CrossRef]
  56. Mikolasek M, Witt CM, Barth J. Adherence to a Mindfulness and Relaxation Self-Care App for Cancer Patients: Mixed-Methods Feasibility Study. JMIR Mhealth Uhealth 2018 Dec 06;6(12):e11271 [FREE Full text] [CrossRef] [Medline]
  57. Achtyes ED, Ben-Zeev D, Luo Z, Mayle H, Burke B, Rotondi AJ, et al. Off-hours use of a smartphone intervention to extend support for individuals with schizophrenia spectrum disorders recently discharged from a psychiatric hospital. Schizophrenia Research 2019 Apr;206:200-208. [CrossRef]
  58. Krause K, Guertler D, Moehring A, Batra A, Eck S, Rumpf H, et al. Feasibility and Acceptability of an Intervention Providing Computer-Generated Tailored Feedback to Target Alcohol Consumption and Depressive Symptoms in Proactively Recruited Health Care Patients and Reactively Recruited Media Volunteers: Results of a Pilot Study. Eur Addict Res 2019 Dec 15;25(3):119-131. [CrossRef] [Medline]
  59. La Porte L, Kim J, Adams M, Zagorsky B, Gibbons R, Silver R, et al. Feasibility of perinatal mood screeningtext messaging on patients? personal smartphones. Arch Womens Ment Health 2019;23(2):181-188. [CrossRef]
  60. Mira A, Soler C, Alda M, Baños R, Castilla D, Castro A, et al. Exploring the Relationship Between the Acceptability of an Internet-Based Intervention for Depression in Primary Care and Clinical Outcomes: Secondary Analysis of a Randomized Controlled Trial. Front Psychiatry 2019;10:325 [FREE Full text] [CrossRef] [Medline]
  61. Smail-Crevier R, Powers G, Noel C, Wang J. Health-Related Internet Usage and Design Feature Preference for E-Mental Health Programs Among Men and Women. J Med Internet Res 2019 Mar 18;21(3):e11224 [FREE Full text] [CrossRef] [Medline]
  62. Crosier B, Brian R, Ben-Zeev D. Using Facebook to reach people who experience auditory hallucinations. J Med Internet Res 2016;18(6):e160 [FREE Full text] [CrossRef]
  63. Kenny R, Dooley B, Fitzgerald A. Ecological Momentary Assessment of Adolescent Problems, Coping Efficacy, and Mood States Using a Mobile Phone App: An Exploratory Study. JMIR Ment Health 2016 Nov 29;3(4):e51. [CrossRef]
  64. Mattila E, Lappalainen R, Välkkynen P, Sairanen E, Lappalainen P, Karhunen L, et al. Usage and Dose Response of a Mobile Acceptance and Commitment Therapy App: Secondary Analysis of the Intervention Arm of a Randomized Controlled Trial. JMIR Mhealth Uhealth 2016 Jul 28;4(3):e90. [CrossRef]
  65. Beatty L, Kemp E, Binnion C, Turner J, Milne D, Butow P, et al. Uptake and adherence to an online intervention for cancer-related distress: older age is not a barrier to adherence but may be a barrier to uptake. Support Care Cancer 2017 Jun;25(6):1905-1914. [CrossRef] [Medline]
  66. Kannisto KA, Korhonen J, Adams CE, Koivunen MH, Vahlberg T, Välimäki MA. Factors Associated With Dropout During Recruitment and Follow-Up Periods of a mHealth-Based Randomized Controlled Trial for Mobile.Net to Encourage Treatment Adherence for People With Serious Mental Health Problems. J Med Internet Res 2017 Feb 21;19(2):e46. [CrossRef] [Medline]
  67. Abel EA, Shimada SL, Wang K, Ramsey C, Skanderson M, Erdos J, et al. Dual Use of a Patient Portal and Clinical Video Telehealth by Veterans with Mental Health Diagnoses: Retrospective, Cross-Sectional Analysis. J Med Internet Res 2018 Nov 07;20(11):e11350. [CrossRef]
  68. Graham S, Irving J, Cano I, Edwards M. Participation with online recovery specific groups - findings from the UK Life in Recovery survey 2015. Alcoholism Treatment Quarterly 2018 Jul 31;36(4):459-481. [CrossRef]
  69. Pavliscsak H, Little JR, Poropatich RK, McVeigh FL, Tong J, Tillman JS, et al. Assessment of patient engagement with a mobile application among service members in transition. J Am Med Inform Assoc 2015 Sep 11;23(1):110-118. [CrossRef]
  70. Schneider BC, Schröder J, Berger T, Hohagen F, Meyer B, Späth C, et al. Bridging the "digital divide": A comparison of use and effectiveness of an online intervention for depression between Baby Boomers and Millennials. J Affect Disord 2018 Aug 15;236:243-251. [CrossRef] [Medline]
  71. Pruitt LD, Vuletic S, Smolenski DJ, Wagner A, Luxton DD, Gahm GA. Predicting post treatment client satisfaction between behavioural activation for depression delivered either in-person or via home-based telehealth. J Telemed Telecare 2019 Sep;25(8):460-467. [CrossRef] [Medline]
  72. March S, Day J, Ritchie G, Rowe A, Gough J, Hall T, et al. Attitudes Toward e-Mental Health Services in a Community Sample of Adults: Online Survey. J Med Internet Res 2018 Feb 19;20(2):e59 [FREE Full text] [CrossRef] [Medline]
  73. Chudy-Onwugaje K, Abutaleb A, Buchwald A, Langenberg P, Regueiro M, Schwartz D, et al. Age Modifies the Association Between Depressive Symptoms and Adherence to Self-Testing With Telemedicine in Patients With Inflammatory Bowel Disease. Inflamm Bowel Dis 2018;24(12):2648-2654. [CrossRef]
  74. Watson H, Levine M, Zerwas S, Hamer R, Crosby R, Sprecher C, et al. Predictors of dropout in face-to-face and internet-based cognitive-behavioral therapy for bulimia nervosa in a randomized controlled trial. Int J Eat Disord 2017;50(5):569-577. [CrossRef]
  75. Stevens R, Polk K, Merrill C, Feng F, Weiss M, Brosnan E, et al. User experience design to enhance the effectiveness of mobile technologies for the treatment of mental health. 2018 Presented at: 2018 Systems and Information Engineering Design Symposium (SIEDS); April 27; Charlottesville, Virginia p. 135-140. [CrossRef]
  76. Kemmeren LL, van Schaik A, Smit JH, Ruwaard J, Rocha A, Henriques M, et al. Unraveling the Black Box: Exploring Usage Patterns of a Blended Treatment for Depression in a Multicenter Study. JMIR Ment Health 2019 Jul 25;6(7):e12707. [CrossRef]
  77. Ervasti M, Kallio J, Määttänen I, Mäntyjärvi J, Jokela M. Influence of Personality and Differences in Stress Processing Among Finnish Students on Interest to Use a Mobile Stress Management App: Survey Study. JMIR Ment Health 2019 May 13;6(5):e10039. [CrossRef]
  78. Crooks RE, Bell M, Patten SB, Wiebe S, Holroyd-Leduc J, Bulloch AG, et al. Mind the gap: Exploring information gaps for the development of an online resource hub for epilepsy and depression. Epilepsy Behav 2017 May;70(Pt A):18-23. [CrossRef] [Medline]
  79. Eisner E, Drake RJ, Berry N, Barrowclough C, Emsley R, Machin M, et al. Development and Long-Term Acceptability of ExPRESS, a Mobile Phone App to Monitor Basic Symptoms and Early Signs of Psychosis Relapse. JMIR Mhealth Uhealth 2019 Mar 29;7(3):e11568. [CrossRef]
  80. Mitchell A, Whittingham K, Steindl S, Kirby J. Feasibility and acceptability of a brief online self-compassion intervention for mothers of infants. Arch Womens Ment Health 2018;21(5):553-561. [CrossRef]
  81. Toscos T, Carpenter M, Drouin M, Roebuck A, Kerrigan C, Mirro M. College Students' Experiences with, and Willingness to Use, Different Types of Telemental Health Resources: Do Gender, Depression/Anxiety, or Stress Levels Matter? Telemedicine and e-Health 2018 Dec;24(12):998-1005. [CrossRef]
  82. Huberty J, Vranceanu A, Carney C, Breus M, Gordon M, Puzia ME. Characteristics and Usage Patterns Among 12,151 Paid Subscribers of the Calm Meditation App: Cross-Sectional Survey. JMIR Mhealth Uhealth 2019 Nov 3;7(11):e15648. [CrossRef]
  83. Nicholas J, Proudfoot J, Parker G, Gillis I, Burckhardt R, Manicavasagar V, et al. The Ins and Outs of an Online Bipolar Education Program: A Study of Program Attrition. J Med Internet Res 2010;12(5):e57. [CrossRef]
  84. Forchuk C, Reiss J, Eichstedt J, Singh D, Collins K, Rudnick A, et al. The Youth-Mental Health Engagement Network: An Exploratory Pilot Study of a Smartphone and Computer-Based Personal Health Record for Youth Experiencing Depressive Symptoms. Int J Ment Health 2016;45(3):205-222. [CrossRef]
  85. Nitsch M, Dimopoulos C, Flaschberger E, Saffran K, Kruger J, Garlock L, et al. A guided online and mobile self-help program for individuals with eating disorders: An iterative engagement and usability study. J Med Internet Res 2016;18(1):e7 [FREE Full text] [CrossRef]
  86. Heiniger L, Smith A, Olver I, Grimison P, Klein B, Wootten A, et al. Development and pilot testing of a web‐based intervention to reduce anxiety and depression in survivors of testicular cancer. Eur J Cancer Care (Engl) 2017;26(6):1-10 [FREE Full text] [CrossRef]
  87. Meyer N, Kerz M, Folarin A, Joyce DW, Jackson R, Karr C, et al. Capturing Rest-Activity Profiles in Schizophrenia Using Wearable and Mobile Technologies: Development, Implementation, Feasibility, and Acceptability of a Remote Monitoring Platform. JMIR Mhealth Uhealth 2018 Oct 30;6(10):e188. [CrossRef]
  88. Burchert S, Alkneme MS, Bird M, Carswell K, Cuijpers P, Hansen P, et al. User-Centered App Adaptation of a Low-Intensity E-Mental Health Intervention for Syrian Refugees. Front. Psychiatry 2019 Jan 25;9(663):1-18. [CrossRef]
  89. Görges F, Oehler C, von Hirschhausen E, Hegerl U, Rummel-Kluge C. GET.HAPPY2 - User perspectives on an internet-based self-management positive psychology intervention among persons with and without depression: Results from a retrospective survey. J Clin Psychol 2020 Jun;76(6):1030-1046. [CrossRef] [Medline]
  90. Kerr D, Ornelas I, Lilly M, Calhoun R, Meischke H. Participant Engagement in and Perspectives on a Web-Based Mindfulness Intervention for 9-1-1 Telecommunicators: Multimethod Study. J Med Internet Res 2019 Jun 19;21(6):e13449 [FREE Full text] [CrossRef] [Medline]
  91. Sayal K, Roe J, Ball H, Atha C, Kaylor-Hughes C, Guo B, et al. Feasibility of a randomised controlled trial of remotely delivered problem-solving cognitive behaviour therapy versus usual care for young people with depression and repeat self-harm: lessons learnt (e-DASH). BMC Psychiatry 2019 Jan 24;19(1):42. [CrossRef] [Medline]
  92. Schuster R, Kalthoff I, Walther A, Köhldorfer L, Partinger E, Berger T, et al. Effects, Adherence, and Therapists' Perceptions of Web- and Mobile-Supported Group Therapy for Depression: Mixed-Methods Study. J Med Internet Res 2019 Apr 28;21(5):e11860 [FREE Full text] [CrossRef] [Medline]
  93. Simblett S, Matcham F, Siddi S, Bulgari V, Barattieri di San Pietro C, Hortas López J, et al. Barriers to and Facilitators of Engagement With mHealth Technology for Remote Measurement and Management of Depression: Qualitative Analysis. JMIR Mhealth Uhealth 2019 Jan 30;7(1):e11325. [CrossRef]
  94. Lim MH, Rodebaugh TL, Eres R, Long KM, Penn DL, Gleeson JFM. A Pilot Digital Intervention Targeting Loneliness in Youth Mental Health. Front. Psychiatry 2019 Aug 23;10:1-13. [CrossRef]
  95. Donkin L, Glozier N. Motivators and motivations to persist with online psychological interventions: a qualitative study of treatment completers. J Med Internet Res 2012 Jun 22;14(3):e91 [FREE Full text] [CrossRef] [Medline]
  96. Kaczmarek LD, Kashdan TB, Kleiman EM, Baczkowski B, Enko J, Siebers A, et al. Who self-initiates gratitude interventions in daily life? An examination of intentions, curiosity, depressive symptoms, and life satisfaction. Personality and Individual Differences 2013 Oct;55(7):805-810. [CrossRef]
  97. Al-Asadi AM, Klein B, Meyer D. Posttreatment attrition and its predictors, attrition bias, and treatment efficacy of the anxiety online programs. J Med Internet Res 2014 Oct 14;16(10):e232 [FREE Full text] [CrossRef] [Medline]
  98. von Brachel R, Hötzel K, Hirschfeld G, Rieger E, Schmidt U, Kosfelder J, et al. Internet-Based Motivation Program for Women With Eating Disorders: Eating Disorder Pathology and Depressive Mood Predict Dropout. J Med Internet Res 2014 Mar 31;16(3):e92. [CrossRef]
  99. Farrer LM, Griffiths KM, Christensen H, Mackinnon AJ, Batterham PJ. Predictors of Adherence and Outcome in Internet-Based Cognitive Behavior Therapy Delivered in a Telephone Counseling Setting. Cogn Ther Res 2013 Oct 27;38(3):358-367. [CrossRef]
  100. Habibović M, Cuijpers P, Alings M, van DVP, Theuns D, Bouwels L, et al. Attrition and adherence in a WEB-Based Distress Management Program for Implantable Cardioverter defibrillator Patients (WEBCARE): randomized controlled trial. J Med Internet Res 2014;16(2):e52 [FREE Full text] [CrossRef] [Medline]
  101. Grubbs KM, Fortney JC, Pyne JM, Hudson T, Moore WM, Custer P, et al. Predictors of Initiation and Engagement of Cognitive Processing Therapy Among Veterans With PTSD Enrolled in Collaborative Care. Journal of Traumatic Stress 2015 Dec 01;28(6):580-584. [CrossRef]
  102. Gould C, Zapata A, Bruce J, Bereknyei Merrell S, Wetherell J, O'Hara R, et al. Development of a video-delivered relaxation treatment of late-life anxiety for veterans. Int. Psychogeriatr 2017 Jun 08;29(10):1633-1645. [CrossRef]
  103. Feijt MA, de Kort YAW, Bongers IM, IJsselsteijn WA. Perceived Drivers and Barriers to the Adoption of eMental Health by Psychologists: The Construction of the Levels of Adoption of eMental Health Model. J Med Internet Res 2018 Apr 24;20(4):e153 [FREE Full text] [CrossRef] [Medline]
  104. Berry N, Lobban F, Bucci S. A qualitative exploration of service user views about using digital health interventions for self-management in severe mental health problems. BMC Psychiatry 2019 Jan 21;19(1):35 [FREE Full text] [CrossRef] [Medline]
  105. Deen TL, Fortney JC, Schroeder G. Patient acceptance of and initiation and engagement in telepsychotherapy in primary care. Psychiatr Serv 2013 Apr 1;64(4):380-384 [FREE Full text] [CrossRef] [Medline]
  106. Fonseca A, Gorayeb R, Canavarro MC. Women’s use of online resources and acceptance of e-mental health tools during the perinatal period. International Journal of Medical Informatics 2016 Oct;94:228-236. [CrossRef]
  107. O'Dea B, Achilles M, Werner-Seidler A, Batterham P, Calear A, Perry Y, et al. Adolescents' Perspectives on a Mobile App for Relationships: Cross-Sectional Survey. JMIR Mhealth Uhealth 2018 Mar 08;6(3):e56 [FREE Full text] [CrossRef] [Medline]
  108. Adkins EC, Zalta AK, Boley RA, Glover A, Karnik NS, Schueller SM. Exploring the potential of technology-based mental health services for homeless youth: A qualitative study. Psychol Serv 2017 Dec;14(2):238-245 [FREE Full text] [CrossRef] [Medline]
  109. Apolinário-Hagen J, Hennemann S, Fritsche L, Drüge M, Breil B. Determinant Factors of Public Acceptance of Stress Management Apps: Survey Study. JMIR Ment Health 2019 Nov 07;6(11):e15373 [FREE Full text] [CrossRef] [Medline]
  110. Nitsch M, Dimopoulos CN, Flaschberger E, Saffran K, Kruger JF, Garlock L, et al. A Guided Online and Mobile Self-Help Program for Individuals With Eating Disorders: An Iterative Engagement and Usability Study. J Med Internet Res 2016 Jan 11;18(1):e7 [FREE Full text] [CrossRef] [Medline]
  111. East M, Havard B, Hastings N. Mental Health Mobile Apps’ Instruction: Technology Adoption Theories Applied in a Mixed Methods Study of Counseling Faculty. Journal of Technology in Human Services 2016 Oct 21;34(4):301-325. [CrossRef]
  112. Lattie EG, Adkins EC, Winquist N, Stiles-Shields C, Wafford QE, Graham AK. Digital Mental Health Interventions for Depression, Anxiety, and Enhancement of Psychological Well-Being Among College Students: Systematic Review. J Med Internet Res 2019 Jul 22;21(7):e12869 [FREE Full text] [CrossRef] [Medline]
  113. Huerta-Ramos E, Marcó-García S, Escobar-Villegas MS, Rubio-Abadal E, Ochoa S, Grasa Bello EM, et al. m-RESIST, a complete m-Health solution for patients with treatmentresistant schizophrenia: a qualitative study of user needs and acceptability in the Barcelona metropolitan area. Actas Esp Psiquiatr 2017 Nov;45(6):277-289 [FREE Full text] [Medline]
  114. Switsers L, Dauwe A, Vanhoudt A, Van Dyck H, Lombaerts K, Oldenburg J. Users’ Perspectives on mHealth Self-Management of Bipolar Disorder: Qualitative Focus Group Study. JMIR Mhealth Uhealth 2018 May 02;6(5):e108. [CrossRef]
  115. Diez-Canseco F, Toyama M, Ipince A, Perez-Leon S, Cavero V, Araya R, et al. Integration of a Technology-Based Mental Health Screening Program Into Routine Practices of Primary Health Care Services in Peru (The Allillanchu Project): Development and Implementation. J Med Internet Res 2018 Mar 15;20(3):e100 [FREE Full text] [CrossRef] [Medline]
  116. Gindidis S, Stewart S, Roodenburg J. Psychologists’ motivations for integrating apps into therapy with secondary school-aged young people. J. Psychol. Couns. Sch 2019 Oct 21;30(1):2-12. [CrossRef]
  117. Allan S, Bradstreet S, Mcleod H, Farhall J, Lambrou M, Gleeson J, et al. Developing a Hypothetical Implementation Framework of Expectations for Monitoring Early Signs of Psychosis Relapse Using a Mobile App: Qualitative Study. J Med Internet Res 2019 Oct 24;21(10):e14366. [CrossRef]
  118. Newman L, Bidargaddi N, Schrader G. Service providers’ experiences of using a telehealth network 12 months after digitisation of a large Australian rural mental health service. International Journal of Medical Informatics 2016 Oct;94:8-20. [CrossRef]
  119. Thorsen MM, Patena JV, Guthrie KM, Spirito A, Ranney ML. Using High-Risk Adolescents′ Voices to Develop a Comprehensible Cognitive Behavioral Therapy–Based Text-Message Program. Behavioral Medicine 2016 Nov 21;44(2):89-99. [CrossRef]
  120. Chęć M, Ligocka M, Maciejewska M, Samochowiec J, Łodygowska E, Samochowiec A. On-line psychological support in the evaluation of specialists and future specialists in Poland. Computers in Human Behavior 2016 Nov;64:703-709. [CrossRef]
  121. Jordan SE, Shearer EM. An exploration of supervision delivered via clinical video telehealth (CVT). Training and Education in Professional Psychology 2019 Nov;13(4):323-330. [CrossRef]
  122. Bucci S, Berry N, Morris R, Berry K, Haddock G, Lewis S, et al. "They Are Not Hard-to-Reach Clients. We Have Just Got Hard-to-Reach Services." Staff Views of Digital Health Tools in Specialist Mental Health Services. Front Psychiatry 2019;10:344 [FREE Full text] [CrossRef] [Medline]
  123. Moessner M, Minarik C, Özer F, Bauer S. Can an internet-based program for the prevention and early intervention in eating disorders facilitate access to conventional professional healthcare? J Ment Health 2016 Oct;25(5):441-447. [CrossRef] [Medline]
  124. Meglic M, Furlan M, Kuzmanic M, Kozel D, Baraga D, Kuhar I, et al. Feasibility of an eHealth Service to Support Collaborative Depression Care: Results of a Pilot Study. J Med Internet Res 2010 Dec 19;12(5):e63. [CrossRef]
  125. Forchuk C, Reiss J, O'Regan T, Ethridge P, Donelle L, Rudnick A. Client perceptions of the mental health engagement network: a qualitative analysis of an electronic personal health record. BMC Psychiatry 2015 Oct 14;15:250 [FREE Full text] [CrossRef] [Medline]
  126. Wozney L, Newton AS, Gehring ND, Bennett K, Huguet A, Hartling L, et al. Implementation of eMental Health care: viewpoints from key informants from organizations and agencies with eHealth mandates. BMC Med Inform Decis Mak 2017 Jun 2;17(1):1-15. [CrossRef]
  127. Wallin EEK, Mattsson S, Olsson EMG. The Preference for Internet-Based Psychological Interventions by Individuals Without Past or Current Use of Mental Health Treatment Delivered Online: A Survey Study With Mixed-Methods Analysis. JMIR Ment Health 2016;3(2):e25 [FREE Full text] [CrossRef] [Medline]
  128. Lal S, Dell'Elce J, Tucci N, Fuhrer R, Tamblyn R, Malla A. Preferences of Young Adults With First-Episode Psychosis for Receiving Specialized Mental Health Services Using Technology: A Survey Study. JMIR Ment Health 2015;2(2):e18 [FREE Full text] [CrossRef] [Medline]
  129. Woolderink M, Bindels JA, Evers SM, Paulus AT, van Asselt AD, van Schayck OC. An Online Health Prevention Intervention for Youth with Addicted or Mentally Ill Parents: Experiences and Perspectives of Participants and Providers from a Randomized Controlled Trial. J Med Internet Res 2015 Dec 02;17(12):e274. [CrossRef]
  130. Laurie J, Blandford A. Making time for mindfulness. International Journal of Medical Informatics 2016 Dec;96:38-50. [CrossRef]
  131. Mitchell AE, Whittingham K, Steindl S, Kirby J. Feasibility and acceptability of a brief online self-compassion intervention for mothers of infants. Arch Womens Ment Health 2018 Oct;21(5):553-561. [CrossRef] [Medline]
  132. Bengtsson J, Nordin S, Carlbring P. Therapists' Experiences of Conducting Cognitive Behavioural Therapy Online vis-à-vis Face-to-Face. Cognitive Behaviour Therapy 2015 Jun 19;44(6):470-479. [CrossRef]
  133. Walsh S, Kaselionyte J, Taylor S, Priebe S. What might affect acceptability of online positive psychology interventions for depression: a qualitative study on patient expectations'. BMC Psychiatry 2018 Jul 27;18(1):240 [FREE Full text] [CrossRef] [Medline]
  134. Urech A, Krieger T, Möseneder L, Biaggi A, Vincent A, Poppe C, et al. A patient perspective on advantages and disadvantages of blended cognitive behaviour therapy for depression: A qualitative content analysis. Psychotherapy Research 2018 Jan 31;29(8):986-998. [CrossRef]
  135. Anastasiadou D, Folkvord F, Serrano-Troncoso E, Lupiañez-Villanueva F. Mobile Health Adoption in Mental Health: User Experience of a Mobile Health App for Patients With an Eating Disorder. JMIR Mhealth Uhealth 2019 May 31;7(6):e12920. [CrossRef]
  136. Saberi P, Dawson Rose C, Wootton A, Ming K, Legnitto D, Jeske M, et al. Use of technology for delivery of mental health and substance use services to youth living with HIV: a mixed-methods perspective. AIDS Care 2020 Aug;32(8):931-939. [CrossRef] [Medline]
  137. Carolan S, de Visser RO. Employees’ Perspectives on the Facilitators and Barriers to Engaging With Digital Mental Health Interventions in the Workplace: Qualitative Study. JMIR Ment Health 2018 Jan 19;5(1):e8. [CrossRef]
  138. Ashwick R, Turgoose D, Murphy D. Exploring the acceptability of delivering Cognitive Processing Therapy (CPT) to UK veterans with PTSD over Skype: a qualitative study. European Journal of Psychotraumatology 2019 Feb 04;10(1):1573128. [CrossRef]
  139. Jonathan G, Carpenter-Song EA, Brian RM, Ben-Zeev D. Life with FOCUS: A qualitative evaluation of the impact of a smartphone intervention on people with serious mental illness. Psychiatr Rehabil J 2019 Jun;42(2):182-189. [CrossRef] [Medline]
  140. Lal S, Nguyen V, Theriault J. Seeking mental health information and support online: experiences and perspectives of young people receiving treatment for first-episode psychosis. Early Intervention in Psychiatry 2016 Jan 26;12(3):324-330. [CrossRef] [Medline]
  141. Pretorius C, Chambers D, Cowan B, Coyle D. Young People Seeking Help Online for Mental Health: Cross-Sectional Survey Study. JMIR Ment Health 2019 Aug 26;6(8):e13524 [FREE Full text] [CrossRef] [Medline]
  142. Ranney M, Thorsen M, Patena JV, Cunningham R, Boyer E, Walton M, et al. "You need to get them where they feel it": Conflicting perspectives on how to maximize the structure of text-message psychological interventions for adolescents. : IEEE Computer Society; 2015 Presented at: 48th Hawaii Int Conf Syst Sci; Jan 5-8; Kauai, Hawaii p. 3247-3255. [CrossRef]
  143. Casey LM, Joy A, Clough BA. The impact of information on attitudes toward e-mental health services. Cyberpsychol Behav Soc Netw 2013 Aug;16(8):593-598. [CrossRef] [Medline]
  144. Sinha Deb K, Tuli A, Sood M, Chadda R, Verma R, Kumar S, et al. Is India ready for mental health apps (MHApps)? A quantitative-qualitative exploration of caregivers' perspective on smartphone-based solutions for managing severe mental illnesses in low resource settings. PLoS One 2018;13(9):e0203353 [FREE Full text] [CrossRef] [Medline]
  145. Chadi N, Weisbaum E, Malboeuf-Hurtubise C, Kohut S, Viner C, Kaufman M, et al. Can the Mindful Awareness and Resilience Skills for Adolescents (MARS-A) Program Be Provided Online? Voices from the Youth. Children (Basel) 2018;5(9):115. [CrossRef]
  146. Kubo A, Altschuler A, Kurtovich E, Hendlish S, Laurent CA, Kolevska T, et al. A Pilot Mobile-Based Mindfulness Intervention for Cancer Patients and Their Informal Caregivers. Mindfulness 2018 Mar 24;9(6):1885-1894. [CrossRef]
  147. Postel M, de Haan HA, ter Huurne ED, Becker E, de Jong CAJ. Effectiveness of a web-based intervention for problem drinkers and reasons for dropout: randomized controlled trial. J Med Internet Res 2010 Dec 16;12(4):e68 [FREE Full text] [CrossRef] [Medline]
  148. Ashford MT, Olander EK, Rowe H, Fisher JR, Ayers S. Feasibility and Acceptability of a Web-Based Treatment with Telephone Support for Postpartum Women With Anxiety: Randomized Controlled Trial. JMIR Ment Health 2018 Apr 20;5(2):e19. [CrossRef]
  149. Chen AT, Wu S, Tomasino KN, Lattie EG, Mohr DC. A multi-faceted approach to characterizing user behavior and experience in a digital mental health intervention. Journal of Biomedical Informatics 2019 Jun;94:103187. [CrossRef]
  150. Wallin E, Norlund F, Olsson EMG, Burell G, Held C, Carlsson T. Treatment Activity, User Satisfaction, and Experienced Usability of Internet-Based Cognitive Behavioral Therapy for Adults With Depression and Anxiety After a Myocardial Infarction: Mixed-Methods Study. J Med Internet Res 2018 Mar 16;20(3):e87 [FREE Full text] [CrossRef] [Medline]
  151. Schuster R, Fichtenbauer I, Sparr VM, Berger T, Laireiter A. Feasibility of a blended group treatment (bGT) for major depression: uncontrolled interventional study in a university setting. BMJ Open 2018 Dec 12;8(3):e018412 [FREE Full text] [CrossRef] [Medline]
  152. Chan C, West S, Glozier N. Commencing and Persisting With a Web-Based Cognitive Behavioral Intervention for Insomnia: A Qualitative Study of Treatment Completers. J Med Internet Res 2017 Feb 10;19(2):e37 [FREE Full text] [CrossRef] [Medline]
  153. Dodd A, Mallinson S, Griffiths M, Morriss R, Jones S, Lobban F. Users' experiences of an online intervention for bipolar disorder: important lessons for design and evaluation. Evid Based Ment Health 2017 Nov;20(4):133-139 [FREE Full text] [CrossRef] [Medline]
  154. Powell L, Parker J, Harpin V. ADHD: Is There an App for That? A Suitability Assessment of Apps for the Parents of Children and Young People With ADHD. JMIR Mhealth Uhealth 2017 Oct 13;5(10):e149. [CrossRef]
  155. Lundgren J, Johansson P, Jaarsma T, Andersson G, Kärner Köhler A. Patient Experiences of Web-Based Cognitive Behavioral Therapy for Heart Failure and Depression: Qualitative Study. J Med Internet Res 2018 Sep 05;20(9):e10302 [FREE Full text] [CrossRef] [Medline]
  156. Huis in het Veld JG, Verkaik R, van Meijel B, Verkade P, Werkman W, Hertogh CM, et al. Self-Management Support and eHealth When Managing Changes in Behavior and Mood of a Relative With Dementia: An Asynchronous Online Focus Group Study of Family Caregivers' Needs. Research in Gerontological Nursing 2018 Mar 02;11(3):151-159. [CrossRef]
  157. Hamblen J, Grubaugh A, Davidson T, Borkman A, Bunnell B, Ruggiero K. An Online Peer Educational Campaign to Reduce Stigma and Improve Help Seeking in Veterans with Posttraumatic Stress Disorder. Telemed J E Health 2019 Jan;25(1):41-47. [CrossRef] [Medline]
  158. Schuster R, Sigl S, Berger T, Laireiter A. Patients' Experiences of Web- and Mobile-Assisted Group Therapy for Depression and Implications of the Group Setting: Qualitative Follow-Up Study. JMIR Ment Health 2018 Jul 11;5(3):e49 [FREE Full text] [CrossRef] [Medline]
  159. Similä H, Immonen M, Toska-Tervola J, Enwald H, Keränen N, Kangas M, et al. Feasibility of mobile mental wellness training for older adults. Geriatr Nurs 2018;39(5):499-505. [CrossRef] [Medline]
  160. Wachtler C, Coe A, Davidson S, Fletcher S, Mendoza A, Sterling L, et al. Development of a Mobile Clinical Prediction Tool to Estimate Future Depression Severity and Guide Treatment in Primary Care: User-Centered Design. JMIR Mhealth Uhealth 2018 Apr 23;6(4):e95 [FREE Full text] [CrossRef] [Medline]
  161. Lipschitz J, Miller CJ, Hogan TP, Burdick KE, Lippin-Foster R, Simon SR, et al. Adoption of Mobile Apps for Depression and Anxiety: Cross-Sectional Survey Study on Patient Interest and Barriers to Engagement. JMIR Ment Health 2019 Jan 25;6(1):e11334 [FREE Full text] [CrossRef] [Medline]
  162. Hartmann R, Sander C, Lorenz N, Böttger D, Hegerl U. Utilization of Patient-Generated Data Collected Through Mobile Devices: Insights From a Survey on Attitudes Toward Mobile Self-Monitoring and Self-Management Apps for Depression. JMIR Ment Health 2019 Apr 03;6(4):e11671 [FREE Full text] [CrossRef] [Medline]
  163. Kenny R, Dooley B, Fitzgerald A. Feasibility of "CopeSmart": A Telemental Health App for Adolescents. JMIR Ment Health 2015 Aug;2(3):e22 [FREE Full text] [CrossRef] [Medline]
  164. Anderson AP, Fellows AM, Binsted KA, Hegel MT, Buckey JC. Autonomous, Computer-Based Behavioral Health Countermeasure Evaluation at HI-SEAS Mars Analog. Aerospace Medicine and Human Performance 2016 Nov 01;87(11):912-920. [CrossRef]
  165. Stiles-Shields C, Montague E, Lattie EG, Kwasny MJ, Mohr DC. What might get in the way: Barriers to the use of apps for depression. DIGITAL HEALTH 2017 Jun 08;3:205520761771382. [CrossRef]
  166. Gonsalves P, Hodgson E, Kumar A, Aurora T, Chandak Y, Sharma R, et al. Design and Development of the "POD Adventures" Smartphone Game: A Blended Problem-Solving Intervention for Adolescent Mental Health in India. Front Public Health 2019;7:238 [FREE Full text] [CrossRef] [Medline]
  167. Povey J, Mills PPJR, Dingwall KM, Lowell A, Singer J, Rotumah D, et al. Acceptability of Mental Health Apps for Aboriginal and Torres Strait Islander Australians: A Qualitative Study. J Med Internet Res 2016;18(3):e65 [FREE Full text] [CrossRef] [Medline]
  168. Hunter-Jones JJ, Gilliam SM, Carswell AL, Hansen NB. Assessing the Acceptability of a Mindfulness-Based Cognitive Therapy Intervention for African-American Women Living with HIV/AIDS. J. Racial and Ethnic Health Disparities 2019 Jul 22;6(6):1157-1166. [CrossRef]
  169. Henshall C, Marzano L, Smith K, Attenburrow M, Puntis S, Zlodre J, et al. A web-based clinical decision tool to support treatment decision-making in psychiatry: a pilot focus group study with clinicians, patients and carers. BMC Psychiatry 2017 Jul 21;17(1):265. [CrossRef] [Medline]
  170. Murnane EL, Cosley D, Chang P, Guha S, Frank E, Gay G, et al. Self-monitoring practices, attitudes, and needs of individuals with bipolar disorder: implications for the design of technologies to manage mental health. J Am Med Inform Assoc 2016 May;23(3):477-484. [CrossRef] [Medline]
  171. Lord S, Moore SK, Ramsey A, Dinauer S, Johnson K. Implementation of a Substance Use Recovery Support Mobile Phone App in Community Settings: Qualitative Study of Clinician and Staff Perspectives of Facilitators and Barriers. JMIR Ment Health 2016 Jun 28;3(2):e24 [FREE Full text] [CrossRef] [Medline]
  172. Caplan S, Sosa Lovera A, Reyna Liberato P. A feasibility study of a mental health mobile app in the Dominican Republic: the untold story. Int J Mental Health 2019 Apr 04;47(4):311-345. [CrossRef]
  173. Apolinário-Hagen J, Vehreschild V, Alkoudmani RM. Current Views and Perspectives on E-Mental Health: An Exploratory Survey Study for Understanding Public Attitudes Toward Internet-Based Psychotherapy in Germany. JMIR Ment Health 2017 Feb 23;4(1):e8. [CrossRef]
  174. Kumarasiri J, Jubb C. Framing of Climate Change Impacts and Use of Management Accounting Practices. AAMJAF 2018 Feb 28;13(2):45-68. [CrossRef]
  175. Huerta-Ramos E, Escobar-Villegas MS, Rubinstein K, Unoka ZS, Grasa E, Hospedales M, M-RESIST Group, et al. Measuring Users' Receptivity Toward an Integral Intervention Model Based on mHealth Solutions for Patients With Treatment-Resistant Schizophrenia (m-RESIST): A Qualitative Study. JMIR Mhealth Uhealth 2016 Sep 28;4(3):e112 [FREE Full text] [CrossRef] [Medline]
  176. Giroux D, Bacon S, King DK, Dulin P, Gonzalez V. Examining Perceptions of a Smartphone-Based Intervention System for Alcohol Use Disorders. Telemedicine and e-Health 2014 Oct;20(10):923-929. [CrossRef]
  177. Gonzales R, Douglas Anglin M, Glik D. Exploring the feasibility of text messaging to support substance abuse recovery among youth in treatment. Health Educ Res 2014 Feb;29(1):13-22 [FREE Full text] [CrossRef] [Medline]
  178. Lattie EG, Ho J, Sargent E, Tomasino KN, Smith JD, Brown CH, et al. Teens Engaged in Collaborative Health: The Feasibility and Acceptability of an Online Skill-Building Intervention for Adolescents at Risk for Depression. Internet Interv 2017 Jun;8:15-26 [FREE Full text] [CrossRef] [Medline]
  179. Miatello A, Mulvale G, Hackett C, Mulvale A, Kutty A, Alshazly F. Data Elicited Through Apps for Health Systems Improvement: Lessons From Using the myEXP Suite of Smartphone and Web Apps. International Journal of Qualitative Methods 2018 Oct;17(1):1-13. [CrossRef]
  180. O'Brien KHM, Battalen A, Sellers C, Spirito A, Yen S, Maneta E, et al. An mHealth approach to extend a brief intervention for adolescent alcohol use and suicidal behavior: Qualitative analyses of adolescent and parent feedback. J Technol Hum Serv 2019;37(4):255-285 [FREE Full text] [CrossRef] [Medline]
  181. Sreejith G, Menon V. Mobile Phones as a Medium of Mental Health Care Service Delivery: Perspectives and Barriers among Patients with Severe Mental Illness. Indian Journal of Psychological Medicine 2019 Oct 01;41(5):428-433. [CrossRef]
  182. Burda C, Haack M, Duarte A, Alemi F. Medication adherence among homeless patients: a pilot study of cell phone effectiveness. J Am Acad Nurse Pract 2012 Nov;24(11):675-681. [CrossRef] [Medline]
  183. Berry N, Bucci S, Lobban F. Use of the Internet and Mobile Phones for Self-Management of Severe Mental Health Problems: Qualitative Study of Staff Views. JMIR Ment Health 2017 Nov 01;4(4):e52. [CrossRef] [Medline]
  184. Whealin JM, Jenchura EC, Wong AC, Zulman DM. How Veterans With Post-Traumatic Stress Disorder and Comorbid Health Conditions Utilize eHealth to Manage Their Health Care Needs: A Mixed-Methods Analysis. J Med Internet Res 2016 Oct 26;18(10):e280 [FREE Full text] [CrossRef] [Medline]
  185. Nicholas J, Fogarty AS, Boydell K, Christensen H. The Reviews Are in: A Qualitative Content Analysis of Consumer Perspectives on Apps for Bipolar Disorder. J Med Internet Res 2017 Apr 07;19(4):e105 [FREE Full text] [CrossRef] [Medline]
  186. Fortuna KL, Storm M, Naslund JA, Chow P, Aschbrenner KA, Lohman MC, et al. Certified Peer Specialists and Older Adults With Serious Mental Illnessʼ Perspectives of the Impact of a Peer-Delivered and Technology-Supported Self-Management Intervention. The Journal of Nervous and Mental Disease 2018;206(11):875-881. [CrossRef]
  187. Pung A, Fletcher SL, Gunn JM. Mobile App Use by Primary Care Patients to Manage Their Depressive Symptoms: Qualitative Study. J Med Internet Res 2018 Sep 27;20(9):e10035 [FREE Full text] [CrossRef] [Medline]
  188. Carr A, Jones J, Mikulich Gilbertson S, Laudenslager M, Kutner J, Kilbourn K, et al. Impact of a Mobilized Stress Management Program (Pep-Pal) for Caregivers of Oncology Patients: Mixed-Methods Study. JMIR Cancer 2019 May 03;5(1):e11406 [FREE Full text] [CrossRef] [Medline]
  189. Jarvis MA, Chipps J, Padmanabhanunni A. "This phone saved my life": Older persons’ experiences and appraisals of an mHealth intervention aimed at addressing loneliness. Journal of Psychology in Africa 2019 May 05;29(2):159-166. [CrossRef]
  190. Klein P, Lawn S, Tsourtos G, van Agteren J. Tailoring of a Smartphone Smoking Cessation App (Kick.it) for Serious Mental Illness Populations: Qualitative Study. JMIR Hum Factors 2019 Sep 03;6(3):e14023 [FREE Full text] [CrossRef] [Medline]
  191. Simblett S, Greer B, Matcham F, Curtis H, Polhemus A, Ferrão J, et al. Barriers to and Facilitators of Engagement With Remote Measurement Technology for Managing Health: Systematic Review and Content Analysis of Findings. J Med Internet Res 2018 Jul 12;20(7):e10480 [FREE Full text] [CrossRef] [Medline]
  192. Aardoom J, Dingemans A, van Ginkel JR, Spinhoven P, Van Furth EF, Van den Akker-van Marle ME. Cost-utility of an internet-based intervention with or without therapist support in comparison with a waiting list for individuals with eating disorder symptoms: a randomized controlled trial. Int J Eat Disord 2016 Dec;49(12):1068-1076. [CrossRef] [Medline]
  193. Carolan S, Harris P, Greenwood K, Cavanagh K. Increasing engagement with an occupational digital stress management program through the use of an online facilitated discussion group: Results of a pilot randomised controlled trial. Internet Interv 2017 Dec;10:1-11 [FREE Full text] [CrossRef] [Medline]
  194. Ho J, Corden ME, Caccamo L, Tomasino KN, Duffecy J, Begale M, et al. Design and evaluation of a peer network to support adherence to a web-based intervention for adolescents. Internet Interventions 2016 Nov;6:50-56. [CrossRef]
  195. Espinosa H, Carrasco A, Moessner M, Cáceres C, Gloger S, Rojas G, et al. Acceptability Study of "Ascenso": An Online Program for Monitoring and Supporting Patients with Depression in Chile. Telemed J E Health 2016 Jul;22(7):577-583. [CrossRef] [Medline]
  196. Clarke J, Proudfoot J, Whitton A, Birch M, Boyd M, Parker G, et al. Therapeutic Alliance With a Fully Automated Mobile Phone and Web-Based Intervention: Secondary Analysis of a Randomized Controlled Trial. JMIR Mental Health 2016 Feb 25;3(1):e10. [CrossRef]
  197. Muuraiskangas S, Harjumaa M, Kaipainen K, Ermes M. Process and Effects Evaluation of a Digital Mental Health Intervention Targeted at Improving Occupational Well-Being: Lessons From an Intervention Study With Failed Adoption. JMIR Ment Health 2016 May 11;3(2):e13 [FREE Full text] [CrossRef] [Medline]
  198. Schroeder J, Wilks C, Rowan K, Toledo A, Paradiso A, Czerwinski M, et al. Pocket skills: A conversational Mobile web app to support dialectical behavioral therapy. : Association for Computing Machinery; 2018 Presented at: CHI'18; April 21-26; Montreal, Canada p. 1-15. [CrossRef]
  199. Kim E, Gellis ZD, Bradway C, Kenaley B. Key Determinants to using Telehealth Technology to Serve Medically Ill and Depressed Homebound Older Adults. Journal of Gerontological Social Work 2018 Jul 24;62(4):451-474. [CrossRef]
  200. Terp M, Jørgensen R, Laursen B, Mainz J, Bjørnes CD. A Smartphone App to Foster Power in the Everyday Management of Living With Schizophrenia: Qualitative Analysis of Young Adults' Perspectives. JMIR Ment Health 2018 Oct 01;5(4):e10157 [FREE Full text] [CrossRef] [Medline]
  201. Williams A, Fossey E, Farhall J, Foley F, Thomas N. Recovery After Psychosis: Qualitative Study of Service User Experiences of Lived Experience Videos on a Recovery-Oriented Website. JMIR Ment Health 2018 May 08;5(2):e37 [FREE Full text] [CrossRef] [Medline]
  202. Saunders K, Bilderbeck A, Panchal P, Atkinson L, Geddes J, Goodwin G. Experiences of remote mood and activity monitoring in bipolar disorder: A qualitative study. Eur Psychiatry 2017 Mar;41:115-121 [FREE Full text] [CrossRef] [Medline]
  203. Shrier LA, Spalding A. "Just Take a Moment and Breathe and Think": Young Women with Depression Talk about the Development of an Ecological Momentary Intervention to Reduce Their Sexual Risk. Journal of Pediatric and Adolescent Gynecology 2017 Feb;30(1):116-122. [CrossRef]
  204. Cernvall M, Sveen J, Bergh Johannesson K, Arnberg F. A pilot study of user satisfaction and perceived helpfulness of the Swedish version of the mobile app PTSD Coach. Eur J Psychotraumatol 2018;9(Suppl 1):1472990 [FREE Full text] [CrossRef] [Medline]
  205. Edbrooke-Childs J, Edridge C, Averill P, Delane L, Hollis C, Craven MP, et al. A Feasibility Trial of Power Up: Smartphone App to Support Patient Activation and Shared Decision Making for Mental Health in Young People. JMIR Mhealth Uhealth 2019 Jun 04;7(6):e11677. [CrossRef]
  206. Schlosser D, Campellone T, Kim D, Truong B, Vergani S, Ward C, et al. Feasibility of PRIME: A Cognitive Neuroscience-Informed Mobile App Intervention to Enhance Motivated Behavior and Improve Quality of Life in Recent Onset Schizophrenia. JMIR Res Protoc 2016 Apr 28;5(2):e77 [FREE Full text] [CrossRef] [Medline]
  207. Garrido S, Cheers D, Boydell K, Nguyen QV, Schubert E, Dunne L, et al. Young People's Response to Six Smartphone Apps for Anxiety and Depression: Focus Group Study. JMIR Ment Health 2019 Oct 02;6(10):e14385 [FREE Full text] [CrossRef] [Medline]
  208. Boggs J, Beck A, Felder J, Dimidjian S, Metcalf C, Segal ZV. Web-based intervention in mindfulness meditation for reducing residual depressive symptoms and relapse prophylaxis: a qualitative study. J Med Internet Res 2014 Mar 24;16(3):e87 [FREE Full text] [CrossRef] [Medline]
  209. Pierce B, Twohig MP, Levin ME. Perspectives on the use of acceptance and commitment therapy related mobile apps: Results from a survey of students and professionals. Journal of Contextual Behavioral Science 2016 Oct;5(4):215-224. [CrossRef]
  210. Puszka S, Dingwall KM, Sweet M, Nagel T. E-Mental Health Innovations for Aboriginal and Torres Strait Islander Australians: A Qualitative Study of Implementation Needs in Health Services. JMIR Ment Health 2016 Sep 19;3(3):e43 [FREE Full text] [CrossRef] [Medline]
  211. Bennett-Levy J, Singer J, DuBois S, Hyde K. Translating E-Mental Health Into Practice: What Are the Barriers and Enablers to E-Mental Health Implementation by Aboriginal and Torres Strait Islander Health Professionals? J Med Internet Res 2017 Jan 11;19(1):e1 [FREE Full text] [CrossRef] [Medline]
  212. Ye J, Shim R, Lukaszewski T, Yun K, Kim SH, Rust G. Telepsychiatry Services for Korean Immigrants. Telemedicine and e-Health 2012 Dec;18(10):797-802. [CrossRef]
  213. Koppe H, van de Mortel TF, Ahern CM. How effective and acceptable is Web 2.0 Balint group participation for general practitioners and general practitioner registrars in regional Australia? A pilot study. Aust J Rural Health 2016 Feb;24(1):16-22 [FREE Full text] [CrossRef] [Medline]
  214. Bohleber L, Crameri A, Eich-Stierli B, Telesko R, von WA. Can We Foster a Culture of Peer Support and Promote Mental Health in Adolescence Using a Web-Based App? A Control Group Study. JMIR Ment Health 2016 Sep 23;3(3):e45 [FREE Full text] [CrossRef] [Medline]
  215. Naslund JA, Aschbrenner KA, Bartels SJ. Wearable devices and smartphones for activity tracking among people with serious mental illness. Mental Health and Physical Activity 2016 Mar;10:10-17. [CrossRef]
  216. Rickard N, Arjmand H, Bakker D, Seabrook E. Development of a Mobile Phone App to Support Self-Monitoring of Emotional Well-Being: A Mental Health Digital Innovation. JMIR Ment Health 2016 Nov 23;3(4):e49 [FREE Full text] [CrossRef] [Medline]
  217. Reger GM, Browne KC, Campellone TR, Simons C, Kuhn E, Fortney JC, et al. Barriers and facilitators to mobile application use during PTSD treatment: Clinician adoption of PE coach. Professional Psychology: Research and Practice 2017 Dec;48(6):510-517. [CrossRef] [Medline]
  218. Whealin JM, King L, Shore P, Spira J. Diverse veterans’ pre- and post-intervention perceptions of home telemental health for posttraumatic stress disorder delivered via tablet. Int J Psychiatry Med 2017 May 09;52(1):3-20. [CrossRef]
  219. Kinner EM, Armer JS, McGregor BA, Duffecy J, Leighton S, Corden ME, et al. Internet-Based Group Intervention for Ovarian Cancer Survivors: Feasibility and Preliminary Results. JMIR Cancer 2018 Jan 15;4(1):e1. [CrossRef]
  220. Michalak EE, Morton E, Barnes SJ, Hole R, CREST.BD, Murray G. Supporting Self-Management in Bipolar Disorder: Mixed-Methods Knowledge Translation Study. JMIR Ment Health 2019 Apr 15;6(4):e13493 [FREE Full text] [CrossRef] [Medline]
  221. Townsend L, Zippay A, Caler K, Forenza B. Technology and Opportunity: People with Serious Mental Illness and Social Connection. Journal of the Society for Social Work and Research 2016 Jun;7(2):371-393. [CrossRef]
  222. Muuraiskangas S, Mattila E, Kyttälä P, Koreasalo M, Lappalainen R. User experiences of a mobile mental well-being intervention among pregnant women. : Springer Verlag; 2016 Presented at: International Symposium on Pervasive Computing Paradigms for Mental Health; November 28-29; Barcelona, Spain p. 140-149. [CrossRef]
  223. Connolly SL, Miller CJ, Koenig CJ, Zamora KA, Wright PB, Stanley RL, et al. Veterans’ Attitudes Toward Smartphone App Use for Mental Health Care: Qualitative Study of Rurality and Age Differences. JMIR Mhealth Uhealth 2018 Aug 22;6(8):e10748. [CrossRef]
  224. Brandt L, Hidalgo L, Diez-Canseco F, Araya R, Mohr D, Menezes P, et al. Addressing Depression Comorbid With Diabetes or Hypertension in Resource-Poor Settings: A Qualitative Study About User Perception of a Nurse-Supported Smartphone App in Peru. JMIR Ment Health 2019 Jun 18;6(6):e11701 [FREE Full text] [CrossRef] [Medline]
  225. East M, Havard B, Hastings N. Mental Health Mobile Apps’ Instruction: Technology Adoption Theories Applied in a Mixed Methods Study of Counseling Faculty. Journal of Technology in Human Services 2016 Oct 21;34(4):301-325. [CrossRef]
  226. Nicholas J, Huckvale K, Larsen ME, Basu A, Batterham PJ, Shaw F, et al. Issues for eHealth in Psychiatry: Results of an Expert Survey. J Med Internet Res 2017 Feb 28;19(2):e55. [CrossRef]
  227. Krog MD, Nielsen MG, Le JV, Bro F, Christensen KS, Mygind A. Barriers and facilitators to using a web-based tool for diagnosis and monitoring of patients with depression: a qualitative study among Danish general practitioners. BMC Health Serv Res 2018 Jun 27;18(1):503 [FREE Full text] [CrossRef] [Medline]
  228. Lemey C, Larsen ME, Devylder J, Courtet P, Billot R, Lenca P, et al. Clinicians’ Concerns About Mobile Ecological Momentary Assessment Tools Designed for Emerging Psychiatric Problems: Prospective Acceptability Assessment of the MEmind App. J Med Internet Res 2019 Apr 25;21(4):e10111. [CrossRef]
  229. Baumel A, Schueller SM. Adjusting an Available Online Peer Support Platform in a Program to Supplement the Treatment of Perinatal Depression and Anxiety. JMIR Mental Health 2016 Mar 21;3(1):e11. [CrossRef]
  230. Navarro P, Bambling M, Sheffield J, Edirippulige S. Exploring Young People’s Perceptions of the Effectiveness of Text-Based Online Counseling: Mixed Methods Pilot Study. JMIR Ment Health 2019 Jul 03;6(7):e13152. [CrossRef]
  231. Saurman E, Perkins D, Roberts R, Roberts A, Patfield M, Lyle D. Responding to mental health emergencies: implementation of an innovative telehealth service in rural and remote new South Wales, Australia. J Emerg Nurs 2011 Sep;37(5):453-459. [CrossRef] [Medline]
  232. Baumel A. Online emotional support delivered by trained volunteers: users' satisfaction and their perception of the service compared to psychotherapy. J Ment Health 2015;24(5):313-320. [CrossRef] [Medline]
  233. Saurman E, Kirby S, Lyle D. No longer 'flying blind': how access has changed emergency mental health care in rural and remote emergency departments, a qualitative study. BMC Health Serv Res 2015 Apr 14;15:156 [FREE Full text] [CrossRef] [Medline]
  234. Nicholas J, Shilton K, Schueller SM, Gray EL, Kwasny MJ, Mohr DC. The Role of Data Type and Recipient in Individuals' Perspectives on Sharing Passively Collected Smartphone Data for Mental Health: Cross-Sectional Questionnaire Study. JMIR Mhealth Uhealth 2019 Apr 05;7(4):e12578 [FREE Full text] [CrossRef] [Medline]
  235. Schuster L, Drennan J, Lings I. Understanding consumers’ decisions to adopt technology-enabled transformative services. The Service Industries Journal 2015 Oct 14;35(15-16):846-864. [CrossRef]
  236. Kasckow J, Zickmund S, Rotondi A, Mrkva A, Gurklis J, Chinman M, et al. Development of telehealth dialogues for monitoring suicidal patients with schizophrenia: consumer feedback. Community Ment Health J 2014 Apr;50(3):339-342 [FREE Full text] [CrossRef] [Medline]
  237. Vereenooghe L, Gega L, Langdon PE. Intellectual disability and computers in therapy: Views of service users and clinical psychologists. Cyberpsychology: Journal of Psychosocial Research on Cyberspace 2017 May 31;11(1):1-15. [CrossRef]
  238. Jorm AF. Mental health literacy: Empowering the community to take action for better mental health. American Psychologist 2012 Apr;67(3):231-243. [CrossRef]
  239. Helsper EJ, Eynon R. Distinct skill pathways to digital engagement. European Journal of Communication 2013 Sep 16;28(6):696-713. [CrossRef]
  240. 404 FE. European Citizens' Digital Health Literacy. European Commission. 2014.   URL: https://ec.europa.eu/commfrontoffice/publicopinion/flash/fl_404_en.pdf [accessed 2020-12-01]
  241. Apolinário-Hagen J, Fritsche L, Bierhals C, Salewski C. Improving attitudes toward e-mental health services in the general population via psychoeducational information material: A randomized controlled trial. Internet Interv 2018 Jun;12:141-149 [FREE Full text] [CrossRef] [Medline]
  242. Possemato K, Kuhn E, Johnson EM, Hoffman JE, Brooks E. Development and refinement of a clinician intervention to facilitate primary care patient use of the PTSD Coach app. Behav. Med. Pract. Policy Res 2016 Feb 22;7(1):116-126. [CrossRef]
  243. Jamison R, Jurcik D, Edwards R, Huang C, Ross E. A Pilot Comparison of a Smartphone App With or Without 2-Way Messaging Among Chronic Pain Patients: Who Benefits From a Pain App? Clin J Pain 2017 Aug;33(8):676-686 [FREE Full text] [CrossRef] [Medline]
  244. Richards P, Simpson S, Bastiampillai T, Pietrabissa G, Castelnuovo G. The impact of technology on therapeutic alliance and engagement in psychotherapy: The therapist's perspective. Clinical Psychologist 2020 Nov 10;22(2):171-181 [FREE Full text] [CrossRef]
  245. Schulze N, Reuter SC, Kuchler I, Reinke B, Hinkelmann L, Stöckigt S, et al. Differences in Attitudes Toward Online Interventions in Psychiatry and Psychotherapy Between Health Care Professionals and Nonprofessionals: A Survey. Telemed J E Health 2018 Nov 09;25(10):926-932. [CrossRef] [Medline]
  246. Simons L, Valentine AZ, Falconer CJ, Groom M, Daley D, Craven MP, et al. Developing mHealth Remote Monitoring Technology for Attention Deficit Hyperactivity Disorder: A Qualitative Study Eliciting User Priorities and Needs. JMIR mHealth uHealth 2016 Mar 23;4(1):e31. [CrossRef]
  247. Drozd F, Haga SM, Lisøy C, Slinning K. Evaluation of the implementation of an internet intervention in well-baby clinics: A pilot study. Internet Interventions 2018 Sep;13:1-7. [CrossRef]
  248. Peters D, Deady M, Glozier N, Harvey S, Calvo RA. Worker Preferences for a Mental Health App Within Male-Dominated Industries: Participatory Study. JMIR Ment Health 2018 Apr 25;5(2):e30. [CrossRef]
  249. Janz NK, Becker MH. The Health Belief Model: A Decade Later. Health Education Quarterly 2016 Sep 04;11(1):1-47. [CrossRef]
  250. Mohr DC, Cuijpers P, Lehman K. Supportive accountability: a model for providing human support to enhance adherence to eHealth interventions. J Med Internet Res 2011;13(1):e30 [FREE Full text] [CrossRef] [Medline]
  251. Schueller SM, Tomasino KN, Mohr DC. Integrating Human Support Into Behavioral Intervention Technologies: The Efficiency Model of Support. Clin Psychol Sci Pract 2016 Nov 17;24(1):27-45. [CrossRef]
  252. De Choudhury M, De S. Mental Health Discourse on reddit: Self-Disclosure, Social Support, and Anonymity Munmun. 2014 Presented at: Proceedings of the Eighth International AAAI Conference on Weblogs and Social Media; June 1–4; Ann Arbor, Michigan p. 71-80.
  253. Venkatesh V, Bala H. Technology Acceptance Model 3 and a Research Agenda on Interventions. Decision Sciences 2008 May;39(2):273-315 [FREE Full text] [CrossRef]
  254. Pung A, Fletcher SL, Gunn JM. Mobile App Use by Primary Care Patients to Manage Their Depressive Symptoms: Qualitative Study. J Med Internet Res 2018 Sep 27;20(9):e10035 [FREE Full text] [CrossRef] [Medline]
  255. Santini ZI, Koyanagi A, Tyrovolas S, Mason C, Haro JM. The association between social relationships and depression: a systematic review. J Affect Disord 2015 Apr 1;175:53-65. [CrossRef] [Medline]
  256. Ng MM, Firth J, Minen M, Torous J. User Engagement in Mental Health Apps: A Review of Measurement, Reporting, and Validity. Psychiatric Services 2019 Jul;70(7):538-544. [CrossRef] [Medline]


CBT: cognitive behavioral therapy
DMHI: digital mental health intervention


Edited by G Eysenbach; submitted 18.09.20; peer-reviewed by A Wasil, M Deady, N Andalibi, CY Lin; comments to author 07.11.20; revised version received 24.12.20; accepted 08.02.21; published 24.03.21

Copyright

©Judith Borghouts, Elizabeth Eikey, Gloria Mark, Cinthia De Leon, Stephen M Schueller, Margaret Schneider, Nicole Stadnick, Kai Zheng, Dana Mukamel, Dara H Sorkin. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 24.03.2021.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.