Factors Influencing Community Participation in Internet Interventions Compared With Research Trials: Observational Study in a Nationally Representative Adult Cohort

Introduction

Background

Digital mental health (DMH) programs are web-based interventions and apps that are designed to deliver evidence-based therapeutic content to individuals, with limited or no in-person clinical support. Low-intensity DMH programs, including self-help DMH programs, can be effective in treating and preventing high prevalence mental health problems such as depression or anxiety [1,2]. However, community uptake and participation in these programs remain poor [3,4], with studies in primary care reporting uptake rates as low as 3%-25% [5]. A range of factors have been proposed to explain this lack of participation, including low levels of awareness of DMH programs and the perception that web-based therapy is less effective than face-to-face care [6]. Thus, the capacity of web-based programs to provide broad-scale prevention and treatment in the community is currently limited, even though most people report that they would be willing to try a DMH program [7,8]. Identifying barriers and enablers to the use of DMH programs and developing strategies to address them are critical for optimizing their potential. An additional limitation of implementing evidence-based interventions, including both digital and in-person therapies, is that outcomes in research trials may not reflect real-world outcomes [9]. Delivery of DMH interventions in practice may not realize the success seen in controlled clinical trials [10], which typically have strict eligibility criteria and support from research staff. By examining the factors associated with engagement in research trials and comparing them with the factors associated with engagement in evidence-based interventions, we might be able to (1) better characterize implementation gaps and (2) understand differences in efficacy and use when interventions move from research to real-world delivery settings, reflecting the limited ecological validity of research trials.

Barriers and Enablers of Engaging With Internet Interventions

Previous research has documented factors influencing the uptake (enablers) or lack of uptake (barriers) of DMH programs. Much of the previous research has described individual factors influencing adherence to or ongoing engagement with the program; these factors include female gender [11], sufficient time [11], less severe symptoms at baseline [12,13], younger age [12], or realization of sufficient benefit without completing the full intervention [11,12,14-16]. Research has also examined factors impeding the uptake or initiation of DMH programs, including a lack of familiarity or acceptability of these types of programs; data security concerns; poor attitudes toward help-seeking; assumptions that e–mental health programs are not as effective as face-to-face therapy; a lack of internet access or anxiety around its use; and other personal factors such as poor education, low knowledge of technology, personality (eg, conscientiousness), and a lack of time [2,4,5,17-22]. Much of this research, with a few recent exceptions [23,24], focuses on examining factors influencing uptake based on intervention completion data, without asking the participants themselves about their perceptions of the barriers and enablers for use. Understanding perceived barriers for potential users may provide insights into the optimal design and implementation of internet interventions. Thus, research is needed to further investigate the factors that impede the uptake of DMH programs and that are perceived to be the most important barriers preventing their use.

Some of the identified factors influencing the uptake of DMH programs may be modifiable, such as the acceptability of DMH programs [19], whereas some factors such as age or personality factors are not. We may be able to improve uptake by designing interventions that target specific groups that would benefit the most [3,13] or by challenging some of the modifiable barriers before an individual commences an e-mental health program [5]. Several studies have developed and evaluated interventions (termed acceptance facilitation interventions or engagement facilitation interventions) that seek to improve uptake by reducing some of these barriers [5,25,26]. These interventions have primarily focused on increasing acceptance, although some have addressed barriers such as perceived needs, privacy concerns, and knowledge [5,25,26]. Although there is evidence that these interventions can increase the acceptability of internet interventions, a recent randomized controlled trial of an engagement facilitation intervention addressing multiple barriers to engagement found no significant increase in uptake or adherence to the subsequently presented e-mental health program [27], despite aligning the design of the engagement facilitation intervention to consumer views and preferences [23]. The limited success of acceptance and engagement facilitation interventions in changing implementation outcomes suggests that engagement in interventions is highly complex and likely involves a combination of factors that interact to influence behavior. The findings also suggest that consumer preferences do not always align with their eventual behavior. However, given the limited investigation of consumer preferences and priorities to date, more work is needed to better understand the drivers of engagement with DMH programs in both research and practice, particularly in samples that are broadly representative of the general population.

Barriers and Enablers of Engaging With Research Trials

A critical issue with DMH programs is that the findings from controlled research trials are not always consistent with outcomes when programs are implemented in clinical or community settings. Research trials benefit from the inclusion of a diverse range of participants, particularly as representation from a variety of groups increases the ability to generalize the findings to those populations [28]. However, many people may experience barriers to participation in research trials. In the general medical literature, barriers to clinical trial participation have included concerns about safety [29] or negative effects on health [30]; general fear or mistrust of medicine; the burden of trial participation [31]; and lack of access by not being offered participation [32], sometimes because of stringent inclusion criteria [33]. Enablers of participation in research trials include positive aspects of health care providers offering participation, perceived benefits, and altruism [31]. Few studies have specifically focused on factors associated with trial participation in the mental health field, although studies have examined factors influencing participation in partnership or cocreation projects, such as Living Labs [34]. From this literature, factors influencing participation in cocreation research include higher levels of socioeconomic status and more positive attitudes toward health and health care [35]. Other barriers to research engagement may include greater illness severity and functional impairments [36] and as with general health, strict eligibility criteria [37,38]. Stringent exclusion criteria are typical in mental health trials, including the trials of DMH programs, potentially resulting in highly selected samples [39]. The risk with such selection is that the effects observed in trials may not generalize when treatments are applied in real-world settings.

In summary, there is existing evidence of a range of factors influencing the uptake of DMH programs, but potential users’ perceived barriers to uptake are less frequently investigated. It also remains unclear which of these factors are the most influential in individuals’ decisions to engage or not engage with a DMH program. There has been no known previous research on the factors that influence participation in DMH research trials. In addition, previous research on factors associated with engagement has typically relied on convenience samples, which may bias the results because of underrepresentation from certain groups of the population. This study sought to fill these critical research gaps by using an Australian adult sample that was nationally representative in terms of age group, gender, and location (proportionally representing both rurality and states and territories of Australia). By comparing the similarities and differences in the factors that influence the uptake of interventions and factors associated with research participation, it may be possible to better understand the limitations of using trial data when implementing interventions in community and clinical settings.

Aims

The primary aims of this study were to identify factors that may influence participation in DMH programs in practice and in research trials of DMH programs (aim 1) and to assess whether these factors are different (aim 2) using survey data from a representative adult sample. Putative factors were selected based on existing evidence of barriers and enablers of engagement with DMH programs and research trials. Identifying these factors in a nationally representative adult sample may inform the targeting of implementation strategies for internet interventions and may be used to identify potential limitations in the ecological validity of research trials. We also explored whether any respondent characteristics influenced how the different factors were ranked (aim 3). Finally, to examine whether the self-rated likelihood of using DMH programs was an adequate proxy for actual uptake of DMH programs, we tested the relationship between participants’ perceptions of likelihood and actual uptake (aim 4), accounting for potential confounding factors.

Methods

Ethics Approval

This study was approved by The Australian National University Human Research Ethics Committee (ANU HREC protocol number 2020/593). Participants were paid according to small scheduled payments for completing surveys via Qualtrics Research Services (QRS) in the form of rewards (typically used for gift cards or airline reward programs of a small value, around Aus $2 (US $1.44) to Aus $4 (US $2.88) depending on the selected reward). Data were collected anonymously, and all participants provided written informed consent before commencing the study.

Finding are reported consistently with the Checklist for Reporting Results of Internet E-Surveys (CHERRIES).

Procedure: Recruitment and Data Collection

Participants were recruited through QRS during November 2020 to December 2020 using quota sampling to obtain a sample that was representative in terms of age group, gender, Australian state and territory populations, and remoteness (urban, regional, or rural area classification) to ensure that the sample collected was broadly representative of the Australian community. Participants from a selection of market research panels were invited by QRS to participate in the surveys. To avoid self-selection bias, survey invitations do not include specific details about the contents of the survey and are instead kept very general. After participants clicked the link from QRS, they were directed to our web page displaying the information sheet on the Qualtrics survey platform. Survey data were entered by participants on the Qualtrics platform. The inclusion criteria were that participants must be (1) living in Australia, (2) aged ≥18 years, and (3) able to read and write in English well enough to complete the survey. Participants were required to read the information sheet describing the key aspects of the study (including survey length, data storage and security, voluntary participation, the purpose of the survey, and the research team), agree that they met the inclusion criteria, and then provide consent to participate before completing the survey. Those who were screened as ineligible or did not provide their consent to participate were excluded from the study and provided with relevant mental health resources they may like to access. Participants could withdraw from the study at any time until completion by closing their web browser window, and the survey took approximately 20-30 minutes to complete. None of the questions were mandatory, except for the demographic questions used to determine the representativeness of the sample, and a back button was provided to review responses. Participants were excluded from the final sample if they failed to respond correctly to the 2 quality check items. QRS used IP addresses, cookies, and panel registrations to minimize repeat and poor-quality responses, excluding responses with completion time <7 minutes. If any participant endorsed the suicide screening item of the Patient Health Questionnaire-9 (PHQ-9) by selecting an option other than “not at all,” they were provided with a pop-up that asked them to telephone relevant crisis services (Lifeline, the Suicide Call Back Service, or emergency services).

Measures

Overview

The usability and technical functionality of the survey were tested by the authors before it was launched. The survey questions were delivered over 15 pages, with the number of questions per page ranging from 1 to 26. Before the section on use of DMH interventions (labeled “internet-based programs”), the following definition was provided: “Internet-based programs are online learning programs or smartphone apps that provide psychological therapy strategies, coping strategies (including mindfulness or relaxation) or support for lifestyle changes to support your mental health and well-being. Internet-based programs teach strategies that are used in face-to-face therapy for reducing symptoms of depression or anxiety.” Similarly, before the section on research participation (labeled “research trials”), a definition was provided: “A research trial is a study that is run by a university, hospital or health provider, testing whether an intervention (such as a pill, device or therapy) works effectively. Participating in a research trial involves having repeated measurements (such as answering surveys or having a medical examination) and being offered a treatment, which may be an active treatment or a control (placebo) treatment. In a research trial, the treatment received by each person is determined by chance – they don't have a choice as to whether they get the active or control treatment.”

Demographic Characteristics

The following demographic characteristics were assessed and included in this study: gender (men, women, and nonbinary), age (18-35, 36-55, and ≥56 years), level of education (high school or less, certificate or diploma, bachelor’s degree, and postgraduate degree), language spoken at home (English and English and other, or other language only), employment status (full time, part time or casual, unemployed, not working owing to study or maternity leave, retirement, etc), and region or area of residence (metropolitan area, regional area, and rural or remote area).

Attitudes and Use of DMH Interventions and Research Participation

We asked participants to rate on a scale of 1 to 5 (strongly disagree=1 to strongly agree=5) the extent to which they agreed with the statements “I am confident in my ability to use the Internet,” and “I want to change the way I think and feel about mental health problems.” We also asked if they had previously used an internet-based program to support their mental health or well-being (yes and no), if they had ever participated in a research trial (yes, maybe or not sure, and no), how likely they would be to participate in a research trial in the future, and how likely they would be to try an internet intervention in 8 hypothetical situations (eg, if a clinician recommended it, out of curiosity, etc) on a 5-point scale of 1 to 5 (extremely unlikely=1 to extremely likely=5). Finally, participants were provided with links to a range of existing DMH programs (Mindspot, myCompass, and MentalHealthOnline), and individuals who clicked on one or more links were identified as an indicator of actual uptake.

Factors Influencing the Use of DMH Programs and Research Participation

We asked participants to rank a list of factors influencing community participation in web-based mental health programs in two settings: (1) use of DMH programs in practice and (2) participation in a research trial of a DMH program. These lists included items such as, “Whether I could be anonymous or maintain my privacy.” The lists were compiled based on the existing evidence of barriers and enablers of engagement in both psychosocial interventions and treatment. Before completing the lists, the participants were provided with brief definitions of internet interventions and research trials. The lists of the factors were the same, although the items were reworded to match the setting. Participants could rank all 15 factors, but we requested that participants rank at least the top 5 factors. Participants ranked at least 5 factors, with most ranking only 5 factors (use of DMH: mean 5.2, SD 1.6; research trials: mean 5.2, SD 1.5), and 13 participants providing no ranking data. To reduce the chances of response bias because of item order, we randomized the presentation order for both lists of factors and hypothetical situations mentioned previously.

Symptom Measures

We used the PHQ-9 [40] to measure depression, the Generalized Anxiety Disorder-7 (GAD-7) [41] to assess anxiety, and the Distress Questionnaire-5 (DQ5) [42] to measure general psychological distress. The first 2 scales assess the frequency of symptoms of major depression (PHQ-9) or generalized anxiety disorder (GAD-7) experienced during the past 2 weeks (4-point scale from not at all=0 to nearly every day=3). Item scores were summed for each scale for the overall severity scores for depression (PHQ-9:9 items; range 0-27) and anxiety (GAD-7:7 items; range 0-21) symptoms. The 5 DQ5 items asked respondents to indicate the frequency of distressing situations, thoughts, and feelings over the previous 30 days using a 5-point scale (1=never, 2=rarely, 3=sometimes, 4=often, and 5=always). Scores were summed and ranged from 5 to 25. For each of these scales, higher scores indicate higher symptom severity, and all have robust psychometric evidence for reliability and validity [42-45]. In this study, all 3 scales had high internal consistency based on Cronbach α of .93 for PHQ-9, .95 for GAD-7, and .94 for DQ5.

Analyses

The ranked factors influencing the use of DMH programs and DMH research trial participation were reverse scored (eg, ranks 1, 2, and 3 correspond to points 3, 2, and 1, respectively) and summed so that higher scores indicated a higher ranked importance of a factor impacting participation. To compare the rankings of each factor across the 2 settings, the proportion of respondents who rated each factor as one of the top 3 factors influencing their participation was compared using McNemar test with Bonferroni correction for 15 comparisons (α=.003). Logistic regression analyses examined factors associated with high rankings (in the top 3) for the 3 most important factors associated with the 2 outcomes, with Bonferroni correction for the 6 models (α=.008). Candidate predictors of rankings included demographic factors, the likelihood of DMH program use (averaged across all scenarios), the likelihood of research trial participation, and mental health symptom scores. Finally, to examine whether the likelihood of use was related to actual uptake, we tested whether the average likelihood of DMH program use was associated with the past use of a DMH program or uptake of a DMH at the end of the survey, adjusting for demographic characteristics.

Results

Participant Characteristics

Table 1 presents the characteristics of the 1094 participants who completed the survey. The view rate was 31.18% (1094/3509) and the completion rate was 45% (n=3509 entered the survey, of which 117, 3.33% did not consent; 939, 26.76% did not meet demographic quotas; and 1359, 38.73% did not complete the survey or failed reliability checks). By design, the sample was representative of the Australian population in terms of age, gender, and location (both state and territory and remoteness). Although only 7.03% (77/1095) reported using the internet to support their mental health, most (655/1094, 59.87%) reported willingness to use a DMH program if it was recommended by a clinician. Most (783/1094, 71.57%) participants were somewhat or extremely likely to use a DMH program in at least one scenario. Across all scenarios, an average of 42.05% (460/1094) reported that they would be “somewhat likely” or “extremely likely” to use a DMH program. The likelihood of participating in a DMH research trial was similar, with 43.51% (476/1094) reporting that they would be “somewhat likely” or “extremely likely” to participate. Fewer than 17% (186/1094) of the participants reported prior participation in a research trial. There was a moderate correlation between the likelihood of using a DMH program and the likelihood of participating in a DMH research trial (r=0.31).

Factors Influencing Engagement With DMH Programs in Community and Research Settings

Table 2 presents the perceived factors influencing community participation in DMH programs compared with the factors influencing participation in trials of DMH programs. Trust in the organization delivering the service or trial was ranked at the top for both settings. Other factors that most highly ranked across both settings included anonymity or privacy, adequate information, and the level of need. Shame or embarrassment and other people being aware of participation or use were the lowest ranked for both settings. A direct comparison of the factors is provided in Table 3, based on the proportion of respondents who rated each factor as one of the top 3 factors influencing their participation. Two factors were more likely to be ranked highly for the use of DMH programs in practice compared with participation in DMH research trials: need for anonymity or privacy and access to other care. In contrast, 3 factors were less likely to be ranked highly in the use of DMH compared with research: need for information, effort required, and time required.

To examine whether there were characteristics associated with high rankings (top 3) of the selected factors, logistic regression models were estimated for the 3 highest-ranked items: trust, anonymity or privacy, and perceived need. The results are presented in Tables 4-6. Overall, there was little variation in rankings across demographic groups, intention to use or participate in DMH programs, and mental health status. The only significant associations were that older people were more likely to highly rank the item “Whether I trusted the organisation that delivers the program” than younger people (2% increased odds per year of age); and those who had higher intentions to use DMH programs were more likely to highly rank the item “Whether I thought I needed support for my mental health” (28% increased odds of high endorsement per one-unit increase in intentions).

Relationship Between DMH Use Intentions and Behaviors

Finally, to examine whether the self-rated likelihood of using DMH programs was a good proxy for actual uptake of DMH programs, we estimated 2 logistic regressions: one on past use of DMH programs and one on uptake of a DMH program within the study. The results of these analyses are presented in Table 7. The analyses indicated that use intentions were a strong indicator of both current and past behaviors. Specifically, there was 2.5-fold increase in the odds of engaging with one of the offered DMH programs per one-point increase in intentions and nearly 3-fold increase in the odds of past DMH program use per one-point increase in intentions. Younger age was also significantly associated with the past use of DMH programs, with a 6% increase in the odds of past use per year of age.

Discussion

Principal Findings

The findings indicate that among a representative Australian adult sample, a few key factors are the common influences of engagement with DMH programs in the context of both use in practice and in research trials. Although these factors may be considered as either “barriers” or “enablers” of uptake, each factor represents both a challenge to clearly articulate the potential difficulties in implementing DMH programs and an opportunity to educate and engage the target audience (ie, both a potential barrier and enabler). The most highly ranked considerations that influence the use of DMH programs were trust in the provider, need for privacy or anonymity, perceived need for support, and information about whether a DMH program is effective.

These outcomes differ somewhat from those of our previous community-based study in which sufficient time and perceived need dominated as key barriers to engagement [23]. Although perceived need was important in this study, the role of trust and need for privacy ranked higher, whereas time constraints had much less influence on use intentions. There are a few possible explanations for the discrepancies. First, we used a ranking approach rather than assigning equal weight to all reported barriers or enablers. Second, our focus was on the uptake or initiation of an intervention rather than on ongoing engagement. In agreement with this, a recent study on university medical students’ attitudes specifically toward the uptake of DMH found that privacy was also a key barrier to engaging in this group [24]. Third, barriers may change over time. For example, public awareness of privacy issues may have increased in recent years in response to problems with electronic medical records or media stories of hacking. Finally, our sample was selected to be representative of the national adult population in Australia in terms of age, gender and location, whereas most previous research on the factors influencing DMH use has relied on convenience samples. It is possible that convenience samples overrepresent people with mental health problem [46] or result in highly educated samples [47] that may have greater existing knowledge of DMH programs. Such biases may skew the preferences and priorities of a sample, leading to conclusions that may neglect the perspectives of underrepresented groups of the community. For example, the mean depression and anxiety symptom scores in this sample were approximately half of those seen in convenience samples [3].

The roles of trust and the need for privacy have been highlighted in previous studies [48-50] and are 2 themes that are likely to intersect [51]. Trust in the provider was the most important consideration both when conducting research trials and when delivering a DMH program in practice. This finding suggests that there may be not only challenges in building trust between providers and end users but also opportunities to promote DMH programs through trusted brands within the mental health sector. Organizations with established reputations for rigor, such as universities, may be trusted more by the public than commercial organizations [52]. Partnerships between stakeholders to ensure that DMH programs meet rigorous standards of safety and efficacy, combined with the reach and infrastructure of the industry and government, may be an important approach for further implementing DMH programs into practice. Privacy and data security have also been identified as key concerns. Clarity around what data are collected and how they are used is likely to be important in promoting uptake, along with emphasizing when programs are delivered anonymously. This information is often lacking or substandard in existing services [48].

The participants also ranked the need for sufficient information on the effectiveness of DMH programs as a key determinant of their use. Marketing DMH programs may benefit from presenting information about the effectiveness of the program in a way that is suitable for diverse audiences [53], without overstating or neglecting existing evidence [23]. Perceived need may be more challenging to address, as it is difficult for programs to engage people experiencing mental health problems if they are unaware that they may benefit from support. Strategies to address this challenge include the implementation of community mental health literacy programs [54], population-based screening with feedback [53,55], and implementation and marketing of DMH programs in diverse clinical and community settings [56]. Novel and effective approaches to research dissemination into the community may also raise awareness of the availability and evidence base of DMH programs. Encouragingly, stigma and embarrassment, along with the perceptions of others, were the least endorsed factors, suggesting that negative attitudes toward people with mental illness and negative attitudes toward treatment may have limited influence on the uptake of digital interventions.

Despite many similarities, there were a few differences in rankings between the use of DMH programs in practice and research. The need for anonymity and privacy was found to be more important for engaging with DMH programs in practice. Having access to other care was also more relevant to DMH in practice. Interestingly, the effort and time required were seen as more important in research trials, indicating that people view research as more burdensome than engaging with a service. This finding may be related to the additional burden of engaging with a research team and completing assessments in the context of a trial. These findings suggest that research cohorts may attract people with greater time and capacity than those who might otherwise engage in DMH programs. Such differences in the profiles of research participants may distort the trial outcomes when DMH programs are implemented in clinical or community settings. Furthermore, information needs were ranked higher in research trials, emphasizing the importance of clearly communicating expectations around participation in trials.

The rankings varied little by demographic characteristics, level of intention to use DMH programs, or mental health symptoms, suggesting that the perceived barriers and enablers to use are relatively homogeneous across the general population. Older adults were more likely to endorse trust as a key concern, suggesting that the reputation of a provider becomes increasingly important when delivering services to older adults. Those who reported a greater likelihood of using DMH programs were more likely to endorse perceived need as a determinant of use, which may indicate that those who are most likely to use programs tend to consider the level of support is needed (eg, no care, informal care, DMH programs, or face-to-face services) based on the severity of their symptoms. However, other demographic factors and mental health symptoms did not influence the relative importance of each factor.

Research on preferences for interventions is sometimes criticized for using behavioral intentions as a proxy for actual behavior. However, we also demonstrated that the likelihood of using a DMH program was strongly associated with both past and current engagement in these programs. These findings suggest that behavioral intentions are likely to be a good proxy for future use, as each 1-point increase in intentions was associated with 2.5-times the odds of engaging with one of the three DMH programs that were offered at the end of the survey. The findings also indicate that people who had engaged in DMH programs in the past had higher intentions for future use, which is reassuring as it suggests a level of satisfaction with past experiences.

Limitations

This study is one of very few to use a nationally representative adult sample to examine the factors associated with engagement in DMH programs. It is also the first, to our knowledge, to concurrently examine barriers and enablers of using DMH programs in practice and in research trials, providing useful information about why trial outcomes may not translate into practice. However, some limitations of this study should be acknowledged. Although we recruited the sample to be representative of key demographic characteristics, selection biases may still exist, such as those related to access to technology, interest in research, and interest in mental health. Potential participants may have withdrawn (or may have been more inclined to participate) when provided with information about the content of the survey, and some may have been motivated to participate by the small incentive. Furthermore, market research panels may not be representative of the community in terms of other characteristics. In particular, panel members may be more accustomed to research, more familiar with digital applications, and have potentially more favorable attitudes toward research than people who do not engage in such panels. As it is extremely challenging to engage research-averse people in research studies, methods to account for nonparticipation [57] may provide further insight into barriers to service engagement. The study was cross-sectional; therefore, the direction of the effects and causation cannot be inferred. Our indicators of the likelihood of using DMH programs were ad hoc, using hypothetical scenarios that may not reflect actual use or specific programs. Nevertheless, these indicators were strongly related to self-reported past use and the choice to engage in a DMH program at the end of the survey. It is also acknowledged that clicking on the link to a DMH program signifies behavior toward using a program, rather than direct engagement with program content, and factors such as previous experience using DMH programs may have influenced the decision to click on links. Our choice of barriers and enablers associated with the decision to engage in a DMH program was intended to be comprehensive, covering the most commonly endorsed factors identified in previous research. However, there may be other factors that we did not consider. Finally, other unmeasured factors such as personality, access to resources, socioeconomic status, and ethnicity may also influence the observed associations.

Conclusions

The factors that influence engagement in DMH programs are largely consistent in the context of both use in practice and research trials, with some key exceptions. Trust in providers, need for privacy or anonymity, perceived need for support, and information about the effectiveness of DMH programs were commonly endorsed in both settings, whereas time commitment was a greater consideration for research trials than for engaging in DMH programs in practice. Bridging the implementation gap between evidence and the use of DMH programs requires clearly communicating to users the legitimacy of the organization delivering the program. Organizations that deliver DMH programs are often successful at either developing evidence for efficacy and safety or marketing a program but not always both [58]. The findings of this study, in the context of growing literature on the implementation of DMH programs, suggest several additional considerations for optimizing the implementation of DMH programs, including providing transparent information about data use, demonstrating the effectiveness of the program, ensuring that programs are developed around the needs of end users [58], and providing clearer dissemination pathways with better marketing of evidence-based programs in community and clinical settings [56].