Background: The implementation of eMental health interventions, especially in the workplace, is a complex process. Therefore, learning from existing implementation strategies is imperative to ensure improvements in the adoption, development, and scalability of occupational eMental health (OeMH) interventions. However, the implementation strategies used for these interventions are often undocumented or inadequately reported and have not been systematically gathered across implementations in a way that can serve as a much-needed guide for researchers.
Objective: The objective of this scoping review was to identify implementation strategies relevant to the uptake of OeMH interventions that target employees and detail the associated barriers and facilitation measures.
Methods: A scoping review was conducted. The descriptive synthesis was guided by the RE-AIM (reach, effectiveness, adoption, implementation, and maintenance) framework and the Consolidated Framework for Implementation Research.
Results: A total of 31 of 32,916 (0.09%) publications reporting the use of the web-, smartphone-, telephone-, and email-based OeMH interventions were included. In all, 98 implementation strategies, 114 barriers, and 131 facilitators were identified. The synthesis of barriers and facilitators produced 19 facilitation measures that provide initial recommendations for improving the implementation of OeMH interventions.
Conclusions: This scoping review represents one of the first steps in a research agenda aimed at improving the implementation of OeMH interventions by systematically selecting, shaping, evaluating, and reporting implementation strategies. There is a dire need for improved reporting of implementation strategies and combining common implementation frameworks with more technology-centric implementation frameworks to fully capture the complexities of eHealth implementation. Future research should investigate a wider range of common implementation outcomes for OeMH interventions that also focus on a wider set of common mental health problems in the workplace. This scoping review’s findings can be critically leveraged by discerning decision-makers to improve the reach, effectiveness, adoption, implementation, and maintenance of OeMH interventions.
Mental health problems experienced by the working population are a global public health issue. Worldwide, more than 210 million people, representing 70% of those affected by common mental health disorders (eg, anxiety and mood disorders) are employed . Several risk factors, including working conditions, workplace culture, and the nature of work, have been linked to occupational mental health [ - ]. Public health emergencies, such as the COVID-19 pandemic, are linked to specific stressors, including the threat of infection, social distancing measures, stigma, and job insecurity, which considerably increase the prevalence of mental health problems in the working population [ ].
Occupational eMental health (OeMH) interventions significantly improve mental health in work settings . OeMH interventions use information and communication technology, including internet- and web-based services, mobile apps, and wearable technologies, to deliver knowledge and services such as psychoeducation, workplace health promotion, psychological and medical treatment, and return to work assistance to employees [ , ]. OeMH interventions have the potential to be more available, accessible, and scalable than traditional interventions [ , ], especially in public health emergencies, leading to physical-distancing policies to contain the spread of threatening conditions such as COVID-19.
However, implementing OeMH interventions is a complex process characterized by unique challenges involving adherence to new and crude regulatory frameworks, interoperability and compatibility with existing systems and procedures, threats to employees, organizational privacy and security, and associated costs . Newly introduced working arrangements in response to public health emergencies, such as the COVID-19 pandemic, could also compound existing implementation challenges and persist after the pandemic ends. Carefully developing and planning implementation strategies, which can be defined as a method or technique used to enhance the adoption, execution plan, and sustainability of an intervention [ ], is therefore essential to guarantee the sustainable uptake of OeMH interventions by employers and employees.
Nonetheless, it is difficult to establish a best practice for the implementation of OeMH interventions. Implementation strategies are often inadequately documented and seldom evaluated and published [, ], especially in comparison with studies on the effectiveness of interventions. Even when reported, implementation strategies have been discussed within a general context, and researchers have called for more tailored implementation strategies that focus on specific contexts [ ], for instance health care [ ]. Context encompasses the environment, broad setting, and circumstances (eg, systems and structures) in which an intervention is implemented and its associated characteristics [ ]. It is a key component of several widely adopted implementation frameworks, as evident in the Consolidated Framework for Implementation Research (CFIR) [ ]. Currently, those implementing new OeMH interventions are likely insufficiently informed about the procedure, strengths, and weaknesses of poorly documented implementation strategies, or uninformed about many potentially useful facilitators in this context. Furthermore, replicating positive results from similar implementations or overcoming barriers encountered in similar contexts would be challenging to achieve [ , ].
Therefore, a compilation of possible implementation strategies for OeMH interventions is critical to fostering improvements in their uptake and can serve as a reference for identifying and overcoming likely barriers and informing the future development of best practices. The objective of this scoping review was to identify implementation strategies relevant to the uptake of OeMH interventions that target employees and detail the associated barriers and facilitation measures. This scoping review would achieve these objectives by mapping the existing literature on the implementation of OeMH interventions and identifying gaps for future research. This work was conducted under the EMPOWER (European Platform to Promote Well-being and Health in the Workplace) project, funded by the European Commission, which investigates the impact of an eMental health platform aimed at preventing common mental health problems and reducing psychological distress in the workplace . It is also one of the series of review papers on different aspects of the knowledge base related to the development of the EMPOWER platform.
A scoping review was conducted to identify implementation strategies relevant to the implementation of OeMH interventions and to describe related barriers and associated facilitation measures. The scoping review is an established method for assessing and mapping the extent of evidence to address and inform practice in a topic area [- ]. The review proceeded through five stages as developed by Arksey and O’Malley [ ], extended by Levac et al [ ], and further modified by Westphaln et al [ ] to accommodate a team-based approach: (1) identifying the research question; (2) identifying relevant studies; (3) selecting studies; (4) charting the data; and (5) collating, summarizing, and reporting the results. Accordingly, this scoping review provides an overview of the existing evidence without a formal assessment of the methodological quality. It is conducted and reported in accordance with the widely adopted PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) [ ] to help ensure a high level of methodological rigor and reporting quality.
Electronic bibliographic databases, including MEDLINE, Scopus, CINAHL Complete, PsycINFO, and Web of Science Core Collection, were searched to find eligible peer-reviewed and gray literature. Search terms were based on concepts related to mental health, digital tools, the workplace, and implementation strategies (). The MEDLINE search strategy ( ) was adapted for other databases using relevant syntax and keywords in consultation with all coauthors who are also experienced researchers in the area. Hand searching of the reference lists of included articles was also completed for further relevant literature not identified during the search of databases. Members of the EMPOWER Consortium (ie, mental health researchers, clinicians, and experts focusing on well-being in the workplace) were also requested to suggest potentially eligible references via email.
Publications were eligible for inclusion if they described implementation strategies (ie, according to Proctor et al ) or related barriers or facilitation measures relevant to the uptake of OeMH interventions targeting employees. For example, all other eligibility criteria being met, approaches with the following characteristics would be considered: aim to introduce and encourage continued use of an intervention; prescribe actions in support of the intervention (eg, adaptations, fiscal strategies, and testing); and ensure that interventions can deliver intended benefits to the relevant organization over time, for instance, creating routine organizational policies or best practices. OeMH interventions are broadly defined here as mental health information and services delivered by information and communication technologies to employees [ , ]. This definition is consistent with the definition of eHealth [ , ], as well as the broader term digital health [ ]. Studies with employed participants aged ≥18 years, that were written in English. and published between January 2010 and May 2021 were considered. Primary research studies, systematic reviews, books, and gray literature (eg, conference proceedings, theses, government documents, and professional publications) were considered. Gray literature, such as commentaries, letters to editors, and editorials, were excluded.
A total of 10 researchers (AO-T, AR, CdM, CT, CMvdFC, DM-K, KS, MdM, MTP, and RMB), including psychologists, health scientists, and health economists, were involved in screening. To ensure consistency across researchers, they attended a web-based training workshop to practice the skills needed to reliably execute screening using the web and an app-based service Rayyan, Qatar Computing Research Institute . A training set of 100 publications was screened by all workshop attendees. Screening decisions (ie, include, maybe, or exclude) were reviewed and discussed to clarify any misunderstandings and identify difficulties using Rayyan QCRI. Instructions not to use the natural language processing–, artificial intelligence (AI)-, and machine learning–based features offered in Rayyan QCRI as well as tips to overcome minor usability shortcomings were given. Screeners were randomly assigned a screening set, and a screener performed a second screening of 20% of titles, abstracts, and full texts, and 100% of the publications that received a maybe screening decision. All screenings were conducted independently to reduce the likelihood of reviewer bias [ ] and inconsistencies in screening decisions were resolved in reconciliation meetings.
Data Extraction and Synthesis of Results
In all, 5 researchers (AR, CdM, CT, MdM, and RMB), including psychologists and health scientists, of the 10 (50%) screeners, were involved in data extraction and attended a web-based training workshop focused on developing consistency across researchers by practicing the skills needed to reliably execute data extraction using a web-based data extraction form. The form was reviewed and improved for clarity regarding the questions asked, user friendliness, and efficiency of data entry. For instance, it was clarified that single-component implementation strategies were to be extracted, and any bundling of strategies (ie, multifaceted strategies) in publications to address a goal were to be noted. Each researcher was randomly assigned an equal number of included records, and a researcher reviewed the extracted data for all the included publications.
A descriptive synthesis was performed, where identified implementation strategies, barriers, and facilitators were collated and later summarized. The synthesis was conducted by 3 (CT, MdM, and RMB, ie, psychologists and health scientists) of the 6 (50%) researchers involved in data extraction and guided by the RE-AIM (reach, effectiveness, adoption, implementation, and maintenance) framework [- ] and the CFIR [ ] and further informed by the Expert Recommendations for Implementing Change [ ]. RE-AIM and CFIR were chosen as they are widely used frameworks in implementation research (IR) [ ] and were deemed by the authors to be the most comprehensive of the recently reviewed implementation frameworks [ ] and most applicable to our objectives. The RE-AIM was originally developed as a framework for reporting findings regarding health promotion and disease management interventions in various settings. RE-AIM is used here to highlight essential strategy components with respect to its five steps: reach—the number of people who are willing to participate in a given initiative; effectiveness—the impact of an intervention on important outcomes (eg, individualistic and economic); adoption—the number of people or organizations who are willing to initiate and deliver an intervention; implementation—fidelity of delivery for the intervention including adaptations, costs, and consistency of delivery; and maintenance—sustained delivery and effects of an intervention after the associated initiative has ended. The CFIR unifies implementation theories to help build a robust implementation knowledge base across a wide range of studies, settings, contexts, and processes. The CFIR was used to provide a comprehensive view of multiple implementation contexts in which factors that might influence intervention implementation and effectiveness could be well detailed. Both frameworks determined the data for extraction: key publication characteristics, strategy definitions, key strategy implementation tasks, implementation processes, barriers and facilitators to strategy implementation, and any other data that holistically captured the complex and multilevel nature of strategy implementation were considered for data collection. Further synthesis of the identified barriers and facilitators produced recommendations for each relevant CFIR construct to improve the implementation of OeMH interventions.
A total of 31 publications were included in this scoping review (). details the methodological process followed, and a detailed itemization of the presented findings is provided in and .
|Citation and year of publication||Study aim and methods (n)||Country of implementation, industry, and participating organizations (n)||Intervention name, aim, and target conditions||Digital technologies used|
|, 2020||To develop, implement, and evaluate the intervention; survey (503), interviews (19), and focus groups (32)||United Kingdom; human health and social work activities; 7||Healthier Outcomes at Work Social Work Project; improve and manage; workplace stress and mental well-being||Smartphone app|
|,a 2020||To describe the intervention’s implementation; protocol—pilot randomized controlled trial (106)||China; human health and social work activities; 1||Step-by-Step F; improve; depressive symptoms and anxiety symptoms||Web-based and smartphone app|
|,a 2020||To describe the evaluation of the intervention’s implementation; protocol—focus groups (N/Rb)||Germany; agriculture, forestry, and fishing; N/Ac||With us in balance; prevent; stress-related disorders, anxiety disorders, mood disorders, substance-related and addictive disorders, insomnia, and chronic pain||Web-based and telephone|
|, 2020||To examine perspectives on the role and legitimacy of the intervention; interviews (32) and focus group (14)||Sweden; N/R; N/A||mWorks; support; common mental disorders||Smartphone app|
|,a 2020||To conduct preliminary evaluation of the intervention; pilot—usability study (81)||Australia; N/Sd; N/R||Anchored app; assess, improve, and monitor; depression, workplace stress, and mental well-being||Smartphone app|
|,a 2020||To rapidly develop and evaluate the intervention; stakeholder consultation groups (97), peer review panel (10), and intervention fidelity and implementation testing (55)||United Kingdom; human health and social work activities; N/R||Psychological Well-being in Healthcare Workers: Mitigating the Impacts of COVID-19; support and manage; workplace stress and mental well-being||Web-based|
|, 2019||To evaluate the feasibility, outcome, and acceptability of the intervention; proof-of-concept—survey (33)||United Kingdom; public administration and defense and compulsory social security; 2||Self-confidence webinar program; improve; mood disorders and depression||Web-based|
|, 2019||To evaluate engagement with the intervention; survey (149)||United States; public administration and defense and compulsory social security; 20||Stress Reduction Training for 9-1-1 Telecommunicators; improve and promote; workplace stress||Web-based|
|, 2019||To conduct formative evaluation of the intervention; interviews (24)||New Zealand; public administration and defense and compulsory social security; N/R||N/R; improve; stigma and discrimination||Web-based|
|, 2018||To evaluate adherence to the intervention; randomized controlled study (563)||Sweden; education; 21||N/R; improve and promote; workplace stress, occupational health, and sleep quality||Web-based|
|, 2018||To evaluate the helpfulness of the intervention; web-based survey (22) and focus groups (2)||United States; human health and social work activities; 1||Paving the Path to Mindfulness Website; improve; burnout and workplace stress||Web-based|
|, 2018||To evaluate acceptance and barriers to the uptake of OeMHe interventions; survey (3294)||N/A; N/A; N/A||N/A; manage; work-related distress||N/S|
|,a 2018||To evaluate the implementation strategy used; controlled trial (221)||The Netherlands; human health and social work activities; 1||Stress Prevention@Work; improve and prevent; workplace stress||Web-based|
|,a 2018||To evaluate the effectiveness of the implementation strategy used; follow-up controlled trial (252)||The Netherlands; human health and social work activities; 1||Stress Prevention@Work (SP@W); assess, improve, and prevent; workplace stress||Web-based|
|,a 2018||To identify key correlates of intention to use OeMH interventions; survey (1364)||China; human health and social work activities; N/A||N/A; N/A; mental health conditions||Web-based and smartphone app|
|,a 2018||To evaluate use of OeMH; log data and survey (1284)||Sweden; N/R; 6||N/R; improve, monitor, promote, and support; workplace stress and mental well-being||Web-based|
|, 2018||To develop and pilot-test the usability, acceptability, feasibility, and preliminary effectiveness of the intervention; prototype testing (21) and effectiveness and feasibility pilot study (84)||Australia; agriculture, forestry and fishing, manufacturing, and logistics; 3||HeadGear; improve; depressive symptoms||Smartphone app|
|,a 2018||To identify facilitators and barriers to engagement with OeMH interventions; interviews (18)||United Kingdom; information and communication, public administration and defense, education, and other service activities; 6||WorkGuru; improve; workplace stress||Web-based|
|,a 2017||To conduct process evaluation of the intervention; survey, log data, interviews, and observations (132)||The Netherlands; N/R; 2||eHealth module embedded in collaborative occupational health care; improve and monitor; mental well-being and return to work||Web-based|
|, 2017||To compare engagement with(out) a discussion group; pilot—3-arm randomized controlled trial (84)||United Kingdom; information and communication, public administration and defense, compulsory social security, education, and third sector organization; 6||WorkGuru; educate, improve, and monitor; workplace stress and nonworkplace stress||Web-based|
|, 2016||To investigate the influence of guidance formats on adherence of the intervention; pooled data from randomized controlled trials (395)||Germany; N/R; N/R||GET.ON Stress; improve and manage; workplace stress||Smartphone app|
|, 2016||To investigate men’s preferences for OeMH interventions’ design features; cross-sectional survey (841)||Canada; N/A; N/A||N/A; N/A; workplace stress and major depression||N/A|
|, 2016||To describe the development, implementation, and outcomes of; survey (1333)||United States; human health and social work activities; 1||Sleep Smart; improve and promote; poor sleep health|
|,a 2016||To describe approaches to and perspectives on OeMH interventions; N/A (N/A)||N/A; N/S; N/A||N/A; N/A; N/A||N/S|
|, 2016||To evaluate the potential effectiveness of the intervention and the effect of an online facilitated discussion group on engagement; protocol—3-arm randomized controlled trial (90)||United Kingdom; N/R; N/A||WorkGuru; educate, improve, and monitor; Workplace stress and nonworkplace stress||Web-based|
|, 2015||To describe the development the intervention; individual (34) and focus group (18) feedback sessions||United States; public administration and defense, compulsory social security, and human health and social work activities; N/R||Coming Home and Moving Forward; improve; stress-related disorders and substance-related and addictive disorders||Web-based|
|, 2014||To investigate users’ views on two different technologies for an OeMH intervention; survey within randomized controlled trial (637)||United Kingdom; transportation and storage, information and communication, and human health and social work activities; 3||Mood GYM; improve; mood disorders||Web-based|
|,a 2014||To contrast the role of differing managerial levels during the implementation of an OeMH; interviews (29)||Sweden; information and communication; public administration and defense; compulsory social security; education; and arts, entertainment, and recreation; 9||N/R; assess, improve, monitor, and promote; mental well-being||Web-based|
|,a 2014||To assess the feasibility of the intervention and explore barriers and /facilitators for the implementation of the intervention; process evaluation alongside a randomized controlled trial (116)||The Netherlands; financial and insurance activities; professional, scientific, and technical activities; public administration and defense; compulsory social security; and education; 6||Happy Work; improve and prevent; depressive symptoms||Web-based|
|,a 2013||To describe the development and implementation of the intervention; N/A (N/A)||International; N/R; N/R||HealthWatch; manage, prevent, and promote; mental well-being||Web-based|
|, 2010||To investigate determinants of high use of the intervention; randomized controlled intervention (303)||Sweden; information and communication and arts, entertainment, and recreation; N/R||N/R; assess, monitor, and promote; workplace stress||Web-based|
aFocused on implementation.
bN/R: not reported.
cN/A: not applicable.
dN/S: not specified.
eOeMH: occupational eMental health.
The 31 included publications comprised 28 journal articles [- , , , - , - ], 2 book chapters [ , ], and a doctoral dissertation [ ] ( and ). Most (25/31, 81%) [ , - , - ] of the included articles were published between 2015 and 2021 and were mainly primary studies (23/31, 74%) [ , , - , , , - , - , ]. Of the 31 included publications on OeMH interventions, 14 (45%) focused on their implementation [ , , , , , - , , , - ] and 17 (55%) did not focus on their implementation but had results or noted implications related to their implementation [ , , - , , - , ].
|Characteristics and citations||Frequency, n (%)|
|Book chapter [, ]||2 (6)|
|Doctoral dissertation ||1 (3)|
|Journal article [- , , , - , - ]||28 (90)|
|2010 ||1 (3)|
|2013 ||1 (3)|
|2014 [- ]||3 (10)|
|2015 ||1 (3)|
|2016 [, - ]||5 (16)|
|2017 [, ]||2 (6)|
|2018 [- ]||9 (29)|
|2019 [- ]||3 (10)|
|2020 [- ]||6 (19)|
|Narrative literature review [, ]||2 (6)|
|Pilot [, , ]||3 (10)|
|Primary study [, , - , , , - , - , ]||23 (74)|
|Protocol [, , ]||3 (10)|
A total of 24 interventions were reported in 27 studies [- , , - , - ] ( ). These interventions were largely web-based (n=16, 67%%) [ - , , , , - , - ] and most aimed to improve (n=19, 79%) [ , , , - , , - , - ] and, to a lesser extent, educate users about mental health problems. Most interventions have focused on stress-related disorders and symptoms (n=17, 71%) [ , - , - , - , ], but a wide range of mental health problems (eg, burnout, anxiety disorders, and substance-related disorders) have also been covered to some extent. Where reported [ , , , - , , - , , , - ], these interventions (n=19, 79%) largely targeted employees in professional occupations (eg, teachers and physicians). Most of these interventions were made available in specific countries, mainly in Europe (n=15, 63%%) [ , , , , , , , , - , , - ], except for one that was available internationally (n=1, 4%) [ ]. Standardized information about these 24 interventions, including year of launch, language, number of employees and employers interested in and who adopted the app, organizational size, and internal policies, was not clearly reported where relevant and could not be accurately extracted in detail.
|Characteristics and citations||Frequency, n (%)|
|Smartphone [, , , , ]||5 (21)|
|Web [- , , , , - , - ]||16 (67)|
|Web and smartphone ||1 (4)|
|Web and telephone ||1 (4)|
|Email ||1 (4)|
|Assess [, , , ]||4 (17)|
|Educate [, ]||1 (4)|
|Improve [, , , - , , - , - ]||19 (79)|
|Manage [, , , , ]||5 (21)|
|Monitor [, , , , , , ]||6 (25)|
|Prevent [, , , ]||4 (17)|
|Promote [, , , , , , ]||7 (29)|
|Support [, , ]||3 (13)|
|Target mental health problem|
|Anxiety disorders and symptoms [, ]||2 (8)|
|Burnout ||1 (4)|
|Chronic pain ||1 (4)|
|Common mental disorders ||1 (4)|
|Mood disorders and symptoms [, , , , , , , ]||8 (33)|
|Return to work ||1 (4)|
|Sleep problems [, , ]||3 (13)|
|Substance-related and addictive disorders [, ]||2 (8)|
|Stigma and discrimination ||1 (4)|
|Stress-related disorders and symptoms [, - , - , - , ]||17 (71)|
|Well-being problems [, , , , , , ]||7 (29)|
|Country of implementation|
|Australia [, ]||2 (8)|
|Canada ||1 (4)|
|China ||1 (4)|
|Germany [, ]||2 (8)|
|International ||1 (4)|
|The Netherlands [, , ]||3 (13)|
|New Zealand ||1 (4)|
|Sweden [, , , , ]||5 (21)|
|United Kingdom [, , , , , , ]||5 (21)|
|United States [, , , ]||4 (17)|
|Target occupational groups|
|Armed forces occupations [, ]||2 (8)|
|Clerical support worker [, , , ]||4 (17)|
|Elementary occupations (eg, cleaners and laborers) ||1 (4)|
|Managers (eg, chief executive officer) [, , ]||3 (13)|
|Not reported [, , , , ]||5 (21)|
|Plant and machine operators and assemblers ||1 (4)|
|Professionals (eg, teachers and physicians) [, , , , , , , - ]||10 (42)|
|Service and sales workers [, , ]||3 (13)|
|Social workers (ie, specifically child and family social workers) ||1 (4)|
|Skilled agricultural, forestry, and fishery workers [, ]||2 (8)|
|Technicians and associate professionals [, , , , , ]||6 (25)|
Overall, 98 examples of implementation strategies were identified (). categorizes these strategies into 17 discrete implementation strategies and maps them onto relevant RE-AIM domains based on the perceived intent of the themes. Each discrete implementation strategy is reported with the percentage and absolute number of defining strategy examples in relation to the 98 examples. Although evaluating effectiveness strategies is beyond the scope of this review, relevant effectiveness data could not be presented as they were largely absent or incomplete.
A total of 36 examples of implementation strategies were extracted from publications that focused on the implementation of OeMH interventions, and 62 from publications that reported results or noted implications related to their implementation. There were no notable differences other than the larger number of examples extracted from the latter group so strategy examples would not be reported separately. Most strategy examples were organized under implementation (61/98, 62%), followed by reach (27/98, 28%), effectiveness (19/98), adoption (17/98, 17%), and maintenance (8/98, 8%). A couple of strategy examples were organized into multiple domains (6/98, 6%). The following sections provide a descriptive summary of the strategy examples categorized in each RE-AIM domain.
|Discrete implementation strategies—proportion of strategy examples; n (%)||Example strategy||Relevant RE-AIM domains|
|Develop and organize implementation quality monitoring systems and act on insights in a timely manner where feasible; 17 (17)||Improve maintenance and adherence through the timely presentation of findings from monthly user feedback surveys where after follow-up actions can be immediately applied to the intervention ||Effectiveness, implementation, and maintenance|
|Assess for readiness and tailor strategies to address identified barriers and benefit from facilitators; 13 (13)||Filled knowledge gaps surrounding the effectiveness of eMental health interventions in the workplace by conducting systematic reviews on relevant topics ||Reach, effectiveness, implementation, and maintenance|
|Use mass media to increase reach; 9 (9)||Users were recruited by sharing information about the intervention through advertisements distributed via email and the organizations’ intranet and magazine ||Reach|
|Capture local knowledge from implementation sites and involve users early in the implementation and intervention development effort; 8 (8)||A consultation process was carried out with users, clinical psychologists, psychiatrists, information technology professionals, and design and user experience specialists to ensure the app’s content and design appealed to a broad range of workers from different industries ||Reach, effectiveness, adoption, implementation, and maintenance|
|Promote adaptability in the intervention to meet local needs without compromising fidelity; 8 (8)||Interventions were improved and adapted to each participating organization based on user feedback ||Reach, effectiveness, adoption, implementation, and maintenance|
|Send reminders; 7 (7)||Automatic email reminders were sent based on user-determined intervals and user inactivity ||Implementation|
|Provide support to users during the intervention; 6 (6)||Users were able to contact the intervention coach at any time to ask for feedback, additional help, or advice and the coach would respond within 24 hours ||Implementation|
|Conduct educational meetings; 5 (5)||Senior and middle management–led introductory seminars with employees that aimed to explain the intervention, secure acceptance, provide answers to questions, and inspire their participation ||Reach, adoption, and implementation|
|Provide incentives; 5 (5)||Users received a certificate of completion and the training was recognized as continuing education toward the renewal of their professional certification ||Reach and implementation|
|Identify and prepare organizational champions who will dedicate themselves to supporting, marketing, and driving the implementation; 4 (4)||Identification of champions at the implementation site facilitated organizational and employee buy-in ||Reach, adoption, implementation, and maintenance|
|Involve senior management; 4 (4)||The program was developed as a quality improvement project by the hospital and all research procedures (ie, retrospectively reviewing these outcomes) were approved by the institutional review board at the hospital ||Reach, adoption, implementation, and maintenance|
|Provide opportunities for users to obtain feedback on progress; 4 (4)||Participants received immediate and automatic tailored feedback and could monitor their own responses and trends over time ||Implementation|
|Stage implementation scale-up; 4 (4)||Conducted a pilot study aimed at assessing the usability, feasibility, acceptability, and preliminary effects of an app-based intervention designed to target depressive symptoms in a stressed working population ||Effectiveness and implementation|
|Customize recruitment activities to enhance reach; 3 (3)||When recruitment efforts did not attract enough participants, executives with the largest workforces in the region and industry were contacted directly via telephone and offered enrollment ||Reach and adoption|
|Develop and distribute educational materials; 3 (3)||All participants who returned the consent form received an email welcoming them to the study and explaining how to log in and use their personal webpage for the stress management program ||Reach, adoption, and implementation|
|Provide immediate opportunities to demonstrate commitment; 3 (3)||Management representatives were offered spots to enroll their organizations immediately after educational meetings about the intervention or to enroll at a later time ||Reach and adoption|
|Use advisory boards and workgroups to provide input and advice on implementation and improvements; 3 (3)||Systematic feedback was sought from researchers, expert clinicians, and veterans on the program and its content ||Implementation|
Mass media services were mainly used to increase reach (9/27, 33%). Examples include email [, , , , , , ], industry publications [ , ], targeted web-based advertisements (eg, Facebook) [ , ], and the organizations’ intranet [ , , ]. The provision of attractive incentives for participation (5/27, 19%) included monetary remuneration [ ], vouchers [ ], points for employee reward schemes [ ], educational credits for professional certifications [ ], and additional medical benefits [ ]. Other strategies included engaging potential users through educational meetings [ ] and materials [ ] (2/27, 7%), employees tasked with the responsibility of supporting the implementation of the intervention [ , ] (2/27, 7%), and well-timed opportunities to commit [ , ] (2/27, 7%). Local barriers to increasing reach among target users were identified through consultations with implementation sites, eligible users, and literature [ , , ] (3/27, 11%), and recruitment activities were later modified to avoid or overcome these barriers where possible [ ] (4/27, 15%).
Strategies to improve the effectiveness of the intervention mainly relied on insights obtained from a diverse group of professionals in relevant fields, representatives of implementation sites, target users, and intervention use data (12/19, 63%). This insight was captured through stakeholder consultations , steering group interviews and focus groups with target users [ ], peer review panels [ ], and user experience research [ , , , , , ] throughout the implementation process. Several strategies adopted an incremental approach to implementation [ , , ] (3/19, 16%), fostered adaptability in the intervention to adequately meet the local needs [ ] (2/19, 11%), or implemented measures to avoid or mitigate identified barriers that could negatively impact the effectiveness of the particular intervention [ , ] (2/19, 11%).
Sharing and discussing details about the proposed intervention with decision-makers was the most commonly used adoption strategy. This involved conducting educational meetings [, , , ] (4/17, 24%) and distributing educational materials about the intervention [ ] (1/17, 6%). Engaging senior management and others from the organization to identify necessary adaptations for intervention to succeed in the organization was also common. These strategies involved organizational stakeholders early in the intervention development process [ , ] (2/17, 12%) to adapt the intervention to meet special organizational needs without compromising fidelity [ ] (1/17, 6%) and address other identified barriers [ ] (1/17, 6%). Some strategies also identified staff members who could dedicate themselves to supporting, marketing, and driving the implementation within the organization, as this was expected to increase the likelihood of success [ , ] (2/17, 12%). The provision of immediate opportunities for decision-makers to confirm their commitment to adopt the intervention was also used [ ] (1/17, 6%).
Implementation strategies focused on adapting interventions and customizing the implementation process to implementation settings, monitoring the consistency of delivery, and providing various forms of support as needed. Implementers underwent training, subscribed to a common protocol, and had their work reviewed to help ensure fidelity. Some implementation strategies were continuously monitored using both qualitative and quantitative methods, including surveys, implementation reviews, process evaluations, and other similar methods [- , , , , , , , ] (16/61, 26%). A diverse group of stakeholders were involved in the assessments across the included studies. These assessments focused on measuring effectiveness, acceptability, and engagement [ - , , , , , , , ] (16/61, 26%). Findings were regularly applied quickly to overcome identified barriers and improve ongoing implementation processes [ , , , ] (10/61, 16%). Some support options included a reminder feature (7/61, 11%) where users could set their own reminder notifications [ ] and be notified when their participation level was too low [ , , ] or when new updates became available [ , ].
Maintenance strategies involved changes at the organizational level, where accommodating work conditions [, ] (2/8, 25%) and support staff [ ] (1/8, 13%) were sometimes arranged. Embedding interventions within existing employee programs was also expected to help sustain the use of the intervention [ ] (1/8, 13%). Special monitoring measures (eg, postintervention acceptability surveys and opportunities for monthly user feedback) were also established to provide insight into how benefits to users could be sustained after the initiative had officially ended [ , ] (2/8, 25%).
Barriers and Facilitators
The included publications reported 114 barriers and 131 facilitation measures (), and 28 barriers were accompanied by facilitation measures. There were no notable differences between barriers and facilitators extracted from publications that focused on the implementation of OeMH interventions (108/217, 49.8%) or that reported results or noted implications related to their implementation (109/217, 50.2%) so these will be reported together. Examples of barriers and facilitators organized by the relevant CFIR domains and associated constructs are provided in the corresponding tables. Most of the 217 identified barriers and facilitation measures were related to key attributes of interventions that influence successful implementation (103/217, 47.5%), followed by the inner setting of the organization (87/217, 40.1%), individual characteristics of target users (25/217, 11.5%), and the outer setting of the organization (2/217, 0.9%). The highest number of barriers were categorized under the inner setting (54/114, 47.4%), followed by intervention characteristics (35/114, 30.7%), individual characteristics of target users (22/114, 19.3%), and the outer setting of the organization (2/114, 1.8%) domains. The highest number of facilitators were categorized under intervention characteristics (77/131, 58.8%), followed by inner setting (44/131, 33.6%), individual characteristics of target users (9/131, 6.9%), and outer setting of the organization (1/131, 0.8%) domains.
Numerous barriers and facilitators were identified regarding how the interventions were bundled, presented, and assembled (ie, design quality and packaging) (). Participants from several studies considered web-based platforms to be an impersonal medium (eg, no face-to-face contact or human interaction) [ , , ], and some saw its use as inappropriate for helping with sensitive topics such as mental health problems [ ]. Several usability issues (eg, poor accessibility, technical issues, unclear navigational elements and user interface, and overly effortful tasks) have also emerged as barriers [ , , , , , ]. Accordingly, ensuring good usability [ , , , , , ] and considering individual factors (eg, high impulsivity benefits from continuous motivational components) [ , , , ] in the design were also often reported as facilitators.
The stakeholders’ perceptions of the evidence supporting the effectiveness of the proposed occupational mental interventions were influenced by several factors. The barriers included between-group contamination due to limited randomization at the individual level, unrepresentativeness of samples used for the general workforce, use of new or adapted measures with low reliability , and type 1 errors [ ]. Identified facilitators focused on the including diverse samples (eg, including underrepresented industries and occupations) [ ], collecting comparable demographic data [ ], including comprehensive engagement measures [ ], presenting interventions based on credible information highly relevant to target employees [ ], using control conditions when evaluating effectiveness [ , ], providing evidence from similar interventions that demonstrate effectiveness [ ], and conducting comprehensive and ongoing process evaluations to inform implementation [ , ].
|Relevant associated construct—proportion of barriers and facilitators; n (%) and brief description||Example of identified barriers||Example of identified facilitators|
|Evidence strength and quality; 15 (6.9); stakeholders’ perceptions of the quality and validity of evidence supporting the belief that the intervention will have desired outcomes||Using newly created or adapted measures demonstrating low reliability negatively impacts the strength of findings ||Providing evidence from other programs and interventions could be a strategy (oral presentations or reading materials) to demonstrate likely effectiveness |
|Relative advantage; 2 (0.9); stakeholders’ perception of the advantage of implementing the intervention versus an alternative solution||Possible low motivation from employers and organization in their employees return to work as they came from small- to medium-sized companies that had insurance for the costs of sickness absence ||The lack of a previous existing intervention for well-being in the organization, except for the intranet, which was difficult to use, so the app resulted to be a huge advantage for employees |
|Adaptability; 4 (1.8); the degree to which an intervention can be adapted, tailored, refined, or reinvented to meet local needs||Materials presented in a modular format that had to be completed start to finish in a single sitting or in a set order ||Possibility to use the program at their own pace |
|Design quality and packaging; 80 (36.9); perceived excellence in how the intervention is bundled, presented, and assembled||Usability was affected by unclear navigational elements and user interface ||Improving usability based on participant and expert feedback |
Strict external policies and failure of interventions to meet patient needs erected several barriers to the implementation of OeMH interventions (). For example, strict legislation and policies regarding privacy and confidentiality were highlighted as potential reasons for the reduced adoption of interventions based on innovative technologies [ ]. Moreover, failure to maintain employees’ confidentiality during these programs was believed to discourage the use of interventions for fear of being vulnerable to privacy breaches by employers [ ]. The sole facilitation measure identified for this CFIR domain also addresses this point by urging implementers to find ways to maintain employee confidentiality [ ].
|Relevant associated construct—proportion of barriers and facilitators; n (%) and brief description||Example of identified barriers||Example of identified facilitators|
|External policy and incentives; 1 (0.5); a broad construct that includes external strategies to spread interventions including policy and regulations (governmental or other central entity), external mandates, recommendations and guidelines, pay for performance, collaboratives, and public or benchmark reporting||The surrounding legislation and policy regulation of privacy and confidentiality may make it difficult to use innovative technology ||—a|
|Patient needs and resources; 1 (0.5); the extent to which patient needs, as well as barriers and facilitators to meet those needs, are accurately known and prioritized by the organization||Reluctancy of the potential participants in participating for fear of demonstrating vulnerability ||Maintaining confidentiality between employee and employer |
aNo facilitator reported.
Many publications have identified the lack of resources dedicated to implementation as a major barrier (). For example, there is a lack of time for employees to use the intervention [ , , , , , , ], funds to meet additional costs [ ], unreliable systems that lead to data loss [ ], inflexible participation times [ ], lack of workspaces to avoid office distractions and private spaces [ ] when completing interventions [ ], low technology (eg, computers and email) adoption by the organization [ ], little support from the app or implementor [ ], and insufficient resources for piloting [ ]. Some interventions were also inadequately adjusted to organizational processes [ , , , ] and insufficiently tailored to the work situation and culture [ , , , , ]. Organizational restructuring has also been identified as a barrier to successful implementation and should be considered during implementation planning [ , , ].
Several facilitators have also been identified. For example, it was recommended for employers to arrange dedicated time for employees to participate in the intervention ; to allow employees flexibility regarding the time, place, and pace when completing the intervention [ ]; to offer an option for employees to use the intervention in a private workspace [ ]; to provide recordings of any live sessions with feedback options [ ]; and to encourage employee access to or ownership of technology (eg, smartphone) in use [ ]. Intervention creators can also support employers with recruitment [ ], by obtaining support from a dedicated organizational support group for implementation [ ], providing lower-cost intervention options (eg, email based) [ ], using reliable data storage methods [ ], and demonstrating cost-effectiveness of the proposed intervention [ ].
|Relevant associated construct—proportion of barriers and facilitators; n (%) and brief description||Example of identified barriers||Example of identified facilitators|
|Structural characteristics; 4 (1.8%); the social architecture, age, maturity, and size of an organization||Personnel shortage, turnover, and organizational restructuring hindered the use of the strategy considerably ||Changes in the organizations should be considered (in light of resulting delays and communication problems) when planning intervention studies |
|Networks and communications; 4 (1.8%); the nature and quality of webs of social networks and the nature and quality of formal and informal communications within an organization||Restrictive internet security settings was a barrier for accessing the intervention ||Conduct onsite testing before implementation |
|Implementation climate; 17 (7.8); the absorptive capacity for change, shared receptivity of involved individuals to an intervention, and the extent to which use of that intervention will be rewarded, supported, and expected within their organization||Alignment with other stakeholders was absent and resulted in poor adherence to the recommended roles and tasks ||Embedding the intervention in a well-established wellness program to benefit from existing infrastructure to promote the intervention; users benefiting from incentive programs |
|Tension for change; 1 (0.5); the degree to which stakeholders perceive the current situation as intolerable or needing change||Some stakeholders may be reluctant to implement new technology as it might threaten their ability to keep their job ||—a|
|Compatibility; 21 (9.7); the degree of tangible fit between meaning and values attached to the intervention by involved individuals; how those align with individuals’ own norms, values, and perceived risks and needs; and how the intervention fits with existing workflows and systems||It was not possible for employees to contact their occupational physician themselves by telephone outside their regular consultations. This could have caused difficulty when an employee struggled with a module in Return@Work and wanted to ask the occupational physician for advice ||Alignment to relevant stakeholders is also important and can be attained by offering ongoing support to leaders at all organizational levels during an implementation |
|Organizational incentives and rewards; 2 (0.9); extrinsic incentives such as goal-sharing awards, performance reviews, promotions, and raises in salary and less tangible incentives such as increased stature or respect||Complimentary gifts (eg, measuring tapes to be used by users with diabetes) with logos and information stimulate discussions and act as reminders ||—|
|Readiness for implementation; 6 (2.8); tangible and immediate indicators of organizational commitment to its decision to implement an intervention||Ensuring fidelity as coaches could not provide good feedback without supervision ||Consult review boards and consider these issues early in the data planning process |
|Leadership engagement; 7 (3.2); commitment, involvement, and accountability of leaders and managers with the implementation||Senior management was not engaged and too much responsibility for implementation was given to the team members who did not prioritize these activities ||Adherence is better when managers are active and engaged |
|Available resources; 25 (11.5); the level of resources dedicated for implementation and ongoing operations including money, training, education, physical space, and time||The intervention required all participants to allocate the same time slot and competed with other time commitments ||Supporting statement from the employers which will suggest to all employees who participate in the study that they will have 1 hour per week over the 8-week period to complete the program |
|Access to knowledge and information; 2 (0.9); ease of access to digestible information and knowledge about the intervention and how to incorporate it into work tasks||Email messages from the decision aid supported the occupational physicians when guiding employees. The email gave them sufficient information and the layout was visually attractive |
aNo facilitator reported.
Characteristics of Individuals
Barriers were related to either the employer or the individual (). Employer-related barriers included the perception of low organizational commitment to addressing issues targeted by the proposed intervention [ ], perceived stigma associated with intervention adoption [ ], and a lack of privacy (eg, sharing information disclosed within the intervention with employers) [ ]. Individual-related barriers included a general lack of motivation and interest in using the intervention [ , ], no opportunities to interact with others during the intervention [ ], poor consistency in using the intervention as directed [ ], poor digital skills [ , , , ], difficulty relating to content [ ], low work ability [ ], and reduction in engagement and adoption due to symptoms associated with medical conditions [ , ]. Proposed facilitators include willingness to seek mental health support [ ], prior experience using an eHealth intervention and interventions that are freely accessible [ ], low technical skill requirement (eg, no authentication) [ ], and content that is available in multiple media formats (eg, printed versions) [ , ].
|Relevant associated construct—proportion of barriers and facilitators; n (%) and brief description||Example of identified barriers||Example of identified facilitators|
|Knowledge and beliefs about the intervention; 7 (3.2); individuals’ attitudes toward and the value placed on the intervention as well as familiarity with facts, truths, and principles related to the intervention||Skepticism toward the independence of the project from the organization ||Maintaining confidentiality between employee and employer |
|Self-efficacy; 12 (5.5); individual belief in their own capabilities to execute courses of action to achieve implementation goals||Lack of computer skills in team members ||The package developed in a free-to-access and simple format that does not require logging in to a system or any specific technical expertise |
|Other personal attributes; 6 (2.8); a broad construct to include other personal traits such as tolerance of ambiguity, intellectual ability, motivation, values, competence, capacity, and learning style||Barriers reported by participants at high risk for a major depressive episode included perceived stigma, lack of interaction with others that is characteristic of eMental health, lack of time, and lack of knowledge ||Willingness to seek professional mental health services |
Summary of Facilitation Measures
The identified facilitation measures were further synthesized and organized by the associated CFIR construct ().
|Associated CFIR construct||Facilitation measure|
|Evidence strength and quality||Strategies must provide evidence of effectiveness regarding the proposed or similar interventions in similar contexts featuring a representative sample of employees and a control group, where feasible, using valid and reliable measures.|
|Relative advantage||Strategies must be perceived to provide an advantage over the implementation of an alternative or no solution.|
|Adaptability||Strategies must allow flexibility on intervention completion times, the pace of progression, access options, and the format of provided materials.|
|Design quality and packaging||Strategies must ensure that the design of the intervention is based on an explicit understanding of users, their tasks, and environments and provides guidance (eg, reminders, knowledge base, progress tracking, and feedback); considers opportunities to integrate intervention features with organizational processes; creates personalized, informative, and nonstigmatizing content that encourages user participation; provides user adaptable content and tasks (ie, increased user control); allows access via additional modalities (eg, ability to print content) and formats (eg, video and audio); includes formative and summative usability testing and accessibility evaluations; highlights a strict approach to privacy and data security; and considers a multichannel recruitment strategy.|
|External policy and incentives||Strategies must identify and comply with applicable privacy legislation and policy regulations.|
|Structural characteristics||Strategies must consider the capacity of stakeholders to complete assigned tasks and account for turnover and other restructuring activities.|
|Networks and communications||Strategies must involve all stakeholders, include onsite testing of required technology, and establish clear communication procedures at the planning stage.|
|Implementation climate||Strategies must be cohesive and compatible with the organization’s culture (eg, high turnover and highly active working environment), ensure that interventions can be used in distraction-free environments (ie, free from excessive noise), account for prior negative experiences with similar interventions, secure support from senior management for strategy implementation, and leverage existing programs by embedding interventions into them.|
|Tension for change||Strategies must consider the impact of implementation on-the-job security of stakeholders and how that affects their perception of proposed changes.|
|Compatibility||Strategies must adequately reflect the implementation needs of the organization and its existing processes and policies; be aligned with stakeholders at different organizational levels; provide adequate separation between work and working with the intervention; and avoid stigmatization, especially of employees with mental health conditions.|
|Organizational incentives and rewards||Strategies should offer incentives for using the intervention and consider incorporating gamification components to offer these incentives.|
|Readiness for implementation||Strategies must ensure that stakeholders are involved in strategy development, aware of the strategy and their role in it, equipped with the necessary tools and access, and adequately trained to implement the strategy.|
|Leadership engagement||Strategies must secure support from all stakeholders, especially an active and engaged senior management who strongly sanctions and advocates for the intervention.|
|Available resources||Strategies must provide organizational support for implementation, intervention support for users, dedicated time and private spaces for completing interventions in the workplace, less time-intensive interventions, alternative options to live-participation activities (eg, live webinar recording), low-cost technology-based options (eg, email) for interventions, reliable cloud data storage, access from varying device types, and implementation cost estimates with demonstrated cost-effectiveness.|
|Access to knowledge and information||Strategies must provide information that sets realistic expectations about the intervention and how to implement it.|
|Knowledge and beliefs about the intervention||Strategies must clearly articulate the role of the organization in the development of the intervention and address privacy and stigmatization concerns associated with using mental health interventions.|
|Self-efficacy||Strategies must accommodate users whose performance is affected by symptoms (eg, lack of motivation) associated with their health conditions (eg, depression) and a lack of confidence using technology.|
|Individual identification with organization||Strategies must consider users’ perception of and level of commitment to the organization.|
|Other personal attributes||Strategies must address a lack of motivation (eg, due to symptoms associated with health conditions) to adopt and consistently use interventions and to seek help.|
Principal Findings and Comparison With Prior Work
The 31 included publications revealed 98 implementation strategies used when implementing OeMH interventions, 114 barriers, and 131 facilitators. The findings support observations [, ] that the reporting of implementation strategies used for eHealth interventions is largely incomplete, nonsystematic, and unstructured. Nonetheless, the findings provide valuable insights into what is known and where knowledge gaps lie in the area.
The OeMH knowledge base does not provide definitive answers regarding the implementation strategies to adopt and when and how it is most effective and efficient to adopt them. For example, the efficacy and cost-effectiveness of using innovative methods such as web-based targeted advertising compared with traditional methods (eg, posters) to increase reach is unclear [, ], despite the former’s success in being more time-efficient [ ] and effective at recruiting hard-to-reach populations [ , ]. Those responsible for implementation must use their judgment about which of the provided strategies would be most appropriate for their circumstances. These findings support the notion that the implementation of eHealth technology (eg, eMental health [eMH] interventions) is often narrowly seen as a postdevelopment activity rather than being a crucial part of the development process [ ]. Nonetheless, this could be partly a consequence of many included studies not specifically or comprehensively investigating implementation and therefore not reporting other details regarding implementation. Alternatively, publication restrictions [ ] (eg, strict word limits) and the multidisciplinary nature of digital health research [ ] may prioritize other study information over details regarding implementation when reporting on digital health interventions.
Barriers and Facilitators
Similar to findings related to medical devices , the findings here also suggest that usability [ ] appears to be the main design consideration in the evaluation of OeMH interventions, with little consideration given to other critical elements of the user experience. Findings regarding the CFIR inner setting domain highlight the need for researchers to articulate potential facilitators, including those that may have failed in one implementation context, as they might work in other contexts. Existing research [ , ] addresses many of the barriers (eg, associated with symptoms associated with mental health problems and limited digital literacy skills) categorized under the CFIR characteristics of individual domains and could provide an easy opportunity to improve implementation if given more consideration during the planning phase. Factors external to the organization (eg, external policies, partners, and competition) are known to greatly hinder or support the successful implementation of technology [ - ] but have been largely undocumented or overlooked by the included publications.
Recency of Work and Coverage of Technologies
Similar to recent eMH reviews focusing on college students  and user engagement [ ], this review also reported an increase since 2015 in eMH intervention studies meeting broad inclusion criteria. Recent reviews [ , ] also found that the eMH interventions described in the included studies were primarily web-based despite the added benefits of mobile apps that are coded for a specific mobile operating system such as iOS and Android (eg, faster, functionality-rich, and offline access) [ ]. This is perhaps because web-based interventions likely cost less to develop and could be accessed via more devices if they were developed in a responsive way [ ]. Emerging technologies, including AI, were considered in our search strategy, but were not used by the OeMH interventions described in the included studies. Nonetheless, this knowledge area is expected to increasingly feature the use of emerging technologies in the near future as the focus extends beyond nascent explorations of their applications for mental health and investigates the optimization of their implementation as well [ ].
Implications and Recommendations for Practice and Future Research
Based on the findings of the scoping review, four practical recommendations could be considered to avoid and mitigate the identified barriers and improve the implementation of OeMH interventions:
- Strategies must demonstrate a relative advantage over alternative solutions and promote flexibility in the delivery of interventions based on an explicit understanding of users, their tasks, and environments.
- Strategies must promote the active engagement of organizational leadership, assess organizational readiness, and ensure compatibility with the organization’s technological infrastructure and culture, in addition to providing desirable incentives and the necessary resources (eg, time and information about the intervention) for users to use the intervention as directed.
- Strategies must ensure transparency regarding the intervention and implications of use and help users build confidence in their ability to benefit from the intervention.
- Strategies must identify and ensure that interventions comply with applicable privacy legislation and policy regulations.
Future IR should continue with the broad aim of understanding what, why, and how OeMH interventions work under real-world conditions, and how to improve their implementation. The findings do not support the prioritization of any one aim over others. However, findings show that IR principles  such as the importance of context (eg, industry, size, and policies) and the people using the research need more attention for OeMH interventions. For example, surprisingly few findings were relevant to CFIR contextual domains (eg, outer setting), which speak to governmental regulations similar to COVID-19–related policies that have a strong influence on working arrangements. In addition, the general lack of detailed, systematic, and standardized reporting on proposed digital health interventions (eg, CONSORT-EHEALTH [Consolidated Standards of Reporting Trials of Electronic and Mobile Health Applications and Online Telehealth]—Expanded CONSORT figure) [ , - ] and the implementation strategies used to achieve these outcomes (eg, Standards for Reporting Implementation Studies) [ , ] need to be remedied for IR to be properly used in this area. Reporting could benefit from subscribing to technology-centric frameworks (eg, the mobile health evidence reporting and assessment checklist [ ] and the integrated technology implementation model [ - ]) that are more comprehensive in capturing key technology implementation factors (eg, accreditation, regulation, technology vendors, individual adoption factors, and interfacing systems). This should allow future studies to replicate and develop theories based on assessments of the implementation strategies used. In addition, any encountered or anticipated barriers and corresponding remedies that might be useful in avoiding these barriers or reducing their negative impact on implementation should also be reported. The development of an OeMH implementation checklist that includes comprehensive reporting guidelines and other prompts to ensure consistency and completeness when implementing these interventions would be beneficial. Future IR should also focus on investigating a wider range of common implementation outcomes (eg, cost-effectiveness and sustainability) [ ] facilitated by implementation strategies for OeMH interventions that also target more common mental health problems in the workplace (eg, anxiety, substance use, and addiction). Issues regarding lack of digital access and digital inequity are an ongoing challenge [ ], although not prominently featured in the results, and should be considered to avoid OeMH interventions contributing to any disparities. This study should also investigate how implementation strategies for OeMH interventions could benefit from emerging technologies. For instance, AI can use usage data to complement existing methods to better identify people who are at a high risk of mental health problems, support health decision-making, and offer resources that meet users’ individual health needs [ ]. This could have a profound positive impact on implementation through improvements in the effectiveness and maintenance of interventions.
Search results were limited to publications in English, and a publication date restriction was imposed from 2010 onwards; however, given the broad search strategy, it is not anticipated that many, if any, potentially eligible publications were missed as a result. The term eMental was coined in 2002 , merely 8 years before this review’s year restriction, and a recent review of 50 publications about OeMH interventions [ ] included 11 publications that were published before 2010 and none were eligible for inclusion in this study. In addition, despite our exhaustive search strategy, 6 publications from 2010 to 2015 compared with 25 from the subsequent 5-year period were eventually included. Incomplete reporting also made it challenging to detail strategies (eg, their effectiveness), barriers, facilitators, and contextual data (eg, industry, organizational size, and employee level) from the included publications and to synthesize these data later. Nonetheless, all researchers involved in data extraction completed the training specifically for this review, followed the same thorough approach, and the extracted data were reviewed at least once by a second researcher. Interrater reliability was not calculated, and reasons for disagreement in screening decisions were not reported, which might have affected the reproducibility of this study [ ]. However, this does not compromise the consistency and accuracy of the screening. Moreover, two 2-hour workshops were conducted with training sessions, and reconciliation meetings were consequently held when there were inconsistencies in screening decisions.
Although multiple implementation strategies can legitimately contribute to multiple RE-AIM domains, adopting a framework with more specificity could potentially be useful for the identification of more targeted strategies. Common implementation models (eg, RE-AIM and CFIR) predate the current development of eHealth, and concerns about their inability to fully capture the complexities of eHealth implementation have been raised  and persist [ ] despite some recent updates [ , ] and clarifications [ ]. Nonetheless, these generic frameworks are useful for guiding data extraction and as tools for making valuable comparisons with other types of interventions. Similar to other scoping reviews, this review reports on the nature and features of the literature on the topic of focus and does not attempt to present a view regarding the appropriateness of the used methods and the strength or quality of evidence. Similarly, the provision of more detailed recommendations would have been premature and potentially misleading, as this was unsupported by the data collected. Further research is needed to determine valid facilitators and how they should be used in the process of OeMH development and delivery on a case-by-case basis while considering contextual factors such as industry, organizational size, employee level, and internal and external policies. Nevertheless, these recommendations could still be particularly relevant for OeMH interventions in comparison with similar interventions in different contexts. Consequently, readers should be mindful that the review cannot determine whether the included studies provide robust or generalizable findings.
This scoping review represents one of the first steps in a research agenda aimed at improving the implementation of OeMH interventions by systematically selecting, shaping, evaluating, and reporting implementation strategies. It has identified 98 implementation strategies, 114 barriers, and 131 facilitation measures related to the implementation of these interventions. A synthesis of these findings offers 19 recommendations that provide initial guidance on how to improve the implementation of OeMH interventions. This scoping review also highlighted the need to combine common implementation models (eg, RE-AIM and CFIR) with more technology-centric frameworks (eg, integrated technology implementation model and the mobile health evidence reporting and assessment checklist) to fully capture the complexities of eHealth implementation. Despite yielding less detailed insight than hoped, owing to incomplete reporting and the adoption of incomprehensive frameworks by the included publications, this scoping review’s findings can still be critically leveraged by discerning decision-makers to improve the reach, effectiveness, adoption, implementation, and maintenance of OeMH interventions.
The research leading to these results received funding from the European Union Horizon 2020 Research and Innovation Programme under grant 848180 and the National Health and Medical Research Council of Australia under grant APP1195937. CdM received funding in the form of a predoctoral grant from Generalitat de Catalunya (PIF-Salut grant, code SLT017/20/000138).
The authors especially thank Dr Beatriz Olaya, Dr Christophe Vanroelen, Ellen Vorstenbosch, Dr Desirée Gutiérrez Marín, and other colleagues from the European Platform to Promote Well-being and Health in the Workplace Consortium for their generous feedback on the manuscript.
Conflicts of Interest
Search concepts and terms and MEDLINE search strategy.DOCX File , 16 KB
Identified examples of implementation strategies organized by relevant reach, effectiveness, adoption, implementation, and maintenance domains.XLSX File (Microsoft Excel File), 18 KB
Identified barriers and facilitators organized by relevant Consolidated Framework for Implementation Research domain and associated construct.XLSX File (Microsoft Excel File), 25 KB
- Depression and other common mental disorders: global health estimates. World Health Organization. 2017. URL: https://apps.who.int/iris/bitstream/handle/10665/254610/WHO-MSD-MER-2017.2-eng.pdf [accessed 2022-05-18]
- Young KP, Kolcz DL, O'Sullivan DM, Ferrand J, Fried J, Robinson K. Health care workers' mental health and quality of life during COVID-19: results from a mid-pandemic, national survey. Psychiatr Serv 2021 Feb 01;72(2):122-128. [CrossRef] [Medline]
- Cahill J, Cullen P, Anwer S, Wilson S, Gaynor K. Pilot Work Related Stress (WRS), effects on wellbeing and mental health, and coping methods. Int J Aerospace Psychol 2021 Jan 14;31(2):87-109. [CrossRef]
- Sampson H, Ellis N. Stepping up: the need for proactive employer investment in safeguarding seafarers’ mental health and wellbeing. Maritime Policy Manag 2020 Dec 30;48(8):1069-1081. [CrossRef]
- Hamouche S. COVID-19 and employees’ mental health: stressors, moderators and agenda for organizational actions. Emerald Open Res 2020 Apr 20;2:15. [CrossRef]
- Phillips EA, Gordeev VS, Schreyögg J. Effectiveness of occupational e-mental health interventions: a systematic review and meta-analysis of randomized controlled trials. Scand J Work Environ Health 2019 Nov 01;45(6):560-576 [FREE Full text] [CrossRef] [Medline]
- Riper H, Andersson G, Christensen H, Cuijpers P, Lange A, Eysenbach G. Theme issue on e-mental health: a growing field in internet research. J Med Internet Res 2010 Dec 19;12(5):e74 [FREE Full text] [CrossRef] [Medline]
- Lehr D, Geraedts A, Perrson Asplund R, Khadjesari Z, Heber E, de Bloom J, et al. Occupational e-mental health: current approaches and promising perspectives for promoting mental health in workers. In: Wiencke M, Cacace M, Fischer S, editors. Healthy at Work: Interdisciplinary Perspectives. Cham, Switzerland: Springer; Aug 27, 2016:257-281.
- Ha SW, Kim J. Designing a scalable, accessible, and effective mobile app based solution for common mental health problems. Int J Human Comput Interact 2020 Apr 26;36(14):1354-1367. [CrossRef]
- Rodriguez-Villa E, Naslund J, Keshavan M, Patel V, Torous J. Making mental health more accessible in light of COVID-19: scalable digital health with digital navigators in low and middle-income countries. Asian J Psychiatr 2020 Dec;54:102433. [CrossRef] [Medline]
- Purtova N, Kosta E, Koops BJ. Laws and regulations for digital health. In: Fricker SA, Thümmler C, Gavras A, editors. Requirements Engineering for Digital Health. Cham, Switzerland: Springer; 2014:47-74.
- Proctor EK, Powell BJ, McMillen JC. Implementation strategies: recommendations for specifying and reporting. Implement Sci 2013 Dec 01;8:139 [FREE Full text] [CrossRef] [Medline]
- Powell BJ, McMillen JC, Proctor EK, Carpenter CR, Griffey RT, Bunger AC, et al. A compilation of strategies for implementing clinical innovations in health and mental health. Med Care Res Rev 2012 Apr;69(2):123-157 [FREE Full text] [CrossRef] [Medline]
- Connolly SL, Hogan TP, Shimada SL, Miller CJ. Leveraging implementation science to understand factors influencing sustained use of mental health apps: a narrative review. J Technol Behav Sci (forthcoming) 2020 Sep 07:1-13 [FREE Full text] [CrossRef] [Medline]
- Graham AK, Lattie EG, Powell BJ, Lyon AR, Smith JD, Schueller SM, et al. Implementation strategies for digital mental health interventions in health care settings. Am Psychol 2020 Nov;75(8):1080-1092 [FREE Full text] [CrossRef] [Medline]
- Dryden-Palmer KD, Parshuram CS, Berta WB. Context, complexity and process in the implementation of evidence-based innovation: a realist informed review. BMC Health Serv Res 2020 Feb 03;20(1):81 [FREE Full text] [CrossRef] [Medline]
- Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci 2009 Aug 07;4:50 [FREE Full text] [CrossRef] [Medline]
- Varsi C, Solberg Nes L, Kristjansdottir OB, Kelders SM, Stenberg U, Zangi HA, et al. Implementation strategies to enhance the implementation of eHealth programs for patients with chronic illnesses: realist systematic review. J Med Internet Res 2019 Sep 27;21(9):e14255 [FREE Full text] [CrossRef] [Medline]
- Pinnock H, Barwick M, Carpenter CR, Eldridge S, Grandes G, Griffiths CJ, StaRI Group. Standards for Reporting Implementation Studies (StaRI): explanation and elaboration document. BMJ Open 2017 Apr 03;7(4):e013318 [FREE Full text] [CrossRef] [Medline]
- EMPOWER: The European platform to promote wellbeing and health in the workplace. EMPOWER Consortium. 2021. URL: https://empower-project.eu/ [accessed 2022-05-18]
- Colquhoun HL, Levac D, O'Brien KK, Straus S, Tricco AC, Perrier L, et al. Scoping reviews: time for clarity in definition, methods, and reporting. J Clin Epidemiol 2014 Dec;67(12):1291-1294. [CrossRef] [Medline]
- Levac D, Colquhoun H, O'Brien KK. Scoping studies: advancing the methodology. Implement Sci 2010 Sep 20;5:69 [FREE Full text] [CrossRef] [Medline]
- Arksey H, O'Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol 2005 Feb;8(1):19-32. [CrossRef]
- Khalil H, Peters M, Godfrey CM, McInerney P, Soares CB, Parker D. An evidence-based approach to scoping reviews. Worldviews Evid Based Nurs 2016 Apr;13(2):118-123. [CrossRef] [Medline]
- Westphaln KK, Regoeczi W, Masotya M, Vazquez-Westphaln B, Lounsbury K, McDavid L, et al. From Arksey and O'Malley and beyond: customizations to enhance a team-based, mixed approach to scoping review methodology. MethodsX 2021 May 7;8:101375 [FREE Full text] [CrossRef] [Medline]
- Tricco AC, Lillie E, Zarin W, O'Brien KK, Colquhoun H, Levac D, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med 2018 Oct 02;169(7):467-473 [FREE Full text] [CrossRef] [Medline]
- Global strategy on digital health 2020-2025. World Health Organization. 2021 Aug 18. URL: https://www.who.int/publications/i/item/9789240020924 [accessed 2022-05-10]
- Eysenbach G. What is e-health? J Med Internet Res 2001;3(2):E20 [FREE Full text] [CrossRef] [Medline]
- Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A. Rayyan-a web and mobile app for systematic reviews. Syst Rev 2016 Dec 05;5(1):210 [FREE Full text] [CrossRef] [Medline]
- Denison HJ, Dodds RM, Ntani G, Cooper R, Cooper C, Sayer AA, et al. How to get started with a systematic review in epidemiology: an introductory guide for early career researchers. Arch Public Health 2013 Aug 07;71(1):21 [FREE Full text] [CrossRef] [Medline]
- Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health 1999 Sep;89(9):1322-1327. [CrossRef] [Medline]
- Glasgow RE, Harden SM, Gaglio B, Rabin B, Smith ML, Porter GC, et al. RE-AIM planning and evaluation framework: adapting to new science and practice with a 20-year review. Front Public Health 2019 Mar 29;7:64 [FREE Full text] [CrossRef] [Medline]
- Holtrop JS, Estabrooks PA, Gaglio B, Harden SM, Kessler RS, King DK, et al. Understanding and applying the RE-AIM framework: clarifications and resources. J Clin Transl Sci 2021 May 14;5(1):e126 [FREE Full text] [CrossRef] [Medline]
- Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci 2015 Feb 12;10:21 [FREE Full text] [CrossRef] [Medline]
- Strifler L, Cardoso R, McGowan J, Cogo E, Nincic V, Khan PA, et al. Scoping review identifies significant number of knowledge translation theories, models, and frameworks with limited use. J Clin Epidemiol 2018 Aug;100:92-102. [CrossRef] [Medline]
- Ravalier JM, Wainwright E, Smyth N, Clabburn O, Wegrzynek P, Loon M. Co-creating and evaluating an app-based well-being intervention: the HOW (healthier outcomes at work) social work project. Int J Environ Res Public Health 2020 Nov 24;17(23):8730 [FREE Full text] [CrossRef] [Medline]
- Liem A, Garabiles MR, Pakingan KA, Chen W, Lam AI, Burchert S, et al. A digital mental health intervention to reduce depressive symptoms among overseas Filipino workers: protocol for a pilot hybrid type 1 effectiveness-implementation randomized controlled trial. Implement Sci Commun 2020 Oct 31;1:96 [FREE Full text] [CrossRef] [Medline]
- Freund J, Titzler I, Thielecke J, Braun L, Baumeister H, Berking M, et al. Implementing Internet- and tele-based interventions to prevent mental health disorders in farmers, foresters and gardeners (ImplementIT): study protocol for the multi-level evaluation of a nationwide project. BMC Psychiatry 2020 Aug 27;20(1):424 [FREE Full text] [CrossRef] [Medline]
- Engdahl P, Svedberg P, Lexén A, Bejerholm U. Role of a digital return-to-work solution for individuals with common mental disorders: qualitative study of the perspectives of three stakeholder groups. JMIR Form Res 2020 Sep 16;4(9):e15625 [FREE Full text] [CrossRef] [Medline]
- Collins DA, Harvey SB, Lavender I, Glozier N, Christensen H, Deady M. A pilot evaluation of a smartphone application for workplace depression. Int J Environ Res Public Health 2020 Sep 16;17(18):6753 [FREE Full text] [CrossRef] [Medline]
- Blake H, Bermingham F, Johnson G, Tabner A. Mitigating the psychological impact of COVID-19 on healthcare workers: a digital learning package. Int J Environ Res Public Health 2020 Apr 26;17(9):2997 [FREE Full text] [CrossRef] [Medline]
- Wan Mohd Yunus WM, Musiat P, Brown JS. Evaluating the feasibility of an innovative self-confidence webinar intervention for depression in the workplace: a proof-of-concept study. JMIR Ment Health 2019 Apr 26;6(4):e11401 [FREE Full text] [CrossRef] [Medline]
- Kerr DC, Ornelas IJ, Lilly MM, Calhoun R, Meischke H. Participant engagement in and perspectives on a web-based mindfulness intervention for 9-1-1 telecommunicators: multimethod study. J Med Internet Res 2019 Jun 19;21(6):e13449 [FREE Full text] [CrossRef] [Medline]
- Davey S, Gordon S, Tester R. Addressing police discrimination regarding mental distress using a service user-led and interpersonal contact/education based ‘e-Learning’. Police Pract Res 2019 Nov 07;22(1):426-442. [CrossRef]
- Villaume K, Tafvelin S, Hasson D. Health-relevant personality traits in relation to adherence to a web-based occupational health promotion and stress management intervention. Int J Workplace Health Manag 2018 Jun 04;11(3):143-158. [CrossRef]
- Rush KE. Paving the path to mindfulness: implementation of a program to reduce stress and burnout in inpatient psychiatric nurses. University of North Carolina Digital Repository. 2018. URL: https://cdr.lib.unc.edu/concern/dissertations/w37637581?locale=en [accessed 2022-05-18]
- Hennemann S, Witthöft M, Bethge M, Spanier K, Beutel ME, Zwerenz R. Acceptance and barriers to access of occupational e-mental health: cross-sectional findings from a health-risk population of employees. Int Arch Occup Environ Health 2018 Apr;91(3):305-316. [CrossRef] [Medline]
- Havermans BM, Boot CR, Brouwers EP, Houtman IL, Anema JR, van der Beek AJ. Process evaluation of a digital platform-based implementation strategy aimed at work stress prevention in a health care organization. J Occup Environ Med 2018 Sep;60(9):e484-e491. [CrossRef] [Medline]
- Havermans BM, Boot CR, Brouwers EP, Houtman IL, Heerkens YF, Zijlstra-Vlasveld MC, et al. Effectiveness of a digital platform-based implementation strategy to prevent work stress in a healthcare organization: a 12-month follow-up controlled trial. Scand J Work Environ Health 2018 Nov 01;44(6):613-621 [FREE Full text] [CrossRef] [Medline]
- Hall BJ, Shi W, Garabiles MR, Chan EW. Correlates of expected eMental Health intervention uptake among Filipino domestic workers in China. Glob Ment Health (Camb) 2018 Apr;5:e33 [FREE Full text] [CrossRef] [Medline]
- Frykman M, Lundmark R, von Thiele Schwarz U, Villaume K, Hasson H. Line managers’ influence on employee usage of a web-based system for occupational health management. Int J Workplace Health Manag 2018 Aug 06;11(4):193-209. [CrossRef]
- Deady M, Johnston D, Milne D, Glozier N, Peters D, Calvo R, et al. Preliminary effectiveness of a smartphone app to reduce depressive symptoms in the workplace: feasibility and acceptability study. JMIR Mhealth Uhealth 2018 Dec 04;6(12):e11661 [FREE Full text] [CrossRef] [Medline]
- Carolan S, de Visser RO. Employees' perspectives on the facilitators and barriers to engaging with digital mental health interventions in the workplace: qualitative study. JMIR Ment Health 2018 Jan 19;5(1):e8 [FREE Full text] [CrossRef] [Medline]
- Volker D, Zijlstra-Vlasveld MC, Brouwers EP, van der Feltz-Cornelis DF. Process evaluation of a blended Web-based intervention on return to work for sick-listed employees with common mental health problems in the occupational health setting. J Occup Rehabil 2017 Jun;27(2):186-194 [FREE Full text] [CrossRef] [Medline]
- Carolan S, Harris PR, Greenwood K, Cavanagh K. Increasing engagement with an occupational digital stress management program through the use of an online facilitated discussion group: results of a pilot randomised controlled trial. Internet Interv 2017 Dec;10:1-11 [FREE Full text] [CrossRef] [Medline]
- Zarski AC, Lehr D, Berking M, Riper H, Cuijpers P, Ebert DD. Adherence to Internet-based mobile-supported stress management: a pooled analysis of individual participant data from three randomized controlled trials. J Med Internet Res 2016 Jun 29;18(6):e146 [FREE Full text] [CrossRef] [Medline]
- Wang J, Lam RW, Ho K, Attridge M, Lashewicz BM, Patten SB, et al. Preferred features of e-mental health programs for prevention of major depression in male workers: results from a Canadian national survey. J Med Internet Res 2016 Jun 06;18(6):e132 [FREE Full text] [CrossRef] [Medline]
- Smith KC, Wallace DP. Improving the sleep of children’s hospital employees through an email-based sleep wellness program. Clin Pract Pediatric Psychol 2016 Sep;4(3):291-305. [CrossRef]
- Carolan S, Harris PR, Greenwood K, Cavanagh K. Increasing engagement with, and effectiveness of, an online CBT-based stress management intervention for employees through the use of an online facilitated bulletin board: study protocol for a pilot randomised controlled trial. Trials 2016 Dec 15;17(1):598 [FREE Full text] [CrossRef] [Medline]
- Possemato K, Acosta MC, Fuentes J, Lantinga LJ, Marsch LA, Maisto SA, et al. A Web-based self-management program for recent combat veterans with PTSD and substance misuse: program development and veteran feedback. Cogn Behav Pract 2015 Aug 01;22(3):345-358 [FREE Full text] [CrossRef] [Medline]
- Schneider J, Sarrami Foroushani P, Grime P, Thornicroft G. Acceptability of online self-help to people with depression: users' views of MoodGYM versus informational websites. J Med Internet Res 2014 Mar 28;16(3):e90 [FREE Full text] [CrossRef] [Medline]
- Hasson H, Villaume K, von Thiele Schwarz U, Palm K. Managing implementation: roles of line managers, senior managers, and human resource professionals in an occupational health intervention. J Occup Environ Med 2014 Jan;56(1):58-65. [CrossRef] [Medline]
- Geraedts AS, Kleiboer AM, Wiezer NM, Cuijpers P, van Mechelen W, Anema JR. Feasibility of a worker-directed Web-based intervention for employees with depressive symptoms. Internet Interv 2014 Jul;1(3):132-140. [CrossRef]
- Hasson D, Villaume K. An automated and systematic Web-based intervention for stress management and organizational health promotion. In: Bauer GF, Jenny GJ, editors. Salutogenic Organizations and Change. Dordrecht, The Netherlands: Springer; 2013:217-237.
- Hasson H, Brown C, Hasson D. Factors associated with high use of a workplace web-based stress management program in a randomized controlled intervention study. Health Educ Res 2010 Aug;25(4):596-607. [CrossRef] [Medline]
- Reagan L, Nowlin SY, Birdsall SB, Gabbay J, Vorderstrasse A, Johnson C, et al. Integrative review of recruitment of research participants through Facebook. Nurs Res 2019;68(6):423-432. [CrossRef] [Medline]
- Kayrouz R, Dear BF, Karin E, Titov N. Facebook as an effective recruitment strategy for mental health research of hard to reach populations. Internet Interv 2016 May;4:1-10 [FREE Full text] [CrossRef] [Medline]
- Whitaker C, Stevelink S, Fear N. The use of Facebook in recruiting participants for health research purposes: a systematic review. J Med Internet Res 2017 Aug 28;19(8):e290 [FREE Full text] [CrossRef] [Medline]
- van Gemert-Pijnen L, Kelders SM, Kip H, Sanderman R. eHealth Research, Theory and Development: A Multidisciplinary Approach. London, UK: Routledge; 2018:247-270.
- Pinnock H, Barwick M, Carpenter CR, Eldridge S, Grandes G, Griffiths CJ, StaRI Group. Standards for Reporting Implementation Studies (StaRI) statement. BMJ 2017 Mar 06;356:i6795 [FREE Full text] [CrossRef] [Medline]
- Agarwal S, LeFevre AE, Lee J, L'Engle K, Mehl G, Sinha C, WHO mHealth Technical Evidence Review Group. Guidelines for reporting of health interventions using mobile phones: mobile health (mHealth) evidence reporting and assessment (mERA) checklist. BMJ 2016 Mar 17;352:i1174. [CrossRef] [Medline]
- Bitkina OV, Kim HK, Park J. Usability and user experience of medical devices: an overview of the current state, analysis methodologies, and future challenges. Int J Ind Ergon 2020 Mar;76:102932. [CrossRef]
- Ergonomics of human-system interaction — Part 304: user performance test methods for electronic visual displays - ISO 9241-304:2008. International Organization for Standardization. 2018. URL: https://www.iso.org/standard/40099.html [accessed 2022-05-18]
- Bernard R, Sabariego C, Cieza A. Difficulties encountered by people with depression and anxiety on the Web: qualitative study and Web-based expert survey. J Med Internet Res 2019 Oct 31;21(10):e12514 [FREE Full text] [CrossRef] [Medline]
- Bernard R, Sabariego C, Cieza A. Barriers and facilitation measures related to people with mental disorders when using the web: a systematic review. J Med Internet Res 2016 Jun 09;18(6):e157 [FREE Full text] [CrossRef] [Medline]
- Schoville RR. Discovery of implementation factors that lead to technology adoption in long-term care. J Gerontol Nurs 2017 Oct 01;43(10):21-26. [CrossRef] [Medline]
- Schoville R, Titler MG. Integrated technology implementation model: examination and enhancements. Comput Inform Nurs 2020 Nov;38(11):579-589. [CrossRef] [Medline]
- Schoville RR, Titler MG. Guiding healthcare technology implementation: a new integrated technology implementation model. Comput Inform Nurs 2015 Mar;33(3):99-E1. [CrossRef] [Medline]
- Lattie EG, Adkins EC, Winquist N, Stiles-Shields C, Wafford QE, Graham AK. Digital mental health interventions for depression, anxiety, and enhancement of psychological well-being among college students: systematic review. J Med Internet Res 2019 Jul 22;21(7):e12869 [FREE Full text] [CrossRef] [Medline]
- Borghouts J, Eikey E, Mark G, De Leon C, Schueller SM, Schneider M, et al. Barriers to and facilitators of user engagement with digital mental health interventions: systematic review. J Med Internet Res 2021 Mar 24;23(3):e24387 [FREE Full text] [CrossRef] [Medline]
- Panhale M. Beginning Hybrid Mobile Application Development. New York, NY, USA: Springer; 2016:15-20.
- Baños RM, Herrero R, Vara MD. What is the current and future status of digital mental health interventions? Span J Psychol 2022 Feb 02;25:e5. [CrossRef] [Medline]
- Peters DH, Adam T, Alonge O, Agyepong IA, Tran N. Implementation research: what it is and how to do it. BMJ 2013 Nov 20;347:f6753. [CrossRef] [Medline]
- Eysenbach G, CONSORT-EHEALTH Group. CONSORT-EHEALTH: improving and standardizing evaluation reports of Web-based and mobile health interventions. J Med Internet Res 2011 Dec 31;13(4):e126 [FREE Full text] [CrossRef] [Medline]
- Eysenbach G. CONSORT-EHEALTH: implementation of a checklist for authors and editors to improve reporting of Web-based and mobile randomized controlled trials. In: Studies in Health Technology and Informatics. Amsterdam, The Netherlands: IOS Press; 2013:657-661.
- Glasgow RE, Huebschmann AG, Brownson RC. Expanding the CONSORT figure: increasing transparency in reporting on external validity. Am J Prev Med 2018 Sep;55(3):422-430. [CrossRef] [Medline]
- Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health 2011 Mar;38(2):65-76 [FREE Full text] [CrossRef] [Medline]
- Crawford A, Serhal E. Digital health equity and COVID-19: the innovation curve cannot reinforce the social gradient of health. J Med Internet Res 2020 Jun 02;22(6):e19361 [FREE Full text] [CrossRef] [Medline]
- Ebert DD, Harrer M, Apolinário-Hagen J, Baumeister H. Digital interventions for mental disorders: key features, efficacy, and potential for artificial intelligence applications. Adv Exp Med Biol 2019;1192:583-627. [CrossRef] [Medline]
- Christensen H, Griffiths K, Evans K. E-Mental Health in Australia: Implications of the Internet and Related Technologies for Policy. Canberra, Australia: Commonwealth Department of Health and Ageing; 2002.
- Belur J, Tompson L, Thornton A, Simon M. Interrater reliability in systematic review methodology: exploring variation in coder decision-making. Sociol Methods Res 2018 Sep 24;50(2):837-865. [CrossRef]
- Heinsch M, Wyllie J, Carlson J, Wells H, Tickner C, Kay-Lambkin F. Theories informing eHealth implementation: systematic review and typology classification. J Med Internet Res 2021 May 31;23(5):e18500 [FREE Full text] [CrossRef] [Medline]
|AI: artificial intelligence|
|CFIR: Consolidated Framework for Implementation Research|
|CONSORT-EHEALTH: Consolidated Standards of Reporting Trials of Electronic and Mobile Health Applications and Online Telehealth|
|eMH: eMental health|
|EMPOWER: European Platform to Promote Well-being and Health in the Workplace|
|IR: implementation research|
|OeMH: Occupational eMental health|
|PRISMA-ScR: Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews|
|RE-AIM: reach, effectiveness, adoption, implementation, and maintenance|
Edited by R Kukafka; submitted 28.10.21; peer-reviewed by M Deady, K Wall, C Guinemer, I Mircheva, C Tickner, C Sinha; comments to author 27.12.21; revised version received 02.03.22; accepted 30.03.22; published 01.06.22Copyright
©Renaldo M Bernard, Claudia Toppo, Alberto Raggi, Marleen de Mul, Carlota de Miquel, Maria Teresa Pugliese, Christina M van der Feltz-Cornelis, Ana Ortiz-Tallo, Luis Salvador-Carulla, Sue Lukersmith, Leona Hakkaart-van Roijen, Dorota Merecz-Kot, Kaja Staszewska, Carla Sabariego. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 01.06.2022.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.