Review
Abstract
Background: Scientific implementation findings relevant to the implementation of internet-delivered cognitive behavioral therapy (iCBT) for depression and anxiety in adults remain sparse and scattered across different sources of published information. Identifying evidence-based factors that influence the implementation of iCBT is key to successfully using iCBT in real-world clinical settings.
Objective: This systematic review evaluated the following: (1) aspects that research articles postulate as important for the implementation of iCBT and (2) aspects relevant to the day-to-day running of iCBT services. A mixed methods systematic review using a convergent synthesis design was conducted to bring together evidence across this sparse literature consisting of divergent scientific article types to investigate the implementation of iCBT for depression and anxiety in adults.
Methods: We searched the PsycINFO, PsycArticles, MEDLINE, CINAHL Complete, and Embase databases for any published peer-reviewed scientific articles that report on the implementation of iCBT for depression or anxiety disorders in adults. A total of 40 articles spanning the case study, commentary, meta-analysis, mixed methods study, pilot randomized controlled trial, randomized controlled trial, qualitative study, quantitative study, review, and systematic review article types were identified as eligible for this mixed methods review. Data were analyzed qualitatively using the descriptive-interpretive approach.
Results: The first domain highlighted the impact of therapist and patient attitudes when implementing iCBT, the superiority of guided iCBT over unguided iCBT, its noninferiority to equivalent face-to-face treatments, and its utility outside of the original target of mild-to-moderate depression and anxiety. In total, 3 subdomains were identified under the second domain: (1) the management of iCBT in the workplace, detailing the importance of managing the iCBT service, related staff, and their motivations for using it; (2) the practice of iCBT in the workplace, describing the therapeutic aspects of iCBT provision, such as the provision of support, the background of supporters, and screening procedures; and (3) contextual considerations, detailing the impact of governmental legislation on therapy conducted over the internet, the lack of an iCBT workforce as a limiting factor, and the cost estimates associated with iCBT provision.
Conclusions: Broadly, the findings describe several aspects that should be taken into account when researchers or practitioners implement iCBT as part of their work. However, the findings should be interpreted with caution, as the articles reviewed spanned many article types, and few of the included studies were directly focused on evaluating the implementation of iCBT. While findings provide insight into important factors to consider during iCBT implementation, these findings and their limitations highlight the need for more implementation-specific research in this area.
doi:10.2196/47927
Keywords
Introduction
Background
Internet-delivered cognitive behavioral therapy (iCBT) for depression and anxiety has been developed to help increase access to evidence-based therapies. There is empirical support for its use in treating depression and anxiety [
- ]. End users experience it positively [ ] and find it to be satisfactory and acceptable [ , , ]. However, disseminating iCBT at scale remains a challenge [ , ], and COVID-19 has brought its relevance to light now more so than ever [ - ]. A 2019 commentary [ ] discussed the evidence-to-practice gap in digital mental health treatments. The authors postulated that the reason for this gap is a lack of knowledge in the field of iCBT around implementing interventions within routine care. They suggested the adoption of implementation science methodologies to bridge this evidence-to-practice gap.Implementation science has been defined as the scientific study of methods to promote the systematic uptake of research findings and other evidence-based practices into routine practice to improve the quality and effectiveness of health services and care [
]. Central to this definition is the problem statement behind it: it takes almost 17 years for health care research to achieve its intended benefit, which is termed as the “evidence-to-practice gap” [ , ]. As a newly emerging academic field, implementation science is largely integrative; it borrows and adapts theories from multiple fields and uses these to understand the determinant mechanisms as to why (or why not) a specific implementation succeeds [ ]. Implementation science theories provide a framework that allows for implementation plans to be developed and relevant outcomes to be measured [ ], and it has been posited that using these methodologies in future studies of iCBT could generate learnings relevant to its real-world application [ , , ].In a recent review focusing on determinants of implementation for eHealth interventions [
], 37 determinants associated with successful implementation were identified. However, it should be noted that “eHealth interventions” in this case contained a wide variety of digitally enabled interventions, including iCBT and psychotherapy delivered via videoconferencing. When comparing iCBT and other eHealth interventions, “complexity” is a factor for consideration, that is, the degree to which an intervention contains multiple components that require interaction from many individuals, from various levels within an organization, to enact the intervention effectively [ ]. iCBT’s level of complexity is highlighted in service illustration articles by Titov et al [ , ]; for example, therapists’ skill set to operate iCBT efficiently (technical knowledge and constructing written messages), revised services delivery pathways, adherence to regulatory frameworks, and newly aligned clinical governance procedures are some elements of how delivering iCBT may differ from more traditional or less complex services. Conversely, although administering psychological therapy through videoconferencing software may require some altering of specific therapeutic skills and technical upskilling [ ], relative complexity across other areas may be lower (eg, referral pathways and wider system integration). Similarly, some authors have illustrated the need for both iCBT [ , ] and telehealth-specific competency frameworks [ ], further illustrating the need for specialized skills to extend the traditional therapist skill set.Attempts to mobilize implementation science information on eHealth interventions, generally to the point of having pragmatic, clinical relevance for iCBT, have been sparse [
]. As a consequence, the availability of implementation findings relevant to iCBT remains low. This study is a mixed methods systematic review that aimed to account for literature that specifically references or can inform factors relevant to the implementation of iCBT, specifically for depression and anxiety treatment in adults. The benefit of a mixed methods systematic review over traditional systematic literature reviewing is that it seeks to extract relevant information across qualitative, quantitative, review, and illustration-based articles. Mixed methods synthesis affords a way to effectively capture this information and synthesize it qualitatively to produce relevant insights into the implementation of iCBT. Therefore, a convergent integrated approach to the mixed methods review was chosen due to its appropriateness over other review methods for the subject; a traditional systematic review on the implementation of iCBT for depression and anxiety would not be appropriate due to insufficient qualitative or quantitative findings to generate insights [ ]. Furthermore, there are no restrictions imposed on the type of evidence included within the synthesis, which aligns with the anticipated variety of articles that would be identified [ ]. The disorder domains of depression and anxiety were chosen due to them being the most substantive areas of research for iCBT.A mixed methods systematic review departs from and complements previous work in the ways mentioned subsequently [
]. First, it will specifically focus on iCBT-based interventions, which can be considered relatively “complex” in their administration [ ]. Second, it will provide a rich description of the current “practice behind the science” by focusing on reportage within the methods, results, and discussion sections of the articles. Third, it will contribute to the existing literature regarding specific implementation strategies that are associated with the use of iCBT [ ]. Finally, it will allow the interpretation of research findings in a way that is hoped to be productive for future implementations specific to iCBT for the treatment of depression and anxiety.Review Objectives
The overarching objective of the review was centered on the pragmatic question of “What can we learn from published peer-reviewed literature about the implementation of iCBT for depression and anxiety?” This objective was further broken down into 2 domains of focus, on which data extraction and subsequent data analysis were based (refer to the Methods section).
The first domain was centered on implementation insights derived from iCBT research. This objective and domain center on understanding the novel information that is often presented in published research and how this information can have relevance and be mobilized for the benefit of iCBT implementation.
The second domain was implementation process considerations for the successful implementation of iCBT, which consisted of establishing the strategies that are used within the literature to facilitate the implementation of iCBT. According to implementation science literature, implementation strategies are methods used to facilitate the implementation of an intervention, where strategies can consist of training packages, management approaches, developing protocols for intervention use, etc [
, ].Methods
Design
Because implementation science information on eHealth interventions that has pragmatic, clinical relevance for iCBT has been sparse and scattered across different article types, a mixed methods systematic review was conducted. This mixed methods systematic review used a convergent integrated approach and was conducted to identify literature that was central to the review objective mentioned earlier [
, , ]. The convergent integrated approach to conducting a mixed methods systematic review consists of “qualitizing” numerical or statistical findings; that is, quantitative findings are extracted and allocated textual descriptions to allow for integration and simultaneous synthesis with other qualitative data. The resulting qualitative data were then analyzed using the descriptive-interpretive approach [ ]. This review was not registered, and a review protocol was not prepared.Search Strategy
The general search strategy used was as follows: (“ICBT” OR “CCBT” OR “internet-delivered CBT” OR “internet-delivered cognitive behavioural therapy” OR “internet-delivered cognitive behaviour therapy” OR “internet-based cognitive behaviour therapy” OR “internet-based cognitive behavioural therapy” OR “internet-administered cognitive behaviour therapy” OR “internet-administered cognitive behavioural therapy” OR “internet-supported cognitive behaviour therapy” OR “internet-supported cognitive behavioural therapy”) AND (“Anx*” OR “depress*” OR “low mood” OR “GAD” OR “phobia” OR “SAD”) AND “Implement*.” Databases searched included PsycINFO, PsycArticles, MEDLINE, CINAHL Complete, and Embase. A full description of terms and derivatives for the different databases is included in
. Search engine limitations required that the search date had to start from 2007. The search was initially conducted in June 2020 (January 1, 2007-June 1, 2020) and further repeated in September 2021 (June 1, 2020-August 31, 2021) to identify any new or relevant publications. The inclusion and exclusion criteria are given in .Inclusion criteria
- Study provides reports on outcomes related broadly to the 2 domains of interest:
- Implementation insights derived from internet-delivered cognitive behavioral therapy (iCBT) research
- Implementation process considerations for the successful implementation of iCBT
- The study types included in the review:
- Empirical research, encompassing pre-post experimental (eg, feasibility or randomized controlled trial), case study, observational, or qualitative designs in naturalistic, nonefficacy settings
- Review-type studies, including systematic, meta, umbrella, narrative, and scoping reviews
- Service-illustration articles that report on the effectiveness of iCBT clinics over time or describe their operating model
- Studies targeting adult patient populations, mental health care workers (eg, clinicians, therapists, and service managers), or prospective users of iCBT
- The study must be conducted in reference to iCBT (eg, patients undertaking iCBT, clinicians or therapists, or patients reporting on their views of iCBT).
- The study must be primarily conducted in reference to depression and anxiety disorders (eg, patients undertaking iCBT for depression and anxiety, clinicians and therapists, or patients reporting on their views of iCBT for depression and anxiety)
Exclusion criteria
- Nonpeer reviewed research
- Research not in the English language
- Protocols
- Dissertations (due to the difficulty in identifying and accessing these at a wide scale)
- Book chapters
- Conference presentations and abstracts
- Research with participants aged <18 years
- Studies reporting only on clinical effectiveness data with no information on the implementation of iCBT reported on in the study.
Screening
The screening consisted of two steps: (1) review the title and abstract and (2) review the full-paper. We chose to first review the titles and abstracts of all identified records due to the nature of this review and the wide range of study types that were anticipated to result from the search. For example, it was noted throughout the reviewing process that articles frequently cited the terms “implementation” or “feasibility” in the title but failed to provide any relevant information under these constructs when abstracts were reviewed. Where articles provided inadequate information in their abstracts (eg, “the results inform the feasibility of implementing iCBT within XYZ context”) to apply the inclusion criteria, DR acted as a second reviewer for these abstracts and consulted with DD (primary reviewer) to make a decision on inclusion or exclusion. Once step 1 was completed, full texts of all articles were reviewed by DD at full text to discern their relevance to the domains of interest. During this review, articles were rejected during full text screening for the following two reasons: (1) incorrect record specification from the databases (eg, conference presentations being mislabeled) and (2) provided little (eg, minor comments relating to the implementation of iCBT within “future research” sections) or no information on implementation. Once all articles were screened and the final dataset established, data extraction commenced.
Data Analysis
Meaning Unit Extraction
Meaning unit extraction began by identifying qualitative meaning units within the methodology, results, and discussion sections of articles relevant to the study objective, that is, learning about the implementation of iCBT for depression and anxiety. Meaning units are discrete data chunks (either paragraphs or sentences) that contribute stand-alone meaning toward a particular research question or objective [
]. Throughout the mixed methods systematic review process, relevant quantitative findings were translated (or “qualitized”) to qualitative meaning units. The resulting qualitative meaning units were identified and subsequently extracted to an Excel (Microsoft Corporation) file for analysis and assigned relevant identifiers. In addition to assigning identifiers, each meaning unit was also assigned brief, textual summary labels that provided a way to quickly identify the information being conveyed by longer meaning units.To address the review objective of learning about the implementation of iCBT for depression and anxiety, 2 main focus areas were identified that allowed for the structuring of the relevant data to guide meaning unit extraction.
The first focus area was the implementation process considerations for the successful implementation of iCBT. It consisted of the following questions: (1) What strategies do articles report on that are related to the process of implementing iCBT (eg, training clinicians or therapists, screening procedures, referral pathways, and service operations)? (2) Do articles report on the impact of these strategies on specific stakeholder groups (eg, patients and clinicians or therapists)? and (3) Do articles acknowledge or cite factors within the context of the implementation (eg, governmental policy, service infrastructure, and funding)?
The second focus area was the implementation insights derived from iCBT research. It involved the following questions: (1) What implications do authors of the included studies cite as important for the future of the implementation of iCBT? and (2) How do authors interpret their findings in discussion sections of articles, and can these interpretations have implications for how iCBT is implemented?
Category Generation
Meaning units within these 2 focus areas were then further analyzed for similarities and clustered together into categories and subcategories grouping meaning units of similar meaning. The categories and subcategories were named to capture the meaning of the meaning units they contained. The process of organizing and naming categories and subcategories was an ongoing activity led by DD and involving feedback from DR and another colleague. The consensus was agreed upon through group discussion, where the core component of each subcategory was established (eg, this subcategory describes x), and the revised category names were generated based on this shared understanding. As per the descriptive-interpretive approach, it was important that progress was continuously audited through group meetings with DD, DR, JP, and LT.
Results
Overview
A total of 40 eligible articles published between 2010 and 2021 were included in the mixed methods synthesis.
provides the types of articles included in this mixed methods systematic review and brief summaries of these included articles. Article types included in this review were case study (1/40, 2%), commentary (4/40, 10%), meta-analysis (5/40, 12%), mixed methods study (3/40, 8%), pilot randomized controlled trial (1/40, 2%), randomized controlled trial (2/40, 5%), qualitative study (5/40, 12%), quantitative study (13/40, 32%), review (3/40, 8%), and systematic review (3/40, 8%). [ , , , , , - ] includes a numbered list of all articles analyzed as part of this mixed methods systematic review. The PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow diagram constructed to illustrate the search findings is illustrated in and the PRISMA checklist is provided in .
Domain and Category Structure
Two domains (and lower subdomains, categories, and subcategories) were identified: (1) implementation insights derived from iCBT research and (2) implementation processes related to the successful implementation of iCBT in care settings (
and ). Subcategories are discussed as talking points within each category as opposed to adding further organizational levels within this study.Category and subcategory | Articles, n (%) | Study reference number | |
Clinician attitudes toward iCBT | |||
Negative attitudes toward iCBT can impact referral rates and patient outcome | 10 (25) | [ | , , , , - , , , ]|
Positive attitudes toward iCBT can increase acceptability and help to grow iCBT in service | 3 (8) | [ | , , ]|
Patient attitudes toward iCBT | |||
Positive attitudes toward iCBT content, support, privacy, and convenience of iCBT can foster engagement | 10 (25) | [ | , , , , , , , - ]|
Attitudes as moderators of clinical outcome, perceived helpfulness, and adherence | 3 (8) | [ | , , ]|
Negative attitudes relate to preference for face-to-face therapy and issues with the utility of iCBT to patient needs | 5 (12) | [ | , , , , ]|
The delivery of internet-delivered therapies can be helped by technological and clinical augmentation | 7 (18) | [ | , , - , , ]|
Specific patient characteristics need to be considered when implementing iCBT | |||
Age is negatively associated with adherence and clinical outcomes in guided iCBT and not associated with symptom deterioration in unguided iCBT | 4 (10) | [ | , , , ]|
The relationship between gender or sex and adherence is unclear in iCBT overall, but gender or sex is not associated with symptom deterioration in unguided iCBT | 2 (5) | [ | , ]|
Patient technological literacy is tentatively positively associated with adherence and clinical outcomes in iCBT | 2 (5) | [ | , ]|
Medication and alcohol use are not associated with iCBT adherence | 1 (2) | [ | ]|
Racial or ethnic minority group membership is negatively associated with adherence to iCBT | 1 (2) | [ | ]|
The relationship between adherence and marital status, employment status, and education level is mixed overall, but they are not associated with symptom deterioration in unguided iCBT | 3 (8) | [ | , , ]|
Having a lower income is positively associated with dropout | 1 (2) | [ | ]|
Comorbidity of disorders can moderate treatment outcome | 1 (2) | [ | ]|
Making sudden clinical gains is associated with greater improvements after the treatment | 1 (2) | [ | ]|
The severity of depression can positively impact clinical outcomes and adherence | 2 (5) | [ | , ]|
Symptoms of depression can negatively impact iCBT adherence | 2 (5) | [ | , ]|
Chronic mental health problems are negatively associated with iCBT adherence | 1 (2) | [ | ]|
Guided iCBT is superior to unguided iCBT in regard to symptom outcomes and adherence | 11 (28) | [ | , , , - , , , , , ]|
iCBT is as effective as face-to-face delivery of the same protocol, yet preference is often for face-to-face treatment | 13 (32) | [ | , , , , , - , , , , ]|
iCBT appears to be effective beyond the original target of mild-to-moderate depression and anxiety | 10 (25) | [ | , , , , , , , , , ]|
Conducting future research that has relevance for iCBT implementation is important | |||
More implementation research is needed to understand the uptake of iCBT within routine care | 9 (22) | [ | , , - , , , , ]|
More research is needed on adverse events to understand the negative effects of iCBT | 2 (5) | [ | , ]|
More research is needed to understand the relationship between adherence and iCBT | 4 (10) | [ | , , , ]
Subdomain and category and subcategory | Articles, n (%) | Studies | ||||
Management of iCBT in the workplace | ||||||
Successful training of supporters is important for the provision of iCBT | 8 (20) | [ | , , , , , , , ]||||
Training stakeholders within the health system is important in creating awareness of iCBT | 2 (5) | [ | , ]||||
Effective management of risk and adverse event management in iCBT is important for its delivery | 10 (25) | [ | , , , , , , , , , ]||||
iCBT should be delivered through secure and interoperable systems that facilitate clinician and client access | 11 (28) | [ | , , , , , , , , - ]||||
Operational considerations for managing iCBT and related staff are important | ||||||
Effective management and leadership support facilitate the implementation | 7 (18) | [ | , , , , , , ]||||
Management of workplace resources is necessary to facilitate staff time to use iCBT | 4 (10) | [ | , , , ]||||
Staff motivation to use iCBT needs to be fostered | 4 (10) | [ | , , , ]||||
Use of routine monitoring of iCBT to convey intervention effectiveness and enhance its delivery | 5 (12) | [ | , , , , ]||||
Effective marketing and service promotion enhance the uptake of iCBT | 6 (15) | [ | , , , , , ]||||
Staff recruitment and retention in iCBT is a challenge that needs to be mitigated against | 2 (5) | [ | , ]||||
Scaling of iCBT within services is challenging and requires multiple considerations (eg, infrastructure, funding, proper testing, and governance) | 7 (18) | [ | , , , , , , ]||||
The practice of iCBT in the workplace | ||||||
Appropriate referral pathways and management of waiting times are important for the delivery of iCBT | 19 (48) | [ | , , , , , , - , , - , , - ]||||
Screening and inclusion criteria for accessing iCBT need to be thoroughly defined | 21 (52) | [ | , , - , , , , , , , , , , , , - ]||||
Considerations of the level of support for patients are crucial in the provision of iCBT | ||||||
Positive impact of support on patients | 11 (28) | [ | , , , , , - , , , ]||||
The quality of support impacts the success of iCBT provision | 13 (32) | [ | , , , - , , , , , , , ]||||
Appropriate considerations should be given to the mediums and modalities of support to fit service and user needs | 20 (50) | [ | , , - , , - , , - , , ]||||
The time demand associated with the provision of support needs to fit service and user needs | 18 (45) | [ | , , , , , - , , , , - , , ]||||
The optimal personal and professional background of the supporter needs to be considered in the provision of iCBT | 17 (43) | [ | , , , - , , - , - , ]||||
Contextual considerations | ||||||
Governmental and health care regulations affect the implementation of iCBT | 10 (25) | [ | , - , , , , , ]||||
Lack of workforce availability for iCBT as a limiting factor in the provision of iCBT | 4 (10) | [ | , , , ]||||
Considering the cost estimates associated with iCBT for patients and health care providers before implementing | 11 (28) | [ | , , - , , , , , , ]
Domain 1: Implementation Insights Derived From iCBT Research
This domain includes categories that contribute to the success of iCBT in either research or routine practice settings or further learning to inform it.
Category 1: Clinician Attitudes Toward iCBT Affect Patient Outcomes and Implementation of iCBT
Clinician attitudes toward iCBT are mixed [
]. Negative attitudes hinder the successful dissemination of iCBT to clients [ ]. These attitudes consist of skepticism about the effectiveness and quality of iCBT [ , ], technological limitations of iCBT [ ], the inability to generate a therapeutic alliance through this medium [ ], preference for face-to-face therapy [ , , ], the perceived lower priority of the intervention in the workplace [ ], and its highly standardized nature being incompatible with other psychological interventions [ , ]. Such negative attitudes can arise from a lack of iCBT exposure or training [ ] and can be transferred to patients and undermine patient outcomes [ ]. Accordingly, there is a need to engage with these negative attitudes to successfully implement iCBT [ , ].In contrast, positive attitudes toward iCBT can increase acceptability and help grow iCBT in service. Positive attitudes include acknowledging the benefits of iCBT in terms of time efficiency, cost-effectiveness, evidence base, program design quality, accessibility, and ability to bridge treatment gaps for those on waiting lists [
]. Professionals with more experience implementing iCBT regard iCBT more positively in terms of its applicability to service provision [ ]. Even health care professionals with little exposure to iCBT are generally positive and accepting toward iCBT but have biases around suitability and large knowledge gaps [ ].Category 2: Patient Attitudes Toward iCBT Affect Engagement, Adherence, and Outcomes
The first subcategory under this larger category relates to patients reporting positive attitudes toward the iCBT treatment, its content, and the associated therapist support they receive [
, , , , , ]. They also report strong motivations to seek out iCBT [ ] and acknowledge advantages in terms of convenience, cost, privacy, and self-directed nature [ , - ]. A second subcategory relates to attitudes, where greater positive initial attitudes predict greater symptom reduction and adherence, and improvement or decline in attitudes during treatment predicts better or worse adherence and outcomes [ , , ]. The third subcategory relates to negative attitudes, which can be a barrier to treatment success [ ]. Such negative patient attitudes include skepticism toward the effectiveness and credibility of iCBT [ , ] as well as toward motivation and accountability to progress through iCBT [ ]. There is also a reported preference for face-to-face therapies over iCBT [ , - ]. Interestingly, offering iCBT as a waiting list treatment can create “unfavorable comparisons” between iCBT and face-to-face therapy, resulting in negative perceptions of iCBT [ ].Category 3: The Delivery of iCBT can be Augmented by Technological and Clinical Design Factors
iCBT has been augmented with novel design elements or treatment strategies to understand their utility and benefit. Such elements and strategies include integrating sensors [
], gamification [ ], transdiagnostic elements [ , , ], iCBT as an add-on or adjunct to existing care pathways [ , , ], incorporating “persuasive technology” components [ ], iCBT use in a blended care model [ , , ], and its use as a first-line intervention to promote interest in further mental health care [ ].Category 4: Patient Characteristics Related to iCBT Outcomes
A variety of demographic, medical, and technological factors have been implicated to affect iCBT and are reported in the subsequent subcategories. There is mixed evidence for age influencing iCBT outcomes wherein null or negative associations have been observed between age and adherence [
, ] and symptom reduction [ , ]. Mixed results are also reported for gender and sex. Females can have greater or similar iCBT adherence to males [ ], and sex was found to be associated with symptom deterioration in unguided iCBT [ ]. Perceived technological literacy has been posited to promote adherence [ , ] and clinical outcomes [ ], though the evidence is unclear regarding the impact of tech literacy on iCBT outcomes [ ]. A review reported that patient-reported medication and alcohol use were not associated with iCBT adherence [ ]. Racial and ethnic minority group membership has been negatively associated with adherence to iCBT [ ]. Both positive and negative associations have been observed for marital status, employment status [ , , ], and education level [ ] with adherence. Individuals with both lower income levels and marital status of single were more likely to drop out from iCBT [ ]. Comorbidity of mental health disorders can reduce the effect of iCBT treatment [ ]. Chronic mental health problems are negatively associated with iCBT adherence [ ].Making sudden, large clinical gains on symptom measures at the start of treatment is associated with greater improvements after the treatment [
]. The severity of depression can positively impact clinical outcomes of depression; for example, higher pretreatment severity results in greater effect sizes [ , ]. Symptoms of depression have also been found to negatively impact iCBT adherence, where symptoms like low motivation have been found to be negatively associated with iCBT adherence [ , ]. Chronic mental health problems are further associated with iCBT adherence, where years of living with chronic mental health conditions was found to be negatively associated with adherence to iCBT [ ].Category 5: Guided iCBT as Superior to Unguided iCBT in Regards to Symptom Outcomes and Adherence
Guided iCBT shows superiority over unguided iCBT in terms of adherence and clinical outcomes [
, , - , , , ]. However, an individual participant data meta-analysis [ ] postulates that the small effects achieved by unguided iCBT are superior to control groups (or no intervention) and can be best used when implemented at scale, such as at the public health level. The therapist element of guided iCBT, in particular, is posited to improve adherence to iCBT [ , ]. Guided iCBT support also fulfills an expressed need to navigate through and explain therapeutic content when patients encounter difficulties [ ].Category 6: iCBT Is as Effective as Face-to-Face Delivery of the Same Protocol, Yet Preference is Often for Face-to-Face Treatment
iCBT produces similar adherence [
, ] and clinical outcomes [ , , , ] to equivalent face-to-face therapy. iCBT has advantages over face-to-face therapy in terms of time efficiency, access rates [ , ], and its ability to deliver standardized treatment [ , ]. However, patients demonstrate a preference for face-to-face treatment over iCBT, which can be a reason for the dropout of iCBT [ , , , , ]. In such instances, these preferences for iCBT over face-to-face treatment can be reduced by introducing a time delay when accessing treatment [ ].Category 7: iCBT Appears to be Effective Beyond the Original Target of Mild-to-Moderate Depression and Anxiety
iCBT is not typically offered for severe presentations of depression and anxiety [
], but real-world data illustrate that a large proportion of patients seen by iCBT clinics have symptoms in this range [ ]. Patients with severe symptoms at baseline can make large clinical gains [ , , , ], show comparable adherence rates to patients with less severe symptoms [ ], and, in some cases, produce larger gains than patients with more severe symptoms [ ]. Studies requiring greater treatment-seeking behaviors tend to recruit individuals with more severe symptoms of depression, illustrating the willingness and motivation of this cohort to initiate treatment [ , ]. After treatment initiation, the effect of higher pretreatment severity on adherence and completion is unclear, with a study positing that higher pretreatment severity may be associated with lower iCBT completion rates [ ]. Similarly, patients whose symptoms are in the subclinical range also benefit from iCBT [ , ]. Those with suicidal ideation are also found to benefit from iCBT [ , , ]. iCBT may also be applicable to conditions where depression is secondary to the presenting problem (eg, addiction, trauma, schizophrenia, and bipolar disorder [ , ]).Category 8: Future Research on Implementation Critical to Advancing iCBT
Several key areas critical to advancing the implementation of iCBT were apparent. First, a plethora of articles stated a lack of research that details the process of implementing iCBT in naturalistic settings; therefore, more research is needed to understand and improve iCBT uptake [
, , , - , , , ]. Second, more research is needed regarding adverse events within iCBT and that current reporting of the adverse events in studies is poor [ , ]. Third, more research is needed to understand the relationship between adherence and iCBT outcomes [ , , ]. It was also suggested that the definition of “dropout” should not be conceptualized in a binary way because varying dosages of iCBT have been found to produce positive clinical change when less than the intended program is completed. A further study stated that high rates of dropout observed in iCBT research should similarly be expected for practical implementations [ ].Domain 2: Implementation Processes Related to the Successful Implementation of iCBT
This domain includes categories of factors that are important for the successful implementation of iCBT.
Subdomain 1: Management of iCBT Day-to-Day Workplace Operations
This subdomain consists of factors important for managing the day-to-day operations of iCBT, with categories pertaining to the training of staff, risk management, marketing and service promotion, IT infrastructure, working with other services, and managing the staff who work in the provision of iCBT.
Category 1: Successful Training of Supporters Is Important for the Provision of iCBT
Effective training of supporters in iCBT requires technical training in the use of the program [
, ], developing competencies around computer skills and web-based written communication [ , , ], and practice providing support to fictional patients [ ]. Training on writing skills (as many iCBTs rely on supporters messaging the patients on the iCBT platform) should occur before patient interactions rather than through trial and error during their interactions with patients [ ]. Training supports, including a manual [ , ], giving clinicians access to training resources [ , ], and providing them with feedback on their written reviews [ ], were considered helpful. A study stated that there are limited opportunities for support training in iCBT [ ].Category 2: Educating Stakeholders Within the Health System Is Important to Create Awareness of iCBT
Training and educating other relevant stakeholders (eg, nonclinical staff, referral providers, and patients) about the benefits of iCBT is important to create awareness of the intervention and its clinical effectiveness [
] increasing the perceived viability of the intervention as a treatment option [ ].Category 3: Effective Management of Risk and Adverse Events in iCBT Is Important for Its Delivery
Successfully implemented iCBT has to be supported by a risk monitoring system (eg, suicidal risk measures) that alerts clinicians to risk (eg, triggered automated messages within the iCBT software) and allows clinicians to act on indicated risk (eg, clinician contacts identified risk cases; [
, , , , , , , ]).Category 4: iCBT Requires Secure, Interoperable Systems That Facilitate Clinician and Client Access
Reviewed articles suggested that iCBT interventions should be hosted on secure servers [
, , ], should be optimized to run on a variety of mediums (tablets, desktops, and phones) [ ], should be integrated with larger patient databases [ , , ], and should have security standards that adhere to relevant governing bodies [ , ]. Internet connection difficulty [ , ], enabling service computers to access iCBT and its related websites, a lack of integration of iCBT apps with health care records [ ], and providing patients access to technology to use iCBT have been cited as limiting factors and may contribute to patient dropout [ , ].Category 5: Operational Considerations for Managing iCBT and Related Staff Are Important for the Successful Implementation
This category details operational factors that impact the successful implementation of iCBT within clinics or workplaces. First, reviewed articles suggest that management and leadership are important to implement iCBT [
] and include activities such as developing guidelines and service procedures [ , , , ], change management [ ], and planning implementation and engaging stakeholders [ , ]. In addition, management of workplace resources is necessary to facilitate staff time to use iCBT. For instance, clinicians with already high workloads may experience time shortages for administering or reviewing the program [ , ]. iCBT clinic managers worry about clinicians balancing iCBT and face-to-face work workloads [ ], and it may be necessary to have a dedicated workforce to support iCBT delivery [ ].Second, staff motivation to use iCBT needs to be fostered, as motivation to use iCBT is “essential” [
] because iCBT proponents facilitate the implementation of iCBT [ ]. However, fostering this motivation and changing the way clinicians practice is difficult [ , , ]. One way to foster this motivation may be through routine monitoring of iCBT outcomes, which can provide persuasive evidence of intervention effectiveness to stakeholders as well as enhance its delivery. Services in Australia and Canada reported that they regularly conducted audits of service effectiveness [ , , ], and such routine monitoring data were used so that staff could evaluate and understand the effectiveness of iCBT in their service [ , ] and to ensure compliance with treatment manual [ ].Third, effective marketing and service promotion are essential to spreading iCBT initiatives [
, ]. Advertisement campaigns (eg, web-based and print media) are frequently successful in recruiting participants for trials and routine care [ , , , ], though such marketing campaigns can require a large quantity of resources [ ]. Recruitment and retention of therapists in iCBT-related positions is also critical. Clinician recruitment can be an issue as some therapists believe that iCBT limits professional freedom due to its highly structured working requirements and that the working conditions are not attractive [ , ].Fourth, scaling of iCBT within services is challenging and requires consideration of multiple factors. The physical infrastructure (eg, internet) must be in place [
], funding needs to be procured [ ], service decision makers must be convinced of the intervention’s feasibility [ ], there needs to be evaluation frameworks for existing and new iCBT programs [ , ], and governance frameworks (eg, clinical, IT, and organizational) must be implemented that adhere to the wider legislative context [ ]. Exploring new service pathways that are developed when considering iCBT services may allow for existing iCBT services to scale their offering [ ]. A study stated that iCBT services should start with a small offering (eg, minimally monitored iCBT) and then acquire human and financial resources over time to build out the service [ ].Subdomain 2: The Practice of iCBT in the Workplace
Category 1: Appropriate Referral Pathways Are Necessary for the Delivery of iCBT
Reviewed articles suggested that successful implementation of iCBT requires the development of appropriate referral pathways, though there are many different referral strategies. Such pathways include self-referral [
, , , , , ], health care provider referral [ , , , , , , , , , , , , ], access to pathways through marketing materials [ , ], contacting patients by email [ ], contacting patients on waiting lists for face-to-face services [ ], or clients applying through a secure website [ ].Category 2: Screening and Inclusion and Exclusion Criteria for iCBT Are Wide Ranging
Successfully implemented programs required patients to complete a web-based [
, , , , , , , , ], in-person [ , , ], or phone screening assessment [ , , ]. As part of the screening, patients were asked about demographic information, mental health symptoms, commitment to iCBT, treatment history, level of risk, internet access, and language proficiency [ , , , , , , , , - ].A variety of inclusion and exclusion criteria for accessing iCBT are reported as follows: no severe depression [
, , , ], no severe anxiety [ ], no chronic or recurrent depression [ ], no dementia [ ], no past history of psychotic symptoms [ , ], aged >18 years [ , , , , , , ], aged between 18 and 65 years [ ], diagnosis of disorder via psychiatric interview or exceeding cutoff on established measure [ , , - , , , ], no comorbid substance abuse [ , , , , , ] or use of benzodiazepines [ ], no suicide risk [ , , , , , , - ], no bipolar disorder, psychosis, or obsessive compulsive disorder [ , , , , , , ], adequate understanding of program language [ , , , , , ], no developmental disorders or other cognitive disabilities [ ], no comorbidities or nonpsychiatric diseases that could cause depressive symptoms [ ], no concurrent treatment [ , , , , ], no change in medication before 1 month of commencing treatment [ ], no email address or technological means to access treatment [ , , , ], patients with low motivation [ , ], and patients to far outside of the geographic location of the clinic [ , , , ].Category 3: Use of, and Processes for, Providing Support That Are Crucial to the Provision of iCBT
Supported iCBT achieves positive clinical outcomes for patients [
, , - ], provides superior clinical and adherence outcomes over unguided interventions [ , , , , ], and is regarded positively by the patients [ , ]. Therapeutic alliance is implicated as a mechanism behind the positive effects of supported interventions [ ]; however, its effects are still unclear in iCBT, as it has been associated with positive outcomes [ ] or no effect [ ].The purpose of support in iCBT is to “recognize and reinforce the participants’ work with the self-help material” [
] and promote engagement with the intervention [ , ]. The supporter in iCBT is posited to assume the role of a motivator, where the iCBT platform delivers the core treatment elements [ , ], and involves therapists monitoring patient progress [ , , , ], responding to their iCBT-related needs [ , , ], and guiding the user through the initial setup [ , ]. Through written support, clinicians can encourage and affirm patients by expressing interest in the thoughts, feelings, and behaviors that patients share [ , ]. The quality of support affects client outcomes. For instance, leniency toward patient accountability (eg, homework completion) is associated with poorer patient outcomes [ ]. Moreover, within written messages, the studies found that misspellings were frequent, emojis and emoticons were seldom used, and less detailed, shorter messages were associated with fewer web-based sessions completed [ ].iCBT support has been delivered in many ways [
]: in person [ , ], over email [ , , ], by telephone [ , , , , , ], through the iCBT platform [ , , , , , ], through videoconferencing software [ ], or by SMS text message [ ]. Support can occur in real-time [ ], on an “on-demand” basis [ ], or asynchronously [ , , , ]. Support can be delivered weekly [ , , - , , - ] or constantly through ongoing therapist monitoring [ ]. Some programs incorporate homework assignments to inform clinicians when conducting support sessions [ , ]. In total, 3 programs implemented “step-wise” access to modules, where new content could not be accessed without completing a supported session [ ], was unlocked 7 days after the completion of the previous module [ ], or released gradually over an 8-week period [ ].Such decisions around how support is delivered and its frequency likely influence clinician time demand. Time spent delivering support varies widely [
], ranging from 10 to 100 minutes per session [ , , , , - , , , , , ] and up to 8 hours per individual per course of treatment [ ]. Support can be delivered weekly [ , , - , - ] or constantly through ongoing therapist monitoring [ ]. Some programs involve 6 to 12 supporter telephone calls [ ], emails from the supporter [ ], or that the supporter contacts the patient ≥1 time a week for 8 weeks [ ]. The end of treatment for some programs was based on a specific time or the number of support sessions received; for example, iCBT was cited across articles to be delivered over a varying course of 7 to 20 weeks [ , , , , , , , ].Category 4: iCBT Can Be Successfully Implemented Across Supporters With a Variety of Personal and Professional Backgrounds
Successful implementation of iCBT included supporters who were volunteer peer-supporters with lived experience of the mental health condition [
], trained volunteers [ ], psychologically trained experts (unspecified qualifications; [ ]), clinical psychologists [ , , , , - , ], psychiatrists [ ], registered or provisionally registered mental health professionals [ , ], graduate students of psychology [ , ], trained health care professionals [ ], psychologists in training [ , ], psychotherapists [ ], social workers [ , ], mental health nurses [ ], nurses [ ], therapists with training on addictions [ ], trained technicians [ , , ], and general practitioners [ ]. There is evidence that untrained technicians [ , ] or novice clinicians [ ] achieve equal outcomes compared to trained clinicians, and that support from a technician is more effective than a waitlist control group [ ].Subdomain 3: Contextual Considerations for iCBT Implementation
Category 1: Governmental and Health Care Regulations Affect Implementation of iCBT
Governmental and health care regulations influence how iCBT can be implemented. An example of this is in Canada, where iCBT has been recognized by the Canadian government, which provides specific funding streams for iCBT services and research [
, , ]. Other countries have implemented policies that incentivize the use of iCBT to improve access to psychological therapy [ , ]. Other regulations impacting iCBT include limitations placed on therapeutic contact taking place over the internet [ , ], requiring iCBT clinics to adhere to existing frameworks for the delivery of therapy [ , , ] and policies around the delivery of therapy over the internet [ , ].Category 2: Lack of Workforce Availability for iCBT as a Limiting Factor in the Provision of iCBT
The lack of workforce availability can limit the implementation of iCBT, particularly because as access to mental health care increases, so does the demand for services [
, ]. One study in Sweden observed only 1 to 2 therapists participating in iCBT initiatives among implementing services and further commented that due to face-to-face resources being expensive and scarce, a dedicated iCBT workforce could resolve this issue in terms of resource and cost [ ]. Another study stated that the presence of trained iCBT professionals in certain health sectors (eg, veteran care in the United States) is rare [ ].Category 3: Cost Models and Cost-Effectiveness of iCBT Maybe an Asset for Successful Implementation
iCBT was provided to patients through the following 5 cost models: free of charge [
], through publicly funded health care systems [ , , , , , , ], subsidized by health care providers [ , ], at a cost to patients when they are not within certain catchment areas or countries [ ], or as part of insurance plans [ ]. The establishment of reimbursement systems for iCBT was cited as an important factor for cost estimates in the future [ , ]. Moreover, it was hypothesized that as iCBT cost-effectiveness becomes more salient, health care providers (public or private) will advocate for it as a first-line intervention to efficiently gatekeep therapeutic resources [ ]. iCBT did not incur extra costs to public health care systems [ ] and was cost-effective (depending on the “willingness to pay” standards of the health care body) [ ]. One study posited that a dedicated iCBT workforce should be developed to create a less expensive alternative [ ].Discussion
Principal Findings
Our mixed methods systematic review highlights the knowledge we have gained from the available literature on the implementation of iCBT. Some of our key findings regarding the process for implementing iCBT include the practice of iCBT, with special reference to determining client eligibility and effectively supporting patients in iCBT. The management of iCBT in the workplace, especially staff and operational considerations, also surfaced as an important process to consider when implementing. Other related findings include the importance of staff training, the management of treatment pathways, security, and factors for consideration within the wider context that impact the implementation of iCBT. In terms of implementation insights, this review has highlighted that clinician and patient attitudes toward iCBT can influence its ability to achieve intended outcomes and the need to continually tailor iCBT for patient benefit and that further research can help to develop our understanding of implementing iCBT successfully.
As would be expected, the practice of iCBT was highlighted as important to the implementation process for iCBT within a mental health service. For instance, what constitutes eligibility for an iCBT intervention manifested in two categories: (1) screening and inclusion criteria for accessing iCBT need to be thoroughly defined and (2) consideration of the usefulness of iCBT beyond the original target of mild-to-moderate depression and anxiety. Historically, eligibility for iCBT has been characterized by low symptom presence (mild to moderate) and no significant risk issues. This approach was sensible while establishing iCBT’s safety and effectiveness as an intervention, subsequently resulting in a well-validated evidence base supporting iCBT for treating depression and anxiety. Consequently, the preponderance of historical eligibility seems to be an artifact in need of revision. This is particularly important in light of the growing body of literature to support iCBT’s applicability to more severe presentations of mental health difficulties [
- ]. In addition, real-world data from iCBT clinics highlight that a substantial proportion of patients accessing these services have presentations within the moderate to severe range [ ].There is a clear need for services to consider the populations they serve (eg, general severity levels and client demographics) and tailor their model of iCBT provision. Still, despite the available evidence, clinical guidelines lag in their support for iCBT in extended service delivery pathways [
]. This situation poses some difficulty for certain services or health systems to innovate their use of digital interventions (eg, the improving access to psychological therapies program in England, which offers treatment based on National Institute for Health and Care Excellence guidelines). Specifically in the English context, the original guideline for the use of iCBT was rolled out in 2004 [ , ] and was updated in 2022. Since then, technologies and research have developed and would suggest the utility of iCBT for the broader population.Our results highlight the importance of the operational aspects of iCBT services. First, the importance of effective management and leadership to support the implementation of iCBT was identified through the review. Transformational leadership approaches, that is, leadership styles associated with motivating and compelling employees to participate in a shared vision [
], have been found to be associated with increased levels of innovation climate, further defined as an organizational climate that is conducive to the adoption of novel, evidence-based practices [ ]. Implementing iCBT requires leaders to navigate interactions across multiple levels of a service and motivate staff to ensure the vision of iCBT is fulfilled. However, the current studies identified do not illustrate in depth the effects of leadership nor was it their primary or secondary focus. Despite this, it is still important that this finding was communicated through this small pool of studies, and more research is needed to inform this gap in knowledge.Training staff in iCBT and increasing their motivation to use it were both cited as important. As an in-service activity, training clinicians and therapists in the use of evidence-based practice has a substantial literature base [
, ]. However, our findings highlight variance in the training delivered to therapists charged with delivering iCBT, ranging from hours to up to a year of continued education, and the components of the training were also not described at length across articles. The wider literature on training stands in contrast to what we identified; training programs for evidence-based practice tend to produce better outcomes (eg, competency, evidence-based practice use, and positive attitudes) when multicomponent approaches are used (eg, workshop, follow-up, and audit of skills acquired) [ ]. To date, no systematic evaluation of iCBT training programs has been conducted, and it has also been cited by one of the included studies that training programs for these interventions are rare [ ]. Similarly, we identified that staff motivation to use iCBT needs to be fostered. This motivation can also be developed through training initiatives, where implementers can illustrate the benefits that iCBT brings to routine clinical practice (eg, improves patient symptoms and access to care, is usable, and is not time consuming), and this activity may potentially influence motivation around intervention use [ , ].Furthermore, routine monitoring of the intervention and its outcomes was also cited as important for the continued development of iCBT within the service. This activity can allow supporters in iCBT to reflect on their own practice to improve service provision, with an article stating that clinicians who administer iCBT desire comprehensive updates regarding iCBT to understand its impact on wider service outcomes [
]. This activity is reminiscent of the construct “reflexive monitoring” from normalization process theory [ ], where individual and group reflections on processes around a specific evidence-based practice can lead to revisions in practice that are adapted to best suit the needs and structures of the service context. The results regarding the operational aspects of iCBT, despite not being widely reported across the literature, indicate that factors associated with evidence-based practice success in the implementation literature are being considered when iCBT is implemented, which is a promising finding. More widespread reporting of this information could be beneficial to practicing professionals when making choices about using iCBT with their patients.Patients tend to be positive about iCBT and the support they receive, but clinician attitudes generally lean toward the negative. From service illustrations, we can infer that clinicians receive significant exposure to iCBT when it is implemented [
, , ], and its effectiveness is grounded in the literature. Negative attitudes can result in the abandonment of the implementation effort due to a lack of acceptance or misunderstandings around the perceived value of the treatment [ , ]. A better understanding of negative clinician attitudes can be attained if iCBT were to be interpreted as a novel, evidence-based practice. A literature search around clinicians’ attitudes toward evidence-based practice provides some insights, including that clinicians rate “other” sources of information (eg, colleague opinion and previous experience) as more impactful than published evidence on their decisions for treatment [ - ]. Fostering attitudes conducive to the uptake of evidence-based practice has been associated with transformational leadership styles [ ] and systematic training initiatives that highlight how the evidence-based practice is integrated with the wider service system [ ], both of which were evident within the current review. However, where there is a disconnect between clinicians and service management or staff does not understand the relative advantage (from the diffusion of innovations) [ ], iCBT over existing practice can subsequently create barriers to evidence-based practice uptake [ , ]. This disconnect is well documented in implementation science theories, such as readiness for change [ ] and implementation climate [ , ], both of which also emphasize the role of attitudes in evidence-based practice use and implementation.iCBTs vary widely in their delivery (guided and self-guided), support time frames, and those who provide the support, but the take-home insight is that patients receive the interventions well in terms of satisfaction and clinical outcomes achieved. This malleability of iCBT, where it can assume many forms yet achieve the intended results, underlines the scalability of the intervention. A narrative review of factors associated with scaling public health interventions described that, once an intervention has proven its effectiveness in both small- and large-scale trials, management and practice factors, such as having systems for monitoring intervention performance, funding, and interacting with stakeholders within the wider health care system, become important for the scaling process [
]. Despite this, an “implementation gap” remains, where effectiveness reductions and high levels of attrition occur when we transition from efficacy settings to real-world service provision [ ]. This further creates a treatment gap within users assigned iCBT as a treatment option, where implementation factors (eg, abandonment due to encountering bugs and not having a care provider to explore these with) may cause attrition, lowering the promise of scalability purported by these treatments [ ].Limitations
In total, 4 main limitations were identified as part of this mixed methods systematic review. First, we used a targeted search strategy to produce a dataset that the authors acknowledge is incomplete due to a lack of proper use of terminology within the field to reference implementation. Therefore, we acknowledge that this review is not definitive on the implementation of iCBT and only reports on relevant factors within the articles identified. There is already a movement to standardize the reporting of digitally delivered psychological treatments in research studies (eg, use of CONSORT [Consolidated Standards of Reporting Trials] for eHealth) [
], and perhaps this should be succeeded by an attempt to standardize how we report implementation learnings too.The second limitation consisted of the “blind spots” associated with the development of the analytic framework that may have resulted from the background of the researchers. The authors mainly come from a background in psychology, and none would consider themselves to be implementation specialists. Other review types (eg, realist, scoping, or narrative reviews) conducted by different research groups may uncover nuances that were otherwise unidentified by this review.
The third limitation of the study relates to the limitations present in the original papers included in this review. The heterogeneous nature of the articles included prevented a formal quality appraisal from being conducted. For instance, we had considered using a tool, such as the Critical Appraisal Skills Programme checklist, but this tool did not provide for a quality review of the narrative-type articles (eg, Schröder et al [
], Andersson et al [ ], Andersson et al [ ], and others) that were included. Furthermore, there are issues around assessing the quality of qualitative evidence within review-type studies and whether studies should be excluded based on perceived quality, which further compounds the issue [ - ]. Thus, we did not assess the quality of the included articles, and no articles were excluded based on methodological flaws. Relatedly, a few articles had a primary objective of exploring a facet of the implementation of iCBT for depression and anxiety, which is important to note when interpreting the results.The fourth and final limitation of the study is whether an expansion of search terms could have been used. For instance, the term “computerized CBT” was not used in the search terms. However, while the search string did not include this term, we did include an abbreviated term for this, “ccbt,” and it is notable that by using this abbreviated term we identified specific studies that referenced computerized cognitive behavioral therapy (eg, Wells et al [
], Wright et al [ ], Kenicer et al [ ], Wright et al [ ], and Grist and Cavanagh [ ]). Regardless, it is possible that the search string may not have completely captured all possible studies.Conclusions
This mixed methods systematic review has identified several strategies for consideration when attempting to implement iCBT. Broadly, these strategies emphasize the importance of effective leadership, managing staff and operations associated with the practice of iCBT, implementing and developing professionals to provide the supported component of iCBT, accounting for context, and deriving implementation insights from novel research contributions in iCBT for patient benefit. Future research into iCBT in real-world settings should endeavor to supply appropriate supplemental information that details the efforts associated with implementing the intervention within care pathways. In tandem, efforts could be made to standardize practices that can support the transferability of learning and scalability through the use of a standardized lexicon of terms that are appropriately used.
Acknowledgments
The authors would like to acknowledge Angel Enrique, senior manager and digital health scientist at SilverCloud Health, for his feedback on early iterations of this review. This research was supported with joint funding from the Irish Research Council and SilverCloud Health under the employment-based postgraduate program (grant EBPPG/2018/198).
Data Availability
Data sharing is not applicable to this paper as no datasets were generated or analyzed during this study.
Authors' Contributions
DD was responsible for conceptualization, data curation, formal analysis, funding acquisition, investigation, methodology, writing the original draft, and reviewing and editing the manuscript. DR was responsible for supervision, formal analysis, funding acquisition, and reviewing and editing the manuscript. GH was responsible for conceptualization, visualization, writing the original draft, and reviewing and editing the manuscript. LT was responsible for funding, supervision, and reviewing and editing the manuscript.
Conflicts of Interest
DD, DR, and GH are employees of SilverCloud Health (now a subsidiary of American Well Corporation). DR and GH are shareholders of American Well Corporation, a company specializing in the development of telehealth solutions for physical and mental health conditions. This study formed part of the doctoral research conducted by DD, which was funded by the Irish Research Council and SilverCloud Health as part of the employment-based postgraduate research program and was hosted by Trinity College Dublin, Ireland. LT is a professor in counseling psychology at Trinity College Dublin and an external research consultant to SilverCloud Health.
Search terms used within mixed methods systematic review.
DOCX File , 34 KBDescription of included articles within mixed methods systematic review.
DOCX File , 36 KBReferences analyzed as part of this mixed methods systematic review.
DOCX File , 257 KBPRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) checklist.
PDF File (Adobe PDF File), 54 KBReferences
- Andrews G, Basu A, Cuijpers P, Craske MG, McEvoy P, English CL, et al. Computer therapy for the anxiety and depression disorders is effective, acceptable and practical health care: an updated meta-analysis. J Anxiety Disord. Apr 2018;55:70-78. [FREE Full text] [CrossRef] [Medline]
- Olthuis JV, Watt MC, Bailey K, Hayden JA, Stewart SH. Therapist-supported Internet cognitive behavioural therapy for anxiety disorders in adults. Cochrane Database Syst Rev. Mar 12, 2016;3(3):CD011565. [FREE Full text] [CrossRef] [Medline]
- Romijn G, Batelaan N, Kok R, Koning J, van Balkom A, Titov N, et al. Internet-delivered cognitive behavioral therapy for anxiety disorders in open community versus clinical service recruitment: meta-analysis. J Med Internet Res. Apr 17, 2019;21(4):e11706. [FREE Full text] [CrossRef] [Medline]
- Wright JH, Owen JJ, Richards D, Eells TD, Richardson T, Brown GK, et al. Computer-assisted cognitive-behavior therapy for depression: a systematic review and meta-analysis. J Clin Psychiatry. Mar 19, 2019;80(2):11. [FREE Full text] [CrossRef] [Medline]
- Jardine J, Earley C, Richards D, Timulak L, Palacios JE, Duffy D, et al. The experience of guided online therapy: a longitudinal, qualitative analysis of client feedback in a naturalistic RCT. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 2020. Presented at: CHI '20; April 25-30, 2020:1-15; Honolulu, HI. URL: https://dl.acm.org/doi/10.1145/3313831.3376254 [CrossRef]
- Cavanagh K, Shapiro DA, Van Den Berg S, Swain S, Barkham M, Proudfoot J. The acceptability of computer-aided cognitive behavioural therapy: a pragmatic study. Cogn Behav Ther. 2009;38(4):235-246. [CrossRef] [Medline]
- Richards D, Timulak L, O'Brien E, Hayes C, Vigano N, Sharry J, et al. A randomized controlled trial of an internet-delivered treatment: its potential as a low-intensity community intervention for adults with symptoms of depression. Behav Res Ther. Dec 2015;75:20-31. [CrossRef] [Medline]
- Vis C, Mol M, Kleiboer A, Bührmann L, Finch T, Smit J, et al. Improving implementation of eMental health for mood disorders in routine practice: systematic review of barriers and facilitating factors. JMIR Ment Health. Mar 16, 2018;5(1):e20. [FREE Full text] [CrossRef] [Medline]
- Folker AP, Mathiasen K, Lauridsen SM, Stenderup E, Dozeman E, Folker MP. Implementing internet-delivered cognitive behavior therapy for common mental health disorders: a comparative case study of implementation challenges perceived by therapists and managers in five European internet services. Internet Interv. Mar 2018;11:60-70. [FREE Full text] [CrossRef] [Medline]
- Druss BG, Cohen AN, Brister T, Cotes RO, Hendry P, Rolin D, et al. Supporting the mental health workforce during and after COVID-19. Psychiatr Serv. Oct 01, 2021;72(10):1222-1224. [CrossRef] [Medline]
- Andersson G, Berg M, Riper H, Huppert JD, Titov N. The possible role of internet-delivered psychological interventions in relation to the COVID-19 pandemic. Clin Psychol Eur. Sep 30, 2020;2(3):e3941. [FREE Full text] [CrossRef] [Medline]
- Rodriguez-Villa E, Naslund J, Keshavan M, Patel V, Torous J. Making mental health more accessible in light of COVID-19: scalable digital health with digital navigators in low and middle-income countries. Asian J Psychiatr. Dec 2020;54:102433. [CrossRef] [Medline]
- Lipschitz J, Hogan TP, Bauer MS, Mohr DC. Closing the research-to-practice gap in digital psychiatry: the need to integrate implementation science. J Clin Psychiatry. May 14, 2019;80(3):2969. [FREE Full text] [CrossRef] [Medline]
- Eccles MP, Mittman BS. Welcome to implementation science. Implementation Sci. Feb 22, 2006;1(1):6. [CrossRef]
- Balas EA, Boren SA. Managing clinical knowledge for health care improvement. Yearb Med Inform. Mar 07, 2018;09(01):65-70. [CrossRef]
- Grant J, Cottrell R, Cluzeau F, Fawcett G. Evaluating "payback" on biomedical research from papers cited in clinical guidelines: applied bibliometric study. BMJ. Apr 22, 2000;320(7242):1107-1111. [FREE Full text] [CrossRef] [Medline]
- Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci. Apr 21, 2015;10:53. [FREE Full text] [CrossRef] [Medline]
- Smith JD, Polaha J. Using implementation science to guide the integration of evidence-based family interventions into primary care. Fam Syst Health. Jun 2017;35(2):125-135. [CrossRef] [Medline]
- Kadesjö Banck J, Bernhardsson S. Experiences from implementation of internet-delivered cognitive behaviour therapy for insomnia in psychiatric health care: a qualitative study applying the NASSS framework. BMC Health Serv Res. Aug 08, 2020;20(1):729. [FREE Full text] [CrossRef] [Medline]
- van der Vaart R, Worm-Smeitink M, Bos Y, Wensing M, Evers A, Knoop H. Implementing guided ICBT for chronic pain and fatigue: a qualitative evaluation among therapists and managers. Internet Interv. Dec 2019;18:100290. [FREE Full text] [CrossRef] [Medline]
- Skivington K, Matthews L, Simpson SA, Craig P, Baird J, Blazeby JM, et al. Framework for the development and evaluation of complex interventions: gap analysis, workshop and consultation-informed update. Health Technol Assess. Sep 2021;25(57):1-132. [FREE Full text] [CrossRef] [Medline]
- Titov N, Hadjistavropoulos HD, Nielssen O, Mohr DC, Andersson G, Dear BF. From research to practice: ten lessons in delivering digital mental health services. J Clin Med. Aug 17, 2019;8(8):1239. [FREE Full text] [CrossRef] [Medline]
- Titov N, Dear B, Nielssen O, Staples L, Hadjistavropoulos H, Nugent M, et al. ICBT in routine care: a descriptive analysis of successful clinics in five countries. Internet Interv. Sep 2018;13:108-115. [FREE Full text] [CrossRef] [Medline]
- Richards D, Viganó N. Online counseling: a narrative and critical review of the literature. J Clin Psychol. Sep 2013;69(9):994-1011. [FREE Full text] [CrossRef] [Medline]
- Friesen LN, Hadjistavropoulos HD, Pugh NE. A qualitative examination of psychology graduate students' experiences with guided Internet-delivered cognitive behaviour therapy. Internet Interv. Apr 2014;1(2):41-48. [CrossRef]
- Terpstra JA, van der Vaart R, Spillekom-van Koulil S, van Dam A, Rosmalen JG, Knoop H, et al. Becoming an eCoach: training therapists in online cognitive-behavioral therapy for chronic pain. Patient Educ Couns. Sep 2018;101(9):1702-1707. [CrossRef] [Medline]
- Hilty DM, Chan S, Torous J, Luo J, Boland RJ. A telehealth framework for mobile health, smartphones, and apps: competencies, training, and faculty development. J Technol Behav Sci. Apr 15, 2019;4(2):106-123. [CrossRef]
- Hadjistavropoulos HD, Nugent MM, Dirkse D, Pugh N. Implementation of internet-delivered cognitive behavior therapy within community mental health clinics: a process evaluation using the consolidated framework for implementation research. BMC Psychiatry. Sep 12, 2017;17(1):331. [FREE Full text] [CrossRef] [Medline]
- Goldsmith MR, Bankhead CR, Austoker J. Synthesising quantitative and qualitative research in evidence-based patient information. J Epidemiol Community Health. Mar 2007;61(3):262-270. [FREE Full text] [CrossRef] [Medline]
- Stern C, Lizarondo L, Carrier J, Godfrey C, Rieger K, Salmond S, et al. Methodological guidance for the conduct of mixed methods systematic reviews. JBI Evid Synth. Oct 2020;18(10):2108-2118. [CrossRef] [Medline]
- Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. Feb 12, 2015;10:21. [FREE Full text] [CrossRef] [Medline]
- Proctor EK, Powell BJ, McMillen JC. Implementation strategies: recommendations for specifying and reporting. Implement Sci. Dec 01, 2013;8:139. [FREE Full text] [CrossRef] [Medline]
- Hong QN, Pluye P, Bujold M, Wassef M. Convergent and sequential synthesis designs: implications for conducting and reporting systematic reviews of qualitative and quantitative evidence. Syst Rev. Mar 23, 2017;6(1):61. [FREE Full text] [CrossRef] [Medline]
- Pluye P, Hong QN. Combining the power of stories and the power of numbers: mixed methods research and mixed studies reviews. Annu Rev Public Health. 2014;35:29-45. [CrossRef] [Medline]
- Elliott R, Timulak L. Descriptive and interpretive approaches to qualitative research. In: Miles J, Gilbert P, editors. A Handbook of Research Methods for Clinical and Health Psychology. Oxford, UK. Oxford Academic Press; 2005:147-160.
- Elliott Jr RK, Timulak L. Essentials of Descriptive-Interpretive Qualitative Research: A Generic Approach. New York, NY. American Psychological Association; 2021.
- El Alaoui S, Hedman E, Kaldo V, Hesser H, Kraepelien M, Andersson E, et al. Effectiveness of Internet-based cognitive-behavior therapy for social anxiety disorder in clinical psychiatry. J Consult Clin Psychol. Oct 2015;83(5):902-914. [FREE Full text] [CrossRef] [Medline]
- Andrews G, Williams AD. Up-scaling clinician assisted internet cognitive behavioural therapy (iCBT) for depression: a model for dissemination into primary care. Clin Psychol Rev. Nov 2015;41:40-48. [CrossRef] [Medline]
- Karyotaki E, Kemmeren L, Riper H, Twisk J, Hoogendoorn A, Kleiboer A, et al. Is self-guided internet-based cognitive behavioural therapy (iCBT) harmful? An individual participant data meta-analysis. Psychol Med. Nov 15, 2018;48(15):2456-2466. [FREE Full text] [CrossRef] [Medline]
- Gellatly J, Chisnall L, Seccombe N, Ragan K, Lidbetter N, Cavanagh K. @Home eTherapy service for people with common mental health problems: an evaluation. Behav Cogn Psychother. Jan 16, 2018;46(1):115-120. [CrossRef] [Medline]
- Richards D, Murphy T, Viganó N, Timulak L, Doherty G, Sharry J, et al. Acceptability, satisfaction and perceived efficacy of "space from depression" an internet-delivered treatment for depression. Internet Interv. Sep 2016;5:12-22. [FREE Full text] [CrossRef] [Medline]
- Woods AP, Stults CB, Terry RL, Rego SA. Strengths and limitations of internet-based cognitive-behavioral treatments for anxiety disorders. Pragmat Case Stud Psychother. Nov 12, 2017;13(3):271-283. [CrossRef]
- Peynenburg VA, Mehta S, Hadjistavropoulos HD. Postsecondary student perceptions and preferences for the treatment of depression and anxiety: comparison of internet-delivered cognitive behaviour therapy to face-to-face cognitive behaviour therapy and medication. Can J Behav Sci. Jul 2020;52(3):220-230. [CrossRef]
- Wells MJ, Owen JJ, McCray LW, Bishop LB, Eells TD, Brown GK, et al. Computer-assisted cognitive-behavior therapy for depression in primary care. Prim Care Companion CNS Disord. Mar 01, 2018;20(2):11. [CrossRef]
- Mathiasen K, Riper H, Andersen TE, Roessler KK. Guided internet-based cognitive behavioral therapy for adult depression and anxiety in routine secondary care: observational study. J Med Internet Res. Nov 28, 2018;20(11):e10927. [FREE Full text] [CrossRef] [Medline]
- Nordgreen T, Gjestad R, Andersson G, Carlbring P, Havik OE. The effectiveness of guided internet-based cognitive behavioral therapy for social anxiety disorder in a routine care setting. Internet Interv. Sep 2018;13:24-29. [FREE Full text] [CrossRef] [Medline]
- Whiteside U, Richards J, Steinfeld B, Simon G, Caka S, Tachibana C, et al. Online cognitive behavioral therapy for depressed primary care patients: a pilot feasibility project. Perm J. 2014;18(2):21-27. [FREE Full text] [CrossRef] [Medline]
- Andersson G, Hedman E. Effectiveness of guided internet-based cognitive behavior therapy in regular clinical settings. Verhaltenstherapie. Aug 22, 2013;23(3):140-148. [CrossRef]
- Kenicer D, McClay CA, Williams C. A national survey of health service infrastructure and policy impacts on access to computerised CBT in Scotland. BMC Med Inform Decis Mak. Sep 08, 2012;12(1):102. [FREE Full text] [CrossRef] [Medline]
- Mol M, Dozeman E, Provoost S, van Schaik A, Riper H, Smit JH. Behind the scenes of online therapeutic feedback in blended therapy for depression: mixed-methods observational study. J Med Internet Res. May 03, 2018;20(5):e174. [FREE Full text] [CrossRef] [Medline]
- Wright JH, McCray LW, Eells TD, Gopalraj R, Bishop LB. Computer-assisted cognitive-behavior therapy in medical care settings. Curr Psychiatry Rep. Sep 07, 2018;20(10):92. [CrossRef] [Medline]
- Schröder J, Berger T, Meyer B, Lutz W, Späth C, Michel P, et al. Impact and change of attitudes toward internet interventions within a randomized controlled trial on individuals with depression symptoms. Depress Anxiety. May 28, 2018;35(5):421-430. [CrossRef] [Medline]
- Gullickson KM, Hadjistavropoulos HD, Dear BF, Titov N. Negative effects associated with internet-delivered cognitive behaviour therapy: an analysis of client emails. Internet Interv. Dec 2019;18:100278. [FREE Full text] [CrossRef] [Medline]
- Grist R, Cavanagh K. Computerised cognitive behavioural therapy for common mental health disorders, what works, for whom under what circumstances? A systematic review and meta-analysis. J Contemp Psychother. Sep 4, 2013;43(4):243-251. [CrossRef]
- Arnberg FK, Linton SJ, Hultcrantz M, Heintz E, Jonsson U. Internet-delivered psychological treatments for mood and anxiety disorders: a systematic review of their efficacy, safety, and cost-effectiveness. PLoS One. May 20, 2014;9(5):e98118. [FREE Full text] [CrossRef] [Medline]
- Schröder J, Berger T, Westermann S, Klein JP, Moritz S. Internet interventions for depression: new developments. Dialogues Clin Neurosci. Apr 01, 2022;18(2):203-212. [CrossRef]
- Andersson G, Titov N, Dear BF, Rozental A, Carlbring P. Internet-delivered psychological treatments: from innovation to implementation. World Psychiatry. Feb 02, 2019;18(1):20-28. [FREE Full text] [CrossRef] [Medline]
- Andersson G. The promise and pitfalls of the internet for cognitive behavioral therapy. BMC Med. Dec 07, 2010;8(1):82. [FREE Full text] [CrossRef] [Medline]
- Wilhelmsen M, Høifødt RS, Kolstrup N, Waterloo K, Eisemann M, Chenhall R, et al. Norwegian general practitioners' perspectives on implementation of a guided web-based cognitive behavioral therapy for depression: a qualitative study. J Med Internet Res. Sep 10, 2014;16(9):e208. [FREE Full text] [CrossRef] [Medline]
- So M, Yamaguchi S, Hashimoto S, Sado M, Furukawa TA, McCrone P. Is computerised CBT really helpful for adult depression?-A meta-analytic re-evaluation of CCBT for adult depression in terms of clinical implementation and methodological validity. BMC Psychiatry. Apr 15, 2013;13(1):113. [FREE Full text] [CrossRef] [Medline]
- Cavanagh K, Seccombe N, Lidbetter N. The implementation of computerized cognitive behavioural therapies in a service user-led, third sector self help clinic. Behav Cogn Psychother. Jul 22, 2011;39(4):427-442. [CrossRef] [Medline]
- Brantnell A, Woodford J, Baraldi E, van Achterberg T, von Essen L. Views of implementers and nonimplementers of internet-administered cognitive behavioral therapy for depression and anxiety: survey of primary care decision makers in Sweden. J Med Internet Res. Aug 12, 2020;22(8):e18033. [FREE Full text] [CrossRef] [Medline]
- Calero Elvira A, Santacreu Ivars M, Marchena Giráldez C, Shih P. Internet-based cognitive behavioural therapy programme with and without videoconference guidance sessions: a randomized controlled trial to treat work-related symptoms of anxiety and depression. Clin Psychol Psychother. Sep 23, 2021;28(5):1230-1242. [CrossRef] [Medline]
- Hadjistavropoulos HD, Peynenburg V, Thiessen DL, Nugent M, Karin E, Staples L, et al. Utilization, patient characteristics, and longitudinal improvements among patients from a provincially funded transdiagnostic internet-delivered cognitive behavioural therapy program: observational study of trends over 6 years. Can J Psychiatry. Mar 12, 2022;67(3):192-206. [FREE Full text] [CrossRef] [Medline]
- Leung LB, Dyer KE, Yano EM, Young AS, Rubenstein LV, Hamilton AB. Collaborative care clinician perceptions of computerized cognitive behavioral therapy for depression in primary care. Transl Behav Med. Aug 07, 2020;10(3):565-572. [FREE Full text] [CrossRef] [Medline]
- Lindegaard T, Seaton F, Halaj A, Berg M, Kashoush F, Barchini R, et al. Internet-based cognitive behavioural therapy for depression and anxiety among Arabic-speaking individuals in Sweden: a pilot randomized controlled trial. Cogn Behav Ther. Jan 30, 2021;50(1):47-66. [FREE Full text] [CrossRef] [Medline]
- Treanor CJ, Kouvonen A, Lallukka T, Donnelly M. Acceptability of computerized cognitive behavioral therapy for adults: umbrella review. JMIR Ment Health. Jul 06, 2021;8(7):e23091. [FREE Full text] [CrossRef] [Medline]
- McCall HC, Sison AP, Burnett JL, Beahm JD, Hadjistavropoulos HD. Exploring perceptions of internet-delivered cognitive behaviour therapy among public safety personnel: informing dissemination efforts. Int J Environ Res Public Health. Aug 19, 2020;17(17):6026. [FREE Full text] [CrossRef] [Medline]
- Pedersen MK, Mohammadi R, Mathiasen K, Elmose M. Internet-based cognitive behavioral therapy for anxiety in an outpatient specialized care setting: a qualitative study of the patients' experience of the therapy. Scand J Psychol. Dec 23, 2020;61(6):846-854. [CrossRef] [Medline]
- Piera-Jiménez J, Etzelmueller A, Kolovos S, Folkvord F, Lupiáñez-Villanueva F. Guided internet-based cognitive behavioral therapy for depression: implementation cost-effectiveness study. J Med Internet Res. May 11, 2021;23(5):e27410. [FREE Full text] [CrossRef] [Medline]
- Robichaud M, Talbot F, Titov N, Dear BF, Hadjistavropoulos HD, Hadjistavropoulos T, et al. Facilitating access to iCBT: a randomized controlled trial assessing a translated version of an empirically validated program using a minimally monitored delivery model. Behav Cogn Psychother. Mar 16, 2020;48(2):185-202. [CrossRef] [Medline]
- Duffy D, Enrique A, Connell S, Connolly C, Richards D. Internet-delivered cognitive behavior therapy as a prequel to face-to-face therapy for depression and anxiety: a naturalistic observation. Front Psychiatry. Jan 9, 2019;10:902. [FREE Full text] [CrossRef] [Medline]
- Richards D, Duffy D, Burke J, Anderson M, Connell S, Timulak L. Supported internet-delivered cognitive behavior treatment for adults with severe depressive symptoms: a secondary analysis. JMIR Ment Health. Oct 02, 2018;5(4):e10204. [FREE Full text] [CrossRef] [Medline]
- Bower P, Kontopantelis E, Sutton A, Kendrick T, Richards DA, Gilbody S, et al. Influence of initial severity of depression on effectiveness of low intensity interventions: meta-analysis of individual patient data. BMJ. Feb 26, 2013;346:f540. [FREE Full text] [CrossRef] [Medline]
- Meyer B, Bierbrodt J, Schröder J, Berger T, Beevers CG, Weiss M, et al. Effects of an Internet intervention (Deprexis) on severe depression symptoms: randomized controlled trial. Internet Interv. Mar 2015;2(1):48-59. [CrossRef]
- The NICE Guideline on the treatment and management of depression (CG90). The National Institute for Health and Care Excellence. 2009. URL: https://pmc.ncbi.nlm.nih.gov/articles/PMC548164/#:~:text=The%20guidelines%20do%20not%20recommend,also%20present%20for%20psychotherapeutic%20interventions [accessed 2024-04-29]
- Anxiety: Management of anxiety (panic disorder, with or without agoraphobia, and generalised anxiety disorder) in adults in primary, secondary and community care (CG22). The National Institute for Health and Care Excellence. 2004. URL: https://www.nice.org.uk/guidance/cg22 [accessed 2024-04-29]
- Depression: management of depression in primary and secondary care (CG23). The National Institute for Health and Care Excellence. 2004. URL: https://www.nice.org.uk/guidance/cg23 [accessed 2024-04-29]
- Judge TA, Bono JE. Five-factor model of personality and transformational leadership. J Appl Psychol. 2000;85(5):751-765. [CrossRef]
- Aarons GA, Sommerfeld DH. Leadership, innovation climate, and attitudes toward evidence-based practice during a statewide implementation. J Am Acad Child Adolesc Psychiatry. Apr 2012;51(4):423-431. [FREE Full text] [CrossRef] [Medline]
- Frank HE, Becker-Haimes EM, Kendall PC. Therapist training in evidence-based interventions for mental health: a systematic review of training approaches and outcomes. Clin Psychol (New York). Sep 2020;27(3):e12330. [FREE Full text] [CrossRef] [Medline]
- Beidas RS, Kendall PC. Training therapists in evidence-based practice: a critical review of studies from a systems-contextual perspective. Clin Psychol (New York). Mar 2010;17(1):1-30. [FREE Full text] [CrossRef] [Medline]
- Sivakumar A, Pan RY, Choi D, Wang AH, Yu CH. "If we got a win-win, you can sell it to everybody": a qualitative study employing normalization process theory to identify critical factors for eHealth implementation and scale-up in primary care. Can J Diabetes. Mar 2022;46(2):181-188. [FREE Full text] [CrossRef] [Medline]
- May C, Finch T. Implementing, embedding, and integrating practices: an outline of normalization process theory. Sociology. Jun 15, 2009;43(3):535-554. [CrossRef]
- Greenhalgh T, Wherton J, Papoutsi C, Lynch J, Hughes G, A'Court C, et al. Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies. J Med Internet Res. Nov 01, 2017;19(11):e367. [FREE Full text] [CrossRef] [Medline]
- Rooshenas L, Elliott D, Wade J, Jepson M, Paramasivan S, Strong S, ACST-2 study group, By-Band-Sleeve study group, Chemorad study group, CSAW study group, Optima prelim study group, et al. POUT study group. Conveying equipoise during recruitment for clinical trials: qualitative synthesis of clinicians' practices across six randomised controlled trials. PLoS Med. Oct 2016;13(10):e1002147. [FREE Full text] [CrossRef] [Medline]
- von Ranson KM, Robinson KE. Who is providing what type of psychotherapy to eating disorder clients? A survey. Int J Eat Disord. Jan 2006;39(1):27-34. [CrossRef] [Medline]
- Stewart RE, Chambless DL. Does psychotherapy research inform treatment decisions in private practice? J Clin Psychol. Mar 2007;63(3):267-281. [CrossRef] [Medline]
- Stewart RE, Chambless DL, Stirman SW. Decision making and the use of evidence based practice: is the three-legged stool balanced? Pract Innov (Wash D C). Mar 2018;3(1):56-67. [FREE Full text] [CrossRef] [Medline]
- Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004;82(4):581-629. [FREE Full text] [CrossRef] [Medline]
- Gadolin C, Andersson T. Healthcare quality improvement work: a professional employee perspective. Int J Health Care Qual Assur. Jun 12, 2017;30(5):410-423. [CrossRef] [Medline]
- Weiner BJ. A theory of organizational readiness for change. Implement Sci. Oct 19, 2009;4:67. [FREE Full text] [CrossRef] [Medline]
- Weiner BJ, Belden CM, Bergmire DM, Johnston M. The meaning and measurement of implementation climate. Implement Sci. Jul 22, 2011;6:78. [FREE Full text] [CrossRef] [Medline]
- Klein KJ, Sorra JS. The challenge of innovation implementation. Acad Manag Rev. Oct 1996;21(4):1055-1080. [CrossRef]
- Milat AJ, Bauman A, Redman S. Narrative review of models and success factors for scaling up public health interventions. Implement Sci. Aug 12, 2015;10:113. [FREE Full text] [CrossRef] [Medline]
- Chambers D, Cantrell A, Booth A. Factors that facilitate the implementation of interventions to reduce preventable hospital admissions with a focus on cardiovascular or respiratory conditions: an evidence map and realist synthesis. National Institute for Health and Care Research. URL: https://njl-admin.nihr.ac.uk/document/download/2033157 [accessed 2019-12-05]
- Pihlaja S, Lahti J, Lipsanen JO, Ritola V, Gummerus EM, Stenberg JH, et al. Scheduled telephone support for internet cognitive behavioral therapy for depression in patients at risk for dropout: pragmatic randomized controlled trial. J Med Internet Res. Jul 23, 2020;22(7):e15732. [FREE Full text] [CrossRef] [Medline]
- Eysenbach G, CONSORT-EHEALTH Group. CONSORT-EHEALTH: improving and standardizing evaluation reports of web-based and mobile health interventions. J Med Internet Res. Dec 31, 2011;13(4):e126. [FREE Full text] [CrossRef] [Medline]
- Thomas J, Harden A, Oakley A, Oliver S, Sutcliffe K, Rees R, et al. Integrating qualitative research with trials in systematic reviews. BMJ. Apr 24, 2004;328(7446):1010-1012. [FREE Full text] [CrossRef] [Medline]
- Higgins JP, Green S. Cochrane handbook for systematic reviews of interventions version 5.1.0. The Cochrane Collaboration. Sep 06, 2011. URL: https://handbook-5-1.cochrane.org/ [accessed 2024-04-29]
- Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol. Jul 10, 2008;8:45. [FREE Full text] [CrossRef] [Medline]
Abbreviations
CONSORT: Consolidated Standards of Reporting Trials |
iCBT: internet-delivered cognitive behavioral therapy |
PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses |
Edited by A Mavragani; submitted 05.04.23; peer-reviewed by L Yang, J Abbas, A AL-Asadi, M Attridge; comments to author 24.02.24; revised version received 28.06.24; accepted 06.08.24; published 28.01.25.
Copyright©Daniel Duffy, Derek Richards, Garrett Hisler, Ladislav Timulak. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 28.01.2025.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research (ISSN 1438-8871), is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.