This paper is in the following e-collection/theme issue:

Digital Health Reviews (1289) e-Learning and Medical Education (691) Technology in Psychiatry/Clinical Psychology Training and Education (27) Theme issue 2018: Health professions digital education: what works best, when, for whom and at what cost? (21) Learning and Education (76) mHealth in Medical Education and Training (77) Evaluation of Medical Education (78) New Methods and Approaches in Medical Education (232)

Published on in Vol 26 (2024)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/54473, first published .
Learning Outcomes of e-Learning in Psychotherapy Training and Comparison With Conventional Training Methods: Systematic Review

Learning Outcomes of e-Learning in Psychotherapy Training and Comparison With Conventional Training Methods: Systematic Review

Learning Outcomes of e-Learning in Psychotherapy Training and Comparison With Conventional Training Methods: Systematic Review

Authors of this article:

Kasperi Mikkonen1, 2 Author Orcid Image ;   Eeva-Eerika Helminen1, 2 Author Orcid Image ;   Samuli I Saarni2, 3 Author Orcid Image ;   Suoma E Saarni2, 3, 4 Author Orcid Image
Article Authors Cited by (1) Tweetations Metrics

    Review

    1Faculty of Medicine, University of Helsinki, Helsinki, Finland

    2Department of Psychiatry, Helsinki University Hospital, Helsinki, Finland

    3Faculty of Medicine and Health Technology, University of Tampere, Tampere, Finland

    4Psychiatry, Wellbeing Services County of Päijät-Häme, Lahti, Finland

    Corresponding Author:

    Kasperi Mikkonen, MA

    Department of Psychiatry

    Helsinki University Hospital

    Välskärinkatu 12

    Helsinki, 00029 HUS

    Finland

    Phone: 358 405561374

    Email: kasperi.mikkonen@gmail.com


    Background: Mental disorders pose a major public health problem in most western countries. The demand for services for common mental health disorders has been on the rise despite the widespread accessibility of medication. Especially, the supply and demand for evidence-based psychotherapy do not align. Large-scale increase of modern psychotherapy is difficult with current methods of training which are often expensive, time consuming, and dependent on a small number of top-level professionals as trainers. E-learning has been proposed to enhance psychotherapy training accessibility, quality, and scalability.

    Objective: This systematic review aims to provide an overview of the current evidence regarding e-learning in psychotherapy training. In particular, the review examines the usability, acceptability, and learning outcomes associated with e-learning. Learning outcomes are assessed in different modalities including trainee experiences, knowledge acquisition, skill acquisition, and application of trained content in daily practice. Furthermore, the equivalence of web-based training and conventional training methods is evaluated.

    Methods: Following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines, a search from Ovid, MEDLINE, PsycINFO, and Scopus databases between 2008 and June 2022 was conducted. Inclusion criteria required studies to describe e-learning systems for psychotherapy training and assess acceptability, feasibility, or learning outcomes. The risk of bias was evaluated for both randomized and nonrandomized studies. Learning outcomes were categorized using the Kirkpatrick model. Effect sizes comparing e-learning and traditional methods were calculated.

    Results: The search yielded 3380 publications, of which 34 fulfilled the inclusion criteria. Positive learning outcomes are generally associated with various e-learning programs in psychotherapy training including trainee satisfaction, knowledge, and skill acquisition, and in application of trained content in clinical practice. Learning outcomes generally show equivalence between e-learning and conventional training methods. The overall effect size, indicating this disparity, was 0.01, suggesting no significant difference. This literature displays a high level of heterogeneity in e-learning solutions and assessment methods.

    Conclusions: e-Learning seems to have good potential to enhance psychotherapy training by increasing access, scalability, and cost-effectiveness while maintaining quality in terms of learning outcomes. Results are congruent with findings related to e-learning in health education in general where e-learning as a pedagogy is linked to an opportunity to carry out learner-centric practices. Recommendations for conducting psychotherapy training programs in blended settings supported by activating learning methods are presented. However, due to the heterogeneity and limitations in the existing literature, further research is necessary to replicate these findings and to establish global standards for e-learning, as well as for the assessment of training outcomes in psychotherapy education. Research is especially needed on the effects of training on patient outcomes and optimal ways to combine e-learning and conventional training methods in blended learning settings.

    J Med Internet Res 2024;26:e54473

    doi:10.2196/54473

    Keywords

    mental health; psychotherapy; digital learning; Kirkpatrick model; e-learning; online health; psychotherapy training; learning outcome; learning outcomes; systematic review; training methods; mental disorders; mental disorder; accessibility; evidence-based; scalability; cost-effectiveness; internet; education 


    Mental disorders pose a global public health problem [1]. Psychotherapy has been found to be effective across a variety of mental disorders [2-4]. However, access to evidence-based psychotherapy in public health care settings is often overly difficult and the supply and demand of evidence-based methods do not align [5]. One widely accepted strategy in meeting the existing and increasing demand of mental health services is to train health care professionals in various psychotherapies. The need for training encompasses both the training of new practitioners and updating the competences of existing professionals following the approach of continuing medical education.

    Closing the gap between the demand for services and available resources through conventional training methods can be challenging. The main barriers in the dissemination of psychotherapy training include cost and time [6]. Conventionally, psychotherapy training is labor-intensive and relies heavily on the time provided by top-level professionals carrying out lectures and workshops that often lead to expensive and time-consuming programs and problems when striving to scale up. While psychotherapy training is linked to positive learning outcomes, the evidence on the impact of training on patient outcomes is mixed [7]. Digital solutions and e-learning have been acknowledged as a potential tool in expanding the reach of training, increasing the quality of training programs, and improving the dissemination of treatment [8-10]. With the comprehensive educational need, using digital solutions may be necessary.

    e-Learning, which is often used synonymously with “digital learning,” “online learning,” and “web-based learning,” can be conceptualized as a framework that combines educational theories and technology or a practical application of educational technology that evolves in tandem with the advancements in technology and the internet [11,12]. The terminology is overlapping, and different labels give rise to diverse expectations [13]. One detail in terminology to highlight is the emphasis on “distance” between the trainer and the trainee, which is not a defining characteristic of e-learning [14]. It is tied to the trainee engagement and whether the training is held synchronously (eg, Zoom lectures) or asynchronously (eg, self-directed and individually paced learning in a web-based environment with multimedia materials and assignments).

    Asynchronous e-learning with an emphasis on the active role of the trainee offers a possible solution for many current problems in psychotherapy training. First, e-learning offers a platform to carry out pedagogical principles that prioritize trainee engagement, as seen in approaches such as the flipped classroom, which involves a transition from trainer-led lectures to individual studying which happens before the synchronous learning event and which has been associated with improvements in learning outcomes compared with conventional training methods in health profession education [15,16]. Second, asynchronous e-learning offers flexibility in terms of time and place, which can remove some of the constraints involved in conventional training methods. Third, e-learning content can be scaled up more easily than conventional training methods. Fourth, while the current training programs are mostly rooted on various therapeutic traditions (or “schools”), which often hinders evidence-based development of psychotherapy, an adequate use of e-learning can be a way to create transparent training content which is easy to update and which fosters therapeutic competence [17-19]. Moreover, while the evidence regarding the cost-effectiveness of e-learning in health education is still in its infancy, e-learning is often associated as a means to increase the cost-effectiveness of training programs in the long term [20].

    e-Learning is a widely used method for training in various industries and is generally associated with positive learning outcomes that are either comparable or superior to conventional training methods [21-23]. The literature examining e-learning in psychotherapy training is emerging, and its promise has been recognized [24]. However, an overview of the current evidence-base remains undetermined, and it remains unspecified how e-learning should be precisely used. For example, what is the role of e-learning in knowledge and skill acquisition, and is the value of e-learning most apparent when implemented as a blended learning curriculum, which combines conventional training methods with e-learning [19]?

    The goals of this systematic literature review are to (1) provide an overview and assess the quality of the current literature; (2) determine the acceptability, usability, and achievable learning outcomes of e-learning in the context of psychotherapy training; and (3) assess how does e-learning compared with conventional training methods. Based on our results, we aim to formulate the current maturity and limitations of e-learning in this domain and discuss what kind of recommendations can be made regarding e-learning and psychotherapy.


    Design

    A systematic review was conducted to extract recently published scientific papers that dealt with e-learning in psychotherapy training. We focused on the last 15 years (from 2008 onward) since rapid software and hardware development has occurred during that period although internet delivered and computerized e-learning as a concept emerged as early as the 1990s [25]. This review followed the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines [26].

    Search Method

    The search was conducted within Ovid MEDLINE, PsycINFO, and Scopus databases. The search strategy included a process where overlapping labels depicting psychotherapy and e-learning were combined. The search was formulated to cover both the training of larger treatment modalities and the specific therapeutic competences. Thus, the search strings were as follows: (psychotherap* or psycholog* or therap* or counsel*) adjacent to (skill* or train* or educat* or compentenc*) OR (interpersonal) adjacent to (skill* or train* or educat* or compentenc*) OR (psychotherap* or psychosoc* or mental health or behavioral health or behavioural health) adjancent to (intervent* or treat* or therap* or evidence-based intervent* or evidence-based treat* or evidence-based therapy* AND (digital* or mobile or web-based or online or distance) adjancent to (learn* or train* or educat* or program* or self-administered learn* or self-administered train* or self-administered educat* or self-administered program*) OR (e-learn* or elearn* or e-educat*). The searches were conducted on all databases in June 2022. To ensure optimal coverage considering the field’s unestablished terminology, a comprehensive array of search terms was used.

    Selection Criteria

    Inclusion

    To be included in this review, the studies should involve (1) a description of an asynchronous e-learning system (ie, a digital learning environment, eg, a website designed for self-directed learning) in the context of training psychotherapy competences and (2) measures of acceptability, feasibility, or learning outcomes.

    Exclusion

    Studies were excluded if they (1) represented an unpublished thesis or dissertation, (2) were not peer-reviewed, (3) were not experimental studies, (4) were unavailable in English, and (5) defined e-learning solely as synchronous distance learning (eg, Zoom lectures).

    Study Selection

    The eligibility of the search results was assessed by the first (KM) and last (SES) authors through three sequential steps: (1) initially, KM meticulously examined the titles and abstracts of all search results, systematically eliminating studies that distinctly met the specified exclusion criteria; (2) in cases where the title or abstract lacked sufficient detail to determine eligibility, KM proceeded to evaluate the full text of the study, applying the same exclusion benchmarks as in the initial phase; and (3) for the final step, KM scrutinized the full texts of all remaining studies, while SES independently reviewed a randomly selected subset of these studies to ensure representativeness. Any differences in opinion were subsequently resolved through a collaborative discussion targeting a unanimous decision. This stepwise approach ensured that studies were disqualified only if they unambiguously satisfied the predefined exclusion criteria. Given our anticipatory search terms’ breadth, a substantial volume of studies not aligning with our inclusion criteria, such as those investigating digital treatment interventions rather than the training of professionals, was anticipated. Hence, to optimize efficiency, the first 2 screening stages were singularly executed by 1 reviewer (KM).

    Data Collection

    Examination of the titles and abstracts of the publications and the assessment for potential inclusion were carried out by first (KM) and last author (SES). Data extraction was carried out by the first author and included (1) target group of the training; (2) description of training content; (3) description of e-learning methods; (4) comparison between training modalities; (5) sample sizes; (6) primary outcome; and (7) summary of results. Furthermore, to calculate a measure of effect size for comparing differences between conventional training methods and e-learning, the means and standard deviations of the learning outcomes reported in the studies were extracted.

    Risk of Bias

    The risk of bias was assessed with the latest Cochrane risk-of-bias tools: RoB 2—tool for randomized trials and ROBINS-I (Risk of Bias in Non-randomized Studies of Interventions) for nonrandomized trials [27,28]. The risk of bias analysis was systematically conducted by the first (KM) and last (SES) authors in an independent manner. Subsequent to this individual assessment, any emergent discrepancies were collaboratively addressed through discussions focused on achieving consensus.

    Framework for Classifying Results

    We classified the results of the primary outcomes of the included studies using the Kirkpatrick model of training evaluation [29], which evaluates training outcomes on 4 levels. The first level (“reaction”) evaluates the training participants’ immediate reaction and feedback regarding the training program including satisfaction, engagement, and relevance. The second (“learning”) evaluates the learning outcomes of the training program including change in attitudes, confidence, knowledge, and skills. The third level (“behavior”) evaluates how much the trainee participants apply the learned knowledge and skills when treating patients in real life. The final level (“results”) evaluates how much the patient outcomes improve because of the training.

    Comparison Between Training Methods

    In order to quantify the magnitude of the difference observed in these results, Hedges g was calculated as a measure of effect size with 95% CIs [30]. Hedges g is comparable with Cohen d but corrects small sample bias. Therefore, Hedges g of 0.2 represents a small effect; 0.5, a moderate effect; and 0.8, a large effect [31]. The effect size was calculated from the studies that included the figures necessary for the calculations. For studies with missing information, the first author of the publication was contacted, and they were asked for the data. An overall effect size describing the equivalence between training methods was formulated as an arithmetic mean of all calculated effect sizes.


    Search Outcome

    The first screening from the databases identified 3380 articles. Duplicates (n=697) were removed using the “Merge items” tool in the “Duplicate items”—folder in “Zotero,” which the authors used as a reference management software. After removing the duplicates, 2683 articles were identified and the eligibility was assessed by KM based on their title, and abstract if necessary, and 2530 articles were excluded as clearly fulfilling the exclusion criteria mentioned in the “Selection Criteria” section of the article (see Figure 1). The remaining 153 articles were read in full-text versions by KM and 79 articles were excluded according to the same criteria. KM reviewed the remaining 74 articles in full, and SES reviewed in full, a random sample of those articles (n=45). The size of random sample required was calculated using Cohen κ for interrater agreement level with the assumptions that 30%-40% of the remaining articles would be included in the final sample, and the interrater agreement is moderate at the least [32]. The calculations were conducted with R with the “CIBinary”—function within the “kappaSize”—library. Based on these calculations, the last author reviewed the random sample of 45 articles. The authors worked blind to each other. The results were compared, and the authors were in substantial agreement on which articles to include and exclude (Cohen κ=0.73). All discrepancies were discussed to a consensus (eg, a detailed discussion whether the target of a training described in a specific study was related to psychotherapy skills or more general interaction skills used in health care). Thirty-four studies were included in the final review.

    ‎
    Figure 1. PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flowchart.

    The articles included in the study are listed in Table 1 for randomized controlled trials (n=17) and Table 2 for nonrandomized (n=17) studies. The tables include the target group of the training, the content of the training, a depiction of the e-learning system used, a comparison between training modalities if applicable, the primary outcomes of the studies, and a summary of the main findings. In all studies, training was targeted to health care providers. The training content was most often related to some form of cognitive behavioral therapy (CBT) and the length of the e-learning programs ranged from 15 minutes to 52 hours. In the studies included, the effects of e-learning are evaluated in within-trainee designs and comparisons with waiting for training and other active training methods. The data were extracted by the first author as they were presented in the original studies.

    Table 1. Randomized controlled trials included in the review.
    StudyTarget groupTraining contentDescription of e-learningComparison and sample sizesPrimary outcomeSummary of results
    Shinohara et al (2022) [33]Midwives and public health nursesPerinatal mental health assessment and empathic communication1-day course with short video scriptsIntervention group which received training (n=58) vs no-training control group (n=57)Improvement of empathic communicationThe intervention group was at a significantly higher level for empathic communication skills compared with the control group (P<.001)
    Nisar et al (2022) [34]Nursing school studentsEvidence-based psychosocial intervention for perinatal depressionA tablet-based, multimedia training program including narrative scriptse-Learning intervention group (n=42) vs conventional specialist–delivered training (n=47)Competence in the trained intervention posttraining and at 3-month follow-upe-Learning was not inferior to specialist-led training (P=.14)
    Taylor et al (2021) [35]Licensed behavioral health providersCognitive Behavioral Therapy for InsomniaA website including video instructions and demonstrations, descriptions of intervention techniques, handouts, and exercisese-Learning group (n=21) vs in-person workshop control (n=23)Knowledge acquisition in CBT-I aKnowledge acquisition was equivalent in both groups (P=.16)
    Sansen et al (2020) [36]Psychotherapists and psychotherapist traineesTrauma-focused cognitive behavioral therapy52-hour web-based training course including written information, animations, video clips, various assignments, and simulated casese-Learning group (n=247) vs waitlist control group (n=252)Subjective ratings of knowledge, confidence, competencies, and willingness to use trauma-focused intervention and performance in a knowledge testSignificantly greater improved subjective ratings of knowledge, confidence, competencies, and willingness, as well as performance in the knowledge test in the e-learning group compared with the control group (all P values <.01)
    Sansen et al (2019) [37]Psychotherapists and psychotherapist traineesTrauma-focused cognitive behavioral therapy52-hour web-based training course including written information, animations, video clips, various assignments, and simulated casese-Learning group (n=247) vs waitlist control group (n=252)Fears and reservation toward conducting trauma-focused methodsA significant reduction of fears and reservations during e-learning (P<.01)
    Kullberg et al (2020) [38]Undergraduate psychology studentsSuicide prevention practice1-hour web-based training including videos of professionals interacting with various patients with suicidal tendenciese-Learning group (n=211) vs waitlist control group (n=187)Self-reported guideline adherence, knowledge acquisition, and provider confidenceSignificantly greater improvements in self-reported knowledge, confidence, and guideline adherence in the e-learning group compared with the control group (all P values <.01). Learning effects were maintained in 3-month follow-up
    Rahman et al (2019) [39]Nonspecialist health workersPsychosocial management of perinatal depressionA tablet-based training course including narrative scripts, reflective assignments, and video role-playse-Learning group (n=30) vs face-to-face training control group (n=30)Competence in interventionNo differences between e-learning and face-to-face training in competence scores post assessment and at 3-month follow-up (P>.05)
    Puspitasari et al (2017) [40]Mental health cliniciansBehavioral activation4 × 40-minute web-based audio-guided presentations of core BAb skillse-Learning intervention group- (n=40) vs trainer-led training (n=37)BA skills competenceSignificantly greater increases in all BA skills in trainer-led training compared with web-based training (P<.0005)
    Stein et al (2015) [41]Mental health cliniciansInterpersonal and social rhythm therapy for bipolar disorder12-hour self-paced web-based traininge-Learning group + 3-6 months of monthly telephone supervision (n=16) vs control group in 2-day workshop and weekly supervision (n=20)Use of IPSRTc techniques during treatmentA significant increase in the use of IPSRT techniques in posttraining which increased in 12-month follow-up (P<.01). The increase was comparable in both groups
    Dimeff et al (2015) [42]Mental health cliniciansTwo core strategies in DBTd: chain analysis and validation

    		Interactive, media-rich web-based training modules with expert commentaries, exercises, simulations, printable handouts, and knowledge checksWeb-based training (e-learning) (n=55) vs ILTe (n=55) vs TMf (n=62)

    		Satisfaction, self-efficacy, motivation, knowledge, clinical proficiency, and clinical useILT was superior compared with web-based training (e-learning) and TM in satisfaction (P<.0001), self-efficacy (P<.001), and motivation (P<.05). Web-based training (e-learning) was the most effective method for increasing knowledge (P<.01). There were no differences between observer-rated proficiency (P=.76) and self-reported clinical use (P=.48), which increased during training
    Harned et al (2014) [43]Mental health providersExposure therapy for anxiety disorders10-hour course incorporating gaming technology and evidence-based instructional design strategies. The learner could choose whether to engage with media-rich didactic materials or simulated clinical scenarios

    		Web-based training (e-learning) (n=60) vs web-based training (e-learning) + brief computerized motivational enhancement intervention (web-based training [e-learning] + MEg) (n=60) vs web-based training (e-learning) + ME + a web-based learning community (web-based training (e-learning) + ME + LCh) (n=61)

    		Knowledge acquisition, attitudes toward exposure therapy, self-efficacy, clinical use, and clinical proficiency

    		All training methods comparably improved self-efficacy and clinical use of exposure therapy. The increase in attitudes in the web-based training (e-learning) + ME + LC group at 6-week follow-up was significantly larger than in the web-based training (e-learning) group (P=.01) as well as in clinical proficiency at 12-week follow-up (P=.04). Web-based training (e-learning)–ME-LC condition was also superior in knowledge acquisition (P values =0.004-0.05)

    Ruzek et al (2014) [44]Mental health clinicians with responsibilities in treating veterans with PTSDiCBTj skills for traumatic stress: motivation enhancement, goal setting, and behavioral task assignment

    		3-module web-based training with interactive exercises, skills demonstrations, downloadable materials, and bibliographiese-Learning group (n=46) vs e-learning + telephone consultation (n=42) vs no-training or training as usual (free to take part in any training activities they would otherwise receive) (n=51)Skills acquisitionSkills improved significantly in both active training groups compared with the control group in motivation enhancement and goal setting (P<.001, P<.005). Web-based training plus supervision was superior compared with web-based training alone in the motivation enhancement module (P=.001) with no significant differences in the 2 other modules.

    Larson et al (2013) [45]Addiction counsellors and supervisorsCognitive behavioral therapyWeb-based training including various exercises, audio-vignettes and role-played treatment sessions, and an assessment between modulese-Learning (n=47) vs training manual (n=52)Adherence to CBT practiceNo statistically significant difference in pass rates and adequacy scores in adherence to CBT practice between groups (P=.19). The posttraining gains in CBT skill adequacy were small
    Bennett-Levy et al (2012) [46]Practicing in a counseling and mental health role with a relevant degree or students closely completing a degreeCognitive behavioral therapy30-module web-based training with didactive text and video demonstrations with case examples and quizzese-Learning group (n=23) vs supported training group with the possibility of six 15-minute support calls (n=24)Completion rate, knowledge acquisition, self-reported skills acquisition, confidence in CBT, and skills use in practiceBoth groups demonstrated similar improvements in objective knowledge test and on self-reported measures of knowledge, skills, confidence, and use of skills at follow-up (improvement all significant at P<.001). Supported training group had a significantly higher completion rate (P<.05).
    Dimeff et al (2011) [47]Community mental health providersDBT distress tolerance skillsA 5-module web-based training with media-rich content, expert insights, practice exercises, and knowledge checks with downloadable handoutse-Learning (n=47) vs training manual (n=43) vs placebo e-learning (n=42)Knowledge acquisition, self-reported self-efficacy, and motivation to use learned skillsWeb-based training and the TM outperformed the control condition in knowledge acquisition (P<.05) and self-efficacy (P<.05) but not in motivation to learn and use the treatment. By the 15-week follow-up, the web-based training outperformed manual in knowledge acquisition (P<.05)
    Dimeff et al (2009) [48]Community mental health providers

    		Dialectical behavior therapyA 20-hour 5-module web-based training with media-rich content, expert insights, clinical vignettes, practice exercises, and knowledge checkse-Learning (n=54) vs training manual (n=49) vs ILT (2-day workshop) (n=47)Satisfaction, knowledge acquisition, confidence, self-efficacy, and application of trained content. Performance in structured role-playHigher satisfaction in web-based training and ILT than in training-manual (P<.001), but no differences between web-based training and ILT (P=.33). Web-based training outperformed the other conditions in knowledge acquisition (P<.05) while there were no differences between ILT and training manual (P values >.90). At posttraining, web-based training and ILT conditions reported more elevated self-efficacy than the manual condition (P<.01) but the differences diminished in follow-up. All training methods resulted in comparable increase in the application of trained content in clinical situations (P=.68). Adherence and competence in role-play scenario improved significantly in all conditions but there was no difference between conditions (P values >.20)
    Weingard et al (2006) [49]Substance abuse counsellorsA cognitive behavioral approach in treating cocaine addictionA 60-minute web-based training module with pre- and posttestse-Learning (n=52) vs face-to-face workshop (n=55) vs delayed training (n=59)Knowledge acquisitionImprovements in knowledge were significant comparable in both web-based training and face-to-face workshops

    aCBT-I: Cognitive Behavioral Therapy for Insomnia.

    bBA: behavioral activation.

    cIPSRT: interpersonal and social rhythm therapy for bipolar disorder.

    dDBT: dialectical behavior therapy.

    eILT: instructor-led training.

    fTM: treatment manual.

    gME: motivational enhancement.

    hLC: learning community.

    iPTSD: posttraumatic stress disorder.

    jCBT: cognitive behavioral therapy.

    Table 2. Nonrandomized trials included in the review.
    StudyTarget groupTraining contentDescription of e-learningComparison and sample size(s)Primary outcomeSummary of results
    Wilkerson et al (2022) [50]Health and mental health professionals, educators, and traineesCognitive Behavioral Therapy for Insomnia6- to 10-hour 10-module web-based training with didactic videos, therapy vignettes, and resources regarding the specific modulesN=2586Training completion rate and knowledge acquisition1203/2586 (46.52%) of registered users started the training and 570/2586 completed the training with a 3-month period with both pre-and posttests complete. A significant increase in knowledge was reported (P<.001)
    Seidler et al (2022) [51]Mental health practitionersEngaging men in therapy and dealing with mental distress and suicidality experiences by males8-hour web-based media-rich asynchronous training program including written information, downloadable worksheets, and videosN=192Improvement in self-reported competencies in engaging male clientsSignificant increase in self-reported competencies (P<.001)
    Soll et al (2021) [52]Postgraduate health care professionalsCognitive behavioral therapy for psychosisPrerecorded lectures, exercises, web-based role-plays, and web-based conferencese-Learning (n=85), in-person training (n=142)Satisfaction and acceptanceParticipants’ satisfaction and acceptance toward the web-based training were noninferior compared with in-person training in all respects except for “room for active participation” and “professional benefit”
    Applebaum et al (2021) [53]Health care professionals working in psychosocial oncologyAcute cancer cognitive therapy6-hour web-based training course including written information and various assignmentsN=46Knowledge acquisitionA significant increase in knowledge (P<.001)
    Gonzalez Salas Duhne et al (2020) [54]Professionals delivering psychological interventionImplementation intentions (“if-then”—planning)A 15-minute video-based training program with pre- and postquestionnairesN=87Knowledge acquisition, acquisition of practical knowledge, and self-reported use of implementation intentionsSignificant increase in knowledge both posttraining and in 1-month follow-up (P<.001). No significant increase in practical knowledge. A significant increase in use of implementation intentions (P=.018)
    Graber (2019) [55]Psychiatric mental health nursing studentsTherapeutic crisis management techniquesA 30-minute presentation in a web-based educational environmente-Learning (n=47), in-person training (n=63)Knowledge acquisition and student satisfactionNo differences between e-learning and in-person training in knowledge acquisition (P=.483) or student satisfaction (P=.87)
    Cramer et al (2019) [56]Mental health professionalsCore competencies in suicide prevention20-hour self-paced narrated presentations with associated resources and handouts, case descriptions, and various assignmentsN=43Knowledge acquisition, attitudes toward self-harm patients, and skills performanceImprovement in knowledge (P<.001), perceived skill mastery (P<.001), capacity to work with suicidal patients (P<.001), objective risk judgment accuracy (P<.001), and reduction of negative feelings about self-harming patients (P<.007)
    Meredith et al (2018) [57]Nurses and occupational therapists in mental health settingsSensory modulation approaches in mental healthA 2-hour, 3 self-paced modules including interactive activities, case scenarios, opportunities of self-reflection, and links to further readingN=121Knowledge acquisition and change in perceived level of knowledge confidence and attitudesSignificant increase in knowledge acquisition, confidence, and attitude toward sensory modulation approaches (all P values <.001)
    German et al (2018) [58]Mental health cliniciansCognitive behavioral therapyA self-paced web-based training including videotaped role-playing, on-screen activities, and quizzese-Learning with peer-led consultation (n=148), in-person training with expert-led consultation (n=214)CBTa competency, knowledge acquisition, and learner retentione-Learning with peer-led consultation was not inferior to in-person training with expert-led consultation in increasing clinical competency. No differences between knowledge acquisition between groups. Trainees in the web-based training program were less likely to complete the training (P<.001)
    Kuhn and Hugo (2017) [59]Psychiatry residentsProlonged exposure to posttraumatic stress disorderA 7- to 8-hour 13 web-based modules including step-by-step instructions, demo videos of expert clinicians, printable scripts, pre- and posttests, and a “common questions” sectionN=12Trainee impressionsThe web-based training used in a blended training program may help address the common problems in traditional training models. The first impressions of the program support its viability, practicality, and efficiency
    Kobak et al (2017) [60]Mental health cliniciansInterpersonal therapy

    		A 4-hour web-based training with animations, graphical illustrations, clinical vignettes, and various exercisesN=26

    		Knowledge acquisitionSignificant increase in knowledge of interpersonal therapy (P<.001)
    Fairburn et al (2017) [61]Mental health cliniciansCognitive Behavioral Therapy for Eating DisordersA 9-hour web-based training providing information on how to implement CBT-Eb with videos, handouts, learning exercises, and tests of knowledge with feedback. A separate digital library of detailed materials of therapy implementation with various subgroupsN=80

    		Competence in CBT-E

    		A significant increase in competence scores after completing the training (P<.001). Of the trainees 42.5% reached a predetermined cutoff point indicating good competence
    Curran et al (2015) [62]Counsellors working with individuals with substance use disordersManualized CBT group treatment for depressionA 12- to 16-hour media-rich and interactive digital trainingN=7

    		Feasibility of self-paced digital training in replacement of in-person training. Observed barriers and facilitators, recommendations for improvement, and concerns of implementationThe main barrier concerning the completion of the training was the lack of protected time. The main facilitators included positive attitudes toward the training, supervisor support, counselor dedication, and beliefs supporting providing services for depression
    Hickey et al (2015) [63]Psychiatry residentsTwo core topics in IS-TDPcTwo asynchronous e-learning modules of “the metapsychology of the unconscious” and “Davanloo’s technique of IS-TDP”e-Learning (n=19), in-person training (n=19)Knowledge acquisitionSignificant improvement in knowledge levels during training in both topics (P<.01). No differences between knowledge acquisition between web-based training and lectures (P<.05)
    Samuelson et al (2014) [64]Primary care providersDetection, assessment, initial management, and referral of patients with PTSDd70-minute web-based training including multimedia didactic content, case presentations, and clinical vignettesN=73Knowledge acquisition, comfort regarding PTSD-related skills, and usage in clinical practiceSignificant increase in PTSD-related knowledge (P<.001) and significantly greater comfort regarding skills compared with pretraining (P<.001). 47% reported using the training content in clinical practice
    Kobak et al (2013) [65]Social workers, psychologists and social work, and psychology studentsCBT for anxiety disordersA 5-hour 10-module web-based training with interactive exercises, animations, graphical illustrations, and expert demonstration videosN=39Knowledge acquisition and satisfactionA significant increase in knowledge in CBT (P<.001) and trainees were satisfied in both the content and the technical aspects of the web-based training
    Martino et al (2011) [66]Counsellors working with substance use disordersMotivational interviewingFour 1-hour modules with instructions for self-practice with reflections on a discussion board managed by a course instructorN=26Adherence and competence in motivational interviewingThe web-based training was a part of a stepwise training program with workshops and supervision after the web-based training which were offered only if the learner showed inadequate competence after the web-based training. The participants who performed adequately after the web-based training did not improve in subsequent training. Participants who showed inadequate competence after the web-based training continued to improve in subsequent training

    aCBT: cognitive behavioral therapy.

    bCBT-E: Cognitive Behavioral Therapy for Eating Disorders.

    cIS-TDP: intensive short-term dynamic psychotherapy.

    dPTSD: posttraumatic stress disorder.

    Risk of Bias

    Risk of bias was assessed both for the randomized and nonrandomized studies included in the review using the Cochrane risk of bias tools (RoB 2 and ROBINS-I). The risk of bias analysis was carried out by the first and last authors (KM, SES). The authors agreed upon the used tools and familiarized themselves with the logic behind them. The first author did the risk of bias analysis and classification. Then the last author examined the classification to identify potential discrepancies. No discrepancies occurred. In the review, 13 out of the 17 randomized studies, which corresponds to 76%, have been categorized as exhibiting a low risk of bias. One of the studies (6%) showed some concerns regarding the risk of bias. Three of the studies (17%) were flagged for having a high risk of bias. One of these studies had a source of bias from the randomization process and the 2 others have a potential source of bias from a large amount of missing outcome data. These results can be viewed in Figure 2. The ROBINS-I classification for risk of bias in nonrandomized studies included in the review can be viewed in Multimedia Appendix 1.

    ‎
    Figure 2. Risk of bias assessment for the randomized trials included in the review using the Cochrane risk of bias tool (RoB 2) [33-49].

    Among the nonrandomized studies included in the review, 16 out of 17 (94%) were flagged for having either a serious or critical risk of bias. The source of bias in all these studies was, at the very least, inadequate control of potential confounding variables. One study [58] was flagged at being only at moderate risk of bias.

    Learning Outcomes According to the Kirkpatrick Model Subcategories

    Kirkpatrick: Reaction

    Out of the 34 articles included in the review, 7 (21%) presented findings related to the “Reaction” level of the Kirkpatrick model. These findings included such concepts as satisfaction, acceptance, relevance, first impressions, viability, practicality, and feasibility conceptualized as observed barriers and facilitators regarding implementation of the trained content. E-Learning was associated with positive changes in these outcomes in training programs for dialectical behavior therapy (DBT) [48], CBT for psychosis [52], therapeutic crisis management [55], CBT for depression [62], prolonged exposure to posttraumatic stress disorder (PTSD) [59], and CBT for anxiety disorders [65].

    Kirkpatrick: Learning

    Out of the studies included in the review, 30 out of 34 (88%) presented findings related to the “Learning” level of the Kirkpatrick model. These included concepts related to acquisition of knowledge and skills, as well changes in attitudes and the level of self-efficacy. E-Learning was associated with positive changes in these outcomes in training programs for perinatal mental health assessment and empathic communication [33], evidence-based intervention in perinatal depression [34], CBT for insomnia [35], trauma-focused CBT [36], suicide prevention practices [38], behavioral activation [40], DBT [42,47,48], exposure therapy for anxiety disorders [43], CBT skills for traumatic stress [44], CBT [45,46,58], CBT for addictions [49], skills in engaging men in therapy [51], CBT addressed for patient with acute cancer [53], implementation intentions [54], therapeutic crisis management techniques [55], suicide prevention [67], sensory modulation approaches [57], interpersonal therapy [60], CBT for eating disorders [61], short-term psychodynamic therapy [63], management of PTSD [55], and motivational interviewing [66].

    Kirkpatrick: Behavior

    Out of the studies included in the review, 7 out of 34 (21%) reported results that can be classified to the “Behavior” level of the Kirkpatrick model which in all cases was reported as application of the trained content in clinical practice. E-Learning was generally associated with positive changes in these outcomes in training programs for interpersonal and social rhythm therapy for bipolar disorder [41], DBT [42], exposure therapy for anxiety disorders [43], CBT [46], and implementation intentions [54].

    Kirkpatrick: Results

    No studies included in this review display results that could be classified in the “Results” level of the Kirkpatrick model.

    Comparison Between e-Learning and Conventional Training Methods

    Thirteen out of 34 (38%) studies, in which 9 were randomized trials [34,35,39-42,46,48,49] and 4 were nonrandomized [52,55,58,63], reported a comparison between a web-based training and training carried out by conventional methods. Of the 13 studies examined, 2 (15%) [42,48] reported results suggesting superiority (related to knowledge acquisition outcomes) for e-learning compared with conventional training. Meanwhile, 3 (23%) [40,42,52] indicated inferiority (concerning outcomes related to skills acquisition, satisfaction, self-efficacy, motivation, opportunity for active participation, and trainee-assessed professional benefit). In addition, 12 (92%) [34,35,39,41,42,46,48,49,52,55,58,63] found no significant difference between e-learning and conventional training methods, reporting equivalence (in terms of outcomes related to knowledge acquisition, skills acquisition, clinical application of the methods trained, self-efficacy, satisfaction, and acceptance rates) at posttraining evaluations. It is worth noting that studies reporting multiple outcomes could be included in more than 1 category.

    Eight out of these 13 studies included the necessary figures to calculate Hedges g. The first authors of the remaining studies were contacted, but none provided the figures or raw data within an acceptable time window of 1 month. The overall mean effect size from all comparisons within the 8 studies (n=30) was 0.01. A forest plot displaying effect sizes and their corresponding 95% CIs is shown in Figure 3. The results are organized and presented in accordance with the Kirkpatrick model categorization.

    ‎
    Figure 3. Forest plot of Hedges g effects sizes depicting the difference between e-learning and conventional training methods in Kirkpatrick levels 1-3 [34,39,42,46,48,52,55,58]. CBT: cognitive behavioral therapy; DBT: dialectical behavior therapy; EBM: evidence-based method.

    Principal Results

    A systematic literature review of e-learning in the context of psychotherapy training was conducted following the PRISMA guidelines. E-Learning was positively associated with learning outcomes in terms of knowledge and skills acquisition as well as implementing learned skills to practice according to the Kirkpatrick levels 1-3. There were no apparent differences between e-learning and conventional training methods. None of the studies examined the impact of training on patient outcomes, and the research demonstrated significant heterogeneity.

    This is the first study to exclusively assess the level of research regarding e-learning within this domain. The review included 34 studies conducted between 2008 and 2022, encompassing a mix of both randomized and nonrandomized research. Up to this point, findings on training outcomes have been reported across various outcomes and the level of evidence is still emerging. The target group, training content, a description of e-learning, sample sizes, primary outcomes, and a summary of results were provided from each article included in the review. Most of the randomized studies were linked with a low risk of bias, whereas nonrandomized studies exhibited a severe risk of bias. The risk of bias in the majority of the nonrandomized studies stemmed from limited control of confounding variables. The execution method of e-learning, the measurement methods used in training outcome assessments, and the trainees were highly heterogenous across the included studies. The high amount of heterogeneity in various aspects of the included literature and small sample sizes in many of the included studies create challenges in synthesizing the findings and drawing clear conclusions. Therefore, it is necessary to process the literature using a higher-lever categorization in learning outcomes (such as the Kirkpatrick model).

    e-Learning within psychotherapy training showed positive associations with several aspects: trainee satisfaction, enhanced learning outcomes—including knowledge and skills acquisition, and the application of trained methods into practice, as observed in within-trainee designs. These findings align with levels 1-3 of the Kirkpatrick model. No outcomes related to patient outcomes were reported (Kirkpatrick level 4). The results of this review point in the direction of equivalence between e-learning and conventional training methods such as lectures and workshops. Training outcomes including trainee satisfaction, knowledge, and skill acquisition, and application of trained content in clinical work were comparable between e-learning and conventional training. The overall effect size signifying this difference was 0.01, indicating no difference.

    In the only study that reported superiority of conventional training methods in skills acquisition e-learning involved solely watching previously recorded lectures rather than actively interacting with content specifically designed as an e-learning program [40]. In the other studies, where the inferiority of e-learning was presented as a result with a more modern approach on e-learning, the inferiority was in the context of trainee satisfaction, self-efficacy, and motivation [42,52]. E-Learning was reported to be superior compared with conventional training only in the context of knowledge acquisition when training DBT [42,48].

    The trainees and the clinical contexts of the trainees varied heavily between included studies. Furthermore, the operationalization of the learning outcomes and the associated measurement methods were not consistent and included both subjective self-ratings of knowledge and competence and more objective measures (eg, quizzes and expert-rated competence scores). In most of the studies, the training content included CBT-based methods, and in some studies interpersonal therapy and short-term psychodynamic therapy. Most of the e-learning solutions described were similar and included case vignettes, demonstrations, multimedia didactic content, and various assignments.

    Strengths and Limitations

    This review has many strengths. These include following the PRISMA guidelines, giving an exhaustive look in the current literature, and that the results of research comparing e-learning and conventional training methods were uniformly aggregated behind a single statistic. As anticipated, studies identified in this review had high heterogeneity of learning methods, therapy content, and outcomes, making systematic comparison difficult. We tried to control this heterogeneity by using the Kirkpatrick model of training evaluation, summarizing the results by calculating effect sizes, and requesting original data from authors to do that. However, it seems that heterogeneity is a feature of this field of study and cannot be easily overcome with post hoc methods; a more uniform research methodology would be needed in future studies.

    This systematic review also has limitations. First, the review did not include gray literature (materials and research produced outside of academic publishing) or literature beyond the scope of the conducted search, which may include information relevant to this study. Second, in addition to the risk of bias assessment, the quality of the research included was not systematically assessed (eg, publication bias). Third, the search method yielded a substantial amount of literature not relevant to the study, which makes the replication of this review challenging. This is mainly attributed to the overlap in vocabulary between literature related to training and psychosocial interventions themselves. This issue also touches upon a possible methodological limitation in our screening process, namely, that the initial screening phases were conducted by a sole reviewer. The purpose of this preliminary screening was to promptly identify and eliminate studies that were unmistakably irrelevant to our review, with a large proportion of such studies investigating digital psychosocial interventions (eg, internet-based cognitive behavioral therapy), a testament to the prevalent overlap in terminologies used. To ensure a balanced review, the authors applied a statistical approach to select a representative sample of studies for evaluation by the second author. We posit that the inclusion of 2 independent raters from the outset, as opposed to 1, during the initial screening phases might have had a negligible effect on the outcomes, considering that all studies with any element of ambiguity were carried forward for full review. Nonetheless, we acknowledge that our approach may be more susceptible to bias compared with methodologies that use multiple independent reviewers for the assessment of all studies. We consider this as a potential limitation and agree that the use of dual raters throughout the process might augment the robustness against bias.

    Conclusions

    Based on this literature review, we conclude that positive learning outcomes are generally associated with various e-learning programs in psychotherapy training including trainee satisfaction, knowledge and skill acquisition, and in application of trained content in clinical practice. Learning outcomes are generally equivalent between e-learning and conventional training methods. This equivalence is interpreted as a supportive finding for e-learning, considering the other benefits of e-learning, such as scalability and consistency of quality. While more research is warranted, and research methodology needs to be more consistent in the description of e-learning solutions and in learning outcome assessment, it seems advisable to use e-learning as a part of psychotherapy training curriculums.

    e-Learning creates an opportunity to scale up training in large and even in nationwide settings. This, in turn, can significantly improve equal and sufficient access to psychotherapy in public health care settings, which are impossible with conventional and unscalable training methods [61]. This can be seen, for example, within the study by Wilkerson et al [50], where it was estimated that 2500 individuals will annually complete an e-learning program in Cognitive Behavioral Therapy for Insomnia, a scale the authors declared as “massive dissemination.” With centralized digital solutions, e-learning might prove itself as the more cost-effective solution than conventional training methods while ensuring quality in training as many learning outcomes are comparable between these methods.

    As the literature is still emerging and current corpus is displaying a high amount of heterogeneity, small sample sizes, and inconsistent description of e-learning solutions and measurement methods in learning outcomes, it would be advisable to use e-learning as a part of blended training programs. By comparison with e-learning results in health education in general, a blended training model might even be the most optimal way to implement e-learning. E-Learning is generally recognized as an opportunity to improve the availability of training programs by eliminating constraints set by time and geographical location and also a way to increase the quality of training programs by shifting the focus of the surrounding pedagogics toward more learning-centric direction. Knowledge acquisition can be supported through e-learning, but refining skills and ensuring competencies may require synchronous interaction with a trainer to reach optimal learning outcomes [67].

    In psychotherapy training, a shift toward minimizing the didactic approach of trainers by facilitating the active role of the trainee in knowledge acquisition is warranted. E-Learning viewed as a pedagogy rather than technology may be an optimal solution in implementing flipped classroom and other activating learning methods. Within these methods, synchronous learning events are held with trainees who have already familiarized themselves with the necessary materials and focus on honing therapeutic and reflective skills and addressing student motivation. Future research needs to focus on replicating the effects of e-learning in trainee satisfaction, learning and behavior, and on studying the effectiveness and cost-effectiveness of training (e-learning or conventional methods) on patient outcomes. Research is especially needed to further determine the optimal way to combine e-learning and conventional training methods. Furthermore, to reach the full potential of e-learning in the context of psychotherapy training, international collaboration is warranted to bring consistency on how e-learning systems are developed, described, and assessed.

    Acknowledgments

    KM conceived and designed the study with the advice from SES. SES, EH, and SIS reviewed, analyzed, and interpreted the data and contributed to drafting, revising, and finalizing the manuscript. All authors read and approved the final version of this manuscript.

    Data Availability

    The data extracted from the included studies are available upon request from the first author.

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    ROBINS-I (Risk of Bias in Non-randomized Studies of Interventions) classification for risk of bias in nonrandomized studies included in the review.

    DOCX File , 255 KB
    Multimedia Appendix 2

    PRSIMA 2020 checklist.

    PDF File (Adobe PDF File), 128 KB
    1. Vigo D, Thornicroft G, Atun R. Estimating the true global burden of mental illness. Lancet Psychiatry. 2016;3(2):171-178. [CrossRef] [Medline]
    2. Barth J, Munder T, Gerger H, Nüesch E, Trelle S, Znoj H, et al. Comparative efficacy of seven psychotherapeutic interventions for patients with depression: a network meta-analysis. Focus (Am Psychiatr Publ). 2016;14(2):229-243. [FREE Full text] [CrossRef] [Medline]
    3. Cuijpers P, Miguel C, Harrer M, Plessen CY, Ciharova M, Ebert D, et al. Cognitive behavior therapy vs. control conditions, other psychotherapies, pharmacotherapies and combined treatment for depression: a comprehensive meta-analysis including 409 trials with 52,702 patients. World Psychiatry. 2023;22(1):105-115. [FREE Full text] [CrossRef] [Medline]
    4. van Dis EAM, van Veen SC, Hagenaars MA, Batelaan NM, Bockting CLH, van den Heuvel RM, et al. Long-term outcomes of cognitive behavioral therapy for anxiety-related disorders: a systematic review and meta-analysis. JAMA Psychiatry. 2020;77(3):265-273. [FREE Full text] [CrossRef] [Medline]
    5. Harvey AG, Gumport NB. Evidence-based psychological treatments for mental disorders: modifiable barriers to access and possible solutions. Behav Res Ther. 2015;68:1-12. [FREE Full text] [CrossRef] [Medline]
    6. Stewart RE, Chambless DL, Baron J. Theoretical and practical barriers to practitioners' willingness to seek training in empirically supported treatments. J Clin Psychol. 2012;68(1):8-23. [FREE Full text] [CrossRef] [Medline]
    7. Knox S, Hill C. Training and supervision in psychotherapy: what we know and where we need to go. In: Bergin and Garfield's Handbook of Psychotherapy and Behavior Change. Hoboken, NJ. John Wiley & Sons Inc; 2021:327-349.
    8. Bennett-Levy J, Perry H. The promise of online cognitive behavioural therapy training for rural and remote mental health professionals. Australas Psychiatry. 2009;17 Suppl 1:S121-S124. [CrossRef] [Medline]
    9. Khanna MS, Kendall PC. Bringing technology to training: web-based therapist training to promote the development of competent cognitive-behavioral therapists. Cogn Behav Pract. 2015;22(3):291-301. [CrossRef]
    10. Fairburn CG, Patel V. The impact of digital technology on psychological treatments and their dissemination. Behav Res Ther. 2017;88:19-25. [FREE Full text] [CrossRef] [Medline]
    11. Guri-Rosenblit S, Gros B. E-learning: confusing terminology, research gaps and inherent challenges. IJEDE Internet. Mar 3, 2011. URL: https://www.ijede.ca/index.php/jde/article/view/729 [accessed 2023-10-17]
    12. Choudhury S, Pattnaik S. Emerging themes in e-learning: a review from the stakeholders' perspective. Comput Educ. 2020;144:103657. [CrossRef]
    13. Moore JL, Dickson-Deane C, Galyen K. e-Learning, online learning, and distance learning environments: Are they the same? Internet Higher Educ. 2011;14(2):129-135. [CrossRef]
    14. Guri-Rosenblit S. ‘Distance education’ and ‘e-learning’: not the same thing. High Educ. 2005;49(4):467-493. [CrossRef]
    15. Hew KF, Lo CK. Flipped classroom improves student learning in health professions education: a meta-analysis. BMC Med Educ. 2018;18(1):38-41. [FREE Full text] [CrossRef] [Medline]
    16. Telford M, Senior E. Healthcare students' experiences when integrating e-learning and flipped classroom instructional approaches. Br J Nurs. 2017;26(11):617-622. [CrossRef] [Medline]
    17. Ruggeri K, Farrington C, Brayne C. A global model for effective use and evaluation of e-learning in health. Telemed J E Health. 2013;19(4):312-321. [FREE Full text] [CrossRef] [Medline]
    18. Rief W. Moving from tradition-based to competence-based psychotherapy. Evid Based Ment Health. 2021;24(3):115-120. [FREE Full text] [CrossRef] [Medline]
    19. Hickey C, McAleer S. Competence in psychotherapy: the role of e-learning. Acad Psychiatry. 2017;41(1):20-23. [CrossRef] [Medline]
    20. Meinert E, Eerens J, Banks C, Maloney S, Rivers G, Ilic D, et al. Exploring the cost of elearning in health professions education: scoping review. JMIR Med Educ. 2021;7(1):e13681. [FREE Full text] [CrossRef] [Medline]
    21. Means B, Toyama Y, Murphy R, Baki M. The effectiveness of online and blended learning: a meta-analysis of the empirical literature. Teachers College Record. 2013;115(3):1-47. [CrossRef]
    22. Yuwono KT, Sujono HD. The effectiveness of e-learning: a meta-analysis. J Phys Conf Ser. 2018;1140(1):012024.
    23. Fontaine G, Cossette S, Maheu-Cadotte M, Mailhot T, Deschênes M-F, Mathieu-Dupuis G, et al. Efficacy of adaptive e-learning for health professionals and students: a systematic review and meta-analysis. BMJ Open. 2019;9(8):e025252. [FREE Full text] [CrossRef] [Medline]
    24. Jackson CB, Quetsch LB, Brabson LA, Herschell AD. Web-Based training methods for behavioral health providers: a systematic review. Adm Policy Ment Health. 2018;45(4):587-610. [FREE Full text] [CrossRef] [Medline]
    25. Li K. The evolution of open learning: a review of the transition from pre-e-learning to the era of e-learning. Knowledge Manag E-Learn. 2018;10(4):408-425.
    26. Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med. 2009;151(4):264-269. [FREE Full text] [CrossRef] [Medline]
    27. Sterne JAC, Savović J, Page MJ, Elbers RG, Blencowe NS, Boutron I, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ. 2019;366:l4898. [FREE Full text] [CrossRef] [Medline]
    28. Sterne JA, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355:i4919. [FREE Full text] [CrossRef] [Medline]
    29. Smidt A, Balandin S, Sigafoos J, Reed VA. The Kirkpatrick model: a useful tool for evaluating training outcomes. J Intellect Dev Disabil. 2009;34(3):266-274. [CrossRef] [Medline]
    30. Hedges L. Distribution theory for Glass's estimator of effect size and related estimators. J Educ Stat. 1981;6(2):107-128. [FREE Full text] [CrossRef]
    31. Cohen J. Statistical Power Analysis for the Behavioral Sciences, Second Edition. Cambridge, MA. Academic Press; 1989.
    32. McHugh ML. Interrater reliability: the kappa statistic. Biochem Med (Zagreb). 2012;22(3):276-282. [FREE Full text] [Medline]
    33. Shinohara E, Ohashi Y, Hada A, Usui Y. Effects of 1-day e-learning education on perinatal psychological support skills among midwives and perinatal healthcare workers in Japan: a randomised controlled study. BMC Psychol. 2022;10(1):133. [FREE Full text] [CrossRef] [Medline]
    34. Nisar A, Yin J, Nan Y, Luo H, Han D, Yang L, et al. Standardising training of nurses in an evidence-based psychosocial intervention for perinatal depression: randomized trial of electronic vs. face-to-face training in China. Int J Environ Res Public Health. 2022;19(7):4094. [FREE Full text] [CrossRef] [Medline]
    35. Taylor D, Dietch J, Pruiksma K, Calhoun C, Milanak M, Wardle-Pinkston S, et al. Developing and testing a web-based provider training for cognitive behavioral therapy of insomnia. Mil Med. 2021;186(Suppl 1):230-238. [CrossRef] [Medline]
    36. Sansen LM, Saupe LB, Steidl A, Fegert JM, Hoffmann U, Neuner F. Development and randomized-controlled evaluation of a web-based training in evidence-based trauma therapy. Prof Psych Res Pract. 2020;51(2):115-124. [CrossRef]
    37. Sansen LM, Saupe LB, Steidl A, Fegert JM, Hoffmann U, Neuner F. Daring to process the trauma: using a web-based training to reduce psychotherapists' fears and reservations around implementing trauma-focused therapy. Eur J Psychotraumatol. 2019;10(1):1696590. [FREE Full text] [CrossRef] [Medline]
    38. Kullberg MJ, Mouthaan J, Schoorl M, de Beurs D, Kenter RMF, Kerkhof AJ. E-Learning to improve suicide prevention practice skills among undergraduate psychology students: randomized controlled trial. JMIR Ment Health. 2020;7(1):e14623. [FREE Full text] [CrossRef] [Medline]
    39. Rahman A, Akhtar P, Hamdani SU, Atif N, Nazir H, Uddin I, et al. Using technology to scale-up training and supervision of community health workers in the psychosocial management of perinatal depression: a non-inferiority, randomized controlled trial. Glob Ment Health (Camb). 2019;6:e8. [FREE Full text] [CrossRef] [Medline]
    40. Puspitasari AJ, Kanter JW, Busch AM, Leonard R, Dunsiger S, Cahill S, et al. A randomized controlled trial of an online, modular, active learning training program for behavioral activation for depression. J Consult Clin Psychol. 2017;85(8):814-825. [CrossRef] [Medline]
    41. Stein BD, Celedonia KL, Swartz HA, DeRosier ME, Sorbero MJ, Brindley RA, et al. Implementing a web-based intervention to train community clinicians in an evidence-based psychotherapy: a pilot study. Psychiatr Serv. 2015;66(9):988-991. [FREE Full text] [CrossRef] [Medline]
    42. Dimeff LA, Harned MS, Woodcock EA, Skutch JM, Koerner K, Linehan MM. Investigating bang for your training buck: a randomized controlled trial comparing three methods of training clinicians in two core strategies of dialectical behavior therapy. Behav Ther. 2015;46(3):283-295. [CrossRef] [Medline]
    43. Harned MS, Dimeff LA, Woodcock EA, Kelly T, Zavertnik J, Contreras I, et al. Exposing clinicians to exposure: a randomized controlled dissemination trial of exposure therapy for anxiety disorders. Behav Ther. 2014;45(6):731-744. [FREE Full text] [CrossRef] [Medline]
    44. Ruzek JI, Rosen RC, Garvert DW, Smith LD, Sears KC, Marceau L, et al. Online self-administered training of PTSD treatment providers in cognitive-behavioral intervention skills: results of a randomized controlled trial. J Trauma Stress. 2014;27(6):703-711. [CrossRef] [Medline]
    45. Larson MJ, Amodeo M, Locastro JS, Muroff J, Smith L, Gerstenberger E. Randomized trial of web-based training to promote counselor use of cognitive behavioral therapy skills in client sessions. Subst Abus. 2013;34(2):179-187. [FREE Full text] [CrossRef] [Medline]
    46. Bennett‐levy J, Hawkins R, Perry H, Cromarty P, Mills J. Online cognitive behavioural therapy training for therapists: outcomes, acceptability, and impact of support. Aust Psychol. 2020;47(3):174-182. [CrossRef]
    47. Dimeff LA, Woodcock EA, Harned MS, Beadnell B. Can dialectical behavior therapy be learned in highly structured learning environments? Results from a randomized controlled dissemination trial. Behav Ther. 2011;42(2):263-275. [CrossRef] [Medline]
    48. Dimeff LA, Koerner K, Woodcock EA, Beadnell B, Brown MZ, Skutch JM, et al. Which training method works best? A randomized controlled trial comparing three methods of training clinicians in dialectical behavior therapy skills. Behav Res Ther. 2009;47(11):921-930. [CrossRef] [Medline]
    49. Weingardt KR, Villafranca SW, Levin C. Technology-based training in cognitive behavioral therapy for substance abuse counselors. Subst Abus. 2006;27(3):19-25. [CrossRef] [Medline]
    50. Wilkerson AK, Wardle-Pinkston S, Dietch JR, Pruiksma KE, Simmons RO, Bunnell BE, et al. Web-based provider training of cognitive behavioral therapy of insomnia: engagement rates, knowledge acquisition, and provider acceptability. Cogn Behav Ther. 2022;51(4):343-352. [CrossRef] [Medline]
    51. Seidler ZE, Wilson MJ, Toogood N, Oliffe JL, Kealy D, Ogrodniczuk JS, et al. Pilot evaluation of the men in mind training program for mental health practitioners. Psychology Men Masculinities. 2022;23(2):257-264. [CrossRef]
    52. Soll D, Fuchs R, Mehl S. Teaching cognitive behavior therapy to postgraduate health care professionals in times of Covid 19—an asynchronous blended learning environment proved to be non-inferior to in-person training. Front Psychol. 2021;12:657234. [FREE Full text] [CrossRef] [Medline]
    53. Applebaum A, Walsh LE, Polacek LC, Benvengo S, Levin T. Acute cancer cognitive therapy online training program: feasibility and impact on clinician knowledge uptake. J Cancer Educ. 2021;36(5):1081-1085. [FREE Full text] [CrossRef] [Medline]
    54. Gonzalez Salas Duhne P, Horan AJ, Ross C, Webb TL, Hardy GE. Assessing and promoting the use of implementation intentions in clinical practice. Soc Sci Med. 2020;265:113490. [CrossRef] [Medline]
    55. Graber J. Comparison of mental health nursing student academic achievement and satisfaction: classroom versus online education in teaching therapeutic crisis management techniques. Issues Ment Health Nurs. 2019;40(3):247-251. [CrossRef] [Medline]
    56. Cramer RJ, Long MM, Gordon E, Zapf PA. Preliminary effectiveness of an online-mediated competency-based suicide prevention training program. Prof Psychol Res Pract. 2019;50(6):395-406. [CrossRef]
    57. Meredith P, Yeates H, Greaves A, Taylor M, Slattery M, Charters M, et al. Preparing mental health professionals for new directions in mental health practice: Evaluating the sensory approaches e-learning training package. Int J Ment Health Nurs. 2018;27(1):106-115. [CrossRef] [Medline]
    58. German RE, Adler A, Frankel SA, Stirman SW, Pinedo P, Evans AC, et al. Testing a web-based, trained-peer model to build capacity for evidence-based practices in community mental health systems. Psychiatr Serv. 2018;69(3):286-292. [CrossRef] [Medline]
    59. Kuhn E, Hugo E. Technology-based blended learning to facilitate psychiatry resident training in prolonged exposure (PE) therapy for PTSD. Acad Psychiatry. 2017;41(1):121-124. [CrossRef] [Medline]
    60. Kobak KA, Lipsitz JD, Markowitz JC, Bleiberg KL. Web-based therapist training in interpersonal psychotherapy for depression: pilot study. J Med Internet Res. 2017;19(7):e257. [FREE Full text] [CrossRef] [Medline]
    61. Fairburn CG, Allen E, Bailey-Straebler S, O'Connor ME, Cooper Z. Scaling up psychological treatments: a countrywide test of the online training of therapists. J Med Internet Res. 2017;19(6):e214. [FREE Full text] [CrossRef] [Medline]
    62. Curran GM, Woo SM, Hepner KA, Lai WP, Kramer TL, Drummond KL, et al. Training substance use disorder counselors in cognitive behavioral therapy for depression: development and initial exploration of an online training program. J Subst Abuse Treat. 2015;58:33-42. [CrossRef] [Medline]
    63. Hickey C, McAleer S, Khalili D. E-learning and traditional approaches in psychotherapy education: comparison. Arch Psychiatry Psychother. 2015;4:48-52. [FREE Full text] [CrossRef]
    64. Samuelson KW, Koenig CJ, McCamish N, Choucroun G, Tarasovsky G, Bertenthal D, et al. Web-based PTSD training for primary care providers: a pilot study. Psychol Serv. 2014;11(2):153-161. [CrossRef] [Medline]
    65. Kobak KA, Craske MG, Rose RD, Wolitsky-Taylor K. Web-based therapist training on cognitive behavior therapy for anxiety disorders: a pilot study. Psychotherapy (Chic). 2013;50(2):235-247. [FREE Full text] [CrossRef] [Medline]
    66. Martino S, Canning-Ball M, Carroll KM, Rounsaville BJ. A criterion-based stepwise approach for training counselors in motivational interviewing. J Subst Abuse Treat. 2011;40(4):357-365. [FREE Full text] [CrossRef] [Medline]
    67. Regmi K, Jones L. A systematic review of the factors—enablers and barriers—affecting e-learning in health sciences education. BMC Med Educ. 2020;20(1):91. [FREE Full text] [CrossRef] [Medline]

    ‎
    CBT: cognitive behavioral therapy
    DBT: dialectical behavior therapy
    PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses
    PTSD: posttraumatic stress disorder
    ROBINS-I: Risk of Bias in Non-randomized Studies of Interventions

    Edited by T de Azevedo Cardoso; submitted 11.11.23; peer-reviewed by R Philippe, S Payne; comments to author 10.04.24; revised version received 22.04.24; accepted 21.05.24; published 29.07.24.

    Copyright

    ©Kasperi Mikkonen, Eeva-Eerika Helminen, Samuli I Saarni, Suoma E Saarni. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 29.07.2024.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research (ISSN 1438-8871), is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.