This paper is in the following e-collection/theme issue:

Digital Health Reviews (1289) Undergraduate Education for Allied Health Professionals (40) Undergraduate Education for Future Doctors (86) Undergraduate Education for Future Healthcare Professionals (32) Graduate and Postgraduate Education for Health Professionals (97) Models and Frameworks in Medical Education (30)

Published on in Vol 26 (2024)

This is a member publication of Imperial College London (Jisc)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/57760, first published .
Spaced Digital Education for Health Professionals: Systematic Review and Meta-Analysis

Spaced Digital Education for Health Professionals: Systematic Review and Meta-Analysis

Spaced Digital Education for Health Professionals: Systematic Review and Meta-Analysis

Authors of this article:

Laura Martinengo1, 2 Author Orcid Image ;   Matthew Song Peng Ng3 Author Orcid Image ;   Tony De Rong Ng3 Author Orcid Image ;   Yi-Ian Ang3 Author Orcid Image ;   Ahmad Ishqi Jabir1, 4 Author Orcid Image ;   Bhone Myint Kyaw5 Author Orcid Image ;   Lorainne Tudor Car1, 6 Author Orcid Image
Article Authors Cited by Tweetations (1) Metrics

    Review

    1Lee Kong Chian School of Medicine, Nanyang Technological University Singapore, Singapore, Singapore

    2Centre for Behavioural and Implementation Sciences Interventions, Yong Loo Lin School of Medicine, National University of Singapore, Singapore, Singapore

    3Yong Loo Lin School of Medicine, National University of Singapore, Singapore, Singapore

    4Future Health Technologies, Singapore-ETH Centre, Campus for Research Excellence And Technological Enterprise, Singapore, Singapore

    5Nanyang Technological University Singapore, Singapore, Singapore

    6Department of Primary Care and Public Health, School of Public Health, Imperial College London, London, United Kingdom

    Corresponding Author:

    Lorainne Tudor Car, MD, PhD

    Lee Kong Chian School of Medicine

    Nanyang Technological University Singapore

    11 Mandalay Rd

    Level 18, Clinical Sciences Builiding

    Singapore, 308232

    Singapore

    Phone: 65 69041258

    Email: l.tudor.car@imperial.ac.uk


    Background: Spaced digital education applies digital tools to deliver educational content via multiple, repeated learning sessions separated by prespecified time intervals. Spaced digital education appears to promote acquisition and long-term retention of knowledge, skills, and change in clinical behavior.

    Objective: The aim of this review was to assess the effectiveness of spaced digital education in improving pre- and postregistration health care professionals’ knowledge, skills, attitudes, satisfaction, and change in clinical behavior.

    Methods: This review followed Cochrane’s methodology and PRISMA (Preferred Reporting Items of Systematic Reviews and Meta-Analyses) reporting guidelines. We searched MEDLINE, Embase, Web of Science, ERIC, PsycINFO, CINAHL, CENTRAL, and ProQuest Dissertation and Theses databases from January 1990 to February 2023. We included randomized controlled trials (RCTs), cluster RCTs, and quasi-RCTs comparing spaced digital education with nonspaced education, spaced nondigital education, traditional learning, or no intervention for pre- or postregistration health care professionals. Study selection, data extraction, study quality, and certainty of evidence were assessed by 2 independent reviewers. Meta-analyses were conducted using random effect models.

    Results: We included 23 studies evaluating spaced online education (n=17, 74%) or spaced digital simulation (n=6, 26%) interventions. Most studies assessed 1 or 2 outcomes, including knowledge (n=15, 65%), skills (n=9, 39%), attitudes (n=8, 35%), clinical behavior change (n=8, 35%), and satisfaction (n=7, 30%). Most studies had an unclear or a high risk of bias (n=19, 83%). Spaced online education was superior to massed online education for postintervention knowledge (n=9, 39%; standardized mean difference [SMD] 0.32, 95% CI 0.13-0.51, I2=66%, moderate certainty of evidence). Spaced online education (n=3, 13%) was superior to massed online education (n=2, 9%) and no intervention (n=1, 4%; SMD 0.67, 95% CI 0.43-0.91, I2=5%, moderate certainty of evidence) for postintervention clinical behavior change. Spaced digital simulation was superior to massed simulation for postintervention surgical skills (n=2, 9%; SMD 1.15, 95% CI 0.34-1.96, I2=74%, low certainty of evidence). Spaced digital education positively impacted confidence and satisfaction with the intervention.

    Conclusions: Spaced digital education is effective in improving knowledge, particularly in substantially improving surgical skills and promoting clinical behavior change in pre- and postregistration health care professionals. Our findings support the use of spaced digital education interventions in undergraduate and postgraduate health profession education. Trial Registration: PROSPERO CRD42021241969;

    Trial Registration: PROSPERO CRD42021241969; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=241969

    J Med Internet Res 2024;26:e57760

    doi:10.2196/57760

    Keywords

    digital education; e-learning; spaced education; spaced learning; spaced digital education; spaced simulation; health care professionals; continuous medical education; systematic review; meta-analysis 


    Background

    Health care service delivery depends on access to a competent workforce. The World Health Organization (WHO) has long warned of severe global shortages of health care workers, particularly in low- and middle-income countries [1,2]. Long working hours, low wages, workplace stress, and lack of professional development opportunities [3,4] appear to drive the health care workforce turnover [3]. An important strategy to increase recruitment and retention of health care workers is providing high-quality education in an affordable, effective, and sustainable way [5].

    Digital education is “the act of teaching and learning by means of digital technologies” [6]. Digital education may promote continuous professional development and improve health care workers’ competencies by offering convenient and adaptable learning tools that can be accessed at any place and time [6-10]. A recent series of systematic reviews and meta-analyses on different digital education modalities [6,11-18] showed that digital education is well received and noninferior to traditional education.

    Spaced Education

    Spaced education is an educational technique that promotes long-term knowledge retention via multiple repeated learning sessions separated by prespecified time intervals [19-21]. Spacing knowledge acquisition is a well-studied and effective learning technique, particularly if it is associated with the active retrieval of information [22]. The beneficial effects of spacing and retrieval appear to be independent of the age of students and the length of the spacing interval, although research seems to suggest that longer spacing intervals are correlated to longer-term knowledge retention [22,23]. Similarly, the effectiveness of spaced education appears to be comparable in online, simulation, and classroom settings [24] or in studies assessing the acquisition of theoretical knowledge and practical skills [22,24]. Spaced education has been increasingly used in undergraduate [25] and postgraduate [24] medical education to improve learners’ knowledge [26] or skills, such as resuscitation in emergency medicine [27,28], surgical techniques [29], and image interpretation in radiology [30]. Spaced education has also been shown to promote changes in clinical practice [31,32]. By combining the accessibility and flexibility of digital education with the benefits of spaced learning in knowledge retention, spaced digital education might offer a feasible, scalable, and effective solution to providing high-quality education for health care professionals [33].

    A recent systematic review [31] assessed the effectiveness of spaced education in continuous development programs for practicing health care professionals, suggesting an improvement in knowledge, skills, clinical behavior, and confidence. However, the review included a variety of research designs and offered a narrative summary of findings. Given the increasing relevance of spaced education in health profession education, particularly when delivered in digital format, we considered it important to update and expand Phillips et al’s [31] systematic review. The aim of this systematic review was to assess the effectiveness of spaced digital education in improving pre- and postregistration health care professionals’ knowledge, skills, attitudes, satisfaction, and clinical behavior change.


    Study Design

    The review followed Cochrane’s methodology [34] and was reported according to the PRISMA (Preferred Reporting Items of Systematic Reviews and Meta-Analyses) checklist [35]. The protocol was registered on the International Prospective Register of Systematic Reviews (PROSPERO; CRD42021241969) on April 14, 2021.

    Identification of Studies

    The search strategy was developed for a series of systematic reviews on the effectiveness of digital health education interventions for the education and training of students and professionals of health care–related professions [5,12-14,17,18,36-47]. The search strategy was developed in collaboration with librarians and information specialists from Karolinska Instituet and has been regularly updated. We searched MEDLINE, CENTRAL, Embase, Web of Science, ERIC, PsycINFO, CINAHL, and ProQuest Dissertation and Theses databases from January 1990 to February 2023. We complemented the search with forward and backward citation searches in all included studies and 2 relevant reviews [24,31]. Multimedia Appendix 1 presents the MEDLINE search strategy.

    Eligibility Criteria

    This systematic review included randomized controlled trials (RCTs), cluster randomized controlled trials (cRCTs), and quasi-RCTs reporting spaced digital education interventions for health profession education. Randomized cross-over trials were excluded due to the high risk of contamination and carryover effects in educational interventions.

    Study participants included preregistration students and postregistration professionals from health professions, such as medicine, dentistry, nursing and midwifery, medical diagnostic and treatment technology, physiotherapy and rehabilitation, and pharmacy, as defined by in the Health Field of Education and Training (091) of the International Standard Classification of Education (ISCED-F) [48]. Practitioners of traditional, alternative, and complementary medicine were excluded.

    We included studies comparing spaced digital education interventions with nonspaced digital education modalities, spaced nondigital education, traditional face-to-face learning, blended learning, or no intervention. We included 2 spaced digital education modalities in this review. Spaced online education was defined for this study as an educational intervention where information, such as case studies or multiple-choice questions, “is presented and repeated over spaced intervals” [24,49] using a digital device, such as a computer, smartphone, or tablet computer. Spaced digital simulation was defined as studies using high-fidelity simulators, including virtual reality (VR)–based instruments, to provide skills training and assessment that was repeated over spaced intervals. There may be 2 or more repetitions of the learning content that should occur in different days.

    Studies were excluded if both intervention and comparison groups described a spaced digital education intervention, if the spacing interval was shorter than 1 day, and if the materials were not repeated over the course of the intervention.

    Outcomes Measured

    We assessed the following primary outcomes, measured using validated or nonvalidated instruments:

    • Postintervention knowledge
    • Postintervention skills
    • Postintervention attitudes or perceptions toward the intervention, as well as the interaction with patients and colleagues
    • Learners’ satisfaction with the intervention
    • Postintervention behavior change in the way learners interact with patients or colleagues or modify their practices

    We also included the following secondary outcomes: patient-related outcomes, cost and cost-effectiveness of the intervention, and negative effects of implementing a spaced education intervention.

    Data Collection and Analysis

    Selection of Studies

    The title and abstract screening for previous updates of the library was performed by pairs of reviewers working independently and in parallel. However, for the February 2023 update of the search strategy, screening was performed by 1 reviewer (author LM) on ASReview [50,51], an open source screening tool using several active learning and machine learning techniques. Following ASReview guidelines, we defined stopping criteria of 1.04% (90/8634 citations) consecutive “irrelevant” records and screening between 8% and 33% of all citations. The stopping criterion for this review was reached after screening 2850 (33.01%) records. Full-text screening for the entire digital health education library was performed by groups of 2 independent reviewers (authors MSPN, TDRN, and YIA) working in parallel. The data extraction and quality assessment of the included studies were also performed by groups of 2 independent reviewers. Differences between reviewers were resolved by discussion or by consultation with a third reviewer (LM) acting as an arbiter.

    Data Extraction and Quality Assessment

    Data from the included studies were extracted using a standardized Microsoft Excel form. We extracted information about the study design, participants, characteristics of the digital health education intervention, primary and secondary outcomes, and pre- and postintervention results, as reported by the primary studies.

    We assessed the methodological quality of the included studies using Cochrane’s Risk of Bias Tool 2 (RoB-2) [52,53], which assesses the risk of bias arising from the randomization process, deviations from the intended interventions, missing outcome data, measurement of the outcome, and selection of the outcome result. For cRCTs, we also assessed the bias arising from the timing of identification or recruitment of participants into clusters. Studies were assessed as having a high risk of bias if at least 1 category was assessed as high risk or multiple domains were assessed as “some concerns.” Studies with 1 domain assessed as “some concerns” were categorized as “some concerns” risk of bias. Finally, studies with all domains assessed as low risk were classified as having a low risk of bias. The risk-of-bias figures were created using robvis, an online tool [54].

    Grading the Quality of the Evidence

    We assessed the quality of evidence and presented it as a summary-of-findings table (Multimedia Appendix 2) for 3 comparisons: spaced online education versus no intervention, spaced online education versus massed online education, and spaced digital simulation versus massed simulation. The evaluation followed the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) criteria; assessed the limitation of the studies (risk of bias), inconsistency, indirectness, and imprecision; and rated the quality of evidence as very low, low, moderate, or high [55]. The assessment was performed using GRADE profiler (GRADEpro) software [55].

    Statistical Analysis

    Study data were analyzed using Cochrane’s Review Manager (RevMan) 5.4.1 [56]. We used the immediate postintervention results in the analyses. To ensure consistent reporting, for outcomes reported as continuous variables, we calculated the standardized mean difference (SMD) and 95% CIs. If studies reported P values, we calculated the SD using RevMan’s inbuilt calculator [56]. If studies reported the median (IQR), we converted those values to means (SDs) following Luo et al [57] and Wan et al [58]. In studies reporting 2 or more groups, for example, studies with more than 1 intervention arm, we combined the groups before the analysis using the Cochrane Handbook for Systematic Reviews of Interventions guidelines [59]. Subsequently, we calculated the SMD and 95% CIs, as per the protocol. Effect size interpretation followed the Hedges’ g statistic and considered an intervention to have a small effect if the SMD was between 0.2 and 0.5, a moderate effect if it was between 0.5 and 0.8, and a large effect if it was 0.8 and above. For studies reporting dichotomous data, we calculated the relative risks (RRs) and associated 95% CIs across studies. For cRCTs, we extracted individual-level data, as reported in the primary studies. If relevant outcome data were missing from the primary studies, we contacted the authors to obtain information. We did not input any missing data. When possible, data were analyzed on an intention-to-treat basis.

    We described our findings narratively using Miller’s classification of clinical competence [60] to classify the outcomes. When possible, we performed a meta-analysis using a random effects model. We used the I2 statistic to evaluate heterogeneity, with I2<25%, 25%-75%, and >75% representing low, moderate, and high heterogeneity, respectively [34].

    We attempted to assess publication bias and perform subgroup analyses; however, the small number of studies in each category and the heterogeneity of data precluded these assessments.


    Characteristics of Included Studies

    The search strategy yielded 85,755 publications, of which 23 (0.03%) studies, reported in 22 papers (0.03%), were included in the review. The study selection process was summarized using a PRISMA flow diagram (Figure 1) [35]. Multimedia Appendix 3 also presents a summary of the included studies, and Multimedia Appendix 4 lists the studies excluded after reviewing the full text.

    ‎
    Figure 1. Study selection flowchart.

    Table 1 presents the characteristics of the included studies. Most studies (n=19, 83%) were RCTs. Participants were randomized into 2 groups, except in 1 (4%) study [61], which randomized participants into 3 groups. Most studies (n=19, 83%) were conducted in high-income countries [49,61-77], 3 (13%) studies were conducted in upper middle–income countries [78-80], and only 1 (4%) study was conducted in a lower middle–income country [81]. The studies included a total of 3371 participants. Study populations comprised postregistration health professionals (n=20, 87%) and preregistration medical [61,76] and nursing [79] students (n=3, 13%). Of the 23 studies, 9 (39%) [61,62,65,66,69,72,74,78,80] reported 1 and 7 (30%) [49,63,67,70,73,76,77] reported 2 primary outcomes. The most common outcome was knowledge (n=15, 65%) [49,62-68,72-75,77,81], followed by skills (n=9, 39%) [61,69,70,72,76,78-81], attitudes (n=8, 35%) [64,68,71-73,75,77,79], behavior change (n=8, 35%) [49,63,67,68,71,75,77], and satisfaction with the intervention (n=7, 31%) [64,70-72,75,76,81].

    Table 1. Characteristics of the included studies.
    CharacteristicsStudies (N=23), n (%)
    Year of publication

    		Before 201913 (57)

    		After 201910 (43)
    Country

    		United States14 (61)

    		Türkiye2 (9)

    		Australia1 (4)

    		Canada1 (4)

    		China1 (4)

    		France1 (4)

    		Nigeria1 (4)

    		Singapore1 (4)

    		United States and Canada1 (4)
    Study design

    		RCTa19 (83)

    		cRCTb3 (13)

    		Quasi-RCT1 (4)
    Participants

    		Students of medicine2 (9)

    		Students of nursing1 (4)

    		Medical and surgical specialty interns and residents (trainee physicians)3 (13)

    		Medical specialty residents (trainee physicians)4 (17)

    		Surgical specialty residents (trainee physicians)5 (22)

    		Practicing physicians2 (9)

    		Multidisciplinary primary care providers6 (26)
    Clinical domain:medical specialties

    		Internal medicine2 (9)

    		Oncology1 (4)

    		Pediatrics2 (9)

    		Primary care5 (22)
    Clinical domain:surgical specialties

    		General surgery2 (9)

    		Obstetrics and gynecology2 (9)

    		Orthopedics2 (9)

    		Urology2 (9)
    Clinical domain: medical and surgical specialties2 (9)
    Intervention: spaced online education

    		Via email10 (43)

    		Using proprietary software (QStream/Moodle)4 (17)

    		Via email with gamified elements1 (4)

    		Via mobile apps1 (4)

    		Via online games1 (4)
    Intervention: spaced digital simulation training

    		High-fidelity simulation4 (17)

    		VRc simulation1 (4)

    		Enhanced with mobile learning1 (4)
    Control: spaced online education

    		No intervention7 (30)

    		Massed online education via email5 (22)

    		Online access to papers2 (9)

    		Massed digital education via app with no alerts/reminders1 (4)

    		Slideshow-based online program1 (4)

    		Standard curriculum1 (4)
    Control: spaceddigital simulation training

    		Nonspaced high-fidelity simulation3 (13)

    		Nonspaced simulation2 (9)

    		No intervention1 (4)
    Number of study outcomes

    		19 (39)

    		27 (30)

    		35 (22)

    		42 (9)
    Category of study outcomes

    		Knowledge15 (65)

    		Skills9 (39)

    		Attitudes8 (35)

    		Satisfaction7 (31)

    		Behavior change8 (35)
    Study quality

    		Low risk of bias11 (48)

    		Unclear risk of bias5 (22)

    		High risk of bias7 (30)

    aRCT: randomized controlled trial.

    bcRCT: cluster randomized controlled trial.

    cVR: virtual reality.

    Characteristics of Interventions

    Spaced online education was described in 17 (74%) studies. The interventions consisted of brief learning modules delivered at prespecified intervals using email (n=10, 59%) [49,63,65,66,69-72,77]; proprietary software, such as Qstream (n=3, 18%) [64,68,74] and Moodle (n=1, 6%) [62]; an online game (n=1, 6%) [75]; and a smartphone app (n=1, 6%) [73]. Typically, in 15 (88%) of the 17 studies, the learning modules presented a case study or a small learning unit, followed by 1 or more multiple-choice questions (MCQs) and the correct answer with an explanation to each question. Alternatively, 2 (12%) [69,70] of the 17 studies offered small learning units without a testing component. The information was repeated 2 or more times at predetermined intervals or according to participant responses in adaptive systems (n=10, 59%) [62,63,67,68,71-74,77]. The duration of the interventions varied from 1 week [75] to 16 months [73].

    Additionally, 6 studies (26%) described spaced digital simulation interventions delivered using high-fidelity simulators (n=4, 67%) [76,78-80], a VR system (n=1, 17%) [61], or low-fidelity simulators enhanced with weekly SMS text messages (n=1, 17%) [81]. The duration of the interventions varied from 2 training sessions of 4 hours each delivered over 1 week (n=1, 17%) [79] to 4 training sessions delivered over 4 weeks (n=2, 33%) [76,80].

    Risk of Bias

    The risk of bias (Figure 2) was low in 9 (39%) studies [49,61,64,66,67,69,70,76,78], with some concerns in 4 (17%) studies [62,63,65,80], and high in 7 (30%) studies [71,72,74,75,77,79]. The most common source of bias was missing outcome data. The risk-of-bias assessment for 3 (13%) cRCTs [68,73,81] reflected some concerns in one study, while it was low in 2 studies.

    ‎
    Figure 2. Risk-of-bias assessment for RCTs and cRCTs. cRCT: cluster randomized controlled trial; RCT: randomized controlled trial.

    Primary Outcomes

    Postintervention Knowledge

    Knowledge was reported by 15 (65%) studies (2924/3371, 86.74%, participants), including 14 (93%) studies on spaced online education [49,62-68,72-75,77] and 1 (7%) study on spaced digital simulation [81]. In total, 12 (80%) of 15 studies assessed knowledge using MCQs [49,62-68,72,74,81], of which 7 (47%) studies used validated questionnaires [49,63,65-68,81]. The other studies used a combination of MCQs and short-answer questions [75], the short-answer component of the College of Family Physicians of Canada [73], or true/false questions developed for the study [77]. Figure 3 [63,72] summarizes the pooled estimates for postintervention knowledge outcomes. Three studies presented knowledge retention data [64,65,81].

    ‎
    Figure 3. Forest plot for postintervention knowledge. Note: SD values for Dolan et al and Shaw et al (Brigham and Women’s Hospital and Massachusetts General Hospital) were derived from P values using RevMan’s inbuilt calculator.
    Spaced Online Education Versus No Intervention

    Of 4 (27%) studies (499/2924, 17.07%, participants) [49,72,74,77], 3 (75%) studies (n=282, 56.5%, participants) [49,72,74] were included in the meta-analysis. The pooled estimate showed no significant difference in knowledge (SMD 0.66, 95% CI –0.21 to 1.54, I2=92%) between the intervention and control groups.

    Another study (133/499, 26.7%, participants) [77], not included in the meta-analysis, randomized primary care providers (PCPs) to receive 2 months of spaced online education or no intervention after watching an educational video on eating disorders and reported no further improvement in knowledge in the intervention group.

    Spaced Online Education Versus Massed Education

    Of the 15 studies, 9 (60%) studies (1691/2924, 57.83%, participants) [62-64,66-68,72,73,75] comparing spaced online education with massed digital education were included in the meta-analysis. The pooled estimate showed a small effect size (SMD 0.32, 95% CI 0.13-0.51, I2=66%) favoring spaced online education.

    Spaced Digital Simulation Versus Massed Simulation

    Ugwa et al [81] compared low-dose onsite simulation enhanced with mobile mentoring with traditional massed simulation. Knowledge improved in the traditional arm immediately after completing the intervention.

    Knowledge Retention

    Of the 15 studies, 2 studies (313/2924, 10.7%, participants) [64,65] comparing spaced online education and massed education reported knowledge retention. House et al [64] reported a nonsignificant effect on knowledge retention at 3 months’ follow-up (SMD 0.46, 95% CI –0.05 to 0.98), while Kerfoot [65] reported a small effect size 2 years after the completion of a trial on spaced versus massed online education (SMD 0.34, 95% CI 0.02-0.67). The pooled estimate showed a small difference in knowledge retention favoring spaced online education over controls (SMD 0.38, 95% CI 0.10-0.65, I2=0%). Figure 4 presents the pooled estimates for knowledge retention.

    ‎
    Figure 4. Forest plot for knowledge retention.

    Additionally, Ugwa et al [81] reported that knowledge retention scores at 3 and 12 months’ follow-up were similar in the spaced digital simulation and control arms (RR 1.02, 95% CI 0.91-1.13).

    Postintervention Skills

    A total of 9 (39%) studies (784/3371, 23.26%, participants) reported learners’ skills, including 3 (33%) studies on spaced online education [69,70,72] and 6 (67%) studies on spaced digital simulation [61,76,78-81]. Postintervention skills were measured using online surveys distributed among medical students to assess the faculty’s [70] or residents’ [69] teaching skills, anonymized video recordings [61,72], task completion reports obtained from the simulation instrument [76,78,80], the Objective Structured Clinical Examination (OSCE) [81], and self-reported questionnaires [79].

    Spaced Online Education Versus No Intervention

    A total of 2 (22%) studies (84/784, 2.4%, participants) [69,70] compared the impact of spaced online education on improving teaching skills. Matzie et al [69] reported that residents receiving the intervention offered more effective feedback to medical students (RR 1.43, 95% CI 1.08-1.90), while Pernar et al [70] reported no difference in faculty’s feedback quality between intervention and control groups (RR 0.96, 95% CI 0.72-1.27). The pooled estimate for both studies reflected a nonsignificant effect of spaced online education on improving teaching skills (RR 1.21, 95% CI 0.99-1.47, I2=74%). Figure 5 presents the forest plot.

    ‎
    Figure 5. Forest plot for postintervention skills (spaced online education vs no intervention).
    Spaced Online Education Versus Massed Education

    Shaw et al [72] assessed interns’ compliance (Brigham and Women’s Hospital: 195/784, 24.9%, participants) with patient safety procedures while completing a central venous catheterization simulation, reporting a nonsignificant effect (SMD 0.34, 95% CI –0.05 to 0.73).

    Spaced Digital Simulation Versus Massed Simulation

    A total of 5 (56%) studies (483/784, 61.6%, participants) [61,76,78,79,81] compared spaced digital simulation with massed simulation (Figure 6 [11]). The interventions aimed to improve nursing students’ skills to manage an adult patient with chronic lymphocytic leukemia [79], residents’ arthroscopic skills [78], medical students’ intramedullary tibial nailing [61] and microsurgical suturing skills [76], and health workers’ maternal and newborn care skills [81]. In addition, 3 (60%) studies [76,79,81] reported skill improvement as overall scores. The pooled estimate of 2 (67%) studies [76,79] showed that spaced digital simulation is superior to massed simulation (SMD 1.24, 95% CI 0.84-1.64, I2=74%) in improving learners’ skills. However, Ugwa et al [81] reported a nonsignificant effect in learners’ skills using a spaced simulation intervention enhanced with mobile mentoring (RR 1.05, 95% CI 0.95-1.16) compared to controls.

    ‎
    Figure 6. Forest plots for learners’ skill outcomes. Note: SD values for Li et al were derived from the SE using RevMan’s inbuilt calculator.

    In addition, 3 (60%) studies [61,76,78] reported itemized skill outcomes, including the time to complete a task, economy of movements, the number of errors, and the number of hints requested, with improved skills representing decreased values. Li et al [78] reported a decrease in task completion time and camera path length but no difference in the extent of cartilage injury in orthopedic residents training in arthroscopic skills using spaced high-fidelity simulation. Orland et al [61] reported a shorter completion time, decreased errors and number of requested hints, and a higher rate of completed tasks in medical students using spaced VR simulation with or without technique guidance compared to controls. Teo et al [76] reported a nonsignificant difference in the time to complete the test strip between students receiving spaced digital simulation and controls. The pooled estimates for the time to complete a task (3, 60%, studies) and the number of errors (2, 40%, studies) showed a nonsignificant difference between intervention and control groups.

    Spaced Digital Simulation Versus No Intervention

    Akdemir et al [80] (22/784, 2.8%, participants) described that a spaced digital simulation intervention to improve gynecology residents’ laparoscopic skills resulted in a significant reduction in the time to complete tasks (SMD –1.50, 95% CI –2.46 to –0.53) and the number of movements (SMD –1.96, 95% CI –3.01 to –0.91) and a nonsignificant change in the number of errors (SMD –0.26, 95% CI –1.10 to 0.58).

    Postintervention Attitudes

    A total of 8 (35%) studies (1539/3371, 45.65%, participants) reported attitudes, including confidence [64,68,71,72], comfort in screening [77], engagement with the intervention (time spent [75] and the number of completed cases [73]), self-efficacy [79], and anxiety [79].

    Spaced Online Education Versus No Intervention

    Raffoul et al [77] reported that PCPs in the intervention group were more comfortable screening for eating disorders. Shaw et al [71] showed that PCPs receiving spaced online education after a conference reported greater confidence in managing 4 conditions presented in the intervention.

    Spaced Online Education Versus Massed Education

    The pooled estimate of 3 (38%) studies showed improved confidence in managing clinical cases [64,68] and patient safety protocols [72] (SMD 0.28, 95% CI 0.01-0.55, I2=43%), as shown in Figure 7 [64,72].

    ‎
    Figure 7. Forest plot for learners’ postintervention confidence (spaced online education vs massed education). Note: SD values for Shaw et al (Brigham and Women’s Hospital) were derived from P values, and SD values for House et al were derived from CIs using RevMan’s inbuilt calculator.

    Additionally, Jaunay et al [75] reported that general practitioners (GPs) who received spaced education via an online game spent 45-60 minutes in the intervention compared to 10-20 minutes in the control group. Grad et al [73] compared 2 versions of a study app presenting family medicine case studies and showed that adding reminders that enabled spaced learning resulted in residents completing more case studies, although the difference was nonsignificant.

    Spaced Digital Simulation Versus Massed Simulation

    Kocyigit and Karagozoglu [79] reported that spaced digital simulation improved nursing students’ self-efficacy (SMD 0.57, 95% CI 0.12-1.02), while there was no significant change in state anxiety (SMD 0.25, 95% CI –0.19 to 0.69) between the study groups.

    Postintervention Satisfaction

    A total of 7 (30%) studies (982/3371, 29.13%, participants) [64,70-72,75,76,81] reported satisfaction with spaced digital education. Satisfaction was assessed using a postintervention survey [75], Likert scale surveys [64,71,72,76], an email-based questionnaire [70], and a satisfaction questionnaire plus focus group discussion [81]. Satisfaction was measured only in the intervention group, except Teo et al [76], who administered the survey to both intervention and control groups and reported similar satisfaction (8.47/10 in spaced digital simulation vs 8.0/10 in massed simulation; P=.23). In general, participants considered spaced digital education engaging and enjoyable [72,75] and said that would like to engage in similar interventions [64,72,75] in the future. Participants also reported greater satisfaction in improved patient outcomes [81]. However, 1 (14%) study [70] reported that surgery faculty were not satisfied with an intervention consisting of weekly emails with tips to improve their teaching skills, because the content was vague and generic and it added to the burden of numerous daily emails.

    Postintervention Behavior Change

    Of the 23 studies, 8 (35%) studies (858/3371, 25.45%, participants) reported a change in clinical behavior, including 7 (88%) studies on spaced online education [49,63,67,68,71,75,77] and 1 (12%) study on spaced digital simulation [79]. The behavior change data were obtained from clinical records [49,63,67,68] and self-reported questionnaires developed by the study authors [71,75,77,79]. Figure 8 [49,63] shows the meta-analyses results.

    ‎
    Figure 8. Forest plot for learners’ postintervention change in clinical behavior. Note: SD values for Dolan et al and Kerfoot et al were derived from P values using RevMan’s inbuilt calculator.
    Spaced Online Education Versus No Intervention

    The pooled estimate from 2 (25%) studies [49,71] showed that spaced online education promotes a change in clinicians’ behavior (SMD 0.63, 95% CI 0.27-0.99, I2=51%). Kerfoot et al [49] showed that spaced online education improved clinicians’ prostate cancer–screening behavior while receiving the intervention (SMD 0.42, 95% CI 0.02-0.83) and during the 72-week follow-up (SMD 0.53, 95% CI 0.12-0.94). Shaw et al [71] reported a change in PCPs’ clinical behavior for 4 common disorders (hypertension, diabetes, nonalcoholic steatohepatitis, and HIV) after receiving a spaced online education intervention (SMD 0.36, 95% CI 0.07-0.66 for hypertension to SMD 0.95, 95% CI 0.65-1.26 for HIV). Finally, Raffoul et al [77] reported that PCPs receiving spaced online education were more comfortable screening for eating disorders, although there was no difference between the intervention and control groups in self-reported screening or referral behavior.

    Spaced Online Education Versus Massed Education

    Dolan et al [63] reported that residents receiving spaced online education appropriately screened more patients for osteoporosis (SMD 0.74, 95% CI 0.11-1.38) and were more likely to treat patients with bisphosphonates (SMD 0.69, 95% CI 0.06-1.32). Jaunay et al [75] reported that GPs receiving an intervention involving a spaced online education game used the new knowledge to improve clinical practice more often (RR 1.45, 95% CI 1.07-1.96), compared to controls. Kerfoot et al [67] demonstrated improved patient outcomes (shorter time to blood pressure target) for PCPs receiving spaced online education (hazard ratio 1.043, 95% CI 1.007-1.081, P=.018). Finally, Mallon et al [68] reported that a spaced online constipation management program delivered to pediatric PCPs resulted in lower patient referrals to specialist pediatric gastroenterology clinics but no reduction in constipation-related visits to the emergency department or urgent care clinics.

    Spaced Digital Simulation Versus Massed Simulation

    Kocyigit and Karagozoglu [79] reported that nursing students’ medical error tendency significantly decreased after 2 spaced digital simulation sessions (80.7 vs 61.9 points, P=0, with higher values reflecting fewer medical errors).


    Principal Findings

    In this systematic review, the first to include a meta-analysis, we found that spaced digital education is effective in improving knowledge, skills, and confidence and promoting change in clinical practice for pre- and postregistration health professionals. In particular, marked improvements were demonstrated in physicians’ adoption of evidence-based practices after engaging in spaced online learning and in students’ and residents’ surgical skills after engaging in spaced digital simulation interventions compared to active or inactive controls. The results for knowledge and behavior change, albeit reported in a smaller number of papers, showed that these improvements are maintained over time. Yet, our results should be interpreted with caution, given the substantial heterogeneity of included studies and the small number of studies populating the different spaced digital education categories.

    Our systematic review categorized the spaced interventions into 2 distinct categories: spaced online education interventions repeatedly delivered small amounts of information while testing learners’ knowledge, while spaced digital simulation interventions provided practical, repeated, and spaced training sessions using high-fidelity simulators to improve learners’ skills. The studies included in this review evaluated 3 learning outcomes well aligned with the 4 levels of Miller’s classification of clinical competence: knowledge (“knows”), skills (“knows how,” “shows how”), and behavior change (“does”) [59]. Although measuring learners’ attitudes and satisfaction with the digital health education intervention is important and may facilitate learners’ engagement with the intervention, knowledge, skills, and behavior change are particularly important to determine the impact of the intervention. Knowledge or skills provide a direct evaluation of the intervention, and selecting one over the other will directly depend on the type of educational intervention content to be delivered. Alternatively, behavior change would more directly reflect the impact of the digital health education intervention on health professionals’ daily practice and the potential benefits to patients.

    The findings reported in this review support the use of spaced digital education interventions in undergraduate and postgraduate health profession education. However, medical schools and providers of continuous professional development programs have yet to fully embrace this learning modality [82-84]. Simultaneously, students are increasingly turning to commercial learning platforms to extend their learning and to prepare for licensing examinations, such as the USMLE (United States Medical Licensing Examination) program [82,85]. These platforms often follow spaced education and active recall principles in the development of learning materials. About one-third of second-year medical students in the United States routinely use online materials not provided by their medical school to prepare for courses and particularly in preparation for the USMLE [86-88], creating a “parallel, medical student–driven curriculum” [86]. As digital health education platforms become more pervasive, it is critical that institutions providing undergraduate and continuous postgraduate development programs adapt their curriculum to incorporate evidence-based learning approaches to improve students’ learning outcomes, while increasing student satisfaction and acceptability of the educational content.

    The studies included in this review described interventions with substantially diverse spacing intervals. However, none of the studies provided a rationale for this differing length. Previous studies from the educational research field have shown that the length of the spacing interval is directly related to the knowledge retention period, suggesting that the ideal interval would be around 10% of the test delay [23,89]. Furthermore, expanding the spacing interval appears to further improve retention [89]. Most of these studies have reported solely on knowledge, such as vocabulary recall [23]. Similar research in health profession education is lacking. Therefore, it is imperative that future research evaluate the role of the spacing effect on knowledge retention in health care professionals, as well as the impact of spaced digital education on other learning outcomes, such as skills or clinical behavior change.

    Spaced online education interventions have been mostly delivered via email, which may not fully leverage more interactive, engaging features often included in other digital interventions. For example, a recent literature review [90] suggested that the use of game design elements in the development of digital health education interventions may increase user engagement and enhance collaboration. In addition, microlearning, an educational approach that consists of brief, focused lessons, is gaining acceptance as a professional development method in the corporate sector and industry [91], encouraging the development of gamified digital platforms customizable to user needs. However, these platforms do not appear to be widely used in health care settings, despite the potential benefits of including multimedia content, as well as reminders, leaderboards, or other gamified elements, to increase user engagement [92].

    Strengths and Limitations

    There are several strengths of this systematic review. We followed gold-standard Cochrane guidelines, and we used a comprehensive search strategy developed by expert librarians that was previously used in a series of systematic reviews on the effectiveness of digital health education modalities in health profession education [5,12-14,17,18,36,38,40,42-47]. The search strategy was supplemented by forward and backward reference screening.

    However, the review also has limitations. First, our search strategy was developed to retrieve clinical trials on digital health education interventions without including spaced education–specific search terms. However, to compensate for this limitation, we added a backward and forward citation search of all included studies. Second, the included studies showed substantial heterogeneity: they reported diverse outcomes and used diverse measurement instruments and reporting variables that limited further data analyses. Third, the low quality of the included studies, reflected in the majority of unclear- and high-risk-of-bias assessments, added uncertainty to the validity of our results.

    Future Research

    Several aspects of spaced digital education for health care professionals warrant further research. First, the optimal spacing interval for spaced interventions needs to be defined, accounting for the type of learning outcome targeted by the intervention (eg, knowledge, skills, behavior change, satisfaction), the health care profession, the career stage (pre-registration students or postregistration professionals), and the intended retention period. Second, the potential impact of newer, more interactive platforms in influencing active engagement of participants, their compliance with the intervention, and the improvement in measured outcomes warrants further investigation. Third, more research is needed to identify the areas of learning where a spaced intervention may be more appropriate. Fourth, studies evaluating possible adverse events associated with interventions are needed as adverse events were not reported in any of the studies included in this review. In addition, cost-effectiveness studies are required to assess the feasibility of large-scale implementation of this learning modality.

    Conclusion

    This systematic review showed that spaced digital education is effective in improving knowledge, skills, and clinical behavior change in pre- and postregistration health care professionals. The improvement of surgical skills using spaced digital simulation and clinical behavior change after spaced online education interventions are particularly noteworthy. These findings suggest that spaced digital education could be effectively added to undergraduate health profession education and continuous professional education programs for practicing professionals. For the successful implementation of spaced digital education interventions in the health professions, we suggest considering the use of newer, more interactive platforms to deliver the interventions. Future studies should evaluate areas of learning where the intervention will be most effective and assess the ideal length of the intervention and the spacing interval.

    Acknowledgments

    We gratefully acknowledge contributions by Ms Yasmin Lynda Munro from the LKCMedicine Medical Library for her assistance in updating the digital health education library in February 2022 and February 2023.

    Data Availability

    The data sets supporting the conclusions of this paper are included within the paper and Multimedia Appendices.

    Authors' Contributions

    LM and LTC had full access to all the data in the study and took responsibility for the integrity of the data and the accuracy of the data analysis. LTC conceptualized and designed the study. LM updated the study library. LM, MSPN, TDRN, YIA, and AIJ selected studies for inclusion, extracted data, and assessed the study quality. LM, BMK, and LTC performed statistical analyses and performed Grading of Recommendations, Assessment, Development and Evaluation (GRADE) assessments. LM, MSPN, TDRN, and YIA drafted the manuscript. All authors reviewed and provided comments for the manuscript. All authors read and approved the final manuscript.

    Conflicts of Interest

    LTC is Associate Editor of JMIR Medical Education.

    Multimedia Appendix 1

    Medline search strategy.

    DOCX File , 24 KB
    Multimedia Appendix 2

    Grading of Recommendations, Assessment, Development and Evaluation (GRADE)–based evaluation: summary of findings.

    DOCX File , 30 KB
    Multimedia Appendix 3

    Characteristics of included studies.

    DOCX File , 49 KB
    Multimedia Appendix 4

    Excluded studies.

    DOCX File , 25 KB
    Multimedia Appendix 5

    PRISMA 2020 checklist.

    DOCX File , 51 KB
    1. Boniol M, Kunjumen T, Nair TS, Siyam A, Campbell J, Diallo K. The global health workforce stock and distribution in 2020 and 2030: a threat to equity and 'universal' health coverage? BMJ Glob Health. Jun 2022;7(6):e009316. [FREE Full text] [CrossRef] [Medline]
    2. Global strategy on human resources for health: workforce 2030. World Health Organization. 2016. URL: https://www.who.int/publications/i/item/9789241511131 [accessed 2023-07-01]
    3. De Vries N, Lavreysen O, Boone A, Bouman J, Szemik S, Baranski K, et al. Retaining healthcare workers: a systematic review of strategies for sustaining power in the workplace. Healthcare (Basel). Jun 29, 2023;11(13):1887. [FREE Full text] [CrossRef] [Medline]
    4. Poon YR, Lin YP, Griffiths P, Yong KK, Seah B, Liaw SY. A global overview of healthcare workers' turnover intention amid COVID-19 pandemic: a systematic review with future directions. Hum Resour Health. Sep 24, 2022;20(1):70. [FREE Full text] [CrossRef] [Medline]
    5. Martinengo L, Yeo NJY, Markandran KD, Olsson M, Kyaw BM, Car LT. Digital health professions education on chronic wound management: a systematic review. Int J Nurs Stud. Apr 2020;104:103512. [FREE Full text] [CrossRef] [Medline]
    6. Car J, Carlstedt-Duke J, Tudor Car L, Posadzki P, Whiting P, Zary N, et al. Digital education in health professions: the need for overarching evidence synthesis. J Med Internet Res. Mar 14, 2019;21(2):e12913. [FREE Full text] [CrossRef] [Medline]
    7. Digital education for building health workforce capacity. World Health Organization. 2020. URL: https://apps.who.int/iris/bitstream/handle/10665/331524/9789240000476-eng.pdf?sequence=1&isAllowed=y [accessed 2023-09-07]
    8. Carroll C, Booth A, Papaioannou D, Sutton A, Wong R. UK health-care professionals' experience of on-line learning techniques: a systematic review of qualitative data. J Contin Educ Health Prof. 2009;29(4):235-241. [CrossRef] [Medline]
    9. Reeves S, Fletcher S, McLoughlin C, Yim A, Patel KD. Interprofessional online learning for primary healthcare: findings from a scoping review. BMJ Open. Aug 04, 2017;7(8):e016872. [FREE Full text] [CrossRef] [Medline]
    10. Rouleau G, Gagnon M, Côté J, Payne-Gagnon J, Hudson E, Dubois C, et al. Effects of e-learning in a continuing education context on nursing care: systematic review of systematic qualitative, quantitative, and mixed-studies reviews. J Med Internet Res. Oct 02, 2019;21(10):e15118. [FREE Full text] [CrossRef] [Medline]
    11. Vaona A, Banzi R, Kwag KH, Rigon G, Cereda D, Pecoraro V, et al. E-learning for health professionals. Cochrane Database Syst Rev. Jan 21, 2018;1(1):CD011736. [FREE Full text] [CrossRef] [Medline]
    12. George PP, Zhabenko O, Kyaw BM, Antoniou P, Posadzki P, Saxena N, et al. Online digital education for postregistration training of medical doctors: systematic review by the Digital Health Education Collaboration. J Med Internet Res. Feb 25, 2019;21(2):e13269. [FREE Full text] [CrossRef] [Medline]
    13. Wahabi HA, Esmaeil SA, Bahkali KH, Titi MA, Amer YS, Fayed AA, et al. Medical doctors' offline computer-assisted digital education: systematic review by the Digital Health Education Collaboration. J Med Internet Res. Mar 01, 2019;21(3):e12998. [FREE Full text] [CrossRef] [Medline]
    14. Kyaw BM, Saxena N, Posadzki P, Vseteckova J, Nikolaou CK, George PP, et al. Virtual reality for health professions education: systematic review and meta-analysis by the Digital Health Education Collaboration. J Med Internet Res. Jan 22, 2019;21(1):e12959. [FREE Full text] [CrossRef] [Medline]
    15. Davids J, Manivannan S, Darzi A, Giannarou S, Ashrafian H, Marcus HJ. Simulation for skills training in neurosurgery: a systematic review, meta-analysis, and analysis of progressive scholarly acceptance. Neurosurg Rev. Aug 2021;44(4):1853-1867. [FREE Full text] [CrossRef] [Medline]
    16. Toale C, Morris M, Kavanagh DO. Training and assessment using the LapSim laparoscopic simulator: a scoping review of validity evidence. Surg Endosc. Mar 2023;37(3):1658-1671. [FREE Full text] [CrossRef] [Medline]
    17. Dunleavy G, Nikolaou CK, Nifakos S, Atun R, Law GCY, Tudor Car L. Mobile digital education for health professions: systematic review and meta-analysis by the Digital Health Education Collaboration. J Med Internet Res. Feb 12, 2019;21(2):e12937. [FREE Full text] [CrossRef] [Medline]
    18. Lall P, Rees R, Law GCY, Dunleavy G, Cotič, Car J. Influences on the implementation of mobile learning for medical and nursing education: qualitative systematic review by the Digital Health Education Collaboration. J Med Internet Res. Feb 28, 2019;21(2):e12895. [FREE Full text] [CrossRef] [Medline]
    19. Cepeda NJ, Coburn N, Rohrer D, Wixted JT, Mozer MC, Pashler H. Optimizing distributed practice: theoretical analysis and practical implications. Exp Psychol. 2009;56(4):236-246. [FREE Full text] [CrossRef] [Medline]
    20. Cepeda NJ, Vul E, Rohrer D, Wixted JT, Pashler H. Spacing effects in learning: a temporal ridgeline of optimal retention. Psychol Sci. Nov 2008;19(11):1095-1102. [CrossRef] [Medline]
    21. Ebbinghaus H. Memory: A Contribution to Experimental Psychology. New York, NY. Teachers College, Columbia University; 1913.
    22. Carpenter SK, Pan SC, Butler AC. The science of effective learning with spacing and retrieval practice. Nat Rev Psychol. Aug 02, 2022;1(9):496-511. [CrossRef]
    23. Cepeda NJ, Pashler H, Vul E, Wixted JT, Rohrer D. Distributed practice in verbal recall tasks: a review and quantitative synthesis. Psychol Bull. May 2006;132(3):354-380. [CrossRef] [Medline]
    24. Versteeg M, Hendriks RA, Thomas A, Ommering BWC, Steendijk P. Conceptualising spaced learning in health professions education: a scoping review. Med Educ. Mar 2020;54(3):205-216. [FREE Full text] [CrossRef] [Medline]
    25. Yeh DD, Park YS. Improving learning efficiency of factual knowledge in medical education. J Surg Educ. 2015;72(5):882-889. [FREE Full text] [CrossRef] [Medline]
    26. Kerfoot BP, DeWolf WC, Masser BA, Church PA, Federman DD. Spaced education improves the retention of clinical knowledge by medical students: a randomised controlled trial. Med Educ. Jan 2007;41(1):23-31. [FREE Full text] [CrossRef] [Medline]
    27. Patocka C, Cheng A, Sibbald M, Duff JP, Lai A, Lee-Nobbee P, et al. A randomized education trial of spaced versus massed instruction to improve acquisition and retention of paediatric resuscitation skills in emergency medical service (EMS) providers. Resuscitation. Aug 2019;141:73-80. [FREE Full text] [CrossRef] [Medline]
    28. Lin Y, Cheng A, Grant VJ, Currie GR, Hecker KG. Improving CPR quality with distributed practice and real-time feedback in pediatric healthcare providers - a randomized controlled trial. Resuscitation. Sep 2018;130:6-12. [CrossRef] [Medline]
    29. Wiseman JJ, Perlmutter JW, Wiseman SM. Learning to learn: spaced education approaches should be adopted to help optimize learning during surgical residency training. Am J Surg. Feb 2023;225(2):439-441. [FREE Full text] [CrossRef] [Medline]
    30. Choe AI, Woodard S, Thompson BM, Walter V, Fotos JS, Kasales CJ. Spaced education: randomized trial comparing learning efficiency of the adaptive versus nonadaptive spaced education systems among radiology residents. J Am Coll Radiol. Jun 2022;19(6):706-710. [FREE Full text] [CrossRef] [Medline]
    31. Phillips JL, Heneka N, Bhattarai P, Fraser C, Shaw T. Effectiveness of the spaced education pedagogy for clinicians' continuing professional development: a systematic review. Med Educ. Sep 2019;53(9):886-902. [FREE Full text] [CrossRef] [Medline]
    32. Yeung J, Djarv T, Hsieh MJ, Sawyer T, Lockey A, Finn J, et al. Education, Implementation and Team Task Force and Neonatal Life Support Task Force of the International Liaison Committee on Resuscitation (ILCOR). Spaced learning versus massed learning in resuscitation - a systematic review. Resuscitation. Nov 2020;156:61-71. [FREE Full text] [CrossRef] [Medline]
    33. Hirsch AW, Nagler J. Spaced education through e-learning for ongoing professional development. AEM Educ Train. Oct 2020;4(4):423-427. [FREE Full text] [CrossRef] [Medline]
    34. Higgins J, Thomas J, Chandler J, Cumpston M, Li T, Page M. Cochrane Handbook for Systematic Reviews of Interventions version 6.3. Cochrane. 2022. URL: https://training.cochrane.org/handbook [accessed 2023-07-20]
    35. Page MJ, Moher D, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. BMJ. Mar 29, 2021;372:n160. [FREE Full text] [CrossRef] [Medline]
    36. Bajpai S, Semwal M, Bajpai R, Car J, Ho AHY. Health professions’ digital education: review of learning theories in randomized controlled trials by the Digital Health Education Collaboration. J Med Internet Res. Mar 12, 2019;21(3):e12912. [FREE Full text] [CrossRef] [Medline]
    37. Brusamento S, Kyaw BM, Whiting P, Li L, Tudor Car L. Digital health professions education in the field of pediatrics: systematic review and meta-analysis by the Digital Health Education Collaboration. J Med Internet Res. Sep 25, 2019;21(9):e14231. [FREE Full text] [CrossRef] [Medline]
    38. Divakar U, Nazeha N, Posadzki P, Jarbrink K, Bajpai R, Ho AHY, et al. Digital education of health professionals on the management of domestic violence: systematic review and meta-analysis by the Digital Health Education Collaboration. J Med Internet Res. May 23, 2019;21(5):e13868. [FREE Full text] [CrossRef] [Medline]
    39. Gentry SV, Gauthier A, L'Estrade Ehrstrom B, Wortley D, Lilienthal A, Tudor Car L, et al. Serious gaming and gamification education in health professions: systematic review. J Med Internet Res. Mar 28, 2019;21(3):e12994. [FREE Full text] [CrossRef] [Medline]
    40. Huang Z, Semwal M, Lee SY, Tee M, Ong W, Tan WS, et al. Digital health professions education on diabetes management: systematic review by the Digital Health Education Collaboration. J Med Internet Res. Feb 21, 2019;21(2):e12997. [FREE Full text] [CrossRef] [Medline]
    41. Kononowicz AA, Woodham LA, Edelbring S, Stathakarou N, Davies D, Saxena N, et al. Virtual patient simulations in health professions education: systematic review and meta-analysis by the Digital Health Education Collaboration. J Med Internet Res. Jul 02, 2019;21(7):e14676. [FREE Full text] [CrossRef] [Medline]
    42. Kyaw BM, Posadzki P, Dunleavy G, Semwal M, Divakar U, Hervatis V, et al. Offline digital education for medical students: systematic review and meta-analysis by the Digital Health Education Collaboration. J Med Internet Res. Mar 25, 2019;21(3):e13165. [FREE Full text] [CrossRef] [Medline]
    43. Kyaw BM, Posadzki P, Paddock S, Car J, Campbell J, Tudor Car L. Effectiveness of digital education on communication skills among medical students: systematic review and meta-analysis by the Digital Health Education Collaboration. J Med Internet Res. Aug 27, 2019;21(8):e12967-e12960. [FREE Full text] [CrossRef] [Medline]
    44. Kyaw BM, Tudor Car L, van Galen LS, van Agtmael MA, Costelloe CE, Ajuebor O, et al. Health professions digital education on antibiotic management: systematic review and meta-analysis by the Digital Health Education Collaboration. J Med Internet Res. Sep 12, 2019;21(9):e14984. [FREE Full text] [CrossRef] [Medline]
    45. Posadzki P, Bala MM, Kyaw BM, Semwal M, Divakar U, Koperny M, et al. Offline digital education for postregistration health professions: systematic review and meta-analysis by the Digital Health Education Collaboration. J Med Internet Res. Apr 24, 2019;21(4):e12968. [FREE Full text] [CrossRef] [Medline]
    46. Semwal M, Whiting P, Bajpai R, Bajpai S, Kyaw BM, Tudor Car L. Digital education for health professions on smoking cessation management: systematic review by the Digital Health Education Collaboration. J Med Internet Res. Mar 04, 2019;21(3):e13000. [FREE Full text] [CrossRef] [Medline]
    47. Tudor Car L, Kyaw BM, Dunleavy G, Smart NA, Semwal M, Rotgans JI, et al. Digital problem-based learning in health professions: systematic review and meta-analysis by the Digital Health Education Collaboration. J Med Internet Res. Feb 28, 2019;21(2):e12945. [FREE Full text] [CrossRef] [Medline]
    48. UNESCO Institute for Statistics. International Standard Classification of Education Fields of Education and Training 2013 (ISCED-F 2013). Montreal, QC. UNESCO Institute for Statistics; 2015.
    49. Kerfoot BP, Lawler EV, Sokolovskaya G, Gagnon D, Conlin PR. Durable improvements in prostate cancer screening from online spaced education a randomized controlled trial. Am J Prev Med. Nov 2010;39(5):472-478. [FREE Full text] [CrossRef] [Medline]
    50. van de Schoot R, de Bruin J, Schram R, Zahedi P, de Boer J, Weijdema F, et al. An open source machine learning framework for efficient and transparent systematic reviews. Nat Mach Intell. Feb 01, 2021;3(2):125-133. [CrossRef]
    51. van Dijk SHB, Brusse-Keizer MGJ, Bucsán CC, van der Palen J, Doggen CJM, Lenferink A. Artificial intelligence in systematic reviews: promising when appropriately used. BMJ Open. Jul 07, 2023;13(7):e072254. [FREE Full text] [CrossRef] [Medline]
    52. Sterne JAC, Savović J, Page MJ, Elbers RG, Blencowe NS, Boutron I, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ. Aug 28, 2019;366:l4898. [FREE Full text] [CrossRef] [Medline]
    53. Higgins J, Savović J, Page M, Elbers R, Sterne J. Chapter 8: assessing risk of bias in a randomized trial. Cochrane Handbook for Systematic Reviews of Interventions version 6.5. 2023. URL: https://training.cochrane.org/handbook/current/chapter-08 [accessed 2024-07-31]
    54. McGuinness LA, Higgins JPT. Risk-of-bias VISualization (robvis): an R package and Shiny web app for visualizing risk-of-bias assessments. Res Synth Methods. Jan 26, 2021;12(1):55-61. [FREE Full text] [CrossRef] [Medline]
    55. Schünemann H, Higgins J, Vist G, Glasziou P, Akl E, Skoetz N, et al. Chapter 14: completing ‘summary of findings’ tables and grading the certainty of the evidence. Cochrane Handbook for Systematic Reviews of Interventions version 6.5. 2023. URL: https://training.cochrane.org/handbook/current/chapter-14 [accessed 2024-09-16]
    56. Cochrane RevMan. The Cochrane Collaboration. URL: https://training.cochrane.org/online-learning/core-software/revman [accessed 2024-09-16]
    57. Luo D, Wan X, Liu J, Tong T. Optimally estimating the sample mean from the sample size, median, mid-range, and/or mid-quartile range. Stat Methods Med Res. Jun 2018;27(6):1785-1805. [CrossRef] [Medline]
    58. Wan X, Wang W, Liu J, Tong T. Estimating the sample mean and standard deviation from the sample size, median, range and/or interquartile range. BMC Med Res Methodol. Dec 19, 2014;14:135. [FREE Full text] [CrossRef] [Medline]
    59. Higgins J, Li T, Deeks J. Chapter 6: choosing effect measures and computing estimates of effect. Cochrane Handbook for Systematic Reviews of Interventions version 6.5. 2023. URL: https://training.cochrane.org/handbook/current/chapter-06 [accessed 2024-09-16]
    60. Miller GE. The assessment of clinical skills/competence/performance. Acad Med. Sep 1990;65(9 Suppl):S63-S67. [CrossRef] [Medline]
    61. Orland MD, Patetta MJ, Wieser M, Kayupov E, Gonzalez MH. Does virtual reality improve procedural completion and accuracy in an intramedullary tibial nail procedure? A randomized control trial. Clin Orthop Relat Res. Sep 2020;478(9):2170-2177. [FREE Full text] [CrossRef] [Medline]
    62. Brateanu A, Strang TM, Garber A, Mani S, Spencer A, Spevak B, et al. Using an adaptive, self-directed web-based learning module to enhance residents' medical knowledge prior to a new clinical rotation. Med Sci Educ. Sep 2019;29(3):779-786. [FREE Full text] [CrossRef] [Medline]
    63. Dolan BM, Yialamas MA, McMahon GT. A randomized educational intervention trial to determine the effect of online education on the quality of resident-delivered care. J Grad Med Educ. Sep 2015;7(3):376-381. [FREE Full text] [CrossRef] [Medline]
    64. House H, Monuteaux MC, Nagler J. A randomized educational interventional trial of spaced education during a pediatric rotation. AEM Educ Train. Apr 2017;1(2):151-157. [FREE Full text] [CrossRef] [Medline]
    65. Kerfoot BP. Learning benefits of on-line spaced education persist for 2 years. J Urol. Jun 2009;181(6):2671-2673. [FREE Full text] [CrossRef] [Medline]
    66. Kerfoot BP, Baker HE, Koch MO, Connelly D, Joseph DB, Ritchey ML. Randomized, controlled trial of spaced education to urology residents in the United States and Canada. J Urol. Apr 2007;177(4):1481-1487. [FREE Full text] [CrossRef] [Medline]
    67. Kerfoot BP, Turchin A, Breydo E, Gagnon D, Conlin PR. An online spaced-education game among clinicians improves their patients' time to blood pressure control: a randomized controlled trial. Circ Cardiovasc Qual Outcomes. May 2014;7(3):468-474. [FREE Full text] [CrossRef] [Medline]
    68. Mallon D, Fei L, Farrell M, Anderson JB, Klein M. Randomized controlled trial of interactive spaced education to support constipation management by pediatric primary care providers. J Pediatr Gastroenterol Nutr. May 01, 2022;74(5):568-574. [FREE Full text] [CrossRef] [Medline]
    69. Matzie KA, Kerfoot BP, Hafler JP, Breen EM. Spaced education improves the feedback that surgical residents give to medical students: a randomized trial. Am J Surg. Feb 2009;197(2):252-257. [FREE Full text] [CrossRef] [Medline]
    70. Pernar LIM, Beleniski F, Rosen H, Lipsitz S, Hafler J, Breen E. Spaced education faculty development may not improve faculty teaching performance ratings in a surgery department. J Surg Educ. 2012;69(1):52-57. [FREE Full text] [CrossRef] [Medline]
    71. Shaw T, Long A, Chopra S, Kerfoot BP. Impact on clinical behavior of face-to-face continuing medical education blended with online spaced education: a randomized controlled trial. J Contin Educ Health Prof. 2011;31(2):103-108. [FREE Full text] [CrossRef] [Medline]
    72. Shaw TJ, Pernar LI, Peyre SE, Helfrick JF, Vogelgesang KR, Graydon-Baker E, et al. Impact of online education on intern behaviour around joint commission national patient safety goals: a randomised trial. BMJ Qual Saf. Oct 2012;21(10):819-825. [FREE Full text] [CrossRef] [Medline]
    73. Grad R, Leger D, Kaczorowski J, Schuster T, Adler S, Aman M, et al. Does spaced education improve clinical knowledge among family medicine residents? A cluster randomized controlled trial. Adv Health Sci Educ Theory Pract. Aug 2021;26(3):771-783. [FREE Full text] [CrossRef] [Medline]
    74. Gyorki DE, Shaw T, Nicholson J, Baker C, Pitcher M, Skandarajah A, et al. Improving the impact of didactic resident training with online spaced education. ANZ J Surg. Jun 2013;83(6):477-480. [FREE Full text] [CrossRef] [Medline]
    75. Jaunay L, Zerr P, Peguin L, Renouard L, Ivanoff A, Picard H, et al. Development and evaluation of a new serious game for continuing medical education of general practitioners (Hygie): double-blinded randomized controlled trial. J Med Internet Res. Nov 20, 2019;21(11):e12669. [FREE Full text] [CrossRef] [Medline]
    76. Teo WZW, Dong X, Yusoff SKBM, Das De S, Chong AKS. Randomized controlled trial comparing the effectiveness of mass and spaced learning in microsurgical procedures using computer aided assessment. Sci Rep. Feb 02, 2021;11(1):2810. [FREE Full text] [CrossRef] [Medline]
    77. Raffoul A, Vitagliano JA, Sarda V, Chan C, Chwa C, Ferreira KB, et al. Evaluation of a one-hour asynchronous video training for eating disorder screening and referral in U.S. pediatric primary care: a pilot study. Int J Eat Disord. Sep 2022;55(9):1245-1251. [FREE Full text] [CrossRef] [Medline]
    78. Li W, Zhang K, Yao S, Xie X, Han W, Xiong W, et al. Simulation-based arthroscopic skills using a spaced retraining schedule reduces short-term task completion time and camera path length. Arthroscopy. Nov 2020;36(11):2866-2872. [FREE Full text] [CrossRef] [Medline]
    79. Kocyigit H, Karagozoglu S. Effects of scenario-based high fidelity and repeated simulation methods on the medical error tendency, self-efficacy and state anxiety levels of nursing students. Int J Caring Sci. 2022;15(2):1168-1177.
    80. Akdemir A, Zeybek B, Ergenoglu AM, Yeniel AO, Sendag F. Effect of spaced training with a box trainer on the acquisition and retention of basic laparoscopic skills. Int J Gynaecol Obstet. Dec 2014;127(3):309-313. [FREE Full text] [CrossRef] [Medline]
    81. Ugwa E, Kabue M, Otolorin E, Yenokyan G, Oniyire A, Orji B, et al. Simulation-based low-dose, high-frequency plus mobile mentoring versus traditional group-based trainings among health workers on day of birth care in Nigeria; a cluster randomized controlled trial. BMC Health Serv Res. Jun 26, 2020;20(1):586. [FREE Full text] [CrossRef] [Medline]
    82. Laynor G. Are spaced repetition study tools changing health professions education? J Electron Resour Med Libr. Aug 08, 2023;20(2-3):94-97. [CrossRef]
    83. Spicer JO, Ujunwa KC, Witt LS, Meka J, Gooding HC. Using instructor-developed study resources to increase evidence-based learning strategies among medical students: a mixed-methods study. Med Teach. Dec 2023;45(12):1380-1386. [FREE Full text] [CrossRef] [Medline]
    84. Wlodarczyk S, Dhaliwal G. The learning sciences meet the learning health system. JAMA Netw Open. Jul 01, 2022;5(7):e2223113. [FREE Full text] [CrossRef] [Medline]
    85. Finn E, Ayres F, Goldberg S, Hortsch M. Brave new e-world: medical students' preferences for and usage of electronic learning resources during two different phases of their education. FASEB Bioadv. May 2022;4(5):298-308. [FREE Full text] [CrossRef] [Medline]
    86. Wu JH, Gruppuso PA, Adashi EY. The self-directed medical student curriculum. JAMA. Nov 23, 2021;326(20):2005-2006. [FREE Full text] [CrossRef] [Medline]
    87. Medical school year two questionnaire 2022 all schools summary report. Association of American Medical Colleges. URL: https://www.aamc.org/data-reports/students-residents/report/year-two-questionnaire-y2q [accessed 2024-01-02]
    88. Gamlin R. Build or buy? Curricular design and external basic sciences learning resources. Acad Med. Apr 2017;92(4):430. [FREE Full text] [CrossRef] [Medline]
    89. Carpenter SK, Cepeda NJ, Rohrer D, Kang SHK, Pashler H. Using spacing to enhance diverse forms of learning: review of recent research and implications for instruction. Educ Psychol Rev. Aug 4, 2012;24(3):369-378. [CrossRef]
    90. Krishnamurthy K, Selvaraj N, Gupta P, Cyriac B, Dhurairaj P, Abdullah A, et al. Benefits of gamification in medical education. Clin Anat. Sep 2022;35(6):795-807. [FREE Full text] [CrossRef] [Medline]
    91. Thillainadesan J, Le Couteur DG, Haq I, Wilkinson TJ. When I say … microlearning. Med Educ. Aug 2022;56(8):791-792. [FREE Full text] [CrossRef] [Medline]
    92. Ng MSP, Jabir AI, Ng TDR, Ang Y, Chia JL, Tan DNH, et al. Evaluating TESLA-G, a gamified, telegram-delivered, quizzing platform for surgical education in medical students: protocol for a pilot randomised controlled trial. BMJ Open. Jun 28, 2023;13(6):e068740. [FREE Full text] [CrossRef] [Medline]

    ‎
    cRCT: cluster randomized controlled trial
    GRADE: Grading of Recommendations, Assessment, Development and Evaluation
    MCQ: multiple-choice question
    PCP: primary care provider
    PRISMA: Preferred Reporting Items of Systematic Reviews and Meta-Analyses
    RR: relative risk
    SMD: standardized mean difference
    USMLE: United States Medical Licensing Examination
    VR: virtual reality

    Edited by N Cahill; submitted 26.02.24; peer-reviewed by A Henri-Bhargava, I Martínez-García, Y Asada; comments to author 21.05.24; revised version received 14.08.24; accepted 26.08.24; published 10.10.24.

    Copyright

    ©Laura Martinengo, Matthew Song Peng Ng, Tony De Rong Ng, Yi-Ian Ang, Ahmad Ishqi Jabir, Bhone Myint Kyaw, Lorainne Tudor Car. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 10.10.2024.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research (ISSN 1438-8871), is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.