This paper is in the following e-collection/theme issue:

Theoretical Frameworks and Concepts (252) Digital Health Reviews (1289) Continuing Medical Education (CME) for Allied Health Professionals (131) Nursing Education and Training (129) New Methods and Approaches in Medical Education (232) Models and Frameworks in Medical Education (30)

Published on in Vol 24, No 3 (2022): March

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/31977, first published .
Digital Education for Health Professionals: An Evidence Map, Conceptual Framework, and Research Agenda

Digital Education for Health Professionals: An Evidence Map, Conceptual Framework, and Research Agenda

Digital Education for Health Professionals: An Evidence Map, Conceptual Framework, and Research Agenda

Authors of this article:

Lorainne Tudor Car1, 2 Author Orcid Image ;   Selina Poon3 Author Orcid Image ;   Bhone Myint Kyaw3 Author Orcid Image ;   David A Cook4 Author Orcid Image ;   Victoria Ward5 Author Orcid Image ;   Rifat Atun6 Author Orcid Image ;   Azeem Majeed2 Author Orcid Image ;   Jamie Johnston7 Author Orcid Image ;   Rianne M J J van der Kleij8 Author Orcid Image ;   Mariam Molokhia9 Author Orcid Image ;   Florian V Wangenheim10 Author Orcid Image ;   Martin Lupton11, 12 Author Orcid Image ;   Niels Chavannes8 Author Orcid Image ;   Onyema Ajuebor13 Author Orcid Image ;   Charles G Prober7 Author Orcid Image ;   Josip Car2, 3 Author Orcid Image
Article Authors Cited by (26) Tweetations (21) Metrics

    Review

    1Lee Kong Chian School of Medicine, Nanyang Technological University, Singapore, Singapore

    2Department of Primary Care and Public Health, School of Public Health, Imperial College London, London, United Kingdom

    3Centre for Population Health Sciences, Lee Kong Chian School of Medicine, Nanyang Technological University, Singapore, Singapore

    4Office of Applied Scholarship and Education Science, School of Continuous Professional Development, Mayo Clinic College of Medicine and Science, Division of General Internal Medicine, Mayo Clinic, Rochester, MN, United States

    5Department of Pediatrics, Lucile Packard Children’s Hospital, Stanford, CA, United States

    6Department of Global Health and Population, Harvard T.H. Chan School of Public Health, Harvard, Boston, MA, United States

    7Stanford Center for Health Education’s Digital MedIC Initiative, Stanford School of Medicine, Stanford University, Stanford, CA, United States

    8Department of Public Health and Primary Care, Leiden University Medical Centre, Leiden, Netherlands

    9School of Population Health and Environmental Sciences, Faculty of Life Sciences and Medicine, King’s College London, London, United Kingdom

    10Department of Management, Technology, and Economics, ETH Zurich, Zurich, Switzerland

    11Faculty of Medicine, Imperial College London, London, United Kingdom

    12The Chelsea and Westminster Hospital, Chelsea, London, United Kingdom

    13Health Workforce Department, World Health Organization, Geneva, Switzerland

    Corresponding Author:

    Josip Car, MD, PhD

    Centre for Population Health Sciences

    Lee Kong Chian School of Medicine

    Nanyang Technological University

    11 Mandalay Road, Clinical Science Building, Novena

    Singapore, 308232

    Singapore

    Phone: 65 69041258

    Email: josip.car@ntu.edu.sg


    Background: Health professions education has undergone major changes with the advent and adoption of digital technologies worldwide.

    Objective: This study aims to map the existing evidence and identify gaps and research priorities to enable robust and relevant research in digital health professions education.

    Methods: We searched for systematic reviews on the digital education of practicing and student health care professionals. We searched MEDLINE, Embase, Cochrane Library, Educational Research Information Center, CINAHL, and gray literature sources from January 2014 to July 2020. A total of 2 authors independently screened the studies, extracted the data, and synthesized the findings. We outlined the key characteristics of the included reviews, the quality of the evidence they synthesized, and recommendations for future research. We mapped the empirical findings and research recommendations against the newly developed conceptual framework.

    Results: We identified 77 eligible systematic reviews. All of them included experimental studies and evaluated the effectiveness of digital education interventions in different health care disciplines or different digital education modalities. Most reviews included studies on various digital education modalities (22/77, 29%), virtual reality (19/77, 25%), and online education (10/77, 13%). Most reviews focused on health professions education in general (36/77, 47%), surgery (13/77, 17%), and nursing (11/77, 14%). The reviews mainly assessed participants’ skills (51/77, 66%) and knowledge (49/77, 64%) and included data from high-income countries (53/77, 69%). Our novel conceptual framework of digital health professions education comprises 6 key domains (context, infrastructure, education, learners, research, and quality improvement) and 16 subdomains. Finally, we identified 61 unique questions for future research in these reviews; these mapped to framework domains of education (29/61, 47% recommendations), context (17/61, 28% recommendations), infrastructure (9/61, 15% recommendations), learners (3/61, 5% recommendations), and research (3/61, 5% recommendations).

    Conclusions: We identified a large number of research questions regarding digital education, which collectively reflect a diverse and comprehensive research agenda. Our conceptual framework will help educators and researchers plan, develop, and study digital education. More evidence from low- and middle-income countries is needed.

    J Med Internet Res 2022;24(3):e31977

    doi:10.2196/31977

    Keywords

    digital education; health professions education; evidence map; systematic review; research questions; conceptual framework; mobile phone 


    The world is faced with a shortage and an unequal distribution of health workforces across low-income, middle-income, and high-income countries [1]. The shortfalls and inequitable distributions affect the likelihood of reaching the United Nations’ third Sustainable Development Goal—health and well-being for all by 2030 [1-5]. To enable an increase in and a more equitable distribution of competent health workforce, there is a need for more effective and accessible health professions education.

    The use of digital technology in health professions education can help in overcoming some of the health workforce–related challenges by providing more accessible, standardized, relevant, timely, and affordable medical education and training [6,7]. Until recently, digital education was perceived as primarily supporting in-person health professions education [8]. The social distancing measures introduced to control the COVID-19 pandemic have dramatically changed the delivery of health professions education worldwide. Many medical schools and health professions education institutions had to pivot to digital education [9,10]. With this sudden shift, research and evidence in digital health professions education have become even more important.

    The evidence on digital education has grown substantially in recent years and has been the subject of many systematic reviews. Existing reviews seem to mostly focus on the effectiveness of different digital education modalities [11-16]. However, the adoption of digital education is complex and includes other research questions, in addition to its effectiveness. It is important to identify evidence that already exists and evidence gaps across the full scope of relevant questions to inform and guide future research and reduce research waste. To address this need, we seek to (1) create a map of existing research, (2) develop a conceptual framework outlining key components of digital education, and (3) highlight specific research questions across a comprehensive research framework. We do this by systematically identifying and analyzing previous systematic reviews on digital health education.


    We used an evidence map methodology to identify and summarize systematic reviews on digital health professions education [17]. We also developed a novel conceptual framework using an established methodological approach [18] and identified specific research questions in alignment with this conceptual framework.

    Study Identification

    To identify relevant systematic reviews on different types of digital health education for health professionals, we used a comprehensive search strategy mentioned in Multimedia Appendix 1, including key terms for participants (eg, health professionals, health personnel, and students), intervention (eg, e-learning, patient simulation, and serious games), and article type (eg, systematic reviews and evidence synthesis). We searched the following major bibliographic databases for studies published between January 1, 2014, and July 21, 2020, without any restrictions on language or study design: MEDLINE (Ovid), Embase (Ovid), Cochrane Library, Educational Research Information Center (EBSCO), and CINAHL (EBSCO). We also manually checked the reference lists of the included systematic reviews for other potentially relevant systematic reviews on digital education for health professionals. In addition, we searched Google, Google Scholar, ResearchGate, and OpenGrey for any other studies that might be relevant for our topic, using keywords systematic review, digital education, health professionals, health professions education, eLearning, and e-learning, and reviewing either the first 10 pages or 500 results.

    Eligibility Criteria

    We included systematic reviews focusing on digital education for health care professionals in preservice (ie, student) and in-service (ie, postdegree, including postgraduate trainees and those in independent practice) positions [19]. This includes disciplines such as medicine, dentistry, nursing and midwifery, medical diagnostic and treatment technology, physiotherapy and rehabilitation, and pharmacy. Practitioners of traditional, alternative, and complementary medicine were excluded. Digital health professions education refers to health professions education that is conducted using digital technology [20] and includes modalities ranging from the basic conversion of content into a digital format (eg, a book converted into a PDF or HTML format) to more complex applications such as mobile education, digital games, virtual patients, and virtual reality (VR). Systematic reviews were included if they focused on ≥1 modality of digital education (as defined in Textbox 1) delivered in a stand-alone or blended format [20]. We defined blended education as education that incorporates aspects of traditional and digital education. Traditional education was defined as education that encompasses the use of nondigital educational materials (eg, textbooks or models) or in-person human interactions. We included systematic reviews of all the types of studies. We excluded older reviews because of the rapid evolution of the field, with the assumption that most of the active research questions from the reviews published >5 years ago would be collated in more recent reviews.

    Digital education technologies and modalities and working definitions and descriptions.

    Offline digital education

    • Education delivery requires no internet or local area network connection and can be delivered through external media, including CD-ROM, external hard disk, and USB stick [21].

    Online digital education

    • Computer-assisted instruction using the internet or a local intranet as the means of delivery, also referred to as online, internet-based, or networked [22,23], includes multiple media formats (eg, text, videos, and images and online discussion (eg, via email, chat, or videoconferencing) and is designed to be primarily delivered on PCs.

    Massive open online course

    • A (free) online course available over the internet to a large number of geographically dispersed participants [24]

    Mobile education (m-Learning)

    • Flexible and accessible learning delivered via personal mobile devices, such as smartphones and tablets [25]

    Serious gaming and gamification

    • Knowledge and training activities are set within a competitive activity. Games are intended to promote the development of knowledge, cognitive skills, or psychomotor skills in a virtual environment [26].

    Virtual reality

    • Interactive exploration of a digital (3D) multimedia environment can reflect a real-world environment (eg, clinic) or an artificial or surreal context (eg, positioning the learner within the human body) [16,27].

    Virtual patient

    • A computer program that simulates real-life clinical scenarios where students take on the role of a health professional and obtain a patient’s history, conduct a physical examination, and make diagnostic and therapeutic decisions [28]

    High-fidelity manikins

    • Realistic, computerized mannequins that mimic elements of human physiology (eg, breathing and heart rhythm) and are used to simulate a real-life clinical scenario [29].

    Blended education

    • The use of digital education modalities in combination with traditional education methods

    Traditional education

    • Education that uses nondigital educational material (eg, textbook or model) or in-person human interaction
    Textbox 1. Digital education technologies and modalities and working definitions and descriptions.

    Study Selection

    The search results from different databases were combined in a single EndNote library, and duplicate citations were removed. A total of 2 review authors (SP and BMK) screened all titles and abstracts for inclusion independently and in duplicate. Disagreements during the title and abstract screening were resolved by consensus. Full texts of articles considered eligible or uncertain based on the title and abstract screening were retrieved and screened independently and in duplicate by the same 2 authors.

    Data Extraction

    From the included systematic reviews, 2 authors (SP and BMK) used a standardized form to independently extract information on the review aim; the study design, participants, interventions, comparators, and outcomes of the original research studies included; the method used to appraise the quality of the included studies (eg, risk of bias) or overall evidence (eg, the Grading of Recommendations Assessment, Development and Evaluation assessment); and recommendations for future research. We classified the outcomes according to the definitions presented in Multimedia Appendix 2. We also extracted information on all additional outcomes reported in the included reviews that did not correspond to our predefined outcome-related definitions. We classified the systematic review in terms of the single digital modality they most focused on, according to our framework (Textbox 1). In most instances, it was clear that a given review focused predominantly on 1 modality. Less often, a review encompassed multiple modalities equally, in which case we classified it as digital education; that is, the use of digital technology in health professions education in general. Finally, we identified recommendations for future research by extracting exact quotes from each review that articulated such recommendations. At every stage, disagreements between the review authors were resolved through discussion and input from the third author (LTC).

    Analysis

    Authors SP and BMK rephrased the quoted research recommendations into research questions and then refined these by applying consistent terminology, removing duplicates, and merging or subdividing themes. Questions that focused on specific digital modalities (eg, online modules) were rewritten to make them relevant to the broader research agenda for digital health education. The final list of research questions was refined by LTC.

    We developed a conceptual framework outlining various digital health professions’ education components according to the methodology described by Jabareen [18]. We consulted and built on our previous conceptual work in this area and existing frameworks for the implementation or adoption of digital education generally [30-39]. We identified these frameworks through a focused literature search on PubMed, Google, and Google Scholar. On the basis of the discussions and consensus among the review authors, key domains and subdomains were finalized. The framework and its components were represented diagrammatically. Recommendations for future research were classified according to the proposed framework in parallel by 2 authors (SP and BMK). Discrepancies were resolved through consensus and with the guidance of the third author (LTC). On the basis of the analysis of the included reviews and the observed gaps in the literature, we outline a research agenda for digital health professions education and present it in the Results section.


    Search Results

    Of 7294 systematic reviews from our initial search, we identified 73 (1%) eligible systematic reviews (Figure 1). Another 4 systematic reviews were identified through Google, Google Scholar, ResearchGate, and OpenGrey. In total, 77 systematic reviews were included for data extraction.

    ‎
    Figure 1. Study flow diagram. ERIC: Educational Research Information Center.
    View this figure

    The number of published systematic reviews increased over time, from 6% (5/77) published in 2014 to 56% (43/77) published in 2019. The systematic reviews focused on digital education for health professions students (17/77, 22% studies), postgraduate or independently practicing (18/77, 23% studies) health professionals, or both (42/77, 55% studies). Most of the systematic reviews were on digital education in surgery (13/77, 17% studies) or health professions education in general (ie, those not focusing on a particular type of practitioner, health care area, or topic; 36/77, 47% studies), followed by nursing (11/77, 14% studies; Table 1; Multimedia Appendix 3 [11-16,32,34,40-109]).

    A breakdown of the digital modalities being investigated in the included systematic reviews is shown in Figure 2. Most systematic reviews focused on digital education in general (22/77, 29%), VR (19/77, 15%), and online education (10/77, 13%). Of the 19 reviews on VR, 17 (89%) were on VR complemented with physical objects or devices such as probes or handles and focused on psychomotor, procedural, or technical skill development. There were fewer reviews published on m-Learning (6/77, 8%), digital game-based learning (3/77, 4%), and virtual patients (2/77, 3%).

    The most common outcomes in the included reviews were health professionals’ knowledge (49/77, 64%), skills (51/77, 66%), attitudes about the clinical topic (13/77, 17%), and satisfaction with digital education (18/77, 23%). Most systematic reviews compared the effectiveness of digital education to traditional education (ie, nondigital; 59/77, 77%) or other forms of digital education (35/77, 45%; Table 1). Most reviews reported only immediate, short-term outcomes; 22% (17/77) of reviews reported the impact of digital education on long-term delayed outcomes; that is, outcomes assessed with delay after the intervention [34,40-55]. Most reviews appraised methods using the Risk of Bias tool [110] only (24/77, 31%), followed by Grading of Recommendations, Assessment, Development, and Evaluation [111] (which also includes the risk of bias assessment; 22/77, 29%) and Medical Education Research Study Quality Instrument [112] (10/77, 13%). Of the 77 studies, 9 (12%) reviews did not report on the quality appraisal of the included evidence, whereas the remaining 14 (18%) reviews used different instruments to determine the evidence quality (Table 1; Multimedia Appendix 3 [11-16,32,34,40-107,109]). The included reviews mostly reported original research to be low or very low quality of evidence or reported unclear or high risk of bias in most studies (Multimedia Appendix 3 [11-16,32,34,40-107,109]). Most systematic reviews reported data from high-income countries (14/77, 18% systematic reviews) or both middle- and high-income countries (42/77, 55% systematic reviews). Only 4% (3/77) of the systematic reviews included studies from low-income countries [11,56,57]. Approximately 29% (22/77) of the included reviews did not report the setting of the included studies.

    Table 1. Characteristics of the included systematic reviews (N=77).
    Characteristics of the systematic reviews and the evidence they includeStudies, n (%)
    Type of participants

    		Medical students5 (6)

    		Medical students and physicians9 (12)

    		Physicians17 (22)

    		Dentistry students3 (4)

    		Dentistry students and dentists2 (3)

    		Nursing students8 (10)

    		Nursing students and nurses3 (4)

    		Mixed students2 (3)

    		Mixed students and HCPsa19 (25)

    		Mixed HCPs9 (12)
    Level of education

    		Postdegree: practicing HCPs10 (13)

    		Postdegree: traineesb5 (6)

    		Postdegree: mix of traineesb and practicing HCPs3 (4)

    		Student17 (22)

    		Mixed student and postdegree42 (55)
    Clinical topics

    		General health professions education23 (30)

    		Surgery14 (18)

    		Nursing8 (10)

    		Life support or trauma management (resuscitation skills)3 (4)

    		Radiology7 (9)

    		Endoscopy3 (4)

    		Other19 (25)
    Setting

    		High-income countries only26 (34)

    		High-income and middle-income countries45 (58)

    		High-, middle-, and low-income countries4 (5)

    		Middle- and low-income countries1 (1)

    		Information not available1 (1)
    Modality

    		Digital education22 (29)

    		Virtual reality19 (25)

    		Online10 (13)

    		Offline6 (8)

    		Mobile learning6 (8)

    		High-fidelity manikins5 (6)

    		Online and offline4 (5)

    		Digital serious games3 (4)

    		Virtual patient2 (3)
    Comparison

    		No intervention25 (32)

    		Traditional education56 (73)

    		Digital intervention35 (45)

    		Other6 (8)
    Quality appraisal

    		Risk of bias24 (31)

    		Grading of Recommendations, Assessment, Development, and Evaluations22 (29)

    		Medical Education Research Study Quality Instrument10 (13)

    		Best Evidence in Medical Education reviews2 (3)

    		The Jadad scale2 (3)

    		Joanna Briggs Institute Meta-Analysis of Statistics Assessment and Review Instrument2 (3)

    		Methodological index for non-randomized studies1 (1)

    		Newcastle–Ottawa Scale1 (1)

    		Other5 (6)

    		Not reported8 (10)
    Outcomes

    		Knowledge49 (64)

    		Skills51 (66)

    		Satisfaction18 (23)

    		Patient outcomes20 (26)

    		Performancec19 (25)

    		Attitude13 (17)

    		Behavioral8 (10)
    Number of included studies in the review

    		<1024 (31)

    		10-1927 (35)

    		20-2910 (13)

    		30-398 (10)

    		≥408 (10)
    Study designs included in the review

    		Randomized controlled trials68 (88)

    		Other experimental studiesd8 (10)

    		Cross-sectional studies5 (6)

    		Qualitative studies3 (4)

    		Pre-post studies12 (16)

    		Cohort studies8 (10)

    		Other or mixede34 (44)
    The conceptual framework domain or subdomain the reviews focus on

    		Education—design77 (100)

    		Education—content77 (100)

    		Education—evaluation9 (12)

    		Education—pedagogy5 (6)

    		Education—engagement3 (4)

    		Context—settings1 (1)

    aHCP: health care professional.

    bIncludes residents, novices, trainees, and fellows.

    cDefined in the included systematic reviews as a combination of skills and behavioral changes as a result of the intervention.

    dIncludes quasi-randomized controlled trials, nonrandomized controlled trials, before-and-after studies, and interrupted time series designs.

    eIncludes study designs not described above or a combination of different study designs.

    ‎
    Figure 2. The number of systematic reviews on different digital modalities according to the year of publication. m-Learning: mobile learning.
    View this figure

    Conceptual Framework of Digital Health Professions Education

    To outline different aspects of digital health professions education and identify gaps in the literature, we developed a novel conceptual framework (Figure 3) grounded in key findings of these systematic reviews together with 7 existing frameworks for digital education general [35-39,113-117] and a framework we developed previously for health professions education [33].

    Broadly, the fundamental domains include an enabling and supportive context, sound infrastructure, and the optimal use of education tools and processes. The context is a combination of institutional norms, sociocultural norms, and settings in which the learner resides, as well as the level of education the learner is at. Subdomains of the context have a direct impact on the infrastructure components required and available for the delivery of digital education—digital and physical spaces, policies and regulatory standards, and human resources. Both context and infrastructure components are important in consideration of health professions education. Learners, individually and as part of a larger group, are at the core of digital health education, and their needs, preferences, prior expertise, and competencies should shape how education is delivered. The interaction among components within and across each layer is dynamic, with different parts being interconnected, as reflected using dotted lines to separate context, infrastructure, education, and learners. Studying and identifying optimal relationships between the components are handled by the research and quality assurance blocks, which are connected to the rest of the framework. Table 2 provides the detailed operational definitions for each domain of the framework.

    ‎
    Figure 3. Conceptual framework of digital health education for healthcare professionals. CME: continuing medical education; CPD: continuing professional development; IT: information technology.
    View this figure
    Table 2. Definitions of digital health professions education conceptual framework components.
    Domain and subdomainDefinition
    Context

    		Sociocultural normsThe acceptability and adoption of digital education as a form and norm of education within the society

    		Institutional normsThe acceptability, impact, considerations, and processes concerning the adoption of digital education at the institutional level

    		SettingsThe setting in which digital health education is conducted or implemented, including clinical or classroom environments; low-, middle-, and high-income countries; and rural or urban environments

    		Level of educationThe impact and integration of digital education with other forms of education (eg, inter- and intraprofessional training opportunities) and clinical work in which participants are engaged
    Infrastructure

    		PhysicalThe physical learning space within which the in-person component of blended digital health education is taking place

    		DigitalThe information and communication technology devices (both hardware and software) to support and create learning environments (virtual environments, digital networks, technological modifications) or media for digital health education, as well as the speed and capacity of internet access

    		RegulatoryPolicies and regulatory standards for health professionals’ licensing and accreditation, as well as those relating to the design and delivery of digital health professions education

    		Human resourcesThe human resources required for digital health education to be maintained and sustained, including educators, administrators, and information technology staff
    Education

    		ModalityThe choice and configuration of digital education modality (eg, online learning and m-Learning) and its potential blending with in-person education

    		Instructional designThe method and practice of digital health professions education encompassing teaching strategies, learning principles, learning outcomes, and the assessment approach

    		ContentHealth professions education area, discipline, theme, or topic delivered via digital education

    		EngagementThe level of communication, interactivity, or immersion of participants taking part in digital health professions education

    		AssessmentMeasurement of digital health professions education conducted at the individual and institutional level to determine its impact on educational and clinical outcomes
    LearnersHealth professionals with distinctive needs, competencies, digital literacy, knowledge, skills, and attitudes toward working and learning, both individually or as a group
    ResearchSystematic study of digital health professions education to create and disseminate new knowledge and allow for more effective and efficient adoption, implementation, and transfer of interventions to various contexts—this encompasses experimental, observational, descriptive, and qualitative research
    Quality assuranceA context-specific and systematic evaluation of practices and procedures to understand the current state and improve the performance of digital health education in a particular setting

    Research Questions From the Included Systematic Reviews

    We identified 318 discrete research questions posed in these 77 articles, from which we distilled a final list of 61 (19.2%) distinct questions covering 14 of the 16 subdomains of the above framework (Table 2; Multimedia Appendix 4 [11-16,32,34,40-95,98-107]). Research questions that spanned multiple subdomains were assigned by investigator consensus to the most relevant single subdomain. None of the included systematic reviews posed questions primarily directed at the physical infrastructure or quality assurance subdomains. We identified 26% (16/61) of questions relating to context, classified into four subdomains: sociocultural norms, institutional norms, settings, and level of education. Approximately 15% (9/61) of research questions (3 per subdomain) were identified for the digital, regulatory, and human resources subdomains within the infrastructure domain. Most of the research questions, 48% (29/61) approximately, were categorized in the education domain, which encompasses modality, instructional design, content, engagement, and evaluation subdomains. Approximately 5% (3/61) of research questions each were categorized in the learners and research domains.

    Classifying Research Questions Addressed by Existing Systematic Reviews

    We also classified the included systematic reviews based on their research questions and using our conceptual framework (Table 3; Multimedia Appendix 3 [11-16,32,34,40-107,109]). The research questions addressed by existing systematic reviews mostly revolved around digital education modality (ie, the effectiveness of various digital education modalities delivered as stand-alone or blended interventions) and content (ie, the effectiveness of digital education within a particular health care area or discipline). Some reviews assessed interactivity (engagement), various aspects of instructional design in digital education, the impact of digital education on institutional outcomes (context—institutional norms), and the impact of setting (eg, low-income and middle-income countries) on learning outcomes.

    Table 3. Research questions identified from the included systematic reviews on health professions digital education.
    Research questions identified from included systematic reviewsConceptual framework domain (subdomain)Systematic reviews’ references
    How do cost and cost-related outcomes influence the adoption of digital technology in health professions education?Context (sociocultural norms)[41,56,58-65]
    How can policy makers be organized to adopt digital education as part of health professions education?Context (sociocultural norms)[56,66]
    How do cultural factors within different countries determine the use of digital education for health professions training?Context (sociocultural norms)[66]
    How does providing access to digital education improve the learning outcomes of health professionals?Context (sociocultural norms)[14,40,41,43,46,53,56,57,66-77]
    What is the long-term cost-effectiveness of digital education compared with traditional education for health professionals?Context (institutional norms)[12-14,16,47,61,78]
    How does health professions’ digital education affect individual and health services outcomes and organizational practice?Context (institutional norms)[11-14, 16, 32, 34, 40, 42-45, 48, 49, 51-53, 56, 60, 62, 64, 67-70, 75, 79-88]
    Is health professions’ digital education more time efficient than traditional education?Context (institutional norms)[46]
    What is the feasibility of implementing digital technology for health professions education in different socioeconomic settings?Context (setting)[13,14,16,43,56,57,78,89,90]
    What are the short- and long-term effects of using digital technology for health professions education in different socioeconomic settings?Context (setting)[32,43,47,50,54,60,78,82,89-91]
    Is digital education for health professionals effective in different socioeconomic settings?Context (setting)[13,40,91,92]
    What are the resource requirements to implement digital education in different socioeconomic settings?Context (setting)[85,93]
    What are the challenges of setting up digital education for health professionals training in different socioeconomic settings?Context (setting)[85]
    What is the differential impact of digital education on the clinical performance of trainee or expert surgeons?Context (level)[94]
    How can digital education for health professionals be integrated into normal work practices?Context (level)[68]
    How can digital technology be incorporated into current health professions’ education and training curriculum to improve learning outcomes?Context (level)[42,46,47,54,61,62,78,90]
    Is digital education effective in improving health professionals’ knowledge and skills performance in the clinical setting?Context (level)[11-14, 16, 32, 34, 40, 42-45, 48, 49, 51-53, 56, 60, 62, 64, 67-70, 75, 79-88]
    Which features of digital education (eg, technical features, fidelity, safety, and adaptability) affect the learning outcomes of health professions education?Infrastructure (digital)[13,95]
    What are the minimum requirements for the digital technology used to achieve the effectiveness of digital health professions education?Infrastructure (digital)[85]
    What are the technical resources needed to deliver digital education to health care professionals?Infrastructure (digital)[61]
    How should educators delivering digital health education be assessed and accredited?Infrastructure (regulatory)[47]
    What are the best practices for the development, evaluation, and use of digital health education in health professions education?Infrastructure (regulatory)[14]
    Is the use of accreditation-related milestones in digital health education effective?Infrastructure (regulatory)[78]
    What digital skills should instructors facilitating digital health education be competent in?Infrastructure (human resources)[47]
    How does the digital competence of teachers affect health professions learning outcomes from digital health education?Infrastructure (human resources)[96]
    What are the workforce resources needed for health professions’ digital education?Infrastructure (human resources)[61]
    What type of instructional design is used in the effective digital education of health professions education?Education (modality)[47,57,83,87,94]
    Which components of digital health education (eg, interactivity and feedback) contribute to enhanced learning outcomes?Education (modality)[45,52,58,67,97]
    What is the optimal use of video-assisted debriefing for health professionals’ simulation-based training?Education (modality)[98]
    How does the design of digital education interventions (eg, format and modality used) in health professions education and training curriculum affect learning outcomes?Education (modality)[34,42,53,64,74,78,93]
    Can digital simulation-based training be used to train nontechnical skills in health professionals?Education (modality)[44,69]
    What is the effectiveness of digital education (mixed or single modality) compared with nondigital education to deliver health professions education?Education (modality)[42,71,98]
    Can digital education complement (ie, blended) or substitute traditional education for health professionals?Education (modality)[54,69,99,100]
    Does digital simulation-based psychomotor skills training provide any benefit to the medical trainee?Education (modality)[46]
    What are the barriers to obtaining digital education materials for health professions education training, and how can they be overcome?Education (content)[66]
    What content should be included in debriefing (eg, digital data) following simulation-based education to achieve improved clinical outcomes?Education (content)[47]
    Can digital education be used to overcome challenges in delivering content-specific topics for health professions education (eg, surgical training in rare pathologic states)?Education (content)[84,89]
    Can digital education be designed to achieve learning outcomes denoted in the Kirkpatrick model?Education (instructional design)[101]
    What learning theories can be used to inform the development of effective digital health professions education?Education (instructional design)[13,14,55,59,63,82,93]
    Is mastery learning via digital education more or as effective as traditional education in terms of clinical psychomotor skills improvement?Education (instructional design)[41,47,48,53,58-60,78,102]
    Is spacing digital simulation–based training more or as effective as traditional education in clinical psychomotor skills development?Education (instructional design)[41,47,48,53,58-60,78,102]
    How does the frequency and duration of digital simulation–based psychomotor skills training affect health professionals’ skills transfer to the clinical setting?Education (instructional design)[41,47,48,53,58-60,78,102]
    What are the optimal duration, frequency, and intensity of digital health professions education programs to affect the learning and clinical outcomes of health professionals?Education (instructional design)[43,54,62,72,83,94,103]
    What pedagogy should be used in the digital education of health professionals to improve their knowledge and skills?Education (instructional design)[11,14,42,95,104]
    What is the effectiveness of using digital education to train and assess nontechnical skills in health care professionals?Education (instructional design)[71,87]
    What is the effectiveness of digital problem–based learning in health professions education?Education (instructional design)[34]
    How does the interactivity of digital education programs affect the learning and clinical outcomes of health professionals?Education (engagement)[53,62,80,91]
    What is the minimal level of haptic feedback required in digital simulation-based training programs to improve health professionals’ psychomotor skills?Education (engagement)[64]
    What are learners’ acceptability of digital education with different levels of interactivity?Education (engagement)[77]
    Which performance metrics or measurement instrument should be used to assess health professionals’ knowledge, skills, attitudes, satisfaction, and clinical outcomes from digital technology–based training programs?Education (assessment)[12, 14, 44, 45, 51-53, 60, 62, 64, 67-71, 73-75, 77, 78, 83, 87, 90, 92, 93, 95, 102, 103, 105]
    What is the ideal approach to assessing health professionals’ knowledge, skills, attitudes, satisfaction, and clinical outcomes from digital technology–based education and training programs?Education (assessment)[12, 14, 44, 45, 51-53, 60, 62, 64, 67-71, 73-75, 77, 78, 83, 87, 90, 92, 93, 95, 102, 103, 105]
    Should the evaluation of digital health education include behavior and clinical outcomes?Education (assessment)[12, 14, 44, 45, 51-53, 60, 62, 64, 67-71, 73-75, 77, 78, 83, 87, 90, 92, 93, 95, 102, 103, 105]
    What is the impact of digital simulation–based training on clinical outcomes in the short and long term?Education (assessment)[71,106]
    How should learning outcomes in the field of digital health professions education be defined and standardized?Education (assessment)[92]
    How does the use of digital education affect health professionals’ clinical decision-making at the point of care?Education (assessment)[68]
    How do health professionals’ prior learning experiences influence the topics that will benefit from the use of digital education?Learner[107]
    What are health professionals’ attitudes toward digital delivery of education and training programs?Learner[16,65,85,91]
    What are health care professionals’ learning needs, and can they be met by the use of digital simulation training?Learner[44]
    What are the methodological requirements for studies assessing digital health education?Research[12, 16, 48, 53, 58, 59, 63, 65, 66, 70, 71, 74, 82, 83, 92, 95, 97]
    How should studies on digital health professions education be reported?Research[12, 16, 48, 53, 58, 59, 63, 65, 66, 70, 71, 74, 82, 83, 92, 95, 97]
    How should studies of digital health professions education be designed to ensure the generalizability of their findings across different settings?Research[12, 16, 48, 53, 58, 59, 63, 65, 66, 70, 71, 74, 82, 83, 92, 95, 97]
    What are the barriers and facilitators that affect the continued adoption of digital tools in health professions education?Context, education, infrastructure, and learner[68]

    Principal Findings

    We present an evidence map of 77 systematic reviews on digital education for health professionals published between 2014 and July 2020. The reviews mostly focused on the effectiveness of various digital education modalities in surgery, health professions education in general, and nursing. Most reviews have focused on online and offline learning. Only a few reviews focused on other digital education modalities such as m-Learning, VR, digital game–based learning, and virtual patients. We developed a novel conceptual framework outlining key components of digital health professions education, namely context, infrastructure, education, learner, research, and quality assurance. Within these reviews, we identified 61 unique recommendations (questions) for future research, focusing primarily on digital education modality, instructional design, and assessment.

    Limitations and Strengths

    Our study has some limitations. First, to cover the most recent evidence in the field of digital education for health care professionals, we excluded studies before 2014; earlier reviews might have identified important research questions that remain unanswered. Our focus on systematic reviews also excluded other article types, such as editorials or viewpoints, which might have identified additional research questions. Second, although extraction and classification of research questions were done in duplicate and using a standardized approach, other classifications could be justified in some instances, which implies a degree of imprecision in the reported frequencies of specific questions. Moreover, our method did not allow us to prioritize the numerous research questions; such prioritization would require input from a representative group of experts and could be the focus of a future study. Third, there are overlaps among different concepts specified within this conceptual framework, which could be delineated and presented differently depending on potential chosen emphasis or entry points. Fourth, reviews classified as online education varied substantially in their inclusion of other modalities (eg, some expressly excluded modalities such as virtual patients, digital games, or massive open online courses, whereas others included these and other modalities). Finally, our novel conceptual framework may require revision as our understanding of this field matures and evolves, additional evidence accumulates, and new technologies emerge.

    Our study also has several strengths, such as a thorough literature search for relevant studies, encompassing several indexed and gray literature databases without restrictions. We followed an established evidence map methodology and performed the steps in duplicate and independently [17]. In the development of our conceptual framework, we drew from the existing frameworks, our previous work, and discussions with experts.

    Integration With Prior Work

    We did not find other frameworks presenting a high-level overview of the use and implementation of digital education for health professionals. Therefore, we drew from the general digital education literature and found several relevant frameworks.(Multimedia Appendix 5). The included frameworks, while providing an overview of digital education, often had additional objectives such as exploring the role of specific stakeholders (eg, the private sector or the ministries), identifying barriers to adoption, or analyzing a particular digital education aspect, setting, or configuration [33,35-37,113-117]. Our framework complements other frameworks by pulling together domains previously presented only in isolation and by adding novel subdomains such as the impact of training levels, the role of regulations and accreditations, and the importance of physical infrastructure (Multimedia Appendix 4 [11-16,32,34,40-95,98-107]).

    Several viewpoint articles have offered research agendas for digital health professions education [28,118-120]. They focus primarily on the design of interventions and research studies in this field, which correspond to the domains of education (modality, instruction design, assessment, and engagement), research (quality of methods and reporting), and context (setting and level of education) in our framework. The agendas espoused in these viewpoints include questions that probe more narrowly and deeply on specific issues relevant to the design and focus of future studies (eg, the choice of comparison intervention and avoidance of confounding, integration of digital education across different institutions, and the need for interdisciplinary collaboration). Our framework was intentionally broad and comprehensive and enabled us to accommodate a variety of additional questions on previously neglected topics.

    Implications for Research and Practice

    Most reviews in our evidence map focused on the effectiveness of digital education interventions and rarely addressed issues around their implementation and adoption. These reviews also mostly compared the effectiveness of digital interventions with that of nondigital education. Findings from studies comparing digital and nondigital education have limited generalizability as these studies cannot account for variance within and between these 2 educational formats [118]. Future research should compare different digital education modalities as such studies are more likely to generate meaningful, generalizable findings. It should also aim to explore potential challenges related to the implementation and adoption of digital education interventions in different settings.

    There is also a need for more methodologically robust research and clearer terminology in this field. The quality of the evidence, as reported in the included reviews, was relatively low, with a limited number of studies measuring skills and knowledge retention. Furthermore, it was at times difficult to determine which modality (or modalities) the included reviews focused on because of poorly explained inclusion criteria. Such ambiguity was particularly common in reviews on e-learning and blended, online, and internet-based education.

    We also express concerns about the paucity of studies from low- and middle-income countries. Such countries could greatly benefit from digital education, especially by using free or low-cost education (eg, massive open online courses). Although some research findings have a universal application (eg, fundamental principles of effective learning), others (such as implementation, infrastructure, and learners) are more context specific. Given the presence of unique needs of low- and middle-income countries (eg, distinct content priorities, learner demographics, and infrastructure), we urge more research in these contexts.

    Our conceptual framework will benefit researchers, funding agencies, and educators, among others. The specific questions identified and classified according to this framework provide a map for future research and can help prioritize original research studies and guide the planning of new or updated systematic reviews. We encourage investigators to broadly consider the questions we identified in this evidence map, especially those specific to areas previously less studied, such as infrastructure, learners, or quality assurance in digital education. Our framework can also be used by funding agencies to better understand the limitations of the existing research and identify areas with limited evidence with the aim of informing their funding calls in this field. Finally, this framework can encourage those developing new courses to anticipate and plan for issues that are important but might be inadvertently overlooked, such as the digital education context, infrastructure, and learners.

    Acknowledgments

    The authors would like to acknowledge the funding support from Nanyang Technological University, Singapore. The authors would also like to acknowledge Ms Thai My Linh and Ms Christina Tan En Hui for their help with the validation of the findings, Ms Chong Zhuo En for help with data compilation, and Ms Yasmin Lynda Munro for providing advice on our search strategy. Although this review includes the findings from the systematic reviews published by members of the Digital Health Education Collaboration, all reviews were given equal treatment during the review process conducted for this publication. The authors alone are responsible for the views expressed in this paper and do not necessarily represent the views, decisions, or policies of the institutions with which they are affiliated.

    Authors' Contributions

    JC, LTC, and OA conceived the idea for the review. SP and BMK screened the studies, extracted the data, and synthesized the findings. LTC, SP, and BMK wrote the manuscript. JC, DAC, VW, RA, AM, JJ, RMJJvdK, MM, FVW, ML, NC, OA, and CGP provided insightful feedback on the manuscript.

    Conflicts of Interest

    None declared.

    ‎
    Multimedia Appendix 1

    MEDLINE (Ovid) search strategy.

    DOCX File , 18 KB
    ‎
    Multimedia Appendix 2

    Educational outcomes reported in the included systematic reviews and their definitions.

    DOCX File , 14 KB
    ‎
    Multimedia Appendix 3

    Characteristics of the included systematic reviews.

    DOCX File , 92 KB
    ‎
    Multimedia Appendix 4

    Research questions identified in the included systematic reviews.

    DOCX File , 193 KB
    ‎
    Multimedia Appendix 5

    Overview of conceptual frameworks on the implementation and adoption of digital education.

    DOCX File , 17 KB
    1. Working for health and growth: investing in the health workforce. World Health Organization. 2016.   URL: https://www.who.int/publications/i/item/9789241511308 [accessed 2022-02-19]
    2. Health workforce. World Health Organization. 2019.   URL: https://www.who.int/health-topics/health-workforce#tab=tab_1 [accessed 2022-02-19]
    3. The world health report 2006. Working together for health. World Health Organization. 2006.   URL: https://www.who.int/workforcealliance/knowledge/resources/whreport_2006/en/ [accessed 2022-02-19]
    4. Scheffler R, Cometto G, Tulenko K, Bruckner T, Liu J, Keuffel EL, et al. Health workforce requirements for universal health coverage and the Sustainable Development Goals – background paper No.1 to the WHO Global Strategy on Human Resources for Health: workforce 2030. Human Resources for Health Observer Series No 17. Geneva: World Health Organization; 2016.   URL: https://apps.who.int/iris/bitstream/handle/10665/250330/9789241511407-eng.pdf [accessed 2022-02-19]
    5. Liu JX, Goryakin Y, Maeda A, Bruckner T, Scheffler R. Global health workforce labor market projections for 2030. Hum Resour Health 2017;15(1):11 [FREE Full text] [CrossRef] [Medline]
    6. Tumlinson K, Jaff D, Stilwell B, Onyango DO, Leonard KL. Reforming medical education admission and training in low- and middle-income countries: who gets admitted and why it matters. Hum Resour Health 2019;17(1):91 [FREE Full text] [CrossRef] [Medline]
    7. Hippe DS, Umoren RA, McGee A, Bucher SL, Bresnahan BW. A targeted systematic review of cost analyses for implementation of simulation-based education in healthcare. SAGE Open Med 2020;8:2050312120913451 [FREE Full text] [CrossRef] [Medline]
    8. Guze PA. Using technology to meet the challenges of medical education. Trans Am Clin Climatol Assoc 2015;126:260-270 [FREE Full text] [Medline]
    9. Goh PS, Sandars J. A vision of the use of technology in medical education after the COVID-19 pandemic. MedEdPublish 2020;9(1):49. [CrossRef]
    10. Armey LE, Hosman L. The centrality of electricity to ICT use in low-income countries. Telecomm Policy 2016;40(7):617-627. [CrossRef]
    11. Dunleavy G, Nikolaou CK, Nifakos S, Atun R, Law GC, Tudor Car L. Mobile digital education for health professions: systematic review and meta-analysis by the digital health education collaboration. J Med Internet Res 2019;21(2):e12937 [FREE Full text] [CrossRef] [Medline]
    12. George PP, Papachristou N, Belisario JM, Wang W, Wark PA, Cotic Z, et al. Online eLearning for undergraduates in health professions: a systematic review of the impact on knowledge, skills, attitudes and satisfaction. J Glob Health 2014;4(1):010406 [FREE Full text] [CrossRef] [Medline]
    13. George PP, Zhabenko O, Kyaw BM, Antoniou P, Posadzki P, Saxena N, et al. Online digital education for postregistration training of medical doctors: systematic review by the digital health education collaboration. J Med Internet Res 2019;21(2):e13269 [FREE Full text] [CrossRef] [Medline]
    14. Gentry SV, Gauthier A, L'Estrade Ehrstrom B, Wortley D, Lilienthal A, Tudor Car L, et al. Serious gaming and gamification education in health professions: systematic review. J Med Internet Res 2019;21(3):e12994 [FREE Full text] [CrossRef] [Medline]
    15. Posadzki P, Bala MM, Kyaw BM, Semwal M, Divakar U, Koperny M, et al. Offline digital education for postregistration health professions: systematic review and meta-analysis by the digital health education collaboration. J Med Internet Res 2019;21(4):e12968 [FREE Full text] [CrossRef] [Medline]
    16. Kyaw BM, Saxena N, Posadzki P, Vseteckova J, Nikolaou CK, George PP, et al. Virtual reality for health professions education: systematic review and meta-analysis by the digital health education collaboration. J Med Internet Res 2019;21(1):e12959 [FREE Full text] [CrossRef] [Medline]
    17. Singh K, Ansari M, Galipeau J, Garritty C, Keely E, Malcolm J, et al. An evidence map of systematic reviews to inform interventions in prediabetes. Can J Diabetes 2012;36(5):281-291. [CrossRef]
    18. Jabareen Y. Building a conceptual framework: philosophy, definitions, and procedure. Int J Qual Methods 2009;8(4):49-62. [CrossRef]
    19. ISCED fields of education and training 2013 (ISCED-F 2013). UNESCO Institute for Statistics. Montreal: UNESCO Institute for Statistics; 2014.   URL: http:/​/uis.​unesco.org/​sites/​default/​files/​documents/​isced-fields-of-education- and-training-2013-en.​pdf [accessed 2022-02-21]
    20. Car J, Carlstedt-Duke J, Tudor Car L, Posadzki P, Whiting P, Zary N, Digital Health Education Collaboration. Digital education in health professions: the need for overarching evidence synthesis. J Med Internet Res 2019;21(2):e12913 [FREE Full text] [CrossRef] [Medline]
    21. Hervatis V, Kyaw BM, Semwal M, Dunleavy G, Tudor Car L, Zary N, et al. Offline and computer-based eLearning interventions for medical students' education. Cochrane Database Syst Rev 2018;2018(10):CD012149. [CrossRef]
    22. Paul P, Toon E, Hadadgar A, Jirwe M, Saxena N, Lim KT, et al. Online- and local area network (LAN)-based eLearning interventions for medical doctors' education. Cochrane Database Syst Rev 2018;2018(10):CD012108. [CrossRef]
    23. Cook DA, Levinson AJ, Garside S, Dupras DM, Erwin PJ, Montori VM. Instructional design variations in internet-based learning for health professions education: a systematic review and meta-analysis. Acad Med 2010;85(5):909-922. [CrossRef] [Medline]
    24. Stuchlikova L, Kósa A. Massive open online courses - challenges and solutions in engineering education. In: IEEE 11th International Conference on Emerging eLearning Technologies and Applications. 2013 Presented at: ICETA '13; October 24-25, 2013; Stara Lesna, Slovakia p. 359-364. [CrossRef]
    25. Crompton H. A historical overview of M-learning: toward learner-centered education. In: Berge ZL, Muilenburg LY, editors. Handbook of mobile learning. Milton Park, UK: Routledge; 2013.
    26. Gentry S, L'Estrade Ehrstrom B, Gauthier A, Alvarez J, Wortley D, van Rijswijk J, et al. Serious gaming and gamification interventions for health professional education. Cochrane Database Syst Rev 2018;2018(6):CD012209. [CrossRef]
    27. Saxena N, Kyaw BM, Vseteckova J, Dev P, Paul P, Lim KT, et al. Virtual reality environments for health professional education. Cochrane Database Syst Rev 2016;2016(2):CD012090. [CrossRef]
    28. Cook DA, Triola MM. Virtual patients: a critical literature review and proposed next steps. Med Educ 2009;43(4):303-311. [CrossRef] [Medline]
    29. Lopreiato JO. Healthcare simulation dictionary. Rockville, MD: Agency for Healthcare and Quality; 2016.
    30. Tiven MB, Fuchs ER, Bazari A, Macquarrie A. Evaluating global digital education: student outcomes framework. Organisation for Economic co-operation and development. 2018.   URL: https://www.oecd.org/pisa/Evaluating-Global-Digital-Education- Student-Outcomes-Framework.pdf [accessed 2022-02-21]
    31. Digital learning and teaching framework. RMIT University. 2017.   URL: https:/​/cpb-ap-se2.​wpmucdn.com/​sites.rmit.edu.au/​dist/​c/​16/​files/​2017/​03/​Draft-Digital-Learning-Framework-28.​3.​2017-vh7abk.​pdf [accessed 2022-02-21]
    32. Kyaw BM, Tudor Car L, van Galen LS, van Agtmael MA, Costelloe CE, Ajuebor O, et al. Health professions digital education on antibiotic management: systematic review and meta-analysis by the digital health education collaboration. J Med Internet Res 2019;21(9):e14984 [FREE Full text] [CrossRef] [Medline]
    33. Tudor Car L, Kyaw BM, Atun R. The role of eLearning in health management and leadership capacity building in health system: a systematic review. Hum Resour Health 2018;16(1):44 [FREE Full text] [CrossRef] [Medline]
    34. Tudor Car L, Kyaw BM, Dunleavy G, Smart NA, Semwal M, Rotgans JI, et al. Digital problem-based learning in health professions: systematic review and meta-analysis by the digital health education collaboration. J Med Internet Res 2019;21(2):e12945 [FREE Full text] [CrossRef] [Medline]
    35. Khan BH. The global e-Learning framework. In: Mishra S, editor. E-Learning (STRIDE Handbook 8). New Delhi, India: Staff Training and Research Institute of Distance Education (STRIDE), IGNOU; 2009:42-51.
    36. Aparicio M, Bacao F, Oliveira T. An e-Learning theoretical framework. J Educ Tech Syst 2016;19(1):292-307.
    37. Andersson A, Grönlund Å. A conceptual framework for e-learning in developing countries: a critical review of research challenges. Electron J Inf Syst Dev Ctries 2009;38(1):1-16 [FREE Full text] [CrossRef]
    38. Wisdom JP, Chor KH, Hoagwood KE, Horwitz SM. Innovation adoption: a review of theories and constructs. Adm Policy Ment Health 2014;41(4):480-502 [FREE Full text] [CrossRef] [Medline]
    39. Greenhalgh T, Wherton J, Papoutsi C, Lynch J, Hughes G, A'Court C, et al. Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies. J Med Internet Res 2017;19(11):e367 [FREE Full text] [CrossRef] [Medline]
    40. Kyaw BM, Posadzki P, Dunleavy G, Semwal M, Divakar U, Hervatis V, et al. Offline digital education for medical students: systematic review and meta-analysis by the digital health education collaboration. J Med Internet Res 2019;21(3):e13165 [FREE Full text] [CrossRef] [Medline]
    41. Jayakumar N, Brunckhorst O, Dasgupta P, Khan MS, Ahmed K. e-Learning in surgical education: a systematic review. J Surg Educ 2015;72(6):1145-1157. [CrossRef] [Medline]
    42. Pei L, Wu H. Does online learning work better than offline learning in undergraduate medical education? A systematic review and meta-analysis. Med Educ Online 2019;24(1):1666538 [FREE Full text] [CrossRef] [Medline]
    43. Viljoen CA, Scott Millar R, Engel ME, Shelton M, Burch V. Is computer-assisted instruction more effective than other educational methods in achieving ECG competence amongst medical students and residents? A systematic review and meta-analysis. BMJ Open 2019;9(11):e028800 [FREE Full text] [CrossRef] [Medline]
    44. Boutefnouchet T, Laios T. Transfer of arthroscopic skills from computer simulation training to the operating theatre: a review of evidence from two randomised controlled studies. SICOT J 2016;2:4 [FREE Full text] [CrossRef] [Medline]
    45. Lapierre A, Bouferguene S, Gauvin-Lepage J, Lavoie P, Arbour C. Effectiveness of interprofessional manikin-based simulation training on teamwork among real teams during trauma resuscitation in adult emergency departments: a systematic review. Simul Healthc 2020;15(6):409-421. [CrossRef] [Medline]
    46. Dromey BP, Peebles DM, Stoyanov DV. A systematic review and meta-analysis of the use of high-fidelity simulation in obstetric ultrasound. Simul Healthc 2021;16(1):52-59 [FREE Full text] [CrossRef] [Medline]
    47. Cheng A, Lockey A, Bhanji F, Lin Y, Hunt EA, Lang E. The use of high-fidelity manikins for advanced life support training--a systematic review and meta-analysis. Resuscitation 2015;93:142-149. [CrossRef] [Medline]
    48. Mok HT, So CF, Chung JW. Effectiveness of high-fidelity patient simulation in teaching clinical reasoning skills. Clin Simul Nurs 2016;12(10):453-467. [CrossRef]
    49. Santos GN, Leite AF, Figueiredo PT, Pimentel NM, Flores-Mir C, de Melo NS, et al. Effectiveness of e-learning in oral radiology education: a systematic review. J Dent Educ 2016;80(9):1126-1139. [Medline]
    50. Tomesko J, Touger-Decker R, Dreker M, Zelig R, Parrott JS. The effectiveness of computer-assisted instruction to teach physical examination to students and trainees in the health sciences professions: a systematic review and meta-analysis. J Med Educ Curric Dev 2017;4:2382120517720428 [FREE Full text] [CrossRef] [Medline]
    51. Ahmet A, Gamze K, Rustem M, Sezen KA. Is video-based education an effective method in surgical education? A systematic review. J Surg Educ 2018;75(5):1150-1158. [CrossRef] [Medline]
    52. Liossi C, Failo A, Schoth DE, Williams G, Howard RF. The effectiveness of online pain resources for health professionals: a systematic review with subset meta-analysis of educational intervention studies. Pain 2018;159(4):631-643. [CrossRef] [Medline]
    53. Tudor Car L, Soong A, Kyaw BM, Chua KL, Low-Beer N, Majeed A. Health professions digital education on clinical practice guidelines: a systematic review by Digital Health Education collaboration. BMC Med 2019;17(1):139 [FREE Full text] [CrossRef] [Medline]
    54. Foronda CL, Fernandez-Burgos M, Nadeau C, Kelley CN, Henry MN. Virtual simulation in nursing education: a systematic review spanning 1996 to 2018. Simul Healthc 2020;15(1):46-54. [CrossRef] [Medline]
    55. Martinengo L, Yeo NJ, Markandran KD, Olsson M, Kyaw BM, Car LT. Digital health professions education on chronic wound management: a systematic review. Int J Nurs Stud 2020;104:103512. [CrossRef] [Medline]
    56. Brusamento S, Kyaw BM, Whiting P, Li L, Tudor Car L. Digital health professions education in the field of pediatrics: systematic review and meta-analysis by the digital health education collaboration. J Med Internet Res 2019;21(9):e14231 [FREE Full text] [CrossRef] [Medline]
    57. Liu Q, Peng W, Zhang F, Hu R, Li Y, Yan W. The effectiveness of blended learning in health professions: systematic review and meta-analysis. J Med Internet Res 2016;18(1):e2 [FREE Full text] [CrossRef] [Medline]
    58. Alaker M, Wynn GR, Arulampalam T. Int J Surg 2016;29:85-94 [FREE Full text] [CrossRef] [Medline]
    59. Khan R, Plahouras J, Johnston BC, Scaffidi MA, Grover SC, Walsh CM. Virtual reality simulation training for health professions trainees in gastrointestinal endoscopy. Cochrane Database Syst Rev 2018;8:CD008237 [FREE Full text] [CrossRef] [Medline]
    60. Koohestani HR, Soltani Arabshahi SK, Fata L, Ahmadi F. The educational effects of mobile learning on students of medical sciences: a systematic review in experimental studies. J Adv Med Educ Prof 2018;6(2):58-69 [FREE Full text] [Medline]
    61. Rambarat CA, Merritt JM, Norton HF, Black E, Winchester DE. Using Simulation to Teach Echocardiography: a Systematic Review. Simul Healthc 2018;13(6):413-419. [CrossRef] [Medline]
    62. Vaona A, Banzi R, Kwag KH, Rigon G, Cereda D, Pecoraro V, et al. E-learning for health professionals. Cochrane Database Syst Rev 2018;1(1):CD011736 [FREE Full text] [CrossRef] [Medline]
    63. Xu X, Posadzki PP, Lee GE, Car J, Smith HE. Digital education for health professions in the field of dermatology: a systematic review by digital health education collaboration. Acta Derm Venereol 2019;99(2):133-138 [FREE Full text] [CrossRef] [Medline]
    64. Rangarajan K, Davis H, Pucher PH. Systematic review of virtual haptics in surgical simulation: a valid educational tool? J Surg Educ 2020;77(2):337-347. [CrossRef] [Medline]
    65. Rourke S. How does virtual reality simulation compare to simulated practice in the acquisition of clinical psychomotor skills for pre-registration student nurses? A systematic review. Int J Nurs Stud 2020;102:103466. [CrossRef] [Medline]
    66. Rasmussen K, Belisario JM, Wark PA, Molina JA, Loong SL, Cotic Z, et al. Offline eLearning for undergraduates in health professions: a systematic review of the impact on knowledge, skills, attitudes and satisfaction. J Glob Health 2014;4(1):010405 [FREE Full text] [CrossRef] [Medline]
    67. Thepwongsa I, Kirby CN, Schattner P, Piterman L. Online continuing medical education (CME) for GPs: does it work? A systematic review. Aust Fam Physician 2014;43(10):717-721 [FREE Full text] [Medline]
    68. Mickan S, Atherton H, Roberts NW, Heneghan C, Tilson JK. Use of handheld computers in clinical practice: a systematic review. BMC Med Inform Decis Mak 2014;14:56 [FREE Full text] [CrossRef] [Medline]
    69. Piromchai P, Avery A, Laopaiboon M, Kennedy G, O'Leary S. Virtual reality training for improving the skills needed for performing surgery of the ear, nose or throat. Cochrane Database Syst Rev 2015(9):CD010198. [CrossRef] [Medline]
    70. Aïm F, Lonjon G, Hannouche D, Nizard R. Effectiveness of virtual reality training in orthopaedic surgery. Arthroscopy 2016;32(1):224-232. [CrossRef] [Medline]
    71. Pfandler M, Lazarovici M, Stefan P, Wucherer P, Weigl M. Virtual reality-based simulators for spine surgery: a systematic review. Spine J 2017;17(9):1352-1363. [CrossRef] [Medline]
    72. Kang J, Seomun G. Evaluating web-based nursing education's effects: a systematic review and meta-analysis. West J Nurs Res 2018;40(11):1677-1697. [CrossRef] [Medline]
    73. Lee H, Min H, Oh SM, Shim K. Mobile technology in undergraduate nursing education: a systematic review. Healthc Inform Res 2018;24(2):97-108 [FREE Full text] [CrossRef] [Medline]
    74. Botelho MG, Agrawal KR, Bornstein MM. An systematic review of e-learning outcomes in undergraduate dental radiology curricula-levels of learning and implications for researchers and curriculum planners. Dentomaxillofac Radiol 2019;48(1):20180027 [FREE Full text] [CrossRef] [Medline]
    75. Kononowicz AA, Woodham LA, Edelbring S, Stathakarou N, Davies D, Saxena N, et al. Virtual patient simulations in health professions education: systematic review and meta-analysis by the digital health education collaboration. J Med Internet Res 2019;21(7):e14676 [FREE Full text] [CrossRef] [Medline]
    76. La Cerra C, Dante A, Caponnetto V, Franconi I, Gaxhja E, Petrucci C, et al. Effects of high-fidelity simulation based on life-threatening clinical condition scenarios on learning outcomes of undergraduate and postgraduate nursing students: a systematic review and meta-analysis. BMJ Open 2019;9(2):e025306 [FREE Full text] [CrossRef] [Medline]
    77. Nagendrababu V, Pulikkotil SJ, Sultan OS, Jayaraman J, Soh JA, Dummer PM. Effectiveness of technology-enhanced learning in Endodontic education: a systematic review and meta-analysis. Int Endod J 2019;52(2):181-192. [CrossRef] [Medline]
    78. Semwal M, Whiting P, Bajpai R, Bajpai S, Kyaw BM, Tudor Car L. Digital education for health professions on smoking cessation management: systematic review by the digital health education collaboration. J Med Internet Res 2019;21(3):e13000 [FREE Full text] [CrossRef] [Medline]
    79. Osborne B, Parange N, Thoirs K. The effectiveness of the use of high fidelity simulators in obstetric ultrasound training: a systematic review. Australas J Ultrasound Med 2015;18(3):107-111 [FREE Full text] [CrossRef] [Medline]
    80. den Harder AM, Frijlingh M, Ravesloot CJ, Oosterbaan AE, van der Gijp A. The importance of human-computer interaction in radiology e-learning. J Digit Imaging 2016;29(2):195-205 [FREE Full text] [CrossRef] [Medline]
    81. Glassman D, Yiasemidou M, Ishii H, Somani BK, Ahmed K, Biyani CS. Effect of playing video games on laparoscopic skills performance: a systematic review. J Endourol 2016;30(2):146-152. [CrossRef] [Medline]
    82. Maertens H, Madani A, Landry T, Vermassen F, Van Herzeele I, Aggarwal R. Systematic review of e-learning for surgical training. Br J Surg 2016;103(11):1428-1437. [CrossRef] [Medline]
    83. Sinclair PM, Kable A, Levett-Jones T, Booth D. The effectiveness of Internet-based e-learning on clinician behaviour and patient outcomes: a systematic review. Int J Nurs Stud 2016;57:70-81. [CrossRef] [Medline]
    84. Lui JT, Hoy MY. Evaluating the effect of virtual reality temporal bone simulation on mastoidectomy performance: a meta-analysis. Otolaryngol Head Neck Surg 2017;156(6):1018-1024. [CrossRef] [Medline]
    85. Rohwer A, Motaze NV, Rehfuess E, Young T. E‐learning of evidence‐based health care (EBHC) to increase EBHC competencies in healthcare professionals: a systematic review. Campbell Syst Rev 2017;13(1):1-147. [CrossRef]
    86. Campbell K, Taylor V, Douglas S. Effectiveness of online cancer education for nurses and allied health professionals; a systematic review using Kirkpatrick evaluation framework. J Cancer Educ 2019;34(2):339-356. [CrossRef] [Medline]
    87. Lee J, Kim H, Kim KH, Jung D, Jowsey T, Webster CS. Effective virtual patient simulators for medical communication training: a systematic review. Med Educ 2020;54(9):786-795. [CrossRef] [Medline]
    88. Vitale SG, Caruso S, Vitagliano A, Vilos G, Di Gregorio LM, Zizolfi B, et al. The value of virtual reality simulators in hysteroscopy and training capacity: a systematic review. Minim Invasive Ther Allied Technol 2020;29(4):185-193. [CrossRef] [Medline]
    89. Divakar U, Nazeha N, Posadzki P, Jarbrink K, Bajpai R, Ho AH, et al. Digital education of health professionals on the management of domestic violence: systematic review and meta-analysis by the digital health education collaboration. J Med Internet Res 2019;21(5):e13868 [FREE Full text] [CrossRef] [Medline]
    90. Barteit S, Guzek D, Jahn A, Bärnighausen T, Jorge MM, Neuhann F. Evaluation of e-learning for medical education in low- and middle-income countries: a systematic review. Comput Educ 2020;145:103726 [FREE Full text] [CrossRef] [Medline]
    91. Kyaw BM, Posadzki P, Paddock S, Car J, Campbell J, Tudor Car L. Effectiveness of digital education on communication skills among medical students: systematic review and meta-analysis by the digital health education collaboration. J Med Internet Res 2019;21(8):e12967 [FREE Full text] [CrossRef] [Medline]
    92. Wahabi HA, Esmaeil SA, Bahkali KH, Titi MA, Amer YS, Fayed AA, et al. Medical doctors' offline computer-assisted digital education: systematic review by the digital health education collaboration. J Med Internet Res 2019;21(3):e12998 [FREE Full text] [CrossRef] [Medline]
    93. Huang Z, Semwal M, Lee SY, Tee M, Ong W, Tan WS, et al. Digital health professions education on diabetes management: systematic review by the digital health education collaboration. J Med Internet Res 2019;21(2):e12997 [FREE Full text] [CrossRef] [Medline]
    94. Rothschild P, Richardson A, Beltz J, Chakrabarti R. Effect of virtual reality simulation training on real-life cataract surgery complications: systematic literature review. J Cataract Refract Surg 2021;47(3):400-406. [CrossRef] [Medline]
    95. Richmond H, Copsey B, Hall AM, Davies D, Lamb SE. A systematic review and meta-analysis of online versus alternative methods for training licensed health care professionals to deliver clinical interventions. BMC Med Educ 2017;17(1):227 [FREE Full text] [CrossRef] [Medline]
    96. Männistö M, Mikkonen K, Kuivila HM, Virtanen M, Kyngäs H, Kääriäinen M. Digital collaborative learning in nursing education: a systematic review. Scand J Caring Sci 2020;34(2):280-292. [CrossRef] [Medline]
    97. Voutilainen A, Saaranen T, Sormunen M. Conventional vs. e-learning in nursing education: a systematic review and meta-analysis. Nurse Educ Today 2017;50:97-103. [CrossRef] [Medline]
    98. Zhang H, Mörelius E, Goh SH, Wang W. Effectiveness of video-assisted debriefing in simulation-based health professions education: a systematic review of quantitative evidence. Nurse Educ 2019;44(3):E1-E6. [CrossRef] [Medline]
    99. Qiao W, Bai Y, Lv R, Zhang W, Chen Y, Lei S, et al. The effect of virtual endoscopy simulator training on novices: a systematic review. PLoS One 2014;9(2):e89224 [FREE Full text] [CrossRef] [Medline]
    100. Berry MC, de M Neto JM, de Souza MI, Figueredo CM, Reher V, Evans JL. Effectiveness of technology-enhanced learning to improve periodontics educational outcomes: a systematic review. J Dent Educ 2020;84(7):830-839. [CrossRef] [Medline]
    101. Lahti M, Hätönen H, Välimäki M. Impact of e-learning on nurses' and student nurses knowledge, skills, and satisfaction: a systematic review and meta-analysis. Int J Nurs Stud 2014;51(1):136-149. [CrossRef] [Medline]
    102. Guedes HG, Câmara Costa Ferreira ZM, Ribeiro de Sousa Leão L, Souza Montero EF, Otoch JP, Artifon EL. Virtual reality simulator versus box-trainer to teach minimally invasive procedures: a meta-analysis. Int J Surg 2019;61:60-68 [FREE Full text] [CrossRef] [Medline]
    103. Kim JH, Park H. Effects of smartphone-based mobile learning in nursing education: a systematic review and meta-analysis. Asian Nurs Res (Korean Soc Nurs Sci) 2019;13(1):20-29 [FREE Full text] [CrossRef] [Medline]
    104. Gorbanev I, Agudelo-Londoño S, González RA, Cortes A, Pomares A, Delgadillo V, et al. A systematic review of serious games in medical education: quality of evidence and pedagogical strategy. Med Educ Online 2018;23(1):1438718 [FREE Full text] [CrossRef] [Medline]
    105. Tarpada SP, Morris MT, Burton DA. E-learning in orthopedic surgery training: a systematic review. J Orthop 2016;13(4):425-430 [FREE Full text] [CrossRef] [Medline]
    106. Kato C, Kataoka Y. Literature review of simulation training for acute obstetric emergencies. J Jpn Acad Midwife 2015;29(1):4-14. [CrossRef]
    107. Zhou Y, Yang Y, Liu L, Zeng Z. [Effectiveness of mobile learning in medical education: a systematic review]. Nan Fang Yi Ke Da Xue Xue Bao 2018;38(11):1395-1400 [FREE Full text] [CrossRef] [Medline]
    108. Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials 1996;17(1):1-12. [CrossRef] [Medline]
    109. Li C, He J, Yuan C, Chen B, Sun Z. The effects of blended learning on knowledge, skills, and satisfaction in nursing students: a meta-analysis. Nurse Educ Today 2019;82:51-57. [CrossRef] [Medline]
    110. Higgins JP, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, Cochrane Bias Methods Group, Cochrane Statistical Methods Group. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ 2011;343:d5928 [FREE Full text] [CrossRef] [Medline]
    111. Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol 2011;64(4):383-394. [CrossRef] [Medline]
    112. Reed DA, Cook DA, Beckman TJ, Levine RB, Kern DE, Wright SM. Association between funding and quality of published medical education research. JAMA 2007;298(9):1002-1009. [CrossRef] [Medline]
    113. Ali S, Uppal MA, Gulliver SR. A conceptual framework highlighting e-learning implementation barriers. Inf Technol People 2018;31(1):156-180. [CrossRef]
    114. Kituyi GM, Tusubira I. A framework for the integration of e-learning in higher education institutions in developing countries. Int J Educ Dev Using Inf Commun Technol 2013;9(2):19-36.
    115. Madar MJ, Willis O. Strategic model of implementing e-learning. Int J Sci Technol Res 2014;3(5):235-238.
    116. Masoumi D, Lindström B. Quality in e-learning: a framework for promoting and assuring quality in virtual institutions. J Comput Assist Learn 2012;28(1):27-41. [CrossRef]
    117. Newton D, Ellis A. An e-learning comparative alignment framework. In: World Conference on E- Learning in Corporate, Government, Healthcare & Higher Education. 2007 Presented at: AACE '07; October 15-19, 2007; Chesapeake, VA.
    118. Cook DA. The research we still are not doing: an agenda for the study of computer-based learning. Acad Med 2005;80(6):541-548. [CrossRef] [Medline]
    119. Ruiz JG, Cook DA, Levinson AJ. Computer animations in medical education: a critical literature review. Med Educ 2009;43(9):838-846. [CrossRef] [Medline]
    120. Cook DA, Beckman TJ. Reflections on experimental research in medical education. Adv Health Sci Educ Theory Pract 2010;15(3):455-464. [CrossRef] [Medline]

    ‎
    VR: virtual reality

    Edited by G Eysenbach; submitted 16.07.21; peer-reviewed by R Bajpai, A Ardekani; comments to author 07.10.21; revised version received 07.11.21; accepted 21.11.21; published 17.03.22

    Copyright

    ©Lorainne Tudor Car, Selina Poon, Bhone Myint Kyaw, David A Cook, Victoria Ward, Rifat Atun, Azeem Majeed, Jamie Johnston, Rianne M J J van der Kleij, Mariam Molokhia, Florian V Wangenheim, Martin Lupton, Niels Chavannes, Onyema Ajuebor, Charles G Prober, Josip Car. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 17.03.2022.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.