This paper is in the following e-collection/theme issue:

Games and Gamification for Health (609) e-Learning and Medical Education (648) Serious Games for Health and Medicine (644) Diabetes Education and Elearning for Health Professionals (11)

Published on in Vol 19, No 3 (2017): March

InsuOnline, an Electronic Game for Medical Education on Insulin Therapy: A Randomized Controlled Trial With Primary Care Physicians

InsuOnline, an Electronic Game for Medical Education on Insulin Therapy: A Randomized Controlled Trial With Primary Care Physicians

InsuOnline, an Electronic Game for Medical Education on Insulin Therapy: A Randomized Controlled Trial With Primary Care Physicians

Authors of this article:

Leandro Arthur Diehl1 Author Orcid Image ;   Rodrigo Martins Souza2 Author Orcid Image ;   Pedro Alejandro Gordan1 Author Orcid Image ;   Roberto Zonato Esteves3, 4 Author Orcid Image ;   Izabel Cristina Meister Coelho4 Author Orcid Image
Article Authors Cited by (31) Tweetations (26) Metrics

    Original Paper

    1Internal Medicine Department, Health Sciences Center, Londrina State University (UEL), Londrina PR, Brazil

    2Games Division, Oniria Software Industry, Londrina PR, Brazil

    3Medicine Department, Maringá State University (UEM), Maringá PR, Brazil

    4Pró-Ensino na Saúde, Pequeno Príncipe College, Pequeno Príncipe Complex, Curitiba PR, Brazil

    *all authors contributed equally

    Corresponding Author:

    Leandro Arthur Diehl, MD, MSc, PhD

    Internal Medicine Department

    Health Sciences Center

    Londrina State University (UEL)

    Depto Clinica Medica/CCS

    Av Robert Koch 60

    Londrina PR, 86038350

    Brazil

    Phone: 55 4333712234

    Fax:55 4333712234

    Email: drgaucho@yahoo.com


    Background: Most patients with diabetes mellitus (DM) are followed by primary care physicians, who often lack knowledge or confidence to prescribe insulin properly. This contributes to clinical inertia and poor glycemic control. Effectiveness of traditional continuing medical education (CME) to solve that is limited, so new approaches are required. Electronic games are a good option, as they can be very effective and easily disseminated.

    Objective: The objective of our study was to assess applicability, user acceptance, and educational effectiveness of InsuOnline, an electronic serious game for medical education on insulin therapy for DM, compared with a traditional CME activity.

    Methods: Primary care physicians (PCPs) from South of Brazil were invited by phone or email to participate in an unblinded randomized controlled trial and randomly allocated to play the game InsuOnline, installed as an app in their own computers, at the time of their choice, with minimal or no external guidance, or to participate in a traditional CME session, composed by onsite lectures and cases discussion. Both interventions had the same content and duration (~4 h). Applicability was assessed by the number of subjects who completed the assigned intervention in each group. Insulin-prescribing competence (factual knowledge, problem-solving skills, and attitudes) was self-assessed through a questionnaire applied before, immediately after, and 3 months after the interventions. Acceptance of the intervention (satisfaction and perceived importance for clinical practice) was also assessed immediately after and 3 months after the interventions, respectively.

    Results: Subjects’ characteristics were similar between groups (mean age 38, 51.4% [69/134] male). In the game group, 69 of 88 (78%) completed the intervention, compared with 65 of 73 (89%) in the control group, with no difference in applicability. Percentage of right answers in the competence subscale, which was 52% at the baseline in both groups, significantly improved immediately after both interventions to 92% in the game group and to 85% in control (P<.001). After 3 months, it remained significantly higher than that at the baseline in both groups (80% in game, and 76% in control; P<.001). Absolute increase in competence score was better with the game (40%) than with traditional CME (34%; P=.01). Insulin-related attitudes were improved both after the game (significant improvement in 4 of 9 items) and after control activity (3 of 9). Both interventions were very well accepted, with most subjects rating them as “fun or pleasant,” “useful,” and “practice-changing.”

    Conclusions: The game InsuOnline was applicable, very well accepted, and highly effective for medical education on insulin therapy. In view of its flexibility and easy dissemination, it is a valid option for large-scale CME, potentially helping to reduce clinical inertia and to improve quality of care for DM patients.

    Trial Registration: Clinicaltrials.gov NCT001759953; https://clinicaltrials.gov/ct2/show/NCT01759953 (Archived by WebCite at http://www.webcitation.org/6oeHoTrBf)

    J Med Internet Res 2017;19(3):e72

    doi:10.2196/jmir.6944

    Keywords

    diabetes mellitus; insulin; video games; medical education; continuing medical education; educational technology 


    Diabetes mellitus (DM) is a main public health problem of 21st century, affecting 422 million adults worldwide [1]. In most countries, most patients with DM are followed and treated by doctors who are not specialists in Endocrinology or diabetes, mainly primary care physicians (PCPs) [2]. However, only 24-56% of patients with DM present good glycemic control [3,4], defined as plasma levels of glycated hemoglobin A1c<7% [5].

    Many factors may be implied in the low frequency of good metabolic control among patients with diabetes, but it is likely that one of the main reasons may be PCPs’ lack of knowledge and confidence on several aspects of DM management [6], specially regarding insulin use [7].

    This gap in PCPs’ competence to treat diabetes with insulin contributes to the problem known as clinical inertia, “the failure to advance therapy when indicated,” [8] resulting in underuse of insulin [9] and poor glycemic control. In fact, previous reports show that there is an average delay of about 3-5 years between the first demonstrations that a patient with DM requires insulin and the actual initiation of insulin therapy [10,11].

    Continuing medical education (CME) on DM and insulin is often advocated as a solution to optimize the knowledge and the practice of PCPs [12]; however, traditional CME activities (such as lectures and group discussions) have small and short-lasting efficacy [13]. Thus, new educational methods are urgently required. Electronic games are a powerful tool for education [14], as they “create a tight marriage among content, game play, and valued ways of thinking and acting” [15]. Reasons for using games include the familiarity of most college students with this medium [16] and their favorable views on the matter [17,18]. Nevertheless, the 2 most compelling arguments sustaining the adoption of games for learning are (1) their potential educational effectiveness [15,16] and (2) their flexibility and easy dissemination [19].

    Good learning games are usually built following the same rules that guide the design of effective learning activities, which include stimulus to players’ intrinsic motivation, practice and repetition, effective feedback, arousal of positive feelings, intensity of the experience, and learner choice and involvement [20,21]. In the medical area, allowing students to practice their skills in a game may increase the safety for real patients [22].

    Also, electronic games are learning resources much more flexible than traditional onsite educational activities, as they can be used in learners’ own equipment, location, and time schedule, rendering them more scalable [23].

    In the field of diabetes, some games for education of patients [24-28] and a few technology-based initiatives for education of health professionals [13,29-34] have been described, but to our knowledge, no game has been previously reported for education of health professionals on diabetes or insulin.

    We have previously described the process of design and development of InsuOnline, the first electronic serious game intended for medical education on insulin therapy for diabetes [35] and a formative assessment of usability and playability of its prototype [36].

    A previous pilot study with medical students and residents has shown that InsuOnline was applicable and well accepted as a tool for medical education on insulin therapy, in what we called “real-world conditions” for self-directed distance learning activities: as a stand-alone resource, played in learners’ own equipment, in learners’ own time schedule, with minimal or no external guidance [36,37].

    The next step in InsuOnline validation process is assessing its educational effectiveness.

    Thus, the objectives of this study were to assess applicability, user acceptance, and effectiveness of InsuOnline for education of PCPs on insulin therapy for diabetes, as compared with a traditional onsite CME session with the same content and duration.


    Trial Design

    We performed an open-label randomized controlled trial to assess the effectiveness of InsuOnline as a method for education of PCPs on insulin therapy for diabetes, as compared with a “traditional” onsite educational activity with the same content and same duration.

    Eligibility Criteria

    Subjects were eligible if they were medical doctors with an active register at a regional Council of Medicine in Brazil, were not specialists in Endocrinology or diabetes, were currently working at a public health care unit as a primary care physician (PCP), and were directly involved in the treatment of patients with diabetes in those facilities, with any degree of computer or gaming literacy.

    Recruitment

    Researchers contacted local public health authorities of some cities from the states of Paraná (Londrina and other cities from the 17th Health Regional, Curitiba and São José dos Pinhais) and Santa Catarina (Blumenau), who agreed to help with recruitment. Those authorities have enlisted PCPs in their area to attend a training session about diabetes, and then we contacted those PCPs via phone, mail, or email to invite them to participate in the study.

    Those who agreed to participate were informed of the research procedures and were sent by email a link to an online informed consent form stored on Google Drive (Multimedia Appendix 1). Respondents were allocated to groups by simple randomization using an online random number generator [38] by the first author. Due to our expectancy of a higher attrition rate in the game group, allocation was made at a 2:1 ratio.

    Settings and Locations Where Data Were Collected

    After filling the informed consent form, participants were randomized and informed about the group they were allocated to and received an email with a link to the baseline questionnaire (stored on Google Drive). Also, participants allocated to control group were sent an email with information about the time and location of their scheduled onsite learning activity. A printed questionnaire was applied to those participants of the onsite learning activity immediately after its ending.

    Participants allocated to game group received an email with instructions on how to download and install the game, after filling the baseline questionnaire, and then were contacted (weekly, if needed) to check if they have finished the game. After that, they were sent the link to another Web-based questionnaire.

    After 3 months, a link to the third questionnaire was sent to participants of both groups.

    If the participants did not answer the questionnaire after a few days, reminders were sent at 3-7–day intervals (initially by email, then by text message, and finally by phone). If they did not answer after up to 6 consecutive reminders, they were considered as loss of follow-up.

    Interventions

    Game

    InsuOnline was developed by transdisciplinary collaboration from a team composed by clinical endocrinologists, game designers, and experts in medical education, with the help of other professionals (programmers, graphic designers, sound editors, etc) when needed at specific points of the process, using the methodology of iterative prototyping [39]. The game was designed to be a tool for education of PCPs on how to best use insulin in the treatment of patients with DM, in a primary health care setting. Educational objectives of the game are presented in Textbox 1. InsuOnline was developed as a 3D app, with simple commands (all player actions were made using a mouse); game engine was built using Unity, and visual elements (scenarios, characters, animations) were designed on Blender 3D creation suite. A detailed description of InsuOnline design and development process can be found in [35]. The version used in this study was a code release (version 1.6.1); screenshots (in Portuguese) are shown in Figures 1 and 2.

    Participants allocated to the game group received a personal login and password to access a website from where the game could be downloaded and installed in players’ own desktops or notebooks with Windows or MacOS, without any cost for study subjects.

    During the game, players take on the role of a young doctor in a primary health care unit, whose mission is to improve glycemic control of an increasing-complexity series of 19 patients with diabetes, usually by means of initiating or adjusting insulin. Patients and situations presented in the game were designed to be similar to scenarios most commonly seen in a primary health care unit. The game gives immediate feedback after each player’s decision, and the player’s progression in the game is only possible if the most adequate decisions are made.

    Several pedagogical elements were included in the game, aiming for the best educational effects [21,40]. These were based on the principles of adult learning and problem-based learning, including motivation, goal-orientation, relevancy-orientation, self-pacing, timely and appropriate feedback, contextualization, and practical (ie, hands-on) approach with active participation of the learner [41-43].

    Recommendations about how to use insulin were selected from main clinical guidelines [5,44-49] and adapted to the reality of Brazilian public primary health care. Only the insulins commonly available in Brazilian public primary health care units (NPH and regular) were made available in the game. The software recorded players’ progress in the game. Participants were allowed to play the game in their preferred time and place, in the number of sessions they wanted. The amount of time necessary to finish the game was about 4 h in a previous study [37]. Usability, playability, and preliminary educational effectiveness of previous versions of the game were previously assessed and reported [36,37]; development was “frozen” during the trial.

    Players were sent minimal written instructions on how to download and play the game, and most did so with minimal or no external guidance. If subjects had any difficulties, researchers were available to give remote assistance by email, text message, or phone, at any moment.

    Educational objectives of InsuOnline.
    • Recognize the goals of glycemic control in adults with DM
    • Know when to start insulin in type 2 DM
    • Know how to start insulin in type 2 DM (“bedtime” scheme)
    • Know how to orientate proper insulin use (storing, injection technique, and devices)
    • Know how to prevent, recognize, and treat hypoglycemia
    • Know how to orientate and to interpret self-monitoring of blood glucose
    • Know when and how to adjust insulin dosage
    • Compare types of insulin (NPH, regular) and know when to use which
    • Know when to intensify insulin therapy in type 2 DM
    • Know how to intensify insulin therapy in type 2 DM (“basal-plus” scheme)
    • Know how to prescribe intensive insulin therapy (“basal-bolus” scheme)
    • Know how to recognize type 1 DM and how to start treatment
    • Know how to recognize diabetic ketoacidosis and to start treatment
    • Recognize the main barriers to insulin initiation and know how to address them
    • Know how to orientate lifestyle changes and how to manage oral antidiabetic drugs in association with insulin
    • Recognize the main factors that require insulin dosage adjustment and how to manage them
    Textbox 1. Educational objectives of InsuOnline.
    ‎
    Figure 1. InsuOnline main characters: the young Dr Lucas (in the left) is the player’s avatar; the nurse Mariana (in the middle) gives useful hints; and the experienced Dr Braga (in the right) is a “mentor” who helps the player in critical moments.
    View this figure
    ‎
    Figure 2. The player’s avatar, Dr Lucas, interacts with one of the patients in the game (dialogue in Portuguese).
    View this figure
    Onsite Learning Activity

    Participants allocated to the “control” group (ie, the active comparator) were scheduled to join a traditional-format onsite learning session, comprising a series of 3 interactive expositive lectures and 2 group case discussions (the first with 3, and the second with 4 cases, which were identical to corresponding InsuOnline levels). PowerPoint slides for the lectures and printed case presentations were previously prepared by the main author, and their content was exactly identical to the recommendations included in the game. To avoid potential biases, lectures were given by a clinical endocrinologist not linked to the research team, previously trained by the main author and familiarized with the didactic material. Also, the onsite learning activity was designed to have the same duration of the game (about 4 h).

    Outcomes

    Outcomes were self-assessed using written questionnaires which were applied to participants of both groups at 3 time points. The questionnaires were composed by different subscales at each time point: (1) at baseline, “competence” and “attitudes” subscales, plus demographic and professional data (for both groups); (2) immediately post intervention, the same “competence” and “attitudes” subscales for both groups, plus “game evaluation” subscale (for game group) or “onsite activity” subscale (for control group); (3) 3 months post intervention, “competence” and “attitudes” again, plus “importance for professional practice” subscale (for both groups).

    The subscale of “competence” included 9 multiple-choice questions to assess factual knowledge about insulin and 11 clinical case vignettes with multiple-choice questions to assess problem-solving skills regarding insulin initiation and adjustment, which were created by our research team based on InsuOnline’s educational objectives (Textbox 1) and content. The subscale of “attitudes” included 9 Likert-type scale questions to assess attitudes and beliefs regarding diabetes and insulin, which were adapted from previous surveys [7,50].

    The subscale of “game evaluation” had 16 Likert-type scale questions to assess playability, user satisfaction, and perceived educational utility of the game, with 1 additional free text item for additional comments from the participants, which were freely adapted from a questionnaire used to assess another game [51].

    The subscale of “onsite activity evaluation” was composed by 10 Likert-type scale questions regarding methodology, user satisfaction, and perceived educational utility of the activity, and 1 free text item to collect additional comments, which were based on the ones from the “game evaluation” subscale, but adapted to an onsite CME session.

    Finally, the subscale of “importance for professional practice” included 5 Likert-type scale questions and 2 free text items to assess participants’ opinion on the actual impact of the educational activity (game or onsite learning session) on their current professional practice, which were also created by authors.

    All subscales were previously tested and refined in our pilot study with medical students and residents [36,37].

    An additional subscale with 5 multiple-choice questions and 7 free text items was applied only to participants allocated to the game group who did not finish the game 60 days after receiving instructions for download or who did not respond to post-intervention questionnaires, to assess reasons why they were unable to play the game, and their perceptions about it.

    All instruments used in this study are available in Multimedia Appendix 2.

    Applicability of each intervention was defined as the proportion of participants allocated to each group that have received the intervention and did finish all tasks required in each intervention.

    The primary outcome of this study was the score in the competence subscale in the immediately post-intervention self-assessment. Secondary outcomes were: score in competence subscale 3 months post-intervention; participants’ attitudes regarding diabetes and insulin post-intervention; participants’ acceptance and satisfaction with learning activities (game or onsite CME session); and applicability of the intervention (game or onsite CME session).

    Statistical Analysis

    Only subjects who filled the baseline and the immediate post-intervention questionnaires were included in the analysis. Data from questionnaires were downloaded from Google Drive online forms or typed from printed questionnaires to worksheets on Excel (Microsoft Corporation) and then analyzed on Epi-Info 7 (Centers for Disease Control, Atlanta, Georgia) for descriptive analysis. Demographic data were compared among groups at baseline using chi-square test for proportions or Student t or Kruskal-Wallis test for continuous variables, as suited. Correlation among baseline scores on the competence subscale and demographic baseline data was assessed using Pearson r or Student t test on SPSS 14.0 (SPSS Inc). Comparison of the scores on competence subscale by group and by time point was performed by repeated-measures analysis of variance (ANOVA), with the Tukey post hoc test when a significant difference was detected, and by simple comparison between 2 groups at each time point using Student t test, on the statistical software R (The R Foundation). Proportion of subjects with scores of 90% or more on competence subscale and the frequency of “agree” answers on the Likert-type scale items for assessing attitudes were compared by group and by time point using chi-square test (followed by post hoc pairwise comparisons using Bonferroni correction when significant). Absolute increase in competence score from baseline to immediately post intervention was estimated on Epi-Info 7. The effect size of the group variable on the competence score was estimated using Cohen d test on R software. Reliability (internal consistency) of the subscales used in this study was evaluated using Cronbach alpha. Comments made by subjects in the free text items were reviewed by content analysis. Statistical significance was defined as P<.05.

    Sample Size

    In order to detect a minimum standard deviation of 0.5 on the score (percentage of right answers) in competence subscale, with 80% of statistical power at 5% of significance level, we estimated a sample size of 128 subjects (64 in each group). Taking into account an expected attrition rate of 40% in the game group (about twice the 17% attrition rate observed in the previous pilot study with students and residents [36,37]) and of 20% in the control group, we decided to enroll at least 90 subjects in the game group and 77 subjects in the control group.

    Ethics and Informed Consent

    Participation was anonymous and voluntary, and all subjects filled an informed consent form (Multimedia Appendix 1), in accordance with Brazilian Health Ministry’s regulations for research on human beings. The study protocol was previously approved by Londrina State University Research Ethics Committee (UEL, #051/2011 and #051/2012), and by all local public health authorities from the cities where the study was performed.


    User Statistics

    A total of 257 primary care physicians were contacted and assessed for eligibility during the recruitment period (July 2014 in Curitiba and São José dos Pinhais, Paraná; August 2014 in the cities from the 17th Health Regional of Paraná State except Londrina; September to October 2014 in Londrina, Paraná; and October 2014 in Blumenau, Santa Catarina); 170 were randomized, and 134 were included in the final analysis (however, 4 subjects from the game group and 7 from the control did not answer the third questionnaire, 3 months post-intervention). The CONSORT diagram [52] for participant flow is shown in Figure 3.

    Baseline characteristics of subjects in both intervention groups were comparable, with no significant differences (Table 1).

    Table 1. Characteristics of subjects allocated to each intervention group (game or control) at the baseline (n=134).
    Characteristics of subjects
    		Game group (n=69)Control group (n=65)
    Gender, n (%)



    		Male35 (50)34 (52)
    Age in years, mean (SD)
    		37.5 (11)38.5 (12.7)
    Location, n (%)



    		Curitiba34 (49)25 (38)

    		17th HRa20 (29)19 (29)

    		Londrina15 (22)18 (28)

    		Blumenau03 (5)
    Years from graduation, median (range)
    		6.5 (0-41)9.0 (0-41)
    Years of experience in primary care, median (range)
    		4.0 (0-38)5.0 (0-41)
    Did residency, n (%)
    		27 (38)25 (39)
    Residency area, n (%)



    		Family and community health8 (30)9 (36)

    		Internal medicine8 (30)4 (16)

    		Ob&Gyn4 (15)2 (8)

    		Surgery4 (15)4 (16)

    		Pediatrics2 (7)3 (12)

    		Other1 (4)3 (12)
    Self-referred number of patients seen per month with: median (range)



    		type 2 DM50 (4-300)50 (1-500)

    		type 2 DM on insulin20 (0-150)10 (1-120)

    		type 1 DM2 (0-100)2 (0-80)

    a17th HR: 17th Health Regional of Paraná state (group of 21 cities around Londrina, with the exception of Londrina).

    ‎
    Figure 3. CONSORT flow diagram.
    View this figure

    Outcomes

    Applicability

    Applicability, defined as the percentage of subjects who received the intervention and finished it, was 78% (69 of 88 subjects) for the game, and 89% (65 of 73 subjects) for the control onsite learning activity, with no significant difference between the two interventions (P=.11, chi-square test). Among the subjects who did not finish the game, 13 responded to our contact; reasons alleged for not finishing InsuOnline were mostly nonrelated to the game, such as lack of access to the Internet, lack of computer, or lack of time for playing it. Only 2 subjects were unable to access or download the game and 1 was unable to visualize it correctly in her computer due to incompatibility with other software (antivirus).

    Among the game group subjects who did finish playing the game, the mean time required for finishing it after download instructions were sent was 21 days (SD 12, range 2-59). No difference was found in game applicability or in the time for finishing the game by age group, location, or gender of the participants.

    Evaluation of the Interventions by Participants

    Both interventions were very well rated by participants, as revealed by their responses to the subscales of “game evaluation” (in the game group) and “onsite activity evaluation” (in the control group), applied immediately after the interventions (most relevant data are summarized in Table 2).

    Comparing similar questions among both groups, significantly more subjects have found the game “fun” (P=.04, chi-square test), without any other difference. The onsite activity was deemed “pleasant” by 83% of control subjects.

    About two-thirds (68%) of game subjects strongly agreed that they have “learned more about insulin with this game than they would learn from a lecture,” and 68 of 69 (98%) would recommend the game for their friends.

    In the free text field for comments, some subjects expressed their intense satisfaction with the game (“it is always better to learn by playing”), and even physicians who had never played an electronic game were able to play it and enjoyed the experience. Also, some criticisms were reported, related to game content (some repetitive or too extensive dialogues, soundtrack too loud) or to the software itself (some “freezes” when the player clicks too fast, incompatibility with some antivirus software).

    Regarding the onsite activity, most comments were also highly positive (“excellent lecture,” “very practical approach,” “very dynamic and practical activity”). Criticisms were about the short duration of the activity (“too much information for one session only,” “it would be better addressed in two consecutive days”).

    Table 2. Evaluation of interventions (game and onsite learning activity) by participants from each study group.
    ItemsGame group (n=69)Control group (n=65)

    		Strongly agree, n (%)Partially agree, n (%)Strongly agree, n (%)Partially agree, n (%)
    The activity was fun48 (70)a18 (26)32 (49)21 (32)
    The activity captured my attention all the time43 (62)25 (36)47 (72)15 (23)
    I would join similar activities in the future62 (90)6 (9)63 (97)1 (2)
    Patients presented in the activity were similar to the ones I see in my practice43 (62)22 (32)46 (71)16 (25)
    The activity increased my knowledge about diabetes66 (96)3 (4)60 (92)5 (8)
    The activity will influence the way I treat patients with diabetes67 (97)2 (3)59 (91)6 (9)

    aP=.04 (χ2 test), compared to control group.

    Competence for Prescribing Insulin

    In relation to the primary outcome of this study, the percentage of right answers in the competence subscale (factual nowledge + problem-solving skills related to insulin) was about 52% at baseline in both groups, and it was significantly increased post-intervention (P<.001, ANOVA for repeated measures) in both groups. In the post hoc Tukey test, this increase was significant in the comparison between the baseline and immediate post-intervention time points (P<.001), between immediate post-intervention and 3 months post-intervention (P<.001), and between 3 months and the baseline (P<.001), which means that scores increased significantly in the immediate post-intervention time compared with baseline, then decreased significantly after 3 months in comparison with immediate post-test, but even with that decrease, 3-month scores remained significantly higher than at baseline.

    When the effects of both factors “time” and “group” were taken into account, a significant difference was found related to “time” (P<.001) and to the iteration “group per time” (P=.02), but no difference was found related to the “group” isolatedly (P=.27, ANOVA for repeated measures). A simple comparison of competence score between the 2 groups at each time point showed no significant difference at baseline (51.5 [SD 15.6%] in the game vs 51.7 [SD 16.6%] in the control group; P=.95) and at 3 months (79.8 [SD 14.4%] in the game vs 76.2 [SD 16.9%] in the control group; P=.20), but a significantly higher score in the game group than in the control group at the immediate post-intervention assessment (respectively: 91.7 [SD 8.9%] vs 85.5 [SD 13.4%]; P=.001, Student t test; Figure 4).

    In fact, absolute increase in competence score (from baseline to immediately post intervention) was higher in the game group (40 [SD 15%]) than in the control group (34 [SD 15%]; P=.01); that difference was marginal at 3 months (28 [SD 14] in the game vs 23 [SD 17] points in the control group; P=.06).

    Also, the frequency of subjects who achieved a 90% or higher score in competence subscale at immediate post-intervention assessment was higher in the game group (53 subjects, or 77%) than in the control group (35 subjects, or 54%; P<.001), with no difference at baseline (0 in game vs 3% in the control group) or 3 months post-intervention (32% in game vs 26% in the control group).

    Cohen d size effect of the game group compared with the control group was 0.4583 (0.093-0.8327) in the immediate post-intervention time [53].

    No differences were observed in competence score by gender, location, residency status, years of experience in primary care, or self-reported mean number of patients with diabetes seen per month; a significant correlation was only observed with participants’ age (Pearson r=−.3314 at the baseline; P<.001; and Pearson r=−.4616 at immediate post-intervention time; P<.001).

    ‎
    Figure 4. Score in competence subscale in both groups (game and control) in the 3 time points (baseline, immediately post-intervention, and 3 months post-intervention; mean and standard error).
    View this figure
    Additional Outcomes

    Significant improvement in diabetes and insulin-related attitudes from the baseline to post-intervention was observed in 4 of the 9 items in the game group, and in 3 of 9 items in the control. In post hoc analyses, most differences occurred between the baseline and immediate posttest, and a few between the baseline and 3 months. An additional statistical difference was observed between 3 months and immediate posttest in the control group in an item that assessed a personal opinion. Results for attitudes subscale are in Multimedia Appendix 3.

    Three months after the interventions, about 80% of the subjects from both the groups stated that the activity had had real impact on their practice and that they knew better what to do when seeing a real patient with diabetes. About 62-75% of participants in both groups also said they were feeling more secure, and found it easier to help patients to improve their glycemic control. In the game group, significantly more subjects said that “it got easier” to orientate DM patients about their therapy, compared with control group (85% vs 66%; P=.03). Results from the subscale of “importance for professional practice” are presented in Multimedia Appendix 4.

    Reliability of the subscales used in this study was estimated to be about 0.7-0.8 for most subscales (Cronbach alpha=.715 for competence subscale, .739 for game evaluation subscale, .649 for onsite activity evaluation, and .850 for importance for professional practice), except for attitudes subscale (Cronbach alpha=.323) [54].

    No harms or adverse effects were reported.


    Principal Findings

    Our results have shown that the game InsuOnline, used in real-world conditions (in players’ own computers and in their own flexible time schedule) was highly effective for education of PCPs on insulin therapy for diabetes, compared with an onsite CME activity with the same content and same approximate duration. In fact, competence improvement was better with the game than with the onsite learning session. InsuOnline was also applicable, with about 80% of the subjects with the more varied degrees of computer or gaming literacy being able to finish it with little or no external help. Both interventions were very well-rated by participants, regarding engagement, realism, and perceived educational value. The good evaluation of the onsite CME session proves that InsuOnline was compared with a valid control activity (a gold-standard active comparator). Three months after both the interventions, doctors from both the groups said that they were feeling more secure and more prepared to help real patients with diabetes in their daily professional practice, with even better results in the game group.

    Games and simulators are being increasingly used for education of health care professionals on various topics [55-57], but, to our knowledge, this is the first report on the educational effectiveness of an electronic game for education of medical doctors about insulin therapy for diabetes.

    Games are a promising way to deliver CME, for many reasons. One reason is their educational adequacy: well-designed games incorporate all the main principles of Adult Education, such as individualized, self-pacing, contextualized learning, with active experimentation and appreciation of previous knowledge [15,16,40-43]. The inclusion of game elements adds entertainment to the learning experience, increasing learners’ motivation to practice and to learn, which renders the learning experience more enjoyable, more engaging, and potentially more effective [40,58]. Games can be designed to simulate an infinite number of medical problems, and are easily updated as needed [15,40]. Finally, games are more flexible than traditional CME, respecting learners’ own time availability and learning pace, and are much more scalable [23]. So, we believe that well-designed and validated games will soon become the gold standard option for delivering large-scale CME.

    Validation of games for health professional education is a recent and growing research area. Most available games for health professionals’ education have not passed through an objective validation process, and many of the published studies on the field are biased [59]. So, there is a great need of good quality studies which can provide solid evidence to support the fast-growing science of games for health. Our randomized-controlled trial was designed to contribute to this body of evidence by following the guidelines for research on health games effectiveness as proposed by Kato, who suggested that studies in this area should attempt to apply the same scientific rigor typical of health sciences [60].

    As there is a variety of proposed methods for validating games for health, we opted to use multiple methods for validation of InsuOnline. One of the simplest validation approaches was proposed by Olsen [61], and includes only 3 aspects: usability, playability, and educational effectiveness. The first two aspects were very well rated for InsuOnline, as previously reported [36], and our current results support the third one. However, the best method to validate a game that was designed for CME is the approach to validation of CME activities described by Moore [62], that proposes a continuum of 7 levels: (1) applicability (user participation), (2) user satisfaction, (3) gain of factual and procedural knowledge (learning), (4) gain of competence, (5) improvement of learner’s performance, (6) improvement of patient health, and (7) improvement of community health. InsuOnline has already met the first 4 criteria. The following 3 steps (change in PCP’s professional performance and improvement of their patients’ glycemic control, and, hopefully, improvement of populational risk of DM complications) will be the focus of future research.

    Attitudes and beliefs of PCPs in our sample were similar, at the baseline, to the ones of family doctors in United States [7] and in Arab-speaking countries in the Middle East [50], although the Brazilian doctors who participated in our study seem to be more often inclined to perceive the initiation of insulin therapy and the training of the patient with DM for using insulin as difficult and complicated tasks (78% and 54% of our subjects, respectively, compared with 66% and 42% in United States, and 48% and 25% in Middle East), which underscores the urgent need for effective educational interventions.

    Reasons for the better results observed with the game than with the onsite learning activity, in this study, can be related to characteristics of the games that foster learning and attitude (and behavioral) changes [63]:

    1. The inclusion of behavioral change procedures into the game, such as: goal setting, decision-making, problem solving, goals re-evaluation, social rewards; and

    2. The use of a story (the game plot), and the inclusion of behavioral change concepts into that story. Players’ identification and empathy for the story protagonist may contribute to change players’ attitudes, reflecting the attitudinal change that occurs with the protagonist character in the game, in a process known as “modeling” [63].

    Also, the higher emotional involvement of the learner with a game than with a traditional CME activity is another likely factor, as affection is a powerful determinant for changing attitudes and behavior [64]. Intrinsic motivation, a very effective learning catalyst, probably is one of the biggest advantages of games in relation to more traditional modalities of education [65]. Our game was designed to produce a high level of players’ engagement, by careful disposition of different game elements (realism, identification with context and characters, increasing-difficulty challenges, humor, clear objectives, immediate and intense feedback for each players’ action, progress monitoring, and rewards). With that, the majority of participants stated that the game was “fun” and that it captured their attention all the time.

    Limitations

    Some limitations of our study should be pointed out. First, blinding of the participants (and researchers) was not feasible due to the nature of the interventions, and that may have affected some of our results, especially the subscales of evaluation of the activities, as many subjects may have found it exciting or unusual to play an electronic game to learn about a medical topic. However, we think that our primary outcome (competence for prescribing insulin) was not compromised, as it was objectively assessed by standardized multiple-choice questions.

    Second, the instruments used in this study were not previously validated, because they were developed by our team to assess the specific outcomes we addressed in this study. The use of customized instruments is strongly recommended by Moreno-Ger [66], who argues that generic questionnaires are usually not useful for assessing games that can be very different in their objectives, target audiences, and needs. At our favor, we can say that our instruments were extensively reviewed by our team of experts in endocrinology and medical education, which warrants their face and content validity, and our subscales have shown acceptable reliability, as measured by their Cronbach alpha in the range of .7 to .8, with an exception of the attitudes subscale, which had very poor internal consistency [54]. The authors decided to maintain this subscale because it had questions similar to the ones used in previous surveys [7,50], allowing their comparison with our results.

    In third place, this study shows only intermediate outcomes (PCPs’ knowledge, skills, attitudes, competence regarding insulin initiation, and adjustment for diabetes), but we don’t know yet if the game will have any impact on actual health professionals’ performance or on their patients’ measures or outcomes (“hard” outcomes), which we expect to assess in future studies.

    Finally, although PCPs’ competence for prescribing insulin is essential, it is very likely that a simple intervention on the education of medical doctors may be insufficient to induce a significant change in the process of care delivered to patients with DM in primary care. A multifaceted intervention, aiming at several aspects of DM management, should probably be more effective to improve care and control of DM on a primary health care level, but it would demand conjugated efforts from government and society to improve issues such as availability of multiprofessional teams, availability of better insulins and oral drugs, availability of reagent strips for self-monitoring of blood glucose, better access to diabetes education, and others.

    Generalizability

    We think our results can be generalized for most primary care physicians worldwide, as our sample included medical doctors from different geographic areas, different age groups, different levels of specialization and experience, and any degree of computer or gaming literacy. Also, our game was tested in what we believe to be the “real-world conditions” for the use of an electronic game for distance CME: in players’ own computers (with a wide range of possible equipment configurations), with very little or no external help from our team, and in their own spare time. Our results show that InsuOnline is a flexible, applicable and scalable option for large-scale CME on diabetes.

    Conclusions

    In this “real-world conditions” randomized controlled trial, InsuOnline, the first electronic serious game designed for medical education on insulin therapy for diabetes, was applicable, very well accepted, and highly effective for education, with even better results for improving primary care physicians’ competence and attitudes related to diabetes and insulin than a gold-standard onsite CME activity with the same content and duration. For this reason, we believe InsuOnline is a valid tool for large-scale CME on DM, with advantages manifested in its easy Web dissemination, customizable content, and accordance with Adult Learning principles. We hope it can contribute to improve PCPs’ performance and optimize the quality of care offered to patients with DM in primary care, eventually improving patients’ glycemic control and reducing the risk of DM complications.

    Acknowledgments

    All authors helped with the writing and review of this manuscript.

    These data (preliminary results of baseline and immediately post-intervention assessments only) were partly presented at the 75th American Diabetes Association Scientific Sessions, at Boston Convention Center, Boston, MA, June 5-9, 2015.

    The authors acknowledge the support of public health authorities in all the cities where this study was performed. We also thank Dr Victor Hugo Ferreira for his invaluable help by giving the lectures and leading the case discussions with the subjects of control group. We thank our student collaborators (Tayná Rolim Galvão Pereira and Caio César Secci), who helped so much with recruitment, contact with study participants, and data collection. We are in debt with Nicholas Bender Haydu and Juliano Barbosa Alves for their invaluable support. Finally, our thanks to all the participants who kindly agreed to join this adventure with us and contributed a few hours of their busy time.

    Conflicts of Interest

    InsuOnline is a copyrighted game. Its design and development was entirely founded by personal resources from the authors LAD and PAG, and from Oniria Software Industry, who are copyright holders. All authors contribute to design and evaluation of the game.

    ‎

    Multimedia Appendix 1

    Informed consent form.

    PDF File (Adobe PDF File), 320KB
    ‎

    Multimedia Appendix 2

    Instruments (questionnaires) used in this study.

    PDF File (Adobe PDF File), 322KB
    ‎

    Multimedia Appendix 3

    Results from the "attitudes" subscale.

    PDF File (Adobe PDF File), 369KB
    ‎

    Multimedia Appendix 4

    Results from the "importance for professional practice" subscale.

    PDF File (Adobe PDF File), 357KB
    ‎

    Multimedia Appendix 5

    CONSORT eHealth checklist V1.6.1.

    PDF File (Adobe PDF File), 871KB
    1. World Health Organization. Global report on diabetes. Geneva: World Health Organization; 2016.
    2. DeWitt DE, Hirsch IB. Outpatient insulin therapy in type 1 and type 2 diabetes mellitus: scientific review. J Am Med Assoc 2003 May 7;289(17):2254-2264. [Medline]
    3. Mendes AB, Fittipaldi JA, Neves RC, Chacra AR, Moreira ED. Prevalence and correlates of inadequate glycaemic control: results from a nationwide survey in 6,671 adults with diabetes in Brazil. Acta Diabetol 2010 Jun;47(2):137-145 [FREE Full text] [CrossRef] [Medline]
    4. Ford ES, Li C, Little RR, Mokdad AH. Trends in A1C concentrations among U.S. adults with diagnosed diabetes from 1999 to 2004. Diabetes Care 2008 Jan;31(1):102-104. [CrossRef] [Medline]
    5. American Diabetes Association. Standards of medical care in diabetes--2014. Diabetes Care 2014 Jan;37(Suppl 1):S14-S80. [CrossRef] [Medline]
    6. Rubin DJ, Moshang J, Jabbour SA. Diabetes knowledge: are resident physicians and nurses adequately prepared to manage diabetes? Endocr Pract 2007;13(1):17-21. [CrossRef] [Medline]
    7. Hayes RP, Fitzgerald JT, Jacober SJ. Primary care physician beliefs about insulin initiation in patients with type 2 diabetes. Int J Clin Pract 2008 Jun;62(6):860-868 [FREE Full text] [CrossRef] [Medline]
    8. Peyrot M, Rubin RR, Lauritzen T, Skovlund SE, Snoek FJ, Matthews DR, International DAWN Advisory Panel. Resistance to insulin therapy among patients and providers: results of the cross-national Diabetes Attitudes, Wishes, and Needs (DAWN) study. Diabetes Care 2005 Nov;28(11):2673-2679. [Medline]
    9. Riddle MC. The underuse of insulin therapy in North America. Diabetes Metab Res Rev 2002;18(Suppl 3):S42-S49. [CrossRef] [Medline]
    10. Nichols GA, Koo YH, Shah SN. Delay of insulin addition to oral combination therapy despite inadequate glycemic control: delay of insulin therapy. J Gen Intern Med 2007 Apr;22(4):453-458 [FREE Full text] [CrossRef] [Medline]
    11. Rubino A, McQuay LJ, Gough SC, Kvasz M, Tennis P. Delayed initiation of subcutaneous insulin therapy after failure of oral glucose-lowering agents in patients with Type 2 diabetes: a population-based analysis in the UK. Diabet Med 2007 Dec;24(12):1412-1418. [CrossRef] [Medline]
    12. Peyrot M, Rubin RR, Khunti K. Addressing barriers to initiation of insulin in patients with type 2 diabetes. Prim Care Diabetes 2010 Apr;4(Suppl 1):S11-S18. [CrossRef] [Medline]
    13. Sharp LK, Lipsky MS. Continuing medical education and attitudes of health care providers toward treating diabetes. J Contin Educ Health Prof 2002;22(2):103-112. [CrossRef] [Medline]
    14. Johnson L, Adams S, Cummins M. NMC. Austin: The New Media Consortium; 2012. The NMC Horizon Report: Higher Education Edition   URL: http://www.nmc.org/publications/horizon-report-2012-higher-ed-edition [accessed 2012-12-21] [WebCite Cache]
    15. Gee JP. What Video Games Have to Teach Us About Learning and Literacy. New York: Palgrave Macmillan; 2004.
    16. Prensky M. Digital game-based learning. New York: McGraw-Hill; 2001.
    17. Kron FW, Gjerde CL, Sen A, Fetters MD. Medical student attitudes toward video games and related new media technologies in medical education. BMC Med Educ 2010;10:50 [FREE Full text] [CrossRef] [Medline]
    18. Diehl LA, de Souza RM, Gordan PA, Esteves RZ, Coelho IC. Gaming habits and opinions of Brazilian medical school faculty and students: what´s next? Games Health J 2014 Apr;3(2):79-85. [CrossRef] [Medline]
    19. Mayo MJ. Games for science and engineering education. Commun ACM 2007 Jul 01;50(7):30. [CrossRef]
    20. Murphy C. Goodgamesbydesign. 2011. Why games work and the science of learning   URL: http://www.goodgamesbydesign.com/Files/WhyGamesWork_TheScienceOfLearning_CMurphy_2011.pdf [accessed 2012-12-21] [WebCite Cache]
    21. Yusoff A. Eprints.soton. Southampton: University of Southampton, Faculty of Engineering, Sciences and Mathematics, Doctoral Thesis; 2010. A conceptual framework for serious games and its validation   URL: http://eprints.soton.ac.uk/171663/1/Thesis_017b.pdf [accessed 2016-10-28] [WebCite Cache]
    22. Ziv A, Small SD, Wolpe PR. Patient safety and simulation-based medical education. Med Teach 2000;22(5):489-495. [CrossRef] [Medline]
    23. Demirbilek M. The use of electronic games in distance learning as a tool for teaching-learning. In: Rogers PL, Berg GA, Boettcher JV, Howard C, Justice L, Schenk KD, editors. Encyclopedia of distance learning. Hershey: IGI Global; 2009:2209-2223.
    24. Brown SJ, Lieberman DA, Germeny BA, Fan YC, Wilson DM, Pasta DJ. Educational video game for juvenile diabetes: results of a controlled trial. Med Inform (Lond) 1997;22(1):77-89. [Medline]
    25. Patterson N, Wolfenstein M, Millar S, Halverson R, Squire K. Wcer.wisc.edu. Madison: University of Wisconsin-Madison; 2011. Games and Simulations for Diabetes Education   URL: http://wcer.wisc.edu/publications/abstract/wcer-working-paper-no.-2011-1 [accessed 2012-12-21] [WebCite Cache]
    26. DeShazo J, Harris L, Pratt W. Effective intervention or child's play? A review of video games for diabetes education. Diabetes Technol Ther 2010 Oct;12(10):815-822. [CrossRef] [Medline]
    27. Lehmann ED. Interactive educational simulators in diabetes care. Med Inform (Lond) 1997;22(1):47-76. [Medline]
    28. Thompson D, Baranowski T, Buday R. Conceptual model for the design of a serious video game promoting self-management among youth with type 1 diabetes. J Diabetes Sci Technol 2010 May 01;4(3):744-749 [FREE Full text] [CrossRef] [Medline]
    29. O'Connor PJ, Sperl-Hillen JM, Johnson PE, Rush WA, Asche SE, Dutta P, et al. Simulated physician learning intervention to improve safety and quality of diabetes care: a randomized trial. Diabetes Care 2009 Apr;32(4):585-590 [FREE Full text] [CrossRef] [Medline]
    30. Harris SB, Leiter LA, Webster-Bogaert S, Van DM, O'Neill C. Teleconferenced educational detailing: diabetes education for primary care physicians. J Contin Educ Health Prof 2005;25(2):87-97. [CrossRef] [Medline]
    31. Tatti P, Lehmann ED. Utility of the AIDA diabetes simulator as an interactive educational teaching tool for general practitioners (primary care physicians). Diabetes Technol Ther 2001;3(1):133-140. [CrossRef] [Medline]
    32. Miller D, Roberts P. Using computer simulation as an insulin prescribing virtual preceptorship. In: Diabetes. Alexandria: American Diabetes Association; 2011 Jul Presented at: 71st American Diabetes Association Scientific Sessions; June 24-28, 2011; San Diego p. A197.
    33. Simation Medical. Learntotreat. Victoria: Simation Medical Ltd; 2012. Learn to Treat Diabetes   URL: http://learntotreat.com [accessed 2012-12-21] [WebCite Cache]
    34. American Diabetes Association, Therasim. Therasim. Durham: Therasim, Inc; 2010. Interactive Patient Simulation   URL: http://therasim.net/ada01/login.php [accessed 2012-12-21] [WebCite Cache]
    35. Diehl LA, Souza RM, Alves JB, Gordan PA, Esteves RZ, Jorge ML, et al. InsuOnline, a serious game to teach insulin therapy to primary care physicians: design of the game and a randomized controlled trial for educational validation. JMIR Res Protoc 2013 Jan 21;2(1):e5 [FREE Full text] [CrossRef] [Medline]
    36. Diehl LA, de Souza RM, Gordan PA, Esteves RZ, Coelho ICM. User assessment of InsuOnLine, a game to fight clinical inertia in diabetes: a pilot study. Games Health J 2015 Oct;4(5):335-343. [CrossRef] [Medline]
    37. Diehl LA, Gordan PA, Esteves RZ, Coelho IC. Effectiveness of a serious game for medical education on insulin therapy: a pilot study. Arch Endocrinol Metab 2015 Oct;59(5):470-473 [FREE Full text] [CrossRef] [Medline]
    38. Randomness and Integrity Services Ltd. Random. Dublin: Randomness and Integrity Services Ltd; 2016. What's this fuss about true randomness?   URL: https://www.random.org/ [accessed 2016-10-27] [WebCite Cache]
    39. Bates B. Game design. Boston: Thomson Course Technology PTR; 2004.
    40. Aldrich C. The complete guide to simulations and serious games: how the most valuable content will be created in the age beyond Gutenberg to Google. San Francisco: Pfeiffer; 2009.
    41. Collins J. Education techniques for lifelong learning: principles of adult learning. Radiographics 2004 Sep;24(5):1483-1489. [CrossRef] [Medline]
    42. Barrows H. Problem-based learning applied to medical education. Springfield: Southern Illinois University School of Medicine; 2000.
    43. Knowles M. The modern practice of adult education: from pedagogy to andragogy. New York: Association Press; 1980.
    44. American Diabetes Association. Practical Insulin: A Handbook for Prescribing Providers. Alexandria: American Diabetes Association; 2011.
    45. Beaser RS. Joslin´s insulin deskbook: designing and initiating insulin treatment programs. Boston: Joslin Diabetes Center; 2008.
    46. Brazilian Diabetes Society. Guidelines from the Brazilian Diabetes Society 2013-2014. São Paulo: AC Farmacêutica; 2014.
    47. Inzucchi SE, Bergenstal RM, Buse JB, Diamant M, Ferrannini E, Nauck M, American Diabetes Association (ADA), European Association for the Study of Diabetes (EASD). Management of hyperglycemia in type 2 diabetes: a patient-centered approach: position statement of the American Diabetes Association (ADA) and the European Association for the Study of Diabetes (EASD). Diabetes Care 2012 Jun;35(6):1364-1379 [FREE Full text] [CrossRef] [Medline]
    48. Lerario AC, Chacra AR, Pimazoni-Netto A, Malerbi D, Gross JL, Oliveira JE, et al. Algorithm for the treatment of type 2 diabetes: a position statement of Brazilian Diabetes Society. Diabetol Metab Syndr 2010 Jun 08;2(1):35 [FREE Full text] [CrossRef] [Medline]
    49. Nathan DM, Buse JB, Davidson MB, Ferrannini E, Holman RR, Sherwin R, American Diabetes Association, European Association for Study of Diabetes. Medical management of hyperglycemia in type 2 diabetes: a consensus algorithm for the initiation and adjustment of therapy: a consensus statement of the American Diabetes Association and the European Association for the Study of Diabetes. Diabetes Care 2009 Jan;32(1):193-203 [FREE Full text] [CrossRef] [Medline]
    50. Lakkis NA, Maalouf GJ, Mahmassani DM, Hamadeh GN. Insulin therapy attitudes and beliefs of physicians in Middle Eastern Arab countries. Fam Pract 2013 Oct;30(5):560-567 [FREE Full text] [CrossRef] [Medline]
    51. Reichlin L, Mani N, McArthur K, Harris AM, Rajan N, Dacso CC. Assessing the acceptability and usability of an interactive serious game in aiding treatment decisions for patients with localized prostate cancer. J Med Internet Res 2011 Jan 12;13(1):e4 [FREE Full text] [CrossRef] [Medline]
    52. Schulz KF, Altman DG, Moher D. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. Br Med J 2010;340:c332 [FREE Full text] [Medline]
    53. Cohen J. Statistical power analysis for the behavioral sciences. Hillsdale: L Erlbaum Associates; 1988.
    54. Bland JM, Altman DG. Cronbach's alpha. Br Med J 1997 Feb 22;314(7080):572 [FREE Full text] [Medline]
    55. Akl EA, Sackett KM, Erdley WS, Mustafa RA, Fiander M, Gabriel C, et al. Educational games for health professionals. Cochrane Database Syst Rev 2013;1:CD006411. [CrossRef] [Medline]
    56. Abdulmajed H, Park YS, Tekian A. Assessment of educational games for health professions: a systematic review of trends and outcomes. Med Teach 2015 Apr;37(Suppl 1):S27-S32. [CrossRef] [Medline]
    57. Nicolaidou I, Antoniades A, Constantinou R, Marangos C, Kyriacou E, Bamidis P, et al. A virtual emergency telemedicine serious game in medical training: a quantitative, professional feedback-informed evaluation study. J Med Internet Res 2015;17(6):e150 [FREE Full text] [CrossRef] [Medline]
    58. Koster R. A theory of fun for game design. Sebastopol: O'Reilly Media; 2013.
    59. Segev A, Rovner M, Appel DI, Abrams AW, Rotem M, Bloch Y. Possible biases of researchers´ attitudes toward video games: publication trends analysis of the medical literature. J Med Internet Res 2016 Jul 18;18(7):e196 [FREE Full text] [CrossRef] [Medline]
    60. Kato PM. Evaluating efficacy and validating games for health. Games Health J 2012 Feb;1(1):74-76. [CrossRef] [Medline]
    61. Olsen T, Procci K, Bowers C. Serious games usability testing: how to ensure proper usability, playability, and effectiveness. In: Design, user experience, and usability: theory, methods, tools and practice : first international conference, DUXU 2011, held as part of HCI International 2011, Orlando, FL, USA, July 9-14, 2011 : proceedings. Berlin: Springer; 2011 Presented at: HCI International; July 9-14, 2011; Orlando p. 625-634.
    62. Moore DE, Green JS, Gallis HA. Achieving desired results and improved outcomes: integrating planning and assessment throughout learning activities. J Contin Educ Health Prof 2009;29(1):1-15. [CrossRef] [Medline]
    63. Baranowski T, Buday R, Thompson DI, Baranowski J. Playing for real: video games and stories for health-related behavior change. Am J Prev Med 2008 Jan;34(1):74-82 [FREE Full text] [CrossRef] [Medline]
    64. Olson JM, Zanna MP. Attitudes and attitude change. Annu Rev Psychol 1993 Jan;44(1):117-154. [CrossRef]
    65. Malone TW, Lepper MR. Making learning fun: a taxonomy of intrinsic motivations for learning. In: Snow RE, Farr MJ, editors. Aptitude, learning, and instruction. Volume 3: conative and affective process analyses. Hillsdale: L. Erlbaum Associates; 1987.
    66. Moreno-Ger P, Torrente J, Hsieh YG, Lester WT. Usability testing for serious games: making informed design decisions with user data. Advances in Human-Computer Interaction 2012;2012:1-13. [CrossRef]

    ‎
    A1c: glycated hemoglobin A1c
    CME: continuing medical education
    DM: diabetes mellitus
    PCP: primary care physician
    RCT: randomized controlled trial

    Edited by G Eysenbach; submitted 04.11.16; peer-reviewed by E da Silva, A Segev; comments to author 08.12.16; revised version received 14.12.16; accepted 13.02.17; published 09.03.17

    Copyright

    ©Leandro Arthur Diehl, Rodrigo Martins Souza, Pedro Alejandro Gordan, Roberto Zonato Esteves, Izabel Cristina Meister Coelho. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 09.03.2017.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.