Delivering Clinical Skin Examination Education to Nurse Practitioners Using an Internet-Based, Microlearning Approach: Development and Feasibility of a Video Intervention

Background: Skin cancer is the most common cancer; survival of the most serious skin cancers and malignant melanomas depends on early detection. Early detection relies on accessibility to clinical skin examination (CSE). Primary care nurse practitioners (PCNPs) are well-positioned to conduct CSEs; however, they require further education on CSE and have time constraints for continuing education. A digitally delivered intervention grounded in microlearning is a promising approach to deliver new information over a brief period. Objective: Our objective was to develop and explore the feasibility of implementing a 1-week digital video intervention with content on CSE skills, defined as melanoma risk assessment, head-to-toe skin examination, and pigmented lesion assessment, for PCNPs. Specific aims were as follows: (1) Aim 1: to develop three microlearning-based melanoma videos with content on CSE that are suitable for digital delivery to PCNPs in various formats and (2) Aim 2: to assess the feasibility of the video intervention, including enrollment and retention rates, adherence, and acceptability and usability of the video intervention. Methods: For Aim 1, the research team created storyboards for videos that addressed each CSE skill. An expert panel of three dermatologists reviewed the storyboards and videos for relevance, comprehension, and clarity using the content validity index (CVI). The panel evaluated the usability of the video intervention delivery by Research Electronic Data Capture (REDCap) and Vimeo using the System Usability Scale (SUS) and technical video production using Beaudin and Quick’s Quality Evaluation of Video (QEV). Aim 2 evaluated enrollment and retention rates of PCNPs, based on metrics from previous studies of CSE in the literature, and video intervention adherence. SUS and the Attitudes toward Web-based Continuing Learning Survey (AWCL) assessed usability and acceptability. Results: CVI scores indicated relevance and clarity for each video: mean scores ranged from 3.79 to 4, where 4 indicated the video was highly relevant and very clear. The integration of REDCap and Vimeo was usable: the SUS score was 96, where 0 was the worst and 100 was the best. The digital delivery of the videos was rated as exceptional on all five technical items: the mean score was 5, where scores ranged from 1 (poor) to 5 (exceptional). Of the 32 PCNPs who were sent emails, 12 enrolled (38%) and, out of these 12, 10 (83%) completed the intervention and the surveys. Video intervention adherence was ≤50%. Participants rated the usability as better (mean 85.8, SD 10.6; better=70-90) and favorably ranked the acceptability of the AWCL’s constructs of perceived usefulness (mean 5.26, SD 0.08), perceived ease of use (mean 5.40, SD 0.41), behavior (mean 5.53, SD 0.12), and affection (mean 5.77, SD 0.04), where scores ranged from 1 (strongly disagree) to 7 (strongly agree). Conclusions: The video intervention was feasible to deliver to PCNPs using a digital, microlearning approach. The findings provide support for using the videos as an intervention in a future pilot randomized trial targeting behavioral CSE outcomes among PCNPs and other primary care providers. JMIR Dermatol 2020 | vol. 3 | iss. 1 | e16714 | p. 1 http://derma.jmir.org/2020/1/e16714/ (page number not for citation purposes) Stratton et al JMIR DERMATOLOGY


Background
The incidence of the deadliest skin cancer, melanoma, has doubled in the United States over the past 20 years [1]. An estimated 96,480 new cases of melanoma were diagnosed in 2019, and 7320 deaths resulted from melanoma [2]. Early detection leads to a greater proportion of removal of thin melanomas (<2 mm in thickness), which is associated with better outcomes [3][4][5]. Early detection is best accomplished by clinical skin examination (CSE), most often performed by a dermatologist. CSE is defined as melanoma risk assessment, head-to-toe skin examination, and pigmented lesion assessment. Currently, there is a shortage of dermatologists-the United States averages just 3.6 dermatologists per 100,000 people [6,7], which has not changed in the last 30 years even as skin cancer incidence continues to rise [8][9][10]. Patients could benefit from the availability of more practitioners, such as nurse practitioners (NPs), to conduct quality CSEs.
Previous research indicates that NPs may lack confidence and skills to perform CSE or assess skin lesions [11][12][13][14]. NPs have demonstrated mixed ability to distinguish lesions that are suspicious for melanoma from nonsuspicious lesions [11,12]; in one study, a majority of NPs stated that they would rather refer any skin lesion to a specialist [15], which is problematic given the shortage of dermatologists. Although NPs' confidence in CSE is low, they believe that primary care providers, in general, help to detect skin cancer early [13]. In a pilot study, NPs showed promise for cultivating good sensitivity (ranging from 50% to 100%, n=4) and excellent specificity (ranging from 99% to 100%, n=4) when asked to identify suspicious lesions [16].
NPs desire more training and resources about skin cancer training [17]. However, a recent systematic review [12] concluded that there are minimal CSE activities for NPs, and the activities that exist are not well explicated. The majority of NP CSE-focused intervention studies had small sample sizes (eg, ranging from 1 to 30) [18][19][20][21], were lengthy (eg, 14 weeks to 6 months) [18], or did not describe NPs' dermatological training [18][19][20][21]. Fewer interventions had modules lasting under an hour (eg, 15-45 minutes) or were self-directed (eg, reviewing pamphlets, posters, and two presentations) [22][23][24]. This manuscript describes the development and feasibility testing of a brief CSE educational video intervention for primary care nurse practitioners (PCNPs).

Purpose Statement
The purpose of this study was to develop and explore the feasibility of implementing a 1-week, digitally delivered video intervention with content on CSE for PCNPs. The videos covered three CSE skills, defined as melanoma risk assessment, head-to-toe skin examination, and pigmented lesion assessment. The specific aims were as follows: (1) Aim 1: to develop three microlearning-based melanoma videos with content on CSE that are suitable for digital delivery to PCNPs in various formats and (2) Aim 2: to assess the feasibility of the video intervention, including enrollment and retention rates, video intervention adherence, and acceptability and usability postcompletion. Table  1 presents the details of the aims, hypotheses, measures, scoring [25][26][27][28][29][30], and outcomes.  [25][26][27] Content validity index (CVI) Aim 1a: to assess content validity of the video intervention using an established method Dermatology experts score usability as better to truly superior 1 (strongly disagree) to 5 (strongly agree) Sum of item scores is calculated and multiplied by 2.5 Scores: concerning (<50), passable (50-69), better (70-90), and truly superior (>90) [28] System Usability Scale (SUS) Aim 1b: to assess the integration of the videos and surveys into Research Electronic Data Capture (REDCap) Dermatology experts give high scores for technical production and navigation (ie, video design, intended content, visual quality, audio quality, and audio-visual relationship)

(poor) to 5 (exceptional)
Option for open-ended comments follows each item [29] Beaudin and Quick's Quality Evaluation of Video (QEV) Aim 1c: to assess the digital delivery of the videos Aim 2: to determine enrollment and retention rates, video intervention adherence, and acceptability and usability postcompletion Enrollment rates N/A a Calculate the number of participants recruited and enrolled compared to those recruited who chose not to consent or enroll Hypothesis 2.1: enrollment rates will be equal to or better than 60% Retention rates N/A Calculate completion of videos and surveys per number of participants enrolled Hypothesis 2.2: retention rates will be greater than 50% Participant completion rates of the videos N/A Vimeo "finishes": counted from start-to-play to viewing of the very last video frame Hypothesis 2.3: video intervention adherence will be greater than or equal to 50% Participants score usability highly 1 (strongly disagree) to 5 (strongly agree) Sum of item scores is calculated and multiplied by 2.5 Scores: concerning (<50), passable (50-69), better (70-90), and truly superior (>90) [28] System Usability Scale (SUS) Hypothesis 2.4: usability scores will be equal to or higher than 70 Participants score acceptability highly 1 (strongly disagree) to 7 (strongly agree) [30] Attitudes toward Webbased Continuing Learning Survey (AWCL) Acceptability: perceived usefulness (5 items), perceived ease of use (4 items), behavior (3 items), and affection (3 items) Hypothesis 2.5: acceptability scores will be equal to or higher than 5 a N/A: not applicable.

Methods
The study design was a one-group, posttest, cross-sectional design. This section describes methods for each study aim. The University of Arizona Institutional Review Board approved the study.

Aim 1: CSE Video Development
Video development was guided by the use of a microlearning conceptual framework and operational transparency. The informational content of the videos was adapted from previous studies [31][32][33]; adaptation focused on key concepts that could be addressed in a short amount of time [34]. Development also included creation of the video storyboards.
Microlearning allows for the dissemination of short, meaningful knowledge, which benefits practitioners with busy schedules. Microlearning is defined as "special moments or episodes of learning while dealing with specific tasks or content and engaging in small but conscious steps" [35]. Research findings document that the use of short content may increase information retention by 20% [36]. Microlearning is for users who have difficulty creating the time to engage in long stretches of learning activities outside of dedicated study times and institutional programs [34].
For operational transparency, we conducted a systematic review to ascertain the rigor of previous CSE interventions [37] using Sidani and Braden's clarifying elements (see Multimedia Appendix 1) [38]. The goal of the intervention was to inform the participants about CSE, enhance their CSE skills, and motivate them to perform CSE. Each of the three videos had a specific learning objective. The specific strategies, respective components, and immediate goals for each video are in Multimedia Appendix 2. The media for intervention delivery were written (ie, reading) and verbal (ie, audio). The format consisted of a video of skills instruction along with a PowerPoint presentation with voiceover. The videos were each 5-10 minutes in length (ie, amount). Vimeo was used to house the videos; this is an online platform and community developed to create, upload, and share videos [39]. Participants viewed each video one time and spaced video viewings within a 1-week period (ie, frequency); therefore, the duration of the intervention was 1 week.
The research team created and reviewed storyboards for each video. The videos were produced in collaboration with video technology experts at the institution and were uploaded to Vimeo. The web application Research Electronic Data Capture (REDCap) maintained the surveys and the separate fields for each video link from Vimeo. REDCap is a secure workflow methodology and software application designed for the development and deployment of digital data capture tools to assist with clinical and translational research [40]. REDCap allowed placement of the Vimeo video link into the survey, enabling viewing of the video within the survey, without having to open a new browser window. The expert panel, which consisted of three dermatologists, evaluated the integration of the videos in REDCap.
The components of each module were assessed using Sidani and Braden's content validity assessment [41] and the content validity index (CVI). The first content validity survey asked the dermatology experts to evaluate the relevance (ie, the degree to which the content has an appropriate sample of activities for the component being measured) and clarity (ie, the extent to which the storyboard is concise, accurate, and direct) of the storyboard content [42]. The dermatology experts could add comments with each item to provide further clarification. Based on the CVI scores and recommendations, video content and activities were refined. The dermatology expert panel reviewed the content validity for a second time while viewing the actual videos 1 month after the first review. They accessed the videos in REDCap and completed the System Usability Scale (SUS).
The five technical production items from Beaudin and Quick's Quality Evaluation of Video (QEV) evaluated the integration of the videos into Vimeo. The following steps helped to confirm that the REDCap and Vimeo platform was functioning appropriately for the delivery of the surveys and videos:

Aim 2: Feasibility
Feasibility was assessed via enrollment and retention rates as well as video intervention adherence, usability, and acceptability (see Table 1). Video adherence was monitored by Vimeo, which calculated the number of plays, number of finishes (ie, participant viewed to the very last video frame), and average percentage of the video watched per module [43]. Participants completed a short satisfaction survey at the end of the study, which consisted of free-text and scaled items.

Sample
A purposive sample of 12 PCNPs was enrolled and 10 PCNPs completed the videos. A sample size of 10 is sufficient for a feasibility study because even a few cases are likely to be very informative with respect to the difficulty of recruitment, the acceptability of the intervention, costs, and logistics [44,45]. Eligibility criteria were as follows: Participants were not excluded from the study based on gender, age, or race. Exclusion criteria were any previous skin cancer continuing education in CSE or training in CSE. Participants were recruited during a statewide meeting (ie, Arizona) and a local NP meeting (see Table 2 for demographic information). Interested NPs received an email that summarized the feasibility study along with a link to the consent form, surveys, and first video. Each email contained a unique link to the intervention. REDCap creates closed surveys, where each unique link is assigned a study ID number. The consent document was a disclosure form that listed the intervention length, investigator identity, and the purpose of the study (see Multimedia Appendix 3).

Data Collection
All data instruments, automated data capture, videos, and contact information were managed using REDCap. The schedule for delivering each video and the posttest survey were automated with REDCap. Human involvement was limited to in-person recruitment and sending the initial email to start the intervention. The videos were incorporated into the questionnaire. Participants completed the posttest survey, along with the usability and acceptability items, after the third video. A total of 1 week after the third video, participants self-reported personal use of CSE in their practice. All outcomes were self-reported though these online surveys and all questions had forced-choice answers. Participants did not have an option to review or change their answers after submission. Prompts were sent daily for up to 2 days if the participant did not view the video within the first 24 hours of enrollment. Participants who failed to submit the survey were not compensated, and the surveys were ineligible for analysis. Each participant who completed the intervention received a US $50 Amazon gift card. Data collection occurred from March to April 2019.

Data Protection
REDCap was developed specifically around the Health Insurance Portability and Accountability Act (HIPAA) Security Rule guidelines. The REDCap electronic data management system at the University of Arizona is housed on two virtual servers: one supporting database services and the other supporting web services. Hardware is located at the University of Arizona's Information Technology Services Center (UITS). The space is physically secured within a keyless entry area. Hardware management and support are provided by UITS. The database server is located behind a firewall and the web server is in a Data Management Zone. REDCap software support is provided by the University of Arizona Center for Biomedical Informatics and Biostatistics. All web-based information transmission is password protected and encrypted in transit. Administration of REDCap is managed through virtual servers located at the University of Arizona College of Medicine [46].

Statistical Analysis
Data from REDCap were exported into SPSS Statistics for Windows, version 26.0 (IBM Corp) [47], for data analysis. Analysis of sample demographic data consisted of descriptive statistics (ie, frequencies and measures of central tendency). For Aim 1a, the CVI was determined by dividing the number of dermatology experts giving the fact or item a score of 3 or 4 and dividing this by the total number of experts (ie, 3) [26]. Scores for the SUS and the QEV were analyzed using descriptive statistics (Aims 1b and 2). For Aim 2, the Attitudes toward Web-based Continuing Learning Survey (AWCL) was used and data analysis consisted of item mean scores, mean construct scores, and correlations between each construct.

Overview
Characteristics of the sample are listed in Table 2.

Aim 1a: Content Validity
The dermatology expert panel conducted two reviews to assess the content validity of the intervention. Content validity scores primarily increased or were consistent during the second round of reviews after the storyboards were adjusted to address the reviewers' recommendations from the first round. The scores of the following components decreased during the second round: right arm nevus count (relevance) and discuss systematic approach (relevance) (see Table 3).

Aims 1b and 1c: Vimeo and REDCap Integration and Digital Delivery
The dermatology expert panel rated the integration of REDCap and Vimeo and the system's usability as truly superior (mean 95.8, SD 7.2), with a range of scores from 87.5 (better) to 100 (truly superior). All expert panel members scored each of the five technical concepts-video design, intended content, visual quality, audio quality, and audio-visual relationship-as exceptional (mean 5, SD 0) (see Multimedia Appendix 4).

Aim 2: Enrollment and Retention Rate and Intervention Adherence
A total of 12 PCNPs consented from a list of 32 emails (38% enrollment rate). Out of 12 participants, 10 completed the intervention and the surveys (83% retention rate). Completion rate of the surveys was 100% (10/10). A total of 50% (5/10) of participants watched the videos in their entirety. Vimeo recorded 6 plays for video 1. Of the 6 participants who played video 1, only 1 (17%) failed to finish the video, ending it 30 seconds before the content was complete. Therefore, the lowest percentage of possible participants who completed all of the videos is 50% (5/10). See Table 4 for the video adherence information for each module.

Aim 2: Usability, Acceptability, and Satisfaction
The mean usability was better (mean 85.8, SD 10.6), with a range of scores from 72.5 (better) to 100 (truly superior).
Acceptability of the intervention was assessed using the AWCL. The mean for each of the constructs-perceived usefulness, perceived ease of use, behavior, and affection-all ranged between somewhat agree and mostly agree (see Table 5).  The overall mean satisfaction with the study and the intervention was 99% (SD 1.87, N=10; 100% = best). All participants (N=10) would watch the videos even if they did not receive compensation, they believed that the length of the video was "just right," and they believed that the content was "just right." Out of 10 participants, 8 (80%) preferred that the videos be accessible for 1 month. When participants were asked what they learned, they cited specific videos (eg, "scaling suspicious vs nonsuspicious lesions") (5/10, 50%) and increased motivation (eg, "11 plus nevus right arm increased likelihood to have 100 plus. So quick and easy to check!") (1/10, 10%), and some identified having already completed skin assessments (eg, "was a great refresher") (2/10, 20%).

Discussion
A strength of this study is the rigorous, transparent development of the videos and the use of an expert panel of dermatologists to ensure valid content. Each specific strategy recommended by Sidani and Braden for operational transparency of the video intervention was outlined (see Multimedia Appendix 2) and graded by the dermatology expert panel using a CVI. This process is absent from many studies describing intervention development [37]. The CVI indicated that the videos were relevant, except for two index scores. Scores for both right arm nevus count and discuss systemic approach decreased from 4 (highly relevant) to 3.67 (between highly relevant and quite relevant). According to the scale, both components were ranked as at least quite relevant and were kept in the videos. One explanation for this alteration is that there was a different, third expert reviewer during the second round. Otherwise, clarity increased overall during the second round of reviews. The expert panel's comments primarily focused on promoting clarity, such as adding the definition of skin cancer to the first video and the definition of a pigmented lesion to the third video. Wording was also adjusted (eg, "get melanoma" to "develop melanoma" and "11 nevi tool" to "right arm nevus count"). The expert panel also offered relevant CSE tips, such as ensuring that the patient removes glasses or hearing aids during the head-to-toe skin examination to better visualize the conchal bowl.
Prior to the dissemination of videos to the PCNPs, the dermatology expert panel evaluated the videos' technical production on Vimeo and their delivery through REDCap. Visual quality, audio quality, and audio-visual relationship were considered exceptional, suggesting that it was feasible to use Vimeo and REDCap to deliver the video intervention.
The hypothesis that the enrollment rate will be equal to or better than 60% was not supported. During the study, 22 NPs were recruited at a conference in November 2018. However, enrollment did not start until February 2019, and just 5 NPs consented after the initial study invitations were sent. An additional 10 participants were recruited at another NP meeting and 8 more participants were enrolled. This highlights challenges in recruiting NPs [48], as well as the importance of a timely follow-up after recruitment.
The hypothesis that the retention rate will be greater than or equal to 50% was supported. Out of 12 recruited participants, 10 (83%) completed the surveys. This is comparable to the retention rate of a prior skin cancer study involving NPs, which reported a retention of 10 out of 14 participants (71%) [13].
Adding strength to this study are the comments from the 2 participants who did not progress past the first video. The first participant stated that she was unable to complete the other videos because of work requirements. She rated the first video favorably, stating that it was very educational. The second participant accessed the first video on her mobile phone without problems but had difficulty accessing the second video, receiving a message that she had already reviewed the video. When she contacted the investigator, she was outside the 1-week time limit for video completion and was withdrawn from the study. Future studies will be formatted to have the REDCap email, which includes the video link, request that participants check their Wi-Fi or cellular connections prior to opening the REDCap link and starting the video.
The hypothesis that video intervention adherence will be greater than or equal to 50% was not supported. Vimeo defines a finish as when the participant views a video to the very last frame. However, the amounts viewed of each video were high: the average amount of video 1 watched was 92% and it was played 6 out of 10 times (60%), the average amount of video 2 watched was 97% and it was played 7 out of 10 times (70%), and the average amount of video 3 watched was 98% and it was played 8 out of 10 times (80%). Participants likely viewed all the content; however, they most likely dropped off at the references and acknowledgements portions of the videos. We were unable to retrospectively connect the times the videos were played according to the Vimeo report with the time the participant finished the REDCap module. However, at least 5 out of 10 PCNPs (50%) fully viewed the content before exiting out of the video and met the "finishing" criteria. Future studies will require active monitoring of the reports to be able to connect the participants' REDCap and Vimeo information or to obtain a time stamp on REDCap. One way to obtain a time stamp would be to keep a daily record of which participants completed the REDCap survey and correlate this record with the time the Vimeo video was viewed. To increase the number of finishes as defined by Vimeo, the references and the acknowledgements will be moved to the beginning of the video. References can be sent with the email invitation and the acknowledgements can be placed at the start of the video.
The hypothesis that usability scores will be equal to or higher than 70 was supported. The PCNP's mean SUS scores were 85, or better, indicating that the participants viewed the usability of REDCap and Vimeo as positive. Similar SUS scores have been recorded for other interventions, such as a web-based simulation in psychiatry residency training (n=16, score 86.5) [49] and a web-based multimedia application called Electrolyte Workshop, which has e-learning tutorials called WalkThru (n=18, score 87.9) and HandsOn (n=27, score 81.5) [50].
Another strength of this study was the use of the microlearning framework to guide intervention development. The hypothesis that acceptability scores for microlearning will be equal to or higher than 5 was supported. Mean scores for the scales of perceived usefulness, perceived ease of use, behavior, and affection all were higher than those reported by Liang et al [30]. These findings suggest that microlearning was an acceptable framework for this intervention. However, further testing is needed to further demonstrate that microlearning is an optimal framework for teaching complex concepts to busy practitioners in a short amount of time.
There were also some limitations to the study. Purposive sampling decreases the generalizability of the results and adds to selection bias [51]. Only the participants who completed the intervention were able to complete the usability, acceptability, and satisfaction surveys. Another limitation to this study was the posttest-only design. This design does not allow for a comparison between the participants before and after the video interventions. However, this design is appropriate for a feasibility study, where the focus is on the practicality of the study [51].
In conclusion, three theory-based, short videos with content on CSE for malignant melanoma were developed that are suitable for internet delivery to PCNPs in various formats. Findings from this feasibility study provide a foundation for the use of microlearning as a guide for delivering brief CSE training to PCNPs. The findings also provide support for using the videos as part of an intervention in future trials targeting behavioral CSE outcomes in PCNPs or other practitioners. This feasibility study provided valuable lessons to inform components of the next research phase, such as the timeliness of enrollment and redesign of the modules to support better measurements of intervention adherence. The long-term goal is to promote the early detection of skin cancer by providing CSE education to PCNPs and ultimately improving skin cancer prognoses.

Acknowledgments
This study was funded by the University of Arizona Cancer Center, Skin Cancer Institute, Seed Grant Program and the College of Nursing Fall 2018 Frederick Lange Memorial Endowment Award. The authors acknowledge Stephen Purcell for video production assistance.

Conflicts of Interest
None declared.