This paper is in the following e-collection/theme issue:

Human Factors and Usability Case Studies (787) Medicine 2.0: Social Media, Open, Participatory, Collaborative Medicine (1641) Focus Groups and Qualitative Research for Human Factors Research (851) Patient Education for Cancer (147) Formative Evaluation of Digital Health Interventions (2410) Children and Young People With Cancer, Pediatric Oncology (59)

Published on in Vol 22, No 7 (2020): July

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/18779, first published .
Development and Usability Evaluation of a Facebook-Based Intervention Program for Childhood Cancer Patients: Mixed Methods Study

Development and Usability Evaluation of a Facebook-Based Intervention Program for Childhood Cancer Patients: Mixed Methods Study

Development and Usability Evaluation of a Facebook-Based Intervention Program for Childhood Cancer Patients: Mixed Methods Study

Authors of this article:

Bu Kyung Park1 Author Orcid Image ;   Ji Yoon Kim2 Author Orcid Image ;   Valerie E Rogers3 Author Orcid Image
Article Authors Cited by (6) Tweetations (3) Metrics

    Original Paper

    1College of Nursing, Research Institute of Nursing Science, Kyungpook National University, Daegu, Republic of Korea

    2Department of Pediatrics, School of Medicine, Kyungpook National University, Daegu, Republic of Korea

    3School of Nursing (retired), University of Maryland Baltimore, Baltimore, MD, United States

    Corresponding Author:

    Bu Kyung Park, PhD

    College of Nursing

    Research Institute of Nursing Science

    Kyungpook National University

    680 Gukchaegosang-ro, Jung-gu

    Daegu, 41944

    Republic of Korea

    Phone: 82 53 420 4929

    Fax:82 53 421 2758

    Email: bukpark@knu.ac.kr


    Background: Childhood cancers previously considered to be incurable now have 5-year survival rates up to 84%. Nevertheless, these patients remain at risk of morbidity and mortality from therapy-related complications. Thus, patient education and self-management strategies for promoting a healthy lifestyle are of tantamount importance for improving short- and long-term health outcomes. A Facebook-based “Healthy Teens for Soaam” (a Korean term for childhood cancers) program was developed to help improve knowledge and self-management practices of teens with cancer related to their disease and treatment.

    Objective: The two-fold purpose of this usability study was (1) to describe the process of developing an 8-week Facebook-based intervention program for teens with cancer, and (2) to evaluate its usability to refine the program.

    Methods: Multiple phases and methods were employed to develop and evaluate the usability of the program. Study phases included: (1) needs assessment through focus group interviews and qualitative content analysis, (2) development of module content, (3) expert review and feedback on module content, (4) Facebook-based program development, (5) usability evaluation by heuristic evaluation, (6) usability evaluation by targeted end-user testing, and (7) modification and final version of the program. Usability of the final version was confirmed through feedback loops of these phases.

    Results: Based on 6 focus group discussion sessions, it was determined that teens with cancer were interested in seeing stories of successful childhood cancer cases and self-management after discharge, and preferred multimedia content over text. Therefore, each Facebook module was redesigned to include multimedia materials such as relevant video clips tailored for teens. Usability assessed by heuristic evaluation and user testing revealed several critical usability issues, which were then revised. Potential end users tested the final program and perceived it to be usable and useful for teens with cancer.

    Conclusions: To our knowledge, “Healthy Teens for Soaam” is the first Facebook-based intervention program for teens with cancer. We actively worked with current childhood cancer patients and survivors to develop and improve this program, achieved good usability, and met the expressed needs and preferences of target end users. This 8-week Facebook-based educational program for teens with cancer, developed as the first step of an upcoming intervention study, will be useful for improving knowledge and self-management strategies of teens.

    J Med Internet Res 2020;22(7):e18779

    doi:10.2196/18779

    Keywords

    pediatric cancer patients; childhood cancer; social network site; Facebook; usability 


    Survival rates associated with childhood cancers continue to improve, partly due to advances in diagnostic techniques and treatment modalities along with clinical research [1]. According to the Surveillance, Epidemiology, and End Results Cancer Statistics Review, the 5-year survival rate of children with leukemia 0-19 years of age in the United States is 84.1% and the 5-year survival rate of patients with childhood brain and central nervous system cancers is 74.8% [2]. In South Korea, the 5-year survival rate among patients with childhood cancers overall was 81.7% in 2014 [3].

    Despite improvements in survival, childhood cancer survivors still face a high risk of therapy-related complications or adverse effects that persist or arise after completion of treatment [4]. Childhood cancer survivors are at significantly increased risk of relapse, second malignancy, and long-term late effects, including cardiovascular and pulmonary dysfunction, endocrine disorders such as metabolic syndrome, and others [5,6]. Specifically, childhood cancer survivors of 5 years or greater have at least one chronic health condition [7]. A study on the late effects of childhood cancer treatment found that the cumulative incidence of a chronic health condition among long-term childhood cancer survivors was 99.9% by 50 years of age, with a nearly 2-fold greater cumulative burden among survivors compared to matched community controls [5].

    Some morbidities are modifiable through preventive healthy behaviors such as physical activity, good nutrition, obesity prevention, and avoidance or cessation of smoking, as well as identification of characteristics that may modify morbidities or adherence to healthy behaviors to better target interventions [8,9]. For example, one study showed that childhood cancer survivors of older age and lower socioeconomic status were less frequently engaged in preventive healthy behaviors [10]. Another study reported that a greater proportion of female survivors smoked compared to teens without cancer [11]. Although childhood cancer survivors have been found to engage in unhealthy behaviors (eg, tobacco, alcohol, drug use, and sexual behaviors) and to be nonadherent to national health behavior guidelines at rates similar to those of their healthy siblings and teens without cancer [12,13], these behaviors are likely to be more consequential for individuals with organ damage secondary to cancer treatment. Moreover, late effects of treatment such as obesity, cancer-related pain, and sensory impairments have been significantly associated with increased risk of comorbid symptoms [14]. Increased comorbidities are associated with decreased quality of life, and with an increased risk of hospitalization and mortality [15]. Thus, the importance of patient education and promoting self-care for a healthy lifestyle for teens with cancer is increasingly becoming recognized to help mitigate complications of cancer and its treatment.

    Children currently undergoing cancer treatment, as well as survivors, require follow-up care for the rest of their lives; thus, the Children’s Oncology Group (COG) has emphasized the importance of regular medical follow-up and has developed the “COG long-term follow-up guidelines for survivors of childhood, adolescent, and young adult cancers” [16]. In addition, various interventions to improve long-term outcomes for teens with cancer have been developed worldwide. For example, the Nursing Discipline of the COG has developed key principles and recommendations for patient and family education practices [17] as well as interventions for teens with cancer, such as nutrition and cooking workshops [18], nutrition and body weight changes [19], systematic intervention for psychological preparation for radiotherapy treatment [20], mobile health intervention to improve adherence and quality of life [21], and a web-based physical activity intervention [22]. However, few intervention studies involving teens with cancer have been conducted in Korea. Existing examples include web-based patient safety education [23], an art intervention for siblings of children with cancers [24], and educational interventions to enhance adherence to prophylactic treatment [25]. Intervention programs for preventing long-term late effects of childhood cancers and their treatment are glaringly absent. In particular, there is a crucial need for interventions that promote self-management strategies for achieving a healthy lifestyle among teens with cancer. This is important, as they will likely survive into adulthood and need to develop the knowledge and skills necessary to prevent or mitigate the late effects of cancers, and thereby improve their health-related quality of life.

    Knowledge about their disease, the importance of treatment adherence, and strategies to improve adherence could be improved through innovative health interventions designed specifically for teens [21]. Among teens with cancer, inadequate information was identified as one of the barriers to compliance with chemotherapy [26]. Adult survivors of childhood cancers were found to lack detailed knowledge of their treatment history and risk for late effects [27]. Therefore, further efforts are needed to educate and empower teens with cancer to gain appropriate knowledge and assume responsibility for their health management [28].

    Social media has been increasingly utilized as a platform for delivering health interventions in childhood cancer care. For example, Watson [29] reported that oncology health care professionals utilized social media to listen, learn, engage, and cocreate to advance cancer care. Teens with cancer have also turned to social media for information about cancer or to interact with peers and others about their diagnosis and its impact [30]. Importantly, the effectiveness of using social media platforms such as Facebook has been reported for this tech-savvy generation. For example, a previous study [31] found that a randomized trial of a Facebook-based physical activity intervention among young adult cancer survivors significantly increased their physical activity. Therefore, we developed an educational program for teens with cancer on Facebook called “Healthy Teens for Soaam” (soaam is the word for pediatric cancers in Korean), as the first step of an intervention that will deliver educational materials with the aim of increasing knowledge and improving self-care practices related to their disease and treatment. The purpose of this usability study was (1) to describe the process of developing this 8-week modular intervention program for teens with cancer and (2) to evaluate its usability to refine the program.


    Study Design

    This study was reviewed and approved by the principal investigator’s institutional review boards (1041078-201606-HRSB-122-01 and 20200026). The study was conducted in 7 phases (Figure 1) and was guided by the Analysis, Design, Development, Implementation, and Evaluation (ADDIE) instructional design approach [32]. This article describes the first 3 phases of the ADDIE approach.

    ‎
    Figure 1. Flow chart of "Healthy Teens for Soaam" program development.
    View this figure

    Phase 1: Needs Assessment Through Focus Group Discussion and Qualitative Content Analysis

    Semistructured focus group discussions were conducted with 12 teens with cancer. Participants were recruited from childhood cancer self-support groups in Korea using convenience and snowball sampling methods. A flyer was posted to the online platforms of self-support groups on popular Korean online platforms (Naver Café, KakaoTalk, and Band), and interested volunteers contacted the researchers by phone or email. Interested participants also suggested other potential participants to the researcher.

    A total of 6 in-person focus group discussion sessions were conducted, during which participants were accompanied by their parent or legal guardian. Each session included 2 to 5 participants, and each lasted for about 60 to 90 minutes. Focus group discussions were facilitated by the principal investigator (BP) and a research assistant. First, the principal investigator explained the overview of the program, distributed handout materials detailing program contents, and asked participants and parents to provide their opinions regarding the contents. For example, they were explicitly prompted to provide suggestions regarding the addition, removal, or emphasis of content. Besides answering questions, participants and parents were encouraged to share their personal experiences with childhood cancer using prompts such as “If you have anything you want to share with us even though it is not directly related to the contents, don’t hesitate to tell us.” Focus group discussions were recorded and transcribed verbatim for qualitative content analysis using NVivo 11 (QRS International, Burlington, MA, USA).

    Focus Group Discussion Data Analysis

    Qualitative content analysis was conducted using a combination of the content analysis method suggested by Krippendorff [33] and the inductive coding approach suggested by Elo and Kyngäs [34]. Coders read the transcripts and freely generated themes. They then developed categories from similar themes as the coding progressed. Major coding rules included the following: (1) a category was created if a list of at least three themes was generated under that category, and (2) multiple sentences from the same participant that referred to the same content were coded as one unit.

    Qualitative content analysis was conducted by the two coders (the principal investigator [BP] with a PhD in nursing and the research assistant with a master’s degree in nursing) to ensure reliability of the analysis. Coders independently analyzed the transcripts. Questions or disagreements regarding the themes or categories were discussed. Revision and refinement of themes and categories continued until content analysis was completed.

    Based on results from focus group discussions, the program module content was organized using the following usability design methods. First, we used group information together (also known as card sorting) after the focus group discussions. We defined the important themes from the focus group discussions and grouped them into several categories such as diagnosis, examination, treatment, social support, and return to society/school. Second, we developed a structure for 8 weekly modules. Third, we developed low-fidelity module prototypes such as paper prototypes. Finally, we established high-fidelity mockups using PowerPoint slides.

    Phase 2: Development of Module Content

    Main topics for the program module content were developed based on the results from the focus group discussion “needs assessment.” Module content was written at a sixth-grade reading level using the Flesch–Kincaid readability test guidelines [35,36]. Content was developed with guidance from published research articles, medical and nursing textbooks, information available from health professional organization websites (eg, Korean Association for Children with Leukemia [37]), patient education materials in hospitals, and consultations with a panel of experts.

    We incorporated usability principles for teen-friendly websites [38] into the program prototype in terms of content, appearance, and navigation. For example, we developed the content using PowerPoint 2016 (Microsoft Corp, Redmond, WA, USA) slides visualized on a single page without scrolling [38,39]. To improve appearance, we used large font sizes and avoided garish color schemes [38,39]. Moreover, slides for each module had their own theme to distinguish them from other modules. Although Facebook posts appear in reverse chronological order (ie, the most recent post appears on the top of users’ News Feeds), this was not considered a potential problem for participants because the module content was designed to be updated weekly and potential target participants were familiar with Facebook navigation.

    Each module included specific learning objectives, structured learning material for fundamentally important content with external web links for further information, multimedia such as video clips or games relevant to the topics, and discussion topics at the end of the modules. Participants were asked to respond to discussion topics by adding comments on Facebook. This served to monitor module participation and sharing of personal experiences for social support. Although not done during testing, in the implementation study, a program moderator (researcher) will provide feedback and encourage interactions among participants on discussion boards. Discussions included the following topics: “Let’s talk about blood transfusion experiences. Have you ever received a blood transfusion? If yes, please tell us about the specific blood product you received, your feelings, and side effects,” and “Have you had any pain while getting treatment in the hospital? Which one was the most painful, and how did you overcome the pain—pain medication, massage, and ice bag?”

    Phase 3: Expert Review and Feedback on Module Content

    The module content was reviewed by four experts currently working in pediatric hematology and oncology units in tertiary hospitals, who each had more than 10 years of experience: two nurses with master’s degrees in pediatric nursing and two pediatric hematology and oncology physicians with PhDs. Reviewers were knowledgeable about the most recent evidence in this highly specialized field and reviewed the module contents for accuracy. They were asked to rate the ease of understanding and appropriateness of information for teens with cancer (1=strongly disagree; 2=disagree; 3=agree; 4=strongly agree). Based on their feedback, module contents were revised and finalized for mockups of the Facebook-based program.

    Phase 4: Facebook-Based Intervention Program Development

    Facebook Group was used to develop the 8-week intervention program, with the “private group” functionality utilized to protect participants’ privacy. With a private group, only people who are invited by the group creator (principal investigator of this study) can join and see who is in the group and what they post [40]. We chose the private group option because, as established by the focus group discussions, many participants did not want to be widely known as cancer patients. Additionally, we customized the group’s privacy options [41], for example, by selecting the “hide group” and “membership approval” options so that only members could find the group and that only the group creator could approve new members.

    To deliver module content, we composed PowerPoint slides and then uploaded each slide to the group’s Photos section. Participants could easily navigate the slides by clicking the “next” and “back” buttons. Considering that the purpose of this program is to deliver accurate and reliable educational content, only group administrators were granted posting permission, but participants were allowed to add comments to each post to share their opinions or contribute to discussions.

    Phase 5: Usability Evaluation by Heuristic Evaluation

    Two experts in the field of usability and human-computer interactions reviewed the Facebook-based program using Nielsen’s heuristics [42,43] adapted and tailored for children’s electronic learning (eLearning) program evaluation [44]. The two usability experts have PhDs in health care informatics and were currently teaching and conducting research on health care informatics in universities, one for 17 years and the other for 5 years. The criteria of Nielsen’s heuristics tailored for eLearning in children were: (1) visibility of system status; (2) match between system and the real world; (3) user control and freedom; (4) consistent and standards; (5) error prevention; (6) recognition rather than recall; (7) flexibility and efficiency of use; (8) esthetic and minimalist design; (9) help users recognize, diagnose, and recover from errors; (10) help and documentation; (11) design attractive screen layout; (12) use appropriate hardware devices; (13) challenge the child; (14) evoke child mental imagery; and (15) support child curiosity [44]. Comments from the heuristic evaluation were analyzed employing content analysis using the inductive coding approach described in Phase 1. The program was revised according to feedback from the heuristic evaluation.

    Phase 6: Usability Evaluation by User Testing

    User testing was conducted using observation, the think-aloud method, voice and screen activity capture using Camtasia 9 (TechSmith, Okemos, MI, USA), and surveys. Participants for this phase were recruited through parents of teens belonging to a childhood cancer self-support group in Korea using convenience and snowball sampling. A flyer was posted to the online platforms of self-support groups on KakaoTalk and Band, and interested volunteers contacted researchers by phone or email. Inclusion criteria were teens with cancer who were (1) aged 13 to 18 years old, (2) diagnosed with any type of childhood cancer, (3) received any type of cancer treatment (eg, radiation therapy, chemotherapy, stem cell transplant), and (4) at any stage of treatment (newly diagnosed through completed treatment). Participants arranged appointments for screening, consenting, and user testing. As the participants were adolescents, both informed consent from their parents or guardians and assent from the teens were obtained prior to participation.

    Procedures

    A total of 11 face-to-face, 1:1 user testing sessions were conducted. Testing was facilitated by the principal investigator in a private room at the study site. Before conducting user testing, the researcher gave instructions about the program and usability evaluation methods such as the think-aloud method and Camtasia screen-capture software. Participants were then provided with an information page that included the Facebook program link, user ID, and password. They were assigned a user testing task to review 2 out of the 8 modules, including logging in, navigating through instructions on the first page to modules, reading content, watching video clips, and adding comments. They were assured that they could stop at any time for any reason. Each session lasted 30 to 40 minutes.

    During evaluation of the program, participants were encouraged to think out loud while their voices were recorded using Camtasia. During pauses from thought verbalization, participants were probed with the following prompts: “What do you think about this? What features do you think need improvement?” At the same time, the researcher documented comments from participants and her own observations using the user testing worksheet.

    Measures

    Before user testing, participants’ demographic characteristics as well as their internet and Facebook usage habits were queried and documented. After user testing, participants filled out a short survey that evaluated electronic health (eHealth) literacy and perceived usability. The eHealth literacy was assessed using the eHealth Literacy Scale (eHEALS) [45], which has 8 items and assesses an individual’s knowledge, comfort, and perceived skills for locating, evaluating, and applying eHealth information for health issues. The Korean version of eHEALS, known as K-eHEALS, was previously tested for reliability and validity [46]. Total eHEALS scores range from 8 to 40, with a higher score indicating better health information literacy. Cronbach α of the original scale was .88 [45] and was also .88 for the Korean version [46]. Cronbach α for this study was .71.

    Perceived usability of the program was assessed using the Perceived Health Website Usability Questionnaire (PHWSUQ) [47]. This questionnaire has 12 items scored using a 7-point Likert scale. It measures three dimensions of usability: satisfaction, ease of use, and usefulness. Cronbach α values for the subscales ranged from .64 to .93, and was .85 for this study.

    Data Analysis

    Quantitative data from the questionnaires were analyzed with descriptive statistics using SPSS Statistics for Windows, version 25.0 (IBM Corp, Armonk, NY, USA). Audio recordings from user testing sessions were transcribed verbatim and analyzed employing the content analysis method described in Phase 1. Qualitative data, collected throughout observation and screen activity and recorded by Camtasia, provided a rich contextual background and a strong source of triangulation for developing themes and generating a comprehensive review of the program’s usability. Thus, screen recordings were reviewed when the transcript did not capture enough usable detail, which improved the accuracy of content analysis.

    Phase 7: Modification and Final Version of the Program

    Modifications regarding content and appearance issues were made on the PowerPoint slides, and the revised slides were uploaded to Facebook (the process was conducted in the order phase 2 to phase 4 to phase 7). If content revisions required expert opinions, relevant experts were contacted to confirm the accuracy of the new content (the process was conducted in the order phase 2 to phase 3 to phase 4 to phase 7).


    Phase 1: Summary of Focus Group Discussions

    Participant characteristics are summarized in Table 1 [48].

    Participant feedback regarding the outline of the program (Table 2) included (1) a desire for stories of successful cases and self-management at home, (2) a preference for multimedia content (eg, video clips) rather than text, (3) requests for more patient safety–related material, (4) an expressed need for detailed program objectives and instructions, and (5) appeals for more information on treatment, prognosis, and medical terminology. One teen expressed their support for the program as follows:

    This is the big problem. The reason I’m so supportive of this education program (Healthy Teens for Soaam) is that there’s a limitation of internet and you narrow down the range a little bit…... I hope the contents go on in the future.

    Additionally, participants freely described and shared their experiences with cancer treatments, and the emerging themes and categories dealt with information needs, support systems, barriers to treatment, facilitators to treatment, return to social life, and health care system issues. Other representative comments from the focus group discussions are provided in Multimedia Appendix 1.

    Table 1. Characteristics of focus group discussion participants (N=12).
    CharacteristicValue
    Age (years), mean (SD)15.2 (5.3)
    Sex, n (%)

    		Male6 (50)

    		Female6 (50)
    Diagnosis, n (%)

    		Leukemia6 (50)

    		Brain tumor2 (17)

    		Aplastic Anemia2 (17)

    		Other2 (17)
    Treatment, n (%)

    		Completed with ongoing outpatient follow up9 (75)

    		Ongoing chemotherapy3 (25)
    Table 2. Content analysis of focus group discussion transcriptions (N=12).
    Theme

    		Frequency of units
    Feedback on the program

    		Add story of successful cases and self-management after discharge9

    		Prefer multimedia (video clips) than text10

    		Add patient safety issues11

    		Need detailed program objectives and instructions13

    		Add information on treatment, prognosis, medical terms20
    Information needs

    		Lack of information or inaccurate information7

    		Current sources of information (eg, other patients, hospital handout, searching the internet)19

    		Useful information during treatment (personal experience)20
    Support system

    		Current support from hospital, school, pediatric cancer associations, self-support groups, etc10

    		Need systematic supporting system for patients and families13

    		Useful support during treatment (personal experience)15
    Barriers to treatment

    		Economic problems5

    		Hospital-related issues (eg, lack of available beds for admission, manpower shortage, miscommunications between health care providers)8

    		Side effects of treatment (eg, chemotherapy, transfusion, infection)25

    		Emotional reactions at the time of diagnosis and during treatment28
    Facilitators to treatment

    		Patients’ insight on diagnosis4

    		Empathy of health care providers6

    		Social support from friends, other patients, family members13
    Return to social life

    		School life4

    		Lack of physical activity6

    		Concerns related to infection prevention, weak immunity10
    Health care system issues

    		Disabled child registration and welfare benefits2

    		Inadequate and poor social welfare system4

    		Health insurance5

    Phases 2 and 3: Facebook-Based 8-Week Intervention Program

    An overview of the final version of the program is provided in Table 3. Content was revised and updated according to focus group discussion feedback. Participants and parents were interested, for example, in the topics of returning to school and the effects of chemotherapy on the adolescent’s fertility; therefore, we added these topics to the modules. Additionally, as participants preferred multimedia over reading text, we searched for movies and Korean dramas depicting stories involving teens with cancer and included links to short video clips in the relevant module content. Animations explaining cancers and their treatment tailored for teenagers were also added for a better understanding of how treatment proceeds.

    Four childhood cancer experts reviewed the content and provided feedback on whether the information was accurate and current, including the pictures and video clips from movies and Korean dramas. Two nurses reviewed relevant medical terminology from academic journals and textbooks, and noted that it differed from the terminology commonly used by participants and parents. For example, granulocyte colony-stimulating factor injections for neutropenia were usually referred to as “count shots” (ie, a shot given when the absolute neutrophil count is low) or “Grasin” among participants and parents. Thus, we also used these words in the program content. Two physicians contributed information on the most recent treatment practices for childhood cancers in tertiary hospitals. For example, a previous version contained only bone marrow transplantation, but we added information about cord blood transplantation and peripheral blood stem cell transplantation.

    Table 3. Overview of the Facebook-based Healthy Teens for Soaam program.
    Module (Week)Contents
    IntroductionProgram purpose, program schedule, how-to-use tutorial video clip
    Module 1 (Week 1)Pediatric cancers characteristics
    -Pediatric cancers statistics (risks and causes of pediatric cancers)
    -Pediatric cancers symptoms, early signs
    -International Childhood Cancer Day (February 15)
    -Psychosocial services and support for children and families
    Module 2 (Week 2)Types of pediatric cancers and their characteristics
    -Bone cancers, brain cancers, leukemia, hepatoblastoma, lymphoma, neuroblastoma, rhabdomyosarcoma, retinoblastoma, Wilms tumor, etc
    Module 3 (Week 3)Diagnostic tests
    -Imaging tests, CTa scans, MRIb, ultrasound, blood tests
    -Bone marrow aspiration and biopsy
    Module 4 (Week 4)Pediatric cancers treatments 1
    -Hickman catheter insertion and management
    -Chemo-port insertion and management
    -Radiation treatment and chemotherapy
    Module 5 (Week 5)Pediatric cancers treatment 2
    -Treatment team and frequently used medical terminology
    -Caring tips for mouth care and oral mucositis
    -Caring tips for infection and bleeding
    -Symptoms of infection, ANCc
    Module 6 (Week 6)Pediatric cancers treatment 3
    -Managing side effects of radiation and chemotherapy
    -Pain, transfusion
    -Growth and development, fertility
    -Stem cell transplantation (BMTd, CBTe, PBSCTf)
    Module 7 (Week 7)Back to school and society
    -Follow-up care after cancers treatment
    -Facilitating school reentry guide
    -Nutrition and daily activity
    Module 8 (Week 8)People around me: family and friends
    Review and summary

    aCT: computed tomography.

    bMRI: magnetic resonance imaging.

    cANC: absolute neutrophil count.

    dBMT: bone marrow transplantation.

    eCBT: cord blood transplantation.

    fPBSCT: peripheral blood stem cell transplantation.

    Phases 4 and 5: Revision After Heuristic Evaluation

    Usability experts pointed out usability problems of the program among three categories of Nielsen’s heuristics tailored for children’s eLearning [44]: (3) user control and freedom, (7) flexibility and efficiency of use, and (10) help and documentation. For example, even though Facebook has a Help menu for user support, the experts suggested that there should be a Help section specific to the program. Therefore, we added a “Help and FAQs” section for dealing with problems encountered while using the program. However, some usability issues were not changeable on the ready-made Facebook platform; these included Facebook advertisements and the reverse chronological order of posts.

    All four childhood cancer experts rated the content highly for its ease of understanding (4=strongly agree) and appropriateness for teens with cancer (4=strongly agree). They also commended the detailed instructions and the neat and consistent overall design of the PowerPoint slides.

    Phases 6 and 7: Revisions After User Testing

    User testing participant characteristics (Table 4) included a mean age of 16.7 years, and the majority had leukemia and were Facebook users (91%). Their mean K‑eHEALS score suggested good eHealth literacy. The mean PHWSUQ score (reflecting the program’s perceived usability) indicated that the program was perceived as usable. All 11 participants completed the user testing tasks and 9 participants were able to navigate through the program with little or no guidance. The mean testing time was 38.8 (SD 3.1) minutes. Representative quotes from user testing are shown in Multimedia Appendix 1.

    Table 4. Characteristics of user testing participants (N=11).
    CharacteristicValue
    Age (years), mean (SD)16.7 (1.1)
    Sex

    		Male7 (64)

    		Female4 (36)
    Diagnosis

    		Leukemia8 (73)

    		Other3 (27)
    Facebook user

    		Yes10 (91)

    		No1 (9)
    K-eHEALSa, mean (SD)27.6 (1.3)
    PHWSUQb, mean (SD)

    		Total PHWSUQ60.1 (2.8)

    		Satisfaction32.9 (4.8)

    		Ease of use15.4 (2.9)

    		Usefulness17.8 (3.4)
    Preferred frequency of content update, n (%)

    		Every day1 (9)

    		2-3 times per week4 (6)

    		Once per week6 (55)

    aK-eHEALS: Korean version of the eHealth Literacy Scale (range 8-40).

    bPHWSUQ: Perceived Health Website Usability Questionnaire (range 12-84 total; 6-42 satisfaction; 3-21 ease-of-use and usefulness).

    The transcript of user testing content analysis identified 20 themes under four categories: program content, program appearance, navigation, and others, which mainly included feedback about the strengths of the program (Table 5). Participants mentioned that they gained some new knowledge from this program, but some perceived the content as difficult to understand or as having low readability because of the medical terminology. Regarding the appearance of the program, participants generally liked the layout, but three users noted, for example, that some text was hidden behind images or that low-resolution figures were difficult to decipher. Nine participants had no issues with navigation of Facebook, but two participants needed help from the researcher. Seven participants expressed their impressions about the discussion section, where they could share personal experiences with participants who had similar experiences. For example, one teen said: “Then if I don’t know the details right now, I think it would be helpful for patients to learn this by Facebook myself.”

    Critical usability issues reported during user testing were reviewed, and, where necessary, revisions were made and confirmed through feedback loop of the phases (Figure 1). Regarding usability issues with “program content,” one participant suggested a movie on a pediatric patient, which was not in our movie list. She mentioned that it had helped her a lot when she underwent treatment. In this case, childhood cancer experts reviewed the movie and included relevant video clips to the module. Some video clips and pictures that were not working or had low graphic resolution were replaced with other video clips and pictures. Usability issues with “program appearance” were mostly minor revisions such as typos, font size, or picture location issues, which were corrected directly on the PowerPoint slides, and then the revised slides were reuploaded on Facebook. There were few usability issues with “navigation.” Participants easily recovered from failed paths, with brief help from the researcher. However, for first-time users of Facebook, we added a short video tutorial of the program navigation. Multimedia Appendix 2 shows a screenshot of the final version of the “Healthy Teens for Soaam” program on Facebook.

    Table 5. Content analysis of user testing transcriptions (N=11).a
    Category, ThemesType of feedbackFrequency of units
    Program Content

    		Reading level of elementary or middle schoolPositive2

    		Explanations on medical terminologies are usefulPositive7

    		Video clips and pictures were helpfulPositive14

    		Easy to understand, good readabilityPositive28

    		Helpful and useful educational contentPositive34

    		Knowledge/information first learned from this programPositive35

    		Need revisions or irrelevant informationNegative2

    		Some video clips and pictures were not working or had low graphic resolutionNegative3

    		Need more information on specific topicsNegative5

    		Content was not easy to understand, low readability (use of difficult medical terminology)Negative7
    Program Appearance

    		Liked layout (eg, font style, font size, sentence length, location of pictures and paragraphs)Positive23

    		SuggestionsNegative3

    		Issues with layout (eg, picture sizes and locations)Negative6
    Navigation

    		Familiar with Facebook navigationPositive6

    		Got lost or failed path (needed help from the researcher)Negative2
    Others (strengths of program)

    		Reasons for not doing or following self-management at homePositive3

    		Provide online social supportPositive4

    		Good sources of information: from health care professionals, school, the internet, family, self-experiencePositive14

    		Improved intention (attention) to know about their diseasePositive15

    		Can share personal experiences on treatmentPositive17

    aRepresentative comments from user testing are shown in Multimedia Appendix 1 (quotations from focus group discussions and user testing).


    Principal Findings

    The aim of this study was to develop a Facebook-based intervention program for childhood cancer participants and to evaluate its usability to guide program refinements. Overall, the evaluation of “Healthy Teens for Soaam” revealed that the program was perceived as usable by our participants, who were representative of our target audience (teens with cancer). Seven phases of development and usability evaluation uncovered usability issues as well as areas to enhance user satisfaction, which were then modified accordingly.

    Childhood cancer participants and families wanted a comprehensive online information source where they could find childhood cancers–related information. Preexisting sources of information (eg, other patients, hospital handouts, searching the internet) were not reliable and potentially provided limited or outdated information. Therefore, focus group discussion participants greatly valued the objectives and content of this program. Moreover, participants supported the discussion topics at the end of each module. Considering the characteristics of adolescence, peer groups are important for their socialization; however, hospital admissions and principles to prevent infection limit their opportunities to participate in peer group activities. Online interactions among teenagers with similar diagnoses provide the next-best opportunity for social support.

    User testing by potential end users revealed that the program content was comprehensive. Previous research has found the two main predictors of noncompliance to chemotherapy to be child resistance and inadequate information [26]. To overcome these barriers, this program provided tailored information to teens with the goals of improving their knowledge and reducing their resistance to treatment by explaining the rationale behind treatments and procedures, and may therefore improve treatment compliance among teens with cancers.

    By adapting the Facebook platform to this program, we took advantage of the following key points: (1) potential end users are familiar with navigation of the program, (2) researchers can reduce costs for developing new online platforms, and (3) researchers may be able to improve treatment fidelity, as teenagers access Facebook almost every day and are continuously connected via the smartphone app. By contrast, the limitations of using Facebook [49] include: (1) design-related limitations (eg, limited freedom and options for text editing and background color schemes) and (2) the inability to control certain features such as advertisements and chronology of posts.

    Next Steps

    The next step of this “Health Teens for Soaam” program will be delivering this 8-week Facebook-based intervention for teens with cancer in South Korea. The purpose will be to increase knowledge and provide accurate, up-to-date information about self-management strategies for improving health and wellness. Through this intervention, we will be able to determine the effectiveness and efficiency of the Facebook-based intervention program.

    A previous Facebook-based intervention program for young adult cancer survivors found that engagement with the Facebook program was variable, and investigators recommended that future research should explore how to promote sustained engagement in online social networking [31,50]. Our program includes discussion sections similar to the previous intervention [31,50], in that relevant discussion topics are included in each module. To foster interaction and social support between participants, a moderator will encourage and remind users of discussion sections. A review of the use of social media for teens with childhood cancer [30] reported that health care providers are increasingly integrating social media into their professional life, and that it provides several advantages for both patients and health care providers. For example, the presence of health care providers presents an opportunity for direct interaction with patients and the ability to provide them with reliable, data-based health information. In turn, the health care providers also learn about the experiences and concerns of teens with cancer in real time. Thus, we will consider inviting pediatric oncology health care providers to serve as additional moderators who can provide expert opinions in response to participant queries. This option will improve our study, promote better participant engagement, and enhance treatment fidelity.

    Strengths and Limitations

    This study had several strengths. We used rigorous qualitative analysis methods, including focus group discussions and user testing transcriptions. Additionally, the use of multiple methods provided a strong source of triangulation and enhanced the reliability of our results, which were informed by both qualitative and quantitative data. We described all development and usability evaluation processes conducted in 7 phases to provide guidance to researchers who want to use Facebook as an intervention platform. According to a recent systematic review of the use of Facebook, only 10 studies have investigated this platform for such interventions [51]. This could be partly because there is no guide on how to utilize this platform, and the task could seem daunting without guidance. Lastly, we used multiple usability evaluation methods. For example, we relied on expert reviews of our content by health care professionals, heuristics evaluation of the Facebook program by human-computer interaction experts, and user testing of the Facebook program by potential end users. Feedback comments from the different evaluation methods guided different aspects of usability issues and improved various facets of our program.

    Our study also had several limitations. Regarding participant recruitment, focus group discussions and user testing employed convenience and snowball sampling methods. Sample sizes were relatively small and participants were from one metropolitan area. We did not apply strict inclusion criteria for user testing participants, instead including participants with any type of cancer and at any stage of treatment (ranging from newly diagnosed to completed chemotherapy). Additionally, the current health status of participants was biased toward teens who were in good health. These limitations could decrease the generalizability of the study findings. In addition, our findings may reflect a response bias, as participants who were already interested in Facebook may have been more likely to participate.

    Acknowledgments

    This study was supported by a National Research Foundation of Korea (NRF) grant funded by the Korean government (MISP and MSIT; 2016R1C1B2013649 and 2020R1C1C1010602).

    Authors' Contributions

    BP was the lead investigator and oversaw implementation of the project. JK contributed to the content development and implementation of the study. VR contributed to the formulation of this manuscript. All authors reviewed and approved the manuscript for publication.

    Conflicts of Interest

    None declared.

    ‎
    Multimedia Appendix 1

    Quotes from focus group discussions and user testing sessions.

    PDF File (Adobe PDF File), 177 KB
    ‎
    Multimedia Appendix 2

    Screenshots from the Healthy Teens for Soaam program.

    PDF File (Adobe PDF File), 441 KB
    1. Cancer in Children and Adolescents. National Cancer Institute. 2018.   URL: https://www.cancer.gov/types/childhood-cancers/child-adolescent-cancers-fact-sheet#r2 [accessed 2020-03-05]
    2. National Cancer Institute. SEER Cancer Statistics Review (CSR) 1975-2016. 2020. 2020 Mar 05.   URL: https://seer.cancer.gov/csr/1975_2016/ [accessed 2020-03-05]
    3. Choi K. Vulnerabilities and Psychosocial Service Needs of Childhood Cancer Survivors and their Caregivers Based on the Cancer Trajectory. Health Soc Welf Rev 2018 Jun;38(2):417-451. [CrossRef]
    4. Long-Term Follow-Up Guidelines for Survivors of Childhood, Adolescent, and Young Adult Cancers. Children’s Oncology Group. 2018.   URL: http://www.survivorshipguidelines.org/pdf/2018/COG_LTFU_Guidelines_v5.pdf [accessed 2020-03-05]
    5. Bhakta N, Liu Q, Ness KK, Baassiri M, Eissa H, Yeo F, et al. The cumulative burden of surviving childhood cancer: an initial report from the St Jude Lifetime Cohort Study (SJLIFE). Lancet 2017 Dec;390(10112):2569-2582. [CrossRef]
    6. Chemaitilly W, Sklar C. Endocrine complications in long-term survivors of childhood cancers. Endocr Relat Cancer 2010 Sep;17(3):R141-R159. [CrossRef] [Medline]
    7. Phillips SM, Alfano CM, Perna FM, Glasgow RE. Accelerating translation of physical activity and cancer survivorship research into practice: recommendations for a more integrated and collaborative approach. Cancer Epidemiol Biomarkers Prev 2014 May;23(5):687-699 [FREE Full text] [CrossRef] [Medline]
    8. Printz C. Majority of survivors of childhood cancer experience comorbidities. Cancer 2015 Jul 15;121(14):2293-2293. [CrossRef] [Medline]
    9. Nathan P, Ford J, Henderson T, Hudson MM, Emmons KM, Casillas JN, et al. Health behaviors, medical care, and interventions to promote healthy living in the Childhood Cancer Survivor Study cohort. J Clin Oncol 2009 May 10;27(14):2363-2373 [FREE Full text] [CrossRef] [Medline]
    10. Tyc V, Hadley W, Crockett G. Prediction of health behaviors in pediatric cancer survivors. Med Pediatr Oncol 2001 Jul;37(1):42-46. [CrossRef] [Medline]
    11. Warner EL, Nam GE, Zhang Y, McFadden M, Wright J, Spraker-Perlman H, et al. Health behaviors, quality of life, and psychosocial health among survivors of adolescent and young adult cancers. J Cancer Surviv 2016 Apr;10(2):280-290 [FREE Full text] [CrossRef] [Medline]
    12. Coccia PF, Altman J, Bhatia S, Borinstein SC, Flynn J, George S, et al. Adolescent and young adult oncology. Clinical practice guidelines in oncology. J Natl Compr Canc Netw 2012 Sep;10(9):1112-1150. [CrossRef] [Medline]
    13. Klosky JL, Howell CR, Li Z, Foster RH, Mertens AC, Robison LL, et al. Risky health behavior among adolescents in the childhood cancer survivor study cohort. J Pediatr Psychol 2012 Jul 16;37(6):634-646 [FREE Full text] [CrossRef] [Medline]
    14. Brinkman TM, Li C, Vannatta K, Marchak JG, Lai J, Prasad PK, et al. Behavioral, Social, and Emotional Symptom Comorbidities and Profiles in Adolescent Survivors of Childhood Cancer: A Report From the Childhood Cancer Survivor Study. J Clin Oncol 2016 Oct 01;34(28):3417-3425 [FREE Full text] [CrossRef] [Medline]
    15. Gijsen R, Hoeymans N, Schellevis FG, Ruwaard D, Satariano WA, van den Bos GA. Causes and consequences of comorbidity: a review. J Clin Epidemiol 2001 Jul;54(7):661-674. [CrossRef] [Medline]
    16. Torres-Espíndola LM, Demetrio-Ríos J, Carmona-Aparicio L, Galván-Díaz C, Pérez-García M, Chávez-Pacheco JL, et al. Comorbidity Index as a Predictor of Mortality in Pediatric Patients With Solid Tumors. Front Pediatr 2019;7:48. [CrossRef] [Medline]
    17. Landier W, Ahern J, Barakat LP, Bhatia S, Bingen KM, Bondurant PG, et al. Patient/Family Education for Newly Diagnosed Pediatric Oncology Patients. J Pediatr Oncol Nurs 2016;33(6):422-431 [FREE Full text] [CrossRef] [Medline]
    18. Beaulieu-Gagnon S, Bélanger V, Meloche C, Curnier D, Sultan S, Laverdière C, et al. Nutrition education and cooking workshops for families of children with cancer: a feasibility study. BMC Nutr 2019;5:52 [FREE Full text] [CrossRef] [Medline]
    19. Nova F, Allenidekania A, Agustini N. The effect of multimedia-based nutrition education on parents' knowledge and body weight change in leukemia children. Enferm Clin 2019 Sep;29(Suppl 2):229-233. [CrossRef] [Medline]
    20. Ångström-Brännström C, Lindh V, Mullaney T, Nilsson K, Wickart-Johansson G, Svärd AM, et al. Parents' Experiences and Responses to an Intervention for Psychological Preparation of Children and Families During the Child's Radiotherapy. J Pediatr Oncol Nurs 2018;35(2):132-148. [CrossRef] [Medline]
    21. Anderson LM, Leonard S, Jonassaint J, Lunyera J, Bonner M, Shah N. Mobile health intervention for youth with sickle cell disease: Impact on adherence, disease knowledge, and quality of life. Pediatr Blood Cancer 2018 Aug;65(8):e27081. [CrossRef] [Medline]
    22. Howell CR, Krull KR, Partin RE, Kadan-Lottick NS, Robison LL, Hudson MM, et al. Randomized web-based physical activity intervention in adolescent survivors of childhood cancer. Pediatr Blood Cancer 2018 Aug;65(8):e27216 [FREE Full text] [CrossRef] [Medline]
    23. Merkle CJ, Chandler DE. Hyperosmolality inhibits exocytosis in sea urchin eggs by formation of a granule-free zone and arrest of pore widening. J Membr Biol 1989 Dec;112(3):223-232. [CrossRef] [Medline]
    24. Jo M, Hong S, Park HR. Effects of Art Intervention Program for Siblings of Children With Cancer: A Pilot Study. J Pediatr Oncol Nurs 2018 May;35(3):178-187. [CrossRef] [Medline]
    25. Na JH, Yoo KY, Kim JY, Park SK, Kim SK, Choi EJ. Educational Interventions to Enhance Adherence to Prophylactic Treatment in Korean Hemophilia Patients. Clin Pediatr Hematol Oncol 2018 Apr 30;25(1):38-42. [CrossRef]
    26. El Malla H, Ylitalo Helm N, Wilderäng U, El Sayed Elborai Y, Steineck G, Kreicbergs U. Adherence to medication: A nation-wide study from the Children's Cancer Hospital, Egypt. World J Psychiatry 2013 Jun 22;3(2):25-33 [FREE Full text] [CrossRef] [Medline]
    27. Ford JS, Chou JF, Sklar CA. Attendance at a survivorship clinic: impact on knowledge and psychosocial adjustment. J Cancer Surviv 2013 Dec 22;7(4):535-543. [CrossRef] [Medline]
    28. Devine K, Viola A, Capucilli P, Sahler O, Andolina J. Factors Associated With Noncompliance With Long-term Follow-up Care Among Pediatric Cancer Survivors. J Pediatr Hematol Oncol 2017 Apr;39(3):167-173 [FREE Full text] [CrossRef] [Medline]
    29. Watson J. Social Media Use in Cancer Care. Semin Oncol Nurs 2018 May;34(2):126-131. [CrossRef] [Medline]
    30. Perales M, Drake E, Pemmaraju N, Wood W. Social Media and the Adolescent and Young Adult (AYA) Patient with Cancer. Curr Hematol Malig Rep 2016 Dec;11(6):449-455 [FREE Full text] [CrossRef] [Medline]
    31. Valle CG, Tate DF, Mayer DK, Allicock M, Cai J. A randomized trial of a Facebook-based physical activity intervention for young adult cancer survivors. J Cancer Surviv 2013 Sep;7(3):355-368 [FREE Full text] [CrossRef] [Medline]
    32. Branch R. Instructional design: The ADDIE approach. Boston, MA: Springer; 2009.
    33. Krippendorff K. Content analysis: an introduction to its methodology. Thousand Oaks, CA: SAGE; 2014.
    34. Elo S, Kyngäs H. The qualitative content analysis process. J Adv Nurs 2008 Apr;62(1):107-115. [CrossRef] [Medline]
    35. O'Mahony M, Smyth B. Using readability tests to predict helpful product reviews. 2010 Apr 28 Presented at: he 9th international conference on Adaptivity, Personalization and Fusion of Heterogeneous Information; 2010; Paris, France p. 28-30.
    36. McClure GM. Readability formulas: Useful or useless? IEEE Trans Profess Commun 1987 Mar;PC-30(1):12-15. [CrossRef]
    37. Korean Childhood Leukemia Foundation.   URL: https://www.kclf.org/en/ [accessed 2020-03-05]
    38. Park BK, Nahm E, Rogers VE. Development of a Teen-Friendly Health Education Program on Facebook: Lessons Learned. J Pediatr Health Care 2016;30(3):197-207. [CrossRef] [Medline]
    39. Joyce A, Nielsen J. Teenager’s UX: designing for teens. NN/g Nielsen Norman Group. 2013.   URL: https://www.nngroup.com/articles/usability-of-websites-for-teenagers/ [accessed 2020-03-05]
    40. What's the difference between a public and private Facebook group and how do I change it? 2020. Facebook Help Center.   URL: https:/​/www.​facebook.com/​help/​286027304749263?helpref=search&sr=3&query=private&search_session_id=c03abc8ad049f3095413a657b8ef5c89 [accessed 2020-03-05]
    41. What are the privacy options for Facebook groups? Facebook Help Center.   URL: https://www.facebook.com/help/220336891328465?helpref=about_content [accessed 2020-03-05]
    42. Nielsen J. Heuristic engineering. Orlando, FL: AP Professioinal; 1994.
    43. Nielsen J. 10 Usability heuristics for user interface design. NN/g Nielsen Norman Group. 2005.   URL: https://www.nngroup.com/articles/ten-usability-heuristics/ [accessed 2020-03-05]
    44. Alsumait AA, Al-Osaimi A. Usability Heuristics Evaluation for Child E-learning Applications. J Softw 2010 Jun 01;5(6):654-661. [CrossRef]
    45. Norman CD, Skinner HA. eHEALS: The eHealth Literacy Scale. J Med Internet Res 2006 Nov 14;8(4):e27 [FREE Full text] [CrossRef] [Medline]
    46. Chung S, Park BK, Nahm E. The Korean eHealth Literacy Scale (K-eHEALS): Reliability and Validity Testing in Younger Adults Recruited Online. J Med Internet Res 2018 Apr 20;20(4):e138 [FREE Full text] [CrossRef] [Medline]
    47. Nahm E, Resnick B, Mills ME. Development and pilot-testing of the perceived health Web Site usability questionnaire (PHWSUQ) for older adults. Stud Health Technol Inform 2006;122:38-43. [Medline]
    48. Park BK. Development and usability evaluation of a Facebook-based program for pediatric cancer patients. 2019 Nov 18 Presented at: American Medical Informatics Association Annual Symposium; 2019; Washington, DC.
    49. Park BK, Nahm E, Rogers VE, Choi M, Friedmann E, Wilson M, et al. A Facebook-Based Obesity Prevention Program for Korean American Adolescents: Usability Evaluation. J Pediatr Health Care 2017;31(1):57-66. [CrossRef] [Medline]
    50. Valle C, Tate D. Engagement of young adult cancer survivors within a Facebook-based physical activity intervention. Transl Behav Med 2017 Dec;7(4):667-679 [FREE Full text] [CrossRef] [Medline]
    51. Whitaker C, Stevelink S, Fear N. The Use of Facebook in Recruiting Participants for Health Research Purposes: A Systematic Review. J Med Internet Res 2017 Aug 28;19(8):e290 [FREE Full text] [CrossRef] [Medline]

    ‎
    ADDIE: Analysis, design, development, implementation, and evaluation
    COG: Children’s Oncology Group
    eHEALS: eHealth Literacy Scale
    eHealth: electronic health
    eLearning: electronic learning
    K-eHEALS: Korean version of the eHealth Literacy Scale
    PHWSUQ: Perceived health website usability questionnaire

    Edited by G Eysenbach; submitted 25.03.20; peer-reviewed by A Roundtree, J Karst, J Jonassaint; comments to author 14.04.20; revised version received 22.05.20; accepted 11.06.20; published 28.07.20

    Copyright

    ©Bu Kyung Park, Ji Yoon Kim, Valerie E Rogers. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 28.07.2020.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.