This paper is in the following e-collection/theme issue:

eHealth Literacy / Digital Literacy (389) Kids' and Adolescents' Use of Technology (92) Co-Design & Participatory Design (56) Health Education for Children and Adolescents (41) Social Media in Adolescence (45) New Methods (943) Focus Groups and Qualitative Research for Human Factors Research (848)

Published on in Vol 26 (2024)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/49453, first published .
Developing an Educational Resource Aimed at Improving Adolescent Digital Health Literacy: Using Co-Design as Research Methodology

Developing an Educational Resource Aimed at Improving Adolescent Digital Health Literacy: Using Co-Design as Research Methodology

Developing an Educational Resource Aimed at Improving Adolescent Digital Health Literacy: Using Co-Design as Research Methodology

Authors of this article:

Callum C Lewis1 Author Orcid Image ;   Melody Taba1, 2 Author Orcid Image ;   Tiffany B Allen1 Author Orcid Image ;   Patrina HY Caldwell1, 3 Author Orcid Image ;   S Rachel Skinner1, 3 Author Orcid Image ;   Melissa Kang4 Author Orcid Image ;   Hamish Henderson5 Author Orcid Image ;   Liam Bray5 Author Orcid Image ;   Madeleine Borthwick5 Author Orcid Image ;   Philippa Collin6 Author Orcid Image ;   Kirsten McCaffery2 Author Orcid Image ;   Karen M Scott1 Author Orcid Image
Article Authors Cited by (2) Tweetations (1) Metrics

    Original Paper

    1Specialty of Child and Adolescent Health, Sydney Medical School, Faculty of Medicine and Health, The University of Sydney, Westmead, Australia

    2Sydney Health Literacy Lab, School of Public Health, Faculty of Medicine and Health, The University of Sydney, Sydney, Australia

    3The Children’s Hospital at Westmead, Westmead, Australia

    4General Practice Clinical School, Faculty of Medicine and Health, The University of Sydney, Sydney, Australia

    5Sydney School of Architecture, Design and Planning, The University of Sydney, Sydney, Australia

    6Institute for Culture and Society, Western Sydney University, Parramatta, Australia

    Corresponding Author:

    Karen M Scott, BEd, MA, PhD

    Specialty of Child and Adolescent Health, Sydney Medical School, Faculty of Medicine and Health

    The University of Sydney

    The Children's Hospital at Westmead, Clinical School

    Locked Bag 4001

    Westmead, 2145

    Australia

    Phone: 61 298453385

    Fax:61 298453389

    Email: karen.scott@sydney.edu.au


    Background: Adolescence is a key developmental period that affects lifelong health and is impacted by adolescents regularly engaging with digital health information. Adolescents need digital health literacy (DHL) to effectively evaluate the quality and credibility of such information, and to navigate an increasingly complex digital health environment. Few educational resources exist to improve DHL, and few have involved adolescents during design. The co-design approach may hold utility through developing interventions with participants as design partners.

    Objective: This project aimed to explore the co-design approach in developing an educational resource to improve adolescents’ DHL.

    Methods: Adolescents (12-17 years old) attended 4 interactive co-design workshops (June 2021-April 2022). Participant perspectives were gathered on DHL and the design of educational resources to improve it. Data generated were analyzed through content analysis to inform educational resource development.

    Results: In total, 27 participants from diverse backgrounds attended the workshops. Insight was gained into participants’ relationship with digital health information, including acceptance of its benefits and relevance, coupled with awareness of misinformation issues, revealing areas of DHL need. Participants provided suggestions for educational resource development that incorporated the most useful aspects of digital formats to develop skills across these domains. The following 4 themes were derived from participant perspectives: ease of access to digital health information, personal and social factors that impacted use, impacts of the plethora of digital information, and anonymity offered by digital sources. Initial participant evaluation of the developed educational resource was largely positive, including useful suggestions for improvement.

    Conclusions: Co-design elicited and translated authentic adolescent perspectives and design ideas into a functional educational resource. Insight into adolescents’ DHL needs generated targeted educational resource content, with engaging formats, designs, and storylines. Co-design holds promise as an important and empowering tool for developing interventions to improve adolescents’ DHL.

    J Med Internet Res 2024;26:e49453

    doi:10.2196/49453

    Keywords

    Adolescent health; digital health literacy; adolescents; online health information; co-design; health education; eHealth literacy; social media 


    Adolescence is a foundational developmental period, influencing lifelong well-being through the establishment of lifestyle and behavioral choices [1]. Adolescents may be exposed to many changes and potential risks, including first encounters with sexual behaviors, substance use, and mental illness, and are, hence, a priority group for health promotion [2]. The digital age has transformed adolescence, representing a new landscape for lived experience and challenges in navigating health concerns [3]. Adolescents frequently seek digital health information [4,5], with a vast majority engaging with it on websites and social media [6], often using mobile devices [7].

    Digital formats provide a private space for health information seeking, which is particularly beneficial for sensitive health domains, including sexual health [8]. The quantity of digital health information available [9] alleviates information poverty, which was previously encountered by adolescents wanting to access targeted health information [10]. The currency of information [9], alongside the ability to share personal perspectives [11], further promotes uptake.

    Despite the frequent use of digital health information by adolescents, awareness of its highly variable quality has engendered a lack of trust [12]. Heightened risks to adolescent health exist due to insufficient regulation of digital health information [13], particularly given poor quality information is frequently disseminated, often driven by commercial interests [14]. This has been accentuated in the current era where misinformation, which adolescents often struggle to identify [15], is frequently published on the internet, including social media, as exemplified by the COVID-19 “infodemic” [16].

    Digital health information can impact adolescents’ engagement with health care providers, including decisions about how and when to attend health facilities [17,18], or decisions to self-manage care [19]. Risks due to poor quality information and delayed health care presentations can potentially cause harm [20]. Conversely, opportunity exists to support health engagement if information is accurate and appropriate [9,21]. Given the importance of digital health information, adolescents’ ability to assess its quality needs to be optimized [22].

    Digital health literacy (DHL), “the ability to find, evaluate, appraise, integrate, and apply health information from online environments” [23] denotes the multifaceted skills required to assess credibility of digital health information [24]. Adolescents often lack these important digital health literacy skills [25]. In particular, adolescents frequently use ineffective heuristics [26], rather than objective criteria [27], to appraise digital health information, often prioritizing relevance and accessibility over credibility [28,29].

    Social cognitive theory (SCT) offers a useful lens through which to analyze DHL. It describes how learning and behavior is influenced by a dynamic interrelationship between a person, their behavior, and their surrounding multifactorial context [30]. Self-efficacy, the notion of individuals trusting their own ability, is a core SCT concept that is understood to be key for effective learning [31], including for building and sustaining DHL skills [32]. However, evidence suggests adolescents frequently overestimate their abilities, perpetuating the impacts of misinformation encountered [27-29]. Education to improve DHL is, therefore, vital, given that higher DHL is associated with positive health behaviors [33], health care engagement, and overall well-being [34,35].

    While adolescents’ DHL deficits are increasingly acknowledged, there are few resources that target its improvement, and the literature on resource development is limited. A 2012 trial of a digital classroom resource, which featured instructional DHL content around nutrition and exercise, recommended that intervention designers work more actively with adolescents to ensure relevance, engagement, and efficacy [36]. Self-directed educational resources that are accessible on demand have been deemed beneficial [37], as have digitized formats, due to the engagement and accessibility afforded by these platforms [38].

    Co-design may offer an effective means to produce such educational resources [29]. Avoiding traditional “top-down” [39] approaches, or mere stakeholder consultations, interventions are instead built with the target population [39] as design partners throughout all phases [40]. As a novel method to close “the translational research gap” [41], co-design may improve the relevance and impact of educational resources by ensuring they are consumer driven and consumer centered [42]. Executed using various formats, past trials have used interactive workshops to involve adolescents [43] and articulate research questions into functional products [39], including most recently for mobile app–based mental health interventions [44,45]. Activities can generate creative solutions, through physical “artefacts,” to inform educational resource generation. Evaluation workshops between participants and designers can then ascertain whether the solution effectively responds to participant needs and perspectives, closing the co-design cycle (Figure 1) [42].

    Having grown up in the digital age, most adolescents are proficient with common digital technologies, such as the internet and social media, though such proficiency may not be transferred to all forms of technology [26]. Nevertheless, most adolescents are able to effectively express their perspectives on DHL issues and corresponding educational needs [26]. As such, co-design with adolescents holds promise for development of web-based educational resources, despite limited existing research with this participant group [39]. Our study aimed to explore the co-design approach, based on Hagen et al [42] (Figure 1) to develop an adolescent DHL educational resource through a series of interactive workshops. Insight was sought into the influence of co-design on educational resource generation by understanding adolescents’ experience with digital health information and DHL needs, and their design preferences.

    ‎
    Figure 1. Framework utilized for the co-design of a digital health literacy education resource (after Hagen et al) [42]. DHL: digital health literacy.

    Overview

    This study involved a qualitative co-design methodology, whereby 4 interactive workshops were held with adolescent participants from June 2021 to April 2022. Data generated were analyzed through inductive content analysis [46] to inform the development of an educational resource with the aim of improving users’ DHL (Figure 2).

    ‎
    Figure 2. Research design flowchart reflecting the multistage co-design development process of the educational resource. DHL: digital health literacy.

    Ethical Considerations

    This research project was approved by Sydney Children’s Hospitals Network Human Research Ethics Committee (2020/ETH00578).

    The participants, who were minors, gave their informed assent, and their parent or guardian gave their informed consent, to participate in this research study. To protect participant privacy, all study data were deidentified prior to analysis.

    Participants were each provided with an AUS $50 gift card (approximately US $35 at the time of the study) for use at a range of stores as compensation for their time.

    Context

    To understand adolescent participants' experiences with digital health information, perspectives on DHL issues, and to ascertain their educational needs, 3 co-design workshops were organized. The workshops also sought to generate design solutions for an educational resource that could develop user skills in appraising digital health information. Interactive activities during workshops generated physical artefacts, such as design maps and sketches, which were collected for data analysis (Table 1) [43].

    These activities were designed in line with SCT to foster participant collaboration in a safe, peer-based environment [30] in which interactive activities enable learning through the dynamic interrelationship of personal attitudes, behavior, and the environment [30]. The activities align with prior literature calling for the development of DHL interventions through co-design, which encompasses use of practical activities to explore and improve adolescents’ self-efficacy and behavior related to DHL [29]. A design meeting was held among the researchers and the software design team developing the resource to translate workshop findings into educational resource designs. An evaluation workshop with adolescent participants was then conducted to obtain participant feedback on a pilot educational resource. The design meeting and evaluation workshop align with the final phases of the co-design framework, which seek to develop, evaluate, and refine interventions with adolescents (Figure 1) [42].

    Co-design facilitators were multidisciplinary academics and research students: a medical educator with experience in eLearning and literacy (KMS), a pediatric clinical academic and child and adolescent health researcher (PHYC), a social scientist focusing on the digital and young people (PC), public health and medical student researchers (MT, CCL, and TBA) and design computing academics and software designers (LB, HH, and MB).

    Table 1. Methods employed during co-design events, including activities used, artefacts generated and alignment with social cognitive theory.
    Co-design event details and purpose and activitiesArtefacts generated
    Workshop 1 (June 2021); face-to-face: identified adolescents’ perspectives on their use of digital health information, both web-based and on social media, as well as the concept of digital health literacy

    • Introduction: adolescent participants annotated a worksheet asking them about the digital health information they use, where they use such resources and reasons for use
    • Social Cognitive Theory (SCT) alignment: Interaction between personal, behavioral, and environmental factors
    		Annotated worksheets

    • Enablers and barriers: participants collaboratively identified enablers and barriers of digital health information use, wrote them on sticky notes and placed these on a wall. These were then categorized and collated by idea type
    • SCT alignment: Influence of environment on behavior and personal factors
    		Affinity diagram

    • Credibility rankings: on A4 posters, participants ranked (in order of credibility) 5 digital health information websites
    • SCT alignment: Influence of personal factors on behavior and environment
    		Annotated or ranked examples

    • Assess and rank: participants assessed the 5 websites using a set of fixed criteria (looks professional, is understandable, I can cross check information) and criteria they generated, and ranked them based on this
    • SCT alignment: Influence of personal factors on behavior and environment
    		Annotated table
    Workshop 2 (November 2021); face-to-face: identified adolescents’ perspectives on their use of digital health information, both web based and on social media, as well as the concept of DHL

    • Misinformation personal stories: participants completed a worksheet with personal examples of people believing misinformation and what made it believable
    • SCT alignment: Influence of environment on behavioral and personal factors
    		Annotated worksheets

    • Social spaces: participants added color-coded stickers to posters of social media platforms they use regularly, use for fun and use to find information. A set of posters of common platforms (eg, Facebook, TikTok) were preprinted, with participants adding a new poster for any additional platforms they use that were not present
    • SCT alignment: Influence of behavior on personal factors and environment
    		Sticker-annotated poster

    • Field notes: researchers noted key points of discussion during all exercises
    		Digital written notes
    Workshop 3(November 2021); face-to-face:ascertain perspectives on what an educational resource seeking to improve adolescents’ DHL should include and how it should work to be effective

    • Brainstorming design ideas: participants identified educational design features and content they believed the design solution should have, wrote them on sticky notes, then added them to a wall of annotated sticky notes. These were then categorized and collated by idea type
    • SCT alignment: Interaction between personal, behavioral, and environmental factors
    		Affinity diagram

    • Resource design maps: in groups, participants identified the features about form and content that they felt a DHL educational resource should include, then sketched these on posters and labeled them
    • SCT alignment: Interaction between personal, behavioral, and environmental factors
    		Annotated posters

    • Resource design sketches: participants sketched layout, appearance, format, content, and storylines that they felt would make an effective DHL educational resource
    • SCT alignment: Interaction between personal, behavioral, and environmental factors
    		Drawings

    • Field notes: researchers noted key points of discussion during all exercises
    		Digital written notes
    Design meeting (December 2021); face-to-face:translate workshop findings into practical design plans through synthesis with the researchers’ educational requirements of the educational resource

    • Priority learning points: learning points from the workshops were written on palm cards. Participants (researchers and designers) added a sticker to those they felt were the most important to include in the educational resource. The cards with the most stickers were then prioritized as key learning points
    • SCT alignment: Interaction between personal, behavioral, and environmental factors
    		Sticker-annotated cards

    • Storyline flowcharts: in groups, participants collaboratively created storylines for the educational resource that they felt would be effective in communicating learning points. These were then drawn as flowcharts on a poster
    • SCT alignment: Interaction between personal, behavioral, and environmental factors
    		Flowcharts

    • Field notes: researchers noted key points of discussion during all exercises
    		Digital written notes
    Pilot Evaluation Workshop (April 2022); virtual: gather participants’ feedback on a trial version of the educational resource, which the design team will use to undertake fine-tuning

    • Resource trial and field notes: participants were guided through the educational resource app, “mis-Adventures,” over a video meeting, enabling them to view and engage with its interactive components. The following question prompts were used to capture perspectives:
      • What did you think about mis-Adventures?
        • story, videos
        • decisions or choices made along the way
        • quizzes, explanations
        • expert advice at the end
        • design or look and feel
      • What did you like best about mis-Adventures?
      • What did you like least about mis-Adventures?
      • How can we improve mis-Adventures?
      • Please tell us about any problems you had using mis-Adventures.
    • Researchers noted down key responses and points of discussion during the exercise
    • SCT alignment: Interaction between personal, behavioral, and environmental factors
    		Digital written notes

    Participants

    Participants were adolescents aged 12-17 years, representing the high school aged population the educational resource was targeted toward. Recruitment involved advertising on flyers, social media, and through word-of-mouth dissemination. The purposive recruitment strategy sought to capture a heterogenous sample despite a small overall size: we aimed to recruit participants who were of differing ages, genders and cultural backgrounds to enhance the diversity of the perspectives obtained [47], and the broad appeal and relevance of the educational resource. For each workshop, recruitment continued until a quota was met, which was set to facilitate small group sizes to enable interactive and collaborative activities. Participants were free to attend as many workshops as they wished: they were reinvited to any subsequent workshops to increase the continuity of the co-design participation cycle [42], while new participants were also invited to ensure diversity of perspectives.

    Data Collection

    Artefacts collected from the workshops (Table 1) were physical documents, featuring writing and drawings (examples in Figure 3). Artefacts were collected at the completion of each workshop, and subsequently scanned for digitization and analysis.

    ‎
    Figure 3. Examples of artefacts obtained from workshops: (A) educational resource design sketches and (B) affinity diagram.

    Data Analysis

    Data analysis involved content analysis of the artefacts generated using NVivo (Release 1.6, QSR International) data management software [48]. This included review and familiarization with the data before iterative coding of the data features in scanned artefacts by CCL using the NVivo pictorial coding tool. An inductive approach was used [46], whereby analysis of the data set informed the codes generated, which comprised recurrent features and ideas in the artefacts. This resulted in the iterative generation of themes, ensuring their relevance to the data [49]. This was completed through an initial categorization of recurrent codes by CCL, with researcher triangulation through analysis meetings with KMS and MT in the first instance, later joined by PHYC, which offered multiple perspectives on coding decisions, enhancing credibility [50]. Through discussion and researcher consensus, similar codes were grouped into concepts and broader categories, which became the themes and subthemes in the data, prior to in-depth analysis and interpretation. Credibility was also developed through obtaining data from multiple workshops with a range of participants.

    Results are displayed through a conceptual mind map, generated through diagrams.net (JGraph Ltd) software [51].


    In total, 27 adolescents participated in the face-to-face co-design workshops: 7 at workshop 1 and 15 at workshops 2 and 3 (which were held on one day [morning then afternoon] with the same participants), and 5 at workshop 4, the pilot evaluation, which was conducted via web-based platforms because of COVID-19 social distancing restrictions. Participants were of diverse backgrounds, with a mix of genders, ages (12-16 years), and ethnicities (Anglo-Australian, Chinese, Egyptian, Italian, Latin American, Lebanese, and Taiwanese). Seven participants attended multiple workshop days, including 1 participant who attended all 4 workshops.

    Participants’ relationship with digital health information was one of duality—indicating an acceptance of its benefits and relevance in their lives, while having an awareness of its dangers in perpetuating misinformation. This revealed areas of DHL educational need, particularly surrounding appraisal of digital health information quality (Figure 4). Participant-derived educational solutions targeting these issues (Figure 5) highlighted the need for an accessible, self-directed digital resource aligning with participants’ priorities for educational flexibility and autonomy. The educational resource designs that were generated incorporated the most useful aspects of the digital domain to enhance adolescents’ engagement, alongside content that targeted credibility issues at the core of DHL (Figure 6). Through the analysis of these issues and design solutions, we identified 4 key themes: ease of access to digital health information, users’ personal and social factors impacting use of DHL, impacts of the plethora of digital health information available, and anonymity offered by digital sources. These themes are detailed below, each with an analysis of participants’ perspectives and needs, and corresponding design solutions, followed by participant feedback on the pilot educational resource that was created.

    ‎
    Figure 4. Digital Health Literacy Issues: themes and subthemes representing adolescents’ experiences and issues with digital health literacy.
    ‎
    Figure 5. Digital Health Literacy (DHL) Resource Design Solutions: themes and subthemes representing design solutions generated in response to identified DHL needs. SCHN: Sydney Children’s Hospitals Network.
    ‎
    Figure 6. Example screenshots from the digital health literacy resource generated through co-design, featuring health professional explainer videos, realistic digital health information use scenarios, and simulated engagement with digital health information across multiple platforms. The individuals featured in these images have provided their written consent for their image to be published.

    Ease of Access to Digital Health Information

    User Perspective and Needs

    Participants perceived ease of access to digital technology as underpinning their frequent use of digital health information, which satisfied their need for succinct, understandable, and engaging information. They preferred visual, and particularly video, formats due to their engaging nature, with younger adolescents especially accessing video platforms such as YouTube and TikTok as their first engagement with a social media network. Participants appreciated animated and image-based content for easily communicating concepts. Some perceived that video features increased credibility, even in cases of misinformation, and the aesthetic appearance of platforms was important in determining quality. Participants identified quizzes as an engaging means of understanding and retaining information. Most participants identified mobile devices as a key driver of use, allowing engagement with information across many times and locations, with search engines offering easy access to perceived credible information. Participants indicated that they appraised search results using simple heuristics, including use of website title format and presence of keywords such as “health” in URLs to predict engagement and perception of credibility. They believed the digital domain offered easier engagement with credible sources. This included “verified” profiles (indicated by a blue tick) on social media platforms, which some believed meant content was fact-checked, when at the time of the study it instead signified that the profile belonged to an authentic identity.

    Co-Designed Educational Solutions

    In line with their desires for engaging, highly interactive and concise content, participants envisaged an educational platform of “choose-your-own-adventure-style” stories, featuring branching storylines and plot decision points. This was implemented with content comprised of “TikTok-style” videos, which adopted the form’s archetypal features, such as short duration (<3 min), fast pace and satirical tone, adding to the educational resource’s cultural appeal. Videos also featured topic experts explaining DHL concepts and strategies for assessing credibility. As participants desired summaries to consolidate learning, this was enacted using a final video in each storyline, encompassing the educational resource’s cardinal learning points. Catering to participants’ device usage, the user interface has a minimalist “mobile-first” design, featuring a scrollable content carousel and smooth page transitions. Shortcut tabs and drop-down menus linking to specific content afford learning flexibility, with an emphasis on visual features to improve accessibility. Quizzes offer points of revision following videos and include identification of features reflecting credibility on emulated health websites. Automated answers with feedback are provided to enhance their educational benefit. Content includes explanations about the differences between credible and noncredible information, and how to screen search results for credibility.

    Personal and Social Factors Impacting Digital Health Information Use

    User Perspective and Needs

    Participants identified a range of personal and social aspects of adolescent users that affect how they engage with digital health information, including psychological, situational, and peer circumstances. The web and social media–based offering of interactive, personalized information was highly appealing, such as through customization of profiles and content. This included recommended content in newsfeeds, and participants relating to others’ experiences through content sharing and facilities such as forums, messaging features, and comments. This was despite participants often finding such content noncredible, identifying social media “influencers” as a pertinent example of how unqualified individuals can propagate misinformation. Platforms that cater to specific interests and cultural and linguistic groups, such as WeChat, which caters for a Chinese population, were popular, along with a preference for educational resources that offer accessibility for the visually and hearing impaired. Participants indicated that family and peers on digital platforms can increase engagement with health information, but also share misinformation. Some found situational factors, such as restrictions to access or content, as a barrier to digital health information use. Conversely, the availability of web and social media–based information meant many only visited health professionals when necessitated by serious situations. Participants indicated that a lack of knowledge and familiarity with specific health topics reduced their engagement and ability to detect incorrect content. For example, they were more likely to trust hyperbolized and inaccurate health content in difficult situations, such as the COVID-19 pandemic. However, participants found that having existing knowledge of the scientific consensus on an issue can aid identification of misinformation.

    Co-Designed Educational Solutions

    The educational resource design was based on storylines featuring a themed journey layout (eg, health preparations before a music festival) to frame users’ progress through it. A live action video format was adopted as a low-resource alternative for participants’ suggestion of an animated avatar with customizable features representing the user. A plot device involved encountering recommended health content on search engine results that leads the character to investigate further, reflecting participants’ experience of often finding health information coincidentally, especially on social media. Captions were developed for video sections to assist users with hearing impairment.

    Three storylines, derived from participants, were developed and included in the educational resource. These include a storyline about finding ways to ease gastrointestinal symptoms, based on participant feedback that stories about sensitive issues are compelling. A plotline about deciding whether to get vaccinated against influenza (as an alternative infectious diseases scenario given participants’ interest in COVID-19) was also featured due to participants finding contextually relevant stories engaging. A third storyline involved searching for ways to ease exercise injuries, which was included due to sports injuries being a common target for health products, and its appeal to male demographics.

    Educational design solutions targeted difficulties with appraising information about these technical topics through simple explanations about identifying a source’s purpose, message, language, author, date, and provider or funding. A disclaimer—to always seek advice from qualified health professionals—was included in the homepage and throughout the resource due to the potentially significant personal impacts of using digital health information.

    Impacts of the Plethora of Digital Information Available

    User Perspective and Needs

    Participants indicated that the variety and magnitude of health information available on the web and social media impacted engagement. This abundance overwhelmed some or resulted in discouragement if the information was hyperbolic or inflammatory. While accessing information through multiple sources enabled cross-checking, which was believed to be universally important in assessing credibility, it sometimes led to participants falsely verifying sources due to the republishing of misinformation. This included mainstream media reporting on misinformation. Having reputable sources directly discredit such stories was regarded as effective by participants. Participants perceived the presence of author qualifications increased trust in a source, while the currency of information was important for its utility. Transparent referencing and links to further credible information were appreciated.

    Co-Designed Educational Solutions

    Content in the educational resource was based on research into areas of DHL educational need undertaken by the research team, aligning with participant perspectives that authorship by health professionals and educators with listed qualifications and associations with health authorities builds trust. This was coupled with participants’ contributions at co-design workshops, which included their learning priorities and identified areas of DHL deficit, to generate relevant learning content.

    Educational design ideas focused on content explaining how to systematically assess source credibility. This included ascertaining whether supporting scientific evidence and multiple perspectives or treatment options are included, the importance of a recent publication date, and how to effectively cross-check information with credible sources. Information about indicators of credible content, such as relevant author health qualifications, was also included. Guided by participants’ suggestions to build trust, the educational resource was developed by experts with listed qualifications and association with well-known health and educational authorities.

    Anonymity Offered by Digital Sources

    User Perspective and Needs

    Participants perceived anonymity when engaging with digital health information to be a core benefit of the digital domain because health topics they viewed were often seen as sensitive or embarrassing. They preferred to use mobile devices because they offered physical privacy from those around them, including peers and family. Participants considered the commercial motivations of web and social media–based publishers as detrimental to credibility, disliking targeted advertisements based on search history as they breached privacy.

    Co-Designed Educational Solutions

    Accordingly, an advertisement-free platform to build trust was implemented. Content about how to identify digital marketing was integrated due to participant interest, including the algorithms and mechanisms behind targeted advertising, and how it exploits users’ confirmation bias to accept false ideas. Content was also included about how integrated health advertising on health information sites can reduce credibility due to conflicts of interest. Content was generated identifying a need to recognize source funding and its impact on information quality, including “influencers” and the commercial partnerships that may underlie their personal stories.

    Educational Resource Feedback

    During the pilot evaluation, many participants recognized their designs and stories in the educational resource, which they appreciated, and indicated it made the resource age-appropriate. They found the user interface to be sleek, visually appealing and easy-to-navigate, responding to their desire for a mobile-first platform. Participants also appreciated the fast pace of the video storylines, finding them concise and engaging, enjoying the “choose-your-own-adventure-style” format. For participants, this interactivity was enhanced by the ability to re-explore storylines and alternate subplots by making different decisions. The narrative and character-driven format were enjoyed, especially through the inclusion of fictitious but believable web and social media–based health personalities.

    Some users would have liked the educational resource to be compatible with desktop formats, along with the inclusion of accessibility features, such as catering for sensory impairment and featuring different languages. Despite finding the educational resource age appropriate, some participants indicated the quizzes were too easy, suggesting additional levels of complexity for advanced users. This mirrored their proposals for more nuanced storylines that could explore further content with a less predictable plot. Additionally, participants desired more dramatic and engaging videos to convey the plot, including more apparent consequences for making poor decisions based on health information. Some suggested that content summary videos, which feature at the end of each storyline, could be personalized to the user’s in-game decision-making to enhance relevance.


    Principal Findings

    The stepwise approach of this study, from initial exploratory workshops to educational design generation, implementation, and evaluation, has enabled the application of the co-design model to the DHL context, yielding a functional and youth-centered product. Co-design elicited authentic adolescent perspectives, encapsulated by the 4 themes regarding digital health information use: ease of access to digital health information, users’ personal and social factors, the plethora of information, and anonymity offered by digital sources. These perspectives encompassed participants’ DHL needs and were addressed through the generation of teaching areas in the educational resource. Targeted educational solutions regarding the most effective modalities and designs were also generated and implemented through integration of participant, researcher, and designer perspectives. The approach enabled the generation of storylines, examples of credible and noncredible sources, and content tone that are highly relevant to an adolescent audience, enhancing potential engagement with and impact of the educational resource.

    The results reflect the benefit of co-design in capturing adolescents’ perspectives on their DHL needs, aligning with research into adolescents’ proficiency in this area, despite limited prior engagement in intervention development [26]. Co-design transformed adolescents’ viewpoints into functional educational resource content. In keeping with the importance of social context in Social Cognitive Theory [30] and to influence DHL learning and self-efficacy [32], adolescents’ perspectives were successfully elicited through interactive workshop activities that prompted collaborative social engagement. Ease of access to digital health information, a well-discussed driver of adolescent use [52], underpinned the educational resource’s teaching points, which included identifying noncredible features of information quickly and effectively. Through these teaching points, the educational resource seeks to improve DHL skillsets by targeting the popular but often ineffective heuristics used to assess credibility, such as website names or professional appearance [26,28]. Identification of the personal and social factors affecting digital health information use, and the impact of the diverse information available, generated content that teaches objective appraisal skills that can be universally used [27]. Storylines about sensitive health issues, such as gastrointestinal symptoms, were implemented as they reflect participants’ preference for the anonymity offered by digital platforms, aligning with prior findings [8].

    The co-design approach also generated effective modalities and designs for the educational resource for engagement with the adolescent target audience. Ease of access underpinned the DHL needs identified and educational solutions adopted, with strong preference for video as a digital health information format [25]. The educational resource design used these findings as a framework, with short “TikTok-style” videos acting as core content within a minimalist mobile-first design, responding to adolescents’ preferred engagement with mobile platforms [7]. Quizzes were incorporated in response to participants suggesting “gamification” [53] features to enhance engagement. In line with Social Cognitive Theory, they may also enhance learning by providing positive feedback to increase user self-efficacy [31]. Similarly, although not implemented, log-in and favorite features, and a themed journey with a customizable avatar, was recurrently suggested by participants, exemplifying how personalization and targeted information brings salience to resources [10].

    Co-design enabled the generation of specific storylines and content tone that are highly relevant to the adolescent audience, enhancing potential engagement and impact of the educational resource. The interactive video “choose-your-own- adventure-style” format was key to enabling ease of use, as was the integration of authentic cultural references, such as “viral” social media trends, which were found to be important social factors underpinning engagement. This is mirrored by Paek and Hove’s [36] prior calls for more effective integration of adolescents’ perspectives to enhance the personal relevance and efficacy of educational interventions. Incorporated plot devices, such as a teenager coincidentally encountering a health topic on a social media feed, which leads them to undertake a search, were found by participants to be very common occurrences, paralleling their frequent use of social media to engage with health information [6]. Similarly, subplots regarding “influencers” and sponsored posts were identified and included in the educational resource to add relevance, given their prominence as a DHL issue [14,52]. The capture and implementation of these nuanced references builds upon the use of co-design in different social and cultural contexts, reflecting its versatility [41].

    The design meeting, which featured both the research and design teams, was an important aspect of this study’s structure, translating workshop findings into formulated content and development plans through collating and filtering participant ideas. For example, in line with recent literature, COVID-19 was a key suggested topic that exemplified DHL needs [15,23] and was featured in multiple suggested storylines to enhance realism. However, the research and resource development teams implemented an alternative influenza vaccine setting as they believed relying on the current COVID-19 context may reduce educational resource longevity. Although this illustrates a disparity between participant and academic ideology [43], the multistage nature of co-design [39] in this instance generated many other implementable features that embody adolescents’ ideas. The use of interactive collaborative activities to structure the design meeting were in line with the Social Cognitive Theoretical framework and co-design approach [43]. The meeting aligned with prior workshop findings that adolescents perceived reputable sources to be effective at debunking misinformation and wanted expert consensus to shape the educational resource. While this stage of the methods may be incongruous with purist co-design theory, which prioritizes participant involvement during every development phase [42], the impact of COVID-19 lockdowns and project time and resource limitations prevented scheduling of an additional workshop to engage with adolescents at this content development stage.

    The feedback obtained through the pilot evaluation workshop was useful in offering an initial indication as to whether the educational resource had adequately responded to the participants’ needs and perspectives. The workshop was largely positive and constructive comments about improvements to the design, plot, and content of the resource led to fine-tuning before finalization. However, time and budget limitations resulted in some of these suggestions being noted but not implemented, presenting areas for further development in subsequent projects. Participants expressed enjoyment in being included in the co-design process, with the final feedback stage important in closing the cycle of participant engagement [42].

    Implications and Recommendations

    This study’s co-design approach demonstrates significant translational benefit through the articulation of adolescents’ DHL perspectives and needs (many of which are in line with previous findings [9,26,29,54]) into implemented educational resource solutions [41]. The qualitative methodology held great utility in authentically exploring participants’ ideas in a comfortable, peer-based environment. Mixed methods evaluation of the educational resource is needed to determine the effectiveness of the co-design approach [55].

    Limitations

    Although small, the participant sample was heterogeneous. Nevertheless, the sample contained some similar demographics; in particular, all participants were from metropolitan areas as planned recruitment in rural and remote regions was canceled because of social restrictions due to the COVID-19 pandemic [56]. Time limitations due to delays in participant recruitment during COVID-19 lockdowns also prevented participant involvement in storyline plot generation. However, as noted by participants in the pilot evaluation workshop, participants’ design ideas were largely adopted and transformed into a functional product. Time and financial constraints limited the educational resource from featuring avatars, animations, and multiple language options, despite participant suggestions. Proposed social networking features, including sharing, forums, and messaging features, were also excluded as their ongoing moderation was beyond logistical constraints. These may reveal potential directions for future development.

    Conclusions

    This study offers novel insights into the implementation and impact of the co-design approach with adolescents to generate DHL educational resources, responding to prior calls for investigation in this area. The workshop format was useful in eliciting authentic adolescent perspectives and enabling their translation to underpin an engaging and functional educational resource that responds to adolescents’ needs. Given these positive initial findings, co-design holds promise as an important tool for developing interventions that could improve DHL in adolescents.

    Acknowledgments

    Sincere thanks to Ms Charlotte Frazer, Ms Mia Rowe and Mr Toby Rowe for their time and acting contributions for the mis-Adventures app. Funding for this study was provided by the Australian Government, through the 2020 Australian Research Council Discovery Project “Enhancing adolescents’ health through digital health literacy.”

    Authors' Contributions

    Conceptualization of this study was by KMS and PHYC, who also received input from SRS, MK and KMC. Funding acquisition was undertaken by KMS in conjunction with PHYC, SRS, MK, and KMC. Project administration was undertaken by MT and KMS, with CCL also supporting. KMS, with PHYC, played key supervision roles. Methodology was developed by KMS and PHYC, with input from SRS, MK, KMC, MT, HH, and PC. MT oversaw resources in this study, with HH, LB, MB, KMS, and CCL also assisting. Investigation was completed by HH, LB, MB, PC, MT, KMS, CCL, PHYC, and TBA. Data curation was completed by CCL, with assistance from MT and KMS. Formal analysis was undertaken by CCL, in collaboration with KMS, MT, and PHYC, who were essential in data validation. The development of mis-Adventures was undertaken by HH, LB, and MB. Visualization of data (Figures 4 and 5) was completed by CCL, with input from KMS, PHYC, and MT. This manuscript was drafted by CCL, in collaboration with KMS, and with critical review and editing from KMS, PHYC, MT, and CCL.

    The authors attest that generative artificial intelligence has not been used in any portion of this manuscript.

    Conflicts of Interest

    None declared.

    1. Sawyer SM, Azzopardi PS, Wickremarathne D, Patton GC. The age of adolescence. Lancet Child Adolesc Health. 2018;2(3):223-228. [CrossRef] [Medline]
    2. Brindis CD, Hair EC, Cochran S, Cleveland K, Valderrama LT, Park MJ. Increasing access to program information: a strategy for improving adolescent health. Matern Child Health J. 2007;11(1):27-35. [CrossRef] [Medline]
    3. Ruan S, Raeside R, Singleton A, Redfern J, Partridge SR. Limited engaging and interactive online health information for adolescents: a systematic review of Australian websites. Health Commun. 2021;36(6):764-773. [CrossRef] [Medline]
    4. Liira H, Saarelma O, Callaghan M, Harbour R, Jousimaa J, Kunnamo I, et al. Patients, health information, and guidelines: A focus-group study. Scand J Prim Health Care. 2015;33(3):212-219. [FREE Full text] [CrossRef] [Medline]
    5. Ettel G, Nathanson I, Ettel D, Wilson C, Meola P. How do adolescents access health information? And do they ask their physicians? Perm J. 2012;16(1):35-38. [FREE Full text] [CrossRef] [Medline]
    6. Armstrong M, Halim NK, Raeside R, Jia SS, Hyun K, Boroumand F, et al. How helpful and what is the quality of digital sources of healthy lifestyle information used by Australian adolescents? a mixed methods study. Int J Environ Res Public Health. 2021;18(23):12844. [FREE Full text] [CrossRef] [Medline]
    7. Devine KA, Viola AS, Coups EJ, Wu YP. Digital health interventions for adolescent and young adult cancer survivors. JCO Clin Cancer Inform. 2018;2:1-15. [FREE Full text] [CrossRef] [Medline]
    8. Waldman L, Ahmed T, Scott N, Akter S, Standing H, Rasheed S. 'We have the internet in our hands': Bangladeshi college students' use of ICTs for health information. Global Health. 2018;14(1):31. [FREE Full text] [CrossRef] [Medline]
    9. Freeman JL, Caldwell PH, Scott KM. The role of trust when adolescents search for and appraise online health information. J Pediatr. 2020;221:215-223.e5. [CrossRef] [Medline]
    10. Gray NJ, Klein JD, Noyce PR, Sesselberg TS, Cantrill JA. Health information-seeking behaviour in adolescence: the place of the internet. Soc Sci Med. 2005;60(7):1467-1478. [CrossRef] [Medline]
    11. O'Neill B, Ziebland S, Valderas J, Lupiáñez-Villanueva F. User-generated online health content: a survey of Internet users in the United Kingdom. J Med Internet Res. 2014;16(4):e118. [FREE Full text] [CrossRef] [Medline]
    12. Henderson EM, Eccleston C. An online adolescent message board discussion about the internet: Use for pain. J Child Health Care. 2015;19(3):412-418. [CrossRef] [Medline]
    13. Ettel DL, Lamanno LR, Neyra SA, Ettel WJ, Ettel GL, Mitchell MK. Teens and technology transforming acne treatment. Perm J. 2017;21:16-192. [FREE Full text] [CrossRef] [Medline]
    14. Qutteina Y, Hallez L, Mennes N, De Backer C, Smits T. What do adolescents see on social media? a diary study of food marketing images on social media. Front Psychol. 2019;10:2637. [FREE Full text] [CrossRef] [Medline]
    15. Hamilton JL, Nesi J, Choukas-Bradley S. Reexamining social media and socioemotional well-being among adolescents through the lens of the COVID-19 pandemic: a theoretical review and directions for future research. Perspect Psychol Sci. 2022;17(3):662-679. [FREE Full text] [CrossRef] [Medline]
    16. Zarocostas J. How to fight an infodemic. Lancet. 2020;395(10225):676. [FREE Full text] [CrossRef] [Medline]
    17. Gazibara T, Cakic M, Cakic J, Grgurevic A, Pekmezovic T. Patterns of online health information seeking after visiting a physician: perceptions of adolescents from high schools in central Belgrade, Serbia. Fam Pract. 2021;38(3):231-237. [CrossRef] [Medline]
    18. Asch JM, Asch DA, Klinger EV, Marks J, Sadek N, Merchant RM. Google search histories of patients presenting to an emergency department: an observational study. BMJ Open. 2019;9(2):e024791. [FREE Full text] [CrossRef] [Medline]
    19. Gazibara T, Cakic J, Cakic M, Grgurevic A, Pekmezovic T. Searching for online health information instead of seeing a physician: a cross-sectional study among high school students in Belgrade, Serbia. Int J Public Health. 2020;65(8):1269-1278. [CrossRef] [Medline]
    20. Jain A, Bickham D. Adolescent health literacy and the Internet: challenges and opportunities. Curr Opin Pediatr. 2014;26(4):435-439. [CrossRef] [Medline]
    21. Coyne I, Prizeman G, Sheehan A, Malone H, While A. An e-health intervention to support the transition of young people with long-term illnesses to adult healthcare services: Design and early use. Patient Educ Couns. 2016;99(9):1496-1504. [CrossRef] [Medline]
    22. Scott K, Caldwell P, Kang M, McCaffery K, Rachel Skinner S. Adolescents' use of dr google: help or hindrance? J Paediatr Child Health. 2018;54(11):1282-1283. [CrossRef] [Medline]
    23. Patil U, Kostareva U, Hadley M, Manganello JA, Okan O, Dadaczynski K, et al. Health literacy, digital health literacy, and COVID-19 pandemic attitudes and behaviors in U.S. college students: implications for interventions. Int J Environ Res Public Health. 2021;18(6):3301. [FREE Full text] [CrossRef] [Medline]
    24. Nutbeam D. Defining, measuring and improving health literacy. Health Evaluation and Promotion. 2015;42(4):450-455.
    25. Holmberg C, Berg C, Dahlgren J, Lissner L, Chaplin JE. Health literacy in a complex digital media landscape: Pediatric obesity patients' experiences with online weight, food, and health information. Health Informatics J. 2019;25(4):1343-1357. [FREE Full text] [CrossRef] [Medline]
    26. Freeman JL, Caldwell PH, Bennett PA, Scott KM. How adolescents search for and appraise online health information: a systematic review. J Pediatr. 2018;195:244-255.e1. [CrossRef] [Medline]
    27. Maitz E, Maitz K, Sendlhofer G, Wolfsberger C, Mautner S, Kamolz L, et al. Internet-based health information-seeking behavior of students aged 12 to 14 years: mixed methods study. J Med Internet Res. 2020;22(5):e16281. [FREE Full text] [CrossRef] [Medline]
    28. McKinnon KA, H Y Caldwell P, Scott KM. How adolescent patients search for and appraise online health information: A pilot study. J Paediatr Child Health. 2020;56(8):1270-1276. [CrossRef] [Medline]
    29. Taba M, Allen TB, Caldwell PH, Skinner SR, Kang M, McCaffery K, et al. Adolescents' self-efficacy and digital health literacy: a cross-sectional mixed methods study. BMC Public Health. 2022;22(1):1223. [FREE Full text] [CrossRef] [Medline]
    30. Bandura A. Social cognitive theory: an agentic perspective. Asian J of Social Psycho. 2002;2(1):21-41. [CrossRef]
    31. Zimmerman B. Self-efficacy and educational development. In: Bandura A, editor. Self-efficacy in changing societies. Cambridge. Cambridge University Press; 1995:202-231.
    32. Holch P, Marwood JR. Ehealth literacy in UK teenagers and young adults: exploration of predictors and factor structure of the ehealth literacy scale (eHEALS). JMIR Form Res. 2020;4(9):e14450. [FREE Full text] [CrossRef] [Medline]
    33. Li S, Cui G, Kaminga AC, Cheng S, Xu H. Associations between health literacy, ehealth literacy, and COVID-19-related health behaviors among chinese college students: cross-sectional online study. J Med Internet Res. 2021;23(5):e25600. [FREE Full text] [CrossRef] [Medline]
    34. van der Vaart R, Drossaert C. Development of the digital health literacy instrument: measuring a broad spectrum of health 1.0 and health 2.0 Skills. J Med Internet Res. 2017;19(1):e27. [FREE Full text] [CrossRef] [Medline]
    35. Rivadeneira MF, Salvador C, Araujo L, Caicedo-Gallardo JD, Cóndor J, Torres-Castillo AL, et al. Digital health literacy and subjective wellbeing in the context of COVID-19: a cross-sectional study among university students in Ecuador. Front Public Health. 2022;10:1052423. [FREE Full text] [CrossRef] [Medline]
    36. Paek H, Hove T. Social cognitive factors and perceived social influences that improve adolescent eHealth literacy. Health Commun. 2012;27(8):727-737. [CrossRef] [Medline]
    37. Schweder S, Raufelder D. Needs satisfaction and motivation among adolescent boys and girls during self-directed learning intervention. J Adolesc. 2021;88:1-13. [CrossRef] [Medline]
    38. Gray NJ, Klein JD, Noyce PR, Sesselberg TS, Cantrill JA. The Internet: a window on adolescent health literacy. J Adolesc Health. 2005;37(3):243. [CrossRef] [Medline]
    39. Collin P, Swist T. From products to publics? The potential of participatory design for research on youth, safety and well-being. Journal of Youth Studies. 2016;19(3):305-318. [CrossRef]
    40. Janiak E, Rhodes E, Foster AM. Translating access into utilization: lessons from the design and evaluation of a health insurance Web site to promote reproductive health care for young women in Massachusetts. Contraception. 2013;88(6):684-690. [CrossRef] [Medline]
    41. Ospina-Pinillos L, Davenport TA, Navarro-Mancilla AA, Cheng VWS, Cardozo Alarcón AC, Rangel AM, et al. Involving end users in adapting a spanish version of a web-based mental health clinic for young people in Colombia: exploratory study using participatory design methodologies. JMIR Ment Health. 2020;7(2):e15914. [FREE Full text] [CrossRef] [Medline]
    42. Hagen P, Collin P, Metcalf A, Nicholas M, Rahilly K, Swainston N. Participatory Design of evidence-based online youth mental health promotion, intervention and treatment. Abbotsford, Victoria. Young and Well Cooperative Research Centre; 2012.
    43. Bagnoli A, Clark A. Focus groups with young people: a participatory approach to research planning. Journal of Youth Studies. 2010;13(1):101-119. [CrossRef]
    44. Li S, Achilles M, Spanos S, Habak S, Werner-Seidler A, O’Dea B. A cognitive behavioural therapy smartphone app for adolescent depression and anxiety: co-design of ClearlyMe. tCBT. 2022;15:1-32. [CrossRef]
    45. Hugh-Jones S, Pert K, Kendal S, Eltringham S, Skelton C, Yaziji N, et al. Adolescents accept digital mental health support in schools: A co-design and feasibility study of a school-based app for UK adolescents. Mental Health & Prevention. Sep 2022;27:200241. [CrossRef]
    46. Hsieh H, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. 2005;15(9):1277-1288. [CrossRef] [Medline]
    47. Patton MQ. Qualitative evaluation and research methods, 2nd ed. California. SAGE Publications, Inc; 1990.
    48. NVivo. Release 1.6 ed. Denver. QSR International Pty Ltd; 2020. URL: https://help-nv.qsrinternational.com/ [accessed 2024-06-05]
    49. Vears DF, Gillam L. Inductive content analysis: A guide for beginning qualitative researchers. Focus on Health Professional Education: A Multi-Professional Journal. 2022;23(1):111-127. [CrossRef]
    50. Jonsen K, Jehn K. Using triangulation to validate themes in qualitative studies. Qualitative research in organizations and management: an international journal. 2009;4(2):123-150.
    51. diagrams.net. Northampton, England. JGraph Ltd; 2022. URL: https://diagrams.net [accessed 2024-06-05]
    52. Lupton D. Young people's use of digital health technologies in the global north: narrative review. J Med Internet Res. 2021;23(1):e18286. [FREE Full text] [CrossRef] [Medline]
    53. Liverpool S, Mota CP, Sales CMD, Čuš A, Carletto S, Hancheva C, et al. Engaging children and young people in digital mental health interventions: systematic review of modes of delivery, facilitators, and barriers. J Med Internet Res. 2020;22(6):e16317. [FREE Full text] [CrossRef] [Medline]
    54. Freeman JL, Caldwell PH, Scott KM. How adolescents trust health information on social media: a systematic review. Acad Pediatr. 2023;23(4):703-719. [FREE Full text] [CrossRef] [Medline]
    55. Morgan DL. Integrating Qualitative and Quantitative Methods: A Pragmatic Approach. United States. SAGE Publications, Inc; 2014:A.
    56. Hagen P, Collin P, Gesling K, Herrod L, Metcalf A, Blanchard M, et al. Enabling Participation: A companion document to the Young and Well Cooperative Research Centre Innovative Methodologies Guide Participatory Design of Evidence-Based Online Youth Mental Health Promotion, Intervention and Treatment. Abbotsford, Victoria. Young and Well Cooperative Research Centre; May 2016:1-9.

    ‎
    DHL: digital health literacy
    SCT: Social cognitive theory

    Edited by T de Azevedo Cardoso, S Ma; submitted 06.06.23; peer-reviewed by PC-I Pang, JR Bautista; comments to author 25.10.23; revised version received 24.11.23; accepted 16.05.24; published 07.08.24.

    Copyright

    ©Callum C Lewis, Melody Taba, Tiffany B Allen, Patrina HY Caldwell, S Rachel Skinner, Melissa Kang, Hamish Henderson, Liam Bray, Madeleine Borthwick, Philippa Collin, Kirsten McCaffery, Karen M Scott. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 07.08.2024.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research (ISSN 1438-8871), is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.