This paper is in the following e-collection/theme issue:

Web-based and Mobile Health Interventions (3092)

Published on in Vol 14, No 1 (2012): Jan-Feb

Identifying Factors for Optimal Development of Health-Related Websites: A Delphi Study Among Experts and Potential Future Users

Identifying Factors for Optimal Development of Health-Related Websites: A Delphi Study Among Experts and Potential Future Users

Identifying Factors for Optimal Development of Health-Related Websites: A Delphi Study Among Experts and Potential Future Users

Authors of this article:

Francine Schneider1 Author Orcid Image ;   Liesbeth van Osch1 ;   Hein de Vries1
Article Authors Cited by (50) Tweetations (30) Metrics

    Original Paper

    CAPHRI, Department of Health Promotion, Maastricht University, Maastricht, Netherlands

    Corresponding Author:

    Francine Schneider, MSc

    CAPHRI

    Department of Health Promotion

    Maastricht University

    P.O. Box 616

    Maastricht, 6200 MD

    Netherlands

    Phone: 31 433882397

    Fax:31 33671032

    Email: francine.schneider@maastrichtuniversity.nl


    Background: The Internet has become a popular medium for offering tailored and targeted health promotion programs to the general public. However, suboptimal levels of program use in the target population limit the public health impact of these programs. Optimizing program development is considered as one of the main processes to increase usage rates.

    Objective: To distinguish factors potentially related to optimal development of health-related websites by involving both experts and potential users. By considering and incorporating the opinions of experts and potential users in the development process, involvement in the program is expected to increase, consequently resulting in increased appreciation, lower levels of attrition, and higher levels of sustained use.

    Methods: We conducted a systematic three-round Delphi study through the Internet. Both national and international experts (from the fields of health promotion, health psychology, e-communication, and technical Web design) and potential users were invited via email to participate. During this study an extensive list of factors potentially related to optimal development of health-related websites was identified, by focusing on factors related to layout, general and risk information provision, questionnaire use, additional services, and ease of use. Furthermore, we assessed the extent to which experts and potential users agreed on the importance of these factors. Differences as well as similarities among experts and potentials users were deduced.

    Results: In total, 20 of 62 contacted experts participated in the first round (32% response rate); 60 of 200 contacted experts (30% response rate) and 210 potential users (95% response rate) completed the second-round questionnaire, and 32 of 60 contacted experts completed the third round (53% response rate). Results revealed important factors consented upon by experts and potential users (eg, ease of use, clear structure, and detailed health information provision), as well as differences regarding important factors consented upon by experts (eg, visual aids, self-monitoring tool, and iterative health feedback) or by potential users only (eg, bread crumb navigation and prevention of receiving spam).

    Conclusions: This study is an important first step in determining the agreed-upon factors that should be taken into account when developing online health promotion programs. The public health impact of these programs will be improved by optimizing the development process in line with these factors.

    J Med Internet Res 2012;14(1):e18

    doi:10.2196/jmir.1863

    Keywords

    Health promotion; Internet interventions; health behavior change; optimal development; Delphi study; experts; users 


    Worldwide, more and more people are accessing the Internet in search of health-related information [1]. It is estimated that globally a minimum of nearly seven million health-related Internet searches are conducted daily [2]. Since Internet penetration rates are still expanding, with currently almost two billion people having access to the Internet, the number of health-related searches is also expected to increase [3]. Therefore, the Internet is a promising channel for offering a broad range of health-related information, such as background information on health, treatment information, medication information, and health behavior information [1].

    Due to this high level of accessibility and its potential to reach large numbers of people [3], the Internet has also become a popular medium in the field of health promotion for offering tailored and targeted health promotion programs [4,5]. As a consequence, in recent years positive effects of online interventions applying computer-tailored techniques have been reported addressing different health behaviors [6], such as physical activity [7,8], nutrition [9,10], smoking cessation [11-14], and alcohol consumption [15,16]. Although these tailored interventions are very promising and have proven to be effective, actual reach is failing to live up to the high expectations [17-22]. Since the public health impact of interventions is determined not only by their efficacy but also by their levels of exposure in the target group [23,24], it is imperative to put effort into optimizing the level of exposure to Internet-delivered lifestyle interventions.

    Successful exposure is partly defined by the level of first-time access of the intervention, also referred to as first use or adoption ([19,24,25]. Besides first use, prolonged use of the intervention is essential. That is, engaging users in the intervention for a substantial amount of time fosters their knowledge of its content and involvement in its effective components, which consequently increases the chances of health behavior change [17,26]. Since health behavior change is a complex and continuous process, achieving sustained behavior change depends on both the intensity of the intervention and the number of times the intervention is visited [27,28]. Due to this high dose–response relationship, ensuring adherence or sustained use of the program is essential to further maximize its effect on subsequent health behavior change [29,30]. Both prolonged and sustained use of the intervention can be influenced not only by user characteristics (eg, demographic characteristics and motivation to use the intervention [17,31,32]) but also by specific strategies to increase adherence (eg, sending periodic reminders [33,34]) and by intervention characteristics (eg, appearance and content of the intervention [33,35]).

    As online behavior change interventions are delivered to the public by using a website or Web-based program, development of the website or program refers to composition of the actual intervention, and requires careful composition of the complete website or program it is embedded in. The website as a mode of delivery is described in the Internet intervention model as consisting of 8 main areas [35]: appearance (eg, the organization of information), behavioral prescriptions (eg, instructions on how to achieve behavior change), burdens (eg, poor navigation applications), content (eg, treatment information), delivery (eg, use of animations, audio, or graphics), message (eg, credibility and likability of the source), participation (eg, degree of interaction or the use of rewards), and assessment (eg, measuring needs of users or adjusting content to personal wishes). Adjusting these characteristics enables tailoring of the website to special needs of the population under consideration. All individual characteristics in these areas should therefore be kept in mind while developing the program.

    The primary characteristics of an intervention are determined during its development process. According to diffusion theory and social marketing principles [25,36], program development is one of the main processes to influence adoption rates of a new product (eg, a new website or online intervention). Besides including experts in the development process, it should also be done in accordance with the needs and wishes of the target group [36]. By considering and incorporating the opinions of potential future users, involvement is expected to increase, consequently resulting in increased appreciation of the intervention, lower levels of attrition, and higher levels of prolonged and sustained use [37]. A new intervention should, therefore, be developed in close collaboration with the target group.

    Although many studies have investigated the effectiveness and appreciation of numerous components of websites that deliver health behavior interventions, results have been inconclusive. Some studies recommend using a multimedia approach [38] or the use of interactive tools, such as surveys, quizzes, and games, whereas other studies seem to contradict these findings [5]. Systematically studying different elements of websites is a very extensive and time-consuming process. Even though some elements of websites should be incorporated at all times, such as accurate and comprehensible information [39], a lot of effort should still be put into making other elements operational [35].

    To date, studies on development of health-related websites have not included perspectives of both experts and users. Hence, this study was a first attempt to identify which elements on health-related websites are perceived as necessary and preferable by these two groups. The current study thus included both experts in the fields of health promotion, health psychology, e-communication, and technical Web design and potential future users. By including elements that are perceived as important by both groups while developing the website, developers may optimize exposure rates of the program [36]. Furthermore, involving different groups of experts will lead to more diverse information regarding health communication and behavior change as well as technical information regarding website development. To identify the potential factors that are related to optimal development of health-related websites, we conducted a three-round Delphi study. The specific aim of the study was to identify factors that are associated with optimal development of health-related websites. Besides identifying these factors, we investigated the degree to which experts and potential future users agreed on the importance of the factors.


    We conducted a three-round Delphi study through the Internet. Due to its systematic nature, a Delphi study is considered to be an appropriate method to derive consensus on health-related issues for which scientific evidence is incomplete or scarce by involving a representative panel of experts [40]. Also, because after each round feedback on group results is provided, the iterative approach allows participants to adjust their opinions when needed. Finally, the structure of the study guarantees anonymity of the participants, thereby preventing conformity biases [41,42].

    The current Delphi study consisted of two substudies. The first substudy (study 1a), including only experts, consisted of three rounds. The first round aimed at providing a list of potential factors related to optimal development of health-related websites by means of an open-ended questionnaire. Next, for the second round, experts were invited to rate the importance of all factors identified in the first round by using a structured questionnaire. Finally, a third round enabled experts to reevaluate their opinions by providing controlled feedback regarding group mean scores, thereby producing consensus. The second substudy (study 1b) included only potential future users and consisted of one round. This round resembled the second round of the first substudy and allowed users to rate the importance of those factors that were identified by experts in the first round by using the same structured questionnaire. We compared second-round results from both studies in order to identify potential differences between experts and potential users regarding the importance they placed on factors related to optimal design.

    Study 1a: Experts

    First Round
    Procedure and Participants

    For the first round, we selected experts from the fields of health promotion, health psychology, e-communication, and technical Web design to obtain a variety of insights from researchers with both theory-based and more practice-based backgrounds. Invited experts with a theory-based background were all first or second authors on scientific papers in the field of eHealth and eHealth promotion published between 2000 and 2009. We used database searches in PsychINFO and Medline to identify experts and to examine reference lists from related papers, book chapters, review studies, and conference abstracts. We selected experts with a practice-based background on the basis of their publications, but also by approaching our own network and by asking responding experts to provide names of important experts in the field.

    This resulted in a list of 62 experts who were invited by email to participate in all three rounds of the Delphi study. The email contained detailed information on the goal and study procedure, as well as a link referring them directly to the first-round questionnaire. Nonresponders received a reminder email after the 3-week response period expired. A total of 20 experts (32% response rate) responded to the invitation.

    Questionnaire

    The first-round questionnaire consisted of 7 open-ended questions. The questions pertained to different subjects related to the development of health-related websites: (1) optimal layout, (2) type of general information provided, (3) type of health risk information provided, (4) ease of use, (5) use of visual aids, and (6) additional information provided. Health-related websites often provide questionnaires to allow visitors the opportunity to assess their own current health status. To provide an accurate update of their health status these questionnaires often tend to be extensive and therefore sensitive to early dropout. Therefore, we also included one open-ended question to gain more insight into factors that contribute to completion of the questionnaires often provided on health-related websites (7). Health-related websites were defined as websites aiming at assisting people to adopt a healthier lifestyle, by offering them important and diverse information regarding health and health-related behaviors (eg, physical activity, smoking, alcohol consumption, and nutrition).

    Data Analysis

    Responses of experts were analyzed, resulting in an extensive list of potential factors related to optimal development of health-related websites. Two researchers independently listed all unique factors and combined similar responses into 1 factor. For those factors on which no agreement was obtained, a third researcher was approached to give a decisive answer.

    Second Round
    Procedure and Participants

    Experts participating in the first round were also invited to participate in the second round. We selected an additional 180 experts by means of the same strategies used for the first round.

    A total of 200 experts received an email inviting them to participate in the second and third rounds. The invitation contained a link that directed experts to the second-round questionnaire. Nonresponders received two reminder emails: after 3 and after 5 weeks. A total of 60 experts (30% response rate) responded to the invitation (Table 1).

    Table 1. First-, second-, and third-round rates of responses of experts
    Field First round (n = 62) Second round (n = 200) Third round (n = 60)
    Invitations Response (%) Invitations Response (%) Invitations Response (%)
    Health promotion 16 7 (44%) 60 18 (30%) 18 10 (56%)
    Health psychology 16 5 (31%) 60 14 (23%) 14 8 (57%)
    E-communication 15 5 (33%) 40 12 (30%) 12 5 (42%)
    Technical Web design 15 2 (13%) 40 16 (40%) 16 9 (56%)
    Total 62 20 (32%) 200 60 (30%) 60 32 (53%)
    Questionnaire

    The second-round questionnaire was composed based on the factors identified by experts in the first round. This resulted in a questionnaire consisting of 85 structured questions. Experts were asked to rate these factors on a 7-point Likert scale ranging from 1 (not at all important) to 7 (extremely important).

    Data Analysis

    The closed-ended questions were analyzed following the standards for analyzing data for a Delphi study, by calculating median scores, also referred to as the 50th percentile score, to determine the importance of the various factors. Furthermore, the interquartile range (IQR) was calculated to assess the degree of agreement between the experts on the importance of the factors [31,41,43]. The IQR represents the distance between the 25th and the 75th percentiles, with a small value indicating a higher degree of agreement. An IQR smaller than 1 is considered to indicate good consensus on a 7-point scale as used in the present study and means that more than 50% of all cases fall within 1 point of one another [44]. To deduce those factors that were considered to be either very or extremely important by the majority of experts (IQR < 1), the cut-off point for importance was a median score of ≥6.

    Third Round
    Procedures and Participants

    All experts participating in the second round (n = 60) were invited to participate in the third and final Delphi round, using the same procedure as in the previous round. A total of 32 experts (53% response rate of second-round participants) responded to the invitation (Table 1).

    Questionnaire

    The third-round questionnaire was an adapted version of the second-round questionnaire, containing only those factors (n = 27) on which no consensus was obtained during the second round (IQR > 1). In line with the Delphi method, additional feedback on second-round group results (median and IQR) was provided, enabling experts to rerate their answers.

    Data Analysis

    The degree of agreement and consensus among experts was measured by computing median scores and IQRs.

    Study 1b: Potential Future Users

    Round 2
    Procedures and Participants

    Potential future users were recruited through an independent, commercial Dutch Internet research panel [45]. From this panel, consisting of approximately 20,000 members, a sample of 220 members were invited to participate in this study. Potential users were invited to take part only in the second round of this Delphi study, since the main goal was to compare users’ opinions with experts’ opinions and not to force consensus among users on the different factors. Respondents from this panel were rewarded for their participation in this study in accordance with the standards of the consumer panel (approximately €5). All participants received an email invitation informing them about the goal and content of the study. The invitation also contained a link that directed participants to the questionnaire. A total of 210 potential future users (95.5% response rate) responded to the invitation (Table 2).

    Table 2. Demographic profile of potential future users (n = 210)
    Demographic characteristic % n
    Age (years)
    Range 19–65
    Mean (SD) 46.49 (47.00)
    Gender
    Male 51 106
    Female 49 104
    Ethnicity
    Native 98 206
    Nonnative 2 4
    Education level
    Low 11 24
    Medium 59 123
    high 30 63
    Income
    Less than average 29 61
    Average 26 54
    More than average 45 94
    Internet use: work or studya
    Frequent 55 116
    Average 12 26
    Infrequent 10 20
    Not applicable 23 48
    Internet use: private usea
    Frequent 96 202
    Average 3 6
    Infrequent 1 2
    Internet use: health purposes
    Frequent 21 45
    Average 69 144
    Infrequent 10 21

    a Frequent user: >3 times/week; average user: every week, but not >3 days/week; infrequent user: <1/week.

    Questionnaire

    The questionnaire assessed demographics such as gender, age, ethnicity, education level, and income. In addition, participants were asked to indicate how often they used the Internet for work or study purposes (1, [almost] never; 6, every day), for private use (1, [almost] never; 6, every day), and for finding information on health-related topics (1, never; 6, very often). For the questions assessing Internet use for work and private purposes, new categories were composed: frequent user (>3 times a week), average user (every week, but not >3 days a week), and infrequent user (<1 a week).

    Furthermore, the questionnaire contained all factors identified by experts in the first round of study 1a, resulting in a questionnaire consisting of 92 questions. Potential future users were asked to rate the importance of all factors on a 7-point Likert scale ranging from 1 (not at all important) to 7 (extremely important).

    Data Analysis

    Data were analyzed according to the same principles used for the second round of study 1a, by calculating median scores and IQRs. Differences in consensus between the expert group and potential future users were analyzed using Wilcoxon signed rank sum tests. Similarities and differences in second-round results between experts and potential future users were further analyzed by using multivariate analysis of variance.


    Study 1a: Experts

    We used experts’ responses from the first round to compose the questionnaire for the second round, and results of the first round are therefore shown as question items. We grouped items with regard to layout, general information content, health risk information content, ease of use, questionnaire completion, visual aids, and additional services. This resulted in a list of 85 factors thought to be related to optimal development of health-related websites. There were 10 factors in the layout category, 11 in general information content, and 16 in health risk information content. A total of 18 factors were mentioned as facilitating ease of use. The remaining factors referred to questionnaire completion (11), visual aids (6), and additional services (13). An overview of all results is given in Multimedia Appendix 1.

    Consensus

    During the second round, consensus was reached (IQR ≤ 1) on 57 factors. After experts rerated their answers during the third round, aided by feedback on second-round group results, consensus was obtained on another 6 factors. In total consensus was obtained on 74% of all factors.

    Importance

    Experts identified 33 factors as being very or extremely important (median ≥ 6): (1) layout, with 5 factors (50% of layout-related factors, eg, professional appearance and use of color), (2) general information provision, with 4 factors (36% of general information-related factors, eg, information on pros and cons of a healthy lifestyle, and tailored information on health), (3) health risk information provision, with 3 factors (19% of health risk-related factors, eg, information on skills that help to decrease health risks and personal advice on how to decrease health risks), (4) ease of use, with 10 factors (56% of factors related to ease of use, eg, availability of an easy log-in procedure and a clear navigation structure), (5) questionnaire completion, with 6 factors (55% of factors related to completion, eg, provision of a progress bar and provision of an option for partial completion), and (6) additional services, with 5 factors (38% of factors related to additional services, eg, provision of a self-monitoring tool and iterative feedback).

    Although consensus was obtained on 3 factors relating to the provision of visual aids (eg, provision of cartoons, pictures, and graphical representations of relevant information), these factors had respective median scores of 4, 5, and ,5 and thus were not considered as extremely or very important. Combining the results on importance and obtained consensus, we can conclude that consensus (IQR ≤ 1) was obtained on the importance (median ≥ 6) of 24 factors (Table 3).

    Table 3. Median scores for important factors in health-related Internet sites on which consensus was obtained by experts, potential users, or botha
    Factor Second round Third round
    Experts (n = 60) Users (n = 120) Experts (n = 32)
    Median IQRb Median IQR Median IQR
    Which factors determine optimal layout?
    1. User friendlyc 7 1 6 2 NAd NA
    2. Lively appearancec 6 1 5 1 NA NA
    3. Use of visual materials, such as pictures, videos, and graphicsc 6 1 5 1 NA NA
    4. Use of colorc 6 1 5 2 NA NA
    5. Professional appearancee 6 2 6 1 6 1
    What kind of general information should definitely be provided on health?
    6. Information on how to attain a healthy lifestylec 6 1 6 1 NA NA
    7. Information on pros and cons of a healthy lifestylec 6 1 5 1 NA NA
    8. Information on health risk behaviorsc 6 1 5 1 NA NA
    9. Personal tailored information on healthc 6 2 5 1 6 1
    What kind of health risk information should definitely be provided?
    10. Information in the form of visual aids, eg, graphsc 6 1 5 1 NA NA
    Which factors determine optimal ease of use?
    11. Clear structuref 7 1 6 1 NA NA
    12. Availability of an easy log-in proceduref 7 1 6 1 NA NA
    13. Use of comprehensive languagec 7 1 6 2 NA NA
    14. Clear navigation structurec 7 1 2 2 NA NA
    15. Simple site designf 6 1 6 1 NA NA
    16. Availability of a helpdeskc 6 1 5 2 NA NA
    17. Availability of a function to customize the site for personal needsc 6 1 4 2 NA NA
    18. Availability of bread crumb navigatione 5 2 6 1 5 2
    19. Availability of contact information for developersc 6 2 5 1 6 1
    Which factors determine whether visitors complete questionnaires provided on the sites?
    20. Clearly structured questionnaire with clear headings and subheadingsf 6 1 6 1 NA NA
    21. Progress barf 6 1 6 1 NA NA
    22. Opportunity to stop completion and proceed at a later timef 6 1 6 1 NA NA
    23. Information on personal benefits of completionc 6 1 5 1 NA NA
    24. Information on relevance of questionnaire completionc 6 1 5 1 NA NA
    25. Guarantee that completion will not result in receiving spame 6 2 7 1 5 1
    26. Use of original questionse 4 1 6 1 NA NA
    What additional services should be provided?
    27. Search enginee 5 1 6 1 NA NA
    28. Opportunity to print or download relevant informationc 6 1 5 1 NA NA
    29. Opportunity for regularly revisiting the sitec 6 1 5 1 NA NA
    30. Tool to self-monitor personal health behavior changec 6 1 5 2 NA NA
    31. Iterative feedback during revisits to assess users against their own previous performancesc 6 1 5 2 NA NA
    32. Privacy statementc 6 2 5 2 6 1

    a Only experts were included in the third round. Statements on which consensus was obtained in the second round were excluded from the third round and results are therefore missing. Only factors on which consensus was obtained, either by experts or users, or both, are displayed (interquartile range ≤ 1).

    b Interquartile range.

    c Factors on which consensus was obtained only by experts.

    d Not applicable.

    e Factors on which consensus was obtained only by potential users.

    f Factors on which consensus was obtained by both experts and potential users.

    Study 1b: Potential Future Users

    Consensus

    Potential future users reached consensus (IQR ≤ 1) on 60 of the 85 factors (71%). An overview of all results is given in Multimedia Appendix 1.

    Importance

    A total of 17 factors were identified as being very or extremely important (median ≥ 6). These factors were mapped into different categories: (1) layout, with 3 factors (30% of factors related to layout, eg, limited amount of distractions and user friendliness), (2) general information provision, with 1 factor (9% of factors related to general information; information on how to attain a healthy lifestyle), (3) ease of use, with 7 factors (39% of factors related to ease of use, eg, availability of an easy log-in procedure and use of comprehensive language), (4) questionnaire completion, with 3 factors (55% of factors related to questionnaire completion, eg, provision of a progress bar and provision of an option for partial completion), and (5) additional services, with 5 factors (38% of factors related to additional services, eg, provision of a self-monitoring tool and iterative feedback).

    Although consensus was obtained on 14 factors relating to ease of use (eg, information on skills that help to decrease health risks, and personal advice on how to decrease health risks), all these factors had median scores of 5 and were not considered to be extremely or very important. Furthermore, all factors related to provision of visual aids had median scores <6, even though consensus was obtained on all 6 factors. Combining the results on importance and obtained consensus, we can conclude that consensus (IQR ≤ 1) was obtained on the importance (median ≥ 6) of 11 factors (Table 3).

    Similarities and Differences Between Experts and Potential Future Users

    The results of the second round revealed that experts agreed on the importance of 24 factors, whereas the majority of potential future users agreed on the importance of 11 factors (median ≥ 6; IQR ≤ 1). The statistical tests showed a nonsignificant (z = –.262, P = .29) difference between the overall level of consensus obtained in the two groups.

    Experts and potential future users both agreed on the importance of 7 factors: (1) general information, with 1 factor (the availability of information on how to attain a healthy lifestyle), (2) ease of use, with 3 factors (clear structure, easy log-in procedure, simple site design), and (3) questionnaire completion, with 3 factors (provision of a progress bar, a clearly structured questionnaire, and an option for partial completion).

    Contrary to these similarities in perceived importance, experts and potential future users’ opinions differed on 22 factors, significantly so on 19 factors (Table 4). The multivariate analysis of variance found a significant effect of group on the importance of factors related to optimal development (F1,268 = 95.95, P < .001, R2 = .90. Univariate F tests revealed that experts rated a lively appearance, the use of visual aids, the use of color, user friendliness (optimal layout), information in the form of visual aids (health risk information), opportunities for customizing the site, availability of a helpdesk, use of comprehensive language, simple site design, clear navigation structure (optimal ease of use), provision of relevance and personal benefits of completing the questionnaire (questionnaire completion), provision of a self-monitoring tool, option for printing and downloading information, stimulation of revisits, and iterative feedback (additional services) as significantly more important factors than did potential future users.

    Conversely, potential future users rated the availability of bread crumb navigation (optimal ease of use), the provision of a guarantee that questionnaire completion will not result in receiving spam, and usage of original questions in questionnaires (questionnaire completion) as more important than did experts.

    Table 4. Univariate F tests for differences in rating the importance of a factor between experts and users
    Factor Mean experts
    (n = 60)     		Mean users
    (n = 210)     		Fη2
    Optimal layout
    Lively appearance 5.48 (0.83) 5.01 (0.98) 11.37*** .04
    Professional appearance 5.87 (0.97) 5.62 (0.81) 3.84 .01
    Use of visual materials, such as pictures, videos, and graphics 5.98 (0.89) 5.21 (0.98) 30.23*** .10
    Use of color 5.55 (0.89) 5.00 (1.05) 13.85*** .05
    User friendly 6.68 (0.57) 6.09 (0.89) 24.08*** .08
    General information
    Information on pros and cons of a healthy lifestyle 5.53 (0.91) 5.36 (0.97) 1.58 .01
    Information on health risk behaviors 5.60 (0.81) 5.43 (0.96) 1.59 .01
    Health risk information
    Information in the form of visual aids, such as graphs 5.71 (1.09) 4.88 (1.06) 28.53*** .10
    Optimal ease of use
    Availability of a function to customize the site for personal needs 5.78 (0.98) 3.92 (1.36) 97.63*** .27
    Availability of a helpdesk 5.68 (0.89) 5.04 (1.06) 18.09*** .06
    Use of comprehensive language 6.50 (0.70) 6.01 (0.91) 14.61*** .05
    Availability of bread crumb navigation 5.05 (1.19) 5.63 (0.97) 15.22*** .05
    Simple site design 6.02 (0.89) 5.65 (0.89) 7.88** .03
    Clear navigation structure 6.57 (0.75) 2.18 (0.95) 1084.30** .80
    Questionnaire completion
    Information on relevance of questionnaire completion 5.77 (0.89) 5.51 (0.89) 3.92* .01
    Information on personal benefits of completion 5.78 (0.92) 5.34 (0.98) 9.95** .04
    Guarantee that completion will not result in receiving spam 5.87 (1.16) 6.29 (0.95) 8.44** .03
    Use of original questions 4.53 (1.19) 5.59 (1.01) 46.50*** .15
    Additional services
    Tool to self-monitor personal health behavior change 5.88 (0.85) 4.84 (1.21) 39.41*** .13
    Opportunity to print or download relevant information 6.18 (0.83) 5.41 (1.15) 23.41*** .08
    Opportunity to regularly revisit the site 6.02 (0.89) 5.15 (1.13) 29.95*** .10
    Iterative feedback during revisits 5.92 (0.79) 5.04 (1.07) 34.45*** .11

    *P < .05, **P < .01, ***P < .001.


    The public health impact of Internet health communication programs is suboptimal in the target population [17,18,46]. Since program development is one of the main processes to influence reach of a new product [25], this study attempted to give both experts and users a say in the development process. By means of a Delphi study, we identified an extensive list of factors potentially related to optimal development of health-related websites, as well as the extent to which experts and users agreed on the importance of these factors. In addition, we deduced similarities and differences in perceived importance between experts and potential users.

    Main Findings

    This study identified an extensive list of factors that might contribute to the development of health-related websites. The importance of a selected set of these factors was stressed by both experts and potential users. Developers should therefore attempt to take these factors into account when developing a health-related website. In particular, the provision of information on attaining a healthy lifestyle was emphasized. Furthermore, a clearly structured website with a simple design and the presence of an easy log-in procedure were brought up as important factors, which corresponds to results from earlier studies [31,39]. Also, with respect to optimizing questionnaire completion, several important factors were identified. To increase questionnaire completion, websites should offer a progress bar and an option for partial completion. These results are also in line with previous findings [47]. Since questionnaire length seems to be inversely related to actual participation as well as completion [48], providing an option for partial completion might be a solution to prevent attrition when using extensive questionnaires. Besides factors agreed upon by both groups, results also indicated that the majority of experts and users significantly differed on the importance of several factors. Due to their specific knowledge on theories in their area of expertise, experts should be involved in the development process to ensure scientific input. However, opinions of potential users must not be neglected because, by engaging users in this process, their involvement will increase. This involvement will subsequently increase the chances of obtaining higher levels of appreciation, prolonged and sustained use, and lower levels of attrition [37].

    The recently introduced Internet intervention model [35] offers an opportunity to classify important factors identified in this study into several main areas that determine how the website is developed and functions. The first area pertains to the appearance of the website, which is one of the first website characteristics visitors are confronted with. Since previous research indicated that more than half of all website visitors are inclined to leave the website within the first 30 seconds [49], the exterior of the website should be appealing in order to attract sufficient visitor attention and prevent early disengagement. In this study, experts highlighted important factors, such as a lively appearance, the use of color, and the use of visual aids, as contributing to the appearance of health-related websites. Furthermore, potential users stressed the importance of developing websites with a professional appearance [39]. The model also emphasizes behavioral prescription, which refers to the instructions users receive on how to change their lifestyle. In the current study, both experts and potential users emphasized the importance of offering information on how to achieve a healthy lifestyle on health-related websites. Another important area of the website is the size of the burden that actual use of the website entails. Results from this study indicated that both experts and users value the presence of an easy log-in procedure. Furthermore, experts stressed the importance of developing a user-friendly website, to decrease the effort visitors must invest in navigating the website. This also entails a simple site design and a clear navigation structure. Furthermore, potential users indicated they appreciated the presence of bread crumb navigation and a search engine, to ensure visible navigation on the website [50]. Content that is provided on the website should be accurate, complete, and readable [2,51]. In line with these findings, experts agreed on the importance of using comprehensive language on health-related websites. Since health-related websites mainly aim at helping or assisting people to adopt a healthy lifestyle, experts stressed that detailed information on the pros and cons of a healthy lifestyle and on health behaviors in general should be included in the website’s content [31,39]. Additionally, potential users indicated the importance of an opportunity to download or print relevant information that is provided on the website. Participation is considered as another important component of the website and focuses on its ability to engage and involve visitors. Providing reinforcement is considered an important strategy to engage visitors and can be a reward for progressing through the website content [35]. The provision of personalized feedback regarding the status of lifestyle behaviors and the opportunity to self-monitor behavior change can be regarded as a form of reinforcement. In line with this, experts stressed the importance of providing a self-monitoring tool to allow visitors to assess their current health behavior status and to track their progress. Finally, assessment refers to the website’s ability to adjust the website to specific user needs and wishes. In this study, experts indicated that an opportunity to adjust the website to personal preferences was highly appreciated. Therefore, visitors should have an opportunity to adjust not only the appearance of the website but also its content, corresponding to their needs and wishes. Tailoring is considered to be an appropriate strategy to adjust health information to personal characteristics and preferences [52]. In addition, previous studies indicated that providing personalized, iterative feedback regarding one’s health behavior might in itself stimulate revisits to the program [31]. In line with these previous findings, the majority of experts agreed on the importance of stimulating revisits and using iterative feedback. To adjust information to personal characteristics and specific user needs, questionnaires are often requisite to obtain detailed information on these topics. As stressed in the introduction, these questionnaires often tend to be extensive and therefore sensitive to early dropout. It is therefore imperative to stimulate questionnaire completion in order to optimize adjustment to visitor’s needs and wishes. Experts indicated that information on relevance and personal benefit [47] of completing a questionnaire should be provided. Potential users indicated that developers should develop original questions and guarantee that questionnaire completion will not result in receiving spam.

    The Internet Intervention model emphasizes two additional areas that were also identified in this Delphi study: mode of delivery and the message. Mode of delivery refers to ways in which the intervention content can be delivered to visitors. Specific strategies that can be used pertain to the use of animations, audio, video, or testimonials. The message area, on the other hand, focuses on the source and style of the message and addresses issues such as the trustworthiness and expertise of the website developers. Even though several factors pertaining to these areas were identified in the Delphi study, the majority of both experts and potential users did not agree on the importance of these factors.

    Limitations

    Several limitations to this study should be considered. First, the development of health-related websites is a very broad topic entailing diverse elements. To obtain detailed information on factors related to each separate element of the development process (eg, deciding on layout, content, and additional services), the questionnaire was divided into 7 categories. This rather broad setup, which is often inherent to the Delphi process, may have limited the specification of in-depth information. As the Delphi method does not allow for further specification of such factors in later rounds, still more in-depth examination of such factors is required. Second, response rates for the expert study ranged between 33% and 53%. Although this range is somewhat low, previous Delphi studies have reported similar response rates [31,43,53]. Suboptimal response rates in our study might have been a consequence of experts being invited to participate in at least two rounds of the Delphi study. Especially, response rates among experts with a practice-based background, coming from the field of technical Web design, were low during the first round. To account for these suboptimal response rates and to balance the input from the various expert fields, we put additional effort into recruiting these experts for participation in the second round. Finally, to ensure dispersion and coverage of answers, as well as optimal participation rates, it is recommendable for future studies to limit the number of questions used in the Delphi study. Incorporating a small number of questions in the first-round questionnaire may allow experts to give more in-depth input and may increase participation in subsequent rounds, thereby diminishing attrition rates.

    Implications

    The currents study is a first step in increasing exposure rates of online health promotion programs. The results of this study need further experimental testing to identify which (combination of) factors ultimately results in the best result. Although the vast number of factors that play a role may hinder the feasibility of a full factorial experimental design, in-depth experimental studies on the importance of some categories of factors are recommended.

    Acknowledgments

    This study was funded by ZonMw, the Netherlands Organization for Health Research and Development (grant number: 120710004). Publication of the manuscript was supported by NWO, the Netherlands Organization for Scientific Research. The authors would like to thank Kim Eijmael and Claire Dolmans for their assistance during the recruitment phases and analysis of the first-round results.

    Conflicts of Interest

    Hein de Vries is scientific director of Vision2Health, a company that licenses evidence-based innovative computer-tailored health communication tools.

    ‎

    Multimedia Appendix 1

    Results of the Delphi study per factor for experts and potential future users (second and third round).

    PDF File (Adobe PDF File), 68KB
    1. Bansil P, Keenan NL, Zlot AI, Gilliland JC. Health-related information on the Web: results from the HealthStyles Survey, 2002-2003. Prev Chronic Dis 2006 Apr;3(2):A36 [FREE Full text] [Medline]
    2. Eysenbach G, Kohler C. What is the prevalence of health-related searches on the World Wide Web? Qualitative and quantitative analysis of search engine queries on the internet. AMIA Annu Symp Proc 2003:225-229. [Medline]
    3. Internet World Stats. Miniwatts Marketing Group. 2011 Oct 6. Internet Usage Statistics: The Internet Big Picture   URL: http://www.internetworldstats.com/stats.htm [accessed 2011-11-25] [WebCite Cache]
    4. Brug J, Oenema A, Kroeze W, Raat H. The internet and nutrition education: challenges and opportunities. Eur J Clin Nutr 2005 Aug;59 Suppl 1:S130-7; discussion S138. [CrossRef] [Medline]
    5. Ritterband LM, Gonder-Frederick LA, Cox DJ, Clifton AD, West RW, Borowitz S. Internet interventions: in review, in use and into the future. Prof Psychol Res Pr 2003;34:527-534. [CrossRef]
    6. Krebs P, Prochaska JO, Rossi JS. A meta-analysis of computer-tailored interventions for health behavior change. Prev Med 2010;51(3-4):214-221. [CrossRef] [Medline]
    7. Vandelanotte C, Spathonis KM, Eakin EG, Owen N. Website-delivered physical activity interventions a review of the literature. Am J Prev Med 2007 Jul;33(1):54-64. [CrossRef] [Medline]
    8. van den Berg MH, Schoones JW, Vliet Vlieland TP. Internet-based physical activity interventions: a systematic review of the literature. J Med Internet Res 2007;9(3):e26 [FREE Full text] [CrossRef] [Medline]
    9. Neville LM, O'Hara B, Milat AJ. Computer-tailored dietary behaviour change interventions: a systematic review. Health Educ Res 2009 Aug;24(4):699-720 [FREE Full text] [CrossRef] [Medline]
    10. Brug J, Glanz K, Van Assema P, Kok G, van Breukelen GJ. The impact of computer-tailored feedback and iterative feedback on fat, fruit, and vegetable intake. Health Educ Behav 1998 Aug;25(4):517-531. [Medline]
    11. Strecher VJ. Computer-tailored smoking cessation materials: a review and discussion. Patient Educ Couns 1999 Feb;36(2):107-117. [Medline]
    12. Myung SK, McDonnell DD, Kazinets G, Seo HG, Moskowitz JM. Effects of Web- and computer-based smoking cessation programs: meta-analysis of randomized controlled trials. Arch Intern Med 2009 May 25;169(10):929-937 [FREE Full text] [CrossRef] [Medline]
    13. Borland R, Balmford J, Hunt D. The effectiveness of personally tailored computer-generated advice letters for smoking cessation. Addiction 2004 Mar;99(3):369-377. [CrossRef] [Medline]
    14. Dijkstra A, De Vries H, Roijackers J. Long-term effectiveness of computer-generated tailored feedback in smoking cessation. Health Educ Res 1998 Jun;13(2):207-214 [FREE Full text] [Medline]
    15. Riper H, Kramer J, Smit F, Conijn B, Schippers G, Cuijpers P. Web-based self-help for problem drinkers: a pragmatic randomized trial. Addiction 2008 Feb;103(2):218-227. [CrossRef] [Medline]
    16. White A, Kavanagh D, Stallman H, Klein B, Kay-Lambkin F, Proudfoot J, et al. Online alcohol interventions: a systematic review. J Med Internet Res 2010;12(5):e62 [FREE Full text] [CrossRef] [Medline]
    17. Eysenbach G. The law of attrition. J Med Internet Res 2005;7(1):e11 [FREE Full text] [CrossRef] [Medline]
    18. De Nooijer JOA, Kloek G, Brug J, De Vries H, De Vries NK. Bevordering van gezondheidsgedrag via internet: nu en in de toekomst [Promotion of Healthy Behavior Through the Internet: Now and in the Future]. Maastricht: Maastricht University; 2005.
    19. Crutzen R. Hard to Get, Hard to Keep: Dissemination of and Exposure to Internet-Delivered Health Behavior Change Interventions Aimed at Adolescents [dissertation]. Maastricht: Maastricht University; 2009.
    20. Glasgow RE, Nelson CC, Kearney KA, Reid R, Ritzwoller DP, Strecher VJ, et al. Reach, engagement, and retention in an Internet-based weight loss program in a multi-site randomized controlled trial. J Med Internet Res 2007;9(2):e11 [FREE Full text] [CrossRef] [Medline]
    21. Evers KE, Cummins CO, Prochaska JO, Prochaska JM. Online health behavior and disease management programs: are we ready for them? Are they ready for us? J Med Internet Res 2005 Jul 1;7(3):e27 [FREE Full text] [CrossRef] [Medline]
    22. Cook C, Heath F, Thompson RL. A meta-analysis of response rates in web- or Internet-based surveys. Educ Psychol Meas 2000;60:821-836. [CrossRef]
    23. de Vries H, Brug J. Computer-tailored interventions motivating people to adopt health promoting behaviours: introduction to a new approach. Patient Educ Couns 1999 Feb;36(2):99-105. [Medline]
    24. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health 1999 Sep;89(9):1322-1327. [Medline]
    25. Rogers E. Diffusion of Innovations. New York, NY: Free Press; 2003.
    26. Danaher BG, Boles SM, Akers L, Gordon JS, Severson HH. Defining participant exposure measures in Web-based health behavior change programs. J Med Internet Res 2006;8(3):e15 [FREE Full text] [CrossRef] [Medline]
    27. Crutzen R, de Nooijer J, Candel MJ, de Vries NK. Adolescents who intend to change multiple health behaviours choose greater exposure to an internet-delivered intervention. J Health Psychol 2008 Oct;13(7):906-911. [CrossRef] [Medline]
    28. Verheijden MW, Jans MP, Hildebrandt VH, Hopman-Rock M. Rates and determinants of repeated participation in a web-based behavior change program for healthy body weight and healthy lifestyle. J Med Internet Res 2007;9(1):e1 [FREE Full text] [CrossRef] [Medline]
    29. Christensen H, Griffiths KM, Farrer L. Adherence in internet interventions for anxiety and depression. J Med Internet Res 2009;11(2):e13 [FREE Full text] [CrossRef] [Medline]
    30. Norman GJ, Zabinski MF, Adams MA, Rosenberg DE, Yaroch AL, Atienza AA. A review of eHealth interventions for physical activity and dietary behavior change. Am J Prev Med 2007 Oct;33(4):336-345. [CrossRef] [Medline]
    31. Brouwer W, Oenema A, Crutzen R, de Nooijer J, de Vries NK, Brug J. An exploration of factors related to dissemination of and exposure to internet-delivered behavior change interventions aimed at adults: a Delphi study approach. J Med Internet Res 2008;10(2):e10 [FREE Full text] [CrossRef] [Medline]
    32. Moorman C, Matulich E. A model of consumers' preventive health behaviors: the role of health motivation and health ability. J Consum Res 1993;20:208-228.
    33. Brouwer W, Kroeze W, Crutzen R, de Nooijer J, de Vries NK, Brug J, et al. Which intervention characteristics are related to more exposure to internet-delivered healthy lifestyle promotion interventions? A systematic review. J Med Internet Res 2011;13(1):e2 [FREE Full text] [CrossRef] [Medline]
    34. Fry JP, Neff RA. Periodic prompts and reminders in health promotion and health behavior interventions: systematic review. J Med Internet Res 2009;11(2):e16 [FREE Full text] [CrossRef] [Medline]
    35. Ritterband LM, Thorndike FP, Cox DJ, Kovatchev BP, Gonder-Frederick LA. A behavior change model for internet interventions. Ann Behav Med 2009 Aug;38(1):18-27. [CrossRef] [Medline]
    36. Cugelman B, Thelwall M, Dawes P. Online interventions for social marketing health behavior change campaigns: a meta-analysis of psychological architectures and adherence factors. J Med Internet Res 2011;13(1):e17 [FREE Full text] [CrossRef] [Medline]
    37. Glasgow RE, Klesges LM, Dzewaltowski DA, Bull SS, Estabrooks P. The future of health behavior change research: what is needed to improve translation of research into health promotion practice? Ann Behav Med 2004 Feb;27(1):3-12. [CrossRef] [Medline]
    38. Cunningham M, Rapee R, Lyneham H. Feedback to a prototype self-help computer program for anxiety disorders in adolescents. Adv Ment Health 2006;5(3):216-224.
    39. Eysenbach G, Köhler C. How do consumers search for and appraise health information on the world wide web? Qualitative study using focus groups, usability tests, and in-depth interviews. BMJ 2002 Mar 9;324(7337):573-577 [FREE Full text] [Medline]
    40. Adler M, Ziglio E. Gazing Into the Oracle: The Delphi Method and Its Application to Social Policy and Public Health. Bristol, PA: Jessica Kingsley Publishers; 1996.
    41. Crutzen R, de Nooijer J, Brouwer W, Oenema A, Brug J, de Vries NK. Internet-delivered interventions aimed at adolescents: a Delphi study on dissemination and exposure. Health Educ Res 2008 Jun;23(3):427-439 [FREE Full text] [CrossRef] [Medline]
    42. Elfeddali I, Bolman C, Mesters I, Wiers RW, de Vries H. Factors underlying smoking relapse prevention: results of an international Delphi study. Health Educ Res 2010 Dec;25(6):1008-1020 [FREE Full text] [CrossRef] [Medline]
    43. De Vet E, Brug J, De Nooijer J, Dijkstra A, De Vries NK. Determinants of forward stage transitions: a Delphi study. Health Educ Res 2005 Apr;20(2):195-205 [FREE Full text] [CrossRef] [Medline]
    44. Linstone HA, Turoff M. Introduction to the Delphi method: techniques and applications. In: Linstone HA, Turoff M, editors. The Delphi Method: Techniques and Applications. Reading, MA: Addison-Wesley; 1975:3-12.
    45. FlyCatcher Internet Research. Homepage   URL: http://www.flycatcher.eu/ [accessed 2011-11-25] [WebCite Cache]
    46. Glasgow RE. eHealth evaluation and dissemination research. Am J Prev Med 2007 May;32(5 Suppl):S119-S126. [CrossRef] [Medline]
    47. Crutzen R. Optimalisering van de betrokkenheid van deelnemers en datakwaliteit in een cohortstudie [Optimizing Participants' Involvement and Data Quality in a Cohort Study]. Maastricht: Maastricht University; 2010.
    48. Galesic M, Bosjnak M. Effects of questionnaire length on participation and indicators of response quality in a Web survey. Public Opin Q 2009;73:349-360.
    49. Brouwer W, Oenema A, Raat H, Crutzen R, de Nooijer J, de Vries NK, et al. Characteristics of visitors and revisitors to an Internet-delivered computer-tailored lifestyle intervention implemented for use by the general public. Health Educ Res 2010 Aug;25(4):585-595 [FREE Full text] [CrossRef] [Medline]
    50. Bognazzini B. Nielsen Norman Group. Ask TOG: First Principles of Interaction Design   URL: http://www.asktog.com/basics/firstPrinciples.html [accessed 2011-11-25] [WebCite Cache]
    51. Eysenbach G, Powell J, Kuss O, Sa ER. Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. JAMA 2002;287(20):2691-2700 [FREE Full text] [Medline]
    52. Kreuter M, Farrell D, Olevitch L, Brennan L. Tailoring Health Messages: Customizing Communication with Computer Technology. Mahwah, NJ: Lawrence Erlbaum Associates; 2000.
    53. Mullen PM. Delphi: myths and reality. J Health Organ Manag 2003;17(1):37-52. [Medline]

    ‎
    IQR: interquartile range

    Edited by G Eysenbach; submitted 07.06.11; peer-reviewed by B Cugelman; comments to author 26.10.11; revised version received 25.11.11; accepted 27.11.11; published 14.02.12

    Copyright

    ©Francine Schneider, Liesbeth van Osch, Hein de Vries. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 14.02.2012.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.