This paper is in the following e-collection/theme issue:

Quality/Credibility of eHealth Information and Trust Issues (485) Public (e)Health, Digital Epidemiology and Public Health Informatics (636) Information Quality in Digital Media (316) Protocols for Public Health Research and Surveillance (58) Mass Media/Social Media Communication and Campaigns (446)

Published on in Vol 18, No 8 (2016): August

Making Quality Health Websites a National Public Health Priority: Toward Quality Standards

Making Quality Health Websites a National Public Health Priority: Toward Quality Standards

Making Quality Health Websites a National Public Health Priority: Toward Quality Standards

Authors of this article:

Theresa Devine1 Author Orcid Image ;   Jordan Broderick2 Author Orcid Image ;   Linda M Harris1 Author Orcid Image ;   Huijuan Wu3 Author Orcid Image ;   Sandra Williams Hilfiker3 Author Orcid Image
Article Authors Cited by (62) Tweetations (52) Metrics

    Original Paper

    1U.S. Department of Health and Human Services (HHS), Office of Disease Prevention and Promotion (ODPHP), Rockville, MD, United States

    2U.S. Department of Health and Human Services (HHS), National Vaccine Program Office, Washington, DC, United States

    3CommunicateHealth, Inc., Northampton, MA, United States

    *all authors contributed equally

    Corresponding Author:

    Huijuan Wu, MSc, MEng, PhD

    CommunicateHealth, Inc.

    26 Market St.

    Northampton, MA, 01060

    United States

    Phone: 1 413 582 0425

    Fax:1 413 517 0545

    Email: huijuan@communicatehealth.com


    Background: Most US adults have limited health literacy skills. They struggle to understand complex health information and services and to make informed health decisions. The Internet has quickly become one of the most popular places for people to search for information about their health, thereby making access to quality information on the Web a priority. However, there are no standardized criteria for evaluating Web-based health information. Every 10 years, the US Department of Health and Human Services' Office of Disease Prevention and Health Promotion (ODPHP) develops a set of measurable objectives for improving the health of the nation over the coming decade, known as Healthy People. There are two objectives in Healthy People 2020 related to website quality. The first is objective Health Communication and Health Information Technology (HC/HIT) 8.1: increase the proportion of health-related websites that meet 3 or more evaluation criteria for disclosing information that can be used to assess information reliability. The second is objective HC/HIT-8.2: increase the proportion of health-related websites that follow established usability principles.

    Objective: The ODPHP conducted a nationwide assessment of the quality of Web-based health information using the Healthy People 2020 objectives. The ODPHP aimed to establish (1) a standardized approach to defining and measuring the quality of health websites; (2) benchmarks for measurement; (3) baseline data points to capture the current status of website quality; and (4) targets to drive improvement.

    Methods: The ODPHP developed the National Quality Health Website Survey instrument to assess the quality of health-related websites. The ODPHP used this survey to review 100 top-ranked health-related websites in order to set baseline data points for these two objectives. The ODPHP then set targets to drive improvement by 2020.

    Results: This study reviewed 100 health-related websites. For objective HC/HIT-8.1, a total of 58 out of 100 (58.0%) websites met 3 or more out of 6 reliability criteria. For objective HC/HIT-8.2, a total of 42 out of 100 (42.0%) websites followed 10 or more out of 19 established usability principles. On the basis of these baseline data points, ODPHP set targets for the year 2020 that meet the minimal statistical significance—increasing objective HC/HIT-8.1 data point to 70.5% and objective HC/HIT-8.2 data point to 55.7%.

    Conclusions: This research is a critical first step in evaluating the quality of Web-based health information. The criteria proposed by ODPHP provide methods to assess website quality for professionals designing, developing, and managing health-related websites. The criteria, baseline data, and targets are valuable tools for driving quality improvement.

    J Med Internet Res 2016;18(8):e211

    doi:10.2196/jmir.5999

    Keywords

    online health information; health literacy; reliability; usability; measurement 


    Most US adults (90%) have limited health literacy skills [1], which means that many struggle to make sense of the complex information and services intended to help prevent disease and promote our health [2-5]. One promising path to improve the health literacy of the adult population is to increase the availability of evidence-based, understandable, easy-to-find sources of health information [6-8].

    While medical experts continue to play a vital role in the health decision–making process, the Internet has quickly become one of the most popular places for people to search for information about their health [9,10]. Research indicates that not only are adults of all generations searching online for health information [11,12], but they are using what they find to make health care decisions, either on behalf of themselves or a loved one [13]. Thus, improving the quality of health-related websites has the potential to improve the health literacy—and the health—of the population [14,15].

    Increasing access to quality online information is a shared priority among national and international policy makers. President Obama’s Digital Government Strategy calls for new and better ways to deliver digital information and services [16] and the World Health Organization has called for greater transparency, privacy and security, codes of conduct, and individual choice and control of health-related websites [17].

    Yet, in spite of the critical role of quality in health-related websites, there are no standardized criteria for assessing it [6,8,18]. Such criteria have been elusive since researchers called for operationalized definitions of quality in their 2002 meta-analysis of empirical website assessments [7]. Therefore, the first steps in achieving real gains in health-related website quality are to establish (1) a standardized approach to defining and measuring the quality of health websites; (2) benchmarks for measurement; (3) baseline data points to capture the current status of website quality; and (4) targets to drive improvement. This paper describes the efforts of the US Department of Health and Human Services’ Office of Disease Prevention and Health Promotion (ODPHP) to establish these four items.


    Defining and Measuring Quality Health Websites for Healthy People

    Every 10 years, ODPHP manages the development of a set of measurable objectives for improving the health of the nation over the coming decade, known as Healthy People. Two objectives (Health Communication and Health Information Technology, HC/HIT-8.1 and 8.2, see Textbox 1) in Healthy People 2020 relate to website quality: one calls for improved information reliability; the other calls for improved website usability. Both were devised and operationalized by experts in health communication and technology. Together, these objectives provide a working definition of website quality and a promising path toward overall quality improvement of Web-based health information.

    Healthy People objectives Health Communication and Health Information Technology (HC/HIT) 8.1 and 8.2.

    HC/HIT-8.1: Increase the proportion of health-related websites that meet three or more evaluation criteria for disclosing information that can be used to assess information reliability

    HC/HIT-8.2: Increase the proportion of health-related websites that follow established usability principles

    Textbox 1. Healthy People objectives Health Communication and Health Information Technology (HC/HIT) 8.1 and 8.2.

    Information Reliability

    Information reliability refers to the accuracy and credibility of website content as well as transparency in the purpose and ownership of the site [7,19]. This information can help users discern the origin and quality of Web-based content [20]. It is one of the most commonly identified indicators of website quality and has been widely referenced by public, private, and nonprofit organizations committed to improving the quality of Web-based health information such as the Medical Library Association, Health on the Net Foundation, Consumers Union, and the Agency for Healthcare Research and Quality [21-24].

    Healthy People objective HC/HIT-8.1 was first introduced in Healthy People 2010 as a developmental objective. In 2005, ODPHP convened a Technical Expert Workgroup to identify reliability criteria based on established Web standards [25,26]. The ODPHP then developed the Website Information Reliability Evaluation Instrument (Multimedia Appendix 1), which includes specific reliability requirements for each of the 6 criteria identified by the Workgroup (see Table 1).

    Table 1. Objective HC/HIT-8.1 criteria and reliability requirements.
    CriteriaReliability requirements
    IdentityName of person or organization responsible for website
    Street address for person or organization responsible for website
    Identified source of funding for website
    PurposeStatement of purpose or mission for website
    Uses and limitations of services provided
    Association with commercial products or services
    Content developmentDifferentiation of advertising from nonadvertising content
    Medical, editorial, or quality review practices or policies
    Authorship of health content (per page of health content)
    PrivacyPrivacy policy
    How personal information is protected
    User feedbackFeedback form or mechanism
    How information from users is useda
    Content updatingDate content created (per page of health content)
    Date content reviewed, updated, modified, or revised (per page of health content)
    Copyright datea

    a Optional requirements.

    Website Usability

    Usability standards tend to fall into three major categories [27]. The first category focuses on how the information is organized, commonly referred to as information architecture. The second category looks at how users navigate the information on a website, known as site design. The third category emphasizes how users interact with content on the website, referred to as content design. These three elements are commonly addressed in both federal and private guidelines on website usability [27-32].

    Usability is an important component of website quality, affecting a user’s ability to access and understand information online [33]. In fact, the design of a website is one of the most important indicators of website credibility and quality for users [34-37].

    In 2012, an expert panel was used to establish an empirical definition of usability for Healthy People 2020. Panel members were selected from academic, private, and government sectors based on their expertise in website usability and health communication. With input from the panel, ODPHP developed the Website Usability Evaluation Instrument (see Multimedia Appendix 2) to measure progress toward Healthy People objective HC/HIT-8.2. The instrument assesses the three aforementioned website usability categories, using 19 established usability principles across 59 task-based usability measures (see Table 2). The Site Design category includes 9 composites that assess basic design elements of the site, including how the site looks, how the site functions, and how a user can interact with the site. The Information Architecture category includes 7 composites that assess how the site content is organized, including navigation, grouping, and labeling. Lastly, the Content Design category includes 3 composites that assess how the content is written and formatted, and includes plain language principles. Each of the 59 measures is rated on a scale of 1 to 4 based on the level of difficulty of performing the task on the website (1 being “task failure” and 4 being “minimal problems”). An average rating score is calculated for each usability principle. The benchmark was set to require an average score of 3.5 for 10 or more of the 19 usability principles.

    Table 2. Objective HC/HIT-8.2 established usability principles and measures.
    CategoriesEstablished usability principles
    Site Design1. Use conventional interaction elements
    2. Make it obvious what is clickable and what is not
    3. Minimize vertical scrolling
    4. Ensure that the Back button behaves predictably
    5. Provide clear feedback signals for actions
    6. Ensure site is accessible for users with disabilities and uses elements of 508 compliance
    7. Provide a simplified user experience
    8. Incorporate multimedia
    9. Offer a functional home page
    Information Architecture10. Present a clear visual hierarchy
    11. Provide easy search functionality
    12. Clearly label content categories
    13. Make pages easy to skim or scan
    14. Make elements on the page easy to read
    15. Visually group related topics
    16. Make sure text and background colors contrast
    Content Design17. Focus the writing on audience and purpose
    18. Use the users’ language; minimize jargon and technical terms
    19. Allow for interaction with the content

    With Healthy People objectives HC/HIT-8.1 and 8.2, ODPHP established standardized criteria to define and measure the quality of health-related websites and set benchmarks for measurement. Next, the criteria were applied to a sample of health-related websites in order to identify baseline data points, which are required for all Healthy People measurable objectives. The reliability (objective HC/HIT-8.1) and usability (objective HC/HIT-8.2) instruments were combined into a single instrument: the National Quality Health Website Survey. Targets were set to drive improvement by the year 2020.

    Sampling

    The ODPHP identified the 100 top-ranked health-related websites (see Multimedia Appendix 3) for 3 months (August-October 2014) from the Alexa Top Sites pool—health category [38]. The data were collected from Alexa on October 14, 2014. Alexa’s traffic ranks are based on the traffic data provided by users in Alexa’s global data panel over a rolling 3-month period. A site’s ranking is based on a combined measure of unique visitors and page views [39]. Websites were considered health related if they had at least three items of health information as it is broadly defined by the e-Health Code of Ethics (see Textbox 2) [40]. Duplicated websites were consolidated with the exception of microsites within the National Institutes of Health such as PubMed and MedlinePlus. A number of websites were excluded from the sample based on the exclusion criteria in Textbox 3.

    Criteria for selecting health websites (from e-Health Code of Ethics).
    • Health information includes information for staying well, preventing and managing disease, and making other decisions related to health and health care.
    • It includes information for making decisions about health products and health services.
    • It may be in the form of data, text, audio, and/or video.
    • It may involve enhancements through programming and interactivity.
    • Health products include drugs, medical devices, and other goods used to diagnose and treat illnesses or injuries or to maintain health. Health products include both drugs and medical devices subject to regulatory approval by agencies such as the US Food and Drug Administration or UK Medicines Control Agency and vitamin, herbal, or other nutritional supplements and other products not subject to such regulatory oversight.
    • Health services include specific, personal medical care or advice; management of medical records; communication between health care providers and/or patients and health plans or insurers or health care facilities regarding treatment decisions, claims, billing for services, and so on; and other services provided to support health care.
    • Health services also include listservs, bulletin boards, chat rooms, and other online venues for the exchange of health information.
    • Like health information, health services may be in the form of data, text, audio, and/or video and may involve enhancements through programming and interactivity.
    Textbox 2. Criteria for selecting health websites (from e-Health Code of Ethics).
    Exclusion criteria.
    • Sites that are not about human beings
    • Sites that are owned or maintained in a foreign country
    • Sites that are specifically for health industry professional development, listing job postings for health professionals or research grants available for health researchers
    • Sites that are designed only to introduce, sell, or support specific medical commercial products or technology solutions for the health or medical industry
    • Sites that provide platforms for laboratory services
    • Sites accessible only to members or paying subscribers who must enter an identifying log-in name and password
    • Sites about beauty or cosmetic products or hairstyles
    • Sites about health or medical education programs
    • Sites about fitness industry professional development or gym memberships
    • Sites providing pharmacy price comparison information
    • Online forums or groups, or other social media platforms for informal discussions regarding health
    Textbox 3. Exclusion criteria.

    The final sample included websites sponsored by three types of organizations: 48 out of 100 websites (48.0%) were for profit, 36 out of 100 (36.0%) were nonprofit, and 16 out of 100 (16.0%) were government.

    Interrater Reliability

    A senior usability researcher and a research associate reviewed the sample websites, following a reviewer training process to reach a certain level of interrater reliability (IRR). The team used Altman’s Kappa Benchmark Scale (see Table 3) [41].

    Table 3. Altman’s Kappa Benchmark Scale.
    Kappa statisticStrength of agreement
    <.20Poor
    .21-.40Fair
    .41-.60Moderate
    .61-.80Good
    .81-1.00Very good

    Objective HC/HIT-8.1 criteria are primarily composed of yes or no questions. Cohen’s kappa was used to measure the agreement for nominal scales. A benchmark kappa score of .80 (a score generally accepted as demonstrating a sufficient degree of IRR) was used for objective HC/HIT-8.1 [41]. The criteria for objective HC/HIT-8.2 are scored on a 4-point scale, making it more difficult to reach a perfect IRR. A benchmark kappa score of .61 was used for objective HC/HIT-8.2. Interclass correlation (ICC) was used for assessing ordinal and interval scales. To ensure IRR, both reviewers assessed the same 6 websites during the initial training process. The IRR was calculated for each assessment; discrepancies were identified and resolved between the 2 reviewers until the IRR scores met the benchmark kappa scores.

    After the initial training process, the reviewers randomly selected 4 additional websites from the sample to measure IRR scores. The IRR scores for the Website Information Reliability Evaluation Instrument (kappa .83) and the Website Usability Evaluation Instrument (ICC .76) both met the benchmarks.

    Website Review Process

    After ensuring IRR, the reviewers divided the remaining 90 websites into 2 equal groups. Each reviewer assessed an equal number of websites.

    To score each item in the Website Information Reliability Evaluation Instrument, reviewers began by randomly selecting 3 pages from different sections of the site. Each of the items in the instrument was scored based on a review of at least 3 different pages and no individual page was reviewed more than once. For items in the instrument that refer to a specific page or feature (eg, home page–related items and search function items), the review also included those specific pages or features. As each review progressed, the reviewer revised the scores for previously scored items as needed. In general, reviewers examined about 150 pages on each website.


    Baseline data points for 2015 were calculated for both Healthy People 2020 website quality objectives.

    Information Reliability Baseline

    For Healthy People objective HC/HIT-8.1, a total of 58 out of 100 health-related websites (58.0%) met 3 or more of the 6 information reliability criteria. Only 2 out of 100 websites (2.0%) met all the criteria, and 1 out of 100 websites (1.0%) met none of the 6 criteria. See Figure 1.

    Figure 2 shows the percentage of health-related websites in compliance with specific criteria. User Feedback (90.0%) had the highest percentage in compliance, followed by Privacy (83.0%). Only 4 out of 100 websites (4.0%) met the criterion for Content Updating.

    Table 4 presents the percentage of websites in compliance by criterion and by information reliability requirements associated with each criterion.

    Table 4. Estimated percentages of websites in compliance, by information reliability criterion and required disclosure elements.
    Criterion and required disclosure elementsNCountPercent (%)SEa (%)Lower bound 95% CI (%)Upper bound 95% CI (%)
    Identity1003737.04.8327.546.5

    		Name1009393.02.5588.098.0

    		Street address1008383.03.7675.690.4

    		Funding sources1004444.04.9634.353.7
    Purpose1005252.05.0042.261.8

    		Purpose or mission1007979.04.0771.087.0

    		Uses and limitations1008282.03.8474.589.5

    		Association with commercial products1007171.04.5462.179.9
    Content Development1001515.03.578.022.0

    		Identify advertising content602846.76.4434.059.3

    		Describe editorial policy1003939.04.8829.448.6

    		Authorship1003838.04.8528.547.5
    Privacy1008383.03.7675.690.4

    		Privacy policy1009696.01.9692.299.8

    		Describe protection of personal information1008383.03.7675.690.4
    User Feedback1009090.03.0084.195.9

    		Feedback mechanism1009090.03.0084.195.9
    Content Updating10044.01.960.27.8

    		Display date created1002525.04.3316.533.5

    		Display date reviewed or updated1002828.04.4919.236.8

    a SE: standard error.

    ‎
    Figure 1. Percentage of websites in compliance, with the number of reliability criteria.
    View this figure
    ‎
    Figure 2. Percentage of websites in compliance, by specific reliability criteria.
    View this figure
    Table 5. Estimated percentages of websites in compliance, by criterion and established usability principles.
    Criterion and established usability principlesNumber (n=100)Percent (%)SEa (%)Lower bound 95% CI (%)Upper bound 95% CI (%)
    Site Design

    		1. Use conventional interaction elements7878.04.1469.986.1

    		2. Make it obvious what is clickable and what is not6868.04.6658.977.1

    		3. Minimize vertical scrolling2424.04.2715.632.4

    		4. Ensure that the Back button behaves predictably100100.00.00100.0100.0

    		5. Provide clear feedback signals for actions3939.44.9129.849.0

    		6. Ensure site is accessible for users with disabilities and uses elements of 508 compliance66.02.371.310.7

    		7. Provide a simplified user experience3030.04.5821.039.0

    		8. Incorporate multimedia7070.04.5861.079.0

    		9. Offer a functional home page3030.04.5821.039.0
    Information Architecture

    		10. Present a clear visual hierarchy4242.04.9432.351.7

    		11. Provide easy search functionality1717.03.769.624.4

    		12. Clearly label content categories2525.04.3316.533.5

    		13. Make pages easy to skim or scan4545.04.9735.254.8

    		14. Make elements on the page easy to read7272.04.4963.280.8

    		15. Visually group related topics4545.04.9735.254.8

    		16. Make sure text and background colors contrast7474.04.3965.482.6
    Content Design

    		17. Focus the writing on audience and purpose3030.04.5821.039.0

    		18. Use the users’ language; minimize jargon and technical terms3232.04.6622.941.1

    		19. Allow for interaction with the content1919.03.9211.326.7

    a SE: standard error.

    ‎
    Figure 3. Percentage of websites in compliance, by the number of usability principles.
    View this figure

    Website Usability Baseline

    For Healthy People objective HC/HIT-8.2, a total of 42 out of 100 health-related websites (42.0%) met 10 or more out of 19 established usability principles. Figure 3 shows the distribution of website compliance by the number of usability principles met. All websites met at least one usability principle. Almost half of the websites (47.0%) met between 9 and 11 principles. None of the websites met 15 or more of the established usability principles. See Table 5 for the percentage of websites in compliance broken down by the 19 usability principles.


    Baseline Data Points

    The 2015 review of health-related websites identified 2 baseline data points. For Healthy People 2020 objective HC/HIT-8.1, a total of 58 out of 100 health-related websites (58.0%) met 3 or more out of 6 reliability criteria. For Healthy People 2020 objective HC/HIT-8.2, a total of 42 out of 100 health-related websites (42.0%) followed 10 or more out of 19 established usability principles.

    This research revealed significant shortcomings in the quality of today’s Web-based health information landscape, particularly in disclosing sources of funding and authorship, clearly differentiating between advertisements and original content, complying with universal accessibility guidelines, providing simple search and print functionality, and minimizing scientific and technical jargon.

    2020 Targets

    The ODPHP set the following targets for 2020 that meet the minimal statistical significance. For Healthy People 2020 objective HC/HIT-8.1, 70.5% of health-related websites will meet 3 or more out of 6 reliability criteria. For Healthy People 2020 objective HC/HIT-8.2, 55.7% of health-related websites will follow 10 or more out of 19 established usability principles.

    Improving the quality of health-related websites is critical to national efforts to promote health literacy and shared decision making. Until now, there has been no standardized approach to defining and measuring the quality of Web-based health information. With Healthy People 2020 objectives HC/HIT-8.1 and 8.2, ODPHP has developed and validated such an approach, and established baseline data points and national benchmarks to track progress over time.

    The 2015 study confirmed that there is, indeed, room for improvement. The ODPHP’s research revealed significant shortcomings in the quality of today’s Web-based health information landscape, particularly in the following areas:

    • Disclosing sources of funding and authorship
    • Clearly differentiating between advertisements and original content
    • Complying with universal accessibility guidelines (eg, Section 508 of the Amendment to the Rehabilitation Act of 1973)
    • Providing simple search and print functionality
    • Minimizing scientific and technical jargon

    Limitations

    This study has several limitations. First, the samples of this study were selected from Alexa Top Sites—health category. The research team had no control over the quality of the website rankings performed by Alexa.

    Additionally, some of the usability principles might change over time. For example, in this study, only 24 out of 100 websites (24.0%) followed the principle of minimizing vertical scrolling. However, with the proliferation of mobile and responsive design technology, users are becoming more accustomed to navigating websites by scrolling. Minimizing vertical scrolling may not remain a usability principle in the future.

    Finally, the survey instrument is somewhat subjective, especially for objective HC/HIT-8.2. Although the research team controlled the reliability by measuring the IRR for several websites in the sample, there was still variation across reviewers that may affect the results.

    Conclusions

    The quality and accessibility of Web-based health information is a key factor in improving access to health services and facilitating informed health decision making. The criteria proposed by ODPHP provide methods to assess the quality of health-related websites and provide baseline data and targets to drive quality improvement. In addition to having implications for website developers and policy makers, this work also points to the need for consumer education related to the quality of Web-based health information. Such education efforts are critical in a time when nearly 3 in 4 Internet users are looking for health information online [42].

    To promote increased quality of health-related websites, ODPHP updated and published Health Literacy Online: A Guide to Simplifying the User Experience, second edition [43]. Health Literacy Online is based on literature related to cognitive processing and online behavior and on usability research with more than 800 participants. It features actionable information that website owners, content writers, designers, and developers can use to create quality health websites. The recommendations in Health Literacy Online provide a clear road map for achieving the Healthy People 2020 objectives to increase the proportion of quality health-related websites (Objective HC/HIT-8). The ODPHP is working with other federal agencies to adopt the principles in Health Literacy Online and is presenting these strategies for improving health websites at national conferences.

    Acknowledgments

    This research was funded by the US Department of Health and Human Services' Office of Disease Prevention and Health Promotion (ODPHP). The ODPHP, the Centers for Disease Control and Prevention’s Office of the Associate Director for Communication, and the Office of the National Coordinator for Health Information Technology co-lead the Healthy People 2020 Health Communication and Health Information Technology topic area. The ODPHP contracted with CommunicateHealth, Inc., to conduct the data collection and analysis discussed in this paper.

    Conflicts of Interest

    None declared.

    ‎

    Multimedia Appendix 1

    Website Information Reliability Evaluation Instrument.

    PDF File (Adobe PDF File), 83KB
    ‎

    Multimedia Appendix 2

    Website Usability Evaluation Instrument.

    PDF File (Adobe PDF File), 74KB
    ‎

    Multimedia Appendix 3

    Sample Websites.

    PDF File (Adobe PDF File), 103KB
    1. Kutner M, Greenberg E, Jin Y, Paulsen C. NCES 2006-483. Washington, DC: U.S. Department of Education, National Center for Education Statistics; 2006. The Health Literacy of America's Adults: Results from the 2003 National Assessment of Adult Literacy   URL: http://files.eric.ed.gov/fulltext/ED493284.pdf [accessed 2016-07-18] [WebCite Cache]
    2. Cline RJ, Haynes KM. Consumer health information seeking on the Internet: The state of the art. Health Educ Res 2001;16(6):671-692 [FREE Full text] [Medline]
    3. Keselman A, Browne AC, Kaufman DR. Consumer health information seeking as hypothesis testing. J Am Med Inform Assoc 2008;15(4):484-495 [FREE Full text] [CrossRef] [Medline]
    4. Diviani N, van den Putte B, Giani S, van Weert JC. Low health literacy and evaluation of online health information: a systematic review of the literature. J Med Internet Res 2015;17(5):e112 [FREE Full text] [CrossRef] [Medline]
    5. McCloud RF, Okechukwu CA, Sorensen G, Viswanath K. Beyond access: barriers to internet health information seeking among the urban poor. J Am Med Inform Assoc 2016 May 20:- Epub ahead of print. [CrossRef] [Medline]
    6. Chung M, Oden RP, Joyner BL, Sims A, Moon RY. Safe infant sleep recommendations on the Internet: let's Google it. J Pediatr 2012;161(6):1080-1084 [FREE Full text] [CrossRef] [Medline]
    7. Eysenbach G, Powell J, Kuss O, Sa ER. Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. JAMA 2002;287(20):2691-2700. [Medline]
    8. Neumark Y, Flum L, Lopez-Quintero C, Shtarkshall R. Quality of online health information about oral contraceptives from Hebrew-language websites. Isr J Health Policy Res 2012;1(1):38 [FREE Full text] [CrossRef] [Medline]
    9. Fox S, Rainie L. Pew Research Center. 2002. Vital decisions: How Internet users decide what information to trust when they or their loved ones are sick   URL: http://www.pewinternet.org/files/old-media//Files/Reports/2002/PIP_Vital_Decisions_May2002.pdf.pdf [accessed 2016-04-21] [WebCite Cache]
    10. Fox S, Jones S. Pew Research Center. 2009. The social life of health information   URL: http://www.pewinternet.org/2009/06/11/the-social-life-of-health-information/ [accessed 2016-04-21] [WebCite Cache]
    11. Neter E, Brainin E. eHealth literacy: extending the digital divide to the realm of health information. J Med Internet Res 2012;14(1):e19 [FREE Full text] [CrossRef] [Medline]
    12. Tennant B, Stellefson M, Dodd V, Chaney B, Chaney D, Paige S, et al. eHealth literacy and Web 2.0 health information seeking behaviors among baby boomers and older adults. J Med Internet Res 2015;17(3):e70 [FREE Full text] [CrossRef] [Medline]
    13. Fox S, Duggan M. Pew Research Center. 2013. Health online   URL: http://www.pewinternet.org/files/old-media/Files/Reports/PIP_HealthOnline.pdf [accessed 2016-04-21] [WebCite Cache]
    14. Fiksdal AS, Kumbamu A, Jadhav AS, Cocos C, Nelsen LA, Pathak J, et al. Evaluating the process of online health information searching: a qualitative approach to exploring consumer perspectives. J Med Internet Res 2014;16(10):e224 [FREE Full text] [CrossRef] [Medline]
    15. Coulter A, Ellins J. Effectiveness of strategies for informing, educating, and involving patients. BMJ 2007;335(7609):24-27 [FREE Full text] [CrossRef] [Medline]
    16. The White House. Whitehouse.gov. 2012. Presidential Memorandum -- Building a 21st Century Digital Government   URL: https:/​/www.​whitehouse.gov/​the-press-office/​2012/​05/​23/​presidential-memorandum-building-21st-century-digital-government [accessed 2016-04-21] [WebCite Cache]
    17. WHO. 2014. The health Internet: governing principles for a .health top-level domain   URL: http://www.who.int/ehealth/programmes/governance/en/index3.html [accessed 2016-04-21] [WebCite Cache]
    18. Kitchens B, Harle CA, Li S. Quality of health-related online search results. Decision Support Systems 2014;57:454-462. [CrossRef]
    19. Kim P, Eng TR, Deering MJ, Maxfield A. Published criteria for evaluating health related web sites: review. BMJ 1999;318(7184):647-649 [FREE Full text] [Medline]
    20. Van der Vaart R, Drossaert CH, de Heus M, Taal E, van de Laar MA. Measuring actual eHealth literacy among patients with rheumatic diseases: a qualitative analysis of problems encountered using Health 1.0 and Health 2.0 applications. J Med Internet Res 2013;15(2):e27 [FREE Full text] [CrossRef] [Medline]
    21. Medical Library Association. Find Good Health Information   URL: http://www.mlanet.org/resources/userguide.html [accessed 2016-04-21] [WebCite Cache]
    22. Health on the Net Foundation. HON. 2014. The commitment to reliable health and medical information on the internet   URL: http://www.hon.ch/HONcode/Patients/Visitor/visitor.html [accessed 2016-04-21] [WebCite Cache]
    23. Consumers Union. Mission   URL: http://consumersunion.org/about/mission/ [accessed 2016-04-21] [WebCite Cache]
    24. The Agency for Healthcare Research and Quality. AHRQ. 2006. Guidelines for Ensuring the Quality of Information Disseminated to the Public   URL: http://www.ahrq.gov/research/publications/ahrq-info-quality-gdlns.html [accessed 2016-04-21] [WebCite Cache]
    25. URAC. Tspwebs. 2001. Health Web Site Standards, Version 1.0   URL: http://webs.tspwebs.net/URAC%20Standards.pdf [accessed 2016-04-27] [WebCite Cache]
    26. Health Improvement Institute. 2005. Health Website Ratings Instrument, Version 2.0   URL: https:/​/dl.​dropboxusercontent.com/​content_link/​XcTWVhlqdUN8rYGqTRDc4uLTG2NOHy54swzmXbVvrgzm619tqqYNeQSgfsgMMFGi/​file [accessed 2016-04-27] [WebCite Cache]
    27. The U.S. Department of Health Human Services. Research-Based Web Design & Usability Guidelines (Enlarge/E). Washington, DC: U.S. Government Printing Office; 2006.
    28. Travis D. Userfocus. 2014. 247 web usability guidelines   URL: http://www.userfocus.co.uk/resources/guidelines.html [accessed 2016-04-21] [WebCite Cache]
    29. UsabilityNet. 2006. Design guidelines for the Web   URL: http://www.usabilitynet.org/tools/webdesign.htm [accessed 2016-04-21] [WebCite Cache]
    30. Sharpened Productions. 2015. 10 Rules of Web Design   URL: http://sharpened.com/web_design_rules [accessed 2016-04-21] [WebCite Cache]
    31. Krug S. Don’t Make Me Think, Revisited: A Common Sense Approach to Web Usability. 3rd edition. Berkeley, CA: New Riders; 2014.
    32. Redish J. Letting Go of the Words: Writing Web Content that Works (Interactive Technologies). 2nd edition. Burlington, MA: Morgan Kaufmann; 2012.
    33. Lee K, Hoti K, Hughes JD, Emmerton LM. Consumer Use of “Dr Google”: A Survey on Health Information-Seeking Behaviors and Navigational Needs. J Med Internet Res 2015;17(12):e288 [FREE Full text] [CrossRef] [Medline]
    34. Weymann N, Härter M, Dirmaier J. Quality of online information on type 2 diabetes: a cross-sectional study. Health Promot Int 2015;30(4):821-831. [CrossRef] [Medline]
    35. Stanford J, Tauber ER, Fogg BJ, Marable L. Ebusiness Forum. 2002. Experts vs. Online Consumers: A Comparative Credibility Study of Health and Finance Web Sites   URL: http://www.ebusinessforum.gr/old/content/downloads/comparativeCredibilityStudy.pdf [accessed 2016-04-15] [WebCite Cache]
    36. Sillence E, Briggs P, Harris PR, Fishwick L. How do patients evaluate and make use of online health information? Soc Sci Med 2007;64(9):1853-1862. [CrossRef] [Medline]
    37. Agree EM, King AC, Castro CM, Wiley A, Borzekowski DL. “It’s Got to Be on This Page”: Age and Cognitive Style in a Study of Online Health Information Seeking. J Med Internet Res 2015;17(3):e79 [FREE Full text] [CrossRef] [Medline]
    38. Alexa. 2014. Top Sites -- health category   URL: http://www.alexa.com/topsites/category/Top/Health [accessed 2014-10-15] [WebCite Cache]
    39. Alexa. 2016. How are Alexa’s traffic rankings determined?   URL: https://support.alexa.com/hc/en-us/articles/200449744-How-are-Alexa-s-traffic-rankings-determined- [accessed 2016-06-10] [WebCite Cache]
    40. Rippen H, Risk A. e-Health Code of Ethics (May 24). J Med Internet Res 2000;2(2):E9 [FREE Full text] [CrossRef] [Medline]
    41. Altman D. Practical Statistics for Medical Research. Boca Raton, FL: CRC Press; 1990.
    42. Pew Research Center. 2013. Health Fact Sheet   URL: http://www.pewinternet.org/fact-sheets/health-fact-sheet/ [accessed 2016-04-21] [WebCite Cache]
    43. U.S. Department of Health and Human Services, Office of Disease Prevention and Health Promotion. Health.gov. 2015. Health Literacy Online: 2nd Edition   URL: http://health.gov/healthliteracyonline/ [accessed 2016-06-10] [WebCite Cache]

    ‎
    HC/HIT: Health Communication and Health Information Technology
    ICC: interclass correlation
    IRR: interrater reliability
    ODPHP: Office of Disease Prevention and Health Promotion

    Edited by G Eysenbach; submitted 19.05.16; peer-reviewed by K Summers, G Reddish; comments to author 08.06.16; revised version received 22.06.16; accepted 23.06.16; published 02.08.16

    Copyright

    ©Theresa Devine, Jordan Broderick, Linda M Harris, Huijuan Wu, Sandra Williams Hilfiker. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 02.08.2016.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.