This paper is in the following e-collection/theme issue:

eHealth Literacy / Digital Literacy (351) Quality/Credibility of eHealth Information and Trust Issues (485) Medicine 2.0'14 Maui/Malaga (Full Paper of Conference Presentation) (20) Medicine 2.0'14 (Maui/Malaga) (18)

Published on in Vol 17, No 5 (2015): May

Low Health Literacy and Evaluation of Online Health Information: A Systematic Review of the Literature

Low Health Literacy and Evaluation of Online Health Information: A Systematic Review of the Literature

Low Health Literacy and Evaluation of Online Health Information: A Systematic Review of the Literature

Authors of this article:

Nicola Diviani1 Author Orcid Image ;   Bas van den Putte1, 2 Author Orcid Image ;   Stefano Giani3 Author Orcid Image ;   Julia CM van Weert1 Author Orcid Image
Article Authors Cited by (330) Tweetations (3) Metrics

    Review

    1Amsterdam School of Communication Research / ASCoR, Department of Communication Science, University of Amsterdam, Amsterdam, Netherlands

    2Trimbos Institute, Netherlands Institute for Mental Health and Addiction, Utrecht, Netherlands

    3University Library, University of Amsterdam, Amsterdam, Netherlands

    Corresponding Author:

    Nicola Diviani, PhD

    Amsterdam School of Communication Research / ASCoR

    Department of Communication Science

    University of Amsterdam

    P.O. Box 15791

    Amsterdam, 1001 NG

    Netherlands

    Phone: 31 6 15254105

    Fax:31 20 5253681

    Email: N.Diviani@uva.nl


    Background: Recent years have witnessed a dramatic increase in consumer online health information seeking. The quality of online health information, however, remains questionable. The issue of information evaluation has become a hot topic, leading to the development of guidelines and checklists to design high-quality online health information. However, little attention has been devoted to how consumers, in particular people with low health literacy, evaluate online health information.

    Objective: The main aim of this study was to review existing evidence on the association between low health literacy and (1) people’s ability to evaluate online health information, (2) perceived quality of online health information, (3) trust in online health information, and (4) use of evaluation criteria for online health information.

    Methods: Five academic databases (MEDLINE, PsycINFO, Web of Science, CINAHL, and Communication and Mass-media Complete) were systematically searched. We included peer-reviewed publications investigating differences in the evaluation of online information between people with different health literacy levels.

    Results: After abstract and full-text screening, 38 articles were included in the review. Only four studies investigated the specific role of low health literacy in the evaluation of online health information. The other studies examined the association between educational level or other skills-based proxies for health literacy, such as general literacy, and outcomes. Results indicate that low health literacy (and related skills) are negatively related to the ability to evaluate online health information and trust in online health information. Evidence on the association with perceived quality of online health information and use of evaluation criteria is inconclusive.

    Conclusions: The findings indicate that low health literacy (and related skills) play a role in the evaluation of online health information. This topic is therefore worth more scholarly attention. Based on the results of this review, future research in this field should (1) specifically focus on health literacy, (2) devote more attention to the identification of the different criteria people use to evaluate online health information, (3) develop shared definitions and measures for the most commonly used outcomes in the field of evaluation of online health information, and (4) assess the relationship between the different evaluative dimensions and the role played by health literacy in shaping their interplay.

    J Med Internet Res 2015;17(5):e112

    doi:10.2196/jmir.4018

    Keywords

    health information seeking; online health information; information quality; health literacy 


    Introduction

    Recent years have witnessed a dramatic increase in consumer online health information seeking [1]. Online health information deserves particular attention because studies on the content of health-related websites have highlighted inaccuracies that raise concerns about the quality of the online health information encountered by consumers [2-6]. The limited accuracy of information is often the result of one of the distinctive features of the Internet, that is, that anyone can potentially publish health-related information. Online health information seeking thus poses several major challenges to health information users, as it requires them to undertake an active role in evaluating a vast amount of often unverified health information on the Internet [7]. As a result, people experiencing difficulties evaluating online health information may be exposed to wrong or incomplete information, which has been shown to be related to adverse health outcomes, such as low participation in screening programs or low adherence to treatments [8]. Evidently, more attention needs to be given to the issue of quality of online health information and in particular to people’s ability to evaluate it [2-6].

    Several guidelines and checklists to improve the quality of online health information have been developed, for example by the Standford Persuasive Tech Lab, the Health On the Net Foundation (HONcode), Web Médica Acreditada, and Centrale santé (Netscoring criteria) (see Kim et al [9] for a summary view). These tools can be useful for Web designers and providers of health information to develop high-quality health websites. At the same time, the guidelines could be used by users as evaluation criteria to assess online health information. However, these criteria are likely known and adopted only by specific segments of the population, resulting in disparities in people’s ability to evaluate online health information. The knowledge gap hypothesis, for instance, states that as a result of increasing mass media exposure, individuals in the higher socioeconomic strata of society tend to acquire information faster than people in lower ones. So the gap in knowledge between the two tends to increase rather than decrease [10]. It is likely that traditionally disadvantaged groups—such as those with lower education or lower health literacy—will be the ones at higher risk for disparities in this context [11].

    Among the determinants of health disparities, people’s health literacy has been proven to play a crucial role in the context of health information seeking. Health literacy has originally been defined as “the cognitive and social skills which determine the motivation and ability of individuals to gain access, to understand, and use information in ways which promote and maintain good health” [12]. A closer look at the different conceptualizations of health literacy proposed in the last years (eg, [12-19]) shows that almost all of them explicitly or implicitly include people’s ability to deal with (ie, obtain, process, evaluate, and use) health information among the skills a person should possess in order to be considered health literate. Several studies have provided evidence of differences in how people with different levels of health literacy seek, find, understand, and use online health information. For instance, low health literate people have been shown to search less for health information, choose different information sources, and have a poorer ability to interpret medication labels or health messages [20-23]. In contrast, little attention has been devoted to how consumers—and in particular those with low health literacy—evaluate online health information [24]. Additionally, to date, no studies have systematically summarized existing evidence on the role of health literacy in the context of the evaluation of online health information.

    The main objective of this study was to address this research gap and provide a comprehensive description of how low health literacy impacts people’s evaluation of online health information. This translates to four distinct research questions aimed at understanding whether and how people’s health literacy is related to their ability to evaluate online health information (RQ1), perceived quality of online health information (RQ2), people’s trust in the Internet as a source of online health information (RQ3), and the use of evaluation criteria for online health information (RQ4).

    The rationale behind RQ1 is the fact that a relationship between health literacy and ability to evaluate the quality of health information has been explicitly or implicitly suggested by several health literacy conceptualizations [12-19] but has not been systematically verified so far.

    Information evaluation, however, does not depend only on the characteristics of the audience (in our case, people’s ability to evaluate online health information). As acknowledged already at the very beginning of scholarly interest in the field of credibility, characteristics of the message and the source can play a role as well [25-28]. This is the reason why RQ2 is about the relationship between health literacy and perceived quality of online health information (message level). Perceived information quality is a multifaceted concept encompassing several dimensions, which in turn can be grouped in different categories [29,30]. For the purposes of this review, we will focus on the dimensions of perceived reliability and accuracy, which Wang and Strong [31] have defined as intrinsic information quality.

    It has been found that perceived information quality does not necessarily imply intention to rely on it [32]. Someone could, for instance, perceive a message as being of high quality but not trust the source because of external factors (such as previous negative experiences) and thus decide not to act on the information. Therefore, RQ3 concerns overall trust in the Internet as a source of health information. In the context of computer-mediated communication, trust has been defined in terms of dependability and is a subjective judgment about whether a person (or, in the case of online health information, a digital object) is worth being relied on [32,33].

    Last, as past research in other fields has shown that people use several different criteria when evaluating online information, including relying on formal (eg, color of the webpage) or contextual (eg, position in Google search results) aspects of the website, or evaluating the information based on previous knowledge [34], RQ4 is aimed at understanding whether health literacy plays a role in the choice and use of these evaluation criteria.

    Answers to these four questions will provide us with a new and more comprehensive understanding of the role played by low health literacy in the evaluation of online information. At the same time, the results of the review will allow us to identify possible areas where consumer education could have an impact on improving low health literate people’s ability to correctly evaluate online health information.


    Data Sources and Search Strategy

    During the third week of January 2014, five academic databases from different relevant disciplines (Medline, PsycINFO, CINAHL, Web of Science, and Communication and Mass Media Complete) were systematically searched for peer-reviewed literature describing consumers’ evaluation of online health information. No time limits were set because the topic of online health information seeking is relatively recent. Search terms used included a combination of Medical Subject Headings (MeSH) terms and free terms covering the four domains, “online information”, “health”, “evaluation”, and “health literacy”, combined with the Boolean operator AND (see Multimedia Appendix 1). A preliminary search showed that the number of articles explicitly mentioning health literacy was limited, so the search was modified to include some of its most common proxies or indicators. Since health literacy has been defined as a set of skills [12], only skills-based indicators (eg, educational attainment, reading ability, or general literacy) were included in the search. These indicators can be considered as proxies of health literacy because of their conceptual similarities (eg, they all refer to teachable skills) and the existence of a direct link between the concepts. Other common, mainly sociodemographic, indicators (eg, age, income, or ethnicity) were excluded because of the more complex nature of their relationship with health literacy. The original search strategy was developed for PsycINFO and subsequently adapted to the peculiarities and requirements of the other databases. These terms were included in the original search, allowing us to refine it. References cited in included articles were reviewed manually, and a Google Scholar and Web of Science search for recent articles citing the ones included in the review was performed in order to identify further additional articles relevant to this review (snowball method).

    Study Inclusion/Exclusion Criteria

    A two-phase screening process was conducted. After de-duplication, the first author (ND) screened titles and abstracts of all retrieved articles in order to identify possible relevant articles (first phase). Abstracts were selected for full text screening (second phase) if they (1) were written in English, (2) reported original results, qualitative or quantitative, (3) studied consumer online health information, (4) mentioned evaluation of the information by consumers/patients, and (5) had been conducted in a low health literacy population (or in a sample of the above-described proxies of low health literacy) OR subgroup analyses were conducted in a sample of low health literate people (or proxies of low health literacy). Excluded were non-empirical articles (such as reviews, commentaries, or editorials), articles describing empirical studies conducted among health care providers, content analyses of websites, quality assessments of websites, and articles reporting on research conducted in samples that were not explicitly described as low health literate (or proxies) and did not present subgroup analyses for the low health literacy (or proxies) group. No selection based on the country where the study was conducted was made, and the same inclusion criteria were used for each country. In order to estimate reliability of the screening process, 10% of the abstracts and all the selected full texts were independently assessed by a second researcher. Initial intercoder agreement (Cohen’s kappa >.70 for both title/abstract and full-text screening) was substantial [35], and all disagreements regarding full texts were resolved during consensus meetings that were held on a regular basis during the whole screening process.

    Data Extraction

    Overview

    Besides the formal characteristics of the included papers—author(s), publication date, study design, study population, and sampling—data were extracted from all articles on the following aspects, which were deemed relevant to answer our research questions.

    Predictors

    Data were extracted about the predictors used in the study and the measures used to assess them. The main predictor of interest was health literacy, which can be assessed using different tools, for example, the Rapid Estimate of Adult Literacy in Medicine (REALM) [36], the Short Test of Functional Health Literacy in Adults (S-TOFHLA) [37,38], or the Newest Vital Signs (NVS) [39]. As mentioned earlier, however, included studies could also describe differences in one of the outcomes of interest related to differences in education or other skills-based proxies for health literacy [40]. Each study could address one or more predictors.

    Outcomes

    Data were extracted on the outcomes addressed in the studies and the measures used to assess them. The four outcomes of interest were (1) ability to evaluate online health information, (2) perceived quality of online health information, (3) trust in the Internet as a source of online health information, and (4) use of evaluation criteria for online health information. Each study could address one or more outcomes.

    Association Between Predictors and Outcomes

    All qualitative or quantitative evidence of (or lack of) an association between one of the predictors and one of the four outcomes of interest were extracted.

    Data Synthesis

    Given the heterogeneity of study designs, samples, predictors, and outcome measures in our articles pool, results could not be synthetized quantitatively using meta-analytic techniques. The findings were therefore synthetized narratively and structured according to the different outcomes under investigation.


    Included Studies

    The initial search resulted in 17,507 articles. In the process of reviewing articles identified through the initial searches, 3 additional articles were identified through cited references, bringing the total to 17,510 articles. After duplicates were removed, the remaining number of articles was 13,632, of which 13,378 were discarded after reviewing titles and abstract. An additional 216 articles were discarded after reviewing the articles in their entirety, resulting in 38 articles [24,41-77]. The whole process is illustrated in detail by the flow diagram in Figure 1.

    ‎
    Figure 1. Flow diagram of the screening process.
    View this figure

    Characteristics of Included Studies

    The basic characteristics of the final pool of articles included in the systematic review are described in Table 1 [41-77]. The 38 studies were published between 2001 and 2013. Most of them were conducted in North America (24/38; 63%), five were conducted in Europe (13%), four (11%) in Asia, four (11%) in Australia, and one in Africa (3%). Study populations varied widely, ranging from the general population to specific patient groups, and so did sample sizes, ranging from N=8 up to N=8586. All studies were non-experimental, with the vast majority being cross-sectional surveys (35/38; 92%), and the remaining were qualitative studies (1 focus group study [60] and 2 qualitative observational studies [45,50]).

    According to commonly used approaches for rating the quality of evidence in systematic reviews (eg, Grades of Recommendation, Assessment, Development and Evaluation [GRADE]; see [78]), the quality level of the evidence has to be considered low because all the studies included in this review are non-interventional in nature.

    Table 1. Characteristics of included studies.
    Author(s), dateCountryStudy typeSampleSample size, N
    AlGahmdi & Moussa, 2012 [41]Saudi ArabiaCross-sectionalRandom sample of male and female outpatients and visitors attending a public University Hospital in Riyadh, Saudi Arabia801
    Bates et al, 2007 [42]United StatesCross-sectionalCommunity-wide convenience sample through intercept survey methods. Participants were recruited at high-traffic areas in a regional hub city in southeastern Ohio519
    Benotsch et al, 2004 [43]United StatesCross-sectionalIndividuals with human immunodeficiency virus recruited from neighborhoods in inner city Atlanta, Georgia324
    Bernhardt et al, 2004 [44]United StatesCross-sectionalIn-person surveys administered to diverse respondents in four different locations in two states, including a small and large city in the Southeastern United States, and a small and large city in the Northeastern United States. Online surveys were administered on a webpage that was promoted to diverse respondents using emails and word-of-mouth858
    Birru et al, 2004 [45]United StatesObservational studySubjects enrolled in a reading assistance program at Bidwell Training Center in Pittsburgh, PA8
    Borzekowski & Rickert, 2001 [46]United StatesCross-sectionalSample of 10th grade students from a diverse community near NY412
    Clayman et al, 2010 [47]United StatesCross-sectionalHispanics-Latinos of the 2005 Health Information National Trends Survey (HINTS) sample496
    Dart, 2008 [48]AustraliaCross-sectionalThree different Australian communities: low socioeconomic sample, mid-high socioeconomic sample, and university sample714
    Dutta-Bergman, 2003 [49]United StatesCross-sectionalStratified random sample of approximately US adults (Porter Novelli HealthStyles database)2636
    Feufel & Stahl, 2012 [50]GermanyObservational studyNative German-speaking adults22

    Gauld & Williams, 2009 [51]Australia & New ZealandCross-sectionalNon-representative sample of Australians and New Zealanders406
    Ghaddar et al, 2012 [52]United StatesCross-sectionalRandom sample of high school students in South Texas261
    Helft et al, 2005 [53]United StatesCross-sectionalConvenience sample of patients from the WMH Oncology Specialty Outpatient Clinic, Indianapolis200
    Hesse et al, 2005 [54]United StatesCross-sectionalNationally representative sample of US adults 18+ (HINTS 2002-03)6369
    Ishikawa et al, 2012 [55]JapanCross-sectionalNationally representative sample of people aged 15-75 years in Japan1311
    Kalichman et al, 2006 [56]United StatesCross-sectionalHIV-positive men and women who use the Internet recruited from AIDS service organizations, health care providers, social service agencies, and infectious disease clinics in inner-city areas of Atlanta, GA419
    Knapp et al, 2011a [57]United StatesCross-sectionalParents whose children with special health care needs were enrolled in Florida’s Medicaid and State Children’s Health Insurance Plan (SCHIP)2371
    Knapp et al, 2011b [58]United StatesCross-sectionalParents whose children are in a pediatric palliative care program in Florida129
    Lawson et al, 2011 [59]New ZealandCross-sectionalSample of New Zealanders drawn from the electoral roll8291
    Mackert et al, 2009 [60]United StatesFocus groupsParents from a midsized city in the southwestern United States 18 years of age or older, at or below median income for the area, who had not completed a 4-year college degree nor worked in the health care field43
    Maguire et al, 2011 [61]AustraliaCross-sectionalAustralian adults with schizophrenia (recruited from both community and inpatient settings) and general practice attendees301
    Maraziene et al, 2012 [63]LithuaniaCross-sectionalRandomly selected sample of Lithuanian citizens1763
    Marrie et al, 2013 [63]United StatesCross-sectionalUS people with multiple sclerosis enrolled (voluntary) in the Consortium of MS Centers developed the North American Research Committee on Multiple Sclerosis (NARCOMS) Registry8586
    Murray et al, 2003 [64]United StatesCross-sectionalHousehold probability sample of US adults (18+) from the 48 contiguous states3209
    Neter & Brainin, 2012 [65]IsraelCross-sectionalAdult (18+) Israeli population4286
    Nguyen & Bellamy, 2006 [66]United StatesCross-sectionalNon-Hispanic Asians and non-Hispanic whites form the 2003 HINTS4395
    Nwagwu, 2007 [67]NigeriaCross-sectionalIn-school, and out-of-school adolescents in Owerri, Nigeria1145
    Oh et al, 2012 [68]United StatesCross-sectionalKorean Americans ≥40 years254
    Richter et al, 2009 [69]GermanyCross-sectionalPatients with inflammatory rheumatic diseases (rheumatoid arthritis, systemic Lupus erythematosus (SLE), spondyloarthritis (SpA) regularly scheduled for a visit in our Rheumatology outpatient clinic at the University Clinic Düsseldorf153
    Smith, 2011 [70]United StatesCross-sectionalNationally representative sample of adults in the United States from the 2008 Annenberg National Health Communication Survey (ANHCS)3656
    Soederberg Miller & Bell, 2012 [71]United StatesCross-sectionalNationally representative sample of US adults from the Health Information National Trends Survey (HINTS)3796
    Van der Vaart et al, 2011 [72]The NetherlandsCross-sectionalSample of patients with rheumatic diseases (Study 1) and stratified sample of the Dutch population (Study 2)277
    van Deursen & van Dijk, 2011 [73]The NetherlandsCross-sectionalStratified random sample of adults (18+) living in the region of Twente, The Netherlands88
    Yan, 2010 [74]Hong Kong & KowloonCross-sectionalConvenience sample recruited in urban public areas including shopping locations (in Hong Kong Island and Kowloon) and subway stations443
    Ye, 2011 [24]United StatesCross-sectionalNationally representative sample of US adults (HINTS)7674
    Zhao, 2010 [75]United StatesCross-sectional2005 HINTS sample (foreign-born and US born)5393
    Zoellner et al, 2009 [76]United StatesCross-sectionalProportional quota sample of adult residents in the Mississippi Delta region.177
    Zulman et al, 2011 [77]United StatesCross-sectionalAdults 50 years of age and older in the United States1450

    Predictors Included in the Studies

    Only four of the included studies (4/38, 11%) specifically described the relationship between health literacy and one or more of the four outcomes of interest [45,52,60,76]. Health literacy was measured using different instruments: the NVS [52,76], the TOFHLA [43], and S-TOFHLA [60].

    The majority of the included studies (33/38, 87%) described the relationship between educational level and one or more of the outcomes. In most of them, educational level was operationalized either as number of years of education or as the highest achieved degree. In some cases, however, other operational definitions were used. One study compared a university sample with a mid-high socioeconomic and a low socioeconomic sample [48], two compared different ethnic groups with different educational levels [66,75], one compared in-school versus out-of-school young people [67], and one last study compared students in different grades and enrolled in programs with or without a health focus within the same school [52].

    Five studies (13%) described the relationship between other skills-based proxies for health literacy and one or more of the outcomes. These skills included reading comprehension [43], comfort speaking English [47], general literacy [45], and ability to understand health information [24]. Although it could be argued that this last skill is conceptually very similar to health literacy, the authors did not define it as such in their paper. In addition, health literacy was not measured with a recognized measure but by a single item. In the fifth study, a distinction was made between skilled (younger than 30 years, higher educational level and more experienced using the Web) and less-skilled (50 or older) participants [50]. As some studies reported results related to more than one predictor, percentages add up to more than 100%, and studies might be referred to more than once throughout the results.

    Outcomes

    Overview

    The majority of the studies included in the review included trust in online health information as an outcome variable (53%, 20/38). The second most common outcome was evaluation ability, included in 26% of the studies (10/38), followed by perceived quality of online health information (defined in the studies as perceived credibility, reliability, accuracy, and worth), which was included in 21% of the studies (8/38). The least frequent outcome variable in the pool of articles was the use of evaluation criteria, which was included in 5 studies only (14%). As some studies reported results on more than one outcome, percentages add up to more than 100%, and studies might be referred to more than once throughout the results.

    Ability to Evaluate Online Health Information

    Our first research question focused on the relationship between health literacy and ability to evaluate online health information. This specific aspect was addressed in 10 (26%) of the 38 studies (see Table 2 [43,45,52,56-58,64,65,72,73]). Evaluation ability was mostly assessed using self-report measures. Five studies [52,57,58,65,72] used the eHealth Literacy scale (eHEALS)—a measure assessing several aspects related to online health information seeking, including people’s perceived ability to evaluate online health resources and to distinguish high quality from low quality online health information—or some of its items [79]. Other studies asked the respondents to self-assess their ability to evaluate online health information by means of a single item measure [64] or qualitatively by means of an open question [45]. Among the studies using objective measures, van Deursen and van Dijk [73] asked their participants to perform an evaluation task derived from the eHEALS. In the other studies, the participants were asked to evaluate websites of varying quality [43,56].

    Two studies assessed the role of health literacy in people’s ability to evaluate online health information. The first study showed that low health literacy was associated with lower eHEALS scores [52], while in the second study low health literacy was shown to be associated with lower quality ratings of a high-quality website and higher quality ratings of a low-quality website [43].

    Six out of the nine studies focusing on educational level showed a positive relationship between educational attainment and perceived or actual ability to evaluate online health information [43,52,57,58,65,73]. The studies by Murray et al [64] and van der Vaart et al [72] did not find any significant difference among different education groups. A last study found that lower education was associated with assigning a higher quality rating to a low-quality website but did not find any association between education and evaluation of a high-quality website [56].

    Last, two studies reported on ability to evaluate online health information in relation to other skills. In a small qualitative study conducted in a sample with low general literacy, 7 out of the 8 respondents reported finding it very easy to locate trustworthy health information on the Internet [45]. The second study identified a positive association between reading ability and correct evaluation of the quality of a high-quality and a low-quality website [43].

    Perceived Quality of Online Health Information

    Eight studies (21%) reported on perceived quality of online health information. Except in one case where a multidimensional scale was used [46], this aspect was measured by means of single-item measures. Most studies did not refer directly to information quality but to one of its dimensions, that is, reliability [51,69,74], perceived accuracy [44,53,67], and perceived worth [46] (see Table 3 [44,46,50,51,53,67,69,74]). No studies on perceived information quality included health literacy as a predictor.

    The majority of the studies focusing on educational level failed to find significant associations between education and perceived quality of online health information. One study found a positive association [53], and one a negative association [44]. Another study found contrasting results: while out-of-school adolescents tended to describe online health information as more accurate than in-school adolescents, the latter group assigned higher overall quality to online health information as compared to out-of-school adolescents [67].

    Only one study reported on the relationship between skills-based proxies for health literacy and perceived quality of online health information. The study, however, did not find any differences among skilled and less-skilled participants: both groups doubted the quality of online health information [50].

    Table 2. Outcome 1: Ability to evaluate online health information.
    Author(s), datePredictorSpecific measure usedResult
    Benotsch et al, 2004a[43]Health literacy (TOFHLA)Quality rating of health information from reputable (JAMA) and unfounded (AIDS cure) webpages (5 dimensions: accuracy, amount of detail, trustworthiness/credibility, relevance, and usefulness)Lower health literacy scores predict higher quality ratings for the AIDS cure webpage (unfounded) and lower quality ratings for the JAMA webpage (reputable) (P<.01).
    Ghaddar et al, 2012b[52]Health literacy (NVS)eHEALScStudents identified as possibly or likely low health literate present significantly lower eHEALS scores than those with adequate health literacy (P<.05).
    Benotsch et al, 2004a[43]Educational levelQuality rating of health information from reputable (JAMA) and unfounded (AIDS cure) webpages (5 dimensions: accuracy, amount of detail, trustworthiness/credibility, relevance, and usefulness)Individuals with fewer years of education assign more credibility to unfounded information (P<.01). Educational level is unrelated to perceived quality of the JAMA webpage.
    Ghaddar et al, 2012b[52]Educational level (different grade levels; health classes)eHEALScFreshmen and sophomore students and those who have not taken a health course have lower eHEALS scores relative to students in higher grade levels and those enrolled in a health course (P<.001). eHEALS scores are significantly lower among students from the non-medical focused campuses compared to the 2 high schools with a focus on medical education (P<.001).
    Kalichman et al, 2006 [56]Educational levelQuality rating of health information from reputable (JAMA) and unfounded (AIDS cure) web pagesLess education predicts assigning higher credibility to unfounded Internet information (P<.001). Education does not have an impact on the evaluation of the reputable webpage.
    Knapp et al, 2011a[57]Educational leveleHEALS (Item 6: “I have the skills I need to evaluate the health resources I find on the Internet” and Item 7: “I can tell high-quality health resources from low-quality health resources on the Internet”)Parents without college education feel less confident in having the skills to evaluate the health resources they find on the Internet (P<.05) and feel less able to tell high-quality health resources from low-quality health resources on the Internet (P<.001) compared to those with college education.
    Knapp et al, 2011b[58]Educational leveleHEALScNot having a high school diploma is associated with a 2.5-point decrease in overall eHealth literacy (P<.05).
    Murray et al, 2003 [64]Educational levelPerceived ability to appraise online health informationNo significant effect of education on self-rated ability in appraising online health information.
    Neter & Brainin, 2012 [65]Educational leveleHEALScLower education is associated with lower eHealth literacy (F1,1274=5.43, P<.02).
    Van der Vaart et al, 2011 [72]Educational leveleHEALScNo significant correlation between educational level and eHEALS scores.
    Van Deursen & Van Dijk, 2011 [73]Educational levelNumber of information tasksd (derived from the eHEALS) completed successfullyEducational level is positively correlated with the number of information tasks completed successfully (β=.56, P<.001).
    Benotsch et al, 2004a[43]Other skills-based proxies for health literacy – Reading comprehensionQuality rating of health information from reputable (JAMA) and unfounded (AIDS cure) webpages (5 dimensions: accuracy, amount of detail, trustworthiness/credibility, relevance, and usefulness)Poorer reading comprehension predicts higher quality ratings for the AIDS cure webpage, whereas higher reading comprehension predicts higher quality ratings for the JAMA webpage (P<.01).
    Birru et al, 2004 [45]Other skills-based proxies for health literacy – Low general literacy (3rd to 8th grade level) only samplePerceived ability to locate trustworthy online health information7 out of 8 subjects report that they find it very easy to locate trustworthy information on the Internet. The eighth subject notes that it is moderately easy to find information that is trustworthy on the Internet.

    aStudy reported three times because it described the impact of health literacy, educational level, and other skills-based proxies for health literacy on the ability to evaluate the credibility of online health information.

    bStudy reported twice because it described the impact of both health literacy and educational level on the ability to evaluate the credibility of online health information.

    cThe eHEALS (eHealth Literacy scale) includes specific items about people’s perceived ability to evaluate the quality of online health information (Item 6: “I have the skills I need to evaluate the health resources I find on the Internet” and Item 7: “I can tell high-quality health resources from low-quality health resources on the Internet”). Specific data for these items are, however, not presented in the paper.

    dInformation tasks included choosing a website or a search system to seek information, defining search options or queries, selecting information on websites or in search results, and evaluating information sources. Specific data for the task “Evaluating information sources” are not presented in the paper.

    Table 3. Outcome 2: Perceived quality of online health information.
    Author(s), datePredictorSpecific measure usedResult
    Bernhardt et al, 2004 [44]Educational levelPerceived accuracy of online health informationLess educated respondents perceive online health information to be more accurate (P<.05).
    Borzekowski & Rickert, 2001 [46]Educational levelComposite assessing perceived worth, trustworthiness, use, and relevance of online health informationNo significant effect of educational level on the outcome.
    Gauld & Williams, 2009 [51]Educational levelPerceived reliability of online health informationEducational level is not correlated to perceived reliability of online health information.
    Helft et al, 2005 [53]Educational levelPerceived accuracy of online health informationLess educated patients are less likely to believe that online health information is accurate (r=.0417; P<.05).
    Nwagwu, 2007 [67]Educational level – In-school vs Out-of-schoolPerceived accuracy and quality of online health informationThe out-of-school groups describes more often the information as accurate. Overall, however, the in-school group assess online health information to be of higher quality more often than the out-of-school.
    Richter et al, 2009 [69]Educational levelPerceived reliability of online informationNo significant effect of education on perceived reliability of online information.
    Yan, 2010 [74]Educational levelPerceived reliability of online health informationNo significant effect of educational level on perceived reliability of online health information.
    Feufel & Stahl, 2012 [50]Other skills-based proxies for health literacy – Skilled (˂30 years of age, had a higher level of education, and were more experienced using the Web) vs less-skilled (≥50 years of age)Attitudes towards the quality of online health informationHealth information seekers in both cohorts doubt the quality of information retrieved online; among poorly skilled seekers, this is mainly because they doubt their skills to navigate vast amounts of information; once a website is accessed, quality concerns disappear in both cohorts.
    Trust in Online Health Information

    Trust in online health information was the outcome measure of more than half of the studies included in this review (20/38, 53%) (Table 4 [24,41,47,48,54,55,59,61,63,66-71,74-76]). Among those, three studies asked the participants to rate their trust in online health information on specific health topics (cancer and nutrition [54,66,76]). All other studies referred to general trust in online health information and assessed it by means of a single-item measure.

    Only one study investigated trust in online health information in relation to health literacy and did not find any significant relationship [76].

    Ten out of the 17 studies reporting on the relationship between educational level and trust in online health information identified a positive association. Among the other studies, two found a negative association [59,66] and four found no significant association [24,48,69,74] between educational level and trust. One study reported contrasting results: Maguire et al [61] found a positive association between education and trust among respondents with schizophrenia but not among non-schizophrenia respondents.

    Two studies reported on the relationship between skills-based proxies for health literacy and trust in online health information. A positive association between comfort speaking English [47] and ease in understanding health information [24] and trust was found.

    Table 4. Outcome 3: Trust in online health information.
    Author(s), datePredictorSpecific measure usedResult
    Zoellner et al, 2009 [76]Health literacy (NVS)Trust in food, diet, or nutrition-related online health informationNo significant effect of health literacy on trust in online health information.
    AlGahmdi & Moussa, 2012 [41]Educational levelTrust in online health informationFewer individuals with lower education always trust online health information (P<.001).
    Dart, 2008 [48]Low socioeconomic (LSE) vs mid-high (MSE) socioeconomic vs universityTrust in online health informationMost respondents in all three groups (LSE, 58.4%; MSE, 63.7%; university, 64.5%) are unsure of the trustworthiness or distrusted online health information (no significance level reported).
    Hesse et al, 2005 [54]Educational levelTrust in cancer-related online health informationEducation is positively associated with trust in cancer-related online information (P<.01).
    Ishikawa et al, 2012 [55]Educational levelTrust in online health informationParticipants with high school education or less report less trust in online health information than those with higher education (OR 0.68, 95% CI 0.51-0.92).
    Lawson et al, 2011 [59]Educational levelTrust in health information from media (including the Internet)Education is negatively associated with trust in health information from media (no statistics reported).
    Maguire et al, 2011 [61]Educational levelTrust in online health informationA lower level of education makes it more than twice less likely that a person with schizophrenia would trust online health information (OR 2.24, P<.01). There are no education-related differences among respondents without schizophrenia.
    Maraziene et al, 2012 [63]Educational levelTrust in online health informationPeople with lower education tend to trust the Internet less than their better educated counterparts (P<.05).
    Marrie et al, 2013 [63]Educational levelTrust in online health informationRespondents with a high school degree or less are less likely to have some/a lot of trust in online health information compared to those with an associate’s degree (OR 1.31, 95% CI 1.10-1.57), bachelor’s degree (OR 1.37, 95% CI 1.17-1.61), and graduate degree (OR 1.30, 95% CI 1.10-1.55).
    Nguyen & Bellamy, 2006 [66]Whites vs Asians (significantly different educational background)Trust in cancer-related online health informationAsians (lower educational level) are more likely to trust cancer-related online information than whites (OR 0.54, P<.05).
    Nwagwu, 2007 [67]In-school vs Out-of-schoolTrust in online health informationOut-of-school respondents report the information as trustworthy less often than the in-schools (no statistics reported).
    Oh et al, 2012 [68]Educational levelTrust in online health informationRespondents with 12 or fewer years of education are 3.1 times less likely to trust online health information a lot than were those with more than 12 years (95% CI 1.1-8.6).
    Richter et al, 2009 [69]Educational levelConfidence in online health informationNo significant effect of education on confidence in online health information.
    Smith, 2011 [70]Educational levelTrust in online health informationEducation is positively associated with trust in online health information (P<.001).
    Soederberg Miller & Bell, 2012 [71]Educational levelTrust in online health informationEducation is significantly correlated with trust in online health information (P<.01)
    Yan, 2010 [74]Educational levelConfidence in online health informationNo significant effect of educational level on confidence in online health information.
    Ye, 2011a [24]Educational levelTrust in online health informationEducational level is not correlated to trust in online health information.
    Zhao, 2010 [75]US-born vs Foreign-born (significantly different educational background)Trust in online health informationForeign-born Hispanics have lower trust in online health information compared with their US-born counterparts (higher educational level) (55% vs 86%, P=.016).
    Zulman et al, 2011 [77]Educational levelTrust in online health informationThose with high school or less are significantly less likely to trust online health information than college graduates (OR 2.47, P<.001).
    Clayman et al, 2010 [47]Other skills-based proxies for health literacy – Comfort speaking EnglishTrust in online health informationThose less comfortable speaking English report lower trust in online health information compared with those more comfortable speaking English (P<.01).
    Ye, 2011a [24]Other skills-based proxies for health literacy – Hard to understand health informationTrust in online health informationThe harder the health information is to understand, the less trust there is in online health information, F1,100=11.85, P<.01; β=.07, SE 0.02).

    aStudy reported twice because it described the impact of both educational level and other skill-based proxies for health literacy on trust in online health information.

    Use of Evaluation Criteria

    Five studies (14%) investigated the last outcome of interest, namely people’s use of evaluation criteria for online health information (Table 5 [42,49-51,60]).

    Only the study by Mackert et al [60] was about health literacy. This study, conducted in a low health literate-only sample, showed that low health literates used position in search results, quality of pictures, celebrity endorsement, and website authorship as criteria to evaluate online health information. With regard to website authorship, the study showed that almost all participants lacked trust in government or religious authorities as sources of online health information, whereas university researchers were generally considered trusted information providers.

    Reliance on website authorship as an evaluation criterion was also found in one of the studies investigating the role of educational level. In his study, Dutta-Bergman [49] showed that educational level was positively associated with trust in information coming from medical universities and federal institutions and negatively associated with trust in information from health insurance companies. No differences related to education were found with regard to trust in online health information coming from the local doctor. Bates et al [42] showed that there was no consistent relationship between educational level and using readability of websites as a criterion to evaluate website quality. This finding suggests that ease of reading is not widely used as a criterion to evaluate the quality of the websites. Gauld and Williams [51] found that respondents with higher educational level were more likely to check credentials (eg, name or qualifications of the author) when evaluating health websites.

    With regard to the role played by other skills-based proxies, Feufel and Stahl [50] showed in their study among skilled and less-skilled people that consistency with search intentions was used as a criterion to evaluate online health information. However, consistency referred to different things in the two groups: for the majority of less-skilled participants, this meant that a website confirmed a priori opinions, while for the majority of skilled participants, this meant that a website yielded the information that was searched for.

    Table 5. Outcome 4: Use of evaluation criteria.
    Author(s), datePredictorEvaluation criteriaResult
    Mackert et al, 2009 [60]Low health literacy (S-TOFHLA) only sampleHeuristics: Website position in search results; picture quality; celebrity endorsement; website authorshipParticipants use heuristics to evaluate online health information quality: position in search results; quality of pictures; celebrity endorsement. Almost-universal lack of trust in the government and in religious figures as sources of online health information. University researchers are trusted sources as information providers.
    Bates et al, 2007 [42]Educational levelWebsite readabilityNo consistent relationship between educational level and using readability of websites as a criterion to evaluate website quality.
    Dutta-Bergman, 2003 [49]Educational levelWebsite authorshipEducation does not impact trust on online health information from the local doctor. Trust in the local hospitals (t=3.83, P<.001) and in health insurance companies (t=1.90, P=.05) as sources of online health information is negatively associated with education. Individuals with higher trust in medical universities (t=11.83, P<.001) and federal sources (t=7.45, P<.001) are more educated than their counterparts.
    Gauld & Williams, 2009 [51]Educational levelWebsite credentialsLower educational level decreases the likelihood to check credentials of health websites.
    Feufel & Stahl, 2012 [50]Other skills-based proxies for health literacy – Skilled (˃30 years of age, had a higher level of education, and were more experienced using the Web) vs less-skilled (≥50 years of age)Consistency with search intentionsOverall, online health information is trusted if consistent with search intentions—among less-skilled participants this means if a website confirms a priori opinions (21/30, 70%) or yields search contents (9/30, 30%), and among skilled participants if a website confirms a priori opinions (4/28, 14%) or yields search contents (24/28, 86%).

    Discussion

    Principal Findings

    People’s health literacy is deemed to play an important role in the context of online health information seeking because according to most definitions it includes the ability to evaluate health information from different sources [12-19]. One of the main aims of this review was to identify and systematically summarize existing literature in order to collect evidence on the effect of low health literacy on the evaluation of online health information. The review provided us with indications of an overall (positive) association between health literacy (or one of its skills-based proxies) and both people’s ability to evaluate online health information (RQ1) and trust in the Internet as a source of health information (RQ3). On the other hand, evidence on the association between health literacy and both perceived quality of online health information (RQ2) and people’s use of evaluation criteria for online health information (RQ4) was inconsistent.

    The limited number and the heterogeneity of studies using health literacy as a predictor makes it hard to get a clear picture of how people’s health literacy levels impact the evaluation of online health information. However, the included studies give us some indications that low health literacy may have a negative impact. Indeed, low health literate individuals use evaluation criteria that do not correspond to the well-established quality criteria [9]. To illustrate, one study reports that they do not trust online health information from the government or that they use the position of a website in search results or the quality of images to evaluate the quality of online health information [60], whereas evidence shows that information provided from institutional sources is usually accurate [80,81] and that position in search results and image quality are not among the criteria that should be used to judge the quality of a website [9]. This could at least partly explain why, as reported in another study [43], low health literate respondents, compared to their high health literate counterparts, have been shown to give higher-quality ratings to low-quality websites, and lower ratings to high-quality websites. At the same time, it has also been found that low health literacy is correlated with lower eHEALS scores (which include people’s perceived ability to evaluate online health information). Confidence in one’s own information skills has been shown to be positively related to information use [82].

    What was observed in the few studies about health literacy was mostly confirmed in the higher amount of studies investigating differences in the evaluation of online health information related to educational level or other health literacy-related skills. Although we recognize that the diversity of measures used and the reliance on dichotomous measures of educational level may limit the accuracy of the results [40], this gives us more confidence in our findings. Included studies have shown that, overall, individuals with lower educational levels have worse actual and self-rated skills to evaluate the quality of online health information and lower trust in online health information compared to their more educated counterparts. Regarding perceived quality of online health information or people’s use of evaluation criteria, the limited number of studies and the diversity of samples and measures, however, does not allow us to draw conclusions about the impact of educational level or other skills-based proxies of health literacy, leaving two of the main research questions of this study mainly unanswered.

    Besides providing us with indications on the role played by people’s health literacy in their evaluation of online health information, our review also highlighted some important gaps and limitations of research in this field. The main gap identified is probably the fact that, despite the undeniable theoretical importance of health literacy for online health information seeking, only four studies have specifically investigated the association between health literacy and the evaluation of online health information. Moreover, one of these studies was conducted in a low health literacy-only sample [60], not allowing comparisons across different health literacy levels. All the other studies in our review compared people with different educational levels or different levels of other literacy-related skills.

    A second important limitation of current research highlighted by the review is the lack of shared definitions and measures. This has been shown to be the case not only for health literacy but also for other more commonly used predictors such as educational level, making comparisons across studies and summaries of existing evidence almost impossible. Additionally, only a few studies measured actual online health information evaluation skills, asking their participants to perform actual evaluation tasks. All the other studies relied on the participants’ self-rated ability. As shown in the study by van der Vaart et al [83], who compared their respondents’ eHEALS scores with an Internet performance test, self-rated ability does not seem to adequately capture people’s actual skills. Further research efforts should thus be devoted to the development and validation of a shared measure of online health information evaluation skills that is better able to reflect people’s real ability in this context.

    Last, among the studies investigating more than one outcome variable, no information was found on the relationship between them. For this reason, no conclusions can be drawn on the interplay among the different evaluative dimensions.

    Limitations

    This review has several limitations. The first limitation consists of the limited number of studies using health literacy as a predictor. Despite the fact that education has been shown to be related to (and it is often used as a proxy for) health literacy (eg, see [84]), this fact limits the extent to which our results can be generalized. A second important limitation of this review is the fact that all the included studies were non-interventional, with the consequence that the overall quality of the evidence has to be considered low [78]. Although causality is not an issue in this context (information evaluation cannot influence people’s literacy level), the uncontrolled nature of cross-sectional studies might fail to account for alternative explanations for the phenomenon under investigation [85]. This adds to the limited generalizability of the findings of this review. Finally, due to the heterogeneity of samples and outcome measures, it was not possible to conduct a meta-synthesis, with the result that no quantitative summary of the evidence could be provided.

    Conclusions and Future Directions

    Despite its limitations, this systematic review on the role of health literacy in the evaluation of online health information has provided us with important insights on this topic. These insights have allowed us to draw some preliminary conclusions and, most importantly, have highlighted the main outcomes and limitations of current research in this relatively new and unexplored field.

    From a research perspective, our findings are to be considered an indication of the fact that health literacy indeed plays a role in the evaluation of online health information and thus this topic is worth more scholarly attention. Based on the results of this review, the future research agenda in this field should include (1) a specific focus on health literacy, (2) more attention to the identification of the different criteria people use to evaluate online health information, (3) the development of shared definitions and measures for the most commonly used outcomes in the field of evaluation of online health information, and (4) the assessment of the relationship between the different evaluative dimensions and the role played by health literacy in shaping their interplay. Only by first addressing these research gaps will it be possible, in line with what has been called for in a recent Cochrane review [86], to develop high-quality interventions to enhance low health literacy individuals’ ability to appraise online health information as well as to develop well-designed randomized controlled trials to investigate their effects. A better understanding of how people appraise online health information is also crucial in view of the investigation of the impact of their information evaluation ability on their interactions with health care providers and ultimately on health outcomes.

    From a practice perspective, the results of this review should be sufficient to urge public health officials and health care providers to start devoting particular attention to the online health information seeking behavior of low health literate citizens and provide them with targeted advice on criteria to correctly assess the quality of the information they find online.

    Acknowledgments

    The authors wish to thank Anne Kimman for her valuable help in the abstract and full-text review process. This study was funded by a personal grant awarded to ND by the Swiss National Science Foundation (grant number P2TIP1_148345). The funding source had no role in study design, data collection, analysis, interpretation, or writing of the report.

    Conflicts of Interest

    None declared.

    ‎

    Multimedia Appendix 1

    Search strategy.

    PDF File (Adobe PDF File), 34KB
    1. Fox S, Duggan M. Health Online 2013. 2013.   URL: http://pewinternet.org/Reports/2013/Health-online.aspx [accessed 2015-04-25] [WebCite Cache]
    2. Berland GK, Elliott MN, Morales LS, Algazy JI, Kravitz RL, Broder MS, et al. Health information on the Internet: accessibility, quality, and readability in English and Spanish. JAMA 2001;285(20):2612-2621 [FREE Full text] [Medline]
    3. Bernstam EV, Walji MF, Sagaram S, Sagaram D, Johnson CW, Meric-Bernstam F. Commonly cited website quality criteria are not effective at identifying inaccurate online information about breast cancer. Cancer 2008 Mar 15;112(6):1206-1213 [FREE Full text] [CrossRef] [Medline]
    4. Eysenbach G, Powell J, Kuss O, Sa ER. Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. JAMA 2002;287(20):2691-2700. [Medline]
    5. Hardey M. 'E-health': the internet and the transformation of patients into consumers and producers of health knowledge. Info, Comm & Soc 2001 Oct;4(3):388-405. [CrossRef]
    6. Stvilia B, Mon L, Yi YJ. A model for online consumer health information quality. J Am Soc Inf Sci Technol 2009;60(9):1781-1791. [CrossRef]
    7. Flanagin AJ, Metzger MJ. Digital Media and Youth: Unparalleled Opportunity and Unprecedented Responsibility. In: Metzer MJ, Flanagin AJ, editors. Digital Media, Youth,Credibility. The John D.Catherine T. MacArthur Foundation Series on Digital Media and Learning. Cambridge, MA: The MIT Press; 2008:5.
    8. Cline RJ, Haynes KM. Consumer health information seeking on the Internet: the state of the art. Health Educ Res 2001 Dec;16(6):671-692 [FREE Full text] [Medline]
    9. Kim P, Eng TR, Deering MJ, Maxfield A. Published criteria for evaluating health related web sites: review. BMJ 1999 Mar 6;318(7184):647-649 [FREE Full text] [Medline]
    10. Tichenor PJ, Donohue GA, Olien CN. Mass media flow and differential growth in knowledge. Public Opin Q 1970;34(2):159-170. [CrossRef]
    11. Viswanath K, Kreuter MW. Health disparities, communication inequalities, and eHealth. Am J Prev Med 2007 May;32(5 Suppl):S131-S133 [FREE Full text] [CrossRef] [Medline]
    12. Nutbeam D. Health promotion glossary. Health Promot 1986 May;1(1):113-127. [Medline]
    13. Ishikawa H, Yano E. Patient health literacy and participation in the health-care process. Health Expect 2008 Jun;11(2):113-122. [CrossRef] [Medline]
    14. Jordan JE, Buchbinder R, Osborne RH. Conceptualising health literacy from the patient perspective. Patient Educ Couns 2010 Apr;79(1):36-42. [CrossRef] [Medline]
    15. Kickbusch I. Health literacy: a search for new categories. Health Promot Int 2002 Mar;17(1):1-2 [FREE Full text] [Medline]
    16. Nutbeam D. Health literacy as a public health goal: A challenge for contemporary health education and communication strategies into the 21st century. Health Promot Internation 2000;15(3):259-267. [CrossRef]
    17. Schulz P, Nakamoto K. Emerging themes in health literacy. Stud Commun Sci 2005;5(2):1-10.
    18. Schulz PJ, Nakamoto K. Health literacy and patient empowerment in health communication: the importance of separating conjoined twins. Patient Educ Couns 2013 Jan;90(1):4-11. [CrossRef] [Medline]
    19. Zarcadoolas C, Pleasant A, Greer DS. Understanding health literacy: An expanded model. Health Promot Internat 2005 Jun;20(2):195-203. [CrossRef]
    20. Anker AE, Reinhart AM, Feeley TH. Health information seeking: a review of measures and methods. Patient Educ Couns 2011 Mar;82(3):346-354. [CrossRef] [Medline]
    21. Hibbard JH, Tusler M. How much do health literacy and patient activation contribute to older adults' ability to manage their health?. Washington, DC: AARP Public Policy Institute; 2005.
    22. Reyna VF, Nelson WL, Han PK, Dieckmann NF. How numeracy influences risk comprehension and medical decision making. Psychol Bull 2009 Nov;135(6):943-973 [FREE Full text] [CrossRef] [Medline]
    23. von WC, Semmler C, Good A, Wardle J. Health literacy and self-efficacy for participating in colorectal cancer screening: The role of information processing. Patient Educ Couns 2009 Jun;75(3):352-357. [CrossRef] [Medline]
    24. Ye Y. Correlates of consumer trust in online health information: findings from the health information national trends survey. J Health Commun 2011 Jan;16(1):34-49. [CrossRef] [Medline]
    25. Hovland CI. Changes in attitude through communication. The Journal of Abnormal and Social Psychology 1951;46(3):424-437. [CrossRef]
    26. Hovland C, Janis I, Kelley H. Communication and persuasion. New Haven, CT: Yale University Press; 1953.
    27. Hovland CI, Lumsdaine CI, Sheffield FD. Experiments on mass communication. Princeton, NJ: Princeton University Press; 1949.
    28. Hovland CI, Weiss W. The influence of source credibility on communication effectiveness. Public Opin Q 1951;15(4):635-650. [CrossRef]
    29. Knight S, Burn JM. Developing a framework for assessing information quality on the world wide web. Informing Science: International Journal of an Emerging Transdiscipline 2005;8(5):159-172.
    30. Rieh SY. Judgment of information quality and cognitive authority in the Web. J Am Soc Inf Sci 2002;53(2):145-161. [CrossRef]
    31. Wang RY, Strong DM. Beyond accuracy: What data quality means to data consumers. J Manage Inf Syst 1996;12(4):5-33.
    32. Fogg B, Tseng H. The elements of computer credibility. 1999 Presented at: SIGCHI conference on human factors in computing systems; 1999; Pittsburgh, PA.
    33. Shneiderman B. Designing trust into online experiences. Commun. ACM 2000;43(12):57-59. [CrossRef]
    34. Fogg B, Marshall J, Laraki O, Osipovich A, Varma C, Fang N. What makes web sites credible? A report on a large quantitative study. 2001 Presented at: SIGCHI conference on human factors in computing systems; 2001; Seattle, WA.
    35. Landis JR, Koch GG. The Measurement of Observer Agreement for Categorical Data. Biometrics 1977 Mar;33(1):159. [CrossRef]
    36. Davis TC, Long SW, Jackson RH, Mayeaux EJ, George RB, Murphy PW, et al. Rapid estimate of adult literacy in medicine: a shortened screening instrument. Fam Med 1993 Jun;25(6):391-395. [Medline]
    37. Parker RM, Baker DW, Williams MV, Nurss JR. The test of functional health literacy in adults: a new instrument for measuring patients' literacy skills. J Gen Intern Med 1995 Oct;10(10):537-541. [Medline]
    38. Nurss J, Parker R, Williams M, Baker D. Directions for administration and scoring and technical data, short test of functional health literacy in adults (STOFHLA-English and STOFHLA-Spanish). Atlanta, GA: Center for the Study of Adult Literacy; 1998.
    39. Weiss BD, Mays MZ, Martz W, Castro KM, DeWalt DA, Pignone MP, et al. Quick assessment of literacy in primary care: the newest vital sign. Ann Fam Med 2005;3(6):514-522 [FREE Full text] [CrossRef] [Medline]
    40. Baker DW. The meaning and the measure of health literacy. J Gen Intern Med 2006 Aug;21(8):878-883 [FREE Full text] [CrossRef] [Medline]
    41. AlGhamdi KM, Moussa NA. Internet use by the public to search for health-related information. Int J Med Inform 2012 Jun;81(6):363-373. [CrossRef] [Medline]
    42. Bates BR, Romina SM, Ahmed R. The effect of improved readability scores on consumers' perceptions of the quality of health information on the internet. J Cancer Educ 2007;22(1):15-20. [CrossRef] [Medline]
    43. Benotsch EG, Kalichman S, Weinhardt LS. HIV-AIDS patients' evaluation of health information on the internet: the digital divide and vulnerability to fraudulent claims. J Consult Clin Psychol 2004 Dec;72(6):1004-1011. [CrossRef] [Medline]
    44. Bernhardt JM, Felter EM. Online pediatric information seeking among mothers of young children: results from a qualitative study using focus groups. J Med Internet Res 2004 Mar 1;6(1):e7 [FREE Full text] [CrossRef] [Medline]
    45. Birru MS, Monaco VM, Charles L, Drew H, Njie V, Bierria T, et al. Internet usage by low-literacy adults seeking health information: an observational analysis. J Med Internet Res 2004 Sep 3;6(3):e25 [FREE Full text] [CrossRef] [Medline]
    46. Borzekowski DL, Rickert VI. Adolescent cybersurfing for health information: a new resource that crosses barriers. Arch Pediatr Adolesc Med 2001 Jul;155(7):813-817. [Medline]
    47. Clayman ML, Manganello JA, Viswanath K, Hesse BW, Arora NK. Providing health messages to Hispanics/Latinos: understanding the importance of language, trust in health information sources, and media use. J Health Commun 2010;15 Suppl 3:252-263 [FREE Full text] [CrossRef] [Medline]
    48. Dart J, Gallois C, Yellowlees P. Community health information sources--a survey in three disparate communities. Aust Health Rev 2008 Feb;32(1):186-196. [Medline]
    49. Dutta-Bergman M. Trusted online sources of health information: differences in demographics, health beliefs, and health-information orientation. J Med Internet Res 2003;5(3):e21 [FREE Full text] [CrossRef] [Medline]
    50. Feufel MA, Stahl SF. What do web-use skill differences imply for online health information searches? J Med Internet Res 2012;14(3):e87 [FREE Full text] [CrossRef] [Medline]
    51. Gauld R, Williams S. Use of the Internet for health information: a study of Australians and New Zealanders. Inform Health Soc Care 2009 Sep;34(3):149-158. [CrossRef] [Medline]
    52. Ghaddar SF, Valerio MA, Garcia CM, Hansen L. Adolescent health literacy: the importance of credible sources for online health information. J Sch Health 2012 Jan;82(1):28-36. [CrossRef] [Medline]
    53. Helft PR, Eckles RE, Johnson-Calley CS, Daugherty CK. Use of the internet to obtain cancer information among cancer patients at an urban county hospital. J Clin Oncol 2005 Aug 1;23(22):4954-4962 [FREE Full text] [CrossRef] [Medline]
    54. Hesse BW, Nelson DE, Kreps GL, Croyle RT, Arora NK, Rimer BK, et al. Trust and sources of health information: the impact of the Internet and its implications for health care providers: findings from the first Health Information National Trends Survey. Arch Intern Med 2005;165(22):2618-2624. [CrossRef] [Medline]
    55. Ishikawa Y, Nishiuchi H, Hayashi H, Viswanath K. Socioeconomic status and health communication inequalities in Japan: a nationwide cross-sectional survey. PLoS One 2012;7(7):e40664 [FREE Full text] [CrossRef] [Medline]
    56. Kalichman SC, Cherry C, Cain D, Weinhardt LS, Benotsch E, Pope H, et al. Health information on the Internet and people living with HIV/AIDS: information evaluation and coping styles. Health Psychol 2006 Mar;25(2):205-210. [CrossRef] [Medline]
    57. Knapp C, Madden V, Wang H, Sloyer P, Shenkman E. Internet use and eHealth literacy of low-income parents whose children have special health care needs. J Med Internet Res 2011;13(3):e75 [FREE Full text] [CrossRef] [Medline]
    58. Knapp C, Madden V, Marcu M, Wang H, Curtis C, Sloyer P, et al. Information seeking behaviors of parents whose children have life-threatening illnesses. Pediatr Blood Cancer 2011 May;56(5):805-811. [CrossRef] [Medline]
    59. Lawson R, Forbes S, Williams J. Patterns of trust in sources of health information. N Z Med J 2011 Jan 21;124(1328):98-105. [Medline]
    60. Mackert M, Kahlor L, Tyler D, Gustafson J. Designing e-health interventions for low-health-literate culturally diverse parents: addressing the obesity epidemic. Telemed J E Health 2009 Sep;15(7):672-677 [FREE Full text] [CrossRef] [Medline]
    61. Maguire PA, Reay RE, Looi JCL, Cubis J, Byrne GJ, Raphael B. Neither the internist nor the Internet: use of and trust in health information sources by people with schizophrenia. Aust N Z J Psychiatry 2011 Jun;45(6):489-497. [CrossRef] [Medline]
    62. Marazienė D, Klumbienė J, Tomkevičiūtė J, Misevičienė I. Sources and reasons for seeking health information by Lithuanian adults. Medicina (Kaunas) 2012;48(7):371-378 [FREE Full text] [Medline]
    63. Marrie RA, Salter AR, Tyry T, Fox RJ, Cutter GR. Preferred sources of health information in persons with multiple sclerosis: degree of trust and information sought. J Med Internet Res 2013;15(4):e67 [FREE Full text] [CrossRef] [Medline]
    64. Murray E, Lo B, Pollack L, Donelan K, Catania J, White M, et al. The impact of health information on the internet on the physician-patient relationship: patient perceptions. Arch Intern Med 2003 Jul 28;163(14):1727-1734. [CrossRef] [Medline]
    65. Neter E, Brainin E. eHealth literacy: extending the digital divide to the realm of health information. J Med Internet Res 2012;14(1):e19 [FREE Full text] [CrossRef] [Medline]
    66. Nguyen GT, Bellamy SL. Cancer information seeking preferences and experiences: disparities between Asian Americans and Whites in the Health Information National Trends Survey (HINTS). J Health Commun 2006;11 Suppl 1:173-180. [CrossRef] [Medline]
    67. Nwagwu WE. The Internet as a source of reproductive health information among adolescent girls in an urban city in Nigeria. BMC Public Health 2007;7:354 [FREE Full text] [CrossRef] [Medline]
    68. Oh KM, Kreps GL, Jun J, Chong E, Ramsey L. Examining the health information-seeking behaviors of Korean Americans. J Health Commun 2012 Aug;17(7):779-801. [CrossRef] [Medline]
    69. Richter JG, Becker A, Koch T, Nixdorf M, Schacher B, Monser R, et al. Internet use in rheumatology outpatients in 2006: gender less important. Clin Exp Rheumatol 2009;27(1):15-21. [Medline]
    70. Smith D. Health care consumer's use and trust of health information sources. Journal of Communication in Healthcare 2011 Oct;4(3):200-210. [CrossRef]
    71. Miller LMS, Bell RA. Online health information seeking: the influence of age, information trustworthiness, and search challenges. J Aging Health 2012 Apr;24(3):525-541. [CrossRef] [Medline]
    72. van der Vaart R, van Deursen AJAM, Drossaert CHC, Taal E, van Dijk JAMG, van de Laar MAFJ. Does the eHealth Literacy Scale (eHEALS) measure what it intends to measure? Validation of a Dutch version of the eHEALS in two adult populations. J Med Internet Res 2011;13(4):e86 [FREE Full text] [CrossRef] [Medline]
    73. van Deursen AJAM, van Dijk JAGM. Internet skills performance tests: are people ready for eHealth? J Med Internet Res 2011;13(2):e35 [FREE Full text] [CrossRef] [Medline]
    74. Yan YY. Online health information seeking behavior in Hong Kong: an exploratory study. J Med Syst 2010 Apr;34(2):147-153. [Medline]
    75. Zhao X. Cancer information disparities between U.S.- and foreign-born populations. J Health Commun 2010;15 Suppl 3:5-21. [CrossRef] [Medline]
    76. Zoellner J, Connell C, Bounds W, Crook L, Yadrick K. Nutrition literacy status and preferred nutrition communication channels among adults in the Lower Mississippi Delta. Prev Chronic Dis 2009 Oct;6(4):A128 [FREE Full text] [Medline]
    77. Zulman DM, Kirch M, Zheng K, An LC. Trust in the internet as a health resource among older adults: analysis of data from a nationally representative survey. J Med Internet Res 2011;13(1):e19 [FREE Full text] [CrossRef] [Medline]
    78. Balshem H, Helfand M, Schünemann HJ, Oxman AD, Kunz R, Brozek J, et al. GRADE guidelines: 3. Rating the quality of evidence. J Clin Epidemiol 2011 Apr;64(4):401-406. [CrossRef] [Medline]
    79. Norman CD, Skinner HA. eHEALS: The eHealth Literacy Scale. J Med Internet Res 2006;8(4):e27 [FREE Full text] [CrossRef] [Medline]
    80. Black PC, Penson DF. Prostate Cancer on the Internet—Information or Misinformation? The Journal of Urology 2006 May;175(5):1836-1842. [CrossRef]
    81. Korfage IJ, van den Bergh RCN, Essink-Bot ML. Deciding on PSA-screening - Quality of current consumer information on the Internet. Eur J Cancer 2010 Nov;46(17):3073-3081. [CrossRef] [Medline]
    82. Dutta MJ, Bodie GD. Web searching for health: Theoretical foundations and connections to health related outcomes. Web Search: Springer Series in Information Science and Knowledge Management 2008;14(4):235-254. [CrossRef]
    83. van der Vaart R, Drossaert CHC, de HM, Taal E, van de Laar MAFJ. Measuring actual eHealth literacy among patients with rheumatic diseases: a qualitative analysis of problems encountered using Health 1.0 and Health 2.0 applications. J Med Internet Res 2013;15(2):e27 [FREE Full text] [CrossRef] [Medline]
    84. Martin LT, Ruder T, Escarce JJ, Ghosh-Dastidar B, Sherman D, Elliott M, et al. Developing predictive models of health literacy. J Gen Intern Med 2009 Nov;24(11):1211-1216 [FREE Full text] [CrossRef] [Medline]
    85. Carlson MDA, Morrison RS. Study design, precision, and validity in observational studies. J Palliat Med 2009 Jan;12(1):77-82 [FREE Full text] [CrossRef] [Medline]
    86. Car J, Lang B, Colledge A, Ung C, Majeed A. Interventions for enhancing consumers' online health literacy. Cochrane Database Syst Rev 2011(6):CD007092. [CrossRef] [Medline]

    ‎
    CINAHL: Cumulative Index to Nursing and Allied Health Literature
    eHEALS: eHealth Literacy Scale
    HINTS: Health Information National Trends Survey
    MeSH: Medical Subject Headings
    S-TOFHLA: Short Test of Functional Health Literacy in Adults
    TOFHLA: Test of Functional Health Literacy in Adults

    Edited by G Eysenbach; submitted 11.11.14; peer-reviewed by S Ghaddar, D Castaneda; comments to author 19.02.15; revised version received 01.03.15; accepted 16.03.15; published 07.05.15

    Copyright

    ©Nicola Diviani, Bas van den Putte, Stefano Giani, Julia CM van Weert. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 07.05.2015.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.