Development of the Digital Health Literacy Instrument: Measuring a Broad Spectrum of Health 1.0 and Health 2.0 Skills

Introduction

Digitization in health care has changed rapidly over the last decades, and online information and (mobile) applications are playing a growing role in health care. Along with these changes, skills to search, select, appraise, and apply online health information and health care-related digital applications are becoming increasingly important for health care consumers. These skills are called digital health literacy [1], or eHealth literacy [2]. The relevance of this form of literacy is demonstrated in recent studies, showing that people’s self-perceived skills to use online information actually affect their health and the quality of their health care, and that a lack of such skills may lead to adverse outcomes [3,4]. Hsu et al. [3] found that digital health literacy skills are associated with various types of health behavior, including healthy eating, exercise, and sleep behavior. Neter and Brainin [4] found relationships between digital health literacy and the presence of chronic illness, perceived self-management skills, and better self-perceived understanding of health status, symptoms, and optional treatments.

A valid measurement instrument on digital health literacy is essential to examine the effects of these skills, both on an individual level and on a population level. On an individual level—for example, in daily clinical practice—a measurement tool could support decisions about the extent to which a patient is able to benefit from particular eHealth tools and interventions [5,6]. Also, it could provide input to coach and train patients who need support in using Web-based health tools [7,8]. On a population level, a proper measurement instrument could provide insight into vulnerable subgroups that face additional challenges in using health care, due to its digitization. For example, previous studies have shown that digital health literacy is related to sociodemographics such as age, education, and income [4,9,10], and studies have shown that certain populations do not have the skills and knowledge to use Web-based health tools for their own benefit and might thereby even become underserved [5,10]. Better insight into populations at risk of low digital health literacy can lead to development and tailoring of health technologies for these specific groups [5,11].

In research, the focus regarding digital health literacy has mainly been on the use of health information that is available on the Internet (Health 1.0). Yet eHealth is a broad concept that extends beyond the use of information alone. More recent applications (so called Health 2.0 applications) offer all sorts of interactive technologies, which support people to communicate about their health (with peers and with health care professionals; eg, via forums or e-consults), to self-monitor their health (eg, via patient portals), and even to receive treatment via the Internet (eg, via Web-based cognitive behavioral therapy) [12]. To measure peoples’ ability to use this broad spectrum of applications, an assessment of very diverging skills is essential, since using interactive Health 2.0 applications asks for a more diverse range of skills than retrieving health information alone does [6,13-15]. A study on the digital health literacy skills of patients with rheumatic diseases found that 6 types of competences are essential to properly use both Health 1.0 and Health 2.0 applications [16]. First, people need operational and navigation skills to use a computer and Internet browser; this involves, for example, using a keyboard, touch screen, and search engine and being able to find one’s way around on the Internet. Second, they need information and evaluation skills to search, appraise, and apply online information; this involves, for example, formulating a correct search query, choosing a reliable search result, understanding the obtained search results, and being able to select the results that are reliable and applicable. To use Health 2.0 applications, people need additional skills related to interactivity on the Web. This encompasses adding self-generated content to the Internet (eg, being able to express oneself in written language) and considering both their own and others’ privacy (eg, knowing who is able to read what one has posted on the Internet) [15,16]. Therefore, when measuring a person’s digital health literacy skills, the ability to interact on the Internet should be taken into account as well.

Studies on digital health literacy up until now have used the 8-item eHealth Literacy Scale (eHEALS) [17], which has been the only validated instrument on these skills for a long time. It provides a reliable insight into the self-reported skills of health care consumers when searching and using online health information. Studies on its validation have shown that it measures 1 overall concept [17,18], or 2 separate concepts: seeking and appraising online information [19,20]. In order to extent the measurement of digital health literacy and to assess the broad spectrum of skills that are involved, we developed a new instrument. The Digital Health Literacy Instrument (DHLI) aims to incorporate the diversity of skills to use both Health 1.0 and Health 2.0 tools [14,16]. To promote the feasibility of assessment, this is done with self-reportage of health care consumers’ perceived skills. Nevertheless, it its known that self-reportage can cause a bias, since people tend to over- or underestimate their own Internet skills [18,21,22]. A study on the predictive validity of the eHEALS has shown that the relationship between people’s own perceived skills and their actual performance on Web-based health-related assignments is only small [18]. To overcome this bias in the DHLI instrument, we strive to measure digital literacy skills more objectively as well.

This study’s objective was to determine the instrument’s reliability and validity, and to explore the value of both the self-report items and the performance-based items. To this extent, we determined distributional properties, internal consistency, test-retest reliability, content validity, and construct validity. The construct validity was assessed by studying the correlation with several concepts that can be assumed to be related. First, we investigated the relation with traditional “digital divide” variables (sociodemographics, Internet use, and use of Web-based health apps). Based on previous studies on health literacy and eHealth literacy, we hypothesized small to moderate (.10-.30) negative correlations with age and positive correlations with education and (health-related) Internet use [23-27]. Second, we studied the relation with health status, as digital health literacy can be assumed to have an important influence on health behavior and health-related choices that people make [3,4]. Due to the low number of studies on this subject, and heterogeneity in how health is measured, the expected correlation needs to be estimated. Taking the broadness of this concept into account and all the other variables that influence it, we expected a small correlation of .20. Third, we measured the relation with existing instruments that measure strongly related concepts, namely the Newest Vital Sign (NVS) [28] and the eHEALS [17]. The NVS aims to measure skills related to health literacy (reading ability, numeracy, and applying information). Since this only implies regular health information and does not include digital skills, we expected a moderate correlation (±.30). The eHEALS measures digital health literacy skills, but only on a Health 1.0 level. It does not assess interactive skills on the Internet; therefore, we expected a moderate to large correlation (±.50).

Methods

Development of the Digital Health Literacy Instrument

The DHLI operationalizes 7 separate skills. The types of skills are based on a study in which patients with rheumatic diseases were asked to perform a wide range of Health 1.0 and 2.0 eHealth assignments (to find and appraise online health information, to use interactive apps to communicate with peer patients, and to use a personal electronic medical record to retrieve disease-related information and monitor their health status). Since that study used a bottom-up method to determine all relevant skills in health-related use of the Internet, this provided a valid starting point for the instrument [16]. While participants were performing these assignments, we recorded a diverse range of problems, which we divided into 6 categories: (1) operational skills, to use the computer and Internet browser, (2) navigation skills, to navigate and orientate on the Web, (3) information searching skills, to use correct search strategies, (4) evaluating reliability and relevance of online information, (5) adding self-generated content to Web-based apps, and (6) protecting and respecting privacy while using the Internet. In designing the instrument, for each skill we formulated 3 items (in Dutch) to measure people’s self-perceived abilities. In the operationalization process, we divided category 4 into 2 separate concepts—evaluating reliability of the information in general, and determining relevance of the information to oneself in a particular situation—resulting in a total of 7 skill categories measured by 21 self-report items. With these self-report items, people score how difficult they perceive certain tasks to be and how often they experience certain problems on the Internet. Each item was scored on a 4-point scale, with response options ranging from “very easy” to “very difficult” and from “never” to “often.” Scores were reversed, so that a higher score represented a higher level of digital health literacy. The 3 items on the skill of protecting privacy were not obligatory to fill in: when respondents did not have any experience with posting messages on social media or other communication portals, they could leave the items blank.

The DHLI was translated into English, using forward and backward translation, according to World Health Organization guidelines [29]. The exact wording of the items can be found below. We calculated subscores for each skill by using the mean of the 3 items on every skill. We calculated a total score by using the total mean, for which answers on at least 18 items were necessary. Additionally, for each skill, we added a performance-based item, using questions that asked the participant to apply the particular skill in a fictional situation (see Multimedia Appendix 1). Typically, the skill items display a “print screen” of a search engine or website and ask the participant a skill-related question that can be answered that can be scored as correct (score=1) or false (score=0). Examples of performance-based questions are what button to press for a certain action, or what piece of information would be most valuable in a certain situation. Each item has 5 answer options: 4 different answers (of which 1 is correct) and an “I don’t know” option (score=0). Each correct answer receives 1 point, adding up to a maximum total score of 7 points. To calculate a total score, at least 6 out of 7 items should be answered.

We tested face validity of this initial instrument among 11 people, using a 3-step test cognitive interview [30]. Participants were asked to think aloud while completing the items, in order to gain insight into their reasoning and decision-making process when answering the questions [31]. After completion, the research leader asked several follow-up probing questions related to the items that had seemed to cause problems in understanding or answering. In this way, we gained insight into the readability and clarity of the items and altered them accordingly. After these initial alterations, we conducted a second pilot test among 8 people. We made only a few minor alterations in wording in this last pilot round.

Design of the Survey Study

We studied the reliability and validity of the instrument in a paper-and-pencil survey study among the general Dutch population. We did not use a Web-based survey, in order not to exclude people with low digital health literacy skills beforehand.

Participants and Procedure

A total of 200 people participated in the study. Inclusion criteria were having Internet access, being fluent in Dutch, and being 18 years of age or older. We recruited participants through convenience sampling using stratification based on age, sex, and educational level to reach an equal distribution on these sociodemographics. Regarding age, the categories were (1) 18-34 years, (2) 35-49 years, (3) 50-64 years, and (4) 65 years or older. Regarding education, the categories were (1) low: no education, primary school only, or lowest level of high school, (2) middle: higher levels of high school or secondary vocational education, and (3) high: bachelor’s degree or higher. On this variable, complete stratification was not feasible, resulting in an overrepresentation of more highly educated respondents.

People who were invited to participate received an invitation letter explaining the inclusion criteria, purpose of the study, its duration (30 minutes), and its voluntary nature. People who consented to take part in the study were contacted in person, by telephone or email, to confirm their interest in the study and to schedule an appointment. The assessment was done at a quiet location (mostly the participant’s home). At the start of the survey (T1) an informed consent form was signed. Participants were asked to fill out the questionnaire and, after that, the research leader assessed the NVS (see Measures section) in a face-to-face setting, which took approximately 4 minutes. We asked all participants 2 weeks later to fill out the DHLI again (T2). After completion of data collection, we raffled off 10 gift certificates of €25 each among the participants at T1.

The study was approved by the Psychology Ethics Committee of Leiden University, Leiden, the Netherlands.

Measures

Besides the DHLI, the survey assessed the participants’ (1) sociodemographics: sex, age, and educational level; (2) Internet use: means of Internet access, frequency of Internet use, and self-rated Internet skills; (3) health-related Internet use; (4) health status; (5) health literacy; and (6) eHealth literacy.

We measured health-related Internet use by asking participants the number of occasions on which they had used several eHealth applications, divided into online information, health-related communication tools (such as a patient forum and e-consult), and treatment-related applications (monitoring, Web-based self-help, mobile phone app), with a total of 12 items. Answer options were “never” (score=0), “once” (score=1), “several times” (score=2), and “often” (score=3). We calculated the sum score by adding up the scores on each item.

We measured health status with 3 subscales of the Dutch version of the RAND 36-Item Health Survey (RAND-36), namely General Health Perceptions, Physical Functioning, and Emotional Well-being [32-34]. These scales contain, respectively, 5, 10, and 5 items on perceived general health and perceived health in relation to others (alpha=.81), experienced limitations due to physical health (alpha=.92), and states of emotional well-being (alpha=.85) [34].

We measured health literacy with the Dutch version of the NVS [28,35]. The instrument consists of 6 items based on a nutrition label from an ice cream container. The NVS measures reading skills, numeracy skills, and the ability to apply information. Each correctly answered item receives 1 point, which can be summed up as a total sum score (alpha=.78).

We measured eHealth literacy with the Dutch version of the eHEALS [17,18]. The eHEALS contains 8 items on self-perceived skills to use online health information, measured by a 5-point Likert scale with response options ranging from “strongly disagree” to “strongly agree.” Total scores of the eHEALS are summed to range from 8-40, with higher scores representing higher self-perceived eHealth literacy (alpha=.93).

Data Analyses

Data were analyzed using IBM SPSS version 23.0 for Windows (IBM Corporation). Cronbach alpha served as a measure of internal consistency, reflecting the (weighted) average correlation of items within the scale [36]. In general, a Cronbach alpha of .7-.8 is regarded as satisfactory for scales to be used as research tools [37]. We calculated item-total correlations using Spearman rho correlations. Distributional properties of the DHLI and the possible subscales were inspected to examine their normality and to identify floor and ceiling effects. We used skewness and kurtosis values, as well as a Kolmogorov-Smirnov test, to assess the distribution of the scores at T1 and T2. Skewness and kurtosis scores between ±1 and significance on the Kolmogorov-Smirnov test indicate no or slight nonnormality [38]. We considered floor or ceiling effects to be present if >15% of the participants scored the worst or the best possible score on the subscales [39]. Paired samples t tests were performed to check for any differences between T1 and T2. To study the test-retest reliability, we calculated intraclass correlation coefficients (ICCs). We assumed a correlation of ≥.70 to be satisfactory [40]. Content validity was assessed with a principal component analysis and varimax rotation to examine the fit with the theoretical 7-factor structure of the instrument. We used expectation-maximization imputations for the missing data. The suitability of using factor analysis on the dataset was assessed using Bartlett test of sphericity (P<.05) and the Kaiser-Meyer-Olkin statistic (recommended value of .6) [38]. We considered factor loadings in excess of .71 to be excellent, .63 to be very good, and .55 to be good [37]. Evidence for construct validity was determined by studying Spearman rho correlations between total scores on the DHLI and sociodemographics, (health-related) Internet use, health status, the NVS, and the eHEALS.

Results

Participants

In total, 200 respondents completed the survey at T1. The response rate on the retest survey was 33.5%; 67 respondents completed the DHLI at T2. Table 1 shows the characteristics of the sample populations at T1 and T2. At T1, 53.5% (107/200) were female. Mean age was 46.4 (SD 19.0) years, and the distribution among the 4 age groups was rather equal, with participants between 18 and 34 years old making up 30.0% (60/200); between 35 and 49, 21.0% (42/200); between 50 and 65, 28.5% (57/200); and 65 and older, 20.5% (41/200). More highly educated people were overrepresented, at 41.5% (83/200) of the total sample.

The largest proportion of the respondents used the Internet frequently (see Table 2) and rated their Internet skills as excellent (n=59, 29.5%) or good (n=81, 40.5%). Most respondents accessed the Internet via a mobile phone (n=166, 83.0%), laptop (n=161, 80.5%), personal computer at home (n=115, 57.5%), or tablet (n=113, 56.5%). Of all respondents, 89.5% (n=179) had ever searched the Internet for health- or treatment-related information. Around half had ever read posts on a health-related peer support forum or social media website (n=103, 51.5%) or a health care review website (n=92, 46.0%). A third had ever used a health-related mobile phone app (n=65, 32.5%). Posting self-generated content on the Internet and using treatment-related apps was reported by a smaller proportion of the sample (between 5.5% and 18.0%, see Table 2). Respondents who filled out the survey at T2 did not differ from the total sample on any of the demographic variables, but did report using the Internet more often (t₁₆₃=1.30, P=.02). This suggests that nonresponse bias might have occurred.

Distributional Properties and Reliability of the Digital Health Literacy Instrument

Table 3 shows the scores and internal consistency of the self-report part of the DHLI. The Cronbach alpha is satisfactory, at .87. The Cronbach alpha of the items on each separate skill are satisfactory as well, indicating that these scales can be used as a subscale in the DHLI (alpha range .70-.89). Only the skill protecting privacy had an unsatisfactory Cronbach alpha score (.57). The item-total correlations (not shown in Table 3) were moderate to large for all items (range .51-.73, P<.001), except for the items on the skill protecting privacy, which showed no significant item-total correlation. Respondents had a total mean score of 3.11 (SD 0.87). Total scores were slightly skewed (–1.004) and showed kurtosis (2.251) due to frequent scores between 2.75 and 3.5. However, the Kolmogorov-Smirnov test was not significant (D₂₀₀=.06, P=.06), indicating that the scores are approximately normally distributed. The highest scores on the subscales were reported on operational skills (mean 3.67, SD 0.59), navigation skills (mean 3.30, SD 0.52), and protecting privacy (mean 3.52, 0.52). Operational skills were strongly skewed (–2,388), with a ceiling effect of 60.0% (120/200) scoring the highest possible score, and showed kurtosis (6.220). Privacy protecting skills were slightly skewed (–1.059), with 16.0% (32/200) scoring the highest possible score and no one scoring the lowest possible score. Since the items of the protecting privacy scale were not obligatory to fill in, the response rate on this scale was lower (n=86). The scores of respondents who completed the DHLI at both T1 and T2 did not differ from the total sample at T1 (test statistics not shown in Table 3). Cronbach alphas of the subscales at T2 were satisfactory, ranging from .68 to .88. The test-retest reliability was satisfactory, with ICC=.77 (P<.001) between T1 and T2 on the total scores and levels of agreement of .49-.81 on the subscales.

The Cronbach alpha of the performance-based items was .47, which means that these items did not together form a single construct and should be interpreted as separate items that measure individual skills. Table 4 shows the number of respondents who answered each performance-based item correctly. Most respondents answered the items correctly. Among the more complex skills, the number of respondents with an incorrect answer increased. The only exception was evaluation reliability, which was answered correctly by 94.5% of the respondents (n=188).

Content Validity of the Digital Health Literacy Instrument

Since the performance-based items did not form a scale together, we further determined the content validity of only the self-report scale. Principal component analysis showed a Bartlett test of sphericity of χ²₂₁₀=2278.360, P<.001, indicating that correlations between items were sufficiently large for this analysis. The correlation matrix showed no correlations higher than .9, indicating an absence of multicollinearity. The Kaiser-Meyer-Olkin measure of sampling adequacy was good (.859), which indicates that the sample size was adequate for factor analysis. Two eigenvalues were lower than 1: navigation skills (0.949) and protecting privacy (0.816). The others exceeded 1, ranging from 1.124 to 7.580. In combination, the scales explained 76% of the variance, varying between 8% and 14% among the subscales. Table 5 shows the factor loadings after rotation. The items clustered among the factors as intended, with satisfactory factor loadings. Only item 9 scored below .55.

Construct Validity of the Self-Report Scale of the Digital Health Literacy Instrument

Table 6 shows the Spearman rho correlations between the total score on the DHLI and the other assessed variables. Overall, age showed a moderate negative correlation, indicating that older age is related to lower digital health literacy. The other variables showed low to high positive correlations, indicating that a higher educational level, Internet use, health-related Internet use, better health status (as measured with the RAND-36), health literacy (as measured with the NVS), and eHealth literacy (as measured with the eHEALS) are related to higher digital health literacy skills.

Discussion

Up until now, measurement instruments on digital health literacy skills have measured only competencies related to searching and using online health information (Health 1.0). No instrument has yet been available that also measures the broader range of skills that are essential to using eHealth applications, including more interactive Health 2.0 skills [17,41]. Moreover, the available instruments are self-report assessments, which provide no information on people’s actual competence level [18]. This paper introduces the newly developed DHLI to assess both Health 1.0 and Health 2.0 skills, using self-reportage and performance-based items.

Our results on the nature and scope of our respondents’ health-related Internet use underscore the need for a broad measurement instrument. Whereas searching for health-related information on the Web was still most common (conducted by >90%), more than half of the respondents also reported using health-related social media or consumer review sites. Looking at the measurement properties of the DHLI, it can be concluded that the instrument indeed measures a wide range of digital health literacy skills. The overall reliability of the self-report scale of the instrument can be concluded to be sufficient, with satisfying Cronbach alpha scores and a proper overall test-retest reliability. Only the results on the skill protecting privacy are less convincing, which indicates that this subscale should be further improved. Furthermore, the content validity is good, with the 7 theoretical subscales represented in 7 separate factors, which together explain the largest part of variance. The distribution of the self-report scale can be assumed to be approximately normal, despite some skewness and kurtosis in the total scale and 2 subscales. People in our sample tended to score mostly in the third and fourth quartile of the answer range, meaning that they perceived their skills to be good to very good.

Among the subscales, operational skills showed a high ceiling effect; the largest proportion of our samples (at T1 and T2) scored the highest possible score on this scale. This indicates that the general population does not experience problems in this area, which is not very surprising because this can be seen as the most basic skill in using the Internet. Nevertheless, from previous studies, it is known that a smaller subgroup in the population does struggle with these skills [14,16], which makes it nonetheless relevant to assess these competencies. Further research needs to consider the instrument’s application to other subgroups for which these skills might be less obvious (such as the elderly and less-educated people) due to less computer experience [7,42]. What is remarkable in relation to the operational skills subscale as well is that the majority of our sample accessed the Internet with a mobile phone, and not with a laptop or personal computer. Operational skills require different competencies, since these devices operate in very diverse ways in terms of knowledge of the function of various buttons, using a cursor, and clicking on items. Therefore, a future developmental step should take this into account and add mobile health skills (mHealth) as well.

In order to measure more than people’s perceived digital health literacy skills, we added a performance-based item to each self-report subscale. Together, the performance-based items showed a low internal consistency, which means that the items should be interpreted individually. The low internal consistency could be explained by the diverse nature of the items. As single items they might be usable to detect specific problems in individuals’ competencies. To test this, further research should determine how applicable these items are among subgroups with low digital health literacy skills and what the discriminant value is among these groups. Possibly, the items should be altered to compose more difficult tasks. In our sample most respondents answered the questions correctly, but the more complex the skill, the larger the proportion of the sample with an incorrect answer. The only exception to this trend was the item on evaluating reliability. We measure this skill by asking the respondent where to check the source of the information on a website. Possibly, this question is too easy and does not represent this skill sufficiently. All in all, these items propose a new method to measure actual digital health literacy skills; from here on their applicability should be improved.

Concerning the validity of the DHLI, the correlations between the self-report scale and related variables were as we expected. The relationship between digital skills and both age and education is still present, possibly due to less computer and Internet use [4,43]. This is confirmed by the positive correlations found between digital skills and both Internet and eHealth use. However, the correlation with education is only low, showing a catch-up in skills by the less educated, narrowing this existing gap. This low correlation might be explained by the high availability of the Internet in general in the Netherlands (Internet access is 92% for less-educated people vs 99% for more highly educated people [44]).

The low, but significant, correlations between health status and digital health literacy indicate a relation between people’s skills in using Web-based health care and their actual health. This is interesting, since it indicates the impact that using eHealth can have on people’s lives. However, no conclusions can be drawn on the causality of this relationship from our data, and the associations found with age and education should be taken into account in this context as well, since these variables are also related to health. Previous studies did find a mediating role of digital health literacy on health behavior [3] and a relationship with self-management of health and interaction with physicians [4]. Future research should reveal more on the impact that digital health literacy has on (physical, mental, and social) health and health behavior, and on how these competencies can be influenced or deployed to improve one’s health.

The correlation between the DHLI and health literacy was moderate, which corresponds with a previous study in which a correlation of r=.36 was found between health literacy and digital health literacy [20]. Since digital health literacy comprises both general health literacy and digital skills, a moderate correlation seems appropriate. The correlation with the eHEALS was moderate to high, which shows there is overlap between the 2 instruments, as expected. Still, it also shows that this new instrument partly measures different skills. To further explore the construct validity of the DHLI, we aim to perform follow-up research on the relationship between scores on this instrument and other health-related factors, such as knowledge on health and disease, health behavior, and self-efficacy in health care [3,4,20].

A limitation of this study that should be taken into account is the overrepresentation of more highly educated respondents, which hinders the translation of these results to the general population. Moreover, as stated before, it is particularly interesting to determine the applicability of this instrument among groups at risk for low digital health literacy. This is, therefore, a large implication for further research. A second limitation, related to the performance-based items, concerns the use of 1 format in the formulation of the items. We used print screens from the Web browser Google Chrome; however, naturally many people use other Web browsers and other operating systems (eg, OS X instead of Windows), which intervene with the validity of the items. When the instrument is assessed digitally, an adaptive test could overcome this problem, so participants can first supply information on their browser use, to which the items can be adjusted. With a paper-and-pencil assessment this could also be done when the instrument is used individually (then the suitable version would be handed to the person), but in a (anonymous) research setting, this will be a persistent problem.

All in all, it can be concluded that the DHLI is acceptable as a new measurement tool to assess digital health literacy, measuring 6 diverse skills. Its self-report scale shows proper reliability and validity. The included performance-based items should be studied and adapted further, to determine their value and their discriminant validity. Future research should examine the acceptability of this instrument in other languages and among different (risk) populations and should explore ways to measure mobile health literacy skills as well.

The Digital Health Literacy Instrument, in both Dutch and English, is available and may be used on request via the corresponding author.