Sources of Traffic and Visitors’ Preferences Regarding Online Public Reports of Quality: Web Analytics and Online Survey Results

Introduction

There is unprecedented interest in making information about provider cost and quality of care publicly available. The Affordable Care Act (ACA) expands coverage to millions, with variable levels of deductibles, leading to increased demand from consumers for cost and quality data [1,2]. In addition, the ACA requires insurance exchanges to create websites that provide comparative health plan performance on quality and cost [3]. As physician performance metrics become publicly available under the ACA’s Value-Based Payment Modifier program [4], the hope is that public reporting will drive provider choice and stimulate greater quality improvement efforts among providers to a degree not seen before. And yet, consumers have been slow to use this information to inform choices [5-7]. However, there is evidence that well-designed reports of quality can influence consumers to choose higher quality providers [8]. Because public reporting has the potential to improve quality, multiple stakeholders are interested in understanding how best to reach consumers and how to provide relevant quality information [1]. This study aims to add to that understanding.

Prior work about the users of public reports of quality has focused on specific user segments (eg, clinicians or patients) and predictors of consumer use. Some clinicians use public reports for internal quality improvement [9,10]. A small proportion of US consumers report seeing comparative hospital or physician quality information, although that number appears to be increasing [5,6]. Studies of sociodemographic predictors of physician rating websites for German consumers were mixed regarding whether education, age, gender, or chronic disease predicted use and awareness of the physician rating websites [11-13].

Despite knowledge that people find online information through a variety of routes, there is no peer-reviewed literature of which we are aware about how people find public quality and cost reports (eg, via search engines searches, links on other websites, direct emails) or whether specific traffic sources are associated with overall traffic. In addition, there has been no information gathered in real time from US online visitors to the reports, which limits our understanding of what is relevant to users as they interact with the reports. Lastly, prior work has not described differences in website experience for different visitors (eg, consumers compared to physicians), not their areas of interest. Because the influence of these reports depends on who finds and uses them, improving the reports’ impact will be difficult without understanding how reports currently garner an audience and without knowing the needs of consumer and physician visitors who find the reports.

In order to address this knowledge gap, we used 2 types of data gathered from a group of public reporting websites of hospital or outpatient provider quality: Web analytics data that passively tracks website visitor traffic and behavior, and an online survey of website visitors. We describe overall traffic to the sites, how visitors arrive at the websites, differences between the sites in traffic sources and types of visitors (eg, consumers vs physicians), visitors’ purposes in going to the website, and their experience while there. The objective of this study is to inform transparency efforts by assessing for potential ways to increase traffic to online public reports of provider-level quality and cost as well as meet the needs of the visitors who find them.

Methods

Setting

The Learning Network of Chartered Value Exchanges (CVEs) is a program that has supported transparency since 2004 and is sponsored by the Agency for Healthcare Research and Quality (AHRQ). The CVEs are multistakeholder state or local quality collaboratives that are investing significant resources in online public reports of hospital and physician quality in their communities. The CVEs involve more than 550 health care leaders and represent more than 124 million lives, more than one-third of the US population [14]. Because of the CVEs’ long experience and broad catchment area, we used the network of CVEs for this study.

All 24 CVE’s were invited to participate, with 22 CVE or CVE-affiliated websites active. Of these, 18 websites participated. The websites were public quality reports of hospitals, outpatient providers (medical groups or clinics), or both. Websites reported on providers within a state, a region in a state, or a county. Quality measures for hospitals were commonly Center for Medicaid and Medicare Services (CMS) measures, although some websites also included measures of maternity and neonatal care. Quality measures for outpatient providers were often Healthcare Effectiveness Data and Information Set (HEDIS) measures and Consumer Assessment of Healthcare Providers and Systems (CAHPS) patient experience measures. Narrative patient comments were not shown on any participating website.

Further descriptions of CVE groups and websites are available [14]. Participating websites agreed to participate on the condition that we not identify the websites and their performance individually. Multimedia Appendix 1 shows a screenshot from one of the websites at the time of the study, whose sponsors gave permission to share it.

Website Analytics Data Collection and Search Term Coding

We gathered Web analytics data from February to August 2011 using Google Analytics [15]. We excluded data from the IP addresses of computers of the report sponsor organization and of any external vendors hosting the online reports.

Collected variables were number of unique visitors and percent of unique visitors arriving via 3 methods: search engine queries (“search traffic”), clicking on a link from a different website (“referral traffic”), or directly entering a URL into a Web browser or clicking on a link in an email, word processing document, or document in Portable Document Format (PDF) (“direct traffic”).

The population density varied among the catchment areas the websites served. To generate population-adjusted website traffic from the absolute number of unique visitors, we used Census Bureau catchment area counts of 100,000 Internet-using households as the denominator under total number of unique visitors to calculate “per capita traffic” to each website [16,17].

We also collected search terms (eg, “best doctors San Francisco”) for all search engine traffic. The primary author (NSB) and a research staff member (RAP) organized the search terms into categories using an iterative process: both investigators read the 50 most common search terms for each website then discussed the various categories of search terms to create a codebook for categories. RAP then coded the top 50 search terms for each website and NSB reviewed the initial coding and discussed any code changes. They then combined codes into larger groupings for the final analysis. The codebook and documentation of the coding process is available on request.

We could not link the Web analytics data to survey responses because the analytics data do not include visitor IP addresses and because they are reported in aggregate.

Survey Development and Content

The primary aim of the survey was to provide information on report visitors’ use and perceptions of the value of the public reporting websites. See Multimedia Appendix 2 for the full Checklist for Reporting Results of Internet E-Surveys (CHERRIES) checklist for reporting of Internet e-surveys and for additional description of survey development [18]. We drafted the AHRQ Public Report [19] surveys based on the authors’ expert knowledge about online public reports and drawing from existing surveys that participating public reporting websites were using.

Survey items fell into the following categories: demographics, purpose of the visit, medical topics of interest, and areas of quality measurement of interest. Visitors were shown questions within each topic tailored to their persona type (eg, patient, friend or family member, health care professional, insurer, employer or labor union, researcher, media, lawyer, legislator). We focused on the results for the consumers (patients or friends or family members) and health care professionals in this paper because these were the largest groups of respondents and are the ones most likely to use the reports for choice or for performance improvement. The surveys are available in Multimedia Appendix 2.

Survey Data Collection and Response Coding

We surveyed participating website visitors from February to August 2011. We used an “open survey” design in which all visitors viewing at least 1 page with access to quality performance measures were offered the opportunity to take the survey. The invitation appeared in a pop-up window with directions to take the survey at the end of the session. See Multimedia Appendix 1 for images of invitation and survey.

We framed the survey respondents as a group interested enough in the website content to spend time on an online survey afterward. The implication is that although there might be a low response rate, the responses we received would be from people who are more likely to be potentially influenced by the report. We adopted this frame because low response rates are a known limitation for website surveys because a proportion of website visitors are searching for other content or have limited attention or time for a survey while online. For instance, Kaiser researchers had a 17% response rate in an online survey of users of a secure online personal health record who were presumably more engaged than one-time visitors to a website [20]. One approach to the known low response rate is to invite only visitors who interact extensively with a website to answer a survey, thus creating a smaller response rate denominator. We chose to invite all visitors because some websites have quality information on only 1 or 2 pages.

To decrease response burden, we programmed the survey software to show each consumer (patient or friend/family member) a randomized set of 3 of 5 item groupings (eg, purpose of visit, demographics, topics of interest), leading to smaller consumer sample sizes for each set of questions than if all consumers had answered all items. As noted in our results, the denominators for each of these item sets only included those who were randomized to see those questions.

We coded free-text survey responses using the existing survey item options and categorizing responses that did not fit into an existing option as “other.” We allowed a new category to be formed if it occurred more than 10 times and put the answers for an existing option into “other” if the option was chosen less than 10 times. The only option affected was the consumer primary purpose of visit, for which “learn about a disease” was included in “other.”

Statistical Analysis

Analysis of Traffic to the Websites

We assessed 2 potential predictors of per capita traffic on the websites. We used the same approach in 2 separate models and websites were the unit of analysis. The predictors were the percent of visitors to the website that were from search engines and, for the websites with >15 survey respondents, the percent of consumer respondents to the survey. We estimated linear regression models. In each model, we included a term for report type (hospital only, physician only, or both), assuming that reports with both types might have higher traffic. We performed sensitivity analyses allowing for clustering by website and using a binary variable for report type (1 provider type vs both provider types).

Analysis of Survey Responses

We calculated response rate statistics: view rate (unique visitors shown the survey/all unique site visitors) and participation rate (number of surveys with at least 1 question answered/unique visitors shown the survey) [18]. See Multimedia Appendix 2 for response rate analysis details. We used a t test to compare health care professional and consumer website experience scores. We used chi-square tests to compare health care professional and consumer primary purposes of visit and other areas of interest. As a sensitivity analysis, to assess whether the associations differed by report type (physician or hospital report), we performed the same analyses using survey responses stratified according to whether the survey was answered from a hospital or a physician reporting page.

All analyses were conducted using Stata 12 (StataCorp LP, College Station, TX, USA). The University of California San Francisco Committee on Human Research approved this study.

Results

Website Analytics Data on Volume and Sources of Visitors

For the 18 websites, there was an average of 1440 unique visitors per month per website, with a total of 116,657 unique visitors to all websites. There was substantial variability in website per capita traffic (range 1-167 unique visitors/100,000 Internet users per month; median 31.8, IQR 15.7-47.2).

The websites commonly reported on hospital quality (89%, 16/18), with 61% (11/18) also reporting on clinic or medical group quality (Table 1). There were fewer reports from the Southern region compared to other regions, with reports approximately evenly split through the rest of the regions. Most reports had a state as a catchment area, with all websites reporting at the provider level (hospitals, clinics, or medical groups).

Visitors arrived most often through a search engine query (63.95%, 74,606/116,657 of unique visitors) (Table 2) and less often through referral from another website (15.80%, 18,432/116,657) or direct links received in an email or in an electronic document (19.99%, 74,606/116,657). There was a positive association between percent of unique visitors arriving from search engines and total unique monthly visitors per 100,000 Internet-using households in the catchment area, adjusted for report type (hospital only, outpatient group only, or both), with a 1-point increase in traffic for every 1.8% point increase in proportion of search traffic (P=.002) (Figure 1). The sensitivity analysis that allowed clustering by website returned similar results.

For most websites (61%, 11/18), the search term by which visitors arrived most often was the name of the website. However, collectively, searches for individual hospitals by name were the bulk of the searches that led to visits across all participating websites (Table 2). Among the 2 websites with the highest traffic, responsible for 56.55% (65,967/116,657) of all unique visitors, the hospital name was the most common search term (37.00%, 17,205/46,500 and 97.23%, 9891/10,172 of search terms used), and search was the most common source of traffic (89.00%, 46,500/52,247 and 74.14%, 10,172/13,720 of unique visitors, respectively).

‎View this figure

Survey Data on Consumer and Health Care Professional Visitors

Of all unique visitors (N=116,657), 49,560 were presented the option to take the survey, resulting in a view rate of 42.48%. Of those who viewed the invitation, 1755 responded, resulting in a participation rate of 3.54%. The number of responses from the websites ranged from 2-287 (mean 97.5, SD 98.6; median 49.5, IQR: 26-143).

There were more consumer respondents (850/1755, 48.43%) than health care professional respondents (551/1755, 31.39%). The remaining respondents were members of the media, employers or labor union members, researchers, insurers, or others who chose a free-text option (354/1755, 20.17%). There was wide variation across websites in proportions of respondents who were consumers (21%-71%) and health care professionals (16%-48%). Figure 2 displays that variation and shows ranking according to per capita traffic. Figure 2 illustrates that there is no association between audience composition and website traffic (P=.56 for regression of proportion of consumers and per capita traffic, with similar results in the clustered sensitivity analysis).

More health care professionals than consumers had a primary purpose of choosing or comparing providers (38.8%, 168/433 vs 25.3%, 108/427) (Table 3), whereas more consumers than health care professionals (27.4%, 117/433 vs 20.3%, 88/427) had a primary purpose of finding quality information on a specific provider (P<.001 for overall comparison). For consumers with a primary purpose to “choose providers” who were also asked whether they did so (n=51), 78.4% (40/51) were likely or very likely to use the information to choose a provider. Only 4.2% (18/433) of health care professionals said that they came to the websites for the purpose of patient referral to a hospital or other health care provider (Table 3). Sensitivity analysis found that these patterns were similar by report type (data not shown).

Few providers (0.9%, 4/433) or consumers (2.6%, 11/427) had a primary purpose of looking at cost information. The interest in this information was more common, with both providers (26.6%, 115/433) and consumers (20.8%, 92/442) desiring cost content to be added to the websites (Table 3). Approximately one-third of consumer respondents indicated interest in adding measures about diseases relevant to them (36.4%, 161/442) or adding written comments from other patients (31.5%, 139/442). Consumer respondents rated their experiences using the website lower than did health care professional respondents (mean 6.2, SD 2.7 vs mean 7.2, SD 2.2 on a scale 0-10, P<.001).

Sensitivity analysis found that these differences in primary purpose by respondent type and in mean experience scores were similar in analyses stratified by report type (hospital report vs outpatient quality report, data not shown).

Consumer respondents were commonly middle aged (58.1%, 194/334 were 45-64 years), white (84.4%, 217/257), and many had private insurance (74.1%, 238/321). Additional respondent characteristics are in Multimedia Appendix 3.

‎View this figure

Discussion

Principal Findings

In this study of Web analytics data and real-time survey data from a multistate group of public reports of quality, we analyze variations in sources of traffic across websites as well as audience type and purpose. We found that overall traffic to the sites is low. Per capita Internet traffic varied extensively across sites, with higher traffic on websites associated with higher proportion of traffic from search engines. Most visitors arrived after a Web search, frequently using search terms for a specific hospital. Although both consumers and health care professionals use the websites to assess provider quality and choose a provider, websites varied substantially in the proportion of consumer or health care professional respondents, and consumer respondents had a less positive experience on average than health care professionals. Our findings speak to 2 mechanisms through which a public report can be influential: achieving a larger audience and meeting the needs of the audience that arrives.

Achieving a Larger Audience

The number of visitors to these websites was low (1440 unique visitors/month) compared to a similar site in the United Kingdom, the NHS Choices website, which posts numerical ratings of hospitals and physicians as well as written comments from patients. NHS Choices has 250,000 page views per month (some of which could be repeat visitors) to the pages that show comparative provider performance [21]. To the extent that public reports influence choice [22], leading to better health, or that public reporting can incentivize providers to improve care [23,24], increasing the use of US public reporting websites will be important. The UK experience implies that it is potentially possible to do so.

In our group of websites, the range of per capita traffic was large, with the 2 websites with the highest per capita traffic achieving most of their traffic via search queries. This, taken with the finding that search traffic was >60% of traffic overall and that a higher proportion of search traffic is statistically significantly associated with higher overall traffic, suggests that increasing search engine traffic is a potentially powerful approach to increase traffic overall. This is supported by data that in 2012-2013 there were 97,000 monthly Google searches using keywords related to hospital quality, suggesting that consumers are actively searching for this information [25].

We also found that very few visits to the websites came from searches for specific diseases. A similar finding in the survey was that few consumer respondents chose “learn about a disease” as their primary purpose for their visit. Taken together, these findings imply that the provision of disease-specific information only (eg, sections titled “learn about your diabetes” without performance measures) may not increase traffic to the site.

Lastly, we found that the 2 websites with the highest search traffic and the highest traffic overall had hospital names as the most common search terms leading to their visits. These websites had also “tagged” (ie, made visible to search engines) the hospital names, whereas the other websites did not. Hence, our data suggest that tagging provider names on a public reporting website may increase the chances of the website being found in a search and lead to more visitors.

Visitor Experience and Preferences

The influence of a public report also depends on the types of people who find the report and whether the content meets their needs. Prior work in this area includes assessments of predictors of reported past use of online physician rating websites and willingness to pay for physician rating sites, examining consumer demographics as well as models of underlying factors motivating consumer use of physician rating websites (digital literacy, perceived ease of use of Internet) [12,26,27]. Our findings add to this work by surveying visitors in real time, including visitors to hospital public reporting websites, assessing physician and consumer perspectives separately, and assessing the purpose of website visitors and their areas of interest. These new contributions may be useful to report sponsors who would like to not only understand predictors of use, but also preferences of their current audience in order to meet their needs. If the traffic on these sites grows, report sponsors or researchers could track whether the user composition changes and whether audience preferences change.

We found that consumer survey respondents on these websites were predominantly older, white, and privately insured. The high proportion of privately insured respondents could be due to low response rate among other respondents or due to low numbers of uninsured or publicly insured visitors. Should the latter be true, one implication is that vulnerable populations who have historically received lower quality of care may access quality information for provider choice less often. Prior research found that African American survey respondents were less likely than white survey respondents to report seeing comparative quality information for hospitals and doctors [28], even though once minorities are aware of reports, they are more likely to use the information [28,29]. To the degree that use of quality information drives health outcomes, disseminating the information to vulnerable populations, as some communities are already trying to do [30], is one approach to avoid widening existing disparities. Additional research is needed regarding whether vulnerable populations have a choice of provider and, if so, the most effective way to disseminate and encourage the use of the quality information to this group [11].

Websites vary in the proportion of consumer and health care professional respondents. Berwick and Coye [31] and Hibbard [9] describe differences in consumer and health care professional pathways through which quality reports may stimulate improvement; hence, both audience types are important. Future research could assess for potential differences between the marketing approaches or website features to explain differences in report visitor composition. We did not find a relationship between the proportion of consumer respondents and per capita traffic, implying that the higher traffic sites are not necessarily succeeding in the consumer populations. Our findings on website experience suggest that additional research is needed to improve the experience for both audiences. Our data show that although there is some overlap in interests between consumers and health care professionals, the consumer website experience was worse than the experience of health care professionals. The needs of health care professionals and consumers may be different enough that a single report will not be adequate to effectively drive quality improvement through the separate pathways Hibbard and Berwick and Coye describe, but additional research is sorely needed in this area.

Few survey respondents indicated that the primary purpose of their visit was to obtain cost information, likely reflecting that most websites lacked cost information. However, 21% of consumers and 27% of health care professionals indicated interest in adding cost information. Since we collected our data, the insurance market has evolved with greater use of high deductible plans and more cost shifting to consumers. In this context, consumer interest in costs may increase [1], especially if there is increased awareness of variation among providers in cost of care [32]. Our data suggest that providing cost information on the websites may be an opportunity to meet the needs of visitors to the public reports.

Consumer respondents reported an interest in seeing patient comments. This is similar to prior literature showing that consumers find patient stories to be as persuasive as quantitative assessments of patient experience [33]. Providing patient comments on provider quality may better meet the needs of consumer visitors. Additional research is needed to understand how to best elicit narrative comments and how to display narrative and quantitative data together to facilitate optimal choices [8,11].

Limitations

The survey participation rate was low (3.6%) leading to a risk of nonresponse bias. This is a known limitation for website surveys in real time [20] and the results represent the only available data from recent visitors to online public reports. The nonresponse bias in our survey might be particularly relevant in the younger age groups who are poorly represented in these data and are often poor responders to Internet and other types of surveys [34]. This is supported by our findings in Multimedia Appendix 3 showing that the consumer respondents were older than the population in the catchment areas who reported using the Internet to research health plans or practitioners in 2011. Additional methods to gather data from consumer groups who do not answer surveys may be necessary to complete our understanding of visitors to public reports. Response bias does not affect the Web analytics data.

Conclusions

Under the ACA, there is new support for health care transparency and a unique opportunity to help consumers choose higher quality providers. If public reports of provider performance and cost are to be effective, consumers and health care professionals need to find them and visitors to the sites need to find what they need there. These new data suggest that online performance reports of physicians and hospitals are not frequently found and, when found, that the website experience can be improved for both health care professionals and consumers. Using specific search engine techniques may garner a larger audience. Developing reports that cover a broader set of medical conditions, that include patient comments, or that provide cost information could enable website sponsors to better meet visitors’ needs.