Do Health Care Providers Use Online Patient Ratings to Improve the Quality of Care? Results From an Online-Based Cross-Sectional Study

Introduction

Over the last years, physician-rating websites (PRWs) have become a popular tool to create more transparency surrounding the quality of health care providers in the United States and other industrialized countries [1-3]. They usually provide structural information about a physician’s office as well as online-derived patient satisfaction results [3]. Regarding the latter, the rating systems usually contain both scaled-rating systems (eg, stars, grades) and narrative comments. Although scaled-rating systems with standardized questions present a more structured way to receive answers about different aspects of care [4], free-text commentaries allow patients to express themselves in their own words [5]. The comments are intended to provide a more complete picture of the patient experience with that provider, incorporating emotional reactions and the meaning that patients ascribe to their experiences [4,6]. When comparing those two features, narrative comments are meant to be more powerful [7] because users are drawing more attention to words than to numbers [8]. Furthermore, narrative comments have been suggested as one possibility to provide performance metrics that are more easily understood, raising the willingness of users to provide a substantial feedback and creating a more personal feedback than other rating formats [4,9,10].

In general, PRWs and other established public reporting instruments (eg, the Wisconsin Collaborative for Healthcare Quality [11], the New York State Coronary Artery Bypass Surgery Report [12], or Nursing Home Compare [13]) intend to improve the quality of health care via two different pathways. According to the first pathway (“selection”), health care services are shifted to caregivers who provide better quality of care. However, there seems to be little or no impact on the selection of health care providers by patients and families or their representatives so far [14]. Nevertheless, a previously published cross-sectional study has shown that 65% of PRW users consulted a particular physician based on the online ratings and 52% avoided consulting a particular physician because of the ratings [15]. In addition, the second pathway (“changes”) describes a mechanism by which providers are motivated to improve their quality of care for patients [16]. Regarding results from the latest systematic reviews, it seems that publicly releasing performance data stimulates quality improvement activity by means of offering new services, changing policies, change in personnel, and an increase in quality improvement activities [14,17].

So far, most of the evidence on whether public reporting instruments improve the quality of care comes from the inpatient sector [12,18,19]; little evidence is available in the outpatient sector [8,11,20]. Even further, no evidence is available on whether PRWs have an impact on the quality of care by motivating physicians to improve their quality of care [21]. In addition, little is known about the characteristics of physicians who use online rating websites or for what purposes [22]. Therefore, it is important to gain a scientific understanding of whether online-based rating websites have the potential to contribute to a better standard of care. In this paper, we present the results of an online-based cross-sectional study that examines (1) which health care providers use online rating websites (2) for what purposes and (3) assesses whether health care providers use online patient ratings to improve patient care.

Methods

Design and Data Source

We conducted an online-based cross-sectional study by surveying outpatient physicians and other outpatient health care providers (eg, midwives, speech and language therapists) who have registered on the German PRW jameda (September 2015). These providers either subscribed to a monthly newsletter that contained an overview of all individual ratings posted on jameda over the previous four weeks or booked a jameda service product. On jameda, providers can register for a free-of-charge basic account that permits them to modify their personal or structural office data and ensures that they receive notification of online ratings on a monthly basis, as well as including the possibility of commenting on the patients’ ratings on the website. Three products are offered on jameda that contain different tools, such as special presentation formats of the practice or the uploading of additional data. To date, jameda is likely to play the most important role in the German PRW movement [23]. For our purposes, the Web service provider of jameda sent out a special issue newsletter via email to registered health care providers (N=25,000) that contained an invitation and an online link to participate in the survey. The newsletter also contained some information about the study and its purpose. As an incentive, we held drawings for four Amazon vouchers with a value of €150 each.

We designed the survey by using QuestBack’s Internet-based Enterprise Feedback Suite survey software. The questionnaire contained 26 questions about PRWs in general and consisted of three parts. After collecting sociodemographic information, the second part asked questions about the knowledge and usage of PRWs. In the third part of the survey, we aimed to assess whether and how health care providers react to online ratings and whether they had implemented any of 20 listed measures in order to improve patient care. Those measures were derived from a systematic review regarding the impact of public reporting on the quality of patient care (described subsequently). We distinguished between scaled-survey questions and narrative comments to investigate which are of greater importance to the providers (see Multimedia Appendix 1 for the developed survey instrument.)

Before conducting the study, the questionnaire was piloted by 25 individuals to ensure the proper working of the survey, the randomization process of some questions, and the comprehensibility of the wording. The pretest resulted in minor adaptations to the wording of some questions. We conducted a systematic search procedure in the databases Medline (via PubMed) and the Cochrane Library (May 2015) to identify studies that have researched the impact of public reporting on the provider behavior with respect to the implementation of quality measures. Our search yielded 12 studies that were related to the outpatient (n=3) or inpatient sector (n=9). However, no study had investigated the usage and impact of online rating websites on provider behavior.

Data Analysis

Results are presented as both mean and standard deviation for continuous variables and as numbers and percentages for categorical variables. We performed comparisons between more than two groups by using a chi-square test (two-sided) for categorical variables and applying the Kruskal-Wallis test for continuous nonparametric variables. The Shapiro-Wilk test was applied to examine the normality of the data distribution. Multilevel multivariate logistic regression models were used to assess how much of the providers’ reaction to implement measures could be explained by demographic and online report card-related variables. The dependent variable indicated whether a health care provider had implemented any measure(s) to improve patient care (yes/no). Regarding the independent variables, we used the following sequence of models: (1) adjusted for demographics (age, gender, marital status, Internet use, medical specialty), (2) adding information regarding any booked jameda service products (yes/no), and (3) adding online report card-related issues (ie, frequency of use, appraisal of the trustworthiness of the online ratings). All statistical analyses were conducted using SPSS version 22.0 (IBM Corp, Armonk, NY, USA). Observed differences were considered statistically significant if P<.05.

Results

In total, our final study sample consisted of 2360 respondents who completed the survey (response rate=9.44%, 2360/25000; completion rate=67.29%, 2360/3507). We excluded 49 participants from subsequent analysis because of extremely short answer times and/or inconsistent answer patterns. The completed online surveys took a mean 9.63 (SD 9.03) minutes. The overall mean age was 49.63 (SD 8.69) years, 66.67% (1560/2340) respondents were male, and the mean duration of practice in the doctors’ office was 12.99 (SD 9.10) years (Table 1).

Regarding the medical specialty, 17.50% (413/2360) of the surveyed sample were general practitioners, 69.36% (1637/2360) were specialists, and 13.14% (310/2360) were other health care-related professions (eg, midwives). The three groups did not differ significantly in terms of age and Internet use, but did in terms of gender, marital status, and years of practice in the office.

Awareness and Usage of Online Rating Websites

Most providers became aware of the websites through the Internet (71.91%, 1697/2360), contact with the providers of PRWs (20.08%, 474/2360), or advertisements (16.74%, 395/2360) (Table 2). Differences regarding the sources of awareness between the three provider groups (ie, general practitioners, specialists, others) could all be shown to be statistically significant (P<.05 each), except for recommendations by friends and relatives as well as others. Furthermore, specialists used PRWs more frequently than general practitioners and other providers did (P<.001). Most providers (87.08%, 2055/2360) used online rating websites to read comments for their individual practice. Almost half of all providers read comments for other providers (48.69%, 1149/2360) and slightly more than one in three providers (35.97%, 849/2360) to know which measures might be implemented to increase patient satisfaction. In addition, 12.08% (285/2360) of respondents stated that they used online ratings for referring patients to other providers. Therefore, numbers for general practitioners (12.1%, 50/413) and specialists (10.38%, 170/1637) were significantly lower than those for other providers (21.0%, 65/310, P<.001). In addition, specialists evaluated their ratings significantly more often than general practitioners and other providers did (P<.001). Most providers evaluated the ratings themselves (84.87%, 2003/2360).

Implemented Measures to Improve Patient Care Because of Online Ratings

Overall, 54.66% (1290/2360) of respondents stated that they had used online ratings to implement some measure(s) to improve patient care (Tables 3 and 4). Numbers for specialists (57.79%, 946/1637) were significantly higher than those for general practitioners (50.1%, 207/413) and other providers (44.2%, 137/310, P<.001). The most widely implemented measures were related to communication with patients (28.77%, 679/2360) or the appointment scheduling process (23.60%, 557/2360). Staff-related measures also played an important role; 10.38% (245/2360) of respondents stated that they had invested in further training of staff and 5.89% (139/2360) had recruited additional staff. Scaled-survey results seemed to have a greater impact on developing measures to improve patient care. The results showed that half of respondents (50.08%, 1182/2360) developed measures from scaled-survey ratings, whereas only 43.90% (1036/2360) stated that they had used information from narrative comments. In line with these results, specialists made significantly more use of the information in both scaled-survey ratings and narrative comments (53.21%, 871/1637 and 47.16%, 772/1637, respectively) than general practitioners (44.6%, 184/413 and 41.7%, 172/413, respectively) and other providers (41.0%, 127/310 and 29.7%, 92/310, respectively; P<.001).

Table 5 presents the results for implementing measures to improve patient care according to the medical specialty (here shown for medical disciplines with at least 20 providers and the seven most frequently implemented measures). As displayed, ophthalmologists (68%, 40/59) and gynecologists (65.4%, 123/188) were most likely to implement any measure to improve patient care. In contrast, the lowest percentages were calculated for psychiatrists (38%, 29/77) and pediatrics (40%, 16/40). Thereby, the mean number of implemented measures (mean 3.06, SD 2.29) was calculated to range between 1.81 (SD 1.05) for pediatrics and 4.29 (SD 3.05) for urologists, respectively. The association between the two variables could be shown to be marginally significant (Spearman P=.47 and P=.07, respectively).

Multilevel logistic regression models were performed to ascertain the effects of providers’ demographics as well as report card-related variables on the likelihood that providers implemented measures to improve patient care (Table 6). Thereby, the dependent variable indicates whether a health care provider had implemented any measure(s) to improve patient care (yes/no).

All models revealed medical specialty to be a significantly associated predictor (P<.001 each). Thereby, the higher odds for specialists were proven to be statistically significant in our baseline model (AOR 1.31, 95% CI 1.04-1.66, P=.03). In contrast, the lower odds for other providers were proven to be statistically significant in our more comprehensive models and ranged between AOR 0.59 (95% CI 0.41-0.85, P<.001) and AOR 0.68 (95% CI 0.48-0.96, P=.03). In addition, the frequency of report card use (P<.001) and the appraisal of the trustworthiness of scaled-survey ratings (P=.02) were determined to be significantly associated predictors. Further regression analyses were run to determine whether the results differed when related only to implemented measures because of scaled-survey ratings or narrative comments (see Multimedia Appendices 3 and ). As presented, the results could be shown to be very robust in terms of significantly associated predictors.

Discussion

The aim of this study was to examine which health care providers use online rating websites and for what purposes, as well as to investigate whether health care providers use online ratings to improve their quality of care. Therefore, we conducted an online-based cross-sectional study by surveying 2360 physicians and other health care providers. Our surveyed sample is slightly younger and contains a lower percentage of female respondents than providers who are registered on jameda or those who work in the German outpatient setting, respectively [24]. However, the distribution of our surveyed physicians across Germany as well as the medical specialties is similar to all physicians in the German outpatient setting (see Multimedia Appendix 5). Nevertheless, our sample may not be representative of all providers in the German outpatient setting despite the similarities in its characteristics. First, compared to the German outpatient setting, approximately 8% (25,000/304,818; 147,948 physicians, 69,370 dentists, and 87,500 other providers) of all providers were invited to participate in this survey, of whom slightly more than 9% completed the survey. Second, providers may differ in further characteristics among those who are registered on jameda and those who are not. For example, those less interested in online-related topics (eg, those without a practice website), or PRWs in general, are less likely to be represented by our results.

Our results show that more than half of the surveyed providers (55%) use online ratings to develop measures to increase the quality of patient care. This result is lower than the findings by Friedberg et al [25], who reported that 83% of interviewed physician group leaders used patient experience to improve the performance in the US outpatient setting. Furthermore, Smith and colleagues [20] showed an increase in the percentage of providers that implemented at least one of 22 possible diabetes improvement interventions from 75% to 94% after the onset of public reporting. The study approach here was quite similar because in both studies a literature-based list of possible interventions was presented. Thereby, the mean number of implemented measures to increase patient care in our study (mean 3.1, SD 2.3) was lower than in the study by Smith et al (mean 8.7, SD 4.5) [20].

The most widely implemented quality measures in our study are related to the communication with patients (28.77%, 679/2360), the appointment scheduling process (23.60%, 557/2360), and the office workflow (21.23%, 501/2360). These findings are in line with the results of Friedberg et al [25], who identified the change in the office workflow (eg, procedures for handling test results or incoming mail) as the most widely implemented initiative (70%), much more common even than in our study. The improvement of the appointment scheduling process was similarly reported by 27% of providers. Other common implemented quality measures in the Friedberg et al study (eg, training nonclinicians: 57%; reassigning staff responsibilities: 45%; or hiring or firing clinicians or staff: 36%) were higher in the US studies. Nevertheless, staff-related measures were implemented by approximately 16% of all respondents in our survey and thus do also play a significant role in the German context. Although 10.38% (245/2360) of the respondents have invested in further training of staff, 5.89% (139/2360) have recruited additional staff, and 3.31% (78/2360) have dismissed staff as a consequence of the online ratings.

Another study showed the adoption of guidelines (87%) and the introduction of patient reminders (82%) to be the most frequently introduced quality measures [11]. Results from our study are far below those percentages and were shown to range between 4% (introduction of guidelines) and 8% (introduction of patient reminders). It is likely that differences in the surveyed samples (ie, individual physicians vs physician group leaders) may account for some of the difference. For example, physician groups in the study of Friedberg et al [25] represent all physicians who work at one or more practices with at least one group-level manager. This manager coordinates the correspondence with health plans and controls the performance of physician groups [25]. Furthermore, successful physician groups share a unified physician-led network, an infrastructure to support group performance, and incentives to engage individual physicians toward group goals [26]. Consequently, these physician groups might be more active in implementing quality measures due to an established infrastructure and developed competencies. In contrast, our sample comprises individual physicians who may not have the infrastructure or the competencies to introduce quality measures on such a large scale. Another study does not present detailed information about the frequency of implemented measures, but states several measures that have been widely implemented, such as the use of phone and mail patient reminders about existing upcoming appointments or to schedule follow-up tests, guideline-based reminders for providers at appointments about services that their patients should receive, or the adoption of care guidelines [20].

As shown, most providers use the websites for reading comments for their individual practice (87.08%, 2055/2360). Of those, 56.69% (1165/2055) stated that they had implemented quality measures because of the online ratings. This also means that 43.31% (890/2055) of those providers did not implement any quality measure(s), possibly because of a lack of time or trust in the online ratings. Others might find it challenging to develop quality measures at all because anonymous ratings sometimes do not provide enough information to learn about quality deficits [27]. Reading other physicians’ ratings out of interest (49%) provides an opportunity for physicians to draw comparisons. This may also have a positive impact on the overall quality of care because some physicians are becoming engaged in implementing quality measures in order to perform better than their colleagues. According to our numbers, 12.08% (285/2360) of the respondents use online ratings for referring patients to other providers. This result exceeds those of other studies, which determined that physicians do not use publicly available quality information [28] or change their referral practices due to public reports [29,30]. The remaining providers might not use online ratings for referral decisions because subjective patient ratings do not accurately display the quality of health care providers [31,32].

We were able to show further that scaled-rating systems seem to have a greater impact on implementing quality measures than narrative comments in absolute terms. This might be because scaled ratings systems offer a more structural approach to rate health care providers in comparison to narrative comments [4]. Therefore, physician-rating systems provide standardized categories that more promptly enable the indication of quality deficits. However, even though narrative comments provide a more complete picture of the patients’ experiences with the health care provider [6], they are more complicated and time-consuming to analyze. For example, it might be easier to determine deficits such as long waiting times for an appointment out of scaled-rating systems. This is because most scaled-rating systems contain this issue, but only 13% of narrative comments address this topic [33].

Finally, even though our results could demonstrate positive effects of public reporting on patient care, several unintended effects should be regarded for the special case of PRWs. For example, public reporting might reduce the access to care because health care providers tend to avoid high-risk patients and prefer low-risk patients (“cherry picking” or “cream skimming”) [29,34]. This effect is likely to be even greater in the case of online rating websites compared to traditional reporting instruments (see previous) because no risk-adjustment of online ratings is carried out [3]. Furthermore, Lindenauer et al [35] mentioned the neglect of more important aspects of care. So far, all PRWs in Germany, and most in the United States, solely contain patient satisfaction as a performance measure. Results for guideline-based treatment (eg, for chronic conditions such as type 2 diabetes ) are not presented on many report cards, meaning ratings for physicians are only based on patient satisfaction scores. From the perspective of a physician, this might be challenging because most (narrative) ratings concern the friendliness and general impression of a physician rather than the medical outcomes [33]. For instance, a physician might neglect the patients’ desire for prescribed antibiotics in the case of a nonbacterial infection. Even though this behavior is correct from a medical point of view, patients might experience this differently and leave a low rating for a physician. Lindenauer et al [35] also stated the unintended risk of an increase in performance rates through coding and documentation (“gaming”). So far, this risk might be of lower importance on most reporting websites because no secondary data are used (which might be gamed). However, online rating websites are vulnerable to other gaming aspects [3,27]. For example, physicians may rate themselves, leave a low rating for their colleagues, or ask the more satisfied patients to leave a rating [27].

Our results demonstrate that health care providers in Germany use PRWs for reading comments for their individual practice. More than half of those providers stated that they had implemented quality measures in order to improve patient care. The most widely implemented quality measures are related to communication with patients, the appointment scheduling process, and office workflow. In addition, staff-related measures also play a significant role in the German context. Scaled-survey results seem to have a greater impact on deriving measures than narrative comments. This might be because the latter are more complicated and time-consuming to analyze. Thus, our results confirm findings from the United States showing that online rating websites may have a positive impact on patient care.

There are some limitations that have to be considered when interpreting the results of this study. First, this study applied a cross-sectional design. Thus, we were able to identify an association between exposure and outcomes, but could not infer cause and effect. Next, even though our sample is representative of those physicians who are registered on the largest German PRW, representation of all practicing health care providers in the outpatient setting in Germany is not achievable (eg, those less interested in online-related topics, those who do not want to deal with this topic in general). In addition, some participants in our sample might be more familiar with Internet-related topics, which may account for some of the high awareness and usage levels of rating websites. Furthermore, the participants were recruited online by the provider of jameda. This means that we only surveyed providers who are active on jameda. Thus, our findings cannot be generalized for providers on other rating websites. For our regression analysis, we used whether providers have implemented any quality measure or not as the binary dependent variable. As shown previously, there were severe differences in the extent of the implementation between the individual measures. Therefore, results might be different if we used the implementation of the individual measures as the dependent variable.