One Decade of Online Patient Feedback: Longitudinal Analysis of Data From a German Physician Rating Website

Introduction

Feedback from patients is an essential element of a patient-oriented health care system [1]. Patients’ views and opinions on the care they have experienced can help health care organizations and professionals identify areas that need to be improved and can also help other patients with decision making when choosing where to receive health care [2]. Health care organizations and professionals can gather patient feedback in a variety of ways, including by conducting patient surveys, audits, interviews, focus groups, and deliberative events [3]. Patients have also always been able to actively share their views and opinions about the care they received with family and friends or with health care organizations and professionals via unsolicited comments or complaints. However, patients increasingly also have the ability to share their views and opinions on the internet and social media [4-7].

Physician rating websites (PRWs) are one of the key opportunities for patients to provide feedback online [4,7]. A systematic search of PRWs in 2018 identified 143 websites from 12 countries; however, the majority of websites were commercially operated in the United States and Germany [8]. Previous research involving PRW ratings in Germany and other countries has highlighted some common themes, including incomplete lists of physicians, a low number of physicians rated, a low number of ratings per physician that are overwhelmingly positive, and unstructured and different rating systems, which has raised concerns about the representativeness, validity, and usefulness of feedback on PRWs [7,9-30]. Medical associations have also often expressed strong opposition to PRWs, concerned that they will be used for doctorbashing or defamation [31,32]. Countries have different legal frameworks with regards to data protection, and previous research suggests that restrictive legal environments (eg, Switzerland) may be having an impact of the types of ratings on PRWs [28,29]. However, the legal basis for PRWs in Germany is reasonably liberal and well established. The Federal Court of Justice of Germany confirmed in 2014 the permissibility of ratings on the basis of the right to freedom of expression and that the anonymity of raters can only be lifted in exceptional cases [33,34]. Research also indicates that PRWs in Germany are having some success in influencing patient decision making and quality improvement [17,35].

However, most studies examining PRWs ratings have typically focused on a certain year (eg, [13,18,21]), a certain medical specialty (eg, [22,23,36-40]), certain cities or regions (eg, [14,26,41]), or with a (more or less) randomly selected sample of physicians or ratings (eg, [14,21,26,36,41]). There is therefore a need for a more comprehensive examination of PRW ratings, to reveal a more generalizable view of ratings and allow trends in rating habits to be identified. As far as we are aware, only 2 studies from the United States [13] and Canada [27] have presented such findings.

This study takes a different approach from most previous studies and analyzes an entire decade of online ratings for all medical specialties on the German PRW, jameda [14,21,26,42] (Please note that the data are not publicly available but may be provided from the provider of the website for research purposes upon request.). Jameda was founded in 2007 and since 2016, has been a wholly owned subsidiary of Burda Digital GmbH. The commercial website provides users with a categorized search function to find suitable physicians, the ability to make appointments with physicians online, the possibility to have video consultations with physicians, an encyclopedia with information from experts on health topics, and an opportunity to rate physicians on a predefined grading system or leave narrative comments. In Germany, a total of 25 PRWs have been identified [8]; however, previous research has indicated that jameda is the German PRW with the highest public awareness, usage, and number of ratings given [4,14,26].

The aim of this study was to examine how ratings posted on the German PRW jameda have developed over the past decade. In particular, it aimed to explore (1) the distribution of ratings according to time-related aspects (year, month, day of the week, and hour of the day) between 2010 and 2019, (2) the number of physicians with ratings, (3) the average number of ratings per physician, (4) the average rating, (5) whether differences exist between medical specialties, and (6) the characteristics of the patients rating physicians.

Methods

Overview

All scaled-survey online ratings that were posted on jameda between 2010 and 2019 were provided by jameda. Ratings on jameda are given according to the 6-point grading system used in German schools (1=very good, 2=good, 3=satisfactory, 4=fair, 5=deficient, and 6=insufficient) [24], in relation to 5 questions: (1) satisfaction with the treatment provided by the physician, (2) the physician’s explanation about the illness and treatment, (3) the relationship of trust with the physician, (4) the time the physician spent with the patient, and (5) friendliness of the physician. Additionally, a mean score (“overall performance”) is calculated based on the results for Q1 to Q5 [24]. The data also contained the physician’s year of birth and medical specialty, as well as the rating patient’s gender, age, and health insurance status.

Statistical Analysis

Descriptive statistics included means and SDs for continuous variables as well as numbers and percentages for categorical variables. To analyze whether differences existed between 2 groups, the Mann-Whitney U test was used for continuous nonparametric variables, and the Kruskal-Wallis test was applied to determine differences between more than 2 groups. The Shapiro-Wilk test was used to examine the normality of the data distribution. Cohen d was calculated to measure the magnitude of the effect size by comparing the standardized difference between the means of 2 groups. All statistical analyses were conducted using SPSS version 22.0 (IBM Corp, Armonk, NY). Differences were considered to be significant if P<.05 and highly significant if P<.001.

Results

Distribution of Ratings and Mean Ratings

In total, 1,906,146 ratings were posted on jameda between 2010 and 2019 (see Table 1). The highest proportions of ratings were left in 2017 (293,744/1,906,146, 15.41%) and 2018 (292,721/1,906,146, 15.36%). In 2019, there was a decline in the number of ratings (232,739/1,906,146, 12.21%) in comparison with the previous years. Ratings were distributed throughout the months of the year relatively equally (minimum in December: 143,620/1,906,146, 7.53%; maximum in March: 173,865/1,906,146, 9.12%), but more variation was found by day of the week (minimum on Saturdays: 123,024/1,906,146, 6.45%; maximum on Tuesdays: 356,128/1,906,146, 18.68%) and by hour of the day (minimum during 3-4 am: 4659/1,906,146, 0.24%; maximum during 11-12 am: 152,606/1,906,146, 8.00%). Likewise, the mean ratings were relatively similar across years (minimum in 2019: mean 1.71, SD 1.52; maximum in 2013: mean 1.83, SD 1.56), months (minimum in January: mean 1.73, SD 1.49; maximum in August: mean 1.77, SD 1.54), and days (minimum on Sunday: mean 1.68, SD 1.45; maximum on Monday: mean 1.78, SD 1.54). However, more variation could be seen by hour of the day (minimum during 7-8 am: mean 1.67, SD 1.43; maximum during 2-3 am and 3-4 am: mean 2.05, SD 1.75 and mean 2.05, SD 1.72, respectively).

Number of Rated Physicians and Ratings Per Rated Physician

Between 2010 and 2019, a total of 127,921 physicians were rated on jameda (see Table 2). The number of rated physicians increased constantly from 19,305 in 2010 to 82,511 in 2018. In 2019, the number of rated physicians decreased to 73,071 rated physicians. The number of ratings that rated physicians received demonstrated an increasing trend. In 2010, 66.94% (12,923/19,305) of all rated physicians were rated only once, 30.88% (5961/19,305) were rated 2-5 times, 1.71% (330/19,305) were rated 6-10 times, and 0.47% (91/19,305) were rated 11-50 times. In 2019, 40.84% (29,843/73,071) of all rated physicians were rated only once, 46.89% (34,262/73,071) were rated 2-5 times, 8.21% (5998/73,071) were rated 6-10 times, 3.93% (2875/73,071) were rated 11-50 times, and 0.13% (93/73,071) were rated more than 50 times. Over the entire decade, 11.43% (14,625/127,921) of all rated physicians were rated once, and 4.23% (5413/127,921) were rated more than 50 times. Please note that the overall numbers cannot be summed up here. For example, one physician received 1 rating in 2010, 3 ratings in 2011, 5 ratings in 2013, 1 rating in 2015, 11 ratings in 2015, 23 ratings in 2017, and 19 ratings in 2019. In sum, this physician was rated 63 times and would be assigned to the category “≥51 Ratings.” Similarly, the overall average number of ratings per rated physician increased from 1.65 (SD 1.56) in 2010 to 3.19 (SD 4.69) in 2019. Comparing the number of ratings and rated physicians with the total number of physicians in the German outpatient sector [43], in 2010, 13.64% (19,305/141,461) of all physicians had been rated on jameda, 21.93% (31,335/142,855) in 2011, 29.22% (42,089/144,058) in 2012, 36.36% (53,065/145,933) in 2013, 42.71% (63,182/147,948) in 2014, 45.56% (68,392/150,106) in 2015, 50.51% (76,773/151,989) in 2016, 51.69% (79,799/154,369) in 2017, and 52.46% (82,511/157,288) in 2018 (see also Multimedia Appendix 1). Thus, more than half of all German physicians have been rated online on jameda each year in Germany since 2016.

Rating Evaluations

Of the 1,906,146 ratings posted between 2010 and 2019, 75.16% (1,432,624/1,906,146) of all ratings were in the best rating category of “very good,” and 5.66% (107,912/1,906,146) of the ratings were in the lowest category of “insufficient” (see Table 3). Furthermore, the percentage of ratings on both ends of the rating scale increased over time, from 71.95% (2010) to 78.17% (2019) for very positive ratings and from 3.91% (2010) to 6.12% (2019) for very negative ratings. However, the overall average rating remained relatively constant. The average rating was 1.73 (SD 1.42) in 2010 and 1.71 (SD 1.52) in 2019, with an overall average of 1.76 (SD 1.53).

With regards to the correlation between the average rating of a rated physician and the number of ratings per physician, physicians with a lower number of ratings tended to receive ratings across the rating scale, while physicians with a higher number of ratings tended to have better ratings (see Figure 1). Physicians with a single rating had a mean rating of 1.58 (SD 1.28). Afterwards, mean ratings get worse with increasing number of ratings. Physicians with between 21 and 50 online ratings received the worst ratings (mean 1.95, SD 0.84). Mean ratings then improve, with physicians having 51-100 ratings receiving a mean rating of 1.79 (SD 0.86) and physicians with more than 100 ratings receiving the best ratings (mean 1.34, SD 0.47; see Table 4).

‎View this figure

Ratings by Medical Specialty

Between 2010 and 2019, general practitioners (343,242), internists (266,899), gynecologists (232,914), and orthopedists (229,481) received the highest number of ratings, while pediatricians (87,330), ophthalmologists (79,699), and urologists (63,703) received the lowest number of ratings (see Table 5). However, according to the relative distribution of ratings, the most frequently rated medical specialties in 2018 were orthopedists (6160/7302, 84.36%); oral maxillofacial surgeons (1017/1257, 80.91%); ear, nose, and throat (ENT) specialists (3559/4479, 79.46%); and dermatologists (3562/4632, 76.90%). In contrast, the least frequently rated medical specialties were radiologists (863/4078, 21.16%) and anesthesiologists (601/4247, 14.15%; see Multimedia Appendix 2). Among the 10 most frequently rated medical specialties, the best rated medical specialties were urologists (mean 1.50, SD 1.29), general practitioners (mean 1.64, SD 1.40), and internists (mean 1.68, SD 1.45). The lowest ratings were given to pediatricians (mean 1.92, SD 1.62), ophthalmologists (mean 2.06, SD 1.74), and dermatologists (mean 2.11, SD 1.77).

Characteristics of Raters

The rating patients were mostly female (56.8%), between 30 and 50 years old (42.6%), and covered by Statutory Health Insurance (81.0%; see Table 6). However, there were some significant differences between genders, age groups, and health insurance status. Male patients gave significantly more favorable ratings than female patients (mean rating 1.61, SD 1.32 vs. mean 1.77, SD 1.48; P<.001). Older patients also gave significantly better ratings than younger patients (P<.001). For example, patients aged 51 years or older left a mean rating of 1.52 (SD 1.22), whereas patients aged 29 years or younger left a mean rating of 1.93 (SD 1.59). Finally, patients covered by private health insurance (mean rating 1.43, SD 1.11) gave significantly more favorable evaluations than did patients covered by statutory health insurance (mean rating 1.75, SD 1.47; P<.001). Nevertheless, effect sizes were small for all groups, varying between 0.114 and 0.289.

Discussion

This study is one of the most comprehensive analyses of PRW ratings conducted to date and has resulted in a number of key findings: (1) just under 2 million ratings were posted on jameda between 2010 and 2019; (2) a total of 127,921 physicians were rated; (3) the overall average number of ratings per rated physicians nearly doubled; (4) three-quarters of all ratings were in the best rating category of “very good,” and the overall average rating remained relatively constant; (5) general practitioners, internists, gynecologists, and orthopedists were the most frequently rated medical specialties; and (6) the rating patients were mostly female, between 30 and 50 years old, and covered by Statutory Health Insurance.

The findings of this study confirm previous research in Germany that indicated that patient ratings show an increasing trend over the past decade [26]. For example, the percentage of all German physicians that had been rated on jameda increased constantly over time from 13.65% (19,305/141,461) in 2010 to 52.46% (82,511/157,288) in 2018. McLennan et al [26] also previously reported that the proportion of physicians from a sample of 298 randomly selected physicians from Hamburg and Thuringia that had been rated at least once had increased between 2010 (range 3.3%-27.8%) and 2014 (range 16.4%-83.2%). Similarly, the average number of ratings per physician also increased between 2010 (range 1.1-3.1) and 2014 (range 1.2-7.5). However, this study only used a small sample from 2 regions in Germany. Overall, there is little international evidence showing the exact development of online ratings over time, which makes it challenging to compare our numbers with those from other similar studies. To the best of our knowledge, more recent studies providing detailed information on a yearly basis are limited. However, 2 studies from the United States [13] and Canada [27] have presented similar findings. First, in 2012, Gao and colleagues [13] showed an increase in the number of rated physicians on RateMDs in the United States from 2475 in 2005 to 112,024 in 2010. Second, Liu and colleagues [27] analyzed a dataset from RateMDs, which included all physicians in Canada in 2018 and showed an increase in the number of ratings for physicians in Canada from 138 in 2005 to 640,603 in 2013. Nevertheless, it should be noted that this study found a plateau in the total number of ratings between 2017 (293,744) and 2018 (292,721). In 2019, a decrease of around 20% in the total number of ratings was seen in comparison with the previous 2 years. In recent years, jameda has implemented and promoted new features on its website (eg, making appointments, video consultations). This has possibly led to lower marketing efforts for collecting online reviews and may also lead to differences from PRWs not offering these addition services. Future studies should investigate whether this latest development can also be observed for other PRWs in Germany and other countries.

This study only provides information regarding jameda. Previous research has demonstrated much lower numbers of both ratings and rated physicians on other German PRWs [4,26]. For example, McLennan and colleagues [26] reported that between 16.4% and 71.1% (mean 41.4%) of physicians were rated on German PRWs overall, compared with 83.2% on jameda. Another study also showed a higher percentage of rated physicians on jameda (90.2%) compared with other relevant German PRWs (32.4% to 61.2%) [4]. Differences in the number of ratings between PRWs can also be shown in the international setting. For example, Trehan and colleagues [44] analyzed online ratings for 250 hand surgeons from the American Society for Surgery of the Hand member directory from 3 PRWs in the United States (HealthGrades, Vitals, RateMDs). Large differences were reported regarding the average number of ratings (13.4, 8.3, and 1.9, respectively) [44]. Further research is required to confirm that this increase in ratings is also true for other PRWs as well.

Furthermore, the percentages of ratings on both ends of the rating scale have increased. This may suggest that a “bimodal” trend in ratings is emerging on jameda, similar to that seen with the rating of products on websites like Amazon where “amateur” reviewers usually only leave a review because they either love or hate a product [45]. It would be helpful if future research examines if this trend continues and can be found on other PRWs, particularly as this trend is usually not seen on PRWs [26], despite qualitative research in Germany finding that a very positive or very negative experience in the health care relationship is a crucial precondition for patients to be willing to rate a physician [46].

Seven years after the first study on online patient ratings on jameda [18], general practitioners, internists, and gynecologists still receive the highest number of ratings in absolute terms. This does not seem surprising due to the high number of physicians in those medical specialty areas in Germany. Similar to previous research [18], it could also be shown that urologists, general practitioners, and internists were likely to receive more favorable ratings on jameda. In contrast, ophthalmologists and dermatologists are still likely to receive far less favorable ratings. This is also in line with the comprehensive analysis by Liu and colleagues [27] from Canada. Previous research findings have also reported that generalists are more likely to have better online ratings than specialists [10,13]. Qualitative research conducted in Germany by McLennan et al [46] found that factors concerning the physician-patient relationship to be some of the most important influencing people’s willingness to rate their physician on PRWs. It is likely that differences in patients’ relationships with physicians in various specialties (eg, duration and frequency of contact and the resulting level of trust) is a key factor for this heterogeneity.

The analysis of such a large number of ratings has also provided a more detailed picture of the association between the number of ratings a physician has and their overall evaluation. Although physicians with only 1 rating tended to have very good ratings (81% of all ratings were in the best rating category), this might potentially be explained, at least in part, by “fake ratings” left by physicians themselves or people connected to the physician. Regardless, it certainly calls into question whether results based on a single rating are meaningful at all [7]. Afterwards, more critical rating results were found. In line with previous studies from Germany [18] and the United States [37], the total performance range was found for physicians with a lower number of ratings. This possibly represents a more realistic picture of patient feedback because the percentage of ratings in the very best rating category declined constantly, and it is also likely that those physicians are not using PRWs as a marketing measure to collect a very high number or ratings [18]. However, in contrast to previous research, physicians who received a higher number of ratings were shown to have better ratings. When there were more than 51 ratings, ratings started to improve again, and physicians with more than 100 ratings received by far the most favorable ratings. It is likely that physician with more than 100 ratings are aware of PRWs and are using them as a marketing tool, potentially specifically asking satisfied patients to leave a (positive) rating on a PRW. However, it is possible that these physicians are simply providing outstanding quality of care, leading to the very favorable ratings on PRWs and, subsequently, more patients choosing to use this physician [18]. Future research should examine which assumption is true [18].

In 2019, Pike et al [37] reported a U-shaped relationship between the number of ratings and the overall rating from the Healthgrades website. A negative relationship between the number of ratings and the overall rating could be seen until physicians achieved 21 ratings; thereafter, a positive relationship was seen. It should be noted that, in contrast to jameda, a lower score on Healthgrades means a worse rating (1=poor; 5=excellent). Although regression analysis on the jameda data did not find a satisfying fit, the study provides further broad-scale evidence on the relationship between the number of ratings and the overall evaluation as discussed earlier in this manuscript.

Limitations

The key limitation of this study is that it analyzed online ratings from only a single German PRW, jameda. Although jameda has shown to be the most frequently used German PRW, there are a total of 25 PRWs in Germany [8], and it is unclear how generalizable the results are to other German PRWs or to other countries. In Germany, it would be particularly helpful for future longitudinal research to examine trends in ratings on PRWs run by public health insurers, as previous research has indicated that these PRWs have been able to quickly establish themselves as some of the most used German PRWs alongside jameda [26]. Another limitation of the study is that it only analyzed publicly available ratings; it is not known how many additional ratings jameda received but did not publish or what efforts jameda made to check whether published ratings are genuine and not fake. Indeed, jameda has often been criticized with regards to the number of fake reviews and its business model that offers physicians paid premium profiles. Recent research has raised concerns that online patient feedback is being inappropriately manipulated by many PRWs and that business models that make PRWs reliant on paying physicians may create financial incentives to suppress negative feedback [47]. Although further work is needed on criteria for determining which feedback is published [47], it is also important to have a comprehensive understanding of the ratings that are being viewed by the public on PRWs.

Conclusion

In conclusion, it can be stated that online ratings have been increasing tremendously over the past decade and seem to have become an essential element for patients to leave feedback on the care they receive. More than half of all physicians have been rated online on jameda each year in Germany since 2016. Indeed, with patients increasingly using the internet in relation to their health care [48], it is likely that online patient feedback will become even more important in the future. With online patient feedback mostly positive, physicians do not have to fear online ratings in general; the commonly expressed concerns regarding PRWs being used for “doctorbashing” or defamation [31] or as “platforms for denunciation” [32] have not proven true. Furthermore, less favorable patient ratings often address important elements of a patient-oriented health care system [1] and can help organizations and professionals identify areas that need to be improved [21].