Scope, Breadth, and Differences in Online Physician Ratings Related to Geography, Specialty, and Year: Observational Retrospective Study

Introduction

Patients’ abilities to discern health care quality are often underappreciated, despite evidence that low patient satisfaction scores and complaints against physicians are linked to increased risk management episodes, malpractice lawsuits, readmission rates, and even increased mortality for selected diagnoses [1-5]. Over the last decade, physician-rating websites have become a popular source of patient satisfaction data [6]. Such websites represent unsolicited reflections of the patient experience with their physicians in comparison to more traditional methods such as surveys. In the United States, the use of physician-rating websites is rapidly increasing, whereas other countries have reported more moderate growth [6,7]. In addition to private online physician websites, government or health insurer-developed sites are also being used in countries such as the United Kingdom and Germany [8,9]. Together, these physician-rating websites may impact patient health care decision making, as data suggests approximately one-third of users have searched for physicians online and report making decisions regarding physician selection based on these ratings [10]. Online physician-rating websites may also impact physician behaviors; over the last five years, physicians have been increasingly responding online to their ratings [11]. Hence, this data source may have significant implications on health care practice and behavior.

Most previous work on online physician ratings has focused on reviewing the frequency and usage among different physician specialties in the United States, China, and Germany [6,12-26], as well as exploring awareness and perceptions among physicians and consumers [10,11,27-29]. More recently, the focus has been to correlate online ratings with quality outcomes or surrogates such as postoperative mortality and surgical volumes with variable findings, depending on the quality outcome in question [6,30-38]. It has been estimated that one in six physicians are rated online, and most ratings are positive [6,12-4,17-19,28]. Although the use of physician-rating websites is increasing overall [6,7,39], for frequency of ratings, US studies have reported that the mean number of ratings per physician is low overall, ranging from two to four ratings per physician [6,17,21]. Several studies have focused on differences in ratings according to specialties [6,14,20-22]. Certain types of physicians, such as obstetricians, dermatologists, surgeons, and family physicians, are more frequently rated than other specialists. Board-certified, younger physicians have been shown to be rated more favorably than non-board-certified, younger physicians [6,14]. Other studies have investigated the relationship between practice location (such as city size) and online ratings. In the United States, physicians in the southern states had a higher likelihood of positive ratings than other parts of the country, whereas others have shown no difference in ratings with respect to practice location and city size [6,20-22].

In Canada, there is currently little information available on the use of physician-rating websites. Our study sought to investigate the nature and trends of online physician ratings in Canada over a nearly 8-year period. The goals of this study were to (1) determine whether online ratings for physicians differed depending on physician specialty, (2) investigate whether physician practice location affected online ratings, and (3) examine possible trends in ratings over time by year of rating. We also compared the number and frequency of ratings by specialty to determine whether certain specialties were rated online more frequently than expected based on their representation in the overall physician population. Based on previous studies, we hypothesized that certain specialties, such as obstetrics and family medicine, would be rated more frequently than others, such as pathology or radiology. We also felt that the quality of ratings would be positive overall and that differences in ratings would exist across specialties and geographic practice location. We suspected that there would be no differences in quality of ratings over time, but that the absolute number of ratings would be steadily increasing over our study period.

Methods

Overview

We accessed a national database of all Canadian physicians rated from January 2005 to September 2013 (N=640,603 ratings) [40]. RateMDs was founded in the United States in 2004 and is currently among the most popular physician-rating websites in Canada and the United States by user traffic [13,18]. No registration or subscription is required to view or post a rating, and there are no incentives to rate a physician. Physicians are rated on a scale of 1 to 5 (described by the website as 1=“terrible,” 2=“poor,” 3=“okay,” 4=“good,” 5=“excellent”). Ratings were given for each of the following domains: staff, punctuality, helpfulness, and knowledge. A mean overall score is posted for each physician. Physician profiles are created or searched for by the rater, and users provide ratings and may provide free-text comments if desired. Our dataset included deidentified data for 57,412 physicians, including specialty, practice region (city and province), date of rating, and scores on each of four domains, from which we calculated an average cumulative rating for each physician. This dataset included all physicians in Canada who were rated on RateMDs during our study period.

Mean number of ratings and mean ratings were calculated for all physicians, each website specialty, and province. To compare the relative proportions of physicians by specialty, we grouped specialties according to Canadian Institute of Health Information (CIHI) categories [41]. We considered “obstetrics and gynecology” as distinct from “surgery” because previous research demonstrated high numbers of ratings for this group [6,14]. We calculated each physician specialty’s online presence by grouping online specialties into CIHI specialty categorizations and divided the number of physicians rated online for that specialty by the total number of physicians in the online database. We then calculated and compared these values to the mean annual number of physicians divided by the total annual physician population for CIHI specialties from 2005 to 2013 (to match our online ratings data period). This allowed us to infer whether a specialty was rated more or less frequently than expected based on the mean annual physician population for that specialty.

Statistics

For statistical analysis, our objective was to recognize and compare differences in favorable versus unfavorable ratings for physician specialty, geographic practice location, and year of rating. We constructed a binary variable indicating whether each rating was greater than, less than, or equal to the median rating, which was 4.5 out of 5. We thus considered whether ratings were in the top 50th percentile of all ratings for one of three predictors: physician specialty, province, and year. For each level of predictor, the proportion of ratings greater than 4.5 was reported with a 95% confidence interval and a P value against the null hypothesis that the true proportion was equal to 0.5. In this way, we were able to stratify specialties, practice location by province, and year of rating according to likelihood of positive ratings. All analyses were performed using prop.test in R version 3.0.2.

Ethics

When submitting research ethics board approval, we were informed that the requirement for ethics approval was waived because data were publicly available.

Results

Findings

From February 2005 to September 2013, there were 640,603 ratings for 57,412 unique physicians. Ratings were generally positive (mean 3.9, SD 1.3). Using the online rating website’s rating descriptions, this translated to a mean rating that fell between “okay” and “good.” The mean number of ratings per physician was 11.2 (SD 10.1; see Table 1). During our study period, the mean annual number of total physicians in Canada was 66,026.1 (SD 5748.2). The largest group of physicians, by medical specialty, was family medicine/general practice (n=30,818 physicians). This group had 370,972 unique ratings and, on average, had 12 ratings per physician, with a mean overall rating of 3.9 (SD 1.3). Internal medicine (including its subspecialties) accounted for 53,818 total ratings of 6677 individual physicians, with 8.1 ratings per physician (SD 7.7) and a mean rating of 3.98 (SD 1.31) out of 5. Surgery (including its subspecialties) included 22,811 total ratings of 2472 individual physicians, with 11.9 ratings per physician (SD 10.7) and an overall mean rating of 4.01 (SD 1.32) out of 5. We found that certain specialties had relatively increased numbers of per-physician ratings, including reproductive endocrinology (mean 19.7, SD 15.2), cosmetics/plastic surgery (mean 16.7, SD 16.1), and obstetrics and gynecology (mean 17.6, SD 16.1). Additionally, certain medical specialties had lower numbers of rated physicians as well as per-physician ratings, including radiologists (total number of rated physicians: 330, mean per-physician ratings 3.0, SD 2.8), pathologists (total number of rated physicians: 13, mean per-physician ratings 4.4, SD 8.1), and medical geneticists (total number of rated physicians: 26, mean per-physician ratings 2.6, SD 4.9; see Table 1).

Differences in Frequencies of Ratings According to Specialty

For each specialty, we calculated the percentage of physicians with online ratings divided by the total online physician population, and compared it to the percentage of physicians in a given specialty divided by the total annual physician populations for CIHI specialties. Certain specialties were more frequently rated than expected based on their proportion in the national population, notably obstetrics and gynecology (4.3% of online cohort vs 2.5% of mean total annual obstetrics and gynecology population), dermatology (1.2% vs 0.8%), family practice (53.7% vs 45.2%), internal medicine (including its subspecialties; 12.0% vs 10.3%), emergency medicine/critical care (2.4% vs 1.1%), and surgery (14.3% vs 10.0%), whereas others were less represented, including anesthesia (1.4% vs 4.1%), radiology (0.6% vs 3.3%), psychiatry (5.0% vs 6.3%), and pathology (<0.01% vs 1.9%; see Table 1).

Differences in Quality of Ratings for Physician Specialty

We investigated whether there were differences in the quality of ratings depending on physician specialty. We found that ratings for certain specialties were more likely to be in the top 50th percentile of all ratings, including cardiac surgery (probability of a rating greater than the median of 4.5 was 78.1%, P<.001), genetics (73.5%, P<.001), nephrology (69.2%, P<.001), radiology (65.3%, P<.001), and vascular surgery (65.1%, P<.001). The bottom four physician specialties included psychiatry (42.2%, P<.001), neurology (42.1%, P<.001), dermatology (37.0%, P<.001), and addiction medicine (35.8%, P<.001; Figure 1; Multimedia Appendix 1). Family medicine/general practice comprised our largest group of physicians in the online cohort, as well as one of the largest groups of physicians represented in the mean annual physician population. Regarding likelihood of a favorable rating, family medicine/general practice was among the bottom seven physician specialties (46.3%, P<.001; Figure 1; Multimedia Appendix 1).

Differences in Frequency of Ratings for Physician Practice Location (by Province)

We found that Ontario had both the highest number of ratings and the highest number of rated unique physicians (244,635 ratings for 20,740 physicians), followed by Quebec (116, 041 for 13,460 physicians), then British Columbia (101,152 ratings for 8398 physicians). The lowest number of ratings for the lowest number of physicians was found in the less densely populated regions of the Northwest Territories/Yukon/Nunavut (802 ratings for 126 physicians) and Prince Edward Island (2534 ratings for 242 physicians).

For most provinces, per-physician number of ratings ranged from 10 to 13, with the exception of Quebec and the Northwest Territories/Yukon/Nunavut (ratings per physician 8.62 and 6.37, respectively).

Differences in Quality of Ratings for Physician Practice Location (by Province)

We also found differences in a physician’s likelihood of a positive rating depending on practice location. Broadly speaking, physicians who practiced medicine in the eastern geographic locations of the country had a higher likelihood of being favorably rated than those who practiced in central or western Canada.

‎View this figure

Specifically, physicians practicing in New Brunswick (56.3%, P<.001), Newfoundland (56.0%, P<.001), Quebec (53.6%, P<.001), Prince Edward Island (53.6%, P<.001), the Northwest Territories/Yukon/Nunavut (52.7%, P=.13), and Nova Scotia (52.7%, P<.001) were more likely to be rated greater than 4.5, whereas those practicing in Saskatchewan (46.4%, P<.001), Ontario (46.9%, P<.001), British Columbia (46.5%, P<.001), Alberta (46.5%, P<.001), and Manitoba (45.6%, P<.001) were likely to be rated 4.5 or lower (Figure 2; Multimedia Appendix 1).

Differences in Online Ratings for Year of Rating

During our study period, there were 640,603 total individual ratings of 27,181 physicians. Over time, the total number of ratings continued to increase; however, we found some important differences in the number of additional new ratings per year (Table 2). In 2005, when the website was still new in Canada, there were only 138 ratings. However, in 2007, 200,650 new ratings were posted before slowly tapering down each subsequent year until 2013, when there were 51,800 new ratings. The year 2007 was also notable in that the mean number of ratings per physician was highest at 5.74 (SD 5.28) before settling at 1 to 3 ratings per physician. In terms of quality of ratings, from 2005 to 2013, physicians were more likely to be rated above the median if rated more recently (ie, in 2013; upper 50th percentile proportion 0.512, P<.001), and the likelihood of favorable ratings increased over time. There were two years (2013 and 2007) when quality of ratings were especially high, whereas for the remaining years the proportion of ratings greater than 4.5 was significantly less than 50% (Figure 3; Multimedia Appendix 1).

‎View this figure

‎View this figure

Discussion

Using national-level data over a nearly 8-year period from the country’s largest physician-rating website, we found that 57,412 unique physicians are rated online and that, overall, ratings are positive. We found differences in ratings with respect to physician specialty, geographic practice location, and year.

To our knowledge, this study is the first to describe the landscape of physician ratings in Canada. This adds to the body of national-level literature on physician-ratings websites in China, Germany, and the United States [6,10,12-14]. Previous studies have focused on either specific specialties or had shorter study periods [20-26]. Overall, our findings are in keeping with previous work that physician ratings are typically positive [6,12-14,17-19,28].

We found that certain specialties (eg, cardiac surgeons and nephrologists) were more likely to be rated in the top 50th percentile of all rated physicians, whereas others (eg, sleep disorder specialists, dermatologists, and addiction medicine specialists) were less likely to be rated as favorably. A variety of physician and patient factors may contribute to such differences. This may be due to differences in patient population as well as differences in patient expectations. For example, surviving a surgery may be a relatively straightforward “rateable” aspect for a surgeon; insight into recognizing the milestones for recovery from addiction with frequent relapses may not be as straightforward. In addition, there are likely more complex interactions between preconceived expectations patients have regarding their physician, their perceived performance of that physician, and their resulting satisfaction—as well described by the expectation-disconfirmation theory in the psychology and consumer marketing literature [42].

Our results add additional information and detail to previous work. Quality of ratings have been shown to be similar for physicians in primary care, medical specialties, surgeons and surgical specialties, and obstetrics and gynecology, but significantly differed for a category of “other physicians,” which included radiologists, pathologists, and anesthesiologists [6]. Others have shown that pediatricians and surgeons had more favorable ratings, although others showed that ratings for generalists did not differ either in quantity or quality from those for subspecialists [17].

In addition to quality of ratings, we also looked at frequency of ratings by specialty. Certain specialties (eg, obstetrics, dermatology, and family medicine) were more commonly rated than others (ie, pathology and radiology), which based on their proportion in the national physician population, overall, in keeping with previous work [6,12,14]. One hypothesis is that patient-physician encounters during surgeries and pregnancies may be discrete care episodes that may be more amenable to appraisal. Also, specialties such as family medicine involve direct physician-patient interaction over time; in contrast, patients rarely interact with their pathologist or radiologist, the two least-rated specialties. Patients may also more readily attribute care to (and hence, rate) a single provider in the case of a surgeon, obstetrician, or dermatologist, as opposed to settings such as inpatient internal medicine, where multiple physicians may collaborate.

We also found differences in the likelihood of a positive rating for geographic location. It seems unlikely that physician quality vastly differs regionally, given the national accreditation and continuing education standards. We noted, in general, that east coast and territory provinces were more likely to have ratings greater than the median (4.5) compared with provinces west of Ontario. There may be geographic differences in rater expectations for a variety of reasons; for example, location may give rise to differences in accessibility to medical care. One interesting hypothesis is that when physicians are scarce, consumers may be more appreciative of access to a physician and this may bias their ratings in a more favorable manner. In addition, we looked at economic prosperity indicators such as gross domestic product by province and found that, overall, lower patient satisfaction is found in more economically prosperous provinces (ie, central and western provinces) [43], in contrast to a theory by Grigoroudis et al [44] that posits that higher patient satisfaction may be explained by economic prosperity. Moreover, other sociologic or cultural phenomenon across locations may lead to variable consumer preference, a well-described marketing phenomenon known as geographic segmentation [45]. Explanations for such differences are likely multifactorial and remain, as yet, unknown. There is limited research on the variability of online physician ratings with geographic practice location. Gao et al [6] reported that physicians in the southern United States were slightly more likely to be rated favorably than those practicing in the rest of the country. However, others have reported no difference in ratings regarding practice location and city size for certain surgical specialties [20-22].

Finally, we found differences for ratings over time. We suspect that this is due to patient factors, rather than physician factors, because we would not expect physician quality to fluctuate dramatically from year to year, and the survey instrument was consistent throughout the time period. Of note, RateMDs was founded in 2004 in the United States and, by 2005, online physician websites were still new in Canada (138 ratings in 2005). By 2007, popularity peaked at 200,650 ratings before stabilizing and decreasing by 2013. It is challenging to explain this phenomenon. It may be that in 2007, online physician ratings finally received public attention, resulting in a flood of “early adopters,” which subsequently waned. There was sufficient popularity of such websites and several prominent nationwide media articles in 2007 that physicians became concerned about their use. One article in a popular national news source reported the Canadian Medical Association’s displeasure at such sites and, in particular, warned of the potential for libel [46-48]. However, our findings suggest that these early users were actually more likely to post favorable ratings. This may be plausible, if only because physician-rating website users in general tend to have more positive views toward the Internet, despite no differences in total quantity of Internet usage from the general population [16]. This is in keeping with our finding that the likelihood of a positive rating was highest in 2007.

Since 2007, ratings stabilized and even decreased in absolute number through to 2013. This finding differs from US data, which shows physician-rating website usage rapidly increasing, although the study period in question spans a 5-year period that ends before this study making comparisons problematic [6]. Based on user traffic to competing physician websites in Canada, it does not appear that increasing popularity of competing websites is the explanation. Compared to the United States, in Canada there is comparatively less consumer choice in physician selection because the avenue to seek subspecialty consultation is via one’s primary care physician rather than self-referral. This may, in turn, be driving a decrease in the popularity of physician-rating websites. This hypothesis has been used to explain the use of physician websites in England; although increasing over time as well, they have demonstrated a more gradual, stable rise in popularity compared to the rapidly accelerating US growth [39].

We acknowledge several limitations to our work. First, although our dataset spans nearly an 8-year period, we are missing data from a period of 3 months (ie, October-December 2013 to complete calendar year 2013). However, we feel a national database of greater than 57,000 physicians for nearly an 8-year period is sufficient to elucidate broad trends. Second, online physician-ratings data may not be generalizable. Rating website users likely differ from the general population by virtue of computer access and ability, and by their inclination to post ratings [30]. In addition, because all physicians are entered into the website by raters, it is possible that a physician may have two unique profiles. This database was deidentified; therefore, we were unable to ensure that duplicate profiles were corrected. Moreover, ratings are anonymously posted, so it is possible that fraudulent ratings exist; however, the website has quality control mechanisms in place to circumvent multiple fraudulent ratings (eg, deleting multiple reviews from a single Web address). Third, we could not control for the possibility that online ratings may, themselves, influence future ratings. For example, when a user logs onto the website to post a rating, their original inclination may be influenced by what has previously been published. Overall, these are issues that are germane to most physician-ratings websites and, on balance, we do not feel these limitations would significantly alter our observations, greatly affect broad trends of average ratings and regional differences, nor affect our conclusions.

This study provides new national-level information on the nature of online physician ratings, particularly regarding specialty, geographic practice location, and changes over time. It remains to be seen whether such trends will continue. The utility of online ratings for ascertaining and evaluating physician quality is still in question—and we would argue that before undertaking these larger questions, a better understanding of the scope and breadth of online physician ratings is required. Our study has shown important differences in how physicians are rated based on a physician’s specialty, practice location, and the year in which the physician is rated. Further studies endeavor to better understand the scope, breadth, and utility of online physician ratings; in the meantime, what we do know is that such websites reflect the unsolicited views of the health care consumer and, as such, remain a valuable data source of the patient experience.