Rejected Online Feedback From a Swiss Physician Rating Website Between 2008 and 2017: Analysis of 2352 Ratings

Introduction

There remains relevant unwarranted variation in health care systems and deficiencies regarding all key aspects of health care [1]. Members of the public, however, have traditionally had few ways of knowing who the “good” health care organizations and professionals are [2]. As part of a wider move toward transparency, public reporting activities have been developed in a number of countries with the aim of providing information about health care organizations or professionals to the public to correct this asymmetry of information in order to inform patient decision-making and drive quality improvement [3-6].

One type of public reporting activity that has been developed in recent decades is physician rating websites (PRWs) [7-10]. PRWs represent a “bottom-up” approach to public reporting, allowing users to post ratings and comments regarding their physician as a source of information for others [11-14]. Although patients have always been able to share their opinions about their physicians with others, the ability to share these opinions via the internet and social media now means that these opinions have the potential to reach a far wider audience. With a growing number of patients utilizing the internet in relation to health care [15], it is expected that PRWs will play an increasingly important role.

A recent systematic search of PRWs internationally analyzed 143 different websites from 12 countries [16] and found that the majority (76.9%) of websites provided options to give feedback both on a predefined quantitative rating scale and as narrative comments. Previous research internationally has often focused on analyzing the ratings and comments publicly available on PRWs. This research has reported that many PRWs have incomplete lists of physicians, a low number of physicians rated, and a low number of ratings per physician that are overwhelmingly positive, which has raised concerns about the representativeness, validity, and usefulness of information on PRWs [14,17]. Furthermore, the medical profession has often expressed concerns that feedback on PRWs will be manipulated for “doctor bashing” or defamation [10].

The first PRWs in Switzerland were established in 2008, at the same time as many international PRWs. However, in comparison with other countries that have established PRWs, there has been limited research conducted on PRWs in Switzerland. This author recently conducted a study involving a random stratified sample of 966 physicians generated from the regions of Zürich and Geneva [18,19]. Selected physicians were searched on a total of four websites (OkDoc, Medicosearch, DocApp, and Google) between November 2017 and July 2018, and it was recorded whether the physician could be found. Moreover, the physician’s rating, the number of ratings and narrative comments, and the text of narrative comments were recorded. As far as the author is aware, this was the first inclusion of Google in a study examining physician ratings internationally.

With regard to the frequency of quantitative ratings and narrative comments on Swiss PRWs, similar issues as those identified in the international literature were found. Many of the selected physicians could not be identified (the proportion of physicians who could be identified ranged from 42.4% on OkDoc to 87.3% on DocApp), few of the identifiable physicians had been rated quantitatively (4.5% on DocApp to 49.8% on Google) or received a narrative comment (4.5% on DocApp to 31.2% on Google) at least once, rated physicians had, on average, a low number of quantitative ratings (1.47 ratings on OkDoc to 3.74 rating on Google) and narrative comments (1.23 comments on OkDoc to 3.03 comments on Google), and all three websites that allowed quantitative ratings had very positive average ratings on a 5-star rating scale (DocApp, 4.71; Medicosearch, 4.69; and Google, 4.41) [18].

With regard to the contents of narrative comments, it was found that the selected physicians had a total of 849 comments [19]. Narrative comments were analyzed and classified according to a theoretical categorization framework previously developed by Emmert et al [10]. In total, 43 subcategories addressing the physician, staff, and practice were identified. None of the PRWs’ comments covered all 43 subcategories of the categorization framework; comments on Google covered 86% of the subcategories, those on Medicosearch covered 72%, those on DocApp covered 60%, and those on OkDoc covered 56%. In total, 2441 distinct issues were identified within the 43 subcategories of the categorization framework; 83.65% of the issues were related to the physician, 6.63% were related to the staff, and 9.70% were related to the practice. Overall, 95% of the subcategories of the categorization framework and 81.60% of the distinct issues identified were concerning aspects of performance (interpersonal skills of the physician and staff, infrastructure, and organization and management of the practice) considered assessable by patients [19]. Furthermore, this research raised concerns that user feedback is being suppressed by Swiss PRWs [18,19], which risks undermining the overall aim of PRWs of providing a reliable source of unbiased information regarding patients’ experiences and satisfaction with physicians.

As far as this author is aware, previous research internationally has only analyzed publicly available feedback on PRWs. However, as it appears that many PRWs are not publishing all the feedback they receive. Analysis of this rejected feedback could provide a better understanding of the types of feedback that are currently not published and whether this is appropriate. This study therefore aimed to examine (1) the number of patient feedback rejected from the Swiss PRW Medicosearch, (2) the evaluation tendencies of the rejected patient feedback, and (3) the types of issues raised in the rejected narrative comments. Gaining a better understanding of feedback rejected by PRWs may help to identify issues in the way PRWs are currently determining which feedback are and are not to be publicly published.

Methods

Sample

Switzerland is a Central European country with a population of about 8.4 million people and 4 official languages (German, French, Italian, and Romansh). The Swiss health care system is highly complex and decentralized, and all Swiss residents are required to purchase basic mandatory health insurance that is offered by competing nonprofit insurers. Mandatory health insurance covers most general practitioner and specialist services, and people not enrolled in managed care plans generally have free choice of professionals [18]. The first PRWs in Switzerland, OkDoc and Medicosearch, were established in 2008. A systematic web-based search conducted in June 2016 identified that the websites DocApp and Google also allow users to view quantitative ratings and/or narrative comments about Swiss physicians in a structured manner without having to open an account or log onto the website [18,19]. It appears that other websites have also subsequently started to allow users to view quantitative ratings and/or narrative comments about Swiss physicians (eg, DeinDoktor and Doctena) [19]. Nevertheless, out of the dedicated Swiss PRWs, Medicosearch appears to be one of the best established and used [18,19]. Medicosearch allows users to search for physicians by location and specialty. Physician profiles provide general information about the physician (specialties, languages spoken, and contact details). In recent years, Medicosearch has shifted its business strategy toward online appointments, where physicians pay a fee and their booking systems are integrated with Medicosearch, allowing patients to book an appointment with a physician directly on Medicosearch. Users can also leave reviews of physicians, but Medicosearch requires both a quantitative rating (5-star rating scale) and a narrative comment in every patient feedback. Although Medicosearch allows negative comments, it informs the concerned physician before publishing it on the website, so that the physician can decide whether to activate the negative feedback function. If the physician refuses, the feedback function is deactivated, removing positive comments as well [19].

As part of a larger project examining Swiss PRWs, the author approached the CEO of Medicosearch, Beat Burger. Discussions confirmed that Medicosearch does not publish all the feedback they receive. The author enquired about the possibility of receiving these rejected feedback for analysis. Medicosearch agreed to provide the rejected feedback to the author in an anonymous form. On October 24, 2017, Medicosearch sent the author an excel file that included all the feedback that Medicosearch had rejected between September 16, 2008, and September 22, 2017. The details of the rated physicians were not included. Medicosearch did not provide any reasons for why the feedback were rejected. The following data were imported into a Statistical Package for the Social Sciences (SPSS version 26 for Windows, IBM Corporation) file: the date the feedback was created, the quantitative rating out of 5, and the narrative comments provided under “title” and “description.” In early 2019, the author inquired with Medicosearch whether it would be possible to receive the rejected feedback from September 23, 2017, to December 31, 2018. Medicosearch told the author on April 11, 2019, that owing to new data-protection rules, Medicosearch had deleted all rejected feedback, and therefore, it was not possible to provide an updated file.

Data Analysis

Medicosearch uses a 5-star rating scale; a rating of 4 to 5 stars was considered a positive rating, 3 stars was considered a neutral rating, and 1 to 2 stars was considered a negative rating. The content of each narrative comment was analyzed and classified by the author according to a theoretical categorization framework of physician-, staff-, and practice-related issues. The categorization framework from Emmert et al was initially used [10], with modifications where necessary. This included removing categories that were not identified in the comments, adding categories that were identified but were not adequately covered by the previous framework, and separating categories (eg, friendliness and caring attitude) that were discussed in comments as distinct issues. Narrative comments were analyzed in their original language. Descriptive statistics included means and standard deviations for continuous variables and percentages for categorical variables. To analyze whether differences existed between German and French comments, chi-squared tests were used. All analyses were performed with the significance level α set to .05 and two-tailed tests, using SPSS version 26.

Results

Characteristics of Ratings

Between September 16, 2008, and September 22, 2017, Medicosearch rejected a total of 2352 patient feedback (Table 1).

The majority of feedback rejected (1754/2352, 74.6%) had narrative comments in German (Table 2). Other rejected feedback had narrative comments in French (275/2352, 11.7%), Italian (31/2352, 1.3%), English (12/2352, 0.5%), and Spanish (1/2352, 0.04%). However, 11.9% (279/2352) of the rejected feedback only provided a quantitative rating with no narrative comment.

Overall, 25% (588/2352) of the quantitative ratings were positive, 18.7% (440/2352) were neutral, and 56% (1316/2352) were negative (Table 3). Additionally, the average rating of the rejected feedback was 2.8 (SD 1.4).

Of the 2073 ratings that provided narrative comments, analysis found that a total of 170 comments were not feedback from a patient concerning the physician, staff, or practice (92 comments were not comprehensible, 29 comments were explicitly labelled as test ratings, 15 comments were about the PRW, 10 comments reported that the person had not visited the physician yet, 10 comments simply reported that the physician’s details were not up to date, eight comments were abusive, four comments were second-hand reports, two comments were asking for advice regarding their or their family member’s condition, and two comments were not concerning a visit to a physician). Consequently, this feedback was excluded from the categorization framework.

The 1903 included narrative comments had a mean length of 158 characters (SD 214), ranging from 1 to 2788 characters. There was a significant difference in the mean character length between positive comments (mean 88, SD 130) and negative comments (mean 193, SD 241) (t₁₃₁₄=−11, P<.001). There was no significant difference in the mean character length between German comments (mean 158, SD 205) and French comments (mean 154, SD 206) (t₁₈₆₂=0.2, P=.82).

Categorization of Issues

Content analysis of the included 1903 narrative comments identified 44 subcategories addressing the physician (n=20), staff (n=9), and practice (n=15) (Textbox 1).

In total, 3804 distinct issues were identified within the 44 subcategories of the categorization framework; 75% (2854/3804) of the issues were related to the physician, 6.4% (242/3804) were related to the staff, and 18.6% (708/3804) were related to the practice (Table 4). The most frequent issue mentioned in the rejected comments was satisfaction with treatment (533/1903, 28%); 73.2% (390/533) of these ratings were negative. Other frequently mentioned issues regarding the physician were as follows: 20.6% (392/1903) of comments provided an overall assessment of the physician (53.8% negative); 18.1% (345/1903) provided a recommendation regarding the physician (76.2% negative); 13.7% (261/1903) referred to the physician’s communication (77.8% negative); 11.6% (220/1903) referred to the physician’s caring attitude (75.9% negative); and 10.6% (203/1903) referred to the physician’s friendliness (73.9% negative). In relation to staff issues, the most frequently mentioned issue was regarding the staffs’ friendliness (109/1903; 5.7%); 43.1% of these ratings were negative. Concerning practice issues, the frequently mentioned issues were as follows: 15.5% (295/1903) of the comments provided an overall assessment (38.6% positive), 8.1% (155/1903) referred to the waiting time within the practice (58.7% negative), and 5% (96/1903) referred to the atmosphere of the practice (40.6% positive).

However, there were some significant differences between German and French comments in a number of subcategories (Multimedia Appendix 1). For instance, German comments referred significantly more often to the physician’s friendliness (χ²₁=5.9, P=.01), being taken seriously by the physician (χ²₁=8.5, P=.002), staff friendliness (χ²₁=7.0, P=.005), waiting time in the practice (χ²₁=6.3, P=.01), and practice atmosphere (χ²₁=6.6, P=.007).

Discussion

Principal Findings

As far as this author is aware, this is the first study internationally to examine feedback that has been rejected from a PRW. The key findings of this study are as follows: (1) the Swiss PRW Medicosearch rejected a total of 2352 patient feedback between September 16, 2008, and September 22, 2017, (2) just over half of all the rejected feedback were negative, and (3) the most frequently mentioned issue in the rejected feedback was satisfaction with treatment. Medicosearch has shown a lot of transparency in providing this rejected feedback for analysis. It is, however, unclear why the majority of the feedback were rejected. This is problematic and raises concerns that online patient feedback are being inappropriately manipulated.

Medicosearch did not provide the reasons why it rejected the feedback, and as far as this author is aware, Medicosearch does not use formal criteria for determining which feedback should be published or rejected. Of the 2352 ratings rejected by Medicosearch, 170 comments were excluded from the categorization framework for various reasons. For example, the feedback were not comprehensible, were explicitly labelled as test ratings, were about the PRW, etc. These would also appear to be legitimate reasons for Medicosearch to reject the feedback. However, the appropriateness of rejecting the remaining 92.7% of feedback is less clear, particularly as they appear to be qualitatively the same as the published feedback for a sample of Swiss physicians recently analyzed [19].

Twelve percent of the rejected feedback only provided a quantitative rating with no narrative comment. Medicosearch requires that both a quantitative rating and a narrative comment be provided in every patient feedback, and this is likely the reason for rejecting this feedback. Narrative comments often provide a richer source of information than quantitative ratings [10]; however, making narrative comments mandatory seems inappropriate. Some patients may not be willing or able to describe what happened in a narrative comment, but may still want to share their satisfaction with their physician with others. It is unclear why these ratings should simply be excluded because the patient did not want to also write a narrative comment.

In terms of the evaluation tendencies of online patient feedback, previous Swiss and international research has found that the published online patient feedback on PRWs is overwhelmingly positive [8,10,14,17-32]. However, recent research also raised concerns that negative feedback is being suppressed by Swiss PRWs [19]. For instance, the PRW OkDoc explicitly states on its website that any negative comments will be deleted, and while Medicosearch allows negative comments, it informs the concerned physician before publishing it online, so the physician can decide whether to activate the negative feedback function [19]. There was therefore an expectation that the majority of the rejected feedback would be negative. However, this analysis of 2352 rejected feedback from Medicosearch found that just over half of all rejected feedback were negative, and the average rejected rating was 2.8 out of 5.

The proportion of rejected negative feedback, however, is substantially higher than the proportion of negative feedback that has been published in the international literature [8,10,14,17-32]. Analysis of the published feedback for a sample of Swiss physicians also reported that only 4.3% (10/234) of the feedback published on Medicosearch was negative and that the average rating was 4.68 out of 5. It is unclear why there is such a large discrepancy between published and rejected negative feedback. It has previously been suggested that Switzerland’s restrictive legal framework regarding data protection may be having a big impact on the types of online patient feedback that are published [18]. However, it may also be that PRWs like Medicosearch are also deciding themselves not to publish a lot of the negative feedback they receive owing to conflicts of interest. Medicosearch has shifted its business strategy toward online appointments, where physicians pay a fee and their booking systems are integrated with Medicosearch, which allows patients to book an appointment with a physician directly on Medicosearch. Consequently, Medicosearch will likely be reluctant to upset paying physicians by publishing too much negative feedback, as their business is now reliant on physicians using their online appointment system.

Users of PRWs can also manipulate online patient feedback, and there is some indication that physicians or practice staff sometimes pose as patients on PRWs to post either positive comments about themselves or negative comments about competitors [33]. Indeed, 25% (588/2352) of the rejected feedback were positive, and it is possible some of these were rejected because Medicosearch suspected that these were fake reviews. However, without a clear and consistent way to determine which feedback is rejected, there is a danger that feedback will be inappropriately rejected.

With regard to the contents of narrative comments, the most frequently mentioned issues identified from the rejected feedback included (1) satisfaction with treatment; (2) the overall assessment of the physician; (3) recommending the physician; (4) the physician’s communication; (5) the physician’s caring attitude; and (6) the physician’s friendliness. In comparison, the top five mentioned issues identified in the analysis of the published feedback for a sample of Swiss physicians were (1) the overall assessment of the physician and the physician’s competence; (2) the physician’s communication; (3) recommending the physician; (4) the physician’s friendliness; and (5) the physician’s caring attitude [19]. This suggests that online patient feedback is raising similar issues, regardless of whether it is published or rejected. Indeed, just like in the analysis of the published feedback for a sample of Swiss physicians [19], it is important to recognize that 95% (42/44) of the subcategories of the categorization framework and 81.5% (3101/3804) of the distinct issues identified were concerning aspects of performance (interpersonal skills of the physician and staff, infrastructure, and organization and management of the practice) that are considered to be assessable by patients.

If online patient feedback is going to be collected, there needs to be clear policies and practices about how this is handled. It cannot be left to the whims of PRWs, who may have financial incentives to suppress negative feedback, to decide which feedback is or is not published online. It has previously been recommended that “there is a need for consensus-based criteria that applies to all Swiss PRWs for determining which comments are and are not to be publicly published and which are clearly publicized so users of PRWs are aware of it” [19]. This analysis of 2352 rejected feedback from Medicosearch further highlights the need for such a consensus-based criteria. To support this, further research is needed to examine how many Swiss PRWs are using criteria for determining which feedback is published or not, what those criteria are, and what measures PRWs are using to address the manipulation of online patient feedback. It appears that research examining these issues would be helpful in most countries that have PRWs.

Limitations

This study has some limitations that should be taken into account when interpreting the results. First, it is unknown how many patient feedback Medicosearch received in total during the time period covered. The author contacted Medicosearch asking for this information but never received a response, and the information is not freely available on the website. It would be helpful to know the proportion of patient feedback that is being rejected. Second, the sample of rejected feedback was only taken from one Swiss PRW. Although Medicosearch is one of the oldest and most used Swiss PRWs, it is unclear how generalizable the results are to other PRWs and other countries. Future research examining whether PRWs are using criteria for determining which feedback is published or not should include all Swiss PRWs. Third, the specialty and sociodemographic information of the rated physicians are unknown, and there may be important differences between the different specialties and physicians. Finally, the sociodemographic information of the rating patients is unknown and may not be representative of Swiss patients in general.