Patient and Provider Experiences With Virtual Care in a Large, Ambulatory Care Hospital in Ontario, Canada During the COVID-19 Pandemic: Observational Study

Introduction

The COVID-19 pandemic has accelerated virtual care use in many jurisdictions, stemming from the need for physical distancing, preservation of personal protective equipment, and the desire to adhere to public health guidance [1-3]. In the province of Ontario, Canada, virtual care adoption was low prior to the COVID-19 pandemic, in large part due to restrictive reimbursement policies [4]. The onset of the pandemic led to the introduction of temporary billing codes allowing for reimbursement of various modalities of virtual care, including videoconferencing across a wide range of platforms as well as telephone visits [5]. During this global crisis, virtual care use in Ontario increased significantly, from 1.6% of total ambulatory visits in the second quarter of 2019 to 70.6% in the second quarter of 2020 [6].

Many studies have been published on patient or provider experiences with virtual care before and during the pandemic [7-12]. Some studies have found positive experiences with virtual visits due to reasons such as convenience and travel time avoided [13-15], while other studies have reported that patients and providers did not find the quality of virtual visits to be better than in-person visits [16]. However, to our knowledge, the literature on patient and provider experiences with virtual care has been mostly limited to small-scale studies localized to a specific clinical program. Furthermore, few have attempted to address potential equity considerations that might contribute to differences in patient experience [17,18]. As virtual care becomes more prevalent, so does the potential issue of the digital divide, in which patients of marginalized populations, such as older age, lower health literacy, non-White ethnic backgrounds, or lower income, may have worse access to or experiences with virtual health services compared with others for reasons such as lack of access to technology or resources in general, discrimination, and limited digital health literacy [19,20].

At Women’s College Hospital (WCH), an academic ambulatory hospital in Toronto, Ontario, Canada, virtual care adoption accelerated during the COVID-19 pandemic. In-person ambulatory visits throughout the hospital were largely replaced with video or telephone appointments. In this study, our objective was to use data from WCH during the pandemic to describe patient and provider experiences with virtual care across various clinical areas and to identify demographic characteristics associated with patient experience.

Methods

Context

WCH is an ambulatory, academic facility in Toronto, Canada and is fully affiliated with the University of Toronto. From an administrative perspective, virtual visits are scheduled almost identically to in-person visits, the only difference being the visit type used. For video visits specifically, the electronic medical record is able to automatically create video encounters with the use of an existing platform (ie, Zoom, a video conferencing service that can be licensed for secure use for health care purposes) when specific video visit types are used. When booking a phone visit, it is clearly identified to clinicians that the visit is to take place by phone. The hospital provided resources on its website, notifying patients of the option of video virtual care and its availability as well as how to use it, including training videos and guides. Otherwise, patients would have been presented with the option during phone calls with administrative staff or notified of a virtual video or phone visit within the appointment notification letter they received. Clinicians were encouraged to curate their environment prior to conducting virtual visits, especially by video. In many instances, clinicians continued to work within the clinic when conducting virtual care, though several clinics or departments worked mostly or entirely from home. Regardless, creating private space and ensuring the use of appropriate technology were encouraged. Clinicians were also encouraged to collect appropriate identifying information from patients, much as would occur during check-in during an in-person visit. On the patient end, the resources provided to them motivated them to treat virtual encounters much as they would in-person encounters, including ensuring their own private space, being in a well-lit area, and minimizing background distractions.

Ethical Approval

This study received ethics approval from the WCH Research Ethics Board (REB # 2019-0191-E).

Data Sources and Population

Data were collected from surveys administered to consenting patients who attended at least one virtual visit (video or telephone) at any clinic within WCH who consented via digital consent to be sent a patient experience survey after their visit. Patients had to be registered in the patient portal to receive the digital consent via the electronic medical record. Patients were offered the opportunity to complete the survey after every virtual visit attended. We analyzed patient experience survey data for video visits from May 2020 to May 2021 (253 patients and 517 responses). As survey deployment for telephone visits was delayed due to staff shortages in the second pandemic wave, we analyzed data for telephone visits since survey launch from October 2020 to May 2021 (147 patients and 209 responses). Provider experience surveys were administered twice, in June 2020 and June 2021, for those who delivered at least one virtual visit at any clinic within the hospital. Both patient and provider surveys included questions regarding demographic characteristics and satisfaction with virtual care, with opportunities for written feedback. Patient and provider survey questions can be found in Multimedia Appendix 1.

Statistical Analysis

Descriptive statistics were used to summarize demographic characteristics and responses to patient experience questions among all patients and new patients (defined as those who had an initial visit with a clinic during the study period) who attended at least one virtual visit at the hospital. The following demographic characteristics were self-identified through survey responses: age, gender, ethnicity, household income, and English-language proficiency. Overall marginalization was determined using the Ontario marginalization index (ON-Marg) [21], which was linked to patient postal codes. ON-Marg is a tool that measures deprivation on multiple levels, including economic, ethnoracial, age-based, and social marginalization. Chi-square tests, Fischer exact tests, and t tests were performed as applicable to compare demographic characteristics and responses to various questions regarding patient experience (eg, helpfulness of the virtual visit, likelihood of recommending virtual visits to a friend) between patients who attended video appointments and patients who attended telephone appointments.

The net promoter score, a metric used to measure a client’s willingness to recommend a company’s product or services [22], was also calculated for video and telephone visits. As the net promoter score can range from –100% to +100%, any score greater than 0% can be considered a desirable score [23]. Ordinal logistic regression analysis was performed on all survey responses to identify any patient demographic variables that may predict their response to select questions on likelihood to recommend virtual visits and perceived helpfulness of the virtual visits. Findings from the regression model compare the odds of choosing an answer in the highest category compared with the other 2 categories (eg, rating of 9-10 compared with ratings of 1-6 and 7-8). Only patients with complete responses to all relevant survey questions were included in the models. Provider ratings to various survey questions on their experience, including the perceived quality of the virtual visit compared with an in-person visit, the amount of time and effort required to conduct the virtual visit, and others, were compared descriptively between June 2020 and June 2021.

Results

Patient Experience

Among all virtual visits for patients registered to the patient portal during the study period, the proportion of individuals who consented to be sent a survey was 1057 of 1872 (56.5%) for video visits and 259 of 358 (72.3%) for phone visits. Among those who consented, 517 of 1057 (48.9%) video visit surveys and 209 of 259 (80.7%) phone visit surveys were completed. Baseline characteristics of all patients who responded to at least one video or telephone survey are reported in Table 1. A total of 253 unique patients completed 517 video surveys, while 147 unique patients completed 209 telephone surveys. There were more women (130/147, 88.4% vs 195/253, 77.1%; P=.005) and older patients (mean age 45.1, SD 15.6 years vs mean 41.6, SD 16.4 years; P=.04) who completed telephone visit surveys compared with video visit surveys. Among survey respondents who were new patients, telephone users were also older than video users (mean age 50.5, SD 17.5 years vs mean 41.3, SD 15.3 years; P=.02; Table S1 in Multimedia Appendix 2). The top clinical departments through which patients attended the virtual visits are listed in Tables S2 and S3 in Multimedia Appendix 2.

Table 2 reports patient responses regarding their experience with virtual care. Most patients found their virtual visit to be very helpful for their health issue (417/517, 80.7% for video and 154/209, 73.7% for telephone). When asked what they would have done if a virtual visit with the doctor was not available, most selected “I would not have sought care at that time” (198/517, 38.3% of video users and 43/209, 20.6% of telephone users), “Scheduled an in-person visit with this doctor” (192/517, 37.1% of video users and 99/209, 47.4% of telephone users), or “See/talk to my family doctor” (145/517, 28.1% of video users and 58/209, 27.8% of telephone users). On a scale of 1 to 10 regarding likelihood to recommend virtual care to a friend, most patients responded with a rating of 8 or above (415/517, 80.3% of video users and 150/209, 71.8% of telephone users). The net promoter score for video visits was higher than that for telephone visits (60.2% vs 40.4%). Findings were similar when only considering new patients (Table S4 in Multimedia Appendix 2). However, the difference in net promoter scores between video and telephone visits was greater for new patients (51.8% for video vs 15.0% for telephone). The majority of patients (405/517, 78.3% for video and 146/209, 69.9% for telephone) preferred to have the option of virtual visits after COVID-19.

We report results for the ordinal logistic regression models in Table 3. In model 1, 255 patients with complete responses were included, and 268 patients were included in model 2. From model 1, ethnicity and age group were significant predictors of the likelihood of recommending video or phone visits to a friend. Specifically, patients who were non-White had lower odds of rating a high score of 9 or 10 compared with White patients (odds ratio [OR] 0.52, 95% CI 0.28-0.99), and patients aged 50 years to 59 years had lower odds of rating a high score of 9 or 10 compared with patients aged 30 years to 39 years (OR 0.26, 95% CI 0.11-0.64). None of the independent variables (ethnicity, family income, overall marginalization, gender, and age) assessed in model 2 appeared to be significant predictors of how helpful the virtual visit was in addressing the patient’s health issue.

Provider Experience

A total of 75 providers completed the survey in June 2020, and 94 providers completed the survey in June 2021 (Table 4). The top 3 survey respondents among providers in 2020 were physicians (47/75, 63%), social workers (7/75, 9%), and psychotherapists (6/75, 8%), while the top 3 provider survey respondents in 2021 were physicians (48/94, 51%), social workers (9/94, 10%), and physiotherapists (8/94, 9%). In both 2020 and 2021, most providers who delivered virtual visits had been practicing for 10 or more years (40/75, 53% in 2020 and 61/94, 65% in 2021).

Responses to the provider experience surveys are shown in Table 5. When asked whether the quality of the virtual visit was similar to that of an in-person visit, 13% (10/75) selected agree or strongly agree in 2020, compared with 28% (26/94) in 2021. In 2020, 67% (50/75) of providers felt that video visits enabled them to sufficiently address their patient’s clinical need compared with 70% (66/94) in 2021. Most providers planned to continue using video visits after the need for physical distancing decreased (53/75, 71% in 2020 and 69/94, 73% in 2021). When asked to rate on a scale of 1 to 10 their likelihood of recommending other providers to do virtual visits for patients, most providers rated a score of 8 or above in 2020 (47/75, 63% for video and 46/75, 61% for telephone) and in 2021 (62/94, 66% for video and 49/94, 52% for telephone). The net promoter scores for video visits were 17.8% in 2020 and 30.4% in 2021, while the net promoter scores for telephone visits were 19.2% in 2020 and 1.1% in 2021.

Discussion

Principal Findings

This study describes the patient and provider experiences with virtual visits during the COVID-19 pandemic across an academic ambulatory hospital in Toronto, Canada. Feedback for virtual visits was generally positive among patients. Video visits were the preferred modality over telephone among many patients, particularly for those who were new patients. However, we found that patients of non-White background were less likely to recommend virtual visits compared with those of White background. Provider experiences with virtual visits were less positive compared with those of their patients, but there was a general improvement in provider feedback from 2020 to 2021.

Comparison With Prior Work

Most patients found that their virtual visit was helpful in addressing their health issue and rated a high score when asked to what degree they would recommend virtual visits to a friend. However, a higher proportion of patients reported video visits to be “very helpful” compared with telephone visits. Similarly, the net promoter score was much higher for video visits compared with telephone visits. In our study, patients who completed telephone visit surveys were generally female and older in age compared with patients who completed video visits. In the literature, older patients are less likely to engage in virtual care than in in-person care and even less likely to choose video than telephone [24,25].

In our study, several patients indicated in the open-ended questions that they would prefer video over telephone due to the ability to see the provider and observe facial expressions and body language. Other studies have also cited the benefit of increased human connection that accompanies video platforms [26]. The difference in net promoter scores was even greater among new patients, with video visits reaching a significantly higher score compared with telephone visits. Video visits enable patients to see their provider, which supports the development of a patient-provider relationship especially for an initial encounter. These results contrast with findings from a systematic review that reported no significant differences in patient satisfaction between video and telephone visits, but they did not stratify initial versus follow-up encounters [27]. This may also be attributed to our sample consisting of younger individuals. It is likely that younger patients prefer video visits more so than older patients who may prefer telephone visits due to ease of access [28].

Despite the generally positive feedback for virtual visits, a small proportion of patients did not find the visit helpful or rated a low recommendation score. The open-ended responses suggest that some patients were unhappy with the delay in their appointment start time and the lack of communication from the clinic in such cases. Published studies have cited other patient criticisms of virtual visits such as technical issues with connection and quality of the call [29], a lack of privacy at home when attending virtual visits [30], and a preference for in-person visits for certain physical health issues or to build a relationship with their provider [31]. Overall, it appears that many patients had positive experiences with their virtual visit(s), with several citing reasons such as convenience and that they were able to save time and money [13].

Findings from the regression model indicate that non-White patients were less likely to recommend virtual visits to a friend compared with White patients. Mixed findings are reported in the literature, with several studies showing no significant differences in patient experience with virtual care by ethnicity [32-34], while others have shown that patients of non-White backgrounds are more likely to have lower satisfaction with virtual visits compared with their White counterparts [17,35]. Upon analysis of the average recommendation scores for video versus telephone visits by ethnicity, shown in Table S5 in Multimedia Appendix 2, Asian and Black patients had similar, if not better, scores than White patients; however, ratings were generally lower for the other ethnic groups. Reasons for this disparity remain unclear and should be investigated in future work. Among all age groups of interest, only patients aged 50 years to 59 years were found to be less likely to recommend virtual visits than patients aged 30 years to 39 years (the reference group). This may be explained by older patients’ preferences toward in-person care or the technological barriers they may encounter with virtual visits [36,37]; however, this association did not persist in the older age groups (those older than 60 years) for reasons unknown. We do note that there were no significant differences in experience found for the other demographic variables assessed (family income, overall marginalization, sex, and most age groups), which may be a positive sign that the delivery of virtual visits may have helped to bridge the gap in equitable health care access in certain ways; for example, lower income patients may find it easier to attend a virtual visit than request time off work to attend in-person care, or older patients with mobility issues may find it easier to attend a virtual visit than an in-person visit.

Our findings are consistent with other studies that found lower satisfaction with virtual care among health care providers compared with patients [38]. Our findings show that the net promoter scores were lower for both video and telephone visits when rated by providers than by patients. Open-ended responses from the provider surveys suggest that many providers felt that they needed to provide a physical examination to adequately address their patient’s health needs, similar to findings from the available literature [39]. Other providers felt that the quality of the virtual visit was lower than that of an in-person visit. Several also cited technical issues, particularly with video visits, as a deterrent for virtual care. Studies examining provider experience with virtual care reported similar reasons for provider dissatisfaction [40], with less than one-half of providers preferring virtual over in-person care [41].

From a provider standpoint, there appeared to be an increase in positive feedback for virtual visits across most survey questions from 2020 to 2021, including quality of virtual visit, time and effort spent on virtual visit, and preference to use virtual care after the pandemic. This may be because, as providers had more experience with virtual care, their self-efficacy may have improved. Another explanation is that the proportion of all visits that were virtual was higher in 2020 than in 2021, so the appropriateness of virtual care for the visit reason was likely also higher in 2021. Several studies have reported that clinicians have a positive outlook on virtual visits [13], particularly within the mental health field [39,42] in which physical examinations play a lesser role in clinical practice compared with other specialties. However, we note that the providers’ net promoter score for telephone visits decreased significantly from 2020 to 2021, while scores for video visits increased. A possible explanation supported by open-text portions of the survey is that providers may have been more comfortable and proficient with providing video visits and preferred the face-to-face connection that can be missing from telephone communication, but this merits further exploration.

Strengths and Limitations

A strength of this study is that it provides the patient and provider perspectives on virtual care in a large ambulatory hospital setting with responses across many clinical specialties and programs. Our findings also offer insights into both patient and provider experiences and into some of the demographic differences in experience with virtual care to identify potential equity issues. Nonetheless, our study does have several limitations. The overall response rate for demographic questions in the video survey was lower than anticipated due to a technical error in survey deployment. Furthermore, despite our equity focus, there are limitations in the demographic insights that can be gleaned. First, although we captured several important demographic variables in this study, we were unable to assess the association between patient experience and other potentially relevant characteristics, such as education level, employment, and immigration status. The low proportion of patients who consented to receive surveys and the fact that surveys could only be sent to patients who had a valid email address and were registered on the portal system would have limited responses, including from certain marginalized and underserviced groups. We also acknowledge the possibility that patients who have fewer positive experiences with virtual visits may be less inclined to complete the survey, which would potentially bias the findings to be more positive. However, our analysis was still able to detect a difference in experience among patients of ethnic minorities. Finally, an electronic survey does not offer a deep understanding of experience, particularly among marginalized groups, as opportunities for feedback is limited and patients may not feel as comfortable sharing their thoughts on the platform due to confidentiality concerns. Future studies of patient experiences with virtual care should include interviews or focus groups with patients from underserved communities. We also surveyed patients seen in specialty clinics within an ambulatory care hospital and did not include primary care patient surveys. Last, these findings reflect the experiences of patients and providers in a single institution within a universal health care system and therefore may not be generalizable to other settings.

Conclusions

This study summarizes the patient and provider experiences with virtual care across an academic ambulatory hospital in Toronto, Canada during the COVID-19 pandemic. Virtual care, comprised of video and telephone visits, was generally well-received among most patients, with many favoring video visits over telephone visits especially for new patients. However, virtual care was less endorsed among many providers. Furthermore, patients with non-White ethnic backgrounds were less likely to recommend virtual visits. These findings provide important contributions regarding understanding overall patient and provider experiences with virtual care as well as predictors of patient experience. Given the prospect of the hybrid modality of care delivery that includes both virtual and in-person options of care delivery post-COVID-19, future work should aim to develop ways to understand factors to improve patient and provider experiences with virtual care and to assess the impact of virtual modalities on patient outcomes and quality of care.