Cost Evaluation of the Ontario Virtual Urgent Care Pilot Program: Population-Based, Matched Cohort Study

Introduction

Although virtual care has been available for several decades, it was not widely used before the COVID-19 pandemic. In Ontario, the most populous province in Canada (2021 population: 14.2 million [1]), the Ontario Telemedicine Network (OTN) was the only fully funded virtual care model prior to the COVID-19 pandemic. Historically, two-thirds of OTN use has been for mental health and substance use [2]. In the second quarter of 2019, OTN accounted for approximately 1.6% [3] of all ambulatory care visits. Following the first wave of the COVID-19 pandemic and the introduction of physician billing codes to support virtual care in Ontario, the percentage of ambulatory care visits in Ontario delivered by virtual care in the second quarter of 2020 has increased to 70.6% [3]. In parallel, between the pre–COVID-19 period (March 12 to July 29, 2019) and the COVID-19 period (March 11 to July 28, 2020), the number of in-person physician office visits decreased by approximately 80% [4].

In the fall of 2020, the Ontario Ministry of Health (MoH) introduced a virtual urgent care (VUC) pilot program to provide alternative access to urgent care services and reduce the need for in-person emergency department (ED) visits for patients with low acuity health concerns [5]. Hospitals interested in offering VUC services were invited to submit a funding application to Ontario Health, Ontario’s health care administrative agency. Following a review and feedback process, the MoH approved $4 million of operational funding to support 14 ED-led VUC programs in the 5 health regions of Ontario. As part of this funding, participating sites were expected to launch their VUC programs within 1 to 2 months of funding approval and participate in a mixed methods evaluation of the VUC pilot program to inform future policy and funding decisions. The characteristics of the VUC sites [6] and patient characteristics [7] are described elsewhere but are briefly summarized here. The 14 VUC sites included a mix of pediatric, northern, urban, academic, and community sites across Ontario. Since sites were responsible for the design and implementation of the VUC program to meet their local needs, models of care differed between sites (eg, clinical and operational governance, triage, technology, staffing, operating hours, marketing, and communications) and sites launched their VUC services at different start dates [6].

There were 22,278 VUC encounters between December 2020 and September 2021, and the median (IQR) duration of the virtual visit was 14 (11-18) minutes [7]. VUC users were more likely to be younger, female, English speaking, and have a postsecondary or higher education and a primary care physician. Most complaints were managed by the VUC provider without the need to refer the patient to an in-person ED visit [7]. Other results based on 1:1 propensity score matching (PSM) methods indicated that 30-day health care use (eg, hospitalization, ED, and physician visits) was similar between 2129 VUC users who presented to an ED within 72 hours of VUC referral and 2129 matched patients presenting to the ED in person [8]. VUC users not promptly referred to an ED (N=14,179) were more likely to have an ED visit within 72 hours, 7 days, and 30 days following the index VUC visit compared to matched in-person ED individuals [8]. However, 30-day hospital admissions were lower for the VUC group not promptly referred to the ED, but the length of hospital admission was longer than for patients presenting to the ED in person [8].

To inform future investments in VUC in Canada and elsewhere, here, we present the results from the costing analysis of the VUC pilot program that consider both funding provided by the MoH in Ontario to operate the VUC program and direct costs associated with health care resource use during the 30 days following each VUC visit. Our research hypotheses were that (1) VUC users promptly referred to an in-person ED by their VUC provider incur higher costs than matched in-person ED users with any type of discharge and (2) VUC users who were not referred to an in-person ED by their VUC provider incur lower 30-day costs than matched in-person ED users discharged home by the ED physician.

Methods

Study Design

We conducted [8] a population-based, matched cohort study of Ontarians who used VUC services provided by 12 of the 14 ED-led pilot sites from December 2020 to September 2021. Records from 2 sites were excluded as 1 site had a provider-to-provider VUC model (ie, physicians or nurses consulting with ED physicians) and the other site was delayed in launching the VUC program. These data sets were linked using unique encoded identifiers and analyzed at ICES (formerly the Institute for Clinical Evaluative Sciences). Patient-level VUC encounter data were linked to administrative health care databases (Multimedia Appendix 1) held at ICES to compare the 30-day costs of VUC versus in-person ED care from an MoH perspective. ICES data contain all health care services covered by the Ontario Health Insurance Plan, which include all hospital care, ED visits, hospital-based outpatient specialty clinics, physician visits for all residents of Ontario (40% of the Canadian population), and prescription drugs for residents aged 65 years and older and social assistance recipients. The analyses and reporting follow the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) guidelines [9].

Ethical Considerations

The study protocol for the collection of VUC data was reviewed and approved by the Research Ethics Board at Sinai Health (21-003E), Toronto, Ontario, Canada. Individual resident consent was not required to access and use provincial health care records for this study, as it was conducted at ICES, an independent, not-for-profit health services research institute whose legal status under Ontario’s Personal Health Information Protection Act authorizes it to collect personal health information, without consent, for the purposes of analysis or compiling statistical information with respect to the management of, evaluation or monitoring of, the allocation of resources to or planning for all or part of the health system. All data analyses were performed at ICES, where the data are securely held in linked, coded (deidentified) form.

Population

The study population included both pediatric and adult Ontario residents visiting 1 of the 12 included VUC pilot programs or any Ontario in-person ED from December 2020 to September 2021. Non-Ontario residents, individuals without a valid Ontario Health Insurance Plan number, and those who left without seeing a VUC provider were excluded from the analyses. We also excluded individuals who did not show up to an in-person ED within 72 hours after being promptly referred to the ED as it was deemed that an ED visit after 72 hours was less likely to be directly related to the initial visit issue (or at least the extent or severity of the issue at that time). As some individuals may have multiple VUC encounters, the first VUC encounter was considered the index VUC visit.

Since VUC users were expected to be different from patients who presented in person to the ED, VUC users were split into those who presented to an ED within 72 hours of VUC referral and those who saw a VUC provider with no referral to an in-person ED (“discharged home” by the VUC provider) [8]. Using propensity scoring matching methods, the first group of VUC users who presented to an ED within 72 hours of VUC referral was matched with a comparable group of individuals presenting in-person to an ED with any discharge disposition (eg, admitted, transferred, and discharged home). The second group of VUC users discharged home was matched with a comparable group of individuals presenting in person to an ED and discharged home by the ED physician. In addition, to generate comparable groups of VUC and in-person ED users based on their discharge disposition, this stratification allowed each matched group to have an equal opportunity to consume health care resources and incur costs.

Primary and Secondary Outcomes

The primary outcome was 30-day MoH costs following the first VUC encounter or matched in-person ED encounter. Secondary outcomes were 30-day health care resource use (hospitalizations, ED visits, same-day surgeries, physician visits, outpatient publicly funded drugs, laboratory tests, and nonphysician visits) and associated health care expenditures.

Statistical Analysis

To match our 2 subgroups of VUC patients with comparable patients who presented in person to an ED, we used a greedy nearest-neighbor 1:1 matching based on the index encounter date (±14 days), day (weekday vs weekend), and time of registration (day: 8 AM-8 PM; night: 8 PM-8 AM), presenting Canadian Emergency Department Information System complaint [10] and the logit of a propensity score. Canadian Emergency Department Information System includes 169 presenting complaints divided into 20 categories [10]. The variables included in the propensity score were age; sex; Statistics Canada Census neighborhood income quintile; urban or rural residence status; Ontario Marginalization Index [11] quintiles related to ethnic concentration, residential instability, material deprivation, and dependency; formal rostering with a family physician; the number of major ACG System Aggregated Diagnosis Groups derived from the Johns Hopkins ACG System version 10 (Health Services Research & Development Center, The Johns Hopkins University, Bloomberg School of Public Health) grouped as 0-4, 5-9, 10-14, 15+; medical conditions derived from validated administrative data algorithms (asthma, congestive heart failure, hypertension, chronic obstructive pulmonary disease, diabetes, and dementia); and the number of physician visits, ED visits, and hospitalizations in the 365 days preceding the index date. A caliper width of 0.2 of the SD of the logit of the propensity score was used. Balance in baseline covariates for each cohort was evaluated using standardized differences, with values less than 0.10 indicating that the groups were well matched [12].

Per-patient mean (SD) 30-day health care expenditures (eg, ED, physician, and hospitalization) expressed in 2020-2021 Canadian dollars (a currency exchange rate of CAD $1=US $0.76 is applicable) were calculated for each cohort using standardized costing algorithms for Ontario administrative health care data in Ontario [13]. The costing methodology uses a bottom-up or micro-costing approach to cost services at the individual level and costs represent amounts paid by the Ontario MoH [13]. Operational funding provided by the MoH to establish the VUC programs was added to the 30-day health care expenditures for the VUC cohort using site-specific information (ie, total operational funds received by each hospital divided by the total number of VUC visits to each hospital). Bootstrap techniques were used to generate 95% CIs associated with the difference in 30-day costs per patient between the VUC cohorts (operational funds plus health care expenditures) and matched in-person ED cohorts (health care expenditures). The same approach was used to analyze differences in secondary outcomes. In addition to the per-patient analyses, we present the total absolute costs for each cohort. All analyses were performed at ICES using linked, encoded data using SAS (version 9.4; SAS Institute).

Results

Study Population

There were 22,278 patient encounters in the pilot program, of which 19,595 patient VUC encounters were available for data linkage. This included 2931 (15%) VUC encounters that resulted in a referral to an in-person ED and 16,664 (85%) VUC encounters that did not result in a referral to an in-person ED. After applying our eligibility criteria, records from 2150 patients promptly referred to the ED and who presented to the ED within 72 hours (Multimedia Appendix 2) and records from 14,498 patients who were not referred to an in-person ED by their VUC provider (Multimedia Appendix 3) were available for the matched analyses. An additional 669 patients promptly referred to the ED by their VUC provider did not present to an ED within 72 hours and were excluded from the matched analysis as per our exclusion criteria. After 1:1 matching, 2129 patients who presented to the ED within 72 hours of a referral by a VUC provider and 14,179 patients seen by a VUC provider with no referral to an in-person ED were matched to an equal number of in-person ED individuals. Our matched VUC cohorts represented 99% (2129/2150) and 98% (14,179/14,498) of all eligible VUC patients, respectively.

VUC Patients Who Were Promptly Referred to the ED and Who Presented to the ED Within 72 Hours and Matched In-Person ED Controls

Pediatric sites accounted for approximately 41% (876/2129) of the cohort of VUC users promptly referred to the ED and who presented to the ED within 72 hours of being referred by their VUC provider. The mean (SD) age of the VUC cohort was 29.1 (24.9) years, and 10% (222/2129) were aged 65 years or older. Approximately 61% (1296/2129) of VUC users were female, 96% (2052/2129) were living in urban areas, and 96% (2050/2129) were rostered to a primary care physician. Upon ED presentation, 80% of acuity scores corresponded to Canadian Triage and Acuity Scale levels 3 (urgent: 1242/2129, 58%), 4 (less urgent: 402/2129, 19%), and 5 (nonurgent: 109/2129, 5%). The most common medical conditions of the VUC cohort were asthma (424/2129, 20%), hypertension (346/2129, 16%), and diabetes (175/2129, 8%). The mean (SD) numbers of in-person ED visits and hospitalizations in the preceding 365 days for this VUC cohort promptly referred to the ED were 1.0 (3.0) and 0.3 (0.7), respectively. Table 1 presents the baseline characteristics of our matched VUC users who presented to an ED within 72 hours of VUC referral and in-person ED cohorts (n=2129 each), which were well-balanced.

The 30-day health care expenditures were similar between the 2 cohorts. However, the 30-day MoH costs per patient were relatively greater for the VUC cohort (mean $2805, SD $7026, including mean $163, SD $99 of operational funds and mean $ 2642, SD $7017 of health care expenditures) compared to the matched in-person ED cohort (mean $2299, SD $6174), resulting in a difference of $506 (95% CI $139-$885) per patient (Table 2).

VUC Patients Not Referred to an In-Person ED and Matched In-Person ED Controls

The baseline characteristics of the VUC cohort not promptly referred to an in-person ED by their VUC provider were similar to VUC users promptly referred to the ED by their VUC provider. Table 3 presents the baseline characteristics of our matched VUC and in-person ED cohorts (n=14,179 each), which were well balanced.

The 30-day mean (SD) MoH health care expenditures per patient were lower with VUC (mean $758, SD $3129) than in-person ED care (mean $1270, SD $3846; P<.001), resulting in a difference of $511 (95% CI $434-$595) per patient in favor of VUC. More than 90% ($480/$511) of this cost difference was explained by a significant reduction in 30-day ED costs ($237/$511, 46% of total cost reduction) and physician visits ($243/$511, 47%). After adding the VUC operational funding (mean $149, SD $95 per VUC user), VUC remained less expensive than in-person ED care. The details are provided in Table 4.

Total Absolute Costs for the VUC and In-Person ED Cohorts

At the cohort level, the absolute 30-day cost associated with the cohort of VUC users who presented to an ED within 72 hours of VUC referral and matched in-person ED patients (n=2129 each) were $6.0 million and $4.9 million, respectively. For the cohort of VUC users not referred to the ED by their VUC provider and the matched cohort of individuals attending in-person to the ED and discharged home by their ED physician (n=14,179 each), the absolute 30-day costs were $12.9 million and $18.0 million, respectively. Overall, the absolute 30-day costs associated with the two VUC cohorts were $18.9 million (ie, $6.0 million + $12.9 million) versus $22.9 million ($4.9 million + $18.0 million) for the 2 in-person ED cohorts.

Discussion

Principal Findings

The data from this Ontario VUC pilot program indicate that approximately 15% (2931/19,595) of VUC encounters were promptly referred to an in-person ED. Our costing analysis of 2129 patients who presented to the ED within 72 hours of a referral by a VUC provider indicated that there were no statistically significant differences in 30-day health care expenditures when compared to matched in-person ED patients. However, when the operational costs to set up the VUC programs were considered, the VUC cohort promptly referred to the ED incurred significantly higher costs. Due to differences in ED and physician costs, the 30-day costs of the cohort of VUC users not promptly referred to the ED were significantly lower than those of a matched cohort of in-person ED users.

These findings are important for several reasons. First, the data from the Ontario VUC pilot program evaluation suggest that VUC models may represent a good investment of government funding, especially if the operational costs to set up and maintain VUC models decrease over time with subsequent implementation efficiencies. However, considering that most VUC complaints were dealt with by the VUC provider with no referral to the ED, other models of VUC where nurse practitioners, physician assistants, or family physicians provide first-line VUC instead of emergency physicians may be more attractive, especially in the context of current severe ED workforce shortages. Finally, this study proposes a costing methodology to deal with both the heterogeneity between VUC users and non-VUC users, which allows more meaningful comparisons.

Comparison With Prior Work

To our knowledge, this study is the first to evaluate the economic costs of VUC in Canada. Although several US studies have shown telemedicine or VUC for low acuity patients save costs from a health system perspective [14-17] or patient perspective [18], these results may not be generalizable to the Canadian health care system and these studies predate the COVID-19 pandemic. While recent Canadian data from April 2021 to March 2022 indicate the number of ED visits has almost returned to pre–COVID-19 levels [19], EDs across Canada continue to experience great pressures due to severe staffing shortages [20]. In this sense, VUC models could serve to complement in-person ED care.

Strengths and Limitations

Compared to previous economic studies of virtual care, this study has several strengths. First, we had access to linked records of all Ontarians who used VUC services across the VUC pilot programs, which allowed us to document the 30-day health care use and costs. Second, we were able to match 98% (16,308/16,648) of all eligible VUC users. Third, we separated the VUC users into 2 cohorts based on their discharge status following the VUC encounter (“discharged home” or “referred to in-person ED”) and we matched each VUC cohort with comparable in-person ED users (eg, the VUC users not referred to an in-person ED by their VUC provider were matched to comparable in-person ED individuals discharged home by their ED physician), which allowed us to decrease the heterogeneity between patients using VUC and in-person ED care. Finally, we used PSM methods and bootstrap techniques to compare matched VUC and in-person ED cohorts and included VUC infrastructure costs in our cost calculations.

This study also has important limitations as patients who choose VUC may be different from patients who attend the ED in person. Since the decision to use VUC versus attending an ED in person is influenced by multiple factors, many of which are not captured in administrative data (eg, self-perceived symptom severity at the time of decision, time and costs to travel to the ED, and satisfaction with prior ED services), comparing users of VUC with those who attend in-person ED is methodologically challenging. To deal with this important limitation and the heterogeneity between VUC users and between VUC and in-person ED users, we split and matched our VUC users by discharge disposition (eg, VUC presenting to the ED within 72 hours of a VUC referral or discharged home) and used PSM methods to compare with in-person ED patients. Despite matching on many measured confounders, important potential confounders were not available to us and the risk for potential unmeasured confounding remains. For example, the pediatric population and the senior population may require assistance with travel to the ED, which may influence decisions to use VUC rather than attend the ED in person. Although we matched the presenting complaint leading to the encounter, we were not able to match on acuity scores for the cohort of VUC users not referred to the ED by their VUC provider. In the VUC pilot, nearly 80% of VUC users screened themselves using a symptom checklist [7]. As such, caution should be expressed when interpreting the results of the matched analyses for the VUC cohort not referred to ED. We also did not know if VUC users would have presented in person to the ED in the absence of the VUC pilot program. They could have gone, for example, to an outpatient urgent care or walk-in clinic. However, compared to other jurisdictions, urgent care clinics are limited in Ontario. For example, while there are approximately 140 hospitals in Ontario, there are less than 20 urgent care centers in Ontario [21]. While there are more walk-in clinics in Ontario than urgent care centers, walk-in clinics do not have ED physicians on staff, and therefore, walk-in clinics may not be an appropriate comparator. In addition, as opposed to hospitalization or ED visits, reporting outpatient clinic data is not mandatory in Ontario [22]. As such, we did not have access to these data to create a control group of individuals attending urgent care outpatient clinics.

While we had access to several administrative databases, privately funded drug expenditures were not included in our analyses nor did we include patients’ time (including economic impact and opportunity costs) or out-of-pocket expenditures (eg, parking and travel), which may have been avoided by the availability of VUC. As this was a pilot project, the population who would use VUC should it become universally available across Ontario may be different. However, we believe the mix of the VUC sites participating in the pilot program is broadly representative of the breadth of ED settings in Ontario. There may be differences in 30-day health care use and costs between models of care (eg, nurse triage vs self-screening), populations (pediatrics vs adults), or settings (community vs academic sites; rural vs urban), which we did not examine. Due to data limitations, we were not able to isolate the physician costs associated with the VUC or in-person ED encounter. Finally, the MoH operational funds used in our calculations may represent an underestimation of the true costs of setting up VUC programs as it did not include any in-kind support provided by the sites. While the operational costs to set up and maintain VUC models may decrease with efficiencies in implementation, the cost of providing VUC is likely to change if different models of care are implemented (eg, nurse practitioners or physician assistants as first VUC contact before escalating to the ED physician if needed) or if the VUC technology becomes more expensive. Generalizations on health system costs within this study are limited to acute and subacute patient outcomes treated within the 30-day window of this study, and results may not be generalizable to other time periods or health care settings. Finally, due to the lack of data, our analyses were limited to costs and did not include economic outcomes (eg, quality-adjusted life-years). Therefore, the cost-effectiveness of the VUC program is unknown and this is left for future research.

Conclusions

This costing analysis supports the use of VUC for low acuity presenting complaints as most patient concerns were addressed without an in-person referral to the ED. However, additional research should evaluate alternative models of VUC to inform future resource allocation and policy decisions.