This paper is in the following e-collection/theme issue:

Digital Health Reviews (1289) Telehealth and Telemonitoring (1671) Clinical Communication, Electronic Consultation and Telehealth (590) Email Communication, Web-Based Communication, Secure Messaging (160) Personal Health Records, Patient-Accessible Electronic Health Records, Patient Portals (573)

Published on in Vol 23, No 5 (2021): May

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/27531, first published .
Impact of Asynchronous Electronic Communication–Based Visits on Clinical Outcomes and Health Care Delivery: Systematic Review

Impact of Asynchronous Electronic Communication–Based Visits on Clinical Outcomes and Health Care Delivery: Systematic Review

Impact of Asynchronous Electronic Communication–Based Visits on Clinical Outcomes and Health Care Delivery: Systematic Review

Authors of this article:

Oliver T Nguyen1, 2 Author Orcid Image ;   Amir Alishahi Tabriz3, 4 Author Orcid Image ;   Jinhai Huo1 Author Orcid Image ;   Karim Hanna5 Author Orcid Image ;   Christopher M Shea6 Author Orcid Image ;   Kea Turner3, 4 Author Orcid Image
Article Authors Cited by (29) Tweetations (24) Metrics

    Review

    1Department of Health Outcomes and Biomedical Informatics, University of Florida, Gainesville, FL, United States

    2Department of Health Services Administration, University of Alabama at Birmingham, Birmingham, AL, United States

    3Department of Health Outcomes and Behavior, Moffitt Cancer Center, Tampa, FL, United States

    4Department of Oncological Sciences, University of South Florida, Tampa, FL, United States

    5Department of Family Medicine, Morsani College of Medicine, University of South Florida, Tampa, FL, United States

    6Department of Health Policy and Management, University of North Carolina at Chapel Hill, Chapel Hill, NC, United States

    Corresponding Author:

    Kea Turner, MPH, MA, PhD

    Department of Health Outcomes and Behavior

    Moffitt Cancer Center

    12902 USF Magnolia Drive, MRC-CANCONT

    Tampa, FL, 33612

    United States

    Phone: 1 (813) 745 5213

    Fax:1 (813) 745 6525

    Email: kea.turner@moffitt.org


    Background: Electronic visits (e-visits) involve asynchronous communication between clinicians and patients through a secure web-based platform, such as a patient portal, to elicit symptoms and determine a diagnosis and treatment plan. E-visits are now reimbursable through Medicare due to the COVID-19 pandemic. The state of evidence regarding e-visits, such as the impact on clinical outcomes and health care delivery, is unclear.

    Objective: To address this gap, we examine how e-visits have impacted clinical outcomes and health care quality, access, utilization, and costs.

    Methods: We conducted a systematic review; MEDLINE, Embase, and Web of Science were searched from January 2000 through October 2020 for peer-reviewed studies that assessed e-visits’ impacts on clinical and health care delivery outcomes.

    Results: Out of 1859 papers, 19 met the inclusion criteria. E-visit usage was associated with improved or comparable clinical outcomes, especially for chronic disease management (eg, diabetes care, blood pressure management). The impact on quality of care varied across conditions. Quality of care was equivalent or better for chronic conditions, but variable quality was observed in infection management (eg, appropriate antibiotic prescribing). Similarly, the impact on health care utilization varied across conditions (eg, lower utilization for dermatology but mixed impact in primary care). Health care costs were lower for e-visits than those for in-person visits for a wide range of conditions (eg, dermatology and acute visits). No studies examined the impact of e-visits on health care access. It is difficult to draw firm conclusions about effectiveness or impact on care delivery from the studies that were included because many used observational designs.

    Conclusions: Overall, the evidence suggests e-visits may provide clinical outcomes that are comparable to those provided by in-person care and reduce health care costs for certain health care conditions. At the same time, there is mixed evidence on health care quality, especially regarding infection management (eg, sinusitis, urinary tract infections, conjunctivitis). Further studies are needed to test implementation strategies that might improve delivery (eg, clinical decision support for antibiotic prescribing) and to assess which conditions can be managed via e-visits.

    J Med Internet Res 2021;23(5):e27531

    doi:10.2196/27531

    Keywords

    telemedicine; telehealth; e-visits; electronic visits; digital care; outcome; delivery; review; access; utilization; cost; patient portal; eHealth 


    Telemedicine—or the delivery of health care at a distance—can improve health care access and quality while reducing health care utilization and costs [1]. For instance, telemedicine can improve access to specialists for patients in rural areas or in underresourced care settings by increasing the convenience and availability of health care (eg, extended hours, decreased wait times) [2-4]. Studies have also demonstrated that telemedicine may achieve comparable clinical outcomes to in-person care across a variety of conditions, such as stroke care [5], heart failure [6,7], hepatitis C [8], and diabetes [8]. Telemedicine can also reduce the utilization of in-person care and reduce health care costs [3,4,9]. Furthermore, some telemedicine types have demonstrated cost-effectiveness [10,11]. Studies have shown that the impact of telemedicine on health care delivery and patient outcomes varies across telemedicine types. Some forms of telemedicine, such as telestroke [12-14], have a strong evidence base while other forms of telemedicine, such as electronic visits (e-visits), are understudied.

    E-visits involve asynchronous communication between clinicians and patients through a secure web-based platform, such as a patient portal. Generally, patients answer questions about their medical history and symptoms through a structured questionnaire and upload photos (if relevant). The data are then reviewed by a clinician, who develops a diagnosis and treatment plan. Although e-visits often involve clinicians who the patient is familiar with, they can also involve a third-party clinician through direct-to-consumer telemedicine. E-visits offer notable benefits, such as allowing patients and clinicians to communicate at a convenient time (ie, eliminating scheduling barriers) and improving documentation of patient-clinician communication (eg, patients can review clinician’s instructions) [15,16]. However, e-visits have a number of implementation barriers, such as those regarding workflow integration (eg, having dedicated clinician time to respond to messages) [16-18], those regarding lack of reimbursement [15,16,19,20], and concerns about the quality of communication from patients (eg, failing to submit sufficient information for diagnosis) [21]. Health care organizations have also raised concerns about the quality of care provided through e-visits, such as the potential for inappropriate antibiotic prescribing and difficulties with providing care without being able to see the patient face-to-face [22-29]. Despite these challenges, studies have reported positive effects of e-visits on health care delivery, such as lower costs [24], comparable follow-up rates to those of in-person care (ie, a proxy for diagnostic accuracy) [30], and comparable or improved patient outcomes (eg, lower uric acid level for patients with gout) [31].

    Because of COVID-19, e-visits are being implemented more frequently and are now reimbursable by Medicare and other payers in the United States [32,33]. As the nation moves forward in telemedicine implementation and health care systems decide whether to integrate telemedicine into future care delivery, it is critical to determine the costs and benefits of telemedicine models, such as e-visits. To date, a systematic review has not been conducted to assess the state of the evidence regarding e-visits. To address this gap, the objective of this review is to summarize the findings of studies examining the association of e-visits with clinical outcomes, quality of care, access to care, utilization, and costs.


    We conducted a systematic review based on PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analysis) guidelines [34].

    Data Sources and Searches

    MEDLINE, Embase, and Web of Science were searched to locate peer-reviewed studies published from January 2000 through October 2020. The start date for the search was chosen because electronic visits were developed recently, and the authors did not anticipate any studies published prior to 2000 (when web-based, patient portal studies emerged). A preliminary search was done to confirm this. Studies were limited to those in the United States as systemic factors, such as reimbursement, may influence the results. All reference lists for included studies were cross-searched. Duplicate studies were removed. Multimedia Appendix 1 lists the search terms used for this search after consultation with a health sciences librarian at the University of Florida.

    Study Selection

    Two reviewers (KT, OTN) independently screened papers identified from the search strategy, and assessed each for inclusion eligibility using a spreadsheet (Excel 2013, Microsoft Inc). Any discrepancy encountered was discussed until a consensus was reached.

    E-visits were defined as any asynchronous electronic visit where a clinician assesses a patient’s health status, makes a diagnosis, and develops a treatment plan via a secure messaging system (eg, patient portal) [24,35]. This definition was based on how recent studies have defined e-visits [24,35]. Studies that defined an e-visit differently were excluded (eg, defines e-visit as real-time, 2-way communication). Studies were included that reported on the impact of e-visits on clinical outcomes, health care quality, access, utilization or costs. Included studies also had to be written in English, empirical (ie, reporting original research), quantitative, and peer reviewed.

    Data Extraction and Quality Assessment

    For each included study, 2 investigators (KT, OTN) noted study design, outcome measurements, care setting, medical conditions studied, sample size, and major findings. Since most of the studies included were observational, the authors used the Risk of Bias Assessment Tool for Nonrandomized Studies to assess criteria specific to observational studies [36]. A P-value <.05 was considered significant.

    Data Synthesis and Analysis

    Due to the heterogeneity of outcome measures used in our included studies, it was infeasible to conduct a meta-analysis. Consequently, findings were qualitatively grouped by outcome type (eg, clinical outcomes, costs).


    After reviewing 1859 studies, a total of 19 studies met our inclusion criteria. Figure 1 shows our study selection process.

    ‎
    Figure 1. PRISMA [34] flowchart.
    View this figure

    Study Characteristics

    Most studies assessed clinical outcomes, quality of care, health care utilization, and costs. None reported access outcomes. Most studies were observational in design, with one study using a randomized experimental design, another study using a case study design, and another 2 studies employing quasi-experimental designs. Sixteen studies used cross-sectional or pooled cross-sectional data. Three studies used a panel design. Table 1 summarizes the study characteristics of included studies, and Table 2 summarizes findings with respect to clinical outcomes, quality of care, access to care, health care utilization, and costs.

    Table 1. Study characteristics.
    CitationTypes of health system measureEntity provisioning e-visitsCare settingMedical conditions treatedSample size, n
    Adamson et al [19]Costs, utilizationHealth care organizationOutpatient clinic, academic medical centerSinusitis, depression, back pain, cough, anxiety, hypertension, abdominal pain, headache, urinary tract infections, influenza, allergic rhinitis, dermatitis, attention-deficit/hyperactive disorder, gastroesophageal reflux disease, vaginitis, upper respiratory infection, insomnia, asthma, contraception, hyperlipidemia2531
    Rohrer et al [37]Costs, utilizationHealth care organizationOutpatient clinicConjunctivitis, sore throat, viral illness, bronchitis, cough390
    Watson et al [38]Clinical outcomesHealth care organizationOutpatient clinicAcne121
    Albert et al [39]Costs, utilizationHealth care organizationOutpatient clinic, academic medical centerBlood pressure management, fractures, diabetes management, skin conditions, prostatitis, pain, sleep issues, vomiting, mononucleosis, hemorrhoids, cold symptoms121
    Courneya et al [40]Quality of care, costs, utilizationInsurerN/AaAcute sinusitis, chronic sinusitis, urinary tract infections, conjunctivitis, viral upper respiratory infection, lower genitourinary system infection, yeast infection, otolaryngology diseases, acne, allergic rhinitis, acute bronchitisReported as more than 40,000
    Mehrotra et al [24]Quality of care, costs, utilizationHealth care organizationOutpatient clinic, academic medical centerSinusitis, urinary tract infections574
    North et al [41]Costs, utilizationHealth care organizationOutpatient clinic, academic medical centerNo specific conditions studied, but e-visit content mentioning chest pain and respiratory complications were monitored892
    Heyworth et al [42]Quality of careHealth care organizationInpatient, Veterans AffairsDiabetes, hypertension, hyperlipidemia, heart disease, prior history of myocardial infarction or stroke51
    Pathipati et al [43]Costs, utilizationHealth care organizationOutpatient clinic, academic medical centerRash, acne, other unspecified dermatological conditions38
    Hawes et al [44]Clinical outcomes, quality of careHealth care organizationOutpatient clinic, academic medical centerDiabetes, anticoagulation management36
    Levine et al [45]Clinical outcomes, costs, utilizationHealth care organizationOutpatient clinic, academic medical centerHypertension1786
    Penza et al [46]Quality of care, costs, utilizationHealth care organizationOutpatient clinic, academic medical centerConjunctivitis505
    Penza et al [47]Clinical outcomes, costs, utilizationHealth care organizationOutpatient clinic, academic medical centerAllergies, upper respiratory infection, cold sores, influenza, lice, conjunctivitis, sinusitis, sore throat, sunburn, tick exposure, urinary tract infections, yeast infection1009
    Player et al [48]Costs, utilizationHealth care organizationOutpatient clinic, academic medical centerAthlete’s foot, allergic skin reaction, eczema, cold sore, shingles rash, marine animal string, jock itch, nosebleed, poison ivy, rash, red eye, ringworm, scabies, hemorrhoids, sunburn, back pain, gout, heartburn, seasonal allergies, travel precaution, prescription refills, sexually transmitted infections, diarrhea, influenza, sinus problems, urinary problems, vaginal irritation or discharge1565
    Rajda et al [49]Costs, utilizationDirect-to-consumer companyN/AAcne vulgaris, atopic dermatitis, onychomycosis, psoriasis vulgaris, rosacea395
    Hertzog et al [30]Costs, utilizationHealth care organizationOutpatient clinic, academic medical centerYeast infection, acne, allergic rhinitis, urinary tract infections, upper respiratory infection, conjunctivitis, oral sores, irritable bowel syndrome, tobacco cessation2691
    Murray et al [50]Clinical outcomes, quality of careHealth care organizationOutpatient clinic, academic medical centerUrinary tract infections300
    Penza et al [51]Clinical outcomes, quality of careHealth care organizationOutpatient clinic, academic medical centerAcute sinusitis300
    Yokose et al [31]Quality of careHealth care organizationOutpatient clinic, academic medical centerGout124

    aN/A: not applicable.

    Table 2. Effects of asynchronous e-visits on clinical outcomes, quality, utilization, and costs.
    Outcome type and citationsSpecific measuresImpact
    Clinical outcomes


    		Hawes et al [44]Abnormal international normalized ratioE-visits were associated with lower abnormal international normalized ratios than in-person visits (P<.05).
    Hawes et al [44]HbA1caE-visits were associated with lower HbA1c values than in-person visits (P<.001).
    Hawes et al [44]Amount of diabetic patients with controlled blood pressureCompared to the preimplementation period, more diabetic patients were observed with controlled blood pressure in the postimplementation period (P<.001).
    Hawes et al [44]Amount of diabetic patients with HbA1c levels of less than 8%Compared to the preimplementation period, more diabetic patients with HbA1c levels of less than 8% were observed in the postimplementation period (P<.0001).
    Hawes et al [44]Amount of diabetic patients with HbA1c levels of less than 7%Compared to the preimplementation period, more diabetic patients with HbA1c levels of less than 7% were observed in the postimplementation period (P<.001).
    Levine et al [45]Systolic blood pressureEquivalent outcomes
    Watson et al [38]Total inflammatory lesion countsEquivalent outcomes
    Watson et al [38]Frontal inflammatory lesion countsEquivalent outcomes
    Watson et al [38]Leeds scoreEquivalent outcomes
    Penza et al [47,51]Mortality ratePenza et al reported only descriptive statistics in both studies, so it is unclear if there are differences in mortality rates between e-visits and in-person visits.
    Murray et al [50]; Penza et al [51]HospitalizationsMurray et al [50] and Penza et al [51] reported only descriptive statistics, so it is unclear if there are differences in the number of related hospitalizations between e-visits and in-person visits.
    Murray et al [50]Antibiotic retreatment rateEquivalent outcomes
    Yokose et al [31]Proportion of patients serum urate levels of less than 6.0 mg/dLE-visits had greater proportions of patients with optimal control of serum urate levels when compared to in-person visits (P<.01).
    Quality of care


    		Hawes et al [44]Amount of diabetic patients receiving aspirin, if clinically indicatedEquivalent outcomes
    Hawes et al [44]Amount of diabetic patients receiving moderate-intensity statinsEquivalent outcomes
    Hawes et al [44]Amount of diabetic patients receiving high-intensity statinsEquivalent outcomes
    Heyworth et al [42]Medication discrepancy discovery rateIt is unclear what the impact is on the rate of discovering medication discrepancies as no P value was reported.
    Mehrotra et al [24]Order rate of diagnostic testE-visits had a lower order rate of diagnostic tests when compared to in-person visits (P<.001).
    Mehrotra et al [24]Order rate of preventive care servicesE-visits had a lower order rate of preventive care services when compared to in-person visits (P<.01).
    Yokose et al [31]Rate that serum urate levels were checkedE-visits had more frequent checks of serum urate levels when compared to in-person visits (P<.05).
    Murray et al [50]; Penza et al [46,51]; Mehrotra et al [24]; Courneya et al [40]Antibiotic prescribing rateMehrotra et al [24] reported that the rate of prescribing antibiotics was higher during e-visits than in-person visits for sinusitis (P<.001) but not for e-visits for urinary tract infections. However, Penza et al [46] and Murray et al [50] saw equivalent outcomes. Penza et al [51] reported e-visits had lower antibiotic prescribing rates than in-person visits (P<.001).
    Courneya et al [40] also investigated the association but did not report a P value, so the impact on antibiotic prescribing rate is unclear.
    Health care utilization


    		Levine et al [45]Overall primary care visit utilizationEquivalent outcomes

    		Levine et al [45]Overall specialist visit utilizationEquivalent outcomes

    		Levine et al [45]Overall emergency department utilizationEquivalent outcomes

    		Levine et al [45]Overall inpatient admissionsEquivalent outcomes

    		Rajda et al [49]Number of specialist procedures done 60- and 90-days after initial consultationE-visits were associated with a lower number of specialist procedures performed 60 and 90 days after an initial consultation when compared to in-person visits (P<.01)

    		Murray et al [50]; Penza et al [51]30-day follow-up rate (planned and unplanned)Equivalent outcomes

    		Penza et al [46]; Pathipati et al [43]; Albert et al [39]; Player et al [48]; Adamson et al [19]Rate of patients who need planned follow-up visitsPenza et al [46] reported that e-visits were associated with higher rates of planned follow-up visits than in-person visits (P<.001).
    Pathipathi et al [43], Albert et al [39], Player et al [48], and Adamson et al [19] reported only the proportion of e-visits that required follow-up visits, so it is unclear what the association of e-visit usage and rate of planned follow-up visits is.

    		Penza et al [47]; Mehrotra et al [24]; Courneya et al [40]; North et al [41]; Hertzog et al [30]Unexpected follow-up encounter rate after initial encounterHertzog et al [30] reported that e-visits were associated with higher unexpected follow-up rates when compared to in-person visits (P<.05). However, Mehrotra et al [24], Courneya et al [40], and North et al [41] found equivalent outcomes. Penza et al [47] reported only descriptive statistics, so it is unclear if there are differences in unexpected follow-up encounter rates between e-visits and in-person visits.
    Health care costs


    		Rajda et al [49]; Courneya et al [40]; Rohrer et al [37]Treatment costsCourneya et al [40] and Rajda et al [49] reported e-visits were associated with lower treatment costs (P<.001). Rohrer et al [37] reported a lower median of costs associated with e-visits than in-person visits (P<.01).

    aHbA1c: hemoglobin A1c.

    Study Quality Assessment Results

    Detailed results of the quality assessment are summarized in Multimedia Appendix 2. Each column describes the quality criterion we assessed. Briefly, most (12/19, 63.2%) studies reported a strategy for minimizing selection bias. Less than half of studies (9/19, 47.4%) reported methods to control for confounders. All studies measured outcomes for e-visits separately from other forms of telemedicine, making it possible to evaluate the unique impact of e-visits on the outcome of interest. Problems with low response rate (<50%) or attrition bias (>10% dropout) were less common (2/19, 10.5%).

    Clinical Outcomes

    Associations between e-visit usage and clinical outcomes were reported by 7 studies. A total of 13 different measures were used among the studies. Overall, the studies reported an association with improved outcomes or null findings across the medical conditions examined.

    Among diabetic patients, one study (n=36 patients) found that e-visits were associated with significantly improved glucose over a 6-month period (–3.4 percentage points in HbA1c, P<.001) [44]. In the same study, e-visits for anticoagulant management were associated with less frequent instances of abnormal international normalized ratio values compared to in-person care (5/104, 5% vs 1/198, 0.5%, P<.05) [44]. An additional study (n=62) reported that patients with gout who received care through e-visits were more likely to have optimal serum rate levels (>6.0 mg/dL) (63.8% vs 33.9%, P<.01) and lower mean serum urate levels (5.5 mg/dL vs 6.7 mg/dL, P<.01) compared to historical controls [31]. In the context of acne and hypertension management, equivalent outcomes were reported between e-visits and in-person visits [38,45].

    Quality of Care

    Across the 8 studies that examined the association of e-visit usage on quality of care, 8 unique measures were observed. Overall, there were mixed effects on the association of e-visit usage on quality of care.

    Mixed results were observed in some quality of care measures of several health care conditions between e-visits and in-person visits [44]. For example, one study (n=36) [44] reported comparable prescribing rates for statins among patients with diabetes across e-visits and in-person visits. One study [24] found that e-visit usage was associated with significantly lower rates of diagnostic procedures for sinusitis (0/475, 0% vs 40/4690, 1%, P=.04) and lower rates of diagnostic procedures for urinary tract infections (8/99, 8% vs 1501/2855, 53%, P<.001) than those for historical controls. The same study [24] also reported that preventive screenings were lower among sinusitis patients receiving an e-visit (1/475, 0.2% vs 155/4690, 3%; P<.001) and urinary tract infection patients’ receiving an e-visit (0/99, 0% vs 214/2855, 7%, P=.005) than in those receiving in-person visits.

    When examining management of acute infections (eg, urinary tract infections), the definitive impact was less clear. One study [24] found that e-visits resulted in higher antibiotic prescribing rates for sinusitis (471/475, 99% vs 4408/4690, 94%, P<.001) but no association for urinary tract infections (98/99, 99% vs 1299/2855, 92%, P=.07), compared to those for historical control. Another study [46] reported that e-visits for conjunctivitis resulted in a significantly lower antibiotic prescribing rate than that of phone visits (26/101, 26% vs 84/202, 42%, P=.006), and a third study [51] found that e-visits for sinusitis resulted in a significantly lower antibiotic prescribing rate (84/150, 56% vs 108/150, 72%, P=.01) than that of in-person visits. Lastly, one study (n=450) [50] reported no difference in antibiotic prescribing rates for treatment of urinary tract infections across e-visits, phone encounters, and in-person visits.

    Access to Care

    No studies investigated the association between e-visits and access to care.

    Health Care Utilization

    Fourteen studies assessed health care utilization associated with e-visits. An overall mixed impact was observed across these studies.

    Some studies evaluated the impact of e-visits on subsequent health care utilization, such as primary care, specialty care, and emergency care [45,49-51]. For example, one study [45] matched 893 e-visits and 893 in-person visits for hypertension and found that e-visits resulted in fewer primary care visits (–0.8 visits, 95% CI 0.3-1.2) compared to in-person visits. The same study [45] also found that usage of specialist visits, emergency department visits, and inpatient admissions were not significantly different across in-person and e-visits. Another study [49] evaluating a teledermatology program found that e-visits were associated with significantly fewer specialty visits at 60-day (15 vs 46, P=.005) and 90-day follow-up (26 vs 74, P=.001) compared to those associated with in-person visits. Another study [40] examining a direct-to-consumer telemedicine program found that the rate of visits that did not require a follow-up visit (ie, resolution rate) was similar for e-visits for sinusitis (90% vs 91%) and conjunctivitis (94% vs 95%) to those for in-person visits.

    Some studies [24,40,41] reported no differences in health care utilization. For example, one study [24] found that e-visits for sinusitis and urinary tract infections had equivalent rates of follow-up visits, phone calls, and emails within 3-week follow-up. Some studies [30,46] reported a higher rate of health care utilization with e-visits. For example, one study [30] found that the rate of follow-up visits was higher for primary care e-visits (59/490, 12%) compared to that for in-person visits (198/2201, 9%; P=.04).

    Health Care Costs

    Three studies [37,40,49] found e-visit usage was associated with lower overall treatment costs than those for in-person visits. One study utilizing claims data found that e-visits for dermatology resulted in lower mean costs at the initial visit (US $59 vs $113, P<.001), at 30-day follow-up ($70 vs $202, P=.03), and at 60-day follow-up ($78 vs $221, P=.02) than those of in-person visits; however, at the 90-day follow-up, costs were comparable to those of an in-person visit ($86 vs $307, P=.08) [49]. One study [40] of a direct-to-consumer e-visit platform reported lower costs for a wide-range of conditions, including sinusitis, conjunctivitis, acne, and ear, nose, and throat infections. As an example, that study (n=9551 visits) used claims data to report that the cost per visit for e-visits was significantly lower for acne management ($178 vs $361, P<.001) compared to in-person care [40]. A third study [37] calculated total reimbursable costs and descriptively reported the median cost per visit was lower for e-visits ($161 vs $219) compared to in-person visits for acute conditions (eg, conjunctivitis, sore throat, bronchitis, viral illness, cough).


    This was a systematic review that assessed the impact of e-visits on clinical outcomes and health care delivery. To our knowledge, this is the first systematic review to assess the state of evidence for asynchronous e-visits. Most studies found that e-visits were associated with lower treatment costs and comparable clinical outcomes to in-person visits. Studies reported mixed effects on health care utilization, and no studies evaluated the impact of e-visits on health care access, suggesting future research is needed in this area. We provide implications for research and practice below.

    Our review found that e-visits may be an adequate substitute for in-person care for chronic disease management. Studies showed that e-visits were effective for the treatment of diabetes, hypertension, and gout. Our findings are consistent with those of previous studies [52] that suggest that other forms of asynchronous communication with providers (eg, secure messaging without a formal e-visit) can improve chronic disease management. Further study, however, is needed to determine the ideal conditions for e-visit implementation. For example, many chronic conditions co-occur with other conditions (eg, diabetes and hypertension), and current studies were not designed to determine whether e-visits are effective for complex patients, such as patients with multiple chronic conditions and older adults who are frail. Additionally, it is important to note that not all types of medical issues may be appropriate for asynchronous management and health care systems may need to implement safeguards to ensure that the right patient is using an e-visit [53]. For example, if patients send a message about an urgent condition, e-visit technology could be harnessed to flag certain keywords (eg, chest pain, breathing difficulties) and display automated pop-up alerts to patients to instruct them to seek care in-person at the office or emergency department [54]. Some health care systems have also implemented guidelines, such as requiring at least one in-person visit prior to an e-visit to ensure that a patient has an established relationship with a clinician [55]. Additional research is needed to determine how e-visits should be implemented (eg, how to prevent inappropriate usage and for which patients does it work best).

    Our review found mixed evidence for the effect of e-visits on quality of care. For example, prior studies [22,24-29] have raised concerns that telemedicine usage can increase inappropriate antibiotic prescribing, but this review found mixed evidence on whether comparable antibiotic prescribing rates were observed between e-visits and in-person visits. This inconsistency may stem from differences in the acute infection that was studied (ie, acute sinusitis, urinary tract infections, ear infections, conjunctivitis). Further research is needed to better understand when e-visits can be used effectively for managing acute infections and what implementation strategies can be used to ensure appropriate antibiotic prescribing (eg, use of clinical decision support). Furthermore, additional research is needed to compare the receipt of low-value care (eg, overutilization of services for sinusitis and urinary tract infections across e-visits and in-person visits). Future studies should examine guidelines, such as the Choosing Wisely guidelines [56,57], to see whether e-visits reduce low-value care compared to in-person visits. One study [24] included in this review found that e-visits were associated with lower rates of diagnostic procedures for sinusitis and urinary tract infections and may offer an advantage in terms of reducing unnecessary health care utilization. Studies in this review did report other quality problems, such as lower use of preventive care services in e-visits compared to in-person visits. This may stem from a lack of practice guidance on how e-visits should be implemented (eg, should providers use e-visits as an opportunity to reinforce messages about preventive care?). Prior studies [16] have noted that lack of practice guidance or standard procedures as barriers to e-visit implementation. These differences in quality of care suggest additional implementation research is needed to test implementation strategies for ensuring quality of care delivered through e-visits is consistent across clinicians.

    Our review found that e-visits had mixed effects on health care utilization compared to in-person visits. These findings are consistent with those of reviews on other forms of telemedicine [6,58-62]. Specifically, telemedicine can reduce health care utilization in certain instances (eg, reduce the need for in-person visits) or it can increase utilization (eg, meet an unmet demand for a patient that was not previously accessing in-person care). Variation in health care utilization may stem from other various factors that are unaccounted for. For example, if a patient did not submit sufficient information to be evaluated through an e-visit, this could lead to greater health care utilization compared to an initial in-person visit where information exchange is synchronous. Future research is needed to determine strategies for ensuring that complete information is elicited from the patient (eg, structured symptom questionnaires). Additional research should test strategies for optimizing patient data collection and patient-clinician communication through e-visit platforms.

    The findings on favorable cost implications align with those of other works that evaluated the effects that synchronous alternatives for care (eg, telemedicine via teleconferencing software) had on cost outcomes for patients [63-70]. Notably, 2 [40,49] of 3 studies used claims data to estimate health care costs; however, insurers commonly include in their contracts with health care organizations contractual adjustments (ie, slightly lower reimbursement rates in exchange for including the health care organization in-network) [71], suggesting that using claims data may underestimate the true cost savings potential of e-visits. Furthermore, many of these studies did not evaluate cost comprehensively (eg, only examined costs of dermatology visits and not visits to providers outside of that delivery system, analyzing average costs per visit instead of average costs of an episode of illness). Additionally, we did not identify any studies that assessed cost-effectiveness, which considers costs relative to outcomes. Future cost-effectiveness studies may also help health care systems make decisions about whether e-visits are worth the investment.

    Additional research is also needed on how e-visits impact health care access and the digital divide. There is growing evidence that patient-level disparities exist across adoption and usage patterns of patient portals [72-76]. Similarly, several studies in this review reported differences in usage based on sex [19,30,39,41,42,48], age [30,42,47], employment status [41,48], and ethnicity [19,41]. Since many patients access e-visits through the patient portal, which has known disparities in uptake [72-76], future studies are needed to test strategies for overcoming disparities in patient portal adoption. Since the Pew Research Center reports 81% of Americans in 2019 owned a smartphone [77], there have been studies that have recently tested whether smartphone access to a patient portal could improve access [78,79]. Future studies should test whether strategies, such as smartphone access, could improve uptake of e-visits.

    Some of the included literature cited implementation barriers in their discussion that should be further explored in future studies (eg, lack of integration into workflow). Clinicians may need assistance with adjusting their workflows when being trained on e-visits [80]. Best practices should be researched and disseminated to help alleviate concerns on increased workload [81]. Furthermore, the adoption of payment models by insurers that reimburse for e-visits may be a crucial facilitator of e-visit uptake [20,82,83]. Since the COVID-19 pandemic has spurred the introduction of insurance coverage of e-visits among Medicare and some private payers, case studies have been published that suggest e-visits are being used more frequently during the COVID-19 pandemic. For example, one health care system reported that the use of e-visits increased by 4000% during the COVID-19 pandemic and that the majority of e-visits were used for remotely managing patients with COVID-19 [84-86]. Further work is needed to evaluate if the reimbursement policies have led to higher utilization of e-visits by patients and health care organizations.

    This systematic review has several limitations. First, a majority of the studies used an observational design, limiting our ability to draw causal conclusions on the effect of using e-visits on quality of care, access to care, costs or clinical outcomes. Second, we found heterogeneity in how studies measured the impact of e-visits, making it impossible to quantitatively pool study estimates. Third, we excluded non-English studies, which may limit our ability to determine the effect of e-visits outside of English-speaking regions. Fourth, the follow-up periods among the studies varied (eg, 2 weeks vs 1 month), limiting our comparisons of findings between studies. Fifth, a majority of the studies examined e-visits that are hosted by the health care delivery organization rather than by a direct-to-consumer vendor or insurer, limiting the generalizability of our results to e-visit programs sponsored by entities external to the health care delivery organization. Lastly, most e-visits occurred in outpatient ambulatory and academic medical center contexts, limiting our ability to comment on inpatient settings.

    Overall, the evidence suggests that e-visits can provide equivalent outcomes to in-person care and reduce health care costs for certain health care conditions. There are still notable quality concerns (eg, inappropriate antibiotic prescribing, underutilization of preventive care) that warrant further study. It is also unknown how e-visits have affected access to care. Furthermore, many studies in the review lacked a rigorous design, making it difficult to draw firm conclusions about the state of evidence regarding e-visits. Future trials should be conducted to test the effectiveness of e-visits and determine what factors drive effective implementation.

    Acknowledgments

    We owe many thanks to Nancy Schaefer for providing her expertise as a health science librarian by providing us a framework on how to search for relevant articles and suggesting search terms to supplement our original search strategy.

    Authors' Contributions

    KT conceptualized the study. OTN and KT drafted the manuscript. All authors participated in the analysis and interpretation of data. AAT, JH, KH, CMS, and KT provided critical revisions to the manuscript. All authors approved the submission.

    Conflicts of Interest

    None declared.

    ‎
    Multimedia Appendix 1

    Search strategy.

    DOCX File , 8 KB
    ‎
    Multimedia Appendix 2

    Quality assessment of studies.

    DOCX File , 8 KB
    1. The Role of Telehealth in an Evolving Health Care Environment: Workshop Summary. Washington, DC: The National Academies Press; Nov 20, 2012.
    2. Zhang D, Wang G, Zhu W, Thapa JR, Switzer JA, Hess DC, et al. Expansion of telestroke services improves quality of care provided in super rural areas. Health Aff (Millwood) 2018 Dec;37(12):2005-2013. [CrossRef] [Medline]
    3. Albritton J, Maddox L, Dalto J, Ridout E, Minton S. The effect of a newborn telehealth program on transfers avoided: a multiple-baseline study. Health Aff (Millwood) 2018 Dec;37(12):1990-1996. [CrossRef] [Medline]
    4. Anderson D, Villagra VG, Coman E, Ahmed T, Porto A, Jepeal N, et al. Reduced cost of specialty care using electronic consultations for Medicaid patients. Health Aff (Millwood) 2018 Dec;37(12):2031-2036. [CrossRef] [Medline]
    5. Johansson T, Wild C. Telemedicine in acute stroke management: systematic review. Int J Technol Assess Health Care 2010 Apr;26(2):149-155. [CrossRef] [Medline]
    6. Pekmezaris R, Tortez L, Williams M, Patel V, Makaryus A, Zeltser R, et al. Home telemonitoring in heart failure: a systematic review and meta-analysis. Health Aff (Millwood) 2018 Dec;37(12):1983-1989. [CrossRef] [Medline]
    7. Lin M, Yuan W, Huang T, Zhang H, Mai J, Wang J. Clinical effectiveness of telemedicine for chronic heart failure: a systematic review and meta-analysis. J Investig Med 2017 Dec;65(5):899-911. [CrossRef] [Medline]
    8. Zhou C, Crawford A, Serhal E, Kurdyak P, Sockalingam S. The impact of project ECHO on participant and patient outcomes: a systematic review. Acad Med 2016 Oct;91(10):1439-1461. [CrossRef] [Medline]
    9. Shah SJ, Schwamm LH, Cohen AB, Simoni MR, Estrada J, Matiello M, et al. Virtual visits partially replaced in-person visits in an ACO-based medical specialty practice. Health Aff (Millwood) 2018 Dec;37(12):2045-2051. [CrossRef] [Medline]
    10. Switzer JA, Demaerschalk BM, Xie J, Fan L, Villa KF, Wu EQ. Cost-effectiveness of hub-and-spoke telestroke networks for the management of acute ischemic stroke from the hospitals' perspectives. Circ Cardiovasc Qual Outcomes 2013 Jan 01;6(1):18-26. [CrossRef] [Medline]
    11. Nelson RE, Saltzman GM, Skalabrin EJ, Demaerschalk BM, Majersik JJ. The cost-effectiveness of telestroke in the treatment of acute ischemic stroke. Neurology 2011 Oct 25;77(17):1590-1598 [FREE Full text] [CrossRef] [Medline]
    12. Baratloo A, Rahimpour L, Abushouk AI, Safari S, Lee CW, Abdalvand A. Effects of telestroke on thrombolysis times and outcomes: a meta-analysis. Prehosp Emerg Care 2018 Jan 18:1-13. [CrossRef] [Medline]
    13. Rubin MN, Wellik KE, Channer DD, Demaerschalk BM. Systematic review of telestroke for post-stroke care and rehabilitation. Curr Atheroscler Rep 2013 Aug;15(8):343. [CrossRef] [Medline]
    14. Kepplinger J, Barlinn K, Deckert S, Scheibe M, Bodechtel U, Schmitt J. Safety and efficacy of thrombolysis in telestroke: a systematic review and meta-analysis. Neurology 2016 Sep 27;87(13):1344-1351. [CrossRef] [Medline]
    15. Bishop TF, Press MJ, Mendelsohn JL, Casalino LP. Electronic communication improves access, but barriers to its widespread adoption remain. Health Aff (Millwood) 2013 Aug;32(8):1361-1367 [FREE Full text] [CrossRef] [Medline]
    16. Hickson R, Talbert J, Thornbury WC, Perin NR, Goodin AJ. Online medical care: the current state of "eVisits" in acute primary care delivery. Telemed J E Health 2015 Feb;21(2):90-96. [CrossRef] [Medline]
    17. Lieu TA, Altschuler A, Weiner JZ, East JA, Moeller MF, Prausnitz S, et al. Primary care physicians' experiences with and strategies for managing electronic messages. JAMA Netw Open 2019 Dec 02;2(12):e1918287 [FREE Full text] [CrossRef] [Medline]
    18. Nazi KM. The personal health record paradox: health care professionals' perspectives and the information ecology of personal health record systems in organizational and clinical settings. J Med Internet Res 2013 Apr;15(4):e70 [FREE Full text] [CrossRef] [Medline]
    19. Adamson SC, Bachman JW. Pilot study of providing online care in a primary care setting. Mayo Clin Proc 2010 Aug;85(8):704-710 [FREE Full text] [CrossRef] [Medline]
    20. Tang PC, Black W, Young CY. Proposed criteria for reimbursing eVisits: content analysis of secure patient messages in a personal health record system. AMIA Annu Symp Proc 2006:764-768 [FREE Full text] [Medline]
    21. Yau G, Williams A, Brown J. Family physicians' perspectives on personal health records: qualitative study. Can Fam Physician 2011 May;57(5):e178-e184 [FREE Full text] [Medline]
    22. Ray KN, Shi Z, Gidengil CA, Poon SJ, Uscher-Pines L, Mehrotra A. Antibiotic prescribing during pediatric direct-to-consumer telemedicine visits. Pediatrics 2019 May;143(5):e20182491 [FREE Full text] [CrossRef] [Medline]
    23. Uscher-Pines L, Mulcahy A, Cowling D, Hunter G, Burns R, Mehrotra A. Access and quality of care in direct-to-consumer telemedicine. Telemed J E Health 2016 Apr;22(4):282-287. [CrossRef] [Medline]
    24. Mehrotra A, Paone S, Martich GD, Albert SM, Shevchik GJ. A comparison of care at e-visits and physician office visits for sinusitis and urinary tract infection. JAMA Intern Med 2013 Jan 14;173(1):72-74 [FREE Full text] [CrossRef] [Medline]
    25. Uscher-Pines L, Mulcahy A, Cowling D, Hunter G, Burns R, Mehrotra A. Antibiotic prescribing for acute respiratory infections in direct-to-consumer telemedicine visits. JAMA Intern Med 2015 Jul;175(7):1234-1235. [CrossRef] [Medline]
    26. Smith KL, Tran D, Westra BL. Sinusitis treatment guideline adherence in the e-visit setting: a performance improvement project. Appl Clin Inform 2016;7(2):299-307 [FREE Full text] [CrossRef] [Medline]
    27. Foster CB, Martinez KA, Sabella C, Weaver GP, Rothberg MB. Patient satisfaction and antibiotic prescribing for respiratory infections by telemedicine. Pediatrics 2019 Sep;144(3):e20190844. [CrossRef] [Medline]
    28. Martinez KA, Rood M, Jhangiani N, Kou L, Boissy A, Rothberg MB. Association between antibiotic prescribing for respiratory tract infections and patient satisfaction in direct-to-consumer telemedicine. JAMA Intern Med 2018 Nov 01;178(11):1558-1560 [FREE Full text] [CrossRef] [Medline]
    29. Sprecher E, Finkelstein JA. Telemedicine and antibiotic use: one click forward or two steps back? Pediatrics 2019 Sep;144(3):e20191585. [CrossRef] [Medline]
    30. Hertzog R, Johnson J, Smith J, McStay FW, da Graca B, Haneke T, et al. Diagnostic accuracy in primary care e-visits: evaluation of a large integrated health care delivery system's experience. Mayo Clin Proc 2019 Jun;94(6):976-984. [CrossRef] [Medline]
    31. Yokose C, Jorge A, D'Silva K, Serling-Boyd N, Matza M, Nasrallah M, et al. Using electronic visits (e-visits) to achieve goal serum urate levels in patients with gout in a rheumatology practice: a pilot study. Semin Arthritis Rheum 2020 Dec;50(6):1382-1386. [CrossRef] [Medline]
    32. Medicare telemedicine health care provider fact sheet. Centers for Medicare & Medicaid Services.   URL: https://www.cms.gov/newsroom/fact-sheets/medicare-telemedicine-health-care-provider-fact-sheet [accessed 2021-04-24]
    33. Use of communication technology-based services during coronavirus/COVID-19 Internet. American Speech-Language-Hearing Association. 2021 Jan 29.   URL: https:/​/www.​asha.org/​Practice/​reimbursement/​medicare/​Use-of-E-Visit-Codes-for-Medicare-Part-B-Services-During-Coronavirus/​ [accessed 2021-04-24]
    34. Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ 2009 Jul 21;339:b2535 [FREE Full text] [CrossRef] [Medline]
    35. Mehrotra A. The convenience revolution for treatment of low-acuity conditions. JAMA 2013 Jul 3;310(1):35-36 [FREE Full text] [CrossRef] [Medline]
    36. Kim SY, Park JE, Lee YJ, Seo H, Sheen S, Hahn S, et al. Testing a tool for assessing the risk of bias for nonrandomized studies showed moderate reliability and promising validity. J Clin Epidemiol 2013 Apr;66(4):408-414. [CrossRef] [Medline]
    37. Rohrer JE, Angstman KB, Adamson SC, Bernard ME, Bachman JW, Morgan ME. Impact of online primary care visits on standard costs: a pilot study. Popul Health Manag 2010 Apr;13(2):59-63. [CrossRef] [Medline]
    38. Watson AJ, Bergman H, Williams CM, Kvedar JC. A randomized trial to evaluate the efficacy of online follow-up visits in the management of acne. Arch Dermatol 2010 Apr;146(4):406-411. [CrossRef] [Medline]
    39. Albert SM, Shevchik GJ, Paone S, Martich GD. Internet-based medical visit and diagnosis for common medical problems: experience of first user cohort. Telemed J E Health 2011 May;17(4):304-308 [FREE Full text] [CrossRef] [Medline]
    40. Courneya PT, Palattao KJ, Gallagher JM. HealthPartners' online clinic for simple conditions delivers savings of $88 per episode and high patient approval. Health Aff (Millwood) 2013 Feb;32(2):385-392. [CrossRef] [Medline]
    41. North F, Crane SJ, Stroebel RJ, Cha SS, Edell ES, Tulledge-Scheitel SM. Patient-generated secure messages and evisits on a patient portal: are patients at risk? J Am Med Inform Assoc 2013 Nov 01;20(6):1143-1149 [FREE Full text] [CrossRef] [Medline]
    42. Heyworth L, Paquin AM, Clark J, Kamenker V, Stewart M, Martin T, et al. Engaging patients in medication reconciliation via a patient portal following hospital discharge. J Am Med Inform Assoc 2014 Feb 01;21(e1):e157-e162 [FREE Full text] [CrossRef] [Medline]
    43. Pathipati AS, Ko JM. Implementation and evaluation of Stanford Health Care direct-care teledermatology program. SAGE Open Med 2016;4:2050312116659089 [FREE Full text] [CrossRef] [Medline]
    44. Hawes E, Lambert E, Reid A, Tong G, Gwynne M. Implementation and evaluation of a pharmacist-led electronic visit program for diabetes and anticoagulation care in a patient-centered medical home. Am J Health Syst Pharm 2018 Jun 15;75(12):901-910. [CrossRef] [Medline]
    45. Levine DM, Dixon RF, Linder JA. Association of structured virtual visits for hypertension follow-up in primary care with blood pressure control and use of clinical services. J Gen Intern Med 2018 Nov;33(11):1862-1867 [FREE Full text] [CrossRef] [Medline]
    46. Penza KS, Murray MA, Myers JF, Maxson J, Furst JW, Pecina JL. Treating pediatric conjunctivitis without an exam: an evaluation of outcomes and antibiotic usage. J Telemed Telecare 2020;26(1-2):73-78. [CrossRef] [Medline]
    47. Penza KS, Murray MA, Pecina JL, Myers JF, Furst JW. Electronic visits for minor acute illnesses: analysis of patient demographics, prescription rates, and follow-up care within an asynchronous text-based online visit. Telemed J E Health 2018 Mar;24(3):210-215. [CrossRef] [Medline]
    48. Player M, O'Bryan E, Sederstrom E, Pinckney J, Diaz V. Electronic visits for common acute conditions: evaluation of a recently established program. Health Aff (Millwood) 2018 Dec;37(12):2024-2030. [CrossRef] [Medline]
    49. Rajda J, Seraly MP, Fernandes J, Niejadlik K, Wei H, Fox K, et al. Impact of direct to consumer store-and-forward teledermatology on access to care, satisfaction, utilization, and costs in a commercial health plan population. Telemed J E Health 2018 Feb;24(2):166-169. [CrossRef] [Medline]
    50. Murray MA, Penza KS, Myers JF, Furst JW, Pecina JL. Comparison of evisit management of urinary symptoms and urinary tract infections with standard care. Telemed J E Health 2020 May;26(5):639-644. [CrossRef] [Medline]
    51. Penza KS, Murray MA, Myers JF, Furst JW, Pecina JL. Management of acute sinusitis via e-visit. Telemed J E Health 2020 Jun 10:epub. [CrossRef] [Medline]
    52. de Jong CC, Ros WJ, Schrijvers G. The effects on health behavior and health outcomes of internet-based asynchronous communication between health providers and patients with a chronic condition: a systematic review. J Med Internet Res 2014 Jan 16;16(1):e19 [FREE Full text] [CrossRef] [Medline]
    53. Lanham HJ, Leykum LK, Pugh JA. Examining the complexity of patient-outpatient care team secure message communication: qualitative analysis. J Med Internet Res 2018 Jul 11;20(7):e218 [FREE Full text] [CrossRef] [Medline]
    54. Ozkaynak M, Johnson S, Shimada S, Petrakis BA, Tulu B, Archambeault C, et al. Examining the multi-level fit between work and technology in a secure messaging implementation. AMIA Annu Symp Proc 2014;2014:954-962 [FREE Full text] [Medline]
    55. Bellon JE, Stevans JM, Cohen SM, James AE, Reynolds B, Zhang Y. Comparing advanced practice providers and physicians as providers of e-visits. Telemed J E Health 2015 Dec;21(12):1019-1026 [FREE Full text] [CrossRef] [Medline]
    56. Tests and treatments for urinary tract infections in older people. Choosing Wisely.   URL: https://www.choosingwisely.org/patient-resources/urinary-tract-infections-in-older-people/ [accessed 2021-04-21]
    57. Treating sinus problems. Choosing Wisely.   URL: https://www.choosingwisely.org/patient-resources/treating-sinus-problems-aaaai/ [accessed 2021-04-24]
    58. Daniel SJ, Wu L, Kumar S. Teledentistry: a systematic review of clinical outcomes, utilization and costs. J Dent Hyg 2013 Dec;87(6):345-352. [Medline]
    59. Boggan JC, Shoup JP, Whited JD, Van Voorhees E, Gordon AM, Rushton S, et al. Effectiveness of acute care remote triage systems: a systematic review. J Gen Intern Med 2020 Jul;35(7):2136-2145. [CrossRef] [Medline]
    60. Hanlon P, Daines L, Campbell C, McKinstry B, Weller D, Pinnock H. Telehealth interventions to support self-management of long-term conditions: a systematic metareview of diabetes, heart failure, asthma, chronic obstructive pulmonary disease, and cancer. J Med Internet Res 2017 May 17;19(5):e172 [FREE Full text] [CrossRef] [Medline]
    61. Bashi N, Karunanithi M, Fatehi F, Ding H, Walters D. Remote monitoring of patients with heart failure: an overview of systematic reviews. J Med Internet Res 2017 Jan 20;19(1):e18 [FREE Full text] [CrossRef] [Medline]
    62. Kao DP, Lindenfeld J, Macaulay D, Birnbaum HG, Jarvis JL, Desai US, et al. Impact of a telehealth and care management program on all-cause mortality and healthcare utilization in patients with heart failure. Telemed J E Health 2016 Jan;22(1):2-11 [FREE Full text] [CrossRef] [Medline]
    63. Grady BJ. A comparative cost analysis of an integrated military telemental health-care service. Telemed J E Health 2002;8(3):293-300. [CrossRef] [Medline]
    64. Russo JE, McCool RR, Davies L. VA telemedicine: an analysis of cost and time savings. Telemed J E Health 2016 Mar;22(3):209-215. [CrossRef] [Medline]
    65. Le Goff-Pronost M, Mourgeon B, Blanchère J, Teot L, Benateau H, Dompmartin A. Real-world clinical evaluation and costs of telemedicine for chronic wound management. Int J Technol Assess Health Care 2018 Jan;34(6):567-575. [CrossRef] [Medline]
    66. Nguyen HV, Tan GSW, Tapp RJ, Mital S, Ting DSW, Wong HT, et al. Cost-effectiveness of a national telemedicine diabetic retinopathy screening program in Singapore. Ophthalmology 2016 Dec;123(12):2571-2580. [CrossRef] [Medline]
    67. Natafgi N, Shane DM, Ullrich F, MacKinney AC, Bell A, Ward MM. Using tele-emergency to avoid patient transfers in rural emergency departments: an assessment of costs and benefits. J Telemed Telecare 2018 Apr;24(3):193-201. [CrossRef] [Medline]
    68. Hayward K, Han SH, Simko A, James HE, Aldana PR. Socioeconomic patient benefits of a pediatric neurosurgery telemedicine clinic. J Neurosurg Pediatr 2019 Oct 25:1-5. [CrossRef] [Medline]
    69. Hale A, Haverhals LM, Manheim C, Levy C. Vet Connect: a quality improvement program to provide telehealth subspecialty care for veterans residing in VA-contracted community nursing homes. Geriatrics (Basel) 2018 Sep 05;3(3):57 [FREE Full text] [CrossRef] [Medline]
    70. Dirnberger J, Waisbren S. Efficacy of telehealth visits for postoperative care at the Minneapolis VA. Am J Surg 2020 Sep;220(3):721-724. [CrossRef] [Medline]
    71. Harris M. Bold consequences: the use and role of write-offs. American Association of Medical Assistants. Chicago, IL: AAMA; 2015 Jan.   URL: http://www.aama-ntl.org/docs/default-source/other/fine-print-bold-consequences_article.pdf [accessed 2021-04-24]
    72. Dendere R, Slade C, Burton-Jones A, Sullivan C, Staib A, Janda M. Patient portals facilitating engagement with inpatient electronic medical records: a systematic review. J Med Internet Res 2019 Apr 11;21(4):e12779 [FREE Full text] [CrossRef] [Medline]
    73. Mold F, Hendy J, Lai Y, de Lusignan S. Electronic consultation in primary care between providers and patients: systematic review. JMIR Med Inform 2019 Dec 03;7(4):e13042 [FREE Full text] [CrossRef] [Medline]
    74. Abd-Alrazaq AA, Bewick BM, Farragher T, Gardner P. Factors that affect the use of electronic personal health records among patients: a systematic review. Int J Med Inform 2019 Jun;126:164-175. [CrossRef] [Medline]
    75. Amante DJ, Hogan TP, Pagoto SL, English TM. A systematic review of electronic portal usage among patients with diabetes. Diabetes Technol Ther 2014 Nov;16(11):784-793. [CrossRef] [Medline]
    76. Grossman LV, Masterson Creber RM, Benda NC, Wright D, Vawdrey DK, Ancker JS. Interventions to increase patient portal use in vulnerable populations: a systematic review. J Am Med Inform Assoc 2019 Aug 01;26(8-9):855-870. [CrossRef] [Medline]
    77. Mobile fact sheet. Pew Research Center. 2021 Apr 07.   URL: https://www.pewresearch.org/internet/fact-sheet/mobile/ [accessed 2021-04-24]
    78. Adler-Milstein J, Longhurst C. Assessment of patient use of a new approach to access health record data among 12 US health systems. JAMA Netw Open 2019 Aug 02;2(8):e199544 [FREE Full text] [CrossRef] [Medline]
    79. Pho KK, Lu R, Gates S, Cai J, Xie D, Xie Y, et al. Mobile device applications for electronic patient portals in oncology. JCO Clin Cancer Inform 2019 Feb;3:1-8 [FREE Full text] [CrossRef] [Medline]
    80. Anderson PM, Hanna R. Defining moments: making time for virtual visits and catalyzing better cancer care. Health Commun 2020 May;35(6):787-791. [CrossRef] [Medline]
    81. Hernandez BF, Morgan BJ, Ish J, Agbator LO, Lindo-Moon S, Stotler FF, et al. Communication preferences and satisfaction of secure messaging among patients and providers in the military healthcare system. Mil Med 2018 Nov 01;183(11-12):e383-e390. [CrossRef] [Medline]
    82. Jung C, Padman R, Shevchik G, Paone S. Who are portal users vs. early e-Visit adopters? A preliminary analysis. AMIA Annu Symp Proc 2011;2011:1070-1079 [FREE Full text] [Medline]
    83. Lee KJ, Finnane A, Soyer HP. Recent trends in teledermatology and teledermoscopy. Dermatol Pract Concept 2018 Jul;8(3):214-223 [FREE Full text] [CrossRef] [Medline]
    84. Annis T, Pleasants S, Hultman G, Lindemann E, Thompson JA, Billecke S, et al. Rapid implementation of a COVID-19 remote patient monitoring program. J Am Med Inform Assoc 2020 Aug 01;27(8):1326-1330 [FREE Full text] [CrossRef] [Medline]
    85. Su MY, Das S. Expansion of asynchronous teledermatology during the COVID-19 pandemic. J Am Acad Dermatol 2020 Dec;83(6):e471-e472 [FREE Full text] [CrossRef] [Medline]
    86. Laurie H, Molly Z, Sanjana R, Megan H, Matthew H, Matthew A, et al. Expanded telehealth options during the COVID pandemic eliminated racial and age disparities in electronic health care use by IBD patients. Am J Gastroenterol 2020 Dec 01;115(Suppl 1):S20. [CrossRef] [Medline]

    ‎
    COVID-19: coronavirus disease 2019
    e-visit: electronic communication–based visit

    Edited by G Eysenbach; submitted 27.01.21; peer-reviewed by K McInnes, W Everett; comments to author 16.02.21; revised version received 02.03.21; accepted 11.04.21; published 05.05.21

    Copyright

    ©Oliver T Nguyen, Amir Alishahi Tabriz, Jinhai Huo, Karim Hanna, Christopher M Shea, Kea Turner. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 05.05.2021.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.