Characteristics of Patients Using Different Patient Portal Functions and the Impact on Primary Care Service Utilization and Appointment Adherence: Retrospective Observational Study

Introduction

The patient-centric care initiative heightened the awareness of health care systems’ responsibility to provide easily accessible ways for patients to engage in their own care and become effective health care partners. Such a mission is expected to be fulfilled with patient portals, where a portal is defined as “a secure online website that allows patients to access their medical records or communicate with their health care providers” [1]. Empowered by the rapid development of health information technologies, patient portals are now widely available and increasingly adopted by patients and providers. Effective use of these portals is expected to result in improved care access, self-management, and care coordination. Furthermore, the US federal government has authorized incentive payments to physicians who demonstrated “meaningful use” of such health information systems [2]. Consequently, patient portal research has garnered growing attention; a spate of reports of portal adoption and enrollee demographics have been published over the past decade. These studies typically described individual portal deployment or analyzed national survey data, such as the Health Information National Trends Survey (HINTS), and they reported characteristics of early portal adopters [3-7]. Along with this, whether the adoption of patient portals affects health care consumption was also investigated. Health care consumption (ie, the usage of various clinical services) is closely related to care access and coordination and is, thus, an important decision factor for service operations. Understanding the impact of portals on health care consumption can facilitate the design of service systems that accommodate patients’ portal usage, leading to enhanced efficiency of service operations and improved patient access to care. However, most reviews reported mixed evidence about the effect of patient portals on health care consumption—whether portal adoption will increase or decrease outpatient office visits was debated [8-12]—and the only consensus was that patient portals were used as a complement rather than a substitute of usual clinical services [13-16]. In addition, the number of appointment no-shows has been chosen to serve as an indicator to infer patient engagement [17], and it has been reported that portal enrollment is significantly related to decreases in appointment no-shows [18-20]. However, such studies mainly captured the association but not the causation between portal enrollment and no-show reduction. It motivated us to carry out a study that could account for measurable confounders and is robust to unmeasured confounders, hence, unveiling the causal effect of portal usage. In particular, we chose to investigate primary care office visit and telephone encounter rates (per quarter) as the measures of primary care service utilization as well as appointment cancellation and no-show rates (per quarter) as the measures of appointment adherence.

Despite the growing interest in portal adoption, there is a notable paucity of follow-up studies describing whether patients use portals actively and how distinct portal functions, such as messaging, laboratory, appointment, and medication, are used after adoption. The successful achievement of the promise to improve care access, self-management, and care coordination is intrinsically linked to the extent to which portals are used. We hypothesized that a patient who actively communicates with physicians using secure messages will benefit more from adopting a patient portal than one who never uses it after adoption. This is evidenced by the literature that states that messaging usage is associated with patient engagement [21,22]. In addition to messaging, the appointment function of portals offers an alternative way to make appointments than by phone calls and makes it easier for patients to reschedule or cancel their appointments. It can be hypothesized that with more freedom to manage appointments, patients will be more adherent to their appointments. Furthermore, with the laboratory and the medication functions of portals, patients can easily access their lab results and refill prescriptions online. Reminders can be sent from portals as an intervention to encourage patients to check their test results or refill their medications. Overall, we believed the convenience brought by patient portals to patients will enable them to be the owners of their health and be actively engaged in their care management. Lastly, we hypothesized that patients’ portal usage behaviors are heterogeneous, and different portal functions might be perceived with distinct values by users with various characteristics. To test these hypotheses, it is necessary to look at how patient characteristics are associated with portal usage and how different portal usage patterns affect patients’ care consumption and adherence to appointments.

The evaluation of how often patients access portals and what they do with them was not given enough attention in the past. Ignoring such a variety behaviors of patients might lead to the misspecification of portal effects on patients. For instance, if a subgroup of patients is doing significantly better with portals, whereas another group is doing worse, aggregating them can potentially lead to a conclusion of “no change.” Furthermore, portal adoption and the subsequent usage can influence patients over time, and the time trend in portal usage effects should not be overlooked; from an operational and a strategic point of view, both short-term and long-term impacts matter. Therefore, instead of solely relying on the observations from cross-sectional data, we sought to examine longitudinal data and focused on not only portal adoption but also on portal usage, aiming to investigate the following: (1) the characteristics of people who are more likely to adopt a patient portal, (2) among patients who have adopted portals, determine who uses portals more often and the characteristics of people associated with different portal function usage behaviors, and (3) whether the primary care service utilization and appointment adherence of patients who have adopted portals are affected by their different portal usage behaviors, featuring both the amount of use and the type of portal functions used. The answers to these questions are vital to informing (1) the design and implementation of patient portals, (2) the service system operations, such as the daily workflow design, and (3) the policy guidelines, such as remuneration models to compensate providers’ portal time.

Methods

Study Setting

Data Source

This study used the data generated by a large primary care patient cohort affiliated with the University of Florida (UF) Health; the data protocol was approved by the UF Institutional Review Board. In 2011, UF Health started offering an electronic patient portal named MyUFHealth, or MyChart, which allows patients to access to portions of their medical records (eg, released test results and after-visit summaries), communicate with the clinical service providers using secure messaging, request prescription refills, and manage outpatient appointments. Monthly clinical service utilization and portal activities of individual patients were generally not frequent; therefore, the time unit used in this study was one quarter: January-March (first quarter, Q1), April-June (second quarter, Q2), July-September (third quarter, Q3), and October-December (fourth quarter, Q4). For instance, Y13 Q3 stands for the third quarter of the year 2013. The study period was from July 1, 2013, to June 30, 2016. During the study period, there were 46,544 UF Health patients who had at least one visit to UF Health family medicine clinics. More than 95% of them came from North Central Florida.

Study Sample

Because the portal accounts of patients under 18 years old are typically managed by their legal guardians, we restricted our analysis to adult patients. We further restricted the study to insured patients who (1) chose UF Health as their primary health care provider, (2) enrolled in UF Health before the start of the study period, and (3) maintained an enrollment status until the end of the study period. As such, their primary care service utilization within the UF Health network can be fully captured. It is worth noting that UF Health is the leading care provider in the study region, and primary care services rendered to insured patients outside of the UF Health network are very limited. In addition, to ensure a contrast of before-after portal adoption and to capture the portal usage effects over time, we defined users in our study as patients who adopted MyUFHealth during periods Y13 Q4 to Y15 Q3. We excluded patients who (1) adopted the portal before the study period, (2) adopted the portal relatively recently (ie, adopted the portal in Y15 Q4 or after), or (3) were temporary users (ie, adopted the portal but closed their accounts before the study ended). These inclusion and exclusion criteria led to 17,580 nonusers and 4312 users.

Variables and Measures

Patients’ demographic and socioeconomic information, including age category, gender, race or ethnicity, marital status, insurance type, and their active problem number (APN), were obtained from their electronic medical records (EMRs). The APN is the number of problems in a patient’s active problem list, which captures patients’ chronic conditions and any ongoing impactful conditions that are resolvable but are important for physicians to be aware of to make clinical decisions. Notably, an ailment like a common cold or flu does not appear in the active problem list, and this list is typically reviewed at each patient encounter and updated—adding or deleting problems—whenever deemed necessary. Accordingly, a patient’s APN is considered as a time-varying confounder to account for individual disease burdens. It should also be noted that patients tend to use care services intensively right after an onset diagnosis of a new health condition and less frequently later, due to the resolution of the triggering health care condition [8]. A patient’s disease process (ie, an onset of a condition, followed by an episode of treatment, possibly including the resolution of the condition) can be nested within the process of portal adoption and subsequent usage. Therefore, we proposed a study design that controls for the time a new diagnosis was made (ie, when a visit type coded as new appeared in the EMR), allowing an assessment of the natural disease process.

To characterize portal usage patterns, we focused on four major portal functions that are regularly accessed by users: messaging (MESG), laboratory (LAB), medication (MED), and appointment (APPT). Patient portal usage is measured by the amount of use per quarter by function type. In particular, variable MESG_it is defined as the count of messaging-related activities by user i at quarter t, such as open a message box, read a message, delete a message, and send a message by patients. Variables LAB_it, MED_it, and APPT_it represent the count of actions related to laboratory activities (eg, check lab test results and request lab test); actions related to medication, such as check medication list and request drug or prescription refill; as well as actions related to appointments, such as appointment scheduling, appointment status checking, and cancel or reschedule an appointment, respectively.

To evaluate how portal usage affects primary care service utilization and appointment adherence, rates of office visits categorized as arrived, cancelled, or no-show, as well as telephone encounters per quarter were measured. Patients’ office visits and telephone encounters within the UF Health network were used as an indicator of their overall primary care service utilization.

User Subgroup Clustering

Patients’ portal activities differ across individuals and time: they might use a specific portal function more or less frequently based on their intrinsic preferences or immediate care needs, which might change with time. Therefore, we considered the portal usage over the course of a postadoption phase as the exposure and the use of primary care services as the outcome. We aimed to investigate the causality by examining the time dynamic behaviors in both exposures and outcomes. To characterize the time-varying exposures, we categorized patients into user subgroups and investigated the makeup of each subgroup, as well as the portal activity features associated with each subgroup.

Specifically, to cluster patients, we defined an activity feature vector (MESG_it,LAB_it,MED_it,APPT_it) (ie, the amount of function usage by user i at t quarters postadoption). This entails a pattern recognition problem with each user i being characterized by a set of ordered vectors , where T_i is the set of observation times postadoption for user i. A naive treatment is to take the average utilization over time and create a compact feature vector. However, this cannot separate the cases where a patient was moderate in messaging utilization in each quarter, versus a patient who did not use messaging except for one quarter of intensive use that brings the average utilization into the moderate level. Therefore, we proposed a two-stage clustering method to mitigate the flaw of averages and allow some assessment of the longitudinal usage patterns.

In the first stage, we characterized the relationships between four functions, for instance, whether there were two or more functions that were frequently used together at any time. The spherical clustering method [23] was employed to cluster activity feature vectors (MESG_it,LAB_it,MED_it,APPT_it). The difference in scales can be addressed by this method. For instance, the overall usage of messaging is one order of magnitude higher than that of medication. As a result, five activity clusters—C_MESG, C_LAB, C_MED, C_APPT, and C_M&L—were identified, which were named after their dominant activities. For instance, if patient i used messaging many times but not so much for the rest of the functions at time t, the activity feature vector will then be labeled with “C_MESG” at time t. For activity feature vectors with MESG and LAB functions used together and more often than others, a label of “C_M&L” was assigned. In addition, a sixth cluster named C_Silent was assigned for any activity feature vector being a vector of zeros. After the label assignment, a patient was then characterized by a |T_i|-dimensional pattern vector, (C_i1, C_i2,..., C_i|Ti| ), where . For example (C_MESG, C_Silent, C_MESG, C_Silent, C_Silent) is a labeled pattern vector for a patient with five observations postadoption (ie, T_i=5). Based on analyzing the data, the order of labels was quite random and, thus, was not featured into user types.

In the second stage, each user was assigned to one user type based on the number of occurrences of various activity clusters (ie, labels) over the postadoption period. For instance, with the above sample patient, cluster C_MESG has a frequency of 2/5, cluster C_Silent has a frequency of 3/5, and the frequency is 0 for the rest of the clusters. A user feature vector (2/5,0,0,0,0,3/5) representing the frequencies of belonging to clusters {C_MESG, C_LAB, C_MED, C_APPT, C_M&L, C_Silent} was created for the patient-level clustering. Because the user feature vector was already normalized, the K-means with Euclidian distance clustering method was used. As a result, five clusters were identified to represent five user types—U_MESG, U_LAB, U_APPT, U_M&L, and U_Silent—named after their dominant activity clusters. For instance, U_Silent represents the type of users who were, in general, inactive postadoption.

To summarize, we created activity feature vectors and collected activity feature vectors of all patients at every quarter after portal adoption to find common patterns, referred to here as activity clusters. This concluded the first-stage clustering. We then labeled each activity feature vector—per patient per time—with a membership (ie, belonging to one out of six activity clusters). This yielded a longitudinal activity pattern vector for each patient. Notably, such pattern vectors were of different dimensions, due to different observational time lengths postadoption. Thus, they were further mapped to a user feature vector with a fixed dimension of six for each patient. The user feature vectors of all patients were then clustered, which led to the final five user subgroups. This concluded the second-stage clustering. The illustration of the two-stage clustering and the descriptive statistics of the clustering results can be found in Multimedia Appendix 1.

Vector Matching Using Propensity Scores

Confounders are the determinants of exposure that are associated with outcomes (ie, variables that potentially affect both outcomes), for example, primary care service utilization, and exposure to different types of interventions (ie, becoming a specific type of user). To reduce the bias in causal inference due to confounders, we matched the users and nonusers using the vector matching method [24]. We first calculated propensity scores by estimating the probability of belonging to a user subgroup using multinomial logit regression (MLR). The covariates of the MLR model include the time-invariant characteristics (ie, age category, gender, race, marital status, and insurance type) and the time-varying variables (ie, APN and primary care office visits categorized as arrived, no-show, or cancellation, as well as telephone encounters) measured at their baseline. Notably, only the most recent marital status was recorded in the system. In addition, insurance types can change over time; however, the change was infrequent in our patient population, due to a relatively limited study time span. Therefore, a patient’s insurance type was treated as a time-invariant variable. The baseline values are observations averaged over the time period before adoption for users, and before Y14 Q4 for nonusers. The matching was based on the propensity score vectors obtained from MLR. A nonuser was matched to a user in one subgroup with a similar propensity score vector. To enhance the matching outcome, we further allowed nonusers to be exactly matched to users upon having multiple candidates available in the same propensity score stratum.

Generalized Linear Model for Heterogeneous Portal Usage Effects

In addition to matching patient demographics, the time-varying disease burden and the dynamic disease process needed to be addressed, which motivated a causal inference study accounting for both time-invariant and time-varying confounders. A panel difference-in-differences (DID) framework using generalized linear models was developed. The framework was similar to that in Zhong et al [25] but was generalized to capture heterogeneous portal usage effects. The detail of the model can be found in Multimedia Appendix 1.

Using such a framework, one can estimate the rate ratios (RRs) between the users and the matched nonusers for the targeted outcomes, including rates (per quarter) of office visits categorized as arrived, cancelled, or no-show, as well as telephone encounters. RRs for different user subgroups after portal adoption with an observational window of up to 10 quarters were obtained. An RR being significantly greater than 1 at a given quarter postadoption implies that the corresponding rate (eg, office visit rate) of the users was significantly larger than that of the nonusers at that quarter, which measures the time-dependent portal effects. In this study, all statistical analyses were performed using R, version 3.3.1 (The R Foundation), with two-sided statistical tests at a .05 significance level.

Results

Patient Portal Adoption

A logistic regression model was built to predict portal adoption; the odds ratios (ORs) obtained are exhibited in Table 1. The following were negatively associated with portal adoption: Hispanic and black or African American race versus white (OR 0.38 vs 0.53, 95% CI 0.19-0.69 vs 0.69-0.92, P=.003 vs P<.001); male gender (OR 0.64, 95% CI 0.59-0.68, P<.001); marital status as not married (ie, single, divorced, widowed, or not having a life partner or a significant other) in contrast to married (ORs 0.30-0.72, 95% CIs 0.22-0.83, all P<.001); and insurance type as not being Blue Cross Blue Shield (ORs 0.23-0.73, 95% CIs 0.12-0.80, all P<.001). Moreover, a high baseline APN (ORs 0.57 and 0.86, 95% CIs 0.51-0.63 and 0.80-0.94, all P<.001) and a high baseline no-show rate (OR 0.29, 95% CI 0.21-0.40, P<.001) were negatively associated with portal adoption. The following were positively associated with portal adoption: being above 30 years of age in contrast to being 19-30 years of age (ORs 1.20-1.28, 95% CIs 1.07-1.52, all P<.01) and having a high baseline telephone encounter rate (OR 1.13, 95% CI 1.06-1.19, P<.001).

Patient Portal Usage

Portal users’ usage summary statistics are presented here. The mean portal log-in rate was 6.85 per user per quarter (SD 12.11) with a median of 3 (IQR 8). The most frequently used portal function (per quarter) was messaging (mean 17.67, SD 34.69; median 5, IQR 20), followed by laboratory (mean 12.22, SD 28.04; median 1, IQR 13), appointment (mean 7.65, SD 19.89; median 1, IQR 7), and medication (mean 1.73, SD 4.05; median 0, IQR 2). The average number of secure messages sent from patients—we counted unique conversation threads, which can include multiple back-and-forth messages—was 1.07 per quarter. It was observed that 1214 users were very active and constantly accessed the portal postadoption. The remaining users did not use the portal in at least one quarter after adopting it.

Patient User Subgroups

After the two-stage clustering, we identified 615, 663, 1006, 536, and 1492 patients in user subgroups, U_LAB (14.3%), U_M&L (15.4%), U_MESG (23.3%), U_APPT (12.4%), and U_Silent (34.6%), respectively. The association between user types and patient characteristics was analyzed using MLR. Patients’ baseline care service utilization and marital status were not significantly associated with user types (P>.05). Married people or people with a life partner or a significant other, although being more likely to adopt portals compared to single people, were not less likely to be U_Silent. The ORs of the significantly relevant patient characteristics obtained from the MLR model are shown in Table 2. It can be seen that age is the most important predictor of user types. The ORs of using appointment functions strictly decrease with age (31-45, 46-64, and 65+ years: ORs 0.52, 0.34, and 0.19, respectively, 95% CIs 0.11-0.67, all P<.001). On the contrary, the intention of using messaging increases with age (31-45 years: OR 0.99, 95% CI 0.76-1.29, P>.05; 46-64 and 65+ years: ORs 1.38 and 1.50, 95% CIs 1.07-1.78 and 1.00-2.23, P=.01 and .04, respectively). Regarding gender, males used the laboratory function less often, such as being type U_M&L (OR 0.61, 95% CI 0.50-0.76, P<.001) and being type U_LAB (OR 0.75, 95% CI 0.61-0.93, P=.01) in contrast to being silent. In addition, male users were more inactive: no ORs were significantly larger than 1. On race and ethnicity, compared to white users, Asian users used the messaging less (OR 0.44, 95% CI 0.26-0.73, P=.002) but used the laboratory more (OR 1.58, 95% CI 1.04-2.39, P=.03). Black or African American users made significantly more appointments via the portal (OR 1.36, 95% CI 1.05-1.76, P=.02). Hispanic users were relatively silent: ORs were insignificant due to a small sample size.

On insurance type, it is interesting to note that although Medicaid and Medicare patients tended not to adopt a portal compared to Blue Cross Blue Shield patients, Medicaid patients shared a similar user type distribution as Blue Cross Blue Shield patients (for all ORs, P>.05). Moreover, Medicare patients used messaging significantly more (OR 1.44, 95% CI 1.03-2.03, P=.04) and were less inactive compared to other insurance types: no ORs were significantly smaller than 1. It should be noted that Medicare patients also include those who are less than 65 years of age but have received Social Security Disability Insurance checks for at least 24 months or have been diagnosed with end-stage renal disease [26]. In our patient population, we have around 28% of Medicare patients who were less than 65 years of age.

Lastly, a heavy disease burden in contrast to a small APN was significantly positively associated with frequent portal usage of any activity types (ORs 1.37-1.76, 95% CIs 1.11-2.22, all P≤.01), which is contrary to the observation that patients with a heavy disease burden tended not to adopt a portal. To demonstrate the quality of matching, the characteristics of the users and the nonusers before and after matching are shown in Table 3.

Primary Care Service Utilization and Appointment Adherence

Overview

Using the panel DID models (see Multimedia Appendix 1), we compared the utilization of primary care services between the matched nonusers and different user subgroups before and after portal adoption. The RRs of the users to the nonusers attributable to portal usage effects at each quarter postadoption are shown in Figure 1, and the corresponding RRs can be found in Table 4. The RRs measure the time-varying difference between the portal users and the matched nonusers at each quarter after portal adoption. To interpret, an RR being 1 suggests that there is no difference between the users and the matched nonusers, and thus there is no significant portal effect.

‎View this figure

Office Visits

For all the user subgroups, the office visit RRs were significantly larger than 1 within 6 months after portal adoption (RRs 1.13-2.1, 95% CIs 1.03-2.25, all P<.001) but were decreasing over time. The difference in office visit rates between the nonusers and most users—except for silent users—2 years postadoption was not significant. For the silent users, their office visit rates were not changed or were slightly lower by around 10% after 6 months postadoption. Patients who frequently used both messaging and laboratory functions had the largest RRs of office visits categorized as arrived, with an approximate 20% increase. Factoring in the APN of different user subgroups, for patients with fewer active health problems, their primary care service utilization was significantly lower after portal adoption. Meanwhile, with a heavy disease burden, the utilization was temporarily increased but was not significantly changed after 2 years postadoption.

Telephone Encounters

The change of telephone encounters was similar to that of office visits. Telephone encounters increased significantly at the time patients adopted portals (RRs 1.93-2.42, 95% CIs 1.62-2.72, all P<.001) and the RRs decreased over time. The silent users’ telephone encounters were significantly lower by 20% or more after 1 year postadoption (RRs 0.68-0.80, 95% CIs 0.50-0.93, all P<.01).

Appointment Cancellation

For all user subgroups, their cancellation rates were not significantly different to the nonusers and there was no trend in cancellation RRs over time.

Appointment No-Show

The no-show rates were significantly lower in most quarters postadoption: that of users were lower by 30% on average than nonusers and were not changed in the remaining quarters. In particular, patients using more messaging and messaging and laboratory combined had a larger reduction in no-show rates (average RR 0.61, minimum RR 0.35, P<.001). In summary, using patient portals is effective in reducing no-shows, but the relationship between portal usage and primary care service utilization is more complex than the simple substitution of online for in-person care.

Sensitivity Analysis

We analyzed the robustness of the results by changing the seeds used for conducting user subgroup clustering. This resulted in changes in user subgroup memberships: about 4.6% (95% CI 4.3-4.8) of patients had different memberships compared to the original clustering. The corresponding results were similar to the original model with respect to the overall RR trends for different outcome measures. The RR estimates and significance were slightly different but did not affect the major conclusions, which validates the robustness of this framework.

Discussion

Principal Findings

Patient Portal Adoption

In terms of the characteristics of portal adopters, users were more likely to be female, white, married, and enrollees of the commercial insurance Blue Cross Blue Shield. Adoption disparities in gender, race, and socioeconomic status were observed, which is consistent with previous studies on social disparities in enrollment and use of patient portals (see Perzynski et al [27], Graetz et al [28], and Kruse et al [29] and the references therein). Surprisingly, instead of alienating the older generation, young adults aged 19-30 years tended not to adopt patient portals. Members of the younger generation are the habitual users of Web-based applications [30]. It suggests that being accustomed to using Internet and other Web portals may not be a powerful predictor of portal adoption. This is a seemingly counterintuitive observation and might be interpreted as young adults not being strongly motivated to use patient portals because they are healthier and have a relatively low level of health care consumption in general [31]. Being tech-savvy is not the driving force for portal adoption and subsequent usage.

Patient Portal Usage

The characteristics of portal adopters were not necessarily the same as those of active portal users. Our findings suggest that the people who potentially enjoy using or need to use patient portals are not aware of, or given enough access to, patient portals. The most important factor driving portal usage intensity is patient disease burden, measured by APN. Patients with a heavy disease burden would use clinical services more frequently and patient portals can provide more convenience for them. Unfortunately, the propensity of portal adoption among the high APN population was not high, although they might be the population that benefits the most from using patient portals. In addition, although Medicare patients did not show a strong intention to adopt portals [27-29,32,33], once they became users, they exhibited a relatively high level of utilization and they were not resistant to using the messaging function of portals for communication. Medicare patients may actually appreciate the value of patient portals but just have barriers to adopt it, which signals a lack of match in the patient portal market. External forces, such as incentives, reward programs, and policy initiatives, are needed to channel patients.

Understanding the unique needs and usage habits of different patient populations can contribute to a better and user-friendly design of the portal that can cater its service and functionality to patients’ various tastes and preferences. For instance, an important factor for predicting user types is age. Comparing the younger and the older generations, we found that their attitudes toward using portals to make an appointment or sending a message differed significantly. Older patients did not favor the appointment function as young people did, possibly because most of their appointments are follow-ups and are made directly after their office visits. However, older patients preferred to send messages to their providers compared to other age groups; the relatively high utilization demonstrated the value they found in messaging. This is possibly because they demand frequent and timely communication with their providers, and messaging is a good complement to telephone encounters and office visits to fulfill their heavy needs.

Lastly, the digital divide between races or ethnicities exists not only in adoption, but also in the subsequent use of portals. Nonwhite patients, in general, tended not to adopt nor actively use a portal. In particular, black and African American patients tended not to adopt a portal, and Hispanic patients were very inactive after adopting portals. In particular, Asian patients exhibited a low level of utilization of the messaging function, implying a language barrier [28,34]. It was also found in other research that racial and ethnic minority groups, especially, reported concerns about privacy and information security and they differed from the advantaged (ie, high socioeconomic status) groups in their knowledge and skills of, and comfort in using, the technology, in addition to their accessibility to the technology infrastructure [6,35-37].

Language barriers, poor health literacy, and a low socioeconomic status, among other barriers, contribute substantially to the digital divide. Addressing these barriers will require patient education, infrastructure enhancement, as well as the technological designs that enable patients to communicate with providers in a secure and convenient way [6]. Providers, especially those serving vulnerable populations, should communicate with patients about portal usage and take time to discuss and demonstrate the technology, such as how to use different portal functions. Policy makers and technology developers should ensure the security, privacy, and ease of use of patient portals and the telehealth infrastructure, factoring in the special needs and the concerns of racial and ethnic minority groups. The heterogeneous adoption and usage behaviors of patients signal that the technology acceptance by people is not uniform and can be compounded by multiple factors, such as the conditions to facilitate the use, the ease of use, and the perceived usefulness [38]. Technology adoption theory would play an important role in guiding the design and development of portal functions that benefit patients with different characteristics and care needs.

Care Service Utilization and Appointment Adherence

First, the portal usage effects are heterogeneous rather than homogeneous. Different user groups behaved in different ways; ignoring such heterogeneity could lead to misspecification of such effects. Patients who frequently used messaging and laboratory together had the largest increase in primary care service utilization, including office visits and telephone encounters, while silent portal users had the largest reduction in using primary care services compared to before adoption. Mixing the two groups will lead to a possible conclusion that the primary care service utilization has not changed, as the positive and negative effects can get cancelled out.

Second, the portal usage effects are dynamic rather than static. Therefore, it is necessary to conduct studies using longitudinal data, not solely relying on observations from cross-sectional studies. The trends of office visit and telephone encounter RRs suggest that the convenience brought by patient portals for supporting better provider-patient interaction might reduce patients’ in-person visits over a longer time frame rather than immediately. This may be due to the fact that patients need time to adapt to portal functionalities, and patient portals influence patients’ health behavior gradually. Both the short-term (ie, a temporary boost) and the long-term (ie, a gradual decline) impacts are critical to informing service operations and guiding policy decisions. Whereas the portal usage was not shown to significantly reduce clinical service consumption immediately, portal activities, such as replying to secure messages, would inevitably increase the provider’s service time. Thus, exploring the payment structure that accommodates the technologically mediated interactions between providers and patients (eg, text messaging, emails, and virtual visits) is instrumental to gaining the buy-in of providers. Policy makers and payers must accordingly recognize and value the amount of time providers spend on interacting and educating patients, particularly the vulnerable and disadvantaged ones, both online and offline.

Lastly, the correct understanding of the heterogeneous and dynamic property of portal usage effects will enable us to carry out targeted and proactive interventions to achieve better patient outcomes. While the user subgroups behaved differently toward their health care consumption, the no-show rates of portal users were, overall, lower compared to that of nonusers, with different magnitudes of change. It also reveals that actively using patient portals, in contrast to being silent, leads to a larger improvement in appointment adherence. To achieve better patient engagement, providers can take the initiative in messaging patients, especially the ones with a high APN, to stimulate their portal usage and, thus, to raise their awareness of care engagement. In addition, elements of gamification can be imbedded into portal functions to encourage and reward patients. Virtual rewards or incentives can be made to patients who exhibit a high level of portal interactions (eg, actively reading the after-visit summary, physicians’ notes, and lab results, as well as participating in portal-based surveys, such as quality-of-care questionnaires and patient-reported outcomes).

Limitations

There are several limitations to our study. First, our causal inference analysis was based on an observational study. Admittedly, no unmeasured confounder is typically assumed to identify causal effects and is difficult to validate. However, even with unmeasured confounders, as long as they are time-invariant, their effects will be “cancelled” owing to the DID study design with the before-after comparison. If confounders are time-varying and measurable, we can treat them in the same fashion as dealing with a patient’s APN (ie, a time-varying confounder) and their disease process (ie, conditioning on their APN and whether a new disease occurred at each quarter). To test the robustness of our framework against unmeasured time-varying confounders, in future work, we plan to develop a trend-in-trends study [39]. In addition, we propose to design a synthetic control test [40] as a sensitivity analysis to evaluate the impact of any unmeasured confounders to our major results. Second, the dataset was limited to UF Health patients. We presented the demographics and socioeconomic characteristics of our patient population as a one-site study so that the results can be compared to other systems with the same or a different patient makeup. We also examined the institutional evaluation and management codes, which represent the severity level of patients: the larger the number, the more complicated the service [41]. It confirmed that our patient population shared a similar patient level in terms of severity to other academic health systems. Therefore, we hope that this patient population can be considered as being representative to generate a general insight. Third, the causal effects of portal usage on patient-reported outcomes and other adherence behaviors (eg, medication adherence) cannot be established using the data collected for this study. Also, we were not able to assess the impact of portal usage on patients’ specialty care consumption and their out-of-the-network urgent care and emergency department visits. We plan to expand the data spectrum to include some of these outcomes as part of the future work. Lastly, the business value and economic impact of patient portal implementation need to be quantified.

Conclusions

In closing, patients differ in their portal adoption and usage behaviors, and the portal effects are heterogeneous and dynamic. There exists a lack of match in the patient portal market in the sense that patients who benefit the most from using patient portals are not actively adopting patient portals. Patient portal usage was confirmed as effective in reducing appointment no-shows. However, to maximize the potential of patient portals, it is paramount to understanding the value that patient portals could bring to patients who have exhibited different characteristics and care needs. Health care delivery planners and administrators should, on the one hand, remove the barriers of adoption for the portal beneficiaries and, on the other hand, incorporate the impact of portal usage into care coordination and workflow design, ultimately aligning patients’ and providers’ needs and functionalities to enhance care delivery.