This paper is in the following e-collection/theme issue:

Clinical Information and Decision Making (1339) Email Communication, Web-Based Communication, Secure Messaging (148) Personal Health Records, Patient-Accessible Electronic Health Records, Patient Portals (543) Digital Health, Telehealth and e-Innovation in Clinical Settings (352) Electronic Health Records (971) Decision Support for Health Professionals (1145) Secondary Use of Clinical Data for Research and Surveillance (325)

Published on in Vol 25 (2023)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/41884, first published .
Provider Interaction With an Electronic Health Record Notification to Identify Eligible Patients for a Cluster Randomized Trial of Advance Care Planning in Primary Care: Secondary Analysis

Provider Interaction With an Electronic Health Record Notification to Identify Eligible Patients for a Cluster Randomized Trial of Advance Care Planning in Primary Care: Secondary Analysis

Provider Interaction With an Electronic Health Record Notification to Identify Eligible Patients for a Cluster Randomized Trial of Advance Care Planning in Primary Care: Secondary Analysis

Authors of this article:

Jessica E Ma1, 2 Author Orcid Image ;   Jared Lowe3 Author Orcid Image ;   Callie Berkowitz4 Author Orcid Image ;   Azalea Kim2 Author Orcid Image ;   Ira Togo5 Author Orcid Image ;   R Clayton Musser2 Author Orcid Image ;   Jonathan Fischer6, 7 Author Orcid Image ;   Kevin Shah8 Author Orcid Image ;   Salam Ibrahim9 Author Orcid Image ;   Hayden B Bosworth2, 6, 8, 10, 11, 12 Author Orcid Image ;   Annette M Totten13 Author Orcid Image ;   Rowena Dolor3 Author Orcid Image
Article Authors Cited by Tweetations (7) Metrics

    Original Paper

    1Geriatric Research, Education, and Clinical Center, Durham Veterans Affairs Health Care System, Durham, NC, United States

    2Division of General Internal Medicine, Department of Medicine, Duke University School of Medicine, Durham, NC, United States

    3Division of General Medicine & Clinical Epidemiology, Department of Medicine, University of North Carolina School of Medicine, Chapel Hill, NC, United States

    4Division of Hematology and Oncology, Department of Medicine, University of North Carolina School of Medicine, Chapel Hill, NC, United States

    5Duke Office of Clinical Research, Durham, NC, United States

    6Department of Community & Family Medicine, Duke University School of Medicine, Durham, NC, United States

    7Duke Population Health Management Office, Durham, NC, United States

    8Department of Medicine, Duke University School of Medicine, Durham, NC, United States

    9Duke Health Performance Services, Duke University Health System, Durham, NC, United States

    10Department of Psychiatry and Behavioral Services, Duke University School of Medicine, Durham, NC, United States

    11Department of Population Health Sciences, Duke University School of Medicine, Durham, NC, United States

    12Center of Innovation to Accelerate Discovery and Practice Transformation, Durham Veterans Affairs Health Care System, Durham, NC, United States

    13Oregon Rural Practice Based Research Network, Oregon Health & Science University School of Medicine, Portland, OR, United States

    Corresponding Author:

    Jessica E Ma, MD

    Geriatric Research, Education, and Clinical Center

    Durham Veterans Affairs Health Care System

    508 Fulton Street

    Durham, NC, 27705

    United States

    Phone: 1 9192860411

    Email: jessica.ma@duke.edu


    Background: Advance care planning (ACP) improves patient-provider communication and aligns care to patient values, preferences, and goals. Within a multisite Meta-network Learning and Research Center ACP study, one health system deployed an electronic health record (EHR) notification and algorithm to alert providers about patients potentially appropriate for ACP and the clinical study.

    Objective: The aim of the study is to describe the implementation and usage of an EHR notification for referring patients to an ACP study, evaluate the association of notifications with study referrals and engagement in ACP, and assess provider interactions with and perspectives on the notifications.

    Methods: A secondary analysis assessed provider usage and their response to the notification (eg, acknowledge, dismiss, or engage patient in ACP conversation and refer patient to the clinical study). We evaluated all patients identified by the EHR algorithm during the Meta-network Learning and Research Center ACP study. Descriptive statistics compared patients referred to the study to those who were not referred to the study. Health care utilization, hospice referrals, and mortality as well as documentation and billing for ACP and related legal documents are reported. We evaluated associations between notifications with provider actions (ie, referral to study, ACP not documentation, and ACP billing). Provider free-text comments in the notifications were summarized qualitatively. Providers were surveyed on their satisfaction with the notification.

    Results: Among the 2877 patients identified by the EHR algorithm over 20 months, 17,047 unique notifications were presented to 45 providers in 6 clinics, who then referred 290 (10%) patients. Providers had a median of 269 (IQR 65-552) total notifications, and patients had a median of 4 (IQR 2-8). Patients with more (over 5) notifications were less likely to be referred to the study than those with fewer notifications (57/1092, 5.2% vs 233/1785, 13.1%; P<.001). The most common free-text comment on the notification was lack of time. Providers who referred patients to the study were more likely to document ACP and submit ACP billing codes (P<.001). In the survey, 11 providers would recommend the notification (n=7, 64%); however, the notification impacted clinical workflow (n=9, 82%) and was difficult to navigate (n=6, 55%).

    Conclusions: An EHR notification can be implemented to remind providers to both perform ACP conversations and refer patients to a clinical study. There were diminishing returns after the fifth EHR notification where additional notifications did not lead to more trial referrals, ACP documentation, or ACP billing. Creation and optimization of EHR notifications for study referrals and ACP should consider the provider user, their workflow, and alert fatigue to improve implementation and adoption.

    Trial Registration: ClinicalTrials.gov NCT03577002; https://clinicaltrials.gov/ct2/show/NCT03577002

    J Med Internet Res 2023;25:e41884

    doi:10.2196/41884

    Keywords

    advance care planning; electronic health record; notification; EHR; provider interaction; primary care; clinical study; referral; notifications; alerts 


    Health care providers and health systems are increasingly recognizing the importance of advance care planning (ACP) [1]. ACP is the process of understanding a patient’s preferences around future medical care and how health care can best align with their personal values and goals [2]. Despite benefits of ACP, including improved quality of life, positive family outcomes, and goal-consistent care, the overall documentation of ACP remains low [3-5].

    There are significant barriers to engaging patients in ACP across all practice settings, such as time limitations, provider skill set, and patient selection [6-9]. In primary care, one challenge is identifying the patients most likely to benefit from ACP, so that limited resources and time may be prioritized. Related to this challenge is determining how to best notify providers which patients have been identified for ACP interventions. One approach is using a notification via the electronic health record (EHR) to prompt providers to consider having an ACP conversation [10,11].

    The approach of “prescreening” patients using the EHR and alerting providers to patient study eligibility is frequently used in clinical trials and offers an efficient and effective alternative to manually screening patient records [12-14]. We used a single EHR notification tool as both a clinical trial alert for participant recruitment and a prompt for providers to engage in ACP]. Although EHR notifications can add value to clinical workflows, EHR and associated notifications can contribute to provider fatigue and burnout [15,16]. Therefore, further research is needed to evaluate how providers respond to EHR notifications, whether notifications lead to desired outcomes, specifically ACP conversation and trial recruitment in this study, and provider likelihood of continuing to use these notifications. The aim of this study is to evaluate an embedded EHR notification to identify patients appropriate for ACP in primary care clinics at 1 health system participating in the Meta-network Learning and Research Center (Meta-LARC) ACP study; we report provider usage and perspectives on this notification tool (hereafter referred to as the Meta-LARC ACP Notification substudy).


    Study Design

    The Meta-LARC ACP study is a Patient-Centered Outcomes Research Institute (PCORI)–funded US-Canadian multisite cluster randomized trial comparing clinician-focused to team-based implementation of the Serious Illness Care Program (SICP) developed by Ariadne Labs. The clinician-based arm focused on the primary care clinician leading to ACP conversations, and the team-based arm used the primary care team (including nurses, medical assistants, and clinicians) to have ACP conversations [17]. Each site determined which specific methods to implement to identify patients appropriate for ACP.

    Within Duke Health, an EHR notification was developed to identify patients who would be appropriate for ACP, remind clinicians to have ACP, as well as to refer patients to the Meta-LARC ACP study. Duke Health is an academic health system serving urban, suburban, and rural populations in the Piedmont region of North Carolina. In the Duke Primary Care clinics, patients are assigned to a usual provider. If the provider is unavailable (eg, urgent same-day visit), then a patient may see any available provider.

    Six Duke Primary Care clinics participated—3 clinics in the clinician-based arm and 3 clinics in the team-based arm—from January 1, 2019, to January 31, 2021. Study recruitment was initially set to end in June 2020 but was extended through January 2021 due to the COVID-19 pandemic. A total of 45 clinicians were trained with Serious Illness Care Program, participated in the study, and received EHR notifications. This included 32 physicians (Doctor of Medicine or Doctor of Osteopathic Medicine), 11 advanced practice practitioners, and 2 population health nurses.

    The objectives of the Meta-LARC ACP Notification substudy are to (1) describe the implementation and usage of the EHR ACP notifications, (2) assess provider interactions with the EHR notifications, (3) evaluate the association of notifications with study referrals and with occurrence and documentation of ACP conversations, and (4) elicit provider perspectives on the EHR notification.

    Development of ACP Notification Tool in the EHR

    We used an ACP scoring system developed at Duke Health to identify patients who may benefit from an ACP conversation. The ACP score adds 1 point for each of the following: (1) age at or older than 70 years, (2) three or more prespecified comorbidities using Systematized Nomenclature of Medicine (SNOMED) clinical terms, and (3) two or more hospitalizations within the health system in the past year [18,19]. Patients with an ACP score of 2 or 3 were considered more likely to benefit from an ACP conversation and were flagged for this study. This ACP score was embedded in the EHR, and a notification (a Best Practice Advisory within the Epic EHR system; Figure 1) appeared when the chart was opened during the encounter with identified patients. As prior diagnoses and hospitalizations were needed to calculate the ACP score, notifications would alert established patients meeting the ACP score criteria. Also, the EHR algorithm did not exclude patients with prior ACP documentation.

    ‎
    Figure 1. Advance care planning (ACP) notification. Meta-LARC: Meta-network Learning and Research Center.

    From May 14, 2019, to January 14, 2021, the notification alerted enrolled providers when the patient presented for their clinic visit and the encounter visit was opened in the EHR. Once the notification appeared, providers were able to perform one of three actions: (1) dismiss the notification temporarily and have it retrigger when the chart was opened again in the same encounter, (2) acknowledge and remove the notification for the encounter, or (3) refer the patient to the study and commit to having an ACP conversation with the patient (Figure 2). If providers acknowledge the notification, providers also have an option of documenting a reason for the action. Provider responses to the notification (dismiss, acknowledge, or referral to study) were measured. While EHR notifications were the primary reminder for providers to have ACP conversations and occurred in the moment during the clinic visit, providers also received weekly emails with a list of their patients who had an ACP score of 2 or 3 and monthly practice facilitation visits during the enrollment period.

    ‎
    Figure 2. Process map with advance care planning (ACP) notification process and provider actions. *ACP score identified patients who may benefit from an ACP conversation. There are a total of 3 points, with 1 point for each of the following: age at or older than 70 years, three or more prespecified comorbidities using Systematized Nomenclature of Medicine clinical terms, and two or more hospitalizations within the health system in the past year. Meta-LARC: Meta-network Learning and Research Center.

    Notifications that occurred outside of the 6 primary care clinics were excluded from the analysis (n=511). These excluded notifications occurred among providers (N=8) who had transitioned out of the primary care clinic and resident trainee providers (N=9) who also practiced outside of the primary care clinic (eg, inpatient medicine service).

    Data Collection and Outcomes

    EHR data were extracted over the study period; additional 2 weeks of data after the conclusion of the notification were included to provide time for providers to act on the notification. These data included provider action to notification (dismiss, acknowledge, and refer patient to study) and any free-text notification comments; patient demographics (age at start of study period, sex, race, and ethnicity); health care utilization (primary care visits and hospitalizations during notification period); primary insurance payor, ambulatory referrals to hospice, and death; and ACP documentation. ACP documentation included ACP notes; legal documents uploaded to the EHR, such as advance directives, health care power of attorney forms, or Physician Orders for Life-Sustaining Treatment (POLST) forms; or the use of ACP billing codes (Current Procedural Terminology [CPT] 99497 and 99498). No specific template was recommended for ACP note documentation; health system templates were available and Epic SmartPhrase codes were used to track documentation in the medical record [20]. POLST is a portable medical order form addressing specific health care treatments in more detail compared to advance directives and Do-Not-Resuscitate orders. ACP notification tool metrics regarding provider usage and response were also extracted from the EHR. For insurance payors, we grouped Medicare and Medicare Advantage into 1 Medicare group. Medicaid pending, in-state Medicaid, and out-of-state Medicaid were in 1 group. All other payors were grouped as commercial and other. Primary care visits were counted as any visit to a primary care clinic.

    Our primary outcome was referral to the study through the EHR notification. Our secondary outcomes were documentation of ACP in the EHR, as indicated by the presence of ACP notes, and usage of related billing codes.

    Notification Free-Text Comments

    We also evaluated an additional free-text response outcome for providers not referring patients. When providers chose to acknowledge responses, providers had an opportunity to leave free-text comments in the notification. Free-text comments were not required to complete the notification. An inductive approach was used to summarize major themes in free-text comments, which included lack of time, not claiming responsibility for the conversation, high visit acuity, and inappropriate timing.

    Provider Survey

    In August 2020 and during the COVID-19 pandemic, approximately 1 year after the implementation of the ACP notification, we distributed a self-administered, closed and open-ended questionnaire soliciting provider perspectives regarding the ACP notification. We asked participants to rate their satisfaction (strongly disagree, disagree, agree, or strongly agree) using the ACP notification, its impact on the delivery of care, and its accuracy in identifying patients for ACP, as well as whether the provider would recommend the implementation of this notification for others. Meta-LARC providers were invited by email to a web-based survey using Qualtrics. A reminder email was sent 2 weeks later. The survey was open for 6 weeks.

    Statistical Analysis

    Descriptive statistics were used for patient and provider characteristics, provider reaction to the ACP notification, and survey results. Results were presented in the following groups: (1) patients referred to the Meta-LARC ACP study, (2) patients who were not referred to the Meta-LARC ACP study, and (3) all patients identified by the ACP notification. Two patients were referred more than once, and each was counted as 1 referral per patient. Percentages were calculated for dichotomous and categorical data and mean and SD or median and IQR for continuous variables. We describe the frequency of occurrence of documentation related to ACP, advance directives, health care power of attorney, or POLST forms. Health care utilization (primary care visits and hospitalizations) were also described.

    A logistic regression was performed using deciles to identify when a significant change in number of notification and referral to study occurred. As the deciles had overlapping notification values, we categorized the notifications into 9 categories (1, 2, 3, 4, 5, 6, 7 to 8, 9 to 13, or greater than 13 notifications). It was found that at 6 notifications there was a significant change in referral response (Multimedia Appendix 1). Therefore, we dichotomized notifications as less than or equal to 5 and greater than 5.

    The chi-square test was used to compare associations between notifications (5 or less notifications and greater than 5 notifications) and (1) referral to the study, (2) ACP note documentation, and (3) ACP billing. We also examined associations between referral to study and (1) ACP documentation and (2) ACP billing. Associations where the primary care provider had an active role in (eg, referral response in the notification, note documentation, and billing) were only evaluated. Therefore, we did not compare associations with advance directives, health care power of attorney, or POLST forms, as these can be placed or submitted by patients and providers outside of the primary care clinic. Lastly, baseline patient characteristics and association with referral to study were evaluated using simple logistic regression models and complete data. Reference groups for the analysis were age (<65 years old), race (White), and sex (female). We excluded ethnicity, as the majority of patients were non-Hispanic, and insurance status, as the majority of patients were Medicare. Odds ratio (OR), CI around OR, and P values were reported. Analyses were performed using R (version 4.0.3; R Foundation for Statistical Computing) [21].

    Ethics Approval

    The Meta-LARC ACP study was reviewed and approved by the Trial Innovation Network Central Institutional Review Board (IRB) at Vanderbilt University (VUMC IRB#181084) for the sites in the United States. As this study analyzes data without additional primary data collection, this secondary analysis was approved by the Duke University IRB (Pro00106329) with a waiver of consent. Data include protected health information data, which are stored in secure university-approved systems. There was no compensation for this additional analysis.


    Overview

    Over the course of study period, 2877 unique patients were identified by the ACP score and triggered an ACP notification, and 45 providers received a total of 17,047 notifications. Patients identified by the ACP score were on average 78.0 (SD 10.5) years old, and the majority were female (n=1678, 58.3%), White (n=1868, 64.9%), and non-Hispanic (n=2822, 98.1%), with Medicare as the primary insurer (n=2629, 91.4%). A total of 290 unique patients (10.1%) were referred to the study by the notification. As of May 25, 2022, a total of 733 (25.5%) of patients identified have died (Table 1).

    Table 1. Characteristics of patients eligible for advance care planning (ACP) and those referred to the study.

    		No referral to study (N=2587, 89.9%)Referral to study (N=290, 10.1%)Overall (N=2877)
    Patient characteristics

    		Age, mean (SD)76.9 (10.7)78.1 (9.2)77.1 (10.6)

    		Gender, n (%)




    		Female1511 (58.4)167 (57.6)1678 (58.3)


    		Male1076 (41.6)123 (42.4)1199 (41.7)

    		Race, n (%)




    		White1694 (65.5)174 (60.0)1868 (64.9)


    		Black813 (31.4)109 (37.6)922 (32.0)


    		Mixed race or other66 (2.6)6 (2.1)72 (2.5)


    		Missing14 (0.5)1 (0.3)15 (0.5)

    		Ethnicity, n (%)




    		Non-Hispanic2537 (98.1)285 (98.3)2822 (98.1)


    		Hispanic11 (0.4)0 (0)11 (0.4)


    		Missing39 (1.5)5 (1.7)44 (1.5)

    		Primary insurance payor, n (%)




    		Medicare2364 (91.4)265 (91.4)2629 (91.4)


    		Medicaid31 (1.2)6 (2.1)37 (1.3)


    		Other167 (6.5)17 (5.9)184 (6.4)


    		Missing25 (1.0)2 (0.7)27 (0.9)
    Notifications

    		Notifications per patient, median (IQR)4 (2-8)2 (1-5)4 (2-8)

    		Provider response to notifications per patient, median (IQR)


    		Acknowledge2 (1-4)0 (0-2)1 (1-4)


    		Dismiss1 (0-5)0.5 (0-2)1 (0-4)
    Health careutilization

    		Primary care provider visits, median (IQR)5 (3-8)7 (4-9)5 (3-8)


    		Missing, n (%)28 (1.1)1 (0.3)29 (1.0%)

    		Hospitalizations (≥2 during study period), n (%)604 (23.4%)60 (20.7)665 (23.1)

    		Hospice referral, n (%)223 (8.6%)39 (13.4)262 (9.1)

    		Death, n (%)643 (24.9%)90 (31.0)733 (25.5)
    ACP documentation

    		ACP note in EHRa, n (%)188 (7.3)125 (43.1)313 (10.9)

    		ACP billing, n (%)


    		994974 (0.2)23 (7.9)27 (0.9)


    		994981 (0.0)1 (0.3)2 (0.1)

    		Health care power of attorney, n (%)123 (4.8)20 (6.9)143 (5.0)

    		Advance directives, n (%)56 (2.2)18 (6.2)74 (2.6)

    		POLSTb, n (%)34 (1.3)5 (1.7)39 (1.4)

    aEHR: electronic health record.

    bPOLST: Physician Orders for Life-Sustaining Treatment.

    Provider Actions and Responses

    Of the 17,047 notifications, the most common response to the notification was dismiss (n=9496, 55.7%), followed by acknowledge (n=7251, 42.5%), and referral to study (n=292, 1.7%). Of note, referral to study should only occur after the provider initiated an ACP conversation, therefore, providers can dismiss or acknowledge the notification until the ACP conversation occurs at a future visit. There was a median of 4 (IQR 2-8) notifications per patient. The maximum number of notifications for any individual patient was 78.

    In total, providers received a median of 269 (IQR 65-552) notifications over the study period. Of those, providers dismissed a median of 62 (IQR 16-179) times, acknowledged a median of 40 (IQR 11-191) times, and referred to the study a median of 2 (IQR 0-10) times. Providers had a median of 61 (IQR 27-135) patients identified by the study algorithm.

    Free-text comments on acknowledge responses were provided for 658 notifications with comments grouped into categories shown in Table 2. The 28 providers (28/45, 62%) who provided comments wrote a median of 3 (IQR 2-20) comments, where 1 provider wrote 332 comments. The most common reason provided was having insufficient time during the clinic visit, accounting for 337 of the 658 comments (51.2%). The other response categories each individually accounted for 8% or less of total responses, with the second most common reason being the provider did not think they were responsible for discussing ACP (53 of 658 comments, 8.1%). A small number of comments—39 of 658 (5.9%)—were associated with ACP being inappropriate for the identified patient.

    Table 2. Provider comments for acknowledge responses (N=658).
    CategoriesValues, n (%)a
    Lack of time337 (51.2)
    Provider was not responsible for conversation (eg, not the primary care provider or followed by oncology)53 (8.1)
    Acute visit (and commonly not primary care provider for the patient)50 (7.6)
    Inappropriate timing (eg, new patient, posthospital, or want to plan for a future visit)49 (7.4)
    Advance care planning previously discussedb40 (6.1)
    Inappropriate for advance care planning39 (5.9)
    Patient was not seen for a routine medical visit (eg, lab visit or telephone encounter)30 (4.6)
    Surrogate decision maker not present (eg, dementia)26 (4.0)
    Patient not interested25 (3.8)
    Other (eg, technical issues)9 (1.4)

    aPercentage calculated from total written comments (N=658).

    bAdvance care planning reported on file for 8 patients.

    A total of 290 patients (290/2877, 10.1%) were referred to the study by the notification. Among the 30 (30/45, 66%) providers who referred patients, these providers referred a median of 4.5 (IQR 2-15.75) patients each. The maximum number of patients referred by 1 provider was 45. Patients with greater than 5 notifications were less likely to be referred to the study (57/1092, 5.2% vs 233/1785, 13.1% of patients with 5 or less notifications; P<.001). Among patients with a notification and complete data (N=2802), Black patients had increased odds of referral to study (OR 1.34, 95% CI 1.04-1.73; P=.02). There was no significant difference in age or sex with referral to the study (Multimedia Appendix 2).

    ACP Documentation and Billing

    Providers wrote 471 ACP notes for a total of 313 patients. A total of 34 providers (34/45, 75%) wrote an ACP note in the EHR with a median of 7.5 (IQR 3-15.5) notes per provider. The maximum number of notes written by 1 provider was 68 during the study period. Thirty-two patients had more than 1 ACP note. Patients were more likely to have an ACP note documented by a study provider if they were referred to the study (125/290, 43.0% referred vs 188/2587, 7.3% not referred; P<.001). There was no significant difference between number of notifications and ACP documentation (189/1785, 10.6% with 5 or less notifications vs 124/1092, 11.3% with greater than 5 notifications; P=.52).

    Seven providers (7/45, 15%) used ACP billing codes (99497 or 99498) for a total of 24 times during the study period. Only 1 provider used the 99498 billing code during the study. Patients had increased likelihood of having ACP billing by a study provider if the patient was referred to the study (23/290, 7.9% referred vs 4/2587, 0.2% not referred; P<.001). There was no significant difference among patients who had greater than 5 notifications and ACP billing (19/1785, 1.1% with 5 or less notifications vs 8/1092, 0.7% with greater than 5 notifications, P=.43).

    Provider Survey Response

    Eleven providers (11/45, 24%) responded to the survey. Results are presented as agree (agree or strongly agree) or disagree (disagree or strongly disagree). The majority of providers agreed that the notification improved delivery of care (7/11, 64%) was valuable to clinical care (7/11, 64%) and also accurately identified patients for ACP (8/11, 73%). While providers would recommend the notification for other clinics (7/11, 64%), fewer providers would continue to use the notification after the conclusion of the study (5/10, 50%). Despite the positive feedback, the notification was burdensome to clinical workflow (9/11, 82%) and difficult to navigate (6/11, 55%; Table 3).

    Table 3. Survey results of 11 providers in the Meta-LARa ACPb substudy.

    		Strongly disagreeDisagreeAgreeStrongly agree
    Improved delivery of care, n (%)1 (9)3 (27)6 (55)1 (9)
    Valuable to clinical care, n (%)1 (9)3 (27)6 (55)1 (9)
    Accuracy for identifying patients, n (%)1 (9)2(18)3 (27)5 (45)
    Burdensome to clinical workflow, n (%)0 (0)2 (18)7 (64)2 (18)
    Difficult to navigate, n (%)2 (18)3 (27)5 (54)1 (9)
    Continue use after study concludes, n (%)2 (20)3 (30)5 (50)0 (0)
    Recommend for other clinics, n (%)1 (9)3 (27)7 (64)0 (0)

    aMeta-LARC: Meta-network Learning and Research Center.

    bACP: advance care planning.

    Participant Enrollment in Meta-LARC Study

    The notification was implemented 5 months after the start of the overall Meta-LARC study; patients may have been referred outside of the notification prior to the implementation as well as during the study. Of the 113 patients who consented to the Meta-LARC study from this site, 100 patients (88.5%) were referred through the EHR notification. Six patients, while identified by the ACP score, were referred outside of the EHR notification. Seven patients were not identified by the ACP score and were referred to the study team outside of the EHR notification.


    Principal Findings

    This Meta-LARC ACP Notification substudy implemented an embedded EHR notification to identify patients appropriate for ACP, referral to a clinical trial, and encourage ACP conversations and documentation. Approximately 10% of the patients identified by the EHR notification were referred to the study. In addition, we found that by the time providers received more than 5 notifications, providers were less likely to refer patients to the study. There was no significant difference in association among patients with more notifications (greater than 5 notifications) and ACP documentation or billing. While providers found the notification added value to their clinical care, the notification was difficult to use and was burdensome to clinical workflow.

    Comparison With Prior Work

    Clinical trials can depend on provider participation for trial recruitment [22], and within the Meta-LARC study, providers would evaluate and refer patients to the trial. In this substudy, we successfully implemented a health system–specific ACP score to identify patients and an EHR notification to alert providers to potential eligibility and facilitate referral to the trial. The majority of patients were referred after notification implementation, which occurred in the fifth month of the year-long trial. This is similar to work done by Embi et al [13] demonstrating increased study referral after implementing automated notifications.

    In addition to being a tool for increasing clinical trial referrals, the EHR notification encouraged providers to engage patients in ACP. The use of an embedded provider EHR notification has been demonstrated in other trials related to ACP in primary care [11,23]. Conversations on ACP are difficult to measure as providers may not document all conversations. Similar to other studies, we use ACP documentation as a surrogate to measure ACP conversations [24,25]. In this substudy, ACP documentation did not change with the number of notifications; in comparison, ACP was more likely documented among patients whom providers referred to the study.

    The number of notifications did not influence the likelihood of having an ACP note documented because of other barriers to conducting ACP in a clinic setting. A study by Tung et al [23] used a clinical decision support system to identify patients for ACP and notify providers. In that study, all providers and patients received a recommendation for ACP. Patients at 2 clinics who received a resource packet prior to the visit were 5 times more likely to complete an advance directive compared to patients who did not receive a packet (21% vs 4%) [23]. While an embedded ACP EHR notification can overcome barriers in clinical practice by identifying appropriate patients, additional interventions for patients, such as patient-facing notifications and resource packets or a nurse navigator, may be needed to increase patient-provider ACP conversations [23,25,26].

    Provider experiences of the ACP notification were predominantly positive. Few comments suggested inappropriate patient identification. Despite the positive response, providers found that the notification occurred too often, was burdensome to the daily work, and was difficult to navigate. This is likely related to the increased number of clicks required to process the notification in the midst of trying to care for a patient during a busy clinical day. If the provider “dismissed” the notification, the notification would reappear during the same encounter. These recurrent notifications can contribute to alert fatigue and increase provider workload and notification desensitization [27-31].

    Furthermore, there are diminishing returns for providers to refer a patient to the study or conduct ACP after 5 notifications. Patients referred to the study were more likely to have less notifications because the notification would stop once the patient was referred to the study. The higher number of notifications for patients not referred to the study appears to be a marker of providers not acting upon or disregarding the notification. More concerning is that the additional notifications are less likely to achieve the original aim of the provider user. When additional notifications no longer prompt the provider action, these notifications may be setting the stage for alert fatigue, where repeated reminders may not increase ACP conversations and instead contribute to provider notification burden. Therefore, implementation of an EHR notification for ACP must be designed with the end user in mind, particularly in the context of the additional demand placed on provider’s attention and to reduce overall burden [32-34]. One potential improvement to this ACP notification may include a limit on the number of notifications over a set period of time. Since providers are less likely to respond after 5 notifications, the notifications might ideally turn off at that time with potential to retrigger in the future—perhaps after 1 year.

    Limitations

    This substudy has a number of limitations. Less than a quarter of providers completed the electronic survey, and 1 provider provided half of the responses within the ACP notification. While response bias may limit conclusions drawn from provider perspectives on the notification, anecdotal provider feedback (from ongoing quarterly site visits conducted as part of the larger Meta-LARC study) has been similar to the feedback observed in the survey and comments. In addition, generalizability of the ACP score and notification may be limited. The EHR notification used an internally developed algorithm to identify primary care patients appropriate for ACP and is specific to our health system’s EHR (Epic) and its current user interface. Additional research is needed to validate the use of the ACP score among the primary care patient population, and this could also increase the generalizability of this algorithm to other primary care clinics within and potentially outside the health system. Lastly, the study design was limited to assessing the provider perspective on the ACP notification and did not collect data on the patient experience of care with the study providers. Future work as part of the Meta-LARC study will be evaluating the patient experience with ACP.

    Conclusions

    The findings of this substudy support the implementation of using an EHR notification to both identify patients for clinical intervention—in this case, ACP—as well as for referral to a clinical study. This substudy highlights how the timing and repetition of the notification influence provider reception and actions. Additional research is needed to better understand how EHR notifications can be optimized for the provider user and improve overall implementation and adoption of EHR notifications for study referrals and ACP.

    Acknowledgments

    We would like to thank Kimberly Leathers, Lauren Arnold, Lorrie Schmid, and Nikita Shah for their contribution to this project. The authors disclosed receipt of the following financial support for the research, authorship, and publication of this paper: The Meta-LARC ACP trial paper was funded through a Patient-Centered Outcomes Research Institute (PCORI) Award (PLC-1609-36277). The statements presented in the paper are solely the responsibility of the authors and do not necessarily represent the views of the PCORI, its Board of Governors or Methodology Committee. JEM’s effort was supported by the United States Department of Veterans Affairs Office of Academic Affiliation postdoctoral fellowship (TPH 21-000) and the Durham Veterans Affairs Center of Innovation to Accelerate Discovery and Practice Transformation (CIN 13-410). Both the Fellowship and Center are funded by the United States Department of Veterans Affairs, which had no involvement in data collection, analysis, and interpretation of results. Portions of this work have been presented at the 2020 Society of General Internal Medicine Annual Meeting, 2020 and 2021 Annual Assembly of American Academy of Hospice and Palliative Medicine, and 2022 Annual Assembly of American Academy of Hospice and Palliative Medicine: State of the Science as electronic poster presentations.

    Data Availability

    The data sets generated and analyzed during this study are not publicly available due to privacy restrictions but are available from the corresponding author upon reasonable request.

    Authors' Contributions

    All authors contributed to the concept and design of this study; JEM and SI contributed to the acquisition of the data; JEM and RD designed and conducted the analysis; JEM, JL, CB, and RD drafted the paper; all authors revised it critically for important intellectual content, approved the version to be published, and participated sufficiently in the work for appropriate portions of the content.

    Conflicts of Interest

    AK is an employee of Accompany Health; the employer had no role in the development or funding of this work. HBB reports research funding through his institution from BeBetter Therapeutics, Boehringer Ingelheim, Esperion, Improved Patient Outcomes, Merck, NHLBI, Novo Nordisk, Otsuka, Sanofi, VA. Elton John Foundation, Hilton Foundation, and Pfizer. He also provides consulting services for Abbott, Esperion, Imatar, Novartis, Sanofi, Vidya, Walmart, Webmed. He was also on the board of directors of Preventric Diagnostics.

    ‎
    Multimedia Appendix 1

    Logistic regression for number of notifications per patient and association with referral to study (N=2877).

    DOCX File , 36 KB
    ‎
    Multimedia Appendix 2

    Unadjusted logistic regression among patients with electronic health record notification and without missing data and association of patient characteristics for referral to study (N=2802).

    DOCX File , 36 KB
    1. Curtis JR, Kross EK, Stapleton RD. The importance of addressing advance care planning and decisions about do-not-resuscitate orders during novel coronavirus 2019 (COVID-19). JAMA 2020;323(18):1771-1772. [CrossRef] [Medline]
    2. Sudore RL, Lum HD, You JJ, Hanson LC, Meier DE, Pantilat SZ, et al. Defining advance care planning for adults: a consensus definition from a multidisciplinary Delphi panel. J Pain Symptom Manage 2017;53(5):821-832.e1 [FREE Full text] [CrossRef] [Medline]
    3. McMahan RD, Tellez I, Sudore RL. Deconstructing the complexities of advance care planning outcomes: what do we know and where do we go? A scoping review. J Am Geriatr Soc 2021;69(1):234-244 [FREE Full text] [CrossRef] [Medline]
    4. Detering KM, Hancock AD, Reade MC, Silvester W. The impact of advance care planning on end of life care in elderly patients: randomised controlled trial. BMJ 2010;340:c1345 [FREE Full text] [CrossRef] [Medline]
    5. Yadav KN, Gabler NB, Cooney E, Kent S, Kim J, Herbst N, et al. Approximately one in three US adults completes any type of advance directive for end-of-life care. Health Aff 2017;36(7):1244-1251. [CrossRef] [Medline]
    6. Howard M, Bernard C, Klein D, Elston D, Tan A, Slaven M, et al. Barriers to and enablers of advance care planning with patients in primary care: survey of health care providers. Can Fam Physician 2018;64(4):e190-e198 [FREE Full text] [Medline]
    7. Risk J, Mohammadi L, Rhee J, Walters L, Ward PR. Barriers, enablers and initiatives for uptake of advance care planning in general practice: a systematic review and critical interpretive synthesis. BMJ Open 2019;9(9):e030275 [FREE Full text] [CrossRef] [Medline]
    8. Hafid A, Howard M, Guenter D, Elston D, Fikree S, Gallagher E, et al. Advance care planning conversations in primary care: a quality improvement project using the serious illness care program. BMC Palliat Care 2021;20(1):122 [FREE Full text] [CrossRef] [Medline]
    9. Ashana DC, D'Arcangelo N, Gazarian PK, Gupta A, Perez S, Reich AJ, et al. "Don't talk to them about goals of care": understanding disparities in advance care planning. J Gerontol A Biol Sci Med Sci 2022;77(2):339-346 [FREE Full text] [CrossRef] [Medline]
    10. Pollak KI, Gao X, Beliveau J, Griffith B, Kennedy D, Casarett D. Pilot study to improve goals of care conversations among hospitalists. J Pain Symptom Manage 2019;58(5):864-870 [FREE Full text] [CrossRef] [Medline]
    11. Rose BL, Leung S, Gustin J, Childers J. Initiating advance care planning in primary care: a model for success. J Palliat Med 2019;22(4):427-431. [CrossRef] [Medline]
    12. Devoe C, Gabbidon H, Schussler N, Cortese L, Caplan E, Gorman C, et al. Use of electronic health records to develop and implement a silent best practice alert notification system for patient recruitment in clinical research: quality improvement initiative. JMIR Med Inform 2019;7(2):e10020 [FREE Full text] [CrossRef] [Medline]
    13. Embi PJ, Jain A, Clark J, Bizjack S, Hornung R, Harris CM. Effect of a clinical trial alert system on physician participation in trial recruitment. Arch Intern Med 2005;165(19):2272-2277 [FREE Full text] [CrossRef] [Medline]
    14. Ni Y, Kennebeck S, Dexheimer JW, McAneney CM, Tang H, Lingren T, et al. Automated clinical trial eligibility prescreening: increasing the efficiency of patient identification for clinical trials in the emergency department. J Am Med Inform Assoc 2015;22(1):166-178 [FREE Full text] [CrossRef] [Medline]
    15. Kruse CS, Mileski M, Dray G, Johnson Z, Shaw C, Shirodkar H. Physician burnout and the electronic health record leading up to and during the first year of COVID-19: systematic review. J Med Internet Res 2022;24(3):e36200 [FREE Full text] [CrossRef] [Medline]
    16. Khairat S, Coleman C, Ottmar P, Jayachander DI, Bice T, Carson SS. Association of electronic health record use with physician fatigue and efficiency. JAMA Netw Open 2020;3(6):e207385 [FREE Full text] [CrossRef] [Medline]
    17. Totten AM, Fagnan LJ, Dorr D, Michaels LC, Izumi SS, Combe A, et al. Protocol for a cluster randomized trial comparing team-based to clinician-focused implementation of advance care planning in primary care. J Palliat Med 2019;22(S1):82-89 [FREE Full text] [CrossRef] [Medline]
    18. Kim A, Lowe J, Greenblatt L, Udofia I, Jackson C, Bolwby L. A pop health approach to advance care planning through primary care: leveraging health IT, data science, and patient navigators. J Gen Intern Med 2018;33(S2):S755-S756.
    19. Kim A, Lowe J. The urgency of now—ask your patients about goals of care. Here’s how health systems can do it. Medium. 2020.   URL: https://medium.com/@azaleakim/the-urgency-of-now-ask-your-patients-about-goals-of-care-b2a877f85cb [accessed 2021-08-24]
    20. Casarett D, Lakis K, Ma JE, Gentry J, Fischer J, Ibrahim S, et al. Goal-concordant care: end-of-life planning conversations for all seriously ill patients. NEJM Catal Innov Care Deliv 2022;3(12):CAT.22.0271. [CrossRef]
    21. R Core Team. R: a language and environment for statistical computing. R Foundation for Statistical Computing. 2020.   URL: https://www.R-project.org/ [accessed 2023-04-13]
    22. Hunninghake DB, Darby CA, Probstfield JL. Recruitment experience in clinical trials: literature summary and annotated bibliography. Control Clin Trials 1987;8(suppl 4):6S-30S. [CrossRef] [Medline]
    23. Tung EE, Vickers KS, Lackore K, Cabanela R, Hathaway J, Chaudhry R. Clinical decision support technology to increase advance care planning in the primary care setting. Am J Hosp Palliat Care 2011;28(4):230-235 [FREE Full text] [CrossRef] [Medline]
    24. Manz CR, Parikh RB, Small DS, Evans CN, Chivers C, Regli SH, et al. Effect of integrating machine learning mortality estimates with behavioral nudges to clinicians on serious illness conversations among patients with cancer: a stepped-wedge cluster randomized clinical trial. JAMA Oncol 2020;6(12):e204759 [FREE Full text] [CrossRef] [Medline]
    25. Gabbard J, Pajewski NM, Callahan KE, Dharod A, Foley KL, Ferris K, et al. Effectiveness of a nurse-led multidisciplinary intervention vs usual care on advance care planning for vulnerable older adults in an accountable care organization: a randomized clinical trial. JAMA Intern Med 2021;181(3):361-369 [FREE Full text] [CrossRef] [Medline]
    26. Bush RA, Pérez A, Baum T, Etland C, Connelly CD. A systematic review of the use of the electronic health record for patient identification, communication, and clinical support in palliative care. JAMIA Open 2018;1(2):294-303 [FREE Full text] [CrossRef] [Medline]
    27. Embi PJ, Leonard AC. Evaluating alert fatigue over time to EHR-based clinical trial alerts: findings from a randomized controlled study. J Am Med Inform Assoc 2012;19(e1):e145-e148 [FREE Full text] [CrossRef] [Medline]
    28. Murphy DR, Meyer AND, Russo E, Sittig DF, Wei L, Singh H. The burden of inbox notifications in commercial electronic health records. JAMA Intern Med 2016;176(4):559-560 [FREE Full text] [CrossRef] [Medline]
    29. Kesselheim AS, Cresswell K, Phansalkar S, Bates DW, Sheikh A. Clinical decision support systems could be modified to reduce 'alert fatigue' while still minimizing the risk of litigation. Health Aff 2011;30(12):2310-2317. [CrossRef] [Medline]
    30. Ancker JS, Edwards A, Nosal S, Hauser D, Mauer E, Kaushal R, with the HITEC Investigators. Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Med Inform Decis Mak 2017;17(1):36 [FREE Full text] [CrossRef] [Medline]
    31. Poly TN, Islam MM, Yang HC, Li YJ. Appropriateness of overridden alerts in computerized physician order entry: systematic review. JMIR Med Inform 2020;8(7):e15653. [CrossRef] [Medline]
    32. Gregory ME, Russo E, Singh H. Electronic health record alert-related workload as a predictor of burnout in primary care providers. Appl Clin Inform 2017;8(3):686-697 [FREE Full text] [CrossRef] [Medline]
    33. Wang E, Major VJ, Adler N, Hauck K, Austrian J, Aphinyanaphongs Y, et al. Supporting acute advance care planning with precise, timely mortality risk predictions. NEJM Catal Innov Care Deliv 2021;2(3):CAT.20.0655. [CrossRef]
    34. Richardson S, Dauber-Decker K, Solomon J, Khan S, Barnaby D, Chelico J, et al. Nudging health care providers' adoption of clinical decision support: protocol for the user-centered development of a behavioral economics-inspired electronic health record tool. JMIR Res Protoc 2023;12:e42653 [FREE Full text] [CrossRef] [Medline]

    ‎
    ACP: advance care planning
    CPT: Current Procedural Terminology
    EHR: electronic health record
    IRB: Institutional Review Board
    Meta-LARC: Meta-network Learning and Research Center
    OR: odds ratio
    PCORI: Patient-Centered Outcomes Research Institute
    POLST: Physician Orders for Life-Sustaining Treatment
    SICP: Serious Illness Care Program
    SNOMED: Systematized Nomenclature of Medicine

    Edited by T Leung; submitted 13.08.22; peer-reviewed by M Howard, D Ose, M Aczon; comments to author 21.11.22; revised version received 17.02.23; accepted 21.03.23; published 12.05.23

    Copyright

    ©Jessica E Ma, Jared Lowe, Callie Berkowitz, Azalea Kim, Ira Togo, R Clayton Musser, Jonathan Fischer, Kevin Shah, Salam Ibrahim, Hayden B Bosworth, Annette M Totten, Rowena Dolor. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 12.05.2023.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.