This paper is in the following e-collection/theme issue:

Personal Health Records, Patient-Accessible Electronic Health Records, Patient Portals (538) Human Factors and Usability Case Studies (739) User-centred Design Case Studies (562) Design and Usability of Clinical Software and EHRs (242)

Published on in Vol 20, No 7 (2018): July

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/10413, first published .
A Decade of Veteran Voices: Examining Patient Portal Enhancements Through the Lens of User-Centered Design

A Decade of Veteran Voices: Examining Patient Portal Enhancements Through the Lens of User-Centered Design

A Decade of Veteran Voices: Examining Patient Portal Enhancements Through the Lens of User-Centered Design

Authors of this article:

Kim M Nazi1 Author Orcid Image ;   Carolyn L Turvey2, 3 Author Orcid Image ;   Dawn M Klein2, 3 Author Orcid Image ;   Timothy P Hogan4, 5 Author Orcid Image
Article Authors Cited by (28) Tweetations (16) Metrics

    Original Paper

    1Veterans and Consumers Health Informatics Office, Office of Connected Care, Veterans Health Administration, US Department of Veterans Affairs, Coxsackie, NY, United States

    2Iowa City VA Health Care System, Comprehensive Access and Delivery Research and Evaluation Center, Iowa City, IA, United States

    3Department of Psychiatry, The University of Iowa Carver College of Medicine, Iowa City, IA, United States

    4Center for Healthcare Organization and Implementation Research (CHOIR), Edith Nourse Rogers Memorial Veterans Hospital, Bedford, MA, United States

    5Division of Health Informatics and Implementation Science, Department of Quantitative Health Sciences, University of Massachusetts Medical School, Worcester, MA, United States

    *all authors contributed equally

    Corresponding Author:

    Kim M Nazi, PhD

    Veterans and Consumers Health Informatics Office, Office of Connected Care

    Veterans Health Administration

    US Department of Veterans Affairs

    44 Mountain View Drive

    Coxsackie, NY, 12051

    United States

    Phone: 1 518 703 6330

    Email: kmnazi@gmail.com


    Background: Health care systems have entered a new era focused on patient engagement. Patient portals linked to electronic health records are recognized as a promising multifaceted tool to help achieve patient engagement goals. Achieving significant growth in adoption and use requires agile evaluation methods to complement periodic formal research efforts.

    Objective: This paper describes one of the implementation strategies that the Department of Veterans Affairs (VA) has used to foster the adoption and sustained use of its patient portal, My HealtheVet, over the last decade: an ongoing focus on user-centered design (UCD). This strategy entails understanding the users and their tasks and goals and optimizing portal design and functionality accordingly. Using a case study approach, we present a comparison of early user demographics and preferences with more recent data and several examples to illustrate how a UCD can serve as an effective implementation strategy for a patient portal within a large integrated health care system.

    Methods: VA has employed a customer experience analytics (CXA) survey on its patient portal since 2007 to enable ongoing direct user feedback. In a continuous cycle, a random sample of site visitors is invited to participate in the Web-based survey. CXA model questions are used to track and trend satisfaction, while custom questions collect data about users’ characteristics, needs, and preferences. In this case study, we performed analyses of descriptive statistics comparing user characteristics and preferences from FY2008 (wherein “FY” means “fiscal year”) to FY2017 and user trends regarding satisfaction with and utilization of specific portal functions over the last decade, as well as qualitative content analysis of user’s open-ended survey comments.

    Results: User feedback has guided the development of enhancements to core components of the My HealtheVet portal including available features, content, interface design, prospective functional design, and related policies. Ten-year data regarding user characteristics and portal utilization demonstrate trends toward greater patient engagement and satisfaction. Administration of a continuous voluntary Web-based survey is an efficient and effective way to capture veterans’ voices about who they are, how they use the patient portal, needed system improvements, and desired additional services.

    Conclusions: Leveraging “voice-of-the-customer” techniques as part of patient portal implementation can ensure that such systems meet users’ needs in ways that are agile and most effective. Through this strategy, VA has fostered significant adoption and use of My HealtheVet to engage patients in managing their health.

    J Med Internet Res 2018;20(7):e10413

    doi:10.2196/10413

    Keywords

    patient portal; user-centered design; eHealth; veteran 


    Background

    Health care systems have entered a new era focused on patient engagement [1-3] described by enthusiasts as “the holy grail of health care” [4] and the “blockbuster drug of the century” [5]. Patient engagement strategies are designed to empower patients to play a more active role in their health care and make informed decisions, improve the patient experience, increase patient satisfaction, and achieve better health outcomes. Patient portals linked to electronic health records (EHRs) are recognized as a promising multifaceted tool to help achieve these patient engagement goals [6-9]. However, the adoption and sustained use of portals has generally fallen short of initial optimism [10-13] even in light of the significant growth in EHRs and tethered patient portals incentivized by Meaningful Use [14]. Positive benefits of portal use have been demonstrated [15-19], and the OpenNotes movement [20] has promoted patient engagement through health records transparency by enabling patient access to provider notes. Evidence indicates that such access improves communication and trust, patient safety, and, potentially, patient outcomes [21-25]. Two large integrated health care systems that launched tethered patient portals in 2003 with significant patient adoption and sustained use are Kaiser Permanente (KP) and the Department of Veterans Affairs (VA). KP’s portal, My Health Manager, is used by more than 5 million members, representing about 70% of adult KP members [26]. VA’s patient portal, My HealtheVet, has more than 4 million registered users (69% of VA patients receiving health care services in FY 2017 [wherein “FY” means “fiscal year”]), with 2.5 million authenticated Premium accounts (42% of VA patients receiving health care services) required for access to all portal features [27]. To better understand what elements are driving this adoption and sustained use, an implementation case study approach is warranted. One of the implementation strategies that are critical to foster the adoption and sustained use of patient portals is an ongoing focus on user-centered design (UCD). This is often accomplished as part of periodic research studies; however, more timely and agile methods are needed to design and evaluate patient portals.

    User-Centered Design

    UCD is a design philosophy and evaluation process that focuses on the end user’s characteristics, needs, preferences, and limitations throughout the design process and development lifecycle [28]. The emphasis of UCD is on understanding the end users and their tasks and goals and optimizing the product to enable the users to fulfill these, rather than requiring users to adapt to the designer’s preferences [29]. UCD of eHealth applications, such as patient portals, necessitates ongoing assessment of user characteristics and preferences and incorporation of assessment insights into ongoing portal development and enhancements. This process includes focusing on what features are considered to be most essential by users [30]. Published compilations of implementation strategies have called for the further development of processes like UCD as a means to obtain and use patient or consumer feedback to support the adoption of innovations and practice change efforts in health care and other settings [31].

    VA has used various methods over the last decade to achieve UCD for My HealtheVet; among them, the principal method has been a continuous, voluntary, and anonymous survey of end users. As a complement to periodic formal research studies [32,33], this ongoing assessment offers the advantage of rapid continuous feedback, which is part of a cyclical process for improvement that entails understanding users, eliciting their input, identifying changes or future design implications, deploying enhancements, and then obtaining feedback to evaluate these enhancements. This method enables VA to obtain ongoing direct feedback from veterans, which can then be leveraged to improve the patient experience.

    About the Department of Veterans Affairs’ Patient Portal

    VA is the largest integrated health care system in the United States and has been a pioneer in enabling patients to access and download their VA medical record data using the Blue Button feature [34,35]. This includes OpenNotes, which are known in My HealtheVet as VA Notes and contain both clinical and mental health providers’ notes [36]. The My HealtheVet patient portal [37] is tethered to the VA EHR and provides a suite of Web-based tools. Veterans self-register to create a basic account and can then self-enter information into their personal health record and access health education resources. VA patients who are matched by the system via the Master Veteran Index are automatically upgraded to an Advanced account and can request VA prescription refills. Patients who complete a one-time process of identity authentication (in person or Web-based) are upgraded to a Premium account and can then access all portal features, including access to health record information and Secure Messaging with VA health care professionals.

    Use of My HealtheVet continues to grow. In fiscal year (FY)2017, portal user activity demonstrated significant increases compared with that in FY2016, including a 20.7% increase in Web-based prescription refills, a 33.9% increase in Secure Messaging exchanges between VA patients and their health care team, and a 38.7% increase in use of the VA Blue Button feature [27].

    In this paper, we examine one of the implementation strategies that VA has used to foster adoption and sustained use of its patient portal over the last decade: an ongoing focus on UCD. This includes iterative use of survey and operational data with user interface redesign to meet the needs and preferences of veteran users. We describe the organization’s implementation strategy for agile UCD and present unique 10-year data on user adoption, characteristics, and utilization to demonstrate trends toward greater patient engagement and satisfaction. Following an initial analysis of portal users and their preferences in 2007 [38], we compared the characteristics of patient portal users one decade later and used a case study approach to present several examples of how user preferences and continuous feedback have informed the evolution of VA’s patient portal.


    Since 2007, VA has used the ForeSee customer experience analytics (CXA) survey tool for the direct measurement of customer satisfaction and prioritization of enhancements. The CXA survey is a standardized method of measuring and monitoring customer satisfaction based on the American Customer Satisfaction Index [39]. The survey methodology uses a psychometric “voice-of-the-customer” technique to assess consumer drivers of satisfaction (look and feel, navigation, site information, site performance, and task processes) and prioritize areas of improvement. In the CXA model, scores are based on data from randomized voluntary Web-based surveys and are reported on a scale of 0 to 100, indicating less to more customer satisfaction. Multiple item measures are combined algorithmically to compile a satisfaction index each time an adequate quantity of data has been collected through completed surveys [40]. The survey tool for the My HealtheVet portal includes standard questions, to allow for trend analysis of core components such as overall satisfaction, and user experience of navigation. The inclusion of custom questions on an as-needed basis further enables the collection of rich data about user demographics, needs, and preferences to address specific and time-sensitive evaluation topics and to inform ongoing design and development efforts.

    The CXA survey is conducted with all veterans using My HealtheVet and is, therefore, a nationwide sample of veteran My HealtheVet users. The survey is implemented on the My HealtheVet portal as a Web-based pop-up browser window inviting a random sample of site visitors to participate. A persistent cookie prevents site visitors who received the survey invitation from being invited again for 90 days. When visitors accept the invitation, the survey presents when they leave the site. The loyalty factor, currently 4 pages, ensures that respondents have experienced multiple pages on the site before being prompted to participate in the survey. The sampling percentage, set at 13% in FY2008 and later changed to 4% in FY2010 due to the large amount of data being collected and increasing survey completion rates, ensures that a minimum number of site visitors are surveyed in order to reduce respondent burden while enabling the collection of adequate data.

    This paper presents selected analyses of the CXA survey data collected over a course of 10 years, including a comparison of data collected early in the implementation of My HealtheVet (FY2008) to more recent data (FY2017), to examine the characteristics of patient portal users and their preferences.

    Data analysis is primarily descriptive and based on forced-choice responses. Analysis of open-ended comments includes a combination of traditional qualitative techniques [41,42] along with keyword clustering to group related comments for further analysis. A variety of strategies are used to then translate insights into iterative improvements, including ongoing data reviews, requirement elaboration, design sessions with key stakeholders, and review of user feedback after deployment of enhancements.


    Overview

    We first present a recent summary of user demographics and characteristics and patterns of portal use and relevant comparisons to previous data. Following our case study approach, we then provide selected examples from the My HealtheVet evaluation program to illustrate how different assessments that capture the voice of the customer have directly informed the evolution of the portal and the addition of new functionality. For FY2008 (October 1, 2007-September 30, 2008), of the surveys presented to site visitors, 17.1% (100,069/585,039) were completed. For FY2017 (October 1, 2016-September 30, 2017), of the surveys presented to site visitors, 68.9% (100,555/146,023) were completed. As completion rates increased over the last decade, the sampling rate was reduced in FY2010 from 13% to 4% in order to minimize respondent burden.

    User Demographics and Characteristics

    Table 1 provides a comparison of user demographics and characteristics for all survey respondents in FY2017 and FY2008. In FY2017, 97% (97,538/100,555) of respondents were veterans compared with 93% (93,064/100,069) in FY2008. Respondents reported having completed higher levels of education, with 40% (39,990/99,974) being college graduates, completing some postgraduate school, or having a graduate or professional degree in FY2017 compared with 34% (732/2152) in FY2008. The proportion of male respondents increased slightly to 93% (90,507/97,319) in FY2017. In FY2017, respondents were generally older, with 64% (59,819/93,467) in the age range of 60-74 years compared with 47% (14,563/30,984) in FY2008; furthermore, 17% (15,889/93,467) of respondents in FY2017 were older than 75 years. This shift in age is also shown in Figure 1.

    While 60% (60,042/100,069) of users in FY2008 reported their period of military service as the Vietnam War, this increased to 67% (67,372/100,555) of users in FY2017. Fewer users self-reported their internet ability as advanced in FY2017 (32,235/53,725, 60%) than in FY2008 (37,848/55,658, 68%), whereas more users reported it as intermediate in FY2017 (19,341/53,725, 36%) than in FY2008 (16,141/55,658, 29%). A greater proportion of respondents reported better health in FY2017, with 34% (33,323/98,007) reporting fair or poor health in FY2017 compared with 39% (15,723/40,315) in FY2008. Although the FY2008 survey did not ask users about health conditions, responses in FY2017 revealed a high prevalence of chronic conditions, including high blood pressure (15,045/22,795, 66%), high cholesterol (14,133/22,795, 62%), arthritis (13,677/22,795, 60%), chronic pain (10,714/22,795, 47%), diabetes (8434/22,795, 37%), stomach or gastrointestinal problems (8434/22,795, 37%), and heart problems (8434/22,795, 37%).

    Table 1. Demographics and characteristics. FY: fiscal year. N/A: Not applicable. VA: Department of Veterans Affairs.
    TypeFY2017FY2008
    Rolea, n (%)100,555100,069

    		Veteran9793

    		Family member35

    		Veteran Service Organization11

    		National Guard or Reserve1N/A

    		General public1<1

    		Other role11

    		VA employee11

    		Non-VA federal employee<11

    		Caregiver (other than family)<1N/A

    		State or local government<1N/A

    		Active duty<1<1

    		News MediaN/A<1
    Highest level of education, n (%)99,9742154

    		Did not complete high school32

    		High school graduate1317

    		Some college or vocational school4444

    		College graduate2119

    		Some postgraduate school65

    		Graduate or professional degree1310
    Health conditionsa, n (%)22,795N/A

    		High blood pressure66N/A

    		High cholesterol62N/A

    		Arthritis of any kind60N/A

    		Chronic pain47N/A

    		Diabetes37N/A

    		Stomach or gastrointestinal problems37N/A

    		Heart problems37N/A

    		Mental health or psychiatric condition34N/A

    		Cancer of any kind29N/A

    		Lung problems (including asthma)25N/A

    		Neurological disorders13N/A

    		Other12N/A

    		Prefer not to answer2N/A
    Age, n (%)93,46730,984

    		Under 20<1<1

    		20-24<1<1

    		25-29<1<1

    		30-34<11

    		35-3912

    		40-4414

    		45-4936

    		50-54510

    		55-59818

    		60-641326

    		65-692814

    		70-74237

    		75-7995

    		80-8453

    		85 or older31
    Gender, n (%)97,31931,020

    		Male9391

    		Female79
    Self-reported health status, n (%)98,00740,315

    		Excellent45

    		Very good2118

    		Good4138

    		Fair2729

    		Poor710
    Self-reported internet ability, n (%)53,72555,658

    		Beginner44

    		Intermediate3629

    		Advanced6068

    aMultiple categories may be selected.

    ‎
    Figure 1. User age ranges for fiscal year (FY)2008 and FY2017.
    View this figure

    Portal Access Patterns and Usage

    Table 2 provides a summary of survey respondents’ self-reported portal access patterns and usage. While the proportion of survey respondents who use VA health care services remained the same from FY2008 to FY2017 (96%), in FY2017 50% (47,066/94,132) of respondents stated that they also use a community non-VA provider. When asked about travel time to the nearest VA facility, 32% respondents reported it to be less than 30 minutes in both FY2008 and FY2017; however, a greater proportion noted fewer minutes of travel time in FY2017 than in FY2008. For example, 37% (19,902/53,788) respondents in FY2008 reported a travel time greater than 60 minutes to the nearest VA facility compared with 24% (12,075/50,313) in FY2017. The proportion of respondents who have a Premium account, offering them access to all portal services, increased significantly from 60% (56,884/94,806) in FY2008 to 77% (73,001/94,806) in FY2017. While a greater number of respondents were first time users in FY2008 (12,074/100,617, 12%) than in FY2017 (4022/100,555, 4%), respondents reported using the portal more frequently in FY2017, with 46% (45,255/100,555) using it about once a month and 29% (29,161/100,555) using it about once a week. When asked about the length of use in FY2017, 63% (63,349/100,5455) respondents reported having used My HealtheVet for more than 2 years.

    User Preferences and Responsive Design

    In keeping with the UCD process, VA has used direct veteran feedback about preferences obtained via the CXA survey to shape the identification and prioritization of portal improvements. In this section, we describe how different types of user feedback have directly informed enhancements to the core components of the system including available features, interface design, content, policy, and prospective functional design of a new feature.

    Additional Services Desired

    UCD principles focus on identifying what features users consider to be essential. One survey question that has been crucial in getting feedback to prioritize portal enhancements over the last decade has been “What additional services would you like to see on My HealtheVet?” As shown in Table 3, additional services desired by users in FY2008 included the ability to view (79,892/92,160, 87%) or schedule (68,395/92,160, 74%) VA Appointments, access information from the VA medical record (67,714/92,160, 73%), and Web-based secure communication with my doctor (58,878/92,160, 64%). Each of these features was subsequently added to the portal (Table 4).

    Secure Messaging implementation began in 2008, which enabled secure Web-based communication with VA health care teams, with the full national release to all VA primary care providers in 2012. Veterans could then also use Secure Messaging to request VA Appointments. The ability to view upcoming VA Appointments was deployed in 2011, with appointment email reminders added in 2015. Building on early access to VA Medication History, VA incrementally expanded the types of information from the VA medical record available in My HealtheVet, for example, preventative Wellness Reminders (2009), VA Chemistry or Hematology Lab Results (2011), VA Immunizations (2012), VA Notes including mental health notes (2013), a more comprehensive Medication List that includes patient-reported non-VA medications (2016), Surgical and Clinical Procedure Notes (2017), and VA Medical Images and Reports (2017).

    Additional services desired by users in FY2017 included the ability to schedule or change VA Appointments directly (52%), a list of health care providers and their contact information (44%), a tool to determine whether different medications are safe when taken together (26%), and the ability to view and pay VA bills or copayments (25%). The ability to schedule or change VA Appointments directly was piloted in FY2017 and is being rolled out to all VA facilities in FY2018. The enhanced VA Health Summary (2017) provides VA patients with a list of their primary health care providers, which will be expanded in FY2018 to include their contact information. Although VA has not yet invested in the development of tools to check medications for potential interactions; this enhancement is being given further consideration in FY2018. In addition, the ability to view a VA Patient Statement and remit payment is also being developed and scheduled for pilot testing in FY2018.

    Patient-Identified Main Improvements

    In addition to eliciting user feedback on additional services desired, the CXA survey also invites open-ended comments in response to the question: “What is the main improvement that you would suggest for the My HealtheVet website?” Below we offer examples of how these comments have led to user-directed improvements.

    With the expansion of Lab and Test Results and the addition of VA Notes in January 2013, one theme that surfaced in the ensuing months was veterans’ desire for more timely access to this information. These comments were crucial in driving VA policy change to reduce the hold period for lab results and progress notes from 7 calendar days after verification to 3 calendar days. This policy change was implemented in June 2013.

    To complement the prioritization of known desired additional services by users, open-ended main improvement comments also allow veterans to suggest needed functional enhancements in their own words. In October 2013, thematic analysis of free-text comments identified the need for multiple functional enhancements including the ability to track delivery of the filled prescriptions, the desire to be notified before automatic log out when the user session was nearing time-out, and the need for improved navigation to complete common tasks. The ability to track delivery of mailed prescriptions by opting-in to receive an email notification was deployed in 2015. Other functional enhancements (session time-out warning, improved navigation, and reduced number of steps to complete common tasks) became core requirements for a major website redesign project. The session time-out warning and ability to extend the session time was deployed as VA migrated to a content management system in October 2016. The incremental deployment of website redesign in October 2016 and September 2017 was significantly informed by veteran main improvement comments:

    Publishing labs and notes within 24 hours of a lab or health visit. Waiting a week for lab results, or a week for Dr and nurse notes is absurd, given that the health problem is “right now,” not right now + seven days, especially when Dr's notes are also instructions for post visit procedures, such as when and how much meds to take, or “If it hasn't improved in three days” see me. Not everyone is “present” at the end of a visit due mostly to anxieties surrounding the visit.
    Table 2. Access patterns. FY: fiscal year. N/A: not applicable. VA: Department of Veterans Affairs.
    Respondent CharacteristicFY2017FY2008
    Use VA health care services, n (%)98,00729,528

    		Yes9696

    		No34

    		Not sure1N/A
    Use community non-VA providers, n (%)94,132N/A

    		Yes50N/A

    		No47N/A

    		Not sure3N/A
    Premium My HealtheVet accounta, n (%)94,806100,617

    		Yes7760

    		No924

    		Not sure1515

    		Not applicableN/A1
    Travel time to nearest VA facilitya, n (%)50,31353,788

    		Less than 30 min3232

    		30-60 min4332

    		61-90 min1420

    		91 min to 2 h69

    		Over 2 h48

    		Not sureN/A1
    Frequency of use, n (%)100,555100,617

    		Daily or more than once a day55

    		About once a week2925

    		About once a month4649

    		About every 6 mo95

    		Less than every 6 mo43

    		First time412

    		Not sure or Do not recall2N/A
    Length of use, n (%)100,555N/A

    		Less than 6 mo9N/A

    		6 mo-less than 1 y6N/A

    		1-2 y19N/A

    		More than 2 y63N/A

    		Not sure or Do not recall3N/A

    aPercentages do not add to 100 due to rounding.

    Table 3. Additional services desired. FY: fiscal year. VA: Department of Veterans Affairs.
    Servicen (%)
    FY201788,308

    		Schedule or change my VA appointments45,695 (52)

    		View a list of my VA health care providers and their contact information38,489 (44)

    		Check to determine if my different medications are safe taken together22,710 (26)

    		View or pay my VA bills or copayments21,768 (25)

    		Use a mobile app for My HealtheVet13,823 (16)

    		Advance check-in for my VA clinic visits12,677 (14)

    		Authorize sharing information with my Non-VA health care provider11,467 (13)

    		Authorize sharing information with my VA health care team8851 (10)

    		Authorize sharing information with other people (eg, family, caregiver)7584 (9)

    		Other6573 (7)

    		More Web-based educational programs5396 (6)

    		Join a Web-based forum to discuss health issues with other veterans3831 (4)
    FY200892,160

    		View my upcoming appointments79,892 (87)

    		Schedule or change my appointments68,395 (74)

    		Look at information in my VA medical record67,714 (73)

    		Web-based, secure communication with my doctor58,878 (64)

    		Checking that different medications I take are safe when used together45,986 (50)

    		Reminders of preventive care I need (eg, shots, cancer screening)34,707 (38)

    		Notification of new content or features on the site32,418 (35)

    		Advance check-in for my VA clinic visits31,863 (35)

    		Monthly email newsletter24,186 (26)

    		Share information that I have stored in My HealtheVet with other people23,088 (25)

    		Advanced directive (eg, living will, durable power of attorney)20,418 (22)

    		Educational programs18,800 (20)

    		Information about the quality of VA health care11,231 (12)

    		Other8791 (10)
    Table 4. My HealtheVet history and feature enhancement milestones. DoD: Department of Defense. EHR: electronic health record. VA: Department of Veterans Affairs.
    YearMilestone
    1999
    • My HealtheVet Pilot at 9 VA Medical Centers
      • ‎
    2003
    • National My HealtheVet Portal deployed
      • ‎
    2004
    • New user registration module deployed
      • ‎
    • Expansion of self-entered data modules
      • ‎
    2005
    • Prescription (Rx) Refill requests
      • ‎
    • Additional self-entered modules
      • ‎
    2006
    • In Person Authentication to Upgrade to Premium Account
      • ‎
    2007
    • Account Activity History
      • ‎
    • Forgot User ID and Password Support
      • ‎
    • Upgraded Health Calendar
      • ‎
    2008
    • Secure Messaging deployed for voluntary provider use
      • ‎
    • Master Veteran Index synchronization
      • ‎
    2009
    • VA Wellness Reminders
      • ‎
    2010
    • VA Blue Button Feature (Download My Data)
      • ‎
    2011
    • VA Appointments
      • ‎
    • VA Allergies
      • ‎
    • VA Chemistry and Hematology Lab Results
      • ‎
    • DoD Military Service Information
      • ‎
    • Display Rx Medication Name
      • ‎
    2012
    • Secure Messaging with all VA primary care providers
      • ‎
    • VA Immunizations
      • ‎
    • Veterans with DoD log-on credential can use to log in to portal (single sign on)
      • ‎
    • Social media content promotion
      • ‎
    2013
    • Expansion of VA EHR data in VA Blue Button Report (eg, VA Notes, VA Radiology Reports, Pathology Reports, Microbiology Lab Results, etc)
      • ‎
    • Basic VA Health Summary added
      • ‎
    • Hold Periods reduced from 7 to 3 calendar days
      • ‎
    • HealtheLiving Assessment (health risk appraisal)
      • ‎
    • Veterans Health Library
      • ‎
    2014
    • Ability to send Secure Messaging attachments
      • ‎
    • Migration to cloud environment for system stability, scalability, and performance
      • ‎
    • Log-in enhancements
      • ‎
    • Display medication images in pharmacy module
      • ‎
    2015
    • Secure Messaging Workload Credit
      • ‎
    • Rx Refill Shipment Email Notification
      • ‎
    • VA Appointment Email Reminders
      • ‎
    • Save Secure Messaging Progress Notes to VA EHR
      • ‎
    • Subscribe to My HealtheVet Newsletter
      • ‎
    2016
    • Content Management System deployed
      • ‎
    • Incremental Redesign: homepage dashboard navigation
      • ‎
    • Session time out warning
      • ‎
    2017
    • Enhanced VA Health Summary with Surgical and Clinical Procedure Notes
      • ‎
    • Incremental Redesign theme deployment
      • ‎
    • VA Medical Images and Reports Pilot
      • ‎
    • Personalized Veteran’s Benefits Handbook
      • ‎
    • Appointment scheduling
      • ‎
    ‎
    Figure 2. Veteran comments about hold periods.
    View this figure
    Why does it take so long for results of lab work, radiology, notes from Drs to show up? It can take a week or more. My Dr already called with the results yesterday but I still can’t see it here. Also saw GI Dr 3 days ago and not notes here. I wish we could access our information sooner.
    Eliminate wait period to view VA Notes, Results, etc. Once the provider has entered the note or viewed the results of test, they should be made available for viewing by the veteran.
    Aligning Content With Patient-Suggested Topics of Interest

    Periodically, an open-ended question is added to the survey asking users about topics of interest for portal content, such as feature articles to ensure that content is directly aligned with veterans’ needs and preferences. An editorial calendar is created to provide articles throughout the year focused on these topics. Topics are also highlighted in a subscription-based monthly electronic newsletter that was developed in 2015 as a user-desired additional service (see Table 3), with more than 500,000 subscribers in FY2017. Examples of topic clusters for August 2009 (N=1809), August 2013 (N=3300), and August 2017 (N=1189) are shown in Figure 2. In 2017, the top user-suggested topics included Health (“general health, age concerns, pre-existing medical issues”), Diabetes (“articles on diabetes and feet or hand or finger neuropathy”), Care (“information on special health care programs for specific conditions”), and Agent Orange (“need more information on Agent Orange exposure and health issues”).

    Prospective Functional Design of a New Feature

    Veteran feedback has also driven the functional design of new features. One feature that is currently being developed is the ability for the users to assign a delegate who can access their account. For example, a spouse or caregiver who may be assisting a veteran patient in managing his or her health. In October 2014, VA convened key stakeholders and subject matter experts to define the business requirements for this feature; however, there was a lack of consensus on a key functional requirement: whether “read-only” access should allow or restrict a delegate’s ability to also print and download data. Using the CXA survey, veterans were asked “If you approve read access for another person to help you manage your personal health information, what would you want that person to be able to do?”

    Of those veterans with a preference to delegate read access to another person, 75% (8194/11,006) would want such access to include print and download capability, while 14% (1541/11,006) would want a delegate to be able to read or view their information on the screen, but not print or download it. With this direct veteran input on desired functional design, requirements were prospectively aligned with user preferences. Data were also collected to assess patient preferences regarding delegating access to health information [43], use of My HealtheVet to transfer information [44], how veterans with non-VA providers use the Blue Button feature to share information with their non-VA providers [45], and the veteran experiences with access to their VA Notes [36].

    Website Redesign and Satisfaction Trends

    Analysis of CXA data over the course of the last decade has been an integral part of the recent My HealtheVet website redesign initiative by enabling a deeper understanding of the end users and their tasks and goals, in keeping with UCD principles. As shown in Table 5, while 75% (75,241/100,617), 24% (23,923/100,617), and 18% (17,899/100,617) users in FY2008 accessed the portal to request a prescription refill, view their medication history, and look up information about a medication, respectively, user goals and tasks in FY2017 have shifted and expanded. Although prescription refill requests remained a predominant task (53,193/100,555, 53%), users also accessed the portal to view their VA Appointments (38,664/100,555, 38%), communicate with their health care team using Secure Messaging (28,952/100,555, 29%), track the delivery status of their medication refills (23,884/100,555, 24%), view their lab or test results (19,382/100,555, 19%), and access their VA health records (11,966/100,555, 12%). An important goal of the culminating website redesign was to improve navigation and usability for these specific core features, and the overall customer satisfaction index score was used as a performance indicator.

    Historical customer satisfaction trends are shown in Figure 3. From October 2007 to October 2015, the aggregate average CXA score was 74, based on 945,480 completed surveys. The average for the 12 months that followed was stable at 76 (N=139,934). While multiple factors impacted customer satisfaction over the last decade, including a period of system performance issues in 2014 that was resolved by improving system architecture, the overall trend toward greater customer satisfaction is evident.

    In October 2016, as part of an incremental website redesign, a dashboard was added to the portal home page to enhance user access to the core features (Figure 4).

    As anticipated, the introduction of changes to the website resulted in an initial decrease in satisfaction (72), followed by satisfaction recovery (75), and subsequent increase to a new high of 79 (Figure 5). A similar pattern was observed with the deployment of additional website redesign changes in September 2017. Satisfaction initially decreased (77), but then recovered to previous levels (79). Satisfaction continued to increase in January 2018 (80).

    Table 5. User-specified goals and tasks. FY: fiscal year. N/A: not applicable. VA: Department of Veterans Affairs.
    Reason for visit or goal trying to accomplishaFY2017 (N=100,555), n (%)FY2008 (N=100,617), n (%)
    Request a prescription refill53,193 (53)75,241 (75)
    View my VA Appointments38,664 (38)N/A
    Use Secure Messaging to communicate with my VA health care team28,952 (29)N/A
    Track the status of my prescription refill delivery27,516 (27)N/A
    View my medication history23,884 (24)23,923 (24)
    View my lab or other test results19,382 (19)N/A
    Access my VA health records or Blue Button or VA Health Summary11,966 (12)N/A
    View my VA Notes (written by my health care team)11,058 (11)N/A
    Look up information about a health condition or medication9393 (9)N/A
    Learn more about features that are available9149 (9)N/A
    Look up information about a medicationN/A17,899 (18)
    Find information about VA benefits9111 (9)6246 (6)
    Enter or keep track of personal information5695 (6)14,507 (14)
    Other5101 (5)9198 (9)
    Enter or keep track of personal health care information (eg, blood pressure)3202 (3)13,125 (13)
    Use the Veterans Health Library (Research a health condition)2648 (3)6367 (6)
    Enter information about my non-VA medications or supplements2288 (2)N/A
    Find a VA facility1646 (2)2206 (2)
    Complete a HealtheLiving Assessment1533 (2)N/A

    aMultiple categories may be selected.

    ‎
    Figure 3. Open-ended comment clusters for topics of interest.
    View this figure
    ‎
    Figure 4. Historical customer satisfaction trends. CXA: customer experience analytics.
    View this figure
    ‎
    Figure 5. Incremental changes to My HealtheVet home page.
    View this figure
    ‎
    Figure 6. Customer experience analytics customer satisfaction index 15-month trend.
    View this figure

    Principal Findings

    The literature on adoption and use of patient portals highlights the need for health care organizations to employ UCD approaches to ensure that portals align with end users’ characteristics, needs, preferences, and goals, and, ultimately, help advance portal implementation. In its commitment to UCD, one method that VA has used to accomplish the above is a continuous survey to elicit a direct feedback from a random sample of veterans who use VA’s patient portal, My HealtheVet. In combination with other methods, such as targeted research studies, the CXA survey has enabled a deeper understanding of portal users and directly informed changes in portal features, functions, policies, and processes. By incorporating the results of this systematic evaluation of the user experience into the portal redesign, VA aims to continue to enhance the ability of My HealtheVet to engage and activate veterans in managing their health.

    Patient Portal Users

    This study compared the characteristics and behaviors of users during the early period of patient portal implementation, 2008, with that of later adopters. This provided a trajectory of how portal use has evolved over a decade. Many aspects remained stable, while others showed clear trends toward portal adoption by populations believed less likely to use patient-facing health technologies. While only 13.4% (89,780/670,000) portal users in FY2008 were VA patients with a Premium account, by FY2017, this increased to 62.5% (2.5 million/4 million) users. Despite early assumptions about older users not adopting and using patient portals [46,47], leading to a gray digital divide [48], the VA experience reveals an increasingly elderly population of users. Within the veteran population, research has shown that VA patients tend to be older and more socioeconomically disadvantaged than veterans who do not rely on VA for care [49]. Although the survey indicates that the majority of users have one or more chronic health conditions and access the portal with increasing frequency, the survey results also suggest a trend toward those with less internet ability and better health also accessing the patient portal. This trend may be a result of the portal expanding the types of transactional services that users find convenient, based on direct veteran input. It also suggests that the portal is engaging a broader segment of the veteran population. Although the proportion of female veterans responding to the survey decreased slightly in FY2017 (from 9% to 7%), the overall population of female veterans was estimated to be 9.4% in 2015. However, only 22.4% used VA health care services [50], which is a key driver for accessing the patient portal. Portal users in FY2017 also tended to have completed higher levels of education than those in FY2008. This may be reflective of changes in the veteran population overall, with the enhanced provision of educational support programs for separating service members. Given that half of the survey respondents in FY2017 reported that they also use community non-VA care providers, VA will need to continue to develop tools that enable effective information sharing across settings of care. Portal functions that support consumer-mediated health information exchange are currently in early field testing [51]. These patient portal user trends align with similar trends for the VA patient population overall in terms of gender (91% male), age (median age of male VA patients, 64 years), and increasing use of VA education benefits [52].

    Incremental Portal Redesign

    Based on user self-report about goals and tasks, a significant redesign of the website was undertaken to enhance navigation to the features aligned with the most common user tasks and to decrease the number of steps to accomplish these. After an initial period of satisfaction decline, anticipated due to the phenomenon of change aversion [53], the satisfaction index recovered and increased. Once users adjusted to the change, they were more satisfied with the new design as measured using the CXA satisfaction index. Looking ahead, there are additional improvements and enhancements that will be important to address.

    Limitations

    It is important to note that the results of the CXA survey reflect the characteristics and perspectives of a random sample of portal users who are invited and opt to participate in the survey and may not be fully generalizable to the larger population. More broadly, the respondent sample represents patient portal users; therefore, other methods are also needed to elicit input from veterans who are not portal users to understand their characteristics and preferences and identify barriers that may exist to system access and use. VA is in the process of adding questions to its patient experience survey, administered to veterans who had a recent medical encounter, to help fill this gap, and ongoing research about veteran preferences for digital tools and services provides complementary insights [32,33]. There may also be data that were not collected in the survey that could be important. Since the survey is anonymous, there is no opportunity to follow up with respondents for more information or clarification. Despite limitations inherent to an anonymous survey, it has the benefit of enabling a continuous flow of direct feedback. While the findings from our case study may not be fully generalizable to other patient populations, the principle of using agile approaches to employ UCD has potential to be a promising implementation strategy for other health care organizations.

    Conclusions

    By leveraging UCD principles, VA has continued to enhance its patient portal and supported its continued implementation, achieving significant growth in adoption and use over the last decade. While quantitative and qualitative research studies are an important component of patient portal evaluation, more agile methods are also needed to complement formal research efforts. As illustrated through this case study, we have found the ongoing administration of a continuous voluntary Web-based survey as an efficient and effective way to capture veteran’s voices about who they are, how they use the patient portal, what improvements are needed, and what additional services are desired. This approach, together with others intended to explore the perspectives of veterans who are not portal users, will help ensure that VA’s health information technology services are developed and enhanced to optimize the benefits to all VA patients. With impending changes to VA’s EHR platform, capturing veteran’s voices is more crucial than ever. More broadly, developing patient portals as an effective patient engagement strategy will require that UCD principles are employed to foster adoption and sustained use. In an era of finite resources, leveraging the “voice-of-the-customer” techniques helps ensure that the portal continues to meet patients’ needs in ways that enhance full participation in their own health care.

    Acknowledgments

    The views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs. KMN was affiliated with the Department of Veterans Affairs during the time of this study but recently retired and is now an independent consultant specializing in Sociology, Communication, and Consumer Health Informatics.

    Authors' Contributions

    KMN was responsible for data collection. All authors (KMN, CLT, DMK, and TPH) contributed to the drafting of the manuscript and revising critically for intellectual content.

    Conflicts of Interest

    None declared.

    1. Barello S, Graffigna G, Vegni E. Patient engagement as an emerging challenge for healthcare services: mapping the literature. Nurs Res Pract 2012;2012:905934 [FREE Full text] [CrossRef] [Medline]
    2. Gephart S, Effken J. Online Journal of Nursing Informatics (OJNI). 2013. Using Health Information Technology to Engage Patients in their Care   URL: http://ojni.org/issues/?p=2848 [accessed 2018-03-15] [WebCite Cache]
    3. James J, Hibbard J, Agres T. Health Affairs. 2013. Health policy brief: patient engagement   URL: https://www.healthaffairs.org/do/10.1377/hpb20130214.898775/full/ [accessed 2018-06-27] [WebCite Cache]
    4. Wilkins S. KevinMD.com. 2012 Jan 27. Patient engagement is the holy grail of health care   URL: https://www.kevinmd.com/blog/2012/01/patient-engagement-holy-grail-health-care.html [accessed 2018-03-14] [WebCite Cache]
    5. Chase D. Forbes. 2012 Sep. Patient engagement is the blockbuster drug of the century   URL: https:/​/www.​forbes.com/​sites/​davechase/​2012/​09/​09/​patient-engagement-is-the-blockbuster-drug-of-the-century/​ [accessed 2018-03-14] [WebCite Cache]
    6. Irizarry T, Shoemake J, Nilsen ML, Czaja S, Beach S, DeVito DA. Patient Portals as a Tool for Health Care Engagement: A Mixed-Method Study of Older Adults With Varying Levels of Health Literacy and Prior Patient Portal Use. J Med Internet Res 2017 Mar 30;19(3):e99 [FREE Full text] [CrossRef] [Medline]
    7. Nazi K, Hogan T, Woods S. Consumer health informatics: engaging and empowering patients and families. In: Finnell J, Dixon B, editors. Clinical Informatics Study Guide. Switzerland: Springer International Publishing; 2016:459-500.
    8. Ricciardi L, Mostashari F, Murphy J, Daniel JG, Siminerio EP. A national action plan to support consumer engagement via e-health. Health Aff (Millwood) 2013 Feb;32(2):376-384. [CrossRef] [Medline]
    9. Sorondo B, Allen A, Fathima S, Bayleran J, Sabbagh I. Patient Portal as a Tool for Enhancing Patient Experience and Improving Quality of Care in Primary Care Practices. EGEMS (Wash DC) 2016;4(1):1262 [FREE Full text] [CrossRef] [Medline]
    10. Fraccaro P, Vigo M, Balatsoukas P, Buchan IE, Peek N, van DVSN. Patient Portal Adoption Rates: A Systematic Literature Review and Meta-Analysis. Stud Health Technol Inform 2017;245:79-83. [Medline]
    11. Goldzweig CL, Orshansky G, Paige NM, Towfigh AA, Haggstrom DA, Miake-Lye I, et al. Electronic patient portals: evidence on health outcomes, satisfaction, efficiency, and attitudes: a systematic review. Ann Intern Med 2013 Nov 19;159(10):677-687. [CrossRef] [Medline]
    12. Irizarry T, DeVito DA, Curran CR. Patient Portals and Patient Engagement: A State of the Science Review. J Med Internet Res 2015;17(6):e148 [FREE Full text] [CrossRef] [Medline]
    13. Tulu B, Trudel J, Strong DM, Johnson SA, Sundaresan D, Garber L. Patient Portals: An Underused Resource for Improving Patient Engagement. Chest 2016 Jan;149(1):272-277. [CrossRef] [Medline]
    14. Centers for Medicare & Medicaid Services (CMS), HHS. Federal Register. 2015 Oct 16. Medicare and Medicaid programs: electronic health record incentive programs. Final rule   URL: https://www.gpo.gov/fdsys/pkg/FR-2015-10-16/pdf/2015-25595.pdf [accessed 2018-06-28] [WebCite Cache]
    15. Henry SL, Shen E, Ahuja A, Gould MK, Kanter MH. The Online Personal Action Plan: A Tool to Transform Patient-Enabled Preventive and Chronic Care. Am J Prev Med 2016 Jul;51(1):71-77. [CrossRef] [Medline]
    16. Lau M, Campbell H, Tang T, Thompson DJS, Elliott T. Impact of patient use of an online patient portal on diabetes outcomes. Can J Diabetes 2014 Feb;38(1):17-21. [CrossRef] [Medline]
    17. Nazi KM, Hogan TP, McInnes DK, Woods SS, Graham G. Evaluating patient access to Electronic Health Records: results from a survey of veterans. Med Care 2013 Mar;51(3 Suppl 1):S52-S56. [CrossRef] [Medline]
    18. Sarkar U, Lyles CR, Parker MM, Allen J, Nguyen R, Moffet HH, et al. Use of the refill function through an online patient portal is associated with improved adherence to statins in an integrated health system. Med Care 2014 Mar;52(3):194-201 [FREE Full text] [CrossRef] [Medline]
    19. Wade-Vuturo AE, Mayberry LS, Osborn CY. Secure messaging and diabetes management: experiences and perspectives of patient portal users. J Am Med Inform Assoc 2013 May 01;20(3):519-525 [FREE Full text] [CrossRef] [Medline]
    20. OpenNotes.   URL: https://www.opennotes.org/ [accessed 2018-05-12] [WebCite Cache]
    21. Delbanco T, Walker J, Bell SK, Darer JD, Elmore JG, Farag N, et al. Inviting patients to read their doctors' notes: a quasi-experimental study and a look ahead. Ann Intern Med 2012 Oct 02;157(7):461-470 [FREE Full text] [CrossRef] [Medline]
    22. Dobscha SK, Denneson LM, Jacobson LE, Williams HB, Cromer R, Woods S. VA mental health clinician experiences and attitudes toward OpenNotes. Gen Hosp Psychiatry 2016;38:89-93. [CrossRef] [Medline]
    23. Gerard M, Fossa A, Folcarelli PH, Walker J, Bell SK. What Patients Value About Reading Visit Notes: A Qualitative Inquiry of Patient Experiences With Their Health Information. J Med Internet Res 2017 Jul 14;19(7):e237 [FREE Full text] [CrossRef] [Medline]
    24. Leveille SG, Walker J, Ralston JD, Ross SE, Elmore JG, Delbanco T. Evaluating the impact of patients' online access to doctors' visit notes: designing and executing the OpenNotes project. BMC Med Inform Decis Mak 2012 Apr 13;12:32 [FREE Full text] [CrossRef] [Medline]
    25. Wright E, Darer J, Tang X, Thompson J, Tusing L, Fossa A, et al. Sharing Physician Notes Through an Electronic Portal is Associated With Improved Medication Adherence: Quasi-Experimental Study. J Med Internet Res 2015 Oct 08;17(10):e226 [FREE Full text] [CrossRef] [Medline]
    26. Garrido T, Raymon B, Wheatley B. Health Affairs. 2016 Apr. Lessons learned from more than a decade in patient portals   URL: https://www.healthaffairs.org/do/10.1377/hblog20160407.054362/full/ [accessed 2018-06-28] [WebCite Cache]
    27. Veterans and Consumers Health Informatics Office. Department of Veterans Affairs. 2018.   URL: https://connectedcare.va.gov/ [accessed 2018-06-28] [WebCite Cache]
    28. Abras C, Maloney-Krichmar D, Preece J. User-centered design. In: Bainbridge W, editor. Encyclopedia of Human-Computer Interaction. Thousand Oaks: Sage Publications; 2004.
    29. Katz-Haaz R. Ten guidelines for user-centered web design. Usability Interface 1998;5(1):1-3.
    30. Morales RM, Casper G, Brennan PF. Patient-centered design. The potential of user-centered design in personal health records. J AHIMA 2007 Apr;78(4):44-46. [Medline]
    31. Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci 2015 Feb 12;10:21 [FREE Full text] [CrossRef] [Medline]
    32. Haun JN, Chavez M, Nazi K, Antinori N, Melillo C, Cotner BA, et al. Veterans' Preferences for Exchanging Information Using Veterans Affairs Health Information Technologies: Focus Group Results and Modeling Simulations. J Med Internet Res 2017 Oct 23;19(10):e359 [FREE Full text] [CrossRef] [Medline]
    33. Haun JN, Chavez M, Nazi KM, Antinori N. Developing a Health Information Technology Systems Matrix: A Qualitative Participatory Approach. J Med Internet Res 2016 Oct 06;18(10):e266 [FREE Full text] [CrossRef] [Medline]
    34. Hogan TP, Nazi KM, Luger TM, Amante DJ, Smith BM, Barker A, et al. Technology-assisted patient access to clinical information: an evaluation framework for blue button. JMIR Res Protoc 2014;3(1):e18 [FREE Full text] [CrossRef] [Medline]
    35. Nazi K. VAntage Point. 2010 Dec 08. VA’s Blue Button: Empowering People with Their Data   URL: https://www.blogs.va.gov/VAntage/866/vas-blue-button-empowering-people-with-their-data/ [accessed 2018-03-14] [WebCite Cache]
    36. Nazi KM, Turvey CL, Klein DM, Hogan TP, Woods SS. VA OpenNotes: exploring the experiences of early patient adopters with access to clinical notes. J Am Med Inform Assoc 2015 Mar;22(2):380-389. [CrossRef] [Medline]
    37. My HealtheVet.   URL: https://www.myhealth.va.gov [accessed 2018-05-12] [WebCite Cache]
    38. Nazi KM. Veterans' voices: use of the American Customer Satisfaction Index (ACSI) Survey to identify My HealtheVet personal health record users' characteristics, needs, and preferences. J Am Med Inform Assoc 2010;17(2):203-211 [FREE Full text] [CrossRef] [Medline]
    39. Wood FB, Siegel ER, Feldman S, Love CB, Rodrigues D, Malamud M, et al. Web evaluation at the US National Institutes of Health: use of the American Customer Satisfaction Index online customer survey. J Med Internet Res 2008 Feb 15;10(1):e4 [FREE Full text] [CrossRef] [Medline]
    40. Fornell C, Johnson MD, Anderson EW, Cha J, Bryant BE. Cornell University. 1996 Oct. The American Customer Satisfaction Index: nature, purpose, and findings   URL: https://scholarship.sha.cornell.edu/cgi/viewcontent.cgi?article=1427&context=articles [accessed 2018-03-14] [WebCite Cache]
    41. Corbin J, Strauss A. Techniques and Procedures for Developing Grounded Theory. In: Basics of Qualitative Research. Thousand Oaks: Sage Publications; 2008.
    42. Green J, Thorogood N. Qualitative Methods for Health Research. Thousand Oaks: Sage Publications; 2009.
    43. Zulman DM, Nazi KM, Turvey CL, Wagner TH, Woods SS, An LC. Patient interest in sharing personal health record information: a web-based survey. Ann Intern Med 2011 Dec 20;155(12):805-810. [CrossRef] [Medline]
    44. Turvey C, Zulman D, Nazi K, Wakefield B, Woods S, Hogan T, et al. Transfer of information from personal health records: a survey of veterans using My HealtheVet. Telemed J E Health 2012 Mar;18(2):109-114. [CrossRef] [Medline]
    45. Turvey C, Klein D, Fix G, Hogan TP, Woods S, Simon SR, et al. Blue Button use by patients to access and share health record information using the Department of Veterans Affairs' online patient portal. J Am Med Inform Assoc 2014;21(4):657-663. [CrossRef] [Medline]
    46. Smith SG, O'Conor R, Aitken W, Curtis LM, Wolf MS, Goel MS. Disparities in registration and use of an online patient portal among older adults: findings from the LitCog cohort. J Am Med Inform Assoc 2015 Jul;22(4):888-895 [FREE Full text] [CrossRef] [Medline]
    47. Graetz I, Gordon N, Fung V, Hamity C, Reed ME. The Digital Divide and Patient Portals: Internet Access Explained Differences in Patient Portal Use for Secure Messaging by Age, Race, and Income. Med Care 2016 Dec;54(8):772-779. [CrossRef] [Medline]
    48. Morris A. E-literacy and the grey digital divide: a review with recommendations. JIL 2007 Dec 13;1(3):13-28. [CrossRef]
    49. RAND Corporation. Centers for Medicare and Medicaid (CMS). 2015 Sep 01. A product of the CMS Alliance to modernize healthcare federally funded research and development center   URL: https://www.va.gov/opa/choiceact/documents/assessments/Assessment_A_Demographics.pdf [accessed 2018-03-14] [WebCite Cache]
    50. National Center for Veterans Analysis and Statistics. Department of Veterans Affairs. 2017. Women Veterans Report: the past, present, and future of Women Veterans   URL: https://www.va.gov/vetdata/docs/SpecialReports/Women_Veterans_2015_Final.pdf [accessed 2018-03-14] [WebCite Cache]
    51. Williams C, Mostashari F, Mertz K, Hogin E, Atwal P. From the Office of the National Coordinator: the strategy for advancing the exchange of health information. Health Aff (Millwood) 2012 Mar;31(3):527-536 [FREE Full text] [CrossRef] [Medline]
    52. National Center for Veterans Analysis and Statistics. Department of Veterans Affairs. 2017. VA Utilization Profile   URL: https://www.va.gov/vetdata/docs/Quickfacts/VA_Utilization_Profile.pdf [accessed 2018-06-28] [WebCite Cache]
    53. Sedley A. GV Library. 2012 Apr 23. Change aversion: why users hate what you launched (and what to do about it)   URL: https:/​/library.​gv.com/​change-aversion-why-users-hate-what-you-launched-and-what-to-do-about-it-2fb94ce65766 [accessed 2018-03-14] [WebCite Cache]

    ‎
    CXA: customer experience analytics
    EHR: electronic health record
    KP: Kaiser Permanente
    UCD: user-centered design
    VA: Department of Veterans Affairs

    Edited by G Eysenbach; submitted 15.03.18; peer-reviewed by B Tulu, L Garvin, T Irizarry; comments to author 05.04.18; revised version received 14.05.18; accepted 16.06.18; published 10.07.18

    Copyright

    ©Kim M Nazi, Carolyn L Turvey, Dawn M Klein, Timothy P Hogan. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 10.07.2018.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.