Published on in Vol 19, No 5 (2017): May

Prioritizing Measures of Digital Patient Engagement: A Delphi Expert Panel Study

Prioritizing Measures of Digital Patient Engagement: A Delphi Expert Panel Study

Prioritizing Measures of Digital Patient Engagement: A Delphi Expert Panel Study

Authors of this article:

Lynn A Garvin1 Author Orcid Image ;   Steven R Simon2, 3, 4 Author Orcid Image

Original Paper

1Department of Health Law, Policy and Management, School of Public Health, Boston University, Boston, MA, United States

2Geriatrics and Extended Care Service, Veterans Affairs Boston Healthcare System, Boston, MA, United States

3The Center for Healthcare Organization and Implementation Research, Veterans Affairs Boston Healthcare System, Boston, MA, United States

4Division of General Medicine and Primary Care, Brigham and Women's Hospital, Boston, MA, United States

*all authors contributed equally

Corresponding Author:

Lynn A Garvin, MBA, PhD

Department of Health Law, Policy and Management

School of Public Health

Boston University

Talbot 342 W

715 Albany Street

Boston, MA, 02118

United States

Phone: 1 617 414 1327


Background: Establishing a validated scale of patient engagement through use of information technology (ie, digital patient engagement) is the first step to understanding its role in health and health care quality, outcomes, and efficient implementation by health care providers and systems.

Objective: The aim of this study was to develop and prioritize measures of digital patient engagement based on patients’ use of the US Department of Veterans Affairs (VA)’s MyHealtheVet (MHV) portal, focusing on the MHV/Blue Button and Secure Messaging functions.

Methods: We aligned two models from the information systems and organizational behavior literatures to create a theory-based model of digital patient engagement. On the basis of this model, we conducted ten key informant interviews to identify potential measures from existing VA studies and consolidated the measures. We then conducted three rounds of modified Delphi rating by 12 national eHealth experts via Web-based surveys to prioritize the measures.

Results: All 12 experts completed the study’s three rounds of modified Delphi ratings, resulting in two sets of final candidate measures representing digital patient engagement for Secure Messaging (58 measures) and MHV/Blue Button (71 measures). These measure sets map to Donabedian’s three types of quality measures: (1) antecedents (eg, patient demographics); (2) processes (eg, a novel measure of Web-based care quality); and (3) outcomes (eg, patient engagement).

Conclusions: This national expert panel study using a modified Delphi technique prioritized candidate measures to assess digital patient engagement through patients’ use of VA’s My HealtheVet portal. The process yielded two robust measures sets prepared for future piloting and validation in surveys among Veterans.

J Med Internet Res 2017;19(5):e182



Patient portals are Web-based platforms that provide patients with access to health information and elements of their medical record and equip them with tools to interact with their clinical teams [1]. Numerous studies have assessed the adoption of patient portals, including the factors that predispose patients to adoption as well as the barriers and challenges they face [2-8]. Moving beyond adoption, measuring the nature and extent of patients’ use of portal tools is a priority for the US Department of Veterans Affairs (VA) and other health systems. This emerging focus on measuring digital patient engagement [9] follows organically from—and is inextricably linked with—measuring clinician meaningful use of health information technology (HIT) as supported by the US Centers for Medicare & Medicaid Services’s “Meaningful Use” rules [10]. However, to date, neither VA nor any other US health system has established nationally validated measures of digital patient engagement to provide an indication of the extent to which patients are genuinely using portal-based tools and the degree to which those tools are engaging patients with their health care. The main objective of this study was therefore to develop and prioritize measures of patients’ experience using functions of a patient portal, with specific attention to the degree to which their experience using those functions promotes engagement with their health care team. To achieve this objective, we used a modified Delphi panel. This systematic approach aggregated experts’ opinions and perceptions of which measures would be most valuable and appropriate for assessing Veterans’ use of Blue Button and Secure Messaging, two salient features of My HealtheVet, VA’s patient portal. [11]. Our study focused on the Blue Button function, which allows Veterans to download their personal health record, and the Secure Messaging function, which enables Veterans to communicate via secure email with their health care team.

This study involved a sequence of three phases: (1) literature review; (2) key informant interviews; and (3) Delphi panel process. For the literature review, we sought to identify prior work that would enable us to design a theory-based model for the study; as such, we researched three literature streams of established frameworks and validated scales of patients': (1) health and health care, (2) use of IT or HIT, and (3) relations with providers and health care systems. This literature review indicated that existing models of patient engagement and of technology adoption did not sufficiently overlap or integrate with each other to provide a framework for measuring patients’ engagement in their health and health care through technology.

We therefore defined digital patient engagement as the value that a patient (or family member or caregiver) assigns to the accrued experience with and results of using a system feature or service plus the expectation of similar future experience and results. On the basis of our theory-based definition, we aligned two established models: (1) Technology Acceptance Model [12] to reflect the functional dimension and (2) Relational Coordination Model to reflect the patient-provider dimension of digital patient engagement [13]. The resulting model included both patient engagement and intent to use and recommend the technology as outcomes reflecting digital patient engagement.

Our resulting theory-based model is shown in Figure 1. The “account type” mentioned in the figure refers to the type of My HealtheVet account the user possesses. Account types included basic, which provided online access to general health information; advanced, which included access to the Blue Button and the ability to view other elements of the personal health record; and premium, which included all of the advanced features but added secure messaging capability.

Figure 1. Digital patient engagement model.
View this figure

For the second phase of the study, in late 2012, we conducted semistructured key informant telephone interviews with principal investigators of all current VA-funded MyHealtheVet (MHV) studies (N=10) to identify existing measures of MHV adoption and use. The ten participants were identified through communication with the MHV Program Office and with researchers in the field. One or both of the authors conducted telephone interviews with each participant.

Following interviews, we conducted the third phase of the study, a modified Delphi process. We followed the methods of previously published studies [14-21] and tailored them to the objectives herein. The modified Delphi technique is an intensive, iterative approach to elicit and refine experts’ opinions on novel conceptual fields with the goals of gaining consensus on candidate measures and evolving the framework. From the literature review and in-depth interviews, we generated a set of candidate measures to be considered by the Delphi panel, with the goal of identifying measures of use of the Blue Button and Secure Messaging that represented digital patient engagement. On the basis of Delphi theory, our in-depth interviews, and the theoretical model, we established success criteria to guide inclusion in the preliminary measures, process goals, and consensus criteria for each round of the Delphi process.

For the Delphi panel, we convened 12 national (US) eHealth experts, who were principally physicians. Our Delphi protocol, conducted in March-October 2013, involved three rounds of panelists’ independent rating of the measures; panelists submitted their ratings through a secure online questionnaire, enabling the research team to score and analyze the results while maintaining panelist anonymity to all but the researchers. To ensure that the process would ultimately yield measures of digital patient engagement, and following procedures established in prior Delphi panel studies, we asked panelists to rate the importance of each proposed measure on an 11-point integer scale, ranging from −5 (strongly disagree) to +5 (strongly agree). Our objective criteria enabled us to accept or reject a measure after each round, or to revise it for retesting in the next round. For acceptance, a measure was required to meet all three of the following conditions: (1) median score ≥+3; (2) interquartile range (IQR) ≤2; and (3) ≤1 outliers (defined as a score of >1.5×IQR from the 25th or 75th percentile). For example, consider the following 12 panelists’ scores for one measure: 0, 0, 1, 2, 3, 3, 3, 3, 4, 4, 4, 5. The median score is 3, satisfying condition (1). The IQR is 2, satisfying condition (2). There are no outliers, that is, no scores lower than −1 and no scores greater than 7 (the latter not being a possible value, given the −5 to +5 scale), thus satisfying condition (3). Therefore, this measure would be accepted and not considered further in subsequent panel iterations. For revision and retesting (in a subsequent round), a measure was required to meet two of the three conditions. If it failed to meet at least two conditions, it was rejected.

All 12 Delphi expert panel members completed the study’s three rounds of measures rating. The final candidate measures of digital patient engagement comprised two similar but separate sets: 58 measures for Secure Messaging and 71 measures for MHV/Blue Button, where Antecedents represented 20 comparable measures for both functions, Processes represented 32 Secure Messaging and 45 MHV/Blue Button measures, and Outcomes represented six comparable measures for Secure Messaging and MHV/Blue Button.

As an example of how the Delphi panel results were used to include or exclude measures, Table 1 shows the four digital patient engagement outcome measures. For each of these four outcome measures, acceptance was achieved in the third and final round of the Panel’s deliberations. Among the four measures, there was only one outlier panel member rating, reflecting high level of agreement on the value of these items in measuring digital patient engagement.

Table 1. Digital patient engagement outcome measures—Delphi panel statistics. Secure Messaging = first statistic and MyHealtheVet /Blue Button = second statistic, reported (in #/# format).
Measure: patient engagementAccept roundIQRaOutlierMedianMeanSDb
I have all the information I need to manage my health and health care.31.5/2.01/03.5/4.03.1/4.02.3/0.9
I am confident in working with my VAchealth care team to manage my health and health care.32.0/1.50/03.0/3.03.7/3.61.0/0.8
I feel in control of my health and health care (such as taking part in decisions or following through on any medication, treatment, or health routine).31.5/2.00/04.0/4.03.5/3.61.4/1.5
I am able to achieve my long-term health and health care goals (such as being self-reliant, living longer and better, or knowing that my family and friends can depend on me).32.0/2.00/03.5/3.53.5/3.61.5/1.6

aIQR: interquartile range.

bSD: standard deviation.

cVA: US Department of Veterans Affairs.

Principal Findings

Measuring how patients use HIT is a high priority for US health care. Nevertheless, existing meaningful use measurements have focused on clinicians’ use of technology with few guidelines for patient adoption and use. While various scales have emerged to assess patient engagement and satisfaction with health care, none has combined patients’ affinity for the technology with patients’ trust in their relationship with clinicians, in person and online, to demonstrate how these variables influence digital patient engagement with their health and health care.

In this national expert panel study using a modified Delphi technique, we consolidated and refined two complementary versions of candidate measures to assess patients’ use of VA's My HealtheVet patient portal, one set of measures for its Secure Messaging feature and another for its Blue Button personal health record and other MHV tools, with the potential for gauging digital patient engagement.

This study offers a number of strengths and innovations. Guided by a theory-based framework, we first developed and refined a new four-item digital patient engagement outcome measure based on Rogers’ “Diffusion of Innovation” model [22]. That is, the four new measures we developed (ie, awareness and understanding, skills and confidence, trial and regular use, and use loyalty and recommendation) map roughly to Rogers’ four stages of diffusion of innovation, namely knowledge, persuasion, decision, and confirmation. Grounding our measures in Rogers’ framework was suggested by and affirmed by the Delphi panel. The four-item digital patient engagement measures also align with Hibbard et al’s patient activation measures [23], paraphrased as (1) belief in having an active role in care, (2) confidence and knowledge to take action, (3) taking action, and (4) staying the course under stress.

As an innovation, the Delphi panel led us to introduce the novel process dimension of patient online care quality. This measure reflects the quality of the interaction of users with the technology.

We strengthened the content validity of the measures, a principal goal of using the Delphi technique, by assuring 100% participation of our content experts across the three rating rounds. To mitigate threats to external validity, such as selection bias, we selected our national experts to reflect a broad perspective of various health systems, diverse patient user groups, and an array of patient portal architectures and features. Generalizability of results also benefited from the inclusion of the expert views of the key informants. We reduced Delphi panel process threats by (1) making study goals and procedural guidelines clear at the start, (2) presenting a fair and transparent rating process with timely survey administration and response to panelist questions, and (3) extending full consideration and discussion on any dissenting opinions by panelists.


A potential limitation of the study is that we employed a panel of experts, rather than patients themselves, to refine and prioritize the measures for digital patient engagement. We chose our approach because we considered the tasks of measure selection to require not only familiarity with the patient portal tools and their role in health care delivery but also a comfort level with the process of questionnaire item development and measurement scales. To ensure that the measures developed in this study truly reflect digital patient engagement, they must be validated among a population of patients who are users of the portal and its functions.


Establishing a valid and reliable scale is the first step to measuring digital patient engagement and its role in health and health care quality, outcomes, and effective, efficient implementation by health care providers and health care systems. This study yielded a robust set of candidate measures of what Veterans value in Blue Button and Secure Messaging. These measures and the scales they constitute can thus be tested empirically to examine their psychometric properties and may ultimately be used in measuring the extent to which patient portals and other patient-facing technologies can engage patients in their health care.

Conflicts of Interest

None declared.

  1. HealthIT. What is a patient portal?   URL: [accessed 2017-05-03] [WebCite Cache]
  2. Agarwal R, Anderson C, Zarate J, Ward C. If we offer it, will they accept? factors affecting patient use intentions of personal health records and secure messaging. J Med Internet Res 2013;15(2):e43 [FREE Full text] [CrossRef] [Medline]
  3. Irizarry T, DeVito DA, Curran CR. Patient portals and patient engagement: a state of the science review. J Med Internet Res 2015;17(6):e148 [FREE Full text] [CrossRef] [Medline]
  4. Kruse CS, Argueta DA, Lopez L, Nair A. Patient and provider attitudes toward the use of patient portals for the management of chronic disease: a systematic review. J Med Internet Res 2015;17(2):e40 [FREE Full text] [CrossRef] [Medline]
  5. Ryan BL, Brown JB, Terry A, Cejic S, Stewart M, Thind A. Implementing and using a patient portal: a qualitative exploration of patient and provider perspectives on engaging patients. J Innov Health Inform 2016 Jul 04;23(2):848 [FREE Full text] [Medline]
  6. Fix GM, Hogan TP, Amante DJ, McInnes DK, Nazi KM, Simon SR. Encouraging patient portal use in the patient-centered medical home: three stakeholder perspectives. J Med Internet Res 2016 Nov 22;18(11):e308 [FREE Full text] [CrossRef] [Medline]
  7. Tavares J, Oliveira T. Electronic health record patient portal adoption by health care consumers: an acceptance model and survey. J Med Internet Res 2016 Mar 02;18(3):e49 [FREE Full text] [CrossRef] [Medline]
  8. Mishuris RG, Stewart M, Fix GM, Marcello T, McInnes DK, Hogan TP, et al. Barriers to patient portal access among veterans receiving home-based primary care: a qualitative study. Health Expect 2015 Dec;18(6):2296-2305. [CrossRef] [Medline]
  9. Dolan B. Mobihealthnews. In-Depth: provider adoption of digital patient engagement tools   URL: [accessed 2017-05-03] [WebCite Cache]
  10. Kruse CS, Bolton K, Freriks G. The effect of patient portals on quality outcomes and its implications to meaningful use: a systematic review. J Med Internet Res 2015;17(2):e44 [FREE Full text] [CrossRef] [Medline]
  11. Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review. PLoS One 2011;6(6):e20476 [FREE Full text] [CrossRef] [Medline]
  12. Venkatesh V, Bala H. Technology acceptance model 3 and a research agenda on interventions. Decis Sci 2008 May;39(2):273-315. [CrossRef]
  13. Gittell JH. New Directions for Relational Coordination Theory. Oxford: Oxford University Press; 2011.
  14. Dalkey N, Helmer O. An experimental application of the DELPHI method to the use of experts. Manag Sci 1963 Apr;9(3):458-467. [CrossRef]
  15. Sackman H. Delphi Technique: Expert Opinion, Forecasting and Group Process. Lexington, MA: Lexington Books; 1975.
  16. Lindsay P, Schull M, Bronskill S, Anderson G. The development of indicators to measure the quality of clinical care in emergency departments following a modified-delphi approach. Acad Emerg Med 2002 Nov;9(11):1131-1139 [FREE Full text] [Medline]
  17. Okoli C, Pawlowski SD. The Delphi method as a research tool: an example, design considerations and applications. Information & Management 2004 Dec;42(1):15-29. [CrossRef]
  18. Akins RB, Tolson H, Cole BR. Stability of response characteristics of a Delphi panel: application of bootstrap data expansion. BMC Med Res Methodol 2005 Dec 01;5:37 [FREE Full text] [CrossRef] [Medline]
  19. Deshpande AM, Shiffman RN, Nadkarni PM. Metadata-driven Delphi rating on the Internet. Comput Methods Programs Biomed 2005 Jan;77(1):49-56. [CrossRef] [Medline]
  20. Landeta J. Current validity of the Delphi method in social sciences. Technol Forecast Soc Change 2006;73(5):467-482.
  21. Schneider F, van Osch L, de Vries H. Identifying factors for optimal development of health-related websites: a Delphi study among experts and potential future users. J Med Internet Res 2012 Feb 14;14(1):e18 [FREE Full text] [CrossRef] [Medline]
  22. Rogers EM. Diffusion of Innovation Theory. Boston, MA: The Free Press; 1995.
  23. Hibbard JH, Stockard J, Mahoney ER, Tusler M. Development of the Patient Activation Measure (PAM): conceptualizing and measuring activation in patients and consumers. Health Serv Res 2004 Aug;39(4 Pt 1):1005-1026 [FREE Full text] [CrossRef] [Medline]

HIT: health information technology
IQR: Interquartile range
MHV: MyHealtheVet
VA: US Department of Veterans Affairs

Edited by G Eysenbach; submitted 30.05.15; peer-reviewed by D Sands, L Laranjo, S Baker; comments to author 19.10.15; revised version received 01.03.17; accepted 18.03.17; published 26.05.17


©Lynn A Garvin, Steven R Simon. Originally published in the Journal of Medical Internet Research (, 26.05.2017.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.