Original Paper
Abstract
Background: Poor discharge preparation during hospitalization may lead to adverse events after discharge. Checklists and videos that systematically engage patients in preparing for discharge have the potential to improve safety, especially when integrated into clinician workflow via the electronic health record (EHR).
Objective: This study aims to evaluate the implementation of a suite of digital health tools integrated with the EHR to engage hospitalized patients, caregivers, and their care team in preparing for discharge.
Methods: We used the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework to identify pertinent research questions related to implementation. We iteratively refined patient and clinician-facing intervention components using a participatory process involving end users and institutional stakeholders. The intervention was implemented at a large academic medical center from December 2017 to July 2018. Patients who agreed to participate were coached to watch a discharge video, complete a checklist assessing discharge readiness, and request postdischarge text messaging with a physician 24 to 48 hours before their expected discharge date, which was displayed via a patient portal and bedside display. Clinicians could view concerns reported by patients based on their checklist responses in real time via a safety dashboard integrated with the EHR and choose to open a secure messaging thread with the patient for up to 7 days after discharge. We used mixed methods to evaluate our implementation experience.
Results: Of 752 patient admissions, 510 (67.8%) patients or caregivers participated: 416 (55.3%) watched the video and completed the checklist, and 94 (12.5%) completed the checklist alone. On average, 4.24 concerns were reported per each of the 510 checklist submissions, most commonly about medications (664/2164, 30.7%) and follow-up (656/2164, 30.3%). Of the 510 completed checklists, a member of the care team accessed the safety dashboard to view 210 (41.2%) patient-reported concerns. For 422 patient admissions where postdischarge messaging was available, 141 (33.4%) patients requested this service; of these, a physician initiated secure messaging for 3 (2.1%) discharges. Most patient survey participants perceived that the intervention promoted self-management and communication with their care team. Patient interview participants endorsed gaps in communication with their care team and thought that the video and checklist would be useful closer toward discharge. Clinicians participating in focus groups perceived the value for patients but suggested that low awareness and variable workflow regarding the intervention, lack of technical optimization, and inconsistent clinician leadership limited the use of clinician-facing components.
Conclusions: A suite of EHR-integrated digital health tools to engage patients, caregivers, and clinicians in discharge preparation during hospitalization was feasible, acceptable, and valuable; however, important challenges were identified during implementation. We offer strategies to address implementation barriers and promote adoption of these tools.
Trial Registration: ClinicalTrials.gov NCT03116074; https://clinicaltrials.gov/ct2/show/NCT03116074.
doi:10.2196/15573
Keywords
Introduction
The transition from hospital to ambulatory care is a vulnerable time for patients and stressful for their caregivers: new treatments have been initiated, conditions require monitoring, and the plan is in flux. Approximately 19% to 28% of patients experience preventable adverse events after discharge, many due to suboptimal monitoring of conditions, medication errors or nonadherence, and failure to execute the recovery plan [
- ]. During hospitalization, discharge planning is often initiated late, and input from patients regarding their preparedness is frequently lacking, which may lead to delays and dissatisfaction [ ]. After discharge, patients report problems related to follow-up, medications, and self-care; have unanswered questions that could have easily been addressed before discharge [ ]; and often feel more relieved than burdened when readmitted [ ]. Lack of patient engagement during the process of discharge preparation may contribute to avoidable adverse events and costly readmissions [ ], particularly those that occur early after hospitalization [ , ].To date, efforts to enhance and standardize discharge practices have typically targeted clinicians [
, ]; interventions directed at patients provide an opportunity to improve patient understanding, self-management, and postdischarge outcomes [ ]. National agencies (eg, Agency for Healthcare Research and Quality [AHRQ], Centers for Medicare and Medicaid Services) are attempting to engage patients and caregivers more broadly by offering access to discharge preparation materials that include checklists for patients [ , ]. Few institutions have determined how best to operationalize these tools for patients. Digital health technology could be leveraged to more proactively engage patients, caregivers, and clinicians during the process of discharge preparation [ - ]; however, currently available patient-facing digital health tools such as patient portals have gaps in functionality with regard to assessing discharge readiness, are not well integrated with the electronic health record (EHR), and present challenges when used during hospitalization [ , - ]. Although it is technically feasible to administer a discharge checklist through a patient portal or mobile device [ , ], hospitals lack knowledge about the potential for adoption and perceived utility of these tools for patients and clinicians in a real-world clinical setting, as well as potential barriers for sustaining the intervention from an organizational perspective.To address this knowledge gap, we designed and developed an interactive patient-centered discharge toolkit (PDTK), a suite of EHR-integrated digital health tools that enabled patients to self-assess and communicate discharge preparedness to their care team and request secure text messaging with a hospital physician after discharge as part of a project funded by the AHRQ. Guided by the Reach, Effectiveness, Adoption, Implementation and Maintenance (RE-AIM) framework, we conducted a mixed methods study to describe the use, adoption, and perceived utility of the PDTK, as well as the key challenges encountered during implementation [
].Methods
Overview
We used RE-AIM (
), a framework designed to address issues related to the implementation of health services and clinical informatics research [ ], to identify research questions to evaluate the feasibility and acceptability of the PDTK ( ) in a real-world clinical setting. Specifically, we employed a variety of quantitative and qualitative methods to assess the Reach and potential for Adoption, while identifying barriers to Implementation and strategies to Maintain the PDTK for hospitalized general medicine patients. Effectiveness of the intervention on outcomes will be evaluated in future studies.Dimension | Methods | Results | |
Reach |
|
|
|
|
|
| |
Effectiveness |
|
|
|
|
|
| |
|
|
| |
Adoption |
|
| |
Implementation |
|
| |
|
|
| |
Maintenance |
|
|
aPDTK: patient-centered discharge toolkit.
Setting and Participants
The PDTK study (Clinicaltrials.gov NCT03116074) was approved by the Partners’ institutional review board and was conducted at Brigham and Women’s Hospital in Boston, MA. The study was conducted on three 30-bed general medicine units from December 2017 through July 2018 in parallel with our AHRQ-funded Patient Safety Learning Laboratory (PSLL; Clinicaltrials.gov NCT02969343) reported elsewhere [
- ]. As part of the PSLL, we integrated a bedside display for patients and clinicians, a patient portal for patients and caregivers, and a safety dashboard for clinicians into our EHR environment (Epic Systems, Inc, Verona, WI) [ , ]. These applications used enterprise data services to obtain clinical data from the EHR in real time [ , - ]. This EHR-integrated digital health infrastructure ( ) served as a platform on which to incorporate enhancements for the independently funded PDTK study. The key focus of the PSLL was to use systems engineering and human factors methods to design, develop, and implement tools to prevent harm in the hospital (eg, falls, catheter-associated urinary tract infections) [ ]. The goal of the PDTK study was to improve safety during transitions out of the hospital by designing, developing, and implementing enhancements to the PSLL infrastructure ( ) based on clinician end-user requirements and organizational priorities.The PDTK comprised enhancements to each of the 3 components of the PSLL technical infrastructure: bedside display, patient portal, and safety dashboard (see
).Each of the 3 study units was codirected by a physician and nurse pair and staffed by its own group of nurses. Patients admitted to each unit were cared for by 1 of 2 geographically localized general medicine teams comprising residents or physician assistants and a supervising hospitalist attending physician [
]. These medical teams rotated approximately every 2 weeks. A few off-service patients (ie, admitted to a service other than general medicine) were admitted to these units under the care of a different attending physician.Any English- and Spanish-speaking patient admitted to these units under the general medicine service was eligible to participate by using any of the patient-facing PDTK components (checklist, video, and secure messaging). For patients who did not have the capacity to consent (as determined by a member of the care team), a caregiver (a designated health care proxy) could participate on their behalf. Patient or caregiver participants were offered access to an acute care patient portal on either personal devices or study-issued mobile devices (iPad Air, Apple, Inc, Cupertino, CA) as part of the concurrent PSLL study [
]. Any clinician (nurse, resident, physician assistant, and attending) caring for a general medicine patient admitted to these units was eligible to participate: all clinicians had access to EHR-integrated digital health infrastructure as part of the concurrent PSLL study and could therefore access the clinician-facing PDTK components (safety dashboard and secure messaging).Component | Description |
EDDa display |
|
Discharge video |
|
Discharge checklist |
|
Clinician dashboard discharge column |
|
Secure messaging postdischarge |
|
aEDD: expected discharge date.
bEHR: electronic health record.
cAPI: application programming interface.
dHIPAA: Health Insurance Portability and Accountability Act.
Iterative Refinement of Intervention Components
In previous work, we engaged patient advisors, clinical stakeholders, information system professionals, and quality and safety leaders to identify gaps in discharge processes [
, ]. For this study, during the design and development phase, we conducted informal workflow observations on study units and interviews with stakeholders to identify end-user requirements for addressing these gaps by engaging patients and clinicians in discharge preparation while aligning with key organizational priorities: engaging patients to improve patient satisfaction, improving expected discharge date (EDD) documentation in the EHR, and reducing 30-day hospital readmissions [ ]. For example, improving EDD accuracy—defined by our institution as the percent of final EDD entries equal to the actual discharge date (ADD)—was an organizational priority for improving operational throughput. Thus, to ensure timeliness of checklist completion and review of checklist responses by the care team, we enhanced the EHR-integrated patient portal, bedside display, and safety dashboard to improve the visibility of the EDD for both patients and clinicians ( ). We presumed that the likelihood of checklist submission by patients and review by clinicians would be dependent on where patients were in their hospital course as well as their currently documented EDD.As in the concurrent PSLL study [
, , ], we applied user-centered design principles to refine patient- and clinician-facing intervention components ( ) to ensure that we addressed end-user needs [ ]. For the discharge checklist ( , left), our goal was to improve structure and organization, validate content, and clarify wording and utility. Key refinements were identified through multiple iterations of the original checklist within our research team (in part based on our experience with a transitions study funded by the Patient-Centered Outcomes Research Institute [ ]), 2 sessions with our hospital’s patient and family advisory council, and a short pilot in which we administered a paper-based prototype to a convenience sample of 10 hospitalized patients and requested feedback. On the basis of the feedback from unit nurses and patient advisors, we also created a video to help hospitalized patients understand the purpose of completing the checklist to prepare for discharge. To develop the discharge video ( , right), we adapted a method previously demonstrated to improve patients’ understanding of their medical condition and care plan [ ]. Finally, we determined that patients would need to watch the video and complete the checklist via one of several workflows: using the patient portal on a hospital-issued mobile device, using their own mobile device, or having research staff coach patients or caregivers to complete the checklist and then submit responses on their behalf.Next, we made discharge process–based enhancements to the PSLL technical infrastructure integrated with our EHR to ensure that patient-reported information from the checklist would be communicated to the care team as the EDD approached. On the basis of the feedback from patients and clinical unit leadership, we confirmed that patients would want their care team to have access to the checklist responses sufficiently before actual discharge to allow time for the care team to review and address any issues (see Enrollment below). Thus, we developed a checklist submission and review process (
) to ensure that checklist responses submitted by patients would be visible for clinicians to review in the EHR in real time.The discharge checklist (
, left), originally created by Coleman [ ], was adapted for our institution [ ], and further refined into a four-domain, 16-item discharge checklist. Questions were simplified, reordered, and separated into four domains: My Understanding, My Medications, My Self-Care Management, and My Follow-up. Dichotomous checklist responses (Yes/No) were determined to be too strict (ie, patients were uncertain of their response and did not want to check yes or no); thus, a third option (Unsure) and a box for a free-text response was added. A caregiver version to be completed by health care proxies, as well as a Spanish version (approved by our hospital’s interpreter services) was created to make the checklist more inclusive of patients who lacked capacity or who did not speak English as their primary language ( ).The discharge video (
, right) was developed by creating an English version of a script in which a clinician guided the viewer through each checklist domain. This script was translated into Spanish and approved by our hospital’s interpreter services. A mobile device (iPad Air, Apple, Inc) was used to film English and Spanish versions. Video editing software (iMovie, Apple, Inc) was used to produce the videos. The videos were uploaded onto a video-hosting site (YouTube, LLC); hyperlinks to these videos were incorporated into the patient portal and REDCap (Research Electronic Data Capture, Nashville, TN; see Checklist Submission section).Checklist Submission
The two options available for patients to complete and submit the checklist were as follows: the patient portal or a Web-based REDCap survey [
, ]. REDCap submissions could be completed by patients on a mobile device (via a hyperlink emailed to the patient), or by study staff who would submit the responses on their behalf. Checklist responses submitted via the patient portal were made visible in the safety dashboard via a shared database. Checklist responses submitted via REDCap were routed to the safety dashboard using the REDCap application programming interface and matched to the corresponding patient using key identifiers (medical record number and admission date). In either case, checklist responses were displayed on the safety dashboard for clinicians within 5 min of submission.Checklist Review
A new discharge column in the safety dashboard displayed key data elements (medical and nonmedical barriers, EDD) from the EHR (Epic Systems, Inc, Verona, WI) using enterprise Web services [
]. EHR data were transformed into clinical decision support using color-coded flags: red=action needed; yellow=risky state; green=guideline compliant; gray=not applicable. For example, a red flag appeared if the current date was past the EDD, no EDD was entered, or the current date was within 24 hours of the EDD, but no checklist had been submitted. A checklist icon appeared on the safety dashboard on successful submission. Checklist items with a no or unsure response were briefly summarized as a keyword on the patient-detail view of the safety dashboard (eg, Meds: Access/Adherence) with a hover-over displaying the specific item answered by the patient (eg, I understand how to get my medications and will take them as prescribed after I leave the hospital). Patients were permitted to enter their mobile phone number during checklist submission to request secure messaging (Imprivata, Lexington, MA) with a hospital physician (attending or senior resident) for up to 7 days after discharge [ , , ]. A link appeared in the safety dashboard for the clinician to initiate a message thread with the patient.Patient and Caregiver Enrollment
The intervention went live in December 2017, starting with a 1-month wash-in period in which we debugged various technical components. During business days, recruitment lists were created in which the care unit and patient approach order were randomized to minimize confounding (eg, always starting enrollment on the top floor of the hospital and ending at the bottom). All patients admitted to intervention units for at least 24 hours with EDDs within 24-48 hours were eligible. Research assistants asked the nurses caring for these patients if they were appropriate to approach, temporarily excluding any patient who was nonverbal, incapacitated, or behaviorally not safe. When an eligible patient was confirmed as not capable of participating, the research assistant attempted to identify a caregiver (a designated health care proxy). Patients and caregivers who were not available for recruitment (eg, off unit getting a test) were reapproached later that day or on subsequent days if they still met eligibility criteria. Research assistants then asked eligible patients (or caregivers) to watch the video and complete the checklist (including entering their mobile phone number to request postdischarge messaging) via the patient portal or REDCap workflow. A patient was considered enrolled on successful submission of the checklist.
Measurements and Data Collection
We used mixed methods to evaluate our implementation experience per each RE-AIM dimension (
). Specifically, we used quantitative methods to measure usage and perceived utility of intervention components (Reach, Adoption, and Implementation). We used qualitative methods to assess barriers to and facilitators of Implementation and Maintenance from the patient and clinician perspective.Usage of Intervention Components (Quantitative)
We captured the number of times the video was watched, the checklist was submitted, and postdischarge messaging was requested by patients. We captured the number and type of patient-reported concerns for each checklist submitted, as well as the number of times clinicians accessed the discharge column on the safety dashboard to view patient-reported concerns and click on the link to initiate postdischarge messaging.
Patient Surveys (Quantitative)
We previously reported usability of the patient portal, which included a discharge module with an earlier, noninteractive version of the checklist [
]. To better understand perceived utility of the PDTK, we asked a convenience sample of enrolled patients to participate in a survey guiding them through each intervention component. The 2-part survey, based on prior work [ ], asked participants to rate their perceived readiness for discharge and willingness to self-assess discharge preparedness on a 5-point Likert scale. After watching the video, completing the checklist, and viewing how clinicians could visualize checklist responses in the safety dashboard, participants rated statements about their confidence that the intervention would facilitate self-identification and communication of discharge concerns to their care team.Patient Interviews (Qualitative)
We verbally consented and conducted semistructured interviews with a convenience sample of English-speaking patients within 24-48 hours of anticipated discharge. Care team members (nurse, physician assistant, and resident or attending) were asked to identify patients within this time frame who were present in their rooms and would be amenable to participating in a brief interview. To reduce sampling bias, potential participants were selected to create a diverse sample based on age, gender, and reason for hospitalization. To minimize selection bias, patients were assured that their participation would not influence their care team’s medical decisions, and care teams were not told which patients agreed to participate. All participants completed the checklist and watched the video either before or at the time of the interview. Study staff (DP and NP) trained in qualitative research methods conducted the interviews using a semistructured interview guide that explored (1) patient experiences completing the checklist and viewing the video, (2) addressal of discharge concerns by the care team, and (3) pros and cons of using these tools. Interviews were digitally recorded, transcribed, and reviewed for accuracy, and conducted until thematic saturation was achieved [
].Clinician Focus Groups (Qualitative)
After the study was completed, we conducted focus groups with physicians, physician assistants, and nurses to assess implementation barriers and facilitators until thematic saturation. Using a structured guide, we asked about EDD entry via the EHR, perception of the checklist workflow, usage of the discharge column of the safety dashboard to review checklist responses and initiate postdischarge messaging, and awareness of patient-facing components (bedside display and patient portal). Focus group discussions were digitally recorded, transcribed, and reviewed for accuracy.
Statistical and Qualitative Analyses
Descriptive statistics were used to report patient demographic and administrative data, quantify patient-reported concerns, calculate the frequency of tool use by patients and clinicians, and quantify survey data. We calculated the proportion of patients (or caregivers) completing the checklist via the patient portal or REDCap. All qualitative data collected from patient and clinician participants were transcribed, openly coded, and analyzed using the constant comparative method [
]. Two researchers (DP and NP) independently coded all transcripts line by line using Word (Microsoft, Inc, Redmond, WA). The code structure was revised as needed to capture novel concepts, adapt, and merge existing concepts; transcripts were coded with each iteration of the codebook, with any discrepancies resolved during consensus meetings. This process was repeated until no novel concepts were identified, at which point, the 2 researchers again independently applied the final code structure to all transcripts. Key themes were identified in a final group consensus meeting with the study staff. Key implementation barriers were identified by study staff via a group consensus approach based on quantitative and qualitative data.Results
Of 752 patient-admissions, the patient (or caregiver) watched the video and completed the checklist in 416 (55.3%), and the patient/caregiver completed the checklist alone in 94 (12.5%). Research assistants made 313 attempts at approaching patients in the remaining 242 patient-admissions; however, the patient was unavailable (126), not appropriate per nurse (97), declined to participate (41), did not speak English or Spanish and no caregiver was available (33), did not respond by email when reminded (8), or encountered technical issues (8). The demographic characteristics of the 67.8% (510/752) patient-admissions (480 unique patients) in which a checklist was submitted and the 32.2% (242/752) patient-admissions (238 unique patients) in which the checklist was not submitted are reported in
. In general, those who did not submit a checklist were older, more often Hispanic and non-English speaking, less often privately insured, had higher diagnosis-related group (DRG) weights and longer lengths of stay, and were typically discharged to a destination other than home.Characteristics | Submitted checklist (n=510) | Did not submit checklist (n=242) | P value | ||||
Unique patients | 480 | 238 | —a | ||||
1 hospitalization | 453 | 234 | |||||
2 or more hospitalizations | 27 | 4 | |||||
Age (years), mean (SD) | 58.6 (17.9) | 62.4 (18.6) | .008b | ||||
Missing | — | 14 | — | ||||
Gender, n (%) | .94c | ||||||
Female | 280 (54.9) | 126 (52.1) | |||||
Missing | — | 14 (5.8) | |||||
Race, n (%) | .13c | ||||||
White | 340 (66.7) | 140 (57.9) | |||||
Nonwhite | 162 (31.7) | 86 (35.5) | |||||
Missing | 8 (1.6) | 16 (6.6) | |||||
Ethnicity, n (%) | <.001 | ||||||
Non-Hispanic | 468 (91.8) | 189 (78.1) | |||||
Hispanic | 33 (6.5) | 36 (14.9) | |||||
Unavailable | 9 (1.7) | 3 (1.2) | |||||
Missing | — | 14 (5.8) | |||||
Primary language, n (%) | <.001c | ||||||
English | 492 (96.5) | 187 (77.3) | |||||
Non-English | 11 (2.2) | 40 (16.5) | |||||
Missing | 7 (1.4) | 15 (6.2) | |||||
Median income by ZIP code | .46c | ||||||
≤US $47,000 | 96 (18.8) | 49 (20.3) | |||||
US $47,001 to US $63,000 | 124 (24.3) | 48 (19.8) | |||||
Greater than US $63,000 | 272 (53.3) | 131 (54.1) | |||||
Missing | 18 (3.5) | 14 (5.8) | |||||
Insurance status, n (%) | <.001c | ||||||
Private | 195 (38.2) | 70 (28.9) | |||||
Public (Medicaid, Medicare) | 305 (59.8) | 134 (55.4) | |||||
Otherc | 10 (2.0) | 18 (7.4) | |||||
Missing | — | 20 (8.3) | |||||
Primary care physician, n (%) | .01c | ||||||
In-network | 229 (44.9) | 125 (51.7) | |||||
Nonnetwork | 280 (54.9) | 103 (42.5) | |||||
Missing | 1 (0.2) | 14 (5.8) | |||||
Elix number of comorbidities, mean (SD) | 4.22 (2.41) | 4.53 (2.45) | .15b | ||||
Missing | — | 14 | |||||
Elix index, comorbidities, n (%) | .03c | ||||||
Less or = 0 | 104 (20.4) | 33 (13.6) | |||||
1 to 5 | 96 (18.8) | 29 (12.0) | |||||
6 to 10 | 94 (18.4) | 26 (10.7) | |||||
11 or more | 216 (42.4) | 106 (43.8) | |||||
Missing | — | 48 (19.8) | |||||
DRGd weight, mean (SD) | 1.83 (1.97) | 2.26 (2.10) | <.001b | ||||
Missing | 10 | 19 | |||||
Length of stay, mean (SD) | 8.78 (7.93) | 11.5 (13.7) | .02b | ||||
Missing | — | 14 | |||||
Discharge destination, n (%) | .003c | ||||||
Home | 410 (80.4) | 133 (55.0) | |||||
Facility | 92 (18.0) | 58 (24.0) | |||||
Other | 7 (1.4) | 2 (0.8) | |||||
Missing | 1 (0.2) | 49 (20.3) | |||||
Readmissions within 30 days, n (%) | .18c | ||||||
Yes | 88 (17.3) | 39 (16.1) | |||||
No | 422 (82.7) | 140 (57.9) | |||||
Missing | — | 63 (26.0) |
aNot applicable.
bP value calculated by Wilcoxon test.
cP value calculated via chi-square test.
dNonstandard insurance or self-insured.
eDRG: diagnosis-related group.
Usage of each PDTK component for the 510 patient-admissions in which a checklist was submitted is reported in
. Although the patient was enrolled in the acute care patient portal in 173 (33.9%) of the 510 patient-admissions, the patient portal was used to submit the checklist in 53 (10.4%); the remainder were submitted via REDCap. The checklist was submitted once and 2 or more times in 492 and 18 patient-admissions, respectively. The median (IQR25,75) days from initial checklist submission to EDD and ADD were 1 (1,2) and 2 (1,5), respectively. On average, 4.24 concerns were reported for each checklist submitted: the most commonly entered concerns by patients were about medications (664/2164, 30.68%) and follow-up (656/2164, 30.31%). The EDD was accurate in 307 (60.2%) of the 510 patient-admissions.Of the 20 patient experience survey participants, 13 (65%) felt well prepared for discharge, and 16 (80%) stated that they would be willing to self-assess discharge preparedness via a checklist. After viewing how the PDTK components functioned, all (100%) completed the checklist, reporting an average of 5.1 concerns (17 understanding of the plan, 30 medications, 21 self-care management, and 34 follow-up); and 7 (35%) requested secure messaging after discharge; 13 (65%) felt that the checklist facilitated self-identification of potential issues before discharge; 15 (75%) believed that their care team would become aware of these issues via the safety dashboard; 10 (50%) felt more confident about what to do to prevent issues after leaving, and 15 (75%) felt confident that they could quickly communicate with a hospital physician via secure messaging should an issue arise postdischarge.
Of the 20 patients approached for semistructured interviews, 12 participated: 7 (58%) were male, 10 (83%) were white, the median age was 70.5 years, and 8 had public insurance. The most common reason for declining to participate was feeling unwell or tired. We identified two overarching themes about discharge preparation: (1) gaps in communication between patients and their care team resulting in patients feeling inadequately informed about their discharge care plan (eg, I wasn’t informed and kept up to date with what was happening and the reason why. I understood on my own basically what was needed because I’ve gone through this before, but if it was the first time, I think I would have been very confused); and (2) despite perceived communication gaps, patients were confident that their care team would address all of their questions and concerns before discharge (eg, I knew everything was going to be done and things were going to be taken care of, but really, I didn’t feel informed, I really didn’t). We also identified key themes regarding patient experiences using the checklist and video components of the PDTK (
).In total, 22 clinicians (8 physicians, 6 physician assistants, 8 nurses; mean age 36.9 years; 14 (14/22, 64%) female) participated in 1 of 3 focus groups from which we identified 3 major themes regarding the safety dashboard component of the PDTK (
): low awareness and variable workflow, lack of optimization, and inconsistent leadership.Metric | Statistic | Comment | |
Discharge video watches, n (%) | 416 (81.6) | Watched before checklist completion, most often the English version | |
Discharge checklist version submitted, n (%) | |||
Patient | 497 (97.5) | —a | |
Caregiver | 13 (2.5) | Consented if patient preferred or did not have capacity | |
Electronic workflow used to submit checklist, n (%) | |||
Web-based REDCap survey | 457 (89.6) | Submitted via a mobile device | |
Patient portal (discharge module) | 53 (10.4) | Could submit the checklist via the portal or REDCap | |
Total number of concerns reported bby domain | 164 | Most frequent items checked no or unsure by domain | |
Understanding the plan, n (%) | 355 (16.4) | Understanding the main reason for hospitalization | |
Medications, n (%) | 664 (30.7) | Understanding changes to the medication regimen and how to get and take medications | |
Self-care, n (%) | 437 (20.2) | Understanding red flag signs and symptoms | |
Follow-up, n (%) | 656 (30.3) | Time and date of appointments, how to get to them | |
Other, n (%) | 52 (2.4) | Unaddressed clinical concerns, nonmedical barriers | |
Safety dashboard discharge column | |||
Viewed by clinical staff during patient-admission, n (%) | 210 (41.2) | Accessed safety dashboard’s patient-detail view or clicked acknowledgment check-box | |
Total number of times accessed, n | 631 | Median (IQR 25,75): 2 (1,4) per patient-admission | |
RN, n (%) | 399 (63.2) | Unit-based bedside nurses | |
MD, n (%) | 180 (28.5) | Attending or resident | |
Administrative, n (%) | 44 (7.0) | Unit clerk | |
Physician assistant, n (%) | 8 (1.3) | Worked on separate nonresident service with attendings | |
Secure postdischarge messaging (n=422c) | |||
Requested by patient, n (%) | 141 (33.4) | Patient must have had mobile phone with a mobile web-browser | |
Initiated by physician, n (%) | 3 (2.1) | 2 attendings, 1 senior resident |
aNot applicable.
bA discharge checklist item for which the response was no or unsure was considered a patient-reported concern.
cDenominator reflects number of patient admissions in which postdischarge messaging was available.
Theme | Description | Quote | |
Checklist and video | |||
Valuable for patients |
|
| |
Patient utility dependent on the timing of administration |
|
| |
Safety dashboard | |||
Low awareness, variable workflow |
|
| |
Lack of optimization |
|
| |
Inconsistent leadership |
|
|
aEDD: expected discharge date.
Discussion
Principal Findings
We used the RE-AIM framework to evaluate the feasibility and acceptability of a suite of EHR-integrated digital health tools to engage patients, caregivers, and clinicians in discharge preparation. Most patients agreed to watch the discharge video and complete the checklist to self-assess discharge preparedness when coached, and we did not encounter significant technical difficulty in our approach. The patient-facing tools were perceived to be valuable by both patients and clinicians, and most patient-reported concerns submitted via the checklist related to medications and follow-up. Clinician use of the safety dashboard discharge column to view these concerns was modest, mostly due to workflow challenges. A large percentage of patients requested postdischarge messaging, but very few clinicians opted in. Themes identified from our qualitative analysis suggest that timing of administration, additional workflow integration, optimization, and leadership are necessary to promote a more robust adoption of these tools.
We attribute the high rate of patient participation to flexible Web-based workflows and facilitation by research assistants. First, the iterative process to develop and refine the checklist and video incorporated feedback from patient advisers and institutional stakeholders, resulting in a product that was relevant and understandable to patients. Next, research assistants functioned as discharge advocates for study participants, guiding them through the process of viewing the video, and completing the checklist. This encouraged participation from patients who might otherwise have not been comfortable or motivated to do so independently. In addition, we included a video component and incorporated a checklist submission process that did not depend on the acute care patient portal: REDCap’s Web-based workflow was useful as a mobile app prototype and also circumvented key barriers to patient portal enrollment and use during hospitalization, such as patients’ dislike of having a separate log-in for the acute care patient portal [
, ]. In contrast, the direct hyperlink to the discharge checklist in REDCap offered more streamlined access (eg, no log-in was needed) for patients, and facilitation by research staff mitigated the perceived burden of submitting the checklist electronically on their own.Although most patient participants perceived that the intervention would facilitate communication regarding discharge concerns to their care team, often this did not occur because of low uptake of the intervention by clinicians; we attribute this to low awareness of the intervention, inconsistent understanding of its purpose and how to use it, and lack of specificity of patient-reported concerns viewable on the safety dashboard. Specifically, although the safety dashboard discharge column logic was vetted by institutional stakeholders to align with hospital priorities (improving EDD accuracy), clinicians often misinterpreted safety dashboard flag colors (eg, green did not signify safe for discharge) and had a different understanding of responsibility for updating the EDD in the EHR because of inconsistent processes. In addition, if clinicians did not access the detailed view for the patient or click on the flag, they did not see the full text of the flag and what it meant, only its color [
]. Finally, variable use of the discharge column by physicians likely led to poor awareness of the link to initiate a secure messaging thread when requested by patients, and many were resistant to using this feature altogether.As one of the first reported attempts at engaging patients, caregivers, and clinicians in discharge preparation using a suite of EHR-integrated digital health tools, findings from our implementation study, guided by the RE-AIM framework, offer several instructive lessons (
). First, the optimal timing of when to complete the checklist is paramount. If completed too early, patients perceive less utility, anticipating that the care team would eventually address their concerns. If completed too late, concerns identified from the patients’ perspective are less likely to be communicated to clinicians, leading to potential deficiencies or delays in addressing them. Second, dashboard flag changes were not linked to relevant EHR data elements because of competing workflows (eg, a newly created process for documenting and escalating discharge barriers to hospital leadership) and technical limitations (eg, lack of a Web service to retrieve readmission risk scores from the EHR). Rather than linking flag changes to EDD documentation and timing, using the safety dashboard to review patient-reported concerns about discharge might be more clinically meaningful in the context of patient-specific readmission risk scores; this would be more consistent with the overall intent of the safety dashboard as a tool for proactively identifying patients at risk for harm [ , ], Third, improving EDD accuracy via this type of intervention is more likely to be achieved if the responsibility for updating EDD resides with clinicians rather than unit clerks and is clear to all parties. Patients and clinicians will then have more confidence in the EDD displayed on the patient portal, bedside display, and safety dashboard. Still, it is noteworthy that EDD accuracy for enrolled patients (307/510, 60.2%) was marginally higher than the EDD accuracy rate for general medicine (965/1702, 56.70%) as a whole. Finally, secure messaging after discharge clearly requires a better understanding of factors predicting whether clinicians will use this feature and how to incentivize its use.Implementation barriers | Strategies to promote adoption | ||
Discharge video | |||
Timing and access of video after admission to the unit |
| ||
Too generic and impersonal |
| ||
Discharge checklist | |||
Timing and administration |
| ||
Patients’ belief that clinicians will address all items |
| ||
Checklist responses out-of-date owing to discharge delays |
| ||
Dashboard discharge column | |||
Variable EHR data entry of key data elements (EDD, medical, nonmedical barriers) |
| ||
Competing QId interventions |
| ||
Poor specificity of patient-reported concerns viewed in the dashboard |
| ||
Secure postdischarge messaging | |||
Physician resistance |
| ||
Managing patient expectations about whether physicians will initiate secure messaging |
|
aEDD: expected discharge date.
bEHR: electronic health record.
cADD: actual discharge date.
dQI: quality improvement
Limitations
Our study has several limitations. First, it was conducted for general medicine patients at a single institution without a control group; therefore, we could not evaluate the impact of our intervention on clinical outcomes. Similarly, our qualitative analyses were performed on convenience samples of participants, which may limit the generalizability of our findings. Second, we used research assistants to coach patients in submitting the checklist; although dedicated discharge advocates are becoming increasingly common, many institutions lack sufficient personnel [
]. In most hospitals, nursing staff could serve patients in this capacity as they are often the first to identify concerns reported by patients preparing for discharge. Third, we identified disparities among those who submitted and those who did not submit a checklist. Clearly, additional work is needed to address disparities in underrepresented groups to fully evaluate the utility of this intervention in a broader population. Finally, this was a hospital-centric intervention—we did not engage primary care physicians. Although our efforts at postdischarge messaging attempted to bridge the transition from inpatient to ambulatory care, the secure messaging vendor used in this study did not offer the ability to communicate with multiple care team members simultaneously, as we previously described [ ]. Nonetheless, seamless communication with key ambulatory clinicians is important during the immediate postdischarge period [ , ].Conclusions
We believe that EHR-integrated digital health tools such as those we described will become increasingly useful as part of an institutional strategy to engage patients, caregivers, and clinicians in improving discharge safety if they simultaneously address key hospital priorities (eg, improving EDD accuracy and mitigating readmission risk). Currently, we are making further enhancements to the intervention components and their implementation. For example, we are reconfiguring the dashboard discharge column logic to more clearly identify patients at high risk for readmission, which should provide context for the types of concerns patients report after completing the checklist. Exploratory features, such as secure messaging with patients, clearly require further investigation to better characterize patient and clinician perceptions of its value and appropriate use after discharge. However, we believe that many of these features will become increasingly utilized to comply with new regulations (eg, Caregiver Advise, Record, Enable [CARE] Act) [
]. Finally, we plan to conduct rigorously designed studies to evaluate the impact of the PDTK on key outcomes during transitions, such as patient activation at discharge, postdischarge health care resource utilization, and hospital readmissions.Acknowledgments
This work was supported by a grant from AHRQ (R21-HS024751). AHRQ had no role in the design or conduct of the study; collection, analysis, or interpretation of data; or preparation or review of the manuscript. The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of AHRQ.
The authors thank the PSLL research team: Patricia C Dykes, Sarah A Collins, Kumiko O Schnock, Ronen Rozenblum, Lisa S Lehmann, Stuart R Lipsitz, Alexandra Businger, Pamela Garabedian, Eli Mlaver, Brittany Couture, Megan Duckworth, Jenzel Espares, and Kerrin Bersani. The PSLL study was funded by the AHRQ (R21-HS024751).
Authors' Contributions
All authors have contributed sufficiently and meaningfully to the conception, design, and conduct of the study; data acquisition, analysis, and interpretation; or drafting, editing, and revising of the manuscript.
Conflicts of Interest
JS was the recipient of an investigator-initiated grant from Mallinckrodt Pharmaceuticals to study opioid-related adverse drug events and an investigator-initiated grant from Portola Pharmaceuticals to study patients who decline inpatient venous thromboembolism prophylaxis. The remaining authors have no competing interests to declare.
All patient safety learning laboratory enhancements. Enlarged images for viewing the details of the technological enhancements.
PPTX File , 13848 KB
All checklist versions. Three versions of the discharge preparedness checklist were created (English speaking patients and caregivers, as well as Spanish speaking patients), listed in this appendix in that order.
DOCX File , 440 KBReferences
- Forster AJ, Murff HJ, Peterson JF, Gandhi TK, Bates DW. The incidence and severity of adverse events affecting patients after discharge from the hospital. Ann Intern Med 2003 Feb 04;138(3):161-167. [CrossRef]
- Forster AJ, Clark HD, Menard A, Dupuis N, Chernish R, Chandok N, et al. Adverse events among medical patients after discharge from hospital. CMAJ 2004 Feb 03;170(3):345-349 [FREE Full text] [Medline]
- Moore C, Wisnivesky J, Williams S, McGinn T. Medical errors related to discontinuity of care from an inpatient to an outpatient setting. J Gen Intern Med 2003 Aug;18(8):646-651. [CrossRef] [Medline]
- Tsilimingras D, Schnipper J, Duke A, Agens J, Quintero S, Bellamy G, et al. Post-discharge adverse events among urban and rural patients of an urban community hospital: a prospective cohort study. J Gen Intern Med 2015 Aug;30(8):1164-1171. [CrossRef]
- Kanaan AO, Donovan JL, Duchin NP, Field TS, Tjia J, Cutrona SL, et al. Adverse drug events after hospital discharge in older adults: types, severity, and involvement of Beers Criteria medications. J Am Geriatr Soc 2013 Nov;61(11):1894-1899 [FREE Full text] [CrossRef] [Medline]
- Forster AJ, Murff HJ, Peterson JF, Gandhi TK, Bates DW. Adverse drug events occurring following hospital discharge. J Gen Intern Med 2005 Apr;20(4):317-323. [CrossRef] [Medline]
- Rojas-García A, Turner S, Pizzo E, Hudson E, Thomas J, Raine R. Impact and experiences of delayed discharge: a mixed-studies systematic review. Health Expect 2017 Feb;21(1):41-56. [CrossRef] [Medline]
- Arora VM, Prochaska ML, Farnan JM, D'Arcy MJ, Schwanz KJ, Vinci LM, et al. Problems after discharge and understanding of communication with their primary care physicians among hospitalized seniors: a mixed methods study. J Hosp Med 2010 Sep;5(7):385-391 [FREE Full text] [CrossRef] [Medline]
- Howard-Anderson J, Busuttil A, Lonowski S, Vangala S, Afsar-Manesh N. From discharge to readmission: understanding the process from the patient perspective. J Hosp Med 2016 Feb 19;11(6):407-412. [CrossRef] [Medline]
- Centers for Medicare & Medicaid Services, Health & Human Services. Medicare program; hospital inpatient prospective payment systems for acute care hospitals and the long-term care hospital prospective payment system and FY 2012 rates; hospitals' FTE resident caps for graduate medical education payment. Final rules. Fed Regist 2011 Aug 18;76(160):51476-51846 [FREE Full text]
- Howard-Anderson J, Lonowski S, Vangala S, Tseng C, Busuttil A, Afsar-Manesh N. Readmissions in the era of patient engagement. JAMA Intern Med 2014 Nov;174(11):1870-1872. [CrossRef] [Medline]
- Graham KL, Auerbach AD, Schnipper JL, Flanders SA, Kim CS, Robinson EJ, et al. Preventability of early versus late hospital readmissions in a national cohort of general medicine patients. Ann Intern Med 2018 Jun 05;168(11):766-774 [FREE Full text] [CrossRef] [Medline]
- Soong C, Daub S, Lee J, Majewski C, Musing E, Nord P, et al. Development of a checklist of safe discharge practices for hospital patients. J Hosp Med 2013 Mar;8(8):444-449. [CrossRef]
- Halasyamani L, Kripalani S, Coleman E, Schnipper J, van Walraven C, Nagamine J, et al. Transition of care for hospitalized elderly patients—development of a discharge checklist for hospitalists. J Hosp Med 2006 Nov;1(6):354-360. [CrossRef]
- Okrainec K, Lau D, Abrams HB, Hahn-Goldberg S, Brahmbhatt R, Huynh T, et al. Impact of patient-centered discharge tools: a systematic review. J Hosp Med 2017 Feb;12(2):110-117. [CrossRef] [Medline]
- Centers for Medicare & Medicaid Services. 2019 Mar. Your Discharge Planning Checklist URL: https://www.medicare.gov/pubs/pdf/11376-discharge-planning-checklist.pdf [accessed 2019-12-01]
- Agency for Healthcare Research and Quality. 2017. Strategy 4: Care Transitions From Hospital to Home: IDEAL Discharge Planning URL: https://www.ahrq.gov/professionals/systems/hospital/engagingfamilies/strategy4/index.html [accessed 2019-12-01]
- Greysen SR, Khanna RR, Jacolbia R, Lee HM, Auerbach AD. Tablet computers for hospitalized patients: a pilot study to improve inpatient engagement. J Hosp Med 2014 Jun;9(6):396-399. [CrossRef] [Medline]
- Dalal AK, Dykes PC, Collins S, Lehmann LS, Ohashi K, Rozenblum R, et al. A Web-based, patient-centered toolkit to engage patients and caregivers in the acute care setting: a preliminary evaluation. J Am Med Inform Assoc 2016 Jan;23(1):80-87. [CrossRef] [Medline]
- Greysen S, Harrison JD, Rareshide C, Magan Y, Seghal N, Rosenthal J, et al. A randomized controlled trial to improve engagement of hospitalized patients with their patient portals. J Am Med Inform Assoc 2018 Dec 01;25(12):1626-1633 [FREE Full text] [CrossRef] [Medline]
- Collins S, Dykes P, Bates DW, Couture B, Rozenblum R, Prey J, et al. An informatics research agenda to support patient and family empowerment and engagement in care and recovery during and after hospitalization. J Am Med Inform Assoc 2018 Feb 01;25(2):206-209. [CrossRef] [Medline]
- Dalal AK, Bates DW, Collins S. Opportunities and challenges for improving the patient experience in the acute and postacute care setting using patient portals: the patient's perspective. J Hosp Med 2017 Dec;12(12):1012-1016. [CrossRef] [Medline]
- Huerta TR, McAlearney AS, Rizer MK. Introducing a patient portal and electronic tablets to inpatient care. Ann Intern Med 2017 Dec 05;167(11):816-817. [CrossRef] [Medline]
- Coleman EA, Berenson RA. Lost in transition: challenges and opportunities for improving the quality of transitional care. Ann Intern Med 2004 Oct 05;141(7):533-536. [CrossRef] [Medline]
- Middleton B, Renner K, Leavitt M. Ambulatory practice clinical information management: problems and prospects. Healthc Inf Manage 1997;11(4):97-112. [Medline]
- Bakken S, Ruland CM. Translating clinical informatics interventions into routine clinical care: how can the RE-AIM framework help? J Am Med Inform Assoc 2009;16(6):889-897 [FREE Full text] [CrossRef] [Medline]
- Mlaver E, Schnipper JL, Boxer RB, Breuer DJ, Gershanik EF, Dykes PC, et al. User-centered collaborative design and development of an inpatient safety dashboard. Jt Comm J Qual Patient Saf 2017 Dec;43(12):676-685. [CrossRef] [Medline]
- Duckworth M, Leung E, Fuller T, Espares J, Couture B, Chang F, et al. Nurse, patient, and care partner perceptions of a personalized safety plan screensaver. J Gerontol Nurs 2017 Apr 01;43(4):15-22. [CrossRef] [Medline]
- Dalal A, Fuller T, Garabedian P, Ergai A, Balint C, Bates D, et al. Systems engineering and human factors support of a system of novel EHR-integrated tools to prevent harm in the hospital. J Am Med Inform Assoc 2019 Jun 01;26(6):553-560. [CrossRef] [Medline]
- Businger A, Fuller T, Schnipper J, Rossetti S, Schnock K, Rozenblum R. Lessons learned implementing a complex and innovative patient safety learning laboratory project in a large academic medical center. J Am Med Inform Assoc 2020 Feb 01;7(2):301-307. [CrossRef]
- Dalal AK, Fuller TE, Garabedian P, Breuer D, Lemonias D, Harry E. An interdisciplinary team to support implementation of a 'system-of-systems' to identify, assess, and mitigate threats to patient safety in real-time. 2018 Apr 08 Presented at: Society of Hospital Medical Conference; April 8-11; Orlando, FL URL: https://www.shmabstracts.com/abstract/an-interdisciplinary-team-to-support-implementation-of-a-system-of-systems-to-identify-assess-and-mitigate-threats-to-patient- safety-in-real-time/
- Dykes PC, Duckworth M, Cunningham S, Dubois S, Driscoll M, Feliciano Z, et al. Pilot testing fall TIPS (Tailoring Interventions for Patient Safety): a patient-centered fall prevention toolkit. Jt Comm J Qual Patient Saf 2017 Aug;43(8):403-413. [CrossRef] [Medline]
- Dykes PC, Rozenblum R, Dalal A, Massaro A, Chang F, Clements M, et al. Prospective evaluation of a multifaceted intervention to improve outcomes in intensive care: the Promoting Respect and Ongoing Safety Through Patient Engagement Communication and Technology Study. Crit Care Med 2017;45(8):e806-e813. [CrossRef] [Medline]
- Mueller SK, Schnipper JL, Giannelli K, Roy CL, Boxer R. Impact of regionalized care on concordance of plan and preventable adverse events on general medicine services. J Hosp Med 2016 Sep;11(9):620-627. [CrossRef] [Medline]
- Dykes PC, Stade D, Chang F, Dalal A, Getty G, Kandala R, et al. Participatory design and development of a patient-centered toolkit to engage hospitalized patients and care partners in their plan of care. AMIA Annu Symp Proc 2014;2014:486-495 [FREE Full text] [Medline]
- Schnock KO, Snyder JE, Fuller TE, Duckworth M, Grant M, Yoon C, et al. Acute care patient portal intervention: portal use and patient activation. J Med Internet Res 2019 Jul 18;21(7):e13336. [CrossRef] [Medline]
- Schnipper JN, Dalal A, Magny-Normilus C, Samal L, Bitton A, Thompson R, et al. Effects of a multi-faceted intervention to improve care transitions within a pioneer accountable care organization. 2016 Presented at: Society of Hospital Medicine Annual Meeting; March 6-9; San Diego, CA URL: https://www.shmabstracts.com/abstract/effects-of-a-multi-faceted- intervention-to-improve-care-transitions-within-a-pioneer-accountable-care-organization/
- Ganguli I, Sikora C, Nestor B, Sisodia RC, Licurse A, Ferris TG, et al. A scalable program for customized patient education videos. Jt Comm J Qual Patient Saf 2017 Nov;43(11):606-610. [CrossRef] [Medline]
- Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009 Apr;42(2):377-381. [CrossRef] [Medline]
- Dalal AK, Schnipper J, Massaro A, Hanna J, Mlaver E, McNally K, et al. A Web-based and mobile patient-centered 'microblog' messaging platform to improve care team communication in acute care. J Am Med Inform Assoc 2017 Apr 01;24(e1):e178-e184. [CrossRef] [Medline]
- Saunders B, Sim J, Kingstone T, Baker S, Waterfield J, Bartlam B, et al. Saturation in qualitative research: exploring its conceptualization and operationalization. Qual Quant 2017 Sep 14;52(4):1893-1907. [CrossRef]
- Glaser BG. The constant comparative method of qualitative analysis. Soc Prob 1965;12(4):436-445. [CrossRef]
- Bersani K, Fuller TE, Garabedian P, Espares J, Mlaver E, Businger A, et al. Use, perceived usability, and barriers to implementation of a patient safety dashboard integrated within a vendor EHR. Appl Clin Inform 2020 Jan 15;11(1):34-45. [CrossRef] [Medline]
- Donzé J, Aujesky D, Williams D, Schnipper JL. Potentially avoidable 30-day hospital readmissions in medical patients: derivation and validation of a prediction model. JAMA Intern Med 2013 Apr 22;173(8):632-638. [CrossRef] [Medline]
- Voss R, Gardner R, Baier R, Butterfield K, Lehrman S, Gravenstein S. The Care Transitions Intervention: translating from efficacy to effectiveness. Arch Intern Med 2011 Jul 25;171(14):1232-1237. [CrossRef] [Medline]
- van Walraven C, Seth R, Austin PC, Laupacis A. Effect of discharge summary availability during post-discharge visits on hospital readmission. J Gen Intern Med 2002 Mar;17(3):186-192. [CrossRef] [Medline]
- Misky GJ, Wald HL, Coleman EA. Post-hospitalization transitions: examining the effects of timing of primary care provider follow-up. J Hosp Med 2010 Sep;5(7):392-397. [CrossRef] [Medline]
- Coleman EA. Family caregivers as partners in care transitions: the Caregiver Advise Record and Enable Act. J Hosp Med 2016 Dec;11(12):883-885. [CrossRef]
Abbreviations
ADD: actual discharge date |
AHRQ: Agency for Healthcare Research and Quality |
DRG: diagnosis-related group |
EHR: electronic health record |
PDTK: patient-centered discharge toolkit |
PSLL: Patient Safety Learning Laboratory |
RE-AIM: Reach, Effectiveness, Adoption, Implementation, and Maintenance |
Edited by G Eysenbach; submitted 22.07.19; peer-reviewed by R Greysen, P Gazarian, M Duckworth, C Vis; comments to author 21.10.19; revised version received 16.12.19; accepted 04.02.20; published 28.04.20
Copyright©Theresa E Fuller, Denise D Pong, Nicholas Piniella, Michael Pardo, Nathaniel Bessa, Catherine Yoon, Robert B Boxer, Jeffrey Lawrence Schnipper, Anuj K Dalal. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 28.04.2020.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.