This paper is in the following e-collection/theme issue:

Clinical Information and Decision Making (1314) Electronic Health Records (961) Usability Evaluation Case Studies (182) Human Factors and Usability Case Studies (739) Clinical Informatics (904) Design and Usability of Clinical Software and EHRs (242)

Published on in Vol 23, No 4 (2021): April

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/25657, first published .
Usability of Electronic Health Record–Generated Discharge Summaries: Heuristic Evaluation

Usability of Electronic Health Record–Generated Discharge Summaries: Heuristic Evaluation

Usability of Electronic Health Record–Generated Discharge Summaries: Heuristic Evaluation

Authors of this article:

Patrice D Tremoulet1, 2 Author Orcid Image ;   Priyanka D Shah2 Author Orcid Image ;   Alisha A Acosta3 Author Orcid Image ;   Christian W Grant1 Author Orcid Image ;   Jon T Kurtz4 Author Orcid Image ;   Peter Mounas5 Author Orcid Image ;   Michael Kirchhoff6 Author Orcid Image ;   Elizabeth Wade7 Author Orcid Image
Article Authors Cited by (10) Tweetations (3) Metrics

    Original Paper

    1Department of Psychology, Rowan University, Glassboro, NJ, United States

    2Device Evaluation, ECRI, Plymouth Meeting, PA, United States

    3Department of Biochemistry, Rowan University, Glassboro, NJ, United States

    4Department of Computer Science, Rowan University, Glassboro, NJ, United States

    5Department of Biological Sciences, Rowan University, Glassboro, NJ, United States

    6Department of Emergency Medicine, Cooper Medical School, Rowan University, Camden, NJ, United States

    7Concord Hospital, Concord, NH, United States

    Corresponding Author:

    Patrice D Tremoulet, PhD

    Department of Psychology

    Rowan University

    201 Mullica Hill Rd

    Robinson Hall Room 115K

    Glassboro, NJ, 08028

    United States

    Phone: 1 8562564500 ext 53777

    Email: tremoulet@rowan.edu


    Background: Obtaining accurate clinical information about recent acute care visits is extremely important for outpatient providers. However, documents used to communicate this information are often difficult to use. This puts patients at risk of adverse events. Elderly patients who are seen by more providers and have more care transitions are especially vulnerable.

    Objective: This study aimed to (1) identify the information about elderly patients’ recent acute care visits needed to coordinate their care, (2) use this information to assess discharge summaries, and (3) provide recommendations to help improve the quality of electronic health record (EHR)–generated discharge summaries, thereby increasing patient safety.

    Methods: A literature review, clinician interviews, and a survey of outpatient providers were used to identify and categorize data needed to coordinate care for recently discharged elderly patients. Based upon those data, 2 guidelines for creating useful discharge summaries were created. The new guidelines, along with 17 previously developed medical documentation usability heuristics, were applied to assess 4 simulated elderly patient discharge summaries.

    Results: The initial research effort yielded a list of 29 items that should always be included in elderly patient discharge summaries and a list of 7 “helpful, but not always necessary” items. Evaluation of 4 deidentified elderly patient discharge summaries revealed that none of the documents contained all 36 necessary items; between 14 and 18 were missing. The documents each had several other issues, and they differed significantly in organization, layout, and formatting.

    Conclusions: Variations in content and structure of discharge summaries in the United States make them unnecessarily difficult to use. Standardization would benefit both patients, by lowering the risk of care transition–related adverse events, and outpatient providers, by helping reduce frustration that can contribute to burnout. In the short term, acute care providers can help improve the quality of their discharge summaries by working with EHR vendors to follow recommendations based upon this study. Meanwhile, additional human factors work should determine the most effective way to organize and present information in discharge summaries, to facilitate effective standardization.

    J Med Internet Res 2021;23(4):e25657

    doi:10.2196/25657

    Keywords

    discharge summary; usability; electronic health record (EHR); care coordination; elderly patients; patient safety; heuristic evaluation; human factors 


    Rising rates of burnout among outpatient clinicians have been linked to the adoption of electronic health records (EHRs) [1,2], which are known to have poor usability [3,4]. Other work has shown that updating EHRs is a burden for many outpatient providers [5,6]. However, few researchers have explored how inpatient providers’ adoption of EHRs impacted outpatient clinicians’ ability to assimilate information about recently discharged patients [7,8].

    Most acute care facilities in the United States currently use EHRs to generate documents intended to communicate important information about patients and their recent acute care visits to outpatient providers, including skilled nursing facilities (SNFs), and long-term care facilities. These may be called transition of care documents, clinical handover documents, end-of-visit summaries, or, most commonly, discharge summaries (DSs). Since each acute care organization’s EHR system is “customized” during implementation, outpatient providers who treat patients that utilize different acute care facilities must extract information from DSs that can vary greatly in content, format, and organization. Moreover, EHR customization is performed by information technology professionals who work for EHR vendors or inpatient providers. These professionals do not possess a complete understanding of the data needs, priorities, and information presentation preferences of outpatient providers. Not surprisingly, acute care and outpatient providers have different impressions about how useful EHR-generated DSs are [9].

    DSs that either make it difficult for outpatient providers to find, or do not include, all of the information needed to coordinate care put patients at risk of adverse events [10]. Elderly patients (65 years old or older) are especially vulnerable: They tend to have more comorbidities and thus are often followed by multiple outpatient specialists, and they are more likely to have multiple postacute care transitions than younger, less complex patients. One study found discrepancies between medication lists from a referring hospital and a home health care agency for all 770 elderly patient participants [11], putting them at risk of medication errors. Other care transition–related adverse events include treatment delays and unnecessary tests [12-14]. However, a literature search yielded only 1 study, conducted in Canada nearly a decade ago, that suggests creating specialized DSs for elderly patients [15].

    A standard method of presenting information in US discharge summaries could boost patient safety by (1) facilitating information transfer, by making it easier for outpatient clinicians to find the data needed to coordinate care for their patients and (2) reducing risk of burnout-related adverse events, by decreasing frustration associated with trying to extract information from poorly organized documents [16]. A first step towards standardizing all US DSs is identifying the data that US outpatient providers need to coordinate care for recently discharged patients who are transitioning to SNFs. We focused on this scenario because we judged that medically complex elderly patients would be more likely to transition from acute care to a SNF setting than to other outpatient care settings. Moreover, the data needed to coordinate care for adults transitioning to SNFs should be a superset of the data needed to coordinate care for other elderly patients and younger adults.

    This study addresses 2 gaps in knowledge: (1) Through qualitative methods, it identifies the information that clinicians in US SNFs need to coordinate care for elderly patients who were recently discharged from acute care and (2) through a well-established human factors technique, called heuristic evaluation, it provides insight into how well US acute care providers’ DSs currently support outpatient providers who coordinate care for elderly patients.


    The study included 2 phases. First, the data requirements of outpatient providers who coordinate care for elderly patients were identified through an exploratory, descriptive effort, and the knowledge gleaned from that effort was applied to specify 2 new guidelines for creating useful DSs. Second, the new guidelines were combined with 17 previously developed medical documentation usability heuristics [7] (see Multimedia Appendix 1) and applied to assess how well 4 examples of elderly patient discharge summaries support care coordination. The second part not only shed light on how difficult it can be for outpatient providers to use the discharge summaries currently being produced by inpatient providers’ EHRs but also yielded several recommendations for improving the quality of EHR-generated discharge summaries. All research processes and procedures for both parts of this study were approved by Rowan University’s Institutional Review Board, and exemptions were granted by the institutional review boards of the 2 hospitals that provided deidentified examples of discharge summaries.

    Development of DS Content Guidelines

    A literature review served as the first step towards identifying the data that are necessary or helpful when coordinating care for a recently discharged elderly patient. Aggregating the items identified at the Transitions of Care Consensus Conference [17], which includes 6 items that the Joint Commission mandates be included in all DSs [18], with the items in the standardized DSs used in Australia [19] and the items recommended in a Canadian study that focused specifically upon creating DSs for elderly patients [15], yielded a list of 26 items. See Multimedia Appendix 2 for items recommended by different sources. Next, 15 outpatient care providers who frequently care for elderly patients were interviewed. These care providers included primary care physicians, nurse practitioners, directors of nursing, social workers, transition-of-care nurses, and medical directors. They were asked to categorize each of those 26 items as “always necessary,” “helpful, but not required,” or “not relevant/distracting” and then invited to name any additional data that they recommend be included in elderly patient DSs. After the 15 interviews, the list had grown to 36 items. See Multimedia Appendix 3 to view the structured interview questions. Those items were included in an online survey sent to 2500 members of a Continuing Care Risk Management community that asked participants to categorize the 36 items using the same 3 categories. See Multimedia Appendix 4 to view the survey questions. A majority of the 58 respondents indicated that all 36 items were helpful and that 29 of the 36 should always be included in elderly patient DSs. Thus, 2 content guidelines were established: One states that each of 29 items should be included in DSs, and the second recommends that the remaining 7 items be considered for inclusion (see Results section).

    Creation of Simulated DSs

    Two hospitals, which use systems from different EHR vendors, each provided 10 deidentified elderly patient DSs that were produced by their EHR system. Two of the DSs from each hospital were randomly selected. Simulations of those 4 DSs, which appeared the same as the original documents (eg, same font size and style, layout, headings), were created to keep heuristic evaluation participants blind to the hospitals. The safe harbor method was applied in accordance with the Health Insurance Portability and Accountability Act of 1996 Privacy Rule. This entailed replacing not only all protected health information about the patient (which had already been deidentified) but also all doctors’ names and all health care organization names with fictitious data. The 4 simulated DSs, which are referred to as H1P1 (Hospital 1, Patient 1), H1P2 (Hospital 1, Patient 2), H2P1 (Hospital 2, Patient 1), and H2P2 (Hospital 2, Patient 2), were then reviewed for validity, including faithful replication of formatting, layout, and organization. Figures 1 and 2 show portions of 2 of the simulated DSs. See Multimedia Appendices 5-8 to view the full simulated DSs.

    ‎
    Figure 1. The top portion of a simulated discharge summary from hospital 1.
    View this figure
    ‎
    Figure 2. The middle portion of a simulated discharge summary from hospital 2.
    View this figure

    Heuristic Evaluation

    Heuristic evaluation is a usability assessment technique in which 3-5 trained experts independently apply a set of design best practices, called heuristics, to identify potential usability problems [20]. The participants also rate the severity of the issues, and then all issues and ratings are analyzed collectively [21]. Participants are not expected to discover the same issues, but together they generally identify the most important usability problems [22]. Proponents of heuristic evaluation recommend developing a set of relevant heuristics for the particular type of item being assessed, which can serve as guidance for developing useable products as well as tools for assessing usability [20].

    In this study, 5 human factors experts independently assessed each of the 4 simulated discharge summaries. Next, 5 clinical experts experienced in providing outpatient care each independently rated the severity of all of the potential issues identified by the human factors team. Clinical experts were also given the opportunity to report any additional issues they found in the simulated discharge summaries. This iterative approach to heuristic evaluation, which reduces the time required by clinical experts, has been used successfully to evaluate medical technology [7,23]. This approach is also consistent with Nielsen’s recommendation that experts first be asked to use heuristics to identify issues and later be asked to rate the severity of all issues [24].

    The human factors team was given a set of 19 assessment tools: 17 previously developed medical document usability heuristics [7] (see Multimedia Appendix 1) and the 2 new discharge summary content guidelines (see Results section) and instructions on how to apply the heuristics to assess discharge summaries. The 5 human factors team members independently identified potential usability issues by looking for violations of the heuristics or the guidelines. Once each team member had evaluated each document, the issues were aggregated into 4 lists (one per simulated discharge document). Then, the items in the lists were paraphrased, and duplicates were removed.

    Each of the 4 issue lists was ordered and grouped based upon the way information was presented in the simulated discharge summaries and then described in a set of slides (see Figure 3). Next, 5 clinical experts independently reviewed the simulated discharge documents, provided severity ratings for each issue, and reported and rated any additional usability issues they identified. The clinical experts were provided with a 4-step severity scale. We adapted Sauro’s 3-point scale [25] by adding a level 0 for “not an issue” and then replacing the term “critical” with the word “severe” for level 3. The latter change was intended to prevent participants from avoiding use of that rating; in prior work, some clinicians were reluctant to use “critical,” reserving that for issues that indisputably cause harm [7]. Table 1 describes the levels and provides an example of an issue at each level.

    ‎
    Figure 3. Screenshot of a slide used to summarize potential issues found by the human factors team.
    View this figure
    Table 1. Descriptions of the 4 levels used to rate the severity of issues found in simulated discharge summaries and examples of issues at each level.
    RatingDescriptionExample issue
    0Not a problemNote stating that contents were produced using Dragon Dictate not needed
    1Minor: hesitation or slight irritationRedundant information found in hospital course
    2Moderate: causes delays and moderate irritationNutritional status on discharge and Do not resuscitate orders “hidden” in relatively lengthy hospital course section
    3Severe: causes frustration or potential for error; must be fixedNo indication of whether medication is temporary or permanent

    In some cases, the evaluators suggested specific solutions to issues (eg, restrict use of all CAPS, never abbreviate medication names, include page numbers in the format “page X of Y”). All solutions were summarized and grouped so that common themes could be extracted. These themes were transformed into recommendations (see Discussion section).


    The knowledge assembled during the literature review, interviews, and survey was applied to develop 2 guidelines for creating useful, easy-to-use discharge summaries. These guidelines are shown in Tables 2 and 3.

    Table 2. First new guideline, with the list of items that should be included in elderly patient discharge summaries and how many of the 4 simulated discharge summaries were missing each item.
    Guideline 1. Include each of the following in elderly patient discharge summariesNumber of documents missing this item (n=4)
    Patient identifiers (eg, LAST name, first name, middle name, date of birth, age, gender, medical record number)2
    Date of admission/discharge0
    Hospital admission diagnosis0
    Principal/primary diagnosis (responsible for the largest portion of the patient\'s stay)3
    List of discharge diagnosis0
    Discharge medications (when patient is discharged from acute care)0
    History of present illness for hospitalization2
    Hospital course (events occurring during patient’s hospital stay)0
    Procedures performed in hospital0
    Laboratory tests and investigation results (including pending results & tests due)3
    Patient physical and cognitive functional ability at discharge3
    Discharge status/Patient\'s discharge condition (how the patient is doing, relevant physical findings, patient\'s health status)4
    Medication on hospital admission4
    (Reasons for) Changes in medication during patient\'s stay in hospital4
    Adverse reactions during stay (including allergies to medications and other allergies)4
    Discharge instructions2
    Appointments after discharge2
    Life-sustaining treatments preferences (DNRa, lifesaving instructions, POLSTb)3
    Nutritional status at discharge from hospital3
    Immunization history4
    Patient demographics (eg, address, phone number)4
    Follow-up issues1
    Emergency contact information4
    Nutritional status at hospital admission4
    Patient’s physical and cognitive functional ability at admission3
    Goals of care and treatment plan during hospital stay4
    Discharging physician contact information4
    Contact information of clinician(s) who consulted patient in hospital4
    Patient weight3

    aDNR: do not resuscitate.

    bPOLST: portable medical orders.

    Table 3. Second new guideline, with the list of items that should be considered to be included in elderly patient discharge summaries and how many of the 4 simulated discharge summaries were missing each item.
    Guideline 2. Consider including the following items in elderly patient discharge summariesNumber of simulated documents missing this item (n=4)
    Family history4
    Social and lifestyle history4
    Free-text comments (a field for clinicians to share miscellaneous notes about the patient)3
    Type of medical devices or equipment (eg, bariatric beds) that are needed for the patient in the SNFa4
    Goals of care and treatment plan post hospital discharge4
    Activities of daily living (ADL) status4
    Wound, skin, fall assessment1

    aSNF: skilled nursing facility.

    When these 2 guidelines were combined with 17 previously developed medical document usability items (shown in Multimedia Appendix 1) and applied to assess the 4 simulated discharge documents, the human factors team identified 98 issues. See Multimedia Appendices 9-12 for the Powerpoint slides used to present these 98 issues to clinical experts. Clinical experts identified 19 additional issues. The average severity rating across all issues and all documents was 1.69, between minor and moderate. The total number of issues and average severity ratings for each of the simulated DSs were as follows: H1P1: 40, 1.91; H1P2: 30, 1.67; H2P1: 21, 1.73; H2P2: 26, 1.36. The entire set of issues was combined and then grouped based upon how they impact the 5 categories of document usability identified in a previous study [7]: readability, comprehensibility, minimalism, content, and organization. The counts and average severity ratings of the issues associated with each of these categories were as follows: comprehensibility: 8, 2.00; content: 42, 1.99; organization: 31, 1.77; readability: 36, 1.27; minimalism: 0. Finally, Table 4 lists all issues with average severity ratings greater than or equal to 2.

    Table 4. Usability issues with average severity ratings ≥2.
    IssueAverage severity rating
    Postoperative patient with no wound care instructions3
    Treatable condition listed in discharge diagnoses, but patient status, plan, and medication missing3
    No indication whether medications are temporary or permanenta3
    Formatting makes medication section difficult to read or understanda3
    Medications need to be prioritizeda3
    Medication frequency missinga3
    Length of time on IVb meds is missinga3
    No diagnosis linked to the medicationsa3
    Medication is missing information on “PRN” (when is it needed?) a3
    New and “old” medications are mixed in with one anothera2.67
    Electronic signature gives name but no phone or email2.67
    Indenting gives impression that patient discharge condition and time spent on discharge are subordinate to physical exam2.5
    Nutritional status and Do Not Resuscitate orders “hidden” in Hospital Course section2.33
    No page numbers2.13
    Follow-up appointments listed in Hospital Course section2.08
    Heading with no content2
    Formatting made significant findings section hard to read2

    aRelated to the section(s) listing medications.

    bIV: intravenous.

    Each of the clinical experts indicated that the simulated discharge summaries were representative of discharge summaries that they had seen before. They further characterized the examples as “fairly good” in quality when asked to compare them to those that they typically encounter. However, they agreed that each had significant room for improvement. They also agreed that it would be beneficial if DSs from all acute care providers could be standardized.


    Principal Findings

    The knowledge developed through this effort may be broadly applied by acute care organizations seeking to assess or improve the usability of their discharge documents. More specifically, acute care providers can work with EHR vendors to apply the 2 guidelines introduced here to help ensure that they produce discharge summaries that contain the information that outpatient providers need to coordinate care for elderly patients. In addition, acute care providers should consider implementing the recommendations listed in the following sections, which are based upon the results of assessing 4 simulated discharge summaries that were based on real documents. These recommendations have been divided into 2 groups: The first contains those that can be implemented in software, and the second contains those that must be fulfilled by humans — though EHR software could definitely provide prompts or help verify these recommendations have been followed. See Multimedia Appendix 13 to review how several of the issues identified by clinical experts are associated with particular recommendations.

    Recommendations

    Textbox 1 and Textbox 2 include the recommendations for adapting EHRs so they produce more useable discharge summaries.

    Recommendations for adapting electronic health records (EHRs).
    1. Require users to make sure that medication information is complete.
      1. For each medication, require and display the following: medication name; medication strength, dose, dosage unit, route of administration, and frequency; indication for the medication (which diagnosis or complaint is targeted by this medication; if PRN, ensure that the “as needed” criteria are defined within the medication section of the discharge summary); start date; end or refill date; indicator of whether the medication is temporary (end at completion of course) or permanent or chronic (will need refills).
      2. If any information (dose, frequency, end or refill date) is missing, prompt users to insert missing information.
    2. Present all medication information clearly and consistently.
      1. Apply formatting and layout to draw attention to needed information (eg, bold the medication name, separate information with spaces or dashes).
      2. Do not use abbreviations.
      3. List medications by generic name only, not a mix of generic and brand names.
      4. Differentiate new start versus continued medications versus medications that should be stopped.
    3. Provide a section for durable medical equipment, so it can be separated from medications.
    4. Require and display contact information so outpatient providers can follow-up with an inpatient provider.
      1. Prompt for an email address or phone number if only a name is given.
    5. Display start and end dates for all procedures.
      1. Prompt for dates if any are missing.
    6. Print page numbers, in the form “xx of yy pages” when documents are printed, and display page numbers when viewed online to facilitate conversations where one person is viewing a print out and one is viewing online.
    7. Use colors or shading that provide sufficient contrast if documents are printed in black and white.
    8. Show all headings, even if there is no content in a section.
      1. When no accompanying text is provided, prompt for input or obtain user’s approval to populate it with “—“ so readers can verify that the section has been intentionally left blank.
    9. Don't allow page breaks between section headings and section content.
    10. Apply consistent font style, font sizing, spacing, layout, indentation, and heading style.
      1. Font size of the text must be at least 12 points to be easily readable. In some cases (eg, older audiences), selective use of a larger font (14 points) may be advisable, since it can help readers more easily see and focus attention on the most important information. Since the font size depends on the font type selected, maintain a size of 16 pixels at minimum [26].
      2. Maintain a line height that is 130% to 150% larger than the font size [27].
      3. Ensure that the section headings and subheadings stand out. Consider increasing the font size, or bold the heading text.
    Textbox 1. Recommendations for adapting electronic health records (EHRs).
    Recommendations that require user action (but that software can prompt or try to verify).
    1. Provide an explanation of patient’s condition that includes at least a grade (eg, poor, good) and add a justification for any deviation from good.
    2. Do not abbreviate medication or procedure names (eg, AMOX for amoxicillin) to ensure absolute clarity of the conveyed information.
    3. Avoid using “all caps,” which makes the content hard to read.
    4. Emphasize important information. Ensure that the date of exam(s) or lab test(s) is prominently mentioned.
    5. Ensure that all procedures undergone during an acute visit are listed. Examples include the use of feeding tubes, dietary restrictions, total parenteral nutrition (TPN), or urinary catheter.
    6. Ensure that the content matches the headings and subheadings within each section.
    Textbox 2. Recommendations that require user action (but that software can prompt or try to verify).

    Acute care providers may also consider working with EHR vendors to redesign after visit summaries (AVSs) based upon the these recommendations, since prior research has revealed that AVSs are frequently used to develop care plans [7]. Using an AVS, which is the document that is presented to a patient upon discharge from acute care, as a surrogate for a DS is not optimal practice. However, given the current practical limitations in EHR system provider-to-provider communication, it is in patients’ best interests for acute care providers to adapt their AVSs so they contain the information that outpatient physicians need to coordinate care for recently discharged patients.

    Limitations

    While this work represents a necessary starting point for eventually standardizing adult patient discharge summaries, it has several limitations. Since the only incentive offered to survey recipients was access to survey results, only 58 out of 2500 (2.32%) of them responded, even after sending multiple reminders. Accordingly, the lists of important “discharge summary components” identified in this study need to be verified by a larger number of outpatient providers. Furthermore, a larger sample of discharge summaries or other usability testing methods may have revealed more issues, leading to additional recommendations. Finally, given the complexity and high level of customization of hospital EHR systems, it is currently unclear how difficult it would be for hospitals to configure their systems to output documents that follow our recommendations or to support users in following them. For example, some current systems are designed to group medication list items into Stop, Start, and Continue categories, which could make it hard to separate durable medical equipment from medications. On the other hand, it would make sense for inpatient providers to request that EHR vendors use the recommendations from this study to make changes to their products that could then be pushed out as software upgrades to all inpatient providers.

    Comparison With Prior Work

    A heuristic evaluation of 4 AVSs generated by acute care providers’ EHR systems revealed formatting and organizational issues that made those documents very difficult to use [7]. The DSs evaluated in this study had fewer formatting and organizational issues, and those issues were rated lower in terms of severity than the ones found in the AVSs. The DSs in this study also had much less irrelevant or unhelpful text than the AVSs in the prior study. On the other hand, there were many more issues related to missing or unclear content in the DSs, and those issues were generally considered to be fairly severe (rated 2-3). This is not surprising because the content guidelines developed for this effort had not been established when the AVSs were evaluated, so their content was only assessed superficially, using the generic heuristics found in Multimedia Appendix 1.

    Australia and the United Kingdom have developed national standards for electronic DSs [18,28], and Canadian researchers have explored standardizing DSs in Nova Scotia [29]. In contrast, most US research aimed at increasing patient safety during care transitions focuses upon improving discharge planning processes (eg, training clinicians to generate more useful discharge documentation [30,31] or involving patients and caregivers in discharge planning [31,32]).

    A few US organizations have developed templates, outlines, or checklists to standardize their own DSs [33-35], but each of them organizes patient information differently. Moreover, even though a 2009 Transitions of Care Consensus Conference produced a list of items that participants recommended be included in transition records [17], recent research indicates that EHR-generated discharge summaries in the United States continue to omit data that outpatient providers need to effectively coordinate care [27,36-39].

    Conclusions

    In summary, this project addresses 2 current gaps in knowledge among inpatient providers and EHR vendors: (1) What data do outpatient providers need to coordinate care for elderly patients recently discharged from acute care facilities to an outpatient facility? and (2) How well do the EHR-generated DSs currently being produced by acute care providers meet outpatient providers’ needs? This work revealed not only that documents currently being produced do not fully meet outpatient provider needs but also that 2 DSs produced by the same acute care provider organization can vary significantly in layout, organization, structure, and content. The current heterogeneity among DSs makes it unnecessarily difficult for outpatient providers to coordinate care for recently discharged patients. This puts patients at risk of adverse events and may contribute to outpatient provider burnout. This work is especially timely given that the Office of the National Coordinator for Health Information Technology and the Centers for Medicare and Medicaid Services recently released proposed rules to facilitate “seamless and secure” electronic transfer of patient data [40]. Seamlessly and securely sharing patient information is not sufficient to ensure high-quality care coordination. Patient data must be delivered in a form that enables clinicians to quickly and easily locate and understand the information most relevant to them. In short, there is an urgent need for additional applied human factors research focused upon improving the quality of the clinical documentation produced by EHR systems. This research should be conducted in parallel with ongoing interoperability efforts, so that once it is possible for EHRs to seamlessly transfer patient data, they will be shared in a concise, well-organized, easy-to-understand form.

    Acknowledgments

    This work was funded by ECRI. A guidance article based upon the recommendations derived from this work was privately published, with access limited to ECRI’s hospital members, in December of 2019. The authors gratefully acknowledge the valuable contributions from the following individuals: Elizabeth Anderson, Paul Anderson, Ellen Bernier, Joseph Conti, Ann Dimapalis, Kelly DiPaolo, Yvonne Fast, Dave Jamison, Susan Jilinski, Heather Kern, Nami Kim, Joseph Libby, Amy Moeller, Ankur Patel, Emily Patterson, Lorraine Possanza, Vic Rose, Firas Saidi, Karen Schoelles, Erin Sparnon, Richard Stefanacci, Ilene Warner-Maron. The authors also wish to thank an anonymous reviewer for suggesting that we add the recommendation that durable medical equipment be separated from medications.

    Authors' Contributions

    PT and PS designed the study, analyzed the data, and generated the recommendations. The human factors evaluation team consisted of PT, AA, CG, JK, and PM. Deidentification of actual elderly patient discharge summaries was performed by MK and EW. PS created the simulated discharge summaries from the deidentified examples. The initial draft of the paper was produced by PT.

    Conflicts of Interest

    None declared.

    ‎
    Multimedia Appendix 1

    Medical Document Usability Heuristics adapted from Tremoulet et al, 2018.

    DOCX File , 15 KB
    ‎
    Multimedia Appendix 2

    Discharge Summary components recommended by different sources.

    DOCX File , 15 KB
    ‎
    Multimedia Appendix 3

    Outpatient Provider Structured Interview Questions.

    DOCX File , 34 KB
    ‎
    Multimedia Appendix 4

    Survey used to help develop two new discharge summary heuristics.

    DOCX File , 25 KB
    ‎
    Multimedia Appendix 5

    Simulated Discharge Summary for patient 1 from hospital 1.

    PDF File (Adobe PDF File), 130 KB
    ‎
    Multimedia Appendix 6

    Simulated Discharge Summary from patient 2 from hospital 1.

    PDF File (Adobe PDF File), 103 KB
    ‎
    Multimedia Appendix 7

    Simulated Discharge Summary for patient 1 from hospital 2.

    PDF File (Adobe PDF File), 73 KB
    ‎
    Multimedia Appendix 8

    Simulated Discharge Summary for patient 2 from hospital 2.

    PDF File (Adobe PDF File), 82 KB
    ‎
    Multimedia Appendix 9

    Potential usability issues in simulated discharge summary for patient 1 from hospital 1.

    PPTX File , 6653 KB
    ‎
    Multimedia Appendix 10

    Potential usability issues in simulated discharge summary for patient 2 from hospital 1.

    PPTX File , 6340 KB
    ‎
    Multimedia Appendix 11

    Potential usability issues in simulated discharge summary for patient 1 from hospital 2.

    PPTX File , 6370 KB
    ‎
    Multimedia Appendix 12

    Potential usability issues in simulated discharge summary for patient 2 from hospital 2.

    PPTX File , 6287 KB
    ‎
    Multimedia Appendix 13

    Locations of several DS issues identified by clinical experts and the recommendations that are related to those issues.

    DOCX File , 17 KB
    1. Downing NL, Bates DW, Longhurst CA. Physician Burnout in the Electronic Health Record Era: Are We Ignoring the Real Cause? Ann Intern Med 2018 Jul 03;169(1):50-51. [CrossRef] [Medline]
    2. Ehrenfeld JM, Wanderer JP. Technology as friend or foe? Do electronic health records increase burnout? Curr Opin Anaesthesiol 2018 Jun;31(3):357-360. [CrossRef] [Medline]
    3. Ellsworth M, Dziadzko M, O'Horo JC, Farrell A, Zhang J, Herasevich V. An appraisal of published usability evaluations of electronic health records via systematic review. J Am Med Inform Assoc 2017 Jan;24(1):218-226 [FREE Full text] [CrossRef] [Medline]
    4. Howe JL, Adams KT, Hettinger AZ, Ratwani RM. Electronic Health Record Usability Issues and Potential Contribution to Patient Harm. JAMA 2018 Mar 27;319(12):1276-1278 [FREE Full text] [CrossRef] [Medline]
    5. Reith TP. Burnout in United States Healthcare Professionals: A Narrative Review. Cureus 2018 Dec 04;10(12):e3681 [FREE Full text] [CrossRef] [Medline]
    6. Ommaya A, Cipriano P, Hoyt D, Horvath KA, Tang P, Paz HL, et al. Care-Centered Clinical Documentation in the Digital Environment: Solutions to Alleviate Burnout. NAM Perspectives 2018 Jan 29;8(1):1 [FREE Full text] [CrossRef]
    7. Tremoulet P, Krishnan R, Karavite D, Muthu N, Regli S, Will A, et al. A Heuristic Evaluation to Assess Use of After Visit Summaries for Supporting Continuity of Care. Appl Clin Inform 2018 Jul 12;9(3):714-724 [FREE Full text] [CrossRef] [Medline]
    8. National Academies of Sciences, Engineering, and Medicine. Taking Action Against Clinician Burnout: A Systems Approach to Professional Well-Being. Washington, DC: The National Academies Press; 2019.
    9. Solan LG, Sherman SN, DeBlasio D, Simmons JM. Communication Challenges: A Qualitative Look at the Relationship Between Pediatric Hospitalists and Primary Care Providers. Acad Pediatr 2016 Jul;16(5):453-459. [CrossRef] [Medline]
    10. Jones CD, Cumbler E, Honigman B, Burke RE, Boxer RS, Levy C, et al. Hospital to Post-Acute Care Facility Transfers: Identifying Targets for Information Exchange Quality Improvement. J Am Med Dir Assoc 2017 Jan;18(1):70-73 [FREE Full text] [CrossRef] [Medline]
    11. Brody AA, Gibson B, Tresner-Kirsch D, Kramer H, Thraen I, Coarr ME, et al. High Prevalence of Medication Discrepancies Between Home Health Referrals and Centers for Medicare and Medicaid Services Home Health Certification and Plan of Care and Their Potential to Affect Safety of Vulnerable Elderly Adults. J Am Geriatr Soc 2016 Nov 27;64(11):e166-e170 [FREE Full text] [CrossRef] [Medline]
    12. Jones CD, Jones J, Bowles KH, Flynn L, Masoudi FA, Coleman EA, et al. Quality of Hospital Communication and Patient Preparation for Home Health Care: Results From a Statewide Survey of Home Health Care Nurses and Staff. J Am Med Dir Assoc 2019 Apr;20(4):487-491 [FREE Full text] [CrossRef] [Medline]
    13. Gadbois EA, Tyler DA, Shield R, McHugh J, Winblad U, Teno JM, et al. Lost in Transition: a Qualitative Study of Patients Discharged from Hospital to Skilled Nursing Facility. J Gen Intern Med 2019 Jan 18;34(1):102-109 [FREE Full text] [CrossRef] [Medline]
    14. Moore AB, Krupp JE, Dufour AB, Sircar M, Travison TG, Abrams A, et al. Improving Transitions to Postacute Care for Elderly Patients Using a Novel Video-Conferencing Program: ECHO-Care Transitions. Am J Med 2017 Oct;130(10):1199-1204. [CrossRef] [Medline]
    15. Kergoat M, Latour J, Julien I, Plante M, Lebel P, Mainville D, et al. A discharge summary adapted to the frail elderly to ensure transfer of relevant information from the hospital to community settings: a model. BMC Geriatr 2010 Sep 23;10(1):69 [FREE Full text] [CrossRef] [Medline]
    16. Tawfik DS, Profit J, Morgenthaler TI, Satele DV, Sinsky CA, Dyrbye LN, et al. Physician Burnout, Well-being, and Work Unit Safety Grades in Relationship to Reported Medical Errors. Mayo Clin Proc 2018 Nov;93(11):1571-1580 [FREE Full text] [CrossRef] [Medline]
    17. Snow V, Beck D, Budnitz T, Miller DC, Potter J, Wears RL, American College of Physicians, Society of General Internal Medicine, Society of Hospital Medicine, American Geriatrics Society, American College of Emergency Physicians, Society of Academic Emergency Medicine. Transitions of Care Consensus Policy Statement American College of Physicians-Society of General Internal Medicine-Society of Hospital Medicine-American Geriatrics Society-American College of Emergency Physicians-Society of Academic Emergency Medicine. J Gen Intern Med 2009 Aug 3;24(8):971-976 [FREE Full text] [CrossRef] [Medline]
    18. Kind AJ, Smith MA. Documentation of Mandated Discharge Summary Components in Transitions from Acute to Subacute Care. In: Henriksen K, Battles JB, Keyes MA, Grady ML, editors. Advances in Patient Safety: New Directions and Alternative Approaches (Vol. 2: Culture and Redesign). Rockville, MD: Agency for Healthcare Research and Quality; 2008:3715   URL: https://www.ncbi.nlm.nih.gov/books/NBK43715/#advances-kind_31.s1
    19. National Guidelines for On-Screen Presentation of Discharge Summaries. Australian Commission on Safety and Quality in Health Care. 2017.   URL: https:/​/www.​safetyandquality.gov.au/​publications-and-resources/​resource-library/​national-guidelines-screen-presentation-discharge-summaries [accessed 2021-03-26]
    20. Nielsen J, Molich R. Heuristic evaluation of user interfaces. 1990 Presented at: CHI90: Conference on Human Factors in Computing; April 1990; Seattle, WA. [CrossRef]
    21. Nielsen J. Reliability of severity estimates for usability problems found by heuristic evaluation. 1992 Presented at: CHI92: Conference on Human Factors in Computing; 1992; Monterey, CA p. 129-130. [CrossRef]
    22. Nielsen J. 10 usability heuristics for user interface design. Nielson Norman Group. 1994 Apr 24.   URL: https://www.nngroup.com/articles/ten-usability-heuristics/ [accessed 2021-03-26]
    23. Tremoulet PD, McManus M, Baranov D. Rendering ICU Data Useful via Formative Evaluations of Trajectory, Tracking, and Trigger (T3) Software. In: Proceedings of the International Symposium on Human Factors and Ergonomics in Health Care. 2017 May 15 Presented at: 2017 International Symposium on Human Factors and Ergonomics in Healthcare; March 2017; New Orleans, LA p. 50-56. [CrossRef]
    24. Nielsen J. Severity ratings for usability problems. Nielson Norman Group. 1994 Nov 01.   URL: https://www.nngroup.com/articles/how-to-rate-the-severity-of-usability-problems/ [accessed 2021-03-26]
    25. Sauro J. Rating the Severity of Usability Problems. MeasuringU. 2013 Jul 30.   URL: https://measuringu.com/rating-severity/ [accessed 2021-03-26]
    26. Section 3: Display content clearly on the page. Health Literacy Online. 2016.   URL: https://health.gov/healthliteracyonline/display/section-3-3/ [accessed 2021-03-26]
    27. Everdell I. Web content. In: Bergstrom JR, Schall AJ, editors. Eye tracking in user experience design. Waltham, MA: Morgan Kaufmann Publishers; 2014:163-186.
    28. Implementation guidance report: e-Discharge standard. Professional Record Standards Body. 2019.   URL: https:/​/theprsb.​org/​wp-content/​uploads/​2019/​02/​eDischarge-Summary-Maintenance-Release-Implementation-Guidance-Report-v2.​1-23.​1.​19.​pdf [accessed 2021-03-26]
    29. Savvopoulos S, Sampalli T, Harding R, Blackmore G, Janes S, Kumanan K, et al. Development of a quality scoring tool to assess quality of discharge summaries. J Family Med Prim Care 2018;7(2):394-400 [FREE Full text] [CrossRef] [Medline]
    30. Ming D, Zietlow K, Song Y, Lee H, Clay A. Discharge summary training curriculum: a novel approach to training medical students how to write effective discharge summaries. Clin Teach 2019 Oct 30;16(5):507-512. [CrossRef] [Medline]
    31. Backman C, Cho-Young D. Engaging patients and informal caregivers to improve safety and facilitate person- and family-centered care during transitions from hospital to home - a qualitative descriptive study. Patient Prefer Adherence 2019 Apr;13:617-626 [FREE Full text] [CrossRef] [Medline]
    32. Medicare and Medicaid Programs; Revisions to Requirements for Discharge Planning for Hospitals, Critical Access Hospitals, and Home Health Agencies, and Hospital and Critical Access Hospital Changes to Promote Innovation, Flexibility, and Improvement in Patient Care. Federal Register: The Daily Journal of the United States Government. 2019 Sep 30.   URL: https:/​/www.​federalregister.gov/​documents/​2019/​09/​30/​2019-20732/​medicare-and-medicaid-programs-revisions-to-requirements-for-discharge-planning-for-hospitals [accessed 2021-03-26]
    33. Discharge Summary Template. University of Tennessee Health Science Center.   URL: https://www.uthsc.edu/pediatrics/clerkship/documents/discharge-summary-template.pdf [accessed 2021-03-26]
    34. Sample Discharge Summary. Creighton University.   URL: https:/​/medschool.​creighton.edu/​fileadmin/​user/​medicine/​Departments/​ob-gyn/​docs/​Sample_Dictation_-_OB_Discharge_Summary.​pdf [accessed 2021-03-26]
    35. Discharge Summary Outline. WVU Medicine.   URL: https:/​/wvumedicine.​org/​ruby-memorial-hospital/​wp-content/​uploads/​sites/​3/​2017/​04/​Discharge-Summary-Outline.​pdf [accessed 2021-03-26]
    36. Dean SM, Gilmore-Bykovskyi A, Buchanan J, Ehlenfeldt B, Kind AJ. Design and Hospitalwide Implementation of a Standardized Discharge Summary in an Electronic Health Record. The Joint Commission Journal on Quality and Patient Safety 2016 Dec;42(12):555-AP11. [CrossRef]
    37. Moy NY, Lee SJ, Chan T, Grovey B, Boscardin WJ, Gonzales R, et al. Development and Sustainability of an Inpatient-to-Outpatient Discharge Handoff Tool: A Quality Improvement Project. The Joint Commission Journal on Quality and Patient Safety 2014 May;40(5):219-227. [CrossRef]
    38. Wimsett J, Harper A, Jones P. Review article: Components of a good quality discharge summary: a systematic review. Emerg Med Australas 2014 Oct 03;26(5):430-438. [CrossRef] [Medline]
    39. Callen JL, Alderton M, McIntosh J. Evaluation of electronic discharge summaries: a comparison of documentation in electronic and handwritten discharge summaries. Int J Med Inform 2008 Sep;77(9):613-620. [CrossRef] [Medline]
    40. Butler M. CMS and ONC Release Proposed Rules on Data Exchange and Interoperability. Journal of AHIMA. 2019 Feb 12.   URL: https://journal.ahima.org/cms-and-onc-release-proposed-rules-on-data-exchange-and-interoperability/ [accessed 2021-03-26]

    ‎
    AVS: after visit summary
    DS: discharge summary
    EHR: electronic health record
    SNF: skilled nursing facility

    Edited by R Kukafka; submitted 10.11.20; peer-reviewed by D Scherer, C Johnson, D Newman; comments to author 25.12.20; revised version received 11.02.21; accepted 16.03.21; published 15.04.21

    Copyright

    ©Patrice D Tremoulet, Priyanka D Shah, Alisha A Acosta, Christian W Grant, Jon T Kurtz, Peter Mounas, Michael Kirchhoff, Elizabeth Wade. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 15.04.2021.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.