This paper is in the following e-collection/theme issue:

Human Factors and Usability Case Studies (739) Design and Usability of Clinical Software and EHRs (242) Electronic Health Records (961) Physician and Healthcare Worker Burnout (97)

Published on in Vol 22, No 12 (2020): December

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/23382, first published .
Perceived Electronic Health Record Usability as a Predictor of Task Load and Burnout Among US Physicians: Mediation Analysis

Perceived Electronic Health Record Usability as a Predictor of Task Load and Burnout Among US Physicians: Mediation Analysis

Perceived Electronic Health Record Usability as a Predictor of Task Load and Burnout Among US Physicians: Mediation Analysis

Authors of this article:

Edward R Melnick1 Author Orcid Image ;   Elizabeth Harry2 Author Orcid Image ;   Christine A Sinsky3 Author Orcid Image ;   Liselotte N Dyrbye4 Author Orcid Image ;   Hanhan Wang5 Author Orcid Image ;   Mickey Todd Trockel6 Author Orcid Image ;   Colin P West7, 8 Author Orcid Image ;   Tait Shanafelt5 Author Orcid Image
Article Authors Cited by (40) Tweetations (35) Metrics

    Original Paper

    1Department of Emergency Medicine, Yale University School of Medicine, New Haven, CT, United States

    2University of Colorado School of Medicine, Aurora, CO, United States

    3Professional Satisfaction and Practice Sustainability, American Medical Association, Chicago, IL, United States

    4Department of Medicine Physician Well-Being Program, Mayo Clinic, Rochester, MN, United States

    5Department of Medicine, Stanford School of Medicine, Palo Alto, CA, United States

    6Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Palo Alto, CA, United States

    7Department of Internal Medicine, Mayo Clinic, Rochester, MN, United States

    8Department of Health Sciences Research, Mayo Clinic, Rochester, MN, United States

    Corresponding Author:

    Edward R Melnick, MD, MHS

    Department of Emergency Medicine

    Yale University School of Medicine

    464 Congress Ave, Suite 260

    New Haven, CT, 06519

    United States

    Phone: 1 (203) 785 5174

    Email: edward.melnick@yale.edu


    Background: Electronic health record (EHR) usability and physician task load both contribute to physician professional burnout. The association between perceived EHR usability and workload has not previously been studied at a national level. Better understanding these interactions could give further information as to the drivers of extraneous task load.

    Objective: This study aimed to determine the relationship between physician-perceived EHR usability and workload by specialty and evaluate for associations with professional burnout.

    Methods: A secondary analysis of a cross-sectional survey of US physicians from all specialties was conducted from October 2017 to March 2018. Among the 1250 physicians invited to respond to the subsurvey analyzed here, 848 (67.8%) completed it. EHR usability was assessed with the System Usability Scale (SUS; range: 0-100). Provider task load (PTL) was assessed using the mental demand, physical demand, temporal demand, and effort required subscales of the National Aeronautics and Space Administration Task Load Index (range: 0-400). Burnout was measured using the Maslach Burnout Inventory.

    Results: The mean scores were 46.1 (SD 22.1) for SUS and 262.5 (SD 71.7) for PTL. On multivariable analysis adjusting for age, gender, relationship status, medical specialty, practice setting, hours worked per week, and number of nights on call per week, physician-rated EHR usability was associated with PTL, with each 1-point increase in SUS score (indicating more favorable) associated with a 0.57-point decrease in PTL score (P<.001). On mediation analysis, higher SUS score was associated with lower PTL score, which was associated with lower odds of burnout.

    Conclusions: A strong association was observed between EHR usability and workload among US physicians, with more favorable usability associated with less workload. Both outcomes were associated with the odds of burnout, with task load acting as a mediator between EHR usability and burnout. Improving EHR usability while decreasing task load has the potential to allow practicing physicians more working memory for medical decision making and patient communication.

    J Med Internet Res 2020;22(12):e23382

    doi:10.2196/23382

    Keywords

    electronic health record; EHR; usability; cognitive load; System Usability Scale; physician task load; NASA Task Load Index; physician; burnout; stress 


    Occupational burnout in physicians is a complex phenomenon with multiple interdependent drivers related to practice efficiency, organizational culture, and personal resilience [1]. In the domain of practice efficiency, there is growing evidence supporting an association between electronic health record (EHR) usability and physician burnout [2-7]. Usability is the extent to which technology can be used effectively, efficiently, and satisfactorily based on its design and integration into a specific context of use [8]. Although the EHR has been lauded as a solution to improve health care quality and safety, there is increasing evidence that EHR usability can cause harm [9-12]. Current EHR usability challenges have resulted in systems that many find unnecessarily complex and prone to error, thereby increasing physician cognitive load and resulting in technological errors that can sometimes reach the patient [10-13]. Partially due to excessive time spent on EHR activities, EHR usability is a specific source of physician dissatisfaction and stress [7,14-16]. The topic has received increasing attention since a 2017 systematic review demonstrated a paucity of published studies and standardized reporting on EHR usability evaluation [17-20]. A recently published cross-sectional national survey of physicians from all specialty disciplines identified a strong association between higher physician-perceived EHR usability (assessed by the industry standard, the System Usability Scale [SUS]) and lower odds of physician burnout [7].

    In addition to usability, the clerical burden associated with documentation, order entry, inbox management, and other EHR administrative tasks (eg, prior authorizations, documentation of care consistent with quality measures) that are not necessarily intrinsic to the practice of medicine also contributes to excessive time spent on the EHR, as well as unnecessary cognitive burden [16,21,22]. Administrative tasks such as these are independent of EHR design—indeed, the EHR may offer advantages by providing a systematic structure and record to such tasks [21]. According to cognitive load theory, tasks like these can overwhelm limited working memory, the process our minds use to input and respond to all information [23]. Cognitive load refers to the amount of working memory used and comprises three components: intrinsic (complexity of the task itself), extraneous (how the task is presented), and germane (the workload of learning the task or content) load [24,25]. When users are overloaded, data is “shed” or lost, which puts users at risk of committing an error [26]. Using the mental demand, physical demand, temporal demand, and effort required subscales of the National Aeronautics and Space Administration Task Load Index (NASA-TLX), a national survey of 4622 physicians conducted by our team identified a strong association between provider task load (PTL) and burnout but did not assess EHR usability [27,28]. Two studies found enhanced EHR usability is associated with reduced physician cognitive load in simulated EHR environments [29,30].

    We propose a conceptual framework in which task load acts as a mediator between EHR usability and professional burnout (Figure 1). For example, a poor EHR interface with patient lab values may present redundant data on the screen, creating unnecessary extraneous load that increases the mental demand of a physician’s task. Ongoing exposure to this interface could contribute to burnout for some physicians. Independent of this task, the poor usability could impede patient communication, thereby diminishing the physician’s sense of purpose in their work, thus leading to burnout [10,31].

    ‎
    Figure 1. Conceptual framework with provider task load (PTL) as a mediator between electronic health record (EHR) usability and professional burnout. MBI: Maslach Burnout Inventory; SUS: System Usability Scale.
    View this figure

    To our knowledge, the association between perceived EHR usability and workload has not previously been studied at a national level. We conducted a cross-sectional analysis of a national survey to determine the association between physician-perceived EHR usability (using the SUS) and the workload of a clinical workday stratified by specialty and practice setting (using the mental demand, physical demand, temporal demand, and effort required subscales of the NASA-TLX, henceforth referred to as PTL). We hypothesized that greater EHR usability scores (as measured by the SUS) would correlate with lower PTL scores. Also, given that both the SUS and PTL have been shown to relate to physician burnout, we hypothesized that they would both remain associated with burnout after adjusting for personal and professional characteristics. Better understanding these interactions could give further information as to the drivers of extraneous task load.


    Study Design, Setting, Participants, and Data Collection

    A secondary analysis of a cross-sectional wellness study of US physicians from all subspecialties was performed. The original survey collected data between October 12, 2017, and March 15, 2018. The sample was assembled using the American Medical Association Physician Masterfile, a nearly complete record of all US physicians, independent of American Medical Association membership. Participation involved voluntary completion of an anonymous electronic survey. Full details of the sampling strategy, recruitment, data collection, and assessment for response bias have previously been reported [32]. Briefly, 30,456 physicians from the Masterfile were invited to participate (Figure 2). Of these, 5197 (17.1%) physicians participated in the study [32]. A random group of 1250 of the participants was invited to complete a subsurvey evaluating their EHR’s usability. Responders who were retired from clinical practice were excluded from the analysis. To evaluate for response bias, an intensive follow-up survey was conducted in a sample of nonresponders. There were 248 (52.1% of 476 invited) participants in the follow-up survey. The Stanford University and Mayo Clinic institutional review boards reviewed and approved the study protocol.

    ‎
    Figure 2. CONSORT (Consolidated Standards of Reporting Trials) diagram of subject enrollment, inclusion criteria, and exclusion criteria. EHR: electronic health record; SUS-TLX: System Usability Scale-Task Load Index.
    View this figure

    Study Measures

    Study measures included items pertaining to participants’ demographic characteristics (age, gender, relationship status), medical specialty, hours worked per week, number of nights on call per week, practice setting, symptoms of burnout, and perception of their EHR’s usability and daily clinical workload.

    EHR Usability

    Physician-perceived EHR usability was measured using the SUS, an industry standard for a quick, reliable measurement of technology usability [33-35]. The SUS includes 10 items, each on a 5-point Likert scale from Strongly Disagree to Strongly Agree that is scored from 0 to 4, with each even-numbered question reverse coded. The items are summed and then multiplied by 2.5 to normalize scores from 0 to 100, with higher scores indicating higher usability. Consistent with convention [34,36], the language of the SUS was modified such that references to “the system” being evaluated were changed to “my EHR.”

    Provider Task Load

    Provider task load was measured using 4 items (the mental demand, physical demand, temporal demand, and effort required subscales) from the NASA-TLX [28]. The rationale for exclusion of the frustration and performance NASA-TLX items from our measurement of PTL is that on principal component analysis with oblimin rotation and Kaiser normalization to determine underlying patterns between the 6 items of the NASA-TLX and the emotional exhaustion and depersonalization scales of the Maslach Burnout Inventory, frustration and performance clustered as one component along with emotional exhaustion and depersonalization scores, suggesting that these domains of the NASA-TLX are measures of work-related distress and would be expected to be collinear with burnout measures. Inclusion of the mental demand, physical demand, temporal demand, and effort required NASA-TLX items is consistent with previous reports assessing workload of physicians and nurses [27,37]. Respondents were prompted to “reflect on a day [they] performed clinical work during the last 1-2 weeks that is representative of a typical current clinical work day” and rate their perception of each subscale demand type on a scale of 0 to 100 (with 100 being the highest level of demand) [38,39]. The 4 scores were summed for a composite score ranging from 0 to 400 [40].

    Burnout

    Burnout was measured using the validated criterion standard, the Maslach Burnout Inventory [41-44]. Respondents with a high score on the emotional exhaustion (≥27) or depersonalization subscale (≥10) were considered to have at least 1 symptom of burnout [41,45-47].

    Statistical Analysis

    Standard descriptive statistics were used to characterize the physician sample that responded to the EHR usability subsurvey and the PTL items of the survey. Associations between variables were evaluated using the Wilcoxon rank sum test (continuous variables) or χ2 test (categorical variables), as appropriate. Univariable linear regression was used to examine the association between EHR usability and PTL. On preliminary analysis controlling for medical specialty, specialties with smaller numbers of participants in the subsurvey had considerable variability. To control for this variation, specialties with fewer than 20 participants were grouped together in a pooled category of “Other” specialties. The Other category included these specialties (number of respondents in parentheses): neurosurgery (9), ophthalmology (8), otolaryngology (10), other (18), physical medicine and rehabilitation (15), preventive medicine & occupational medicine (4), radiation oncology (5), and urology (7). Two multivariable linear regression analyses were performed to investigate whether the relationship between SUS score and burnout is mediated by PTL. Demographic and professional factors included in the multivariable regression analyses were age, gender, relationship status, hours worked per week, medical specialty, nights on call, and practice setting. All tests were 2-sided, with a type I error level of .05. Analyses were completed using R statistical software (R Foundation for Statistical Computing, Vienna, Austria, Version 3.6.0) with the exception of the mediation analyses, which were conducted using the PROCESS version 3 macro for SPSS (IBM Corporation) [48].


    Participants

    There were 5197 responders to the full survey. A randomly selected group of 1250 of these responders received the EHR usability subsurvey. Among these responders, the 848 individuals (67.8%) who responded to all SUS and PTL items were included in the present analysis. The demographic characteristics (age, gender, and medical specialty) of the respondents included in this analysis were generally similar to the full survey respondents and US physicians (Table 1).

    Table 1. Demographic characteristics of survey responders.
    CharacteristicEHR usability subsurvey responders (N=848)
    Gender, n (%)

    		Male493 (58.1)

    		Female348 (41.0)

    		Other1 (0.1)

    		Missing6 (0.7)
    Age (years), median (IQR)53.0 (42.0, 61.0)
    Age (years), n (%)

    		<3561 (7.2)

    		35-44200 (23.6)

    		45-54183 (21.6)

    		55-64259 (30.5)

    		>/=65123 (14.5)

    		Missing22 (2.6)
    Specialty, n (%)

    		Anesthesiology36 (4.2)

    		Dermatology23 (2.7)

    		Emergency medicine54 (6.4)

    		Family medicine55 (6.5)

    		Radiology37 (4.4)

    		Neurology32 (3.8)

    		Obstetrics and gynecology42 (5.0)

    		Pathology25 (2.9)

    		Psychiatry51 (6.0)

    		Other76 (9.0)

    		General internal medicine77 (9.1)

    		Internal medicine subspecialty113 (13.3)

    		General pediatrics46 (5.4)

    		Pediatric subspecialty53 (6.2)

    		General surgery33 (3.9)

    		General surgery subspecialty58 (6.8)

    		Orthopedic surgery35 (4.1)

    		Missing2 (0.2)
    Hours worked per week, median (IQR)50.0 (40.0, 60.0)
    Hours worked per week, n (%)

    		<40 h126 (14.9)

    		40-49 h171 (20.2)

    		50-59 h215 (25.4)

    		60-69 h187 (22.1)

    		70-79 h76 (9.0)

    		>80 h70 (8.3)

    		Missing3 (0.4)
    Nights on call per week, median (IQR)1.0 (0.0, 2.0)
    Primary practice setting, n (%)

    		Private practice388 (45.8)

    		Academic medical center274 (32.3)

    		Veterans hospital16 (1.9)

    		Active military practice9 (1.1)

    		Other160 (18.9)

    		Missing1 (0.1)
    Relationship status, n (%)

    		Single105 (12.4)

    		Married687 (81.0)

    		Partnered39 (4.6)

    		Widowed/widower8 (0.9)

    		Missing9 (1.1)

    SUS and PTL Scores

    Physician-perceived EHR usability and PTL scores were similar to those previously reported in the primary analyses of this survey [7,27]. For this analysis, the mean SUS score was 46.1 (SD 22.1; range: 0-100; IQR 30-62.5), and the mean composite PTL score was 262.5 (SD 71.7; range: 0-400; IQR 215-315) with mean subscale scores of 70.9 (SD 20.8) for mental demand, 47.8 (SD 27.3) for physical demand, 68.3 (SD 24.9) for time demand, and 75.6 (SD 21.1) for effort required. On univariate analysis, each 1-point increase in SUS score (indicating greater EHR usability) was associated with a 0.55-point decrease in PTL score (P<.001; Figure 3). The relationship between perceived EHR usability and composite PTL scores is shown for the 17 specialty discipline categories in Figure 4.

    ‎
    Figure 3. Scatter plot of provider task load (PTL) sum scores (range: 0-400) by System Usability Scale (SUS) scores (range: 0-100) with regression line.
    View this figure
    ‎
    Figure 4. Scatter plot of provider task load (PTL) as assessed by the modified NASA-TLX and electronic health record usability as assessed by the System Usability Scale (SUS) by specialty with regression line. Note that higher SUS indicates more favorable usability, whereas higher PTL indicates increased task load (less favorable).
    View this figure

    Multivariable and Mediation Analyses

    On multivariable analysis adjusting for age, gender, relationship status, medical specialty, practice setting, hours worked per week, and number of nights on call per week, EHR SUS scores were associated with PTL. Each 1-point increase in SUS score was associated with a 0.57-point decrease in composite PTL score (P<.001; Table 2). Gender; age; practicing emergency medicine, anesthesiology, or psychiatry; hours worked per week; and number of nights on call per week were also associated with composite PTL scores in this model. Being female, practicing emergency medicine or anesthesiology, more hours worked per week, and more nights on call per week were all associated with higher PTL as a measure of workload, whereas being older and practicing psychiatry were both associated with lower workload.

    Table 2. Predictors of provider task load in multivariable linear regression models among practicing physicians in 2017.
    PredictorCoefficient (SE)P value
    SUSa score−0.57 (0.03).001
    Gender (reference: male)
    Female12.59 (1.68).03
    Missing/other19.26 (10.24).58
    Age, for each year older−0.45 (0.07).048
    Specialty (reference: general internal medicine)
    Anesthesiology30.6 (4.18).03
    Dermatology13.68 (5.03).42
    Emergency medicine47.04 (3.83)<.001
    Family medicine−2.87 (3.71).82
    Radiology13.33 (4.25).36
    Neurology−1.18 (4.4).94
    Obstetrics and gynecology9.1 (4).5
    Pathology−4.13 (4.83).8
    Psychiatry−29.11 (3.84).03
    Other−9.66 (3.42).4
    Internal medicine subspecialty8.83 (3.13).41
    General pediatrics−0.76 (3.99).96
    Pediatric subspecialty−14.01 (3.84).28
    General surgery−1.23 (4.4).93
    General surgery subspecialty−6.76 (3.71).59
    Orthopedic surgery−9.47 (4.24).51
    Hours worked per week, for each additional hour0.98 (0.05)<.001
    Nights on call per week, for each additional call3.26 (0.37).01
    Primary practice setting (reference: private practice)
    Academic medical center−0.62 (1.77).92
    Veterans hospital2.22 (5.22).9
    Active military practice−31.44 (6.9).18
    Other4.5 (2.07).52
    Relationship status (reference: single)
    Married14.12 (2.24).06
    Partnered19.95 (3.93).13
    Widowed/widower46.33 (8.61).11

    aSUS: System Usability Scale.

    In the first ordinary least squares regression model of the mediation analysis (Figure 5), higher SUS was significantly related to lower PTL scores (β=−.537, 95% CI −0.755 to −0.319; SE 0.111; P<.001). In the second logistic regression model, which included SUS and PTL as predictor variables of burnout, both SUS and PTL were significantly associated with burnout (OR 0.978, 95% CI 0.972 to 0.985 and OR 1.009, 95% CI 1.007 to 1.011, respectively). The bootstrap confidence intervals derived from 5000 samples indicated that the indirect effect of PTL on the association between SUS and burnout was significant (effect=−0.005, 95% CI −0.007 to −0.003). From this result, higher SUS was associated with lower PTL, which was associated with lower odds of burnout.

    ‎
    Figure 5. Mediation analysis results showing the quantitative relationships between provider task load (PTL) as a mediator between electronic health record (EHR) usability and professional burnout. The values for EHR usability to professional burnout display the association between the System Usability Scale (SUS) and Maslach Burnout Inventory (MBI) after controlling for PTL and, parenthetically, the association between SUS and MBI when PTL is not statistically controlled for. *log-odds, since this is a logistic regression.
    View this figure

    Principal Results

    The findings of this national study indicate that physician-perceived EHR usability scores are strongly associated with physician daily task load. After adjusting for multiple personal and professional characteristics, individual physicians’ composite PTL scores were 0.55 points lower for each 1 point more favorable EHR usability, as measured by the SUS. The mediation analysis supports the proposed conceptual framework in which task load acts as a mediator between EHR usability and professional burnout. Despite the strong statistically significant associations found, the amount of variability in PTL as a measure of workload explained by EHR usability was small (R2=0.03). This finding indicates that factors other than EHR usability appear to be the primary drivers of physician workload as assessed by the mental demand, physical demand, temporal demand, and effort required subscales of the NASA-TLX. In the primary analysis of this survey, age, gender, medical specialty, hours worked per week, number of nights on call per week, and practice setting were all associated with PTL as a measure of workload [27]. To put these findings in context, a 10-point lower PTL was associated with 30% lower odds of burnout in the primary analysis [27]. An 18-point higher average SUS score (associated with a 10-point lower PTL in this analysis) would give the EHR a grade of D (instead of F) and be above the score of Microsoft Excel (Microsoft Corporation, 2009) [7].

    Exploring this relationship and variation in EHR usability and PTL as a measure of workload by specialty reveals that some specialties with higher task load rated their EHRs more favorably (eg, anesthesiology and dermatology) while other specialties associated with lower task load (eg, orthopedic surgery and family medicine) rated their EHRs less favorably. This finding suggests the relationship between EHR usability and task load may not be due to physicians in specialties with higher task load rating their EHR less favorably. For example, anesthesiologists may benefit from the EHR’s ability to provide certain information rapidly in the operating room, yet still have higher task load due to factors in the anesthesiology clinical context that are not EHR-related. However, only three specialties (emergency medicine, anesthesiology, and psychiatry) remained predictive of task load after controlling for EHR usability, gender, age, hours worked per week, and number of nights on call per week. The analysis is likely underpowered to explain specific differences between individual specialties. On balance, SUS appears to have only a small influence on task load, although the effect of SUS on PTL varies by specialty.

    Comparison With Prior Work

    It is also notable that in this analysis, lower EHR usability and higher workload were both associated with the odds of burnout after controlling for multiple personal and professional characteristics, suggesting that these are distinct domains that both represent potential improvement targets to reduce physician burnout. A study of 46 participants’ perception of the usability and task load of three popular websites found no association between usability and task load as assessed by the SUS and NASA-TLX [49]. Important differences between that study and the present study were that SUS scores were only in the high range (compared to the EHR) and that they measured both domains for individual tasks as opposed to in aggregate as we have here. The primary analysis of this national survey of 5197 physicians found that age, gender, relationship status, hours worked per week, and practicing certain medical specialties were all associated with the odds of burnout [32]. In this analysis, gender was not significantly predictive of the odds of burnout. Given this finding and the growing literature on the association of gender on physician burnout [32,50,51], future work should explore the interactions between these variables and further evaluate how EHR usability and task load vary by personal and professional characteristics. For example, a mixed-methods assessment of 25 intensive care physicians running simulated cases found gender-based differences in perceived EHR workload stress, satisfaction, and usability as assessed by the NASA-TLX and SUS [52]. Modest redesign of computerized reminders was associated with improved usability and decreased mental workload for 16 nurses in a simulated environment as assessed by the NASA-TLX [53]. A randomized crossover trial of 7 pediatric surgeons reported improved SUS scores and lower cognitive workload scores when order sets were systematically developed [54]. Future work could explore the relationship between EHR usability, specific task load, and professional burnout across and between different medical specialty disciplines and EHR vendor products as well as further differentiate the contribution of administrative burden to these areas by specialty. Policy makers should also explore the potential savings to the health care system that might be realized by improving EHR usability and how removing required administrative tasks that are not intrinsic to the practice of medicine could decrease task load and burnout.

    Limitations

    This study is subject to several limitations. First, although the association between SUS and PTL scores is statistically significant, cross-sectional data cannot prove causation or the potential direction of effect. Second, with all survey research, response bias and representativeness of the study sample are potential limitations. An assessment for response bias was employed in this study, including evaluation of a random sample of nonresponders to the initial survey who participated in an incentivized follow-up survey. Although women were more likely to participate in the EHR usability subsurvey, participants were generally representative of US physicians with respect to age, years in practice, and prevalence of burnout. Third, self-reported data is subject to limitations [55,56]. Physician perceptions of usability and workload may not accurately reflect reality [20]. For example, one dimension of usability is error tolerance, a system’s ability to prevent or recover from an error; due to poor usability, EHR users may not be aware of errors they make in the EHR [11,12]. In addition, individual respondents could have a tendency to provide similar responses to different questions, which could contribute to a common method bias [57]. Fourth, although the SUS and NASA-TLX are considered the industry standards to assess technology usability and task load, they are intended to assess a specific task within a single system, not an entire class of software or an entire day of clinical work. Instead, this analysis provides a global assessment of EHR usability and provider task load and, therefore, cannot account for differences between specific tasks or vendors’ products and may be subject to recall bias. Fifth, physician respondents could conflate EHR usability issues with regulatory and clerical demands that manifest in the EHR but may be unrelated to EHR user interface design [21,22]. If that is the case, the association of clerical burden with task load and burnout could be stronger than that of EHR usability [7,16].

    Despite these limitations, as the first national study exploring the relationship between EHR usability and workload in practice across medical specialties and practice settings, this analysis adds an important dimension to existing knowledge about factors associated with PTL and physician burnout. Our findings are consistent with multiple smaller studies of task load in both simulated [29,30] and real world settings [58,59] suggesting that more usable EHR interfaces are associated with lower cognitive load. Technology changes physicians’ information gathering and reasoning strategies, thereby shaping cognitive behavior [60]. Cognitive load theory should be taken into consideration to design EHR interfaces and workflows that meet users’ needs and are free from unnecessary extraneous cognitive load, such that the EHR becomes part of a physician work environment more closely resembling a “manageable cockpit” that is streamlined, ergonomic, and safe [10,61,62].

    Conclusion

    As assessed by US physicians using standardized metrics of technology and workload, a strong association was observed between EHR usability and task load, with more favorable usability associated with lower task load. Although EHR usability was significantly related to workload, the effect size was small, indicating that factors other than the EHR appear to be the primary drivers of workload. Both outcomes were also associated with the odds of burnout. Efforts to address physician burnout should attend to both improving EHR usability and addressing other drivers of task load, particularly extraneous load. Improving EHR usability while decreasing extraneous task load has the potential to allow practicing physicians more available working memory for medical decision making and patient communication. Specific areas to target could include consolidating the display of related information, reducing redundancy of information, and increasing standardization [63,64].

    Acknowledgments

    The authors would like to thank Bidisha Nath, MD, MPH, for her contribution to the literature search and manuscript coordination. Funding for this study was provided by the Stanford Medicine WellMD Center, the American Medical Association, and the Mayo Clinic Department of Medicine Program on Physician Well-Being. ERM is supported in part by two American Medical Association Practice Transformation Initiatives and the National Institute on Drug Abuse of the National Institutes of Health under Award Number UH3DA047003. The content is solely the responsibility of the authors and does not necessarily represent the official views of the American Medical Association or the National Institutes of Health.

    Authors' Contributions

    ERM, EH, CAS, and TS significantly contributed to the conception and design of the study. TS, LND, CAS, MTT, and CPW acquired the data. ERM, EH, HW, and TS analyzed the data. ERM drafted the initial manuscript. All authors were involved in data interpretation, revised the manuscript, and approved the final version submitted for publication. HW and TS had access to the data in the study and take responsibility for data integrity and accuracy. ERM takes responsibility for all aspects of the work.

    Conflicts of Interest

    None declared.

    1. Bohman B, Dyrbye L, Sinsky C, Linzer M, Olson K, Babbott S, et al. Physician well-being: the reciprocity of practice efficiency, culture of wellness, and personal resilience.: NEJM Catalyst; 2017.   URL: https://catalyst.nejm.org/doi/full/10.1056/CAT.17.0429 [accessed 2020-12-07]
    2. Olson K, Sinsky C, Rinne S, Long T, Vender R, Mukherjee S, et al. Cross-sectional survey of workplace stressors associated with physician burnout measured by the Mini-Z and the Maslach Burnout Inventory. Stress Health 2019 Apr;35(2):157-175. [CrossRef] [Medline]
    3. Gardner RL, Cooper E, Haskell J, Harris DA, Poplau S, Kroth PJ, et al. Physician stress and burnout: the impact of health information technology. J Am Med Inform Assoc 2019 Feb 01;26(2):106-114. [CrossRef] [Medline]
    4. Tai-Seale M, Dillon EC, Yang Y, Nordgren R, Steinberg RL, Nauenberg T, et al. Physicians' Well-Being Linked To In-Basket Messages Generated By Algorithms In Electronic Health Records. Health Aff (Millwood) 2019 Jul;38(7):1073-1078. [CrossRef] [Medline]
    5. Adler-Milstein J, Zhao W, Willard-Grace R, Knox M, Grumbach K. Electronic health records and burnout: Time spent on the electronic health record after hours and message volume associated with exhaustion but not with cynicism among primary care clinicians. J Am Med Inform Assoc 2020 Apr 01;27(4):531-538. [CrossRef] [Medline]
    6. Micek MA, Arndt B, Tuan W, Trowbridge E, Dean SM, Lochner J, et al. Physician Burnout and Timing of Electronic Health Record Use. ACI Open 2020 Feb 06;04(01):e1-e8. [CrossRef]
    7. Melnick ER, Dyrbye LN, Sinsky CA, Trockel M, West CP, Nedelec L, et al. The Association Between Perceived Electronic Health Record Usability and Professional Burnout Among US Physicians. Mayo Clin Proc 2020 Mar;95(3):476-487 [FREE Full text] [CrossRef] [Medline]
    8. Bevan N, Carter J, Harker S. Human-Computer Interaction Design and Evaluation. ISO 9241-11 Revised: What Have We Learnt About Usability Since 1998?: Springer International Publishing; 2015:143-151.
    9. Zahabi M, Kaber DB, Swangnetr M. Usability and Safety in Electronic Medical Records Interface Design: A Review of Recent Literature and Guideline Formulation. Hum Factors 2015 Aug;57(5):805-834. [CrossRef] [Medline]
    10. Gellert G, Webster S, Gillean J, Melnick E, Kanzaria H. Should US doctors embrace electronic health records? BMJ 2017 Dec 24;356:j242. [Medline]
    11. Ratwani RM, Savage E, Will A, Fong A, Karavite D, Muthu N, et al. Identifying Electronic Health Record Usability And Safety Challenges In Pediatric Settings. Health Aff (Millwood) 2018 Dec;37(11):1752-1759. [CrossRef] [Medline]
    12. Ratwani RM, Savage E, Will A, Arnold R, Khairat S, Miller K, et al. A usability and safety analysis of electronic health records: a multi-center study. J Am Med Inform Assoc 2018 Sep 01;25(9):1197-1201. [CrossRef] [Medline]
    13. Howe JL, Adams KT, Hettinger AZ, Ratwani RM. Electronic Health Record Usability Issues and Potential Contribution to Patient Harm. JAMA 2018 Dec 27;319(12):1276-1278 [FREE Full text] [CrossRef] [Medline]
    14. Friedberg MW, Chen PG, Van Busum KR, Aunon F, Pham C, Caloyeras J, et al. Factors Affecting Physician Professional Satisfaction and Their Implications for Patient Care, Health Systems, and Health Policy. Rand Health Q 2014;3(4):1 [FREE Full text] [Medline]
    15. Babbott S, Manwell LB, Brown R, Montague E, Williams E, Schwartz M, et al. Electronic medical records and physician stress in primary care: results from the MEMO Study. J Am Med Inform Assoc 2014 Feb;21(e1):e100-e106 [FREE Full text] [CrossRef] [Medline]
    16. Shanafelt TD, Dyrbye LN, Sinsky C, Hasan O, Satele D, Sloan J, et al. Relationship Between Clerical Burden and Characteristics of the Electronic Environment With Physician Burnout and Professional Satisfaction. Mayo Clin Proc 2016 Jul;91(7):836-848. [CrossRef] [Medline]
    17. Walji MF, Kalenderian E, Piotrowski M, Tran D, Kookal KK, Tokede O, et al. Are three methods better than one? A comparative assessment of usability evaluation methods in an EHR. Int J Med Inform 2014 May;83(5):361-367 [FREE Full text] [CrossRef] [Medline]
    18. Ratwani R, Zachary Hettinger A, Kosydar A, Fairbanks R, Hodgkins M. A framework for evaluating electronic health record vendor user-centered design and usability testing processes. J Am Med Inform Assoc 2017 Apr 01;24(e1):e35-e39 [FREE Full text] [CrossRef] [Medline]
    19. Ratwani RM, Reider J, Singh H. A Decade of Health Information Technology Usability Challenges and the Path Forward. JAMA 2019 Feb 26;321(8):743-744. [CrossRef] [Medline]
    20. Ellsworth MA, Dziadzko M, O'Horo JC, Farrell AM, Zhang J, Herasevich V. An appraisal of published usability evaluations of electronic health records via systematic review. J Am Med Inform Assoc 2017 Jan;24(1):218-226. [CrossRef] [Medline]
    21. Downing NL, Bates DW, Longhurst CA. Physician Burnout in the Electronic Health Record Era: Are We Ignoring the Real Cause? Ann Intern Med 2018 Jul 03;169(1):50-51. [CrossRef] [Medline]
    22. Tutty MA, Carlasare LE, Lloyd S, Sinsky CA. The complex case of EHRs: examining the factors impacting the EHR user experience. J Am Med Inform Assoc 2019 Jul 01;26(7):673-677 [FREE Full text] [CrossRef] [Medline]
    23. Sweller J. Cognitive Load Theory. In: Psychology of Learning and Motivation. San Diego, CA: Elsevier Academic Press; 2011:37-76.
    24. Paas F, Renkl A, Sweller J. Cognitive Load Theory and Instructional Design: Recent Developments. Educational Psychologist 2003 Mar;38(1):1-4 [FREE Full text] [CrossRef]
    25. DeLeeuw K, Mayer R. A comparison of three measures of cognitive load: Evidence for separable measures of intrinsic, extraneous, and germane load. Journal of Educational Psychology 2008;100(1):223-234 [FREE Full text] [CrossRef]
    26. Staal M. Stress, Cognition, and Human Performance: A Literature Review and Conceptual Framework. 2004.   URL: https://ntrs.nasa.gov/search.jsp?R=20060017835 [accessed 2020-12-07]
    27. Harry E, Sinsky C, Dyrbye LN, Makowski MS, Trockel M, Tutty M, et al. Physician Task Load and the Risk of Burnout Among US Physicians in a National Survey. Jt Comm J Qual Patient Saf 2020 Oct 04;S1553-7250(20):30246-30244. [CrossRef] [Medline]
    28. Hart SG. Nasa-Task Load Index (NASA-TLX); 20 Years Later. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting. 2016 Nov 05 Presented at: Human Factors and Ergonomics Society Annual Meeting; 2006; Washington, DC p. 904-908   URL: http://journals.sagepub.com/doi/abs/10.1177/154193120605000909 [CrossRef]
    29. Mazur LM, Mosaly PR, Moore C, Marks L. Association of the Usability of Electronic Health Records With Cognitive Workload and Performance Levels Among Physicians. JAMA Netw Open 2019 Apr 05;2(4):e191709 [FREE Full text] [CrossRef] [Medline]
    30. Pollack AH, Pratt W. Association of Health Record Visualizations With Physicians' Cognitive Load When Prioritizing Hospitalized Patients. JAMA Netw Open 2020 Jan 03;3(1):e1919301 [FREE Full text] [CrossRef] [Medline]
    31. Flanagan ME, Militello LG, Rattray NA, Cottingham AH, Frankel RM. The Thrill Is Gone: Burdensome Electronic Documentation Takes Its Toll on Physicians' Time and Attention. J Gen Intern Med 2019 Jul;34(7):1096-1097 [FREE Full text] [CrossRef] [Medline]
    32. Shanafelt TD, West CP, Sinsky C, Trockel M, Tutty M, Satele DV, et al. Changes in Burnout and Satisfaction With Work-Life Integration in Physicians and the General US Working Population Between 2011 and 2017. Mayo Clin Proc 2019 Sep;94(9):1681-1694 [FREE Full text] [CrossRef] [Medline]
    33. Brooke J. In: Jordan PW, Weerdmeester BA, McClelland IL, editors. SUS: A Quick and Dirty Usability Scale. London: Taylor and Francis; 1996.
    34. Bangor A, Kortum PT, Miller JT. An Empirical Evaluation of the System Usability Scale. International Journal of Human-Computer Interaction 2008 Jul 30;24(6):574-594. [CrossRef]
    35. Brooke J. SUS: a retrospective. Journal of Usability Studies 2013;8(2):29-40 [FREE Full text]
    36. Lewis J, Sauro J. The Factor Structure of the System Usability Scale. In: Human Centered Design. Berlin, Heidelberg: Springer; 2009:94-103.
    37. Tubbs-Cooley HL, Mara CA, Carle AC, Gurses AP. The NASA Task Load Index as a measure of overall workload among neonatal, paediatric and adult intensive care nurses. Intensive Crit Care Nurs 2018 Jun;46:64-69. [CrossRef] [Medline]
    38. Rutledge T, Stucky E, Dollarhide A, Shively M, Jain S, Wolfson T, et al. A real-time assessment of work stress in physicians and nurses. Health Psychol 2009 Mar;28(2):194-200. [CrossRef] [Medline]
    39. Weigl M, Müller A, Vincent C, Angerer P, Sevdalis N. The association of workflow interruptions and hospital doctors' workload: a prospective observational study. BMJ Qual Saf 2012 May;21(5):399-407. [CrossRef] [Medline]
    40. Rölfing JD, Nørskov JK, Paltved C, Konge L, Andersen SAW. Failure affects subjective estimates of cognitive load through a negative carry-over effect in virtual reality simulation of hip fracture surgery. Adv Simul (Lond) 2019;4:26 [FREE Full text] [CrossRef] [Medline]
    41. Maslach C, Jackson S, Leiter M. Maslach Burnout Inventory Manual. Palo Alto, CA: Consulting Psychologists Press; 1997.
    42. Rafferty JP, Lemkau JP, Purdy RR, Rudisill JR. Validity of the Maslach Burnout Inventory for family practice physicians. J Clin Psychol 1986 May;42(3):488-492. [CrossRef] [Medline]
    43. Lee RT, Ashforth BE. A meta-analytic examination of the correlates of the three dimensions of job burnout. J Appl Psychol 1996 Apr;81(2):123-133. [CrossRef] [Medline]
    44. Leiter M, Durup J. The discriminant validity of burnout and depression: A confirmatory factor analytic study. Anxiety, Stress, & Coping 2007 May 29;7(4):357-373. [CrossRef]
    45. Shanafelt TD, Bradley KA, Wipf JE, Back AL. Burnout and self-reported patient care in an internal medicine residency program. Ann Intern Med 2002 Mar 05;136(5):358-367. [CrossRef] [Medline]
    46. Thomas NK. Resident burnout. JAMA 2004 Dec 15;292(23):2880-2889. [CrossRef] [Medline]
    47. Rosen IM, Gimotty PA, Shea JA, Bellini LM. Evolution of sleep quantity, sleep deprivation, mood disturbances, empathy, and burnout among interns. Acad Med 2006 Jan;81(1):82-85. [CrossRef] [Medline]
    48. Hayes A. Introduction to Mediation, Moderation, and Conditional Process Analysis. In: A Regression-Based Approach Second Edition. New York: Guilford Press; 2017.
    49. Longo L. Experienced mental workload, perception of usability, their interaction and impact on task performance. PLoS One 2018;13(8):e0199661 [FREE Full text] [CrossRef] [Medline]
    50. Shanafelt TD, Boone S, Tan L, Dyrbye LN, Sotile W, Satele D, et al. Burnout and satisfaction with work-life balance among US physicians relative to the general US population. Arch Intern Med 2012 Oct 08;172(18):1377-1385. [CrossRef] [Medline]
    51. Shanafelt TD, Hasan O, Dyrbye LN, Sinsky C, Satele D, Sloan J, et al. Changes in Burnout and Satisfaction With Work-Life Balance in Physicians and the General US Working Population Between 2011 and 2014. Mayo Clin Proc 2015 Dec;90(12):1600-1613. [CrossRef] [Medline]
    52. Khairat S, Coleman C, Ottmar P, Bice T, Koppel R, Carson SS. Physicians' gender and their use of electronic health records: findings from a mixed-methods usability study. J Am Med Inform Assoc 2019 Dec 01;26(12):1505-1514 [FREE Full text] [CrossRef] [Medline]
    53. Saleem JJ, Patterson ES, Militello L, Anders S, Falciglia M, Wissman JA, et al. Impact of clinical reminder redesign on learnability, efficiency, usability, and workload for ambulatory clinic nurses. J Am Med Inform Assoc 2007;14(5):632-640 [FREE Full text] [CrossRef] [Medline]
    54. Avansino J, Leu MG. Effects of CPOE on provider cognitive workload: a randomized crossover trial. Pediatrics 2012 Sep;130(3):e547-e552. [CrossRef] [Medline]
    55. Donaldson S, Grant-Vallone E. Understanding self-report bias in organizational behavior research. J Bus Psychol 2002;17(2):245-260. [CrossRef]
    56. Hunt M, Auriemma J, Cashaw ACA. Self-report bias and underreporting of depression on the BDI-II. J Pers Assess 2003 Feb;80(1):26-30. [CrossRef] [Medline]
    57. Conway JM, Lance CE. What Reviewers Should Expect from Authors Regarding Common Method Bias in Organizational Research. J Bus Psychol 2010 May 22;25(3):325-334. [CrossRef]
    58. Ariza F, Kalra D, Potts HW. How Do Clinical Information Systems Affect the Cognitive Demands of General Practitioners? Usability Study with a Focus on Cognitive Workload. J Innov Health Inform 2015 Nov 20;22(4):379-390 [FREE Full text] [CrossRef] [Medline]
    59. Agha Z. Quantifying electronic medical record usability to improve clinical workflow-final report (Prepared by the Veterans Medical Research Foundation under Grant No R01 HS021290). Rockville, MD: Agency for Healthcare Research and Quality; 2017.
    60. Patel VL, Kushniruk AW, Yang S, Yale JF. Impact of a computer-based patient record system on data collection, knowledge organization, and reasoning. J Am Med Inform Assoc 2000;7(6):569-585 [FREE Full text] [CrossRef] [Medline]
    61. Ray J, Ratwani R, Sinsky C, Frankel R, Friedberg M, Powsner S, et al. Six habits of highly successful health information technology: powerful strategies for design and implementation. J Am Med Inform Assoc 2019 Oct 01;26(10):1109-1114 [FREE Full text] [CrossRef] [Medline]
    62. Sinsky C, Privitera M. Creating a "Manageable Cockpit" for Clinicians: A Shared Responsibility. JAMA Intern Med 2018 Jun 01;178(6):741-742. [CrossRef] [Medline]
    63. de Jong T. Cognitive load theory, educational research, and instructional design: some food for thought. Instr Sci 2009 Aug 27;38(2):105-134. [CrossRef]
    64. Sinsky CA, Rule A, Cohen G, Arndt BG, Shanafelt TD, Sharp CD, et al. Metrics for assessing physician activity using electronic health record log data. J Am Med Inform Assoc 2020 Apr 01;27(4):639-643 [FREE Full text] [CrossRef] [Medline]

    ‎
    EHR: electronic health record
    NASA-TLX: National Aeronautics and Space Administration Task Load Index
    PTL: provider task load
    SUS: System Usability Scale

    Edited by G Eysenbach, R Kukafka; submitted 12.08.20; peer-reviewed by S Rinne, R Hilliard, J Brooke, C Son; comments to author 02.09.20; revised version received 15.10.20; accepted 07.12.20; published 22.12.20

    Copyright

    ©Edward R Melnick, Elizabeth Harry, Christine A Sinsky, Liselotte N Dyrbye, Hanhan Wang, Mickey Todd Trockel, Colin P West, Tait Shanafelt. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 22.12.2020.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.