This paper is in the following e-collection/theme issue:

Human Factors and Usability Case Studies (787) Physician and Healthcare Worker Burnout (109) Adoption and Change Management of eHealth Systems (712) Occupational Health and Ergonomics/Prevention at the Workplace (238) Electronic Health Records (1053) Clinical Informatics (1022) Design and Usability of Clinical Software and EHRs (268)

Published on in Vol 23, No 7 (2021): July

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/26817, first published .
Frustration With Technology and its Relation to Emotional Exhaustion Among Health Care Workers: Cross-sectional Observational Study

Frustration With Technology and its Relation to Emotional Exhaustion Among Health Care Workers: Cross-sectional Observational Study

Frustration With Technology and its Relation to Emotional Exhaustion Among Health Care Workers: Cross-sectional Observational Study

Authors of this article:

Daniel S Tawfik1 Author Orcid Image ;   Amrita Sinha1 Author Orcid Image ;   Mohsen Bayati2, 3 Author Orcid Image ;   Kathryn C Adair4 Author Orcid Image ;   Tait D Shanafelt5, 6 Author Orcid Image ;   J Bryan Sexton4, 7 Author Orcid Image ;   Jochen Profit1, 8 Author Orcid Image
Article Authors Cited by (15) Tweetations (16) Metrics

    Original Paper

    1Department of Pediatrics, Stanford University School of Medicine, Palo Alto, CA, United States

    2Operations, Information, and Technology, Stanford University Graduate School of Business, Stanford, CA, United States

    3Department of Biomedical Informatics, Stanford University School of Medicine, Stanford, CA, United States

    4Duke Center for Healthcare Safety and Quality, Duke University Health System, Durham, NC, United States

    5Department of Medicine, Stanford University School of Medicine, Stanford, CA, United States

    6WellMD Center, Stanford University School of Medicine, Stanford, CA, United States

    7Department of Psychiatry, Duke University School of Medicine, Duke University Health System, Durham, NC, United States

    8California Perinatal Quality Care Collaborative, Palo Alto, CA, United States

    Corresponding Author:

    Daniel S Tawfik, MS, MD

    Department of Pediatrics

    Stanford University School of Medicine

    770 Welch Road

    Suite 435

    Palo Alto, CA, 94304

    United States

    Phone: 1 650 723 9902

    Email: dtawfik@stanford.edu


    Background: New technology adoption is common in health care, but it may elicit frustration if end users are not sufficiently considered in their design or trained in their use. These frustrations may contribute to burnout.

    Objective: This study aimed to evaluate and quantify health care workers’ frustration with technology and its relationship with emotional exhaustion, after controlling for measures of work-life integration that may indicate excessive job demands.

    Methods: This was a cross-sectional, observational study of health care workers across 31 Michigan hospitals. We used the Safety, Communication, Operational Reliability, and Engagement (SCORE) survey to measure work-life integration and emotional exhaustion among the survey respondents. We used mixed-effects hierarchical linear regression to evaluate the relationship among frustration with technology, other components of work-life integration, and emotional exhaustion, with adjustment for unit and health care worker characteristics.

    Results: Of 15,505 respondents, 5065 (32.7%) reported that they experienced frustration with technology on at least 3-5 days per week. Frustration with technology was associated with higher scores for the composite Emotional Exhaustion scale (r=0.35, P<.001) and each individual item on the Emotional Exhaustion scale (r=0.29-0.36, P<.001 for all). Each 10-point increase in the frustration with technology score was associated with a 1.2-point increase (95% CI 1.1-1.4) in emotional exhaustion (both measured on 100-point scales), after adjustment for other work-life integration items and unit and health care worker characteristics.

    Conclusions: This study found that frustration with technology and several other markers of work-life integration are independently associated with emotional exhaustion among health care workers. Frustration with technology is common but not ubiquitous among health care workers, and it is one of several work-life integration factors associated with emotional exhaustion. Minimizing frustration with health care technology may be an effective approach in reducing burnout among health care workers.

    J Med Internet Res 2021;23(7):e26817

    doi:10.2196/26817

    Keywords

    frustration with technology; emotional exhaustion; professional burnout; work-life integration; biomedical technology; work-life balance; user-centered design; electronic health records; medical informatics applications; health information systems 


    Technological innovation has expanded the horizons of medicine in recent decades [1], but these advances have been accompanied by increased clerical burden among physicians and other health care providers. Electronic health record (EHR) adoption has improved emphasis on quality monitoring, billing accuracy, and research, but it has often resulted in redundant documentation and inefficient workflows [2-6]. As a result, physicians have greatly increased interactions with a variety of health care information technologies (HITs), with their interactions commonly encompassing a combination of direct clinical and nonclinical goals [7]. For example, outpatient physicians now spend up to twice as much time interacting with EHRs as they do with patients [8,9].

    Concurrently, symptoms of burnout, including emotional exhaustion, have risen to epidemic proportions, now affecting over 500,000 US physicians and costing the US health care system $4-5 billion annually [10-12]. Fundamentally, medicine is a human-centered endeavor, and technology can enable or distract from this focus. Unfortunately, almost 50% of physicians believe they spend an excessive amount of time on clerical tasks, and many physicians believe EHRs contribute to burnout [13-15]. High EHR task load, time spent on EHRs, and automated in-basket messages have been associated with greater emotional exhaustion [16-21]. Furthermore, the National Academy of Medicine recognizes the associations of administrative burden, technology usability, and time pressure on burnout (all of which may be attributed as EHR or HIT factors), yet the mechanisms underlying these associations are not well described [22].

    One potential mechanism relating to HIT use and burnout may be frustration with technology—an emotional reaction to an obstacle preventing the fulfillment of a perceived need [23]. HIT is at risk for inducing frustration among health care workers, by virtue of its complex interfaces, frequent updates as capabilities improve, and deployment within a high-stakes environment that provides limited opportunity for dedicated training [24-26]. If frustration with technology contributes to emotional exhaustion, this would indicate an opportunity to prioritize reducing frustration through better design, training, and implementation as a mechanism to combat burnout [27].

    We sought to quantify health care workers’ frustration with technology and its relationship with emotional exhaustion, after controlling for measures of work-life integration (WLI) that may indicate excessive job demands. We hypothesized that higher frustration with technology corresponds to higher emotional exhaustion.


    Study Design and Population

    This cross-sectional, observational study is a secondary analysis of the Safety, Communication, Operational Reliability, and Engagement (SCORE) survey distributed via email through the Michigan Health and Hospital Association Keystone Center in 2015 as part of their routine patient safety and quality measurement, allowing a single response per user [28,29]. All employees working 0.5 full-time equivalents or higher in any Michigan hospital for 4 consecutive weeks prior to the survey administration were invited to participate. Confidentiality was assured to the respondents, participation in the survey was voluntary, and no incentives were offered. No questions were randomized or adapted to responses in real-time, no completeness check was enforced, and reviewing of answers was allowed. No cookies, internet protocol address checks, or log files were used to exclude responses. All surveys that contained responses to the scales measuring WLI and emotional exhaustion (described below) were analyzed. Surveys with “not applicable” or missing responses to either of these two scales were excluded. This study was not considered human subjects research by Stanford University and was approved by the institutional review board at Duke University Medical Center (Pro00033155).

    Measures

    The SCORE survey measures common workplace issues and work setting norms [28,29], including WLI, Emotional Exhaustion, Local Leadership, Learning Environment, Burnout Climate, Teamwork Climate, and Safety Climate scales [28,30,31]. SCORE also contains workforce engagement subscales and demographic questions (ie, number of years in specialty, job position, shift type, and length).

    WLI and Frustration With Technology

    We assessed WLI using a scale primarily focused on tangible frequencies of activities reflecting the interaction between work and personal responsibilities [28,29,32,33]. Each WLI item begins with the phrase “During the past week, how often did this occur?” The WLI items are as follows: (1) skipped a meal, (2) ate a poorly balanced meal, (3) worked through a shift with no breaks, (4) arrived home late from work, (5) had difficulty sleeping, (6) slept less than 5 hours in a night, and (7) changed personal/family plans due to work.

    This WLI scale was originally validated as a 7-item scale as described above (Cronbach α=.79 in a validation study [29] and α=.81 in the current data set) and later updated with an additional eighth item assessing the frequency that one “felt frustrated with technology” as a key indicator of the ability of technology to facilitate efficient workflows and minimize conflicts between work and personal responsibilities (Cronbach α=.83 in a validation study [28] and α=.81 in the current data set). The full 8-item WLI scale is the current standard, but for the purposes of this study, we calculated WLI scores using the previously validated 7-item WLI scale, separately considering the additional item relating to frustration with technology as our primary independent variable of interest.

    Each item spans a 4-point Likert scale (“rarely or none of the time,” “some or a little of the time,” “occasionally or a moderate amount of time,” and “all of the time”). For ease of interpretation, when assessing global correlations, we transposed the mean score of the 7 WLI items onto a scale of 0 to 100 and reversed the valence, with 100 indicating a favorable score (high WLI) and 0 indicating a poor score (low WLI). We similarly transposed frustration with technology onto a scale of 0 to 100, with 100 indicating a poor score (high frustration) and 0 indicating a favorable score (low frustration). For a secondary analysis comparing aggregated scores by work setting, we divided work settings into four quartiles based on the mean frustration with technology score.

    Emotional Exhaustion

    The 5-item Emotional Exhaustion scale (Cronbach α=.92) of the SCORE survey is composed of 4 items adapted from the Emotional Exhaustion subscale of the Maslach Burnout Inventory (r=0.96-0.98 with the 9-item scale) and a fifth item developed to align with the job demands-resources model of burnout [34-36]. This scale has been validated for use among health care workers as a burnout metric (eg, “Events in this work setting affect my life in an emotionally unhealthy way”), with demonstrated content, internal consistency, and consequence validity for this purpose [28-31,37]. Responses span a 5-point Likert scale from “disagree strongly” to “agree strongly.” We calculated each individual’s emotional exhaustion score by transposing the mean score of the five items onto a scale of 0 to 100, in line with a previous study [38].

    The complete SCORE survey alongside derivation, scoring procedures, and reliability data for each of its scales is available on the internet [39].

    Classifications

    Individual responses were also categorized by their specific work setting, based on self-reported work location, such as St. Elsewhere Hospital 5 South, Pleasantview Pediatrics Clinic, and Mercy Health Systems Billing Department (fictional names). Each work setting thus reflects a grouping of respondents who work together as a team, regardless of each respondent’s individual role. Work settings were classified as direct patient care (clinical) or indirect patient care (nonclinical, including administrative or billing departments). Work settings providing direct patient care were further classified as either intensive/emergency or acute; surgical or medical; and inpatient, outpatient, or mixed inpatient/outpatient. To maintain confidentiality and reduce risk of response bias from small samples, respondents from work settings with fewer than 5 total respondents were excluded from correlation and regression analyses.

    Statistical Analysis

    Descriptive statistics are presented as mean and SD values, or frequencies and percentages as appropriate. We compared group means by performing two-tailed t tests. We evaluated agreement in frustration with technology scores within work settings by using weighted Cohen kappa agreement analysis. As a first step insight-generation analysis, we evaluated Pearson correlations among survey items by using mean scores aggregated by work setting and weighted by the number of responses, avoiding the assumptions of identically distributed observations across work settings and of nested results (eg, health care workers within work settings) [40]. In our primary analysis, we evaluated the independent relations between frustration with technology score, other measures of WLI, and the outcome of emotional exhaustion by using a single mixed-effect generalized linear regression model, with work setting as random intercept, and job position, number of years in specialty, and work setting classifiers as fixed effects. We also performed, as a sensitivity analysis, a secondary validation to control for any available potential confounding factor that the primary regression may have omitted. We leveraged the statistical machine learning method lasso [41,42] to select relevant covariates from a large set of 36 potential covariates and re-ran our regression. Analyses were performed using Stata/IC software (version 15.1; StataCorp LLC). We used simple Bonferroni correction to account for multiple comparisons. With a total of 11 comparisons (8 independent items in the regression model, plus 3 t tests of adjacent quartiles) and a desired family-wise error rate of <0.05, two-tailed P values <.0045 were considered statistically significant.


    Of the 23,853 distributed surveys, 16,797 were returned (70.4% response rate). Of these, 915 indicated that technology use was “not applicable” to them and 377 had incomplete responses, resulting in 15,505 complete responses for further analysis. Descriptive statistics are presented in Table 1. The most frequently represented positions among all respondents (N=15,505) were nurses (n=4316, 27.8%), technologists and technicians (n=1890, 12.2%), and administrative support personnel (n=1800, 11.6%). The majority of respondents (n=10,284, 66.3%) reported 5 or more years in their current specialty. Nearly half (n=7286, 47.0%) of the respondents were from units not providing direct patient care, 9.0% (n=1398) were from units providing intensive or emergency care, and 10.0% (n=1559) were from units providing surgical care. Of the 1140 work settings represented, 818 (71.8 %) had 5 or more unique respondents and were included in regression analyses.

    Table 1. Characteristics of survey respondents (N=15,505) from 1140 different work settings (818 work settings had 5 or more respondents).
    CharacteristicParticipant, n (%)
    Position

    		Nurse4316 (27.8)

    		Technologist/Technician1890 (12.2)

    		Admin support1800 (11.6)

    		Admin/Manager1238 (8.0)

    		Clinical support839 (5.4)

    		Therapist696 (4.5)

    		Nurses’ aide626 (4.0)

    		Physician431 (2.9)

    		Environmental support288 (1.9)

    		Pharmacist226 (1.5)

    		Physician assistant105 (0.7)

    		Other3050 (19.7)
    Years in specialty

    		0-23056 (20.0)

    		3-41933 (12.7)

    		5-103374 (22.1)

    		11-203684 (24.1)

    		21 or more3226 (21.1)
    Setting

    		Indirect patient care7286 (47.0)

    		Direct patient care8219 (53.0)


    		Acute care6821 (44.0)


    		ICUa1398 (9.0)


    		Medical6660 (43.0)


    		Surgical1559 (10.0)


    		Inpatient4344 (28.0)


    		Mixed3045 (19.6)


    		Outpatient830 (5.4)
    Shift

    		Day10,979 (70.8)

    		Night2250 (14.5)

    		Swing817 (5.3)

    		Other1214 (7.8)
    Shift length (hours)

    		87889 (50.9)

    		101272 (8.2)

    		124091 (26.4)

    		Flexible874 (5.6)

    		Other1211 (7.8)
    Frustration with technology

    		Rarely6310 (40.7)

    		A little4130 (26.6)

    		Occasionally2815 (18.2)

    		Always2250 (14.5)

    aICU: intensive care unit.

    WLI scores ranged from 0 to 100 (higher score favorable), with a mean score of 68.4 (SD 23.4) and median score of 71.4 (IQR 52.4-85.7). Separately, frustration with technology was reported as “none/rarely” by 6310 (40.7%) of the 15,505 respondents, “some/a little” by 4130 (26.6%), “occasionally/moderate” by 2815 (18.2%), and “all the time” by 2250 (14.5%) respondents. Frustration with technology scores ranged from 0 to 100 (lower score favorable), with a mean score of 35.0 (SD 35.9), and the score was higher among direct clinical care providers (mean 36.8, SD 36.4) than indirect providers (mean 32.9, SD 35.3; P<.001). The distribution of frustration with technology scores among 818 work settings, with corresponding WLI scores, is shown in Multimedia Appendix 1. The mean frustration with technology scores by job type are presented in Figure 1; the highest scores were reported by physicians, pharmacists, physician assistants, and nurses. Agreement in frustration with technology within work settings was low, with a combined weighted Cohen κ=0.04. Emotional exhaustion scores ranged from 0 to 100 (lower score favorable), with a mean score of 41.6 (SD 30.7).

    ‎
    Figure 1. Frustration with technology scores by job position. Data shown as mean values and 95% confidence limits of the mean, with the reference line at a population mean of 35.03.
    View this figure

    Work setting correlations among frustration with technology, WLI items, and emotional exhaustion responses are illustrated in Multimedia Appendix 2 and tabulated in Multimedia Appendix 3. Frustration with technology was positively correlated with the full Emotional Exhaustion scale (r=0.35) as well as each individual item on the scale (r=0.29-0.36). The reverse-transposed WLI scale was negatively correlated with the Emotional Exhaustion scale and its individual items (r=–0.55 to –0.63). Each individual WLI item was correlated with the Emotional Exhaustion scale and its individual items, with the smallest correlations for “arrived home late from work” (r=0.32-0.41) and the largest correlations for “had difficulty sleeping” (r=0.57-0.65).

    In our primary analysis, frustration with technology and six of the seven WLI items were independently related to emotional exhaustion in multivariable modeling, each associating with a 0.34- to 2.06-point increase in the emotional exhaustion score for each 10-point change (on a 100-point scale), as shown in Table 2. Frustration with technology was associated with a 1.23-point (95% CI 1.07-1.38) increase in the emotional exhaustion score with each 10-point change, and it was second only to difficulty sleeping out of the WLI items most strongly associated with emotional exhaustion. For example, an increase in frustration with technology score from 30 to 40 would correspond to a 1.23-point increase in the emotional exhaustion score, all else being equal. The frequency of sleeping less than 5 hours a night was the only WLI item that was not independently associated with emotional exhaustion. Results were similar when stratified by direct patient care versus indirect patient care. The results of our sensitivity analysis, as shown in Multimedia Appendix 4, are aligned with our primary regression model.

    Table 2. Frustration with technology and work-life integration as independent predictors of emotional exhaustion.
    Work-Life Integration scale itemβa95% CIP value
    Felt frustrated by technology1.231.07 to 1.38<.001
    During the past week, how often did this occur?

    		Had difficulty sleeping2.061.88 to 2.25<.001

    		Changed personal/family plans because of work.990.80 to 1.18<.001

    		Worked through a day/shift without any breaks.870.69 to 1.05<.001

    		Arrived home late from work.820.64 to 1.00<.001

    		Ate a poorly balanced meal.670.49 to 0.85<.001

    		Skipped a meal.340.14 to 0.54.001

    		Slept less than 5 hours in a night.01–0.18 to 0.19.94

    aEstimates via a single multivariable mixed model with work setting as random intercept. Beta coefficients reflect the change in emotional exhaustion score for each 10-point increase in frustration or work-life integration item (100-point scale) evaluated among 12,528 respondents in 818 work settings, adjusted for job type, years of experience, patient care type (intensive care vs not, surgical vs not, inpatient vs not), and direct patient care vs indirect patient care.

    Our secondary analysis relating frustration with technology and emotional exhaustion aggregated within work settings is shown in Figure 2. Work settings with higher mean frustration with technology scores (distributed into quartiles) had higher emotional exhaustion scores. Mean emotional exhaustion scores ranged from 29.0 (SD 18.7, 95% CI 25.6-32.3) in the lowest quartile to 47.3 (SD 19.1, 95% CI 43.9-50.7) in the highest quartile. Results were similar when stratified by direct patient care versus indirect patient care.

    ‎
    Figure 2. Emotional exhaustion scores, stratified by quartile of the technology frustration scores for each work setting, shown for all respondents (A) and stratified by direct patient care versus indirect patient care (B). Data are shown as mean values and upper 95% confidence limits, with results of t tests of adjacent quartiles.
    View this figure

    This study found that frustration with technology varies with health care worker role and among individuals within work settings. Frustration with technology and 6 of 7 WLI items are independently associated with emotional exhaustion. Although frustration with technology was higher among direct clinical providers, similar relationships with emotional exhaustion were apparent for respondents engaged in direct patient care compared to those engaged in indirect patient care.

    These results highlight and build on the evidence relating health care workers’ user experience with well-being outcomes [16,43,44]. HIT differs from consumer technology in that the purchaser (eg, health care administration) often has different priorities than the end users (health care workers), making user experience less incentivized as a driver in technology development. Although the purchaser’s incentives (eg, improving patient care quality and communication, facilitating accurate reimbursement, and developing research and analytics infrastructure) are undeniably important, our findings highlight the need for attention to user experience with HIT [45]. Recently, physician-reported EHR usability was reported to be in the bottom 10th percentile relative to system usability scores in other industries, but physicians reporting better EHR usability had lower odds of burnout [26].

    Frustration with technology may arise from discrepancies between expectations and reality [24]. Particularly in settings in which EHRs or other HITs were rapidly implemented, health care workers’ expectations of the benefits of these technologies may not have been accurately set, features may not have been fully explained, or efficient use of the interface may not have been taught [46,47]. The Health Information Technology for Economic and Clinical Health (HITECH) Act of 2009 that incentivized the adoption of HIT was effective at encouraging transition to EHRs, but rapid adoption may have precluded adequate attention to end-user input, development of interoperability standards, setting realistic expectations, or education on the use of new technologies [1,48-50].

    Emotional exhaustion, the key construct of burnout evaluated in this study, reflects a depleted state arising from excessive demands, continuous stress, or insufficient resources [34,35,51-54]. If an EHR requires many clicks or otherwise inefficient workflows to accomplish tasks, this would translate to increased job demands. Alternatively, if inadequate training results in a health care worker not knowing how to use the interface effectively, this lack of knowledge translates to insufficient resources to use the available tools. In both cases, frustration with technology may develop as an indicator of imbalanced job demands and resources.

    In our primary analysis, frustration with technology was found to be associated with burnout independently of other markers of WLI. This association was not as strong as the one observed for difficulty sleeping, consistent with observations that sleep disturbance may itself be an indicator of psychological distress [55]. However, it was similar in magnitude to the associations for other items reflecting excessive workload or workplace inefficiencies, such as changing personal/family plans because of work, working a shift without any breaks, or arriving home late from work. This pattern of findings suggests that frustration with technology is more closely related to an imbalance between job demands and resources, and it is less similar to psychological distress markers.

    At the work-setting level, average frustration with technology was significantly associated with higher emotional exhaustion, which indicates a potential climate-like effect of frustration with technology, similar to that observed for WLI [28]. However, agreement among respondents from the same work setting was low, suggesting that much of frustration with technology is rooted at the individual level rather than the work-setting level (ie, the individual’s experience with technology rather than problems with the technology itself). Even work settings with the highest average frustration with technology scores still contained individuals reporting low frustration, suggesting against the notion that frustration is inevitable or inextricably linked to HIT. Different individuals may have vastly different experiences using the same technology, related to their specific worker tasks, training, personal expectations, comfort with the technology, and acceptance of change [56].

    Of note, frustration with technology was only moderately correlated with frustration with one’s job in general, indicating that frustration with technology can occur without generalized job frustration and that individuals may remain motivated to take action to reduce their frustrations. This pattern of findings suggests that ensuring all individuals receive adequate support and training may be effective at reducing frustration with technology, rather than focusing only on making extensive changes to the technology or the user interface itself.

    Several practical steps may be taken to reduce the frequency of frustration with technology among health care workers. Although details will require tailoring to specific settings, tasks, and technologies available, our findings suggest an approach of educating and supporting individuals experiencing the most frequent frustrations. Recent studies have described the use of supplemental EHR training to improve comfort with technology and ability to work more efficiently [57,58]. Interventions at the work-setting level may also include measuring and reducing individual workloads, such as employing scribes to assist with clinical documentation, transitioning to team-based documentation and inbox management, or automating data-entry tasks [59-72]. Finally, interventions to reduce workloads placed on the system could have the broadest benefits, but it would likely require changes to current payment structures that promote lengthy documentation and labor-intensive payment authorizations [73].

    This study must be interpreted in the context of its design. As a cross-sectional, observational study, it cannot determine causality or directionality of the observed correlations (ie, whether frustration with technology induces burnout, burnout amplifies frustrations, and/or an external factor influences both). Although we were able to adjust for many potential confounders in our regression model, it is possible that residual confounding from unobserved variables such as age or prior experience with technology remains. Emotional exhaustion was also evaluated, but it does not capture other manifestations of burnout such as depersonalization. The 100-point emotional exhaustion scale we used differs from scores generated by other burnout instruments and cannot be used to directly compare effect sizes from studies using different instruments; however, methods to approximate estimates from disparate instruments have been previously described [74]. Although we used the 7-item WLI scale in this study, the 8-item version is commonly used, internally consistent (Cronbach α=.81 for this sample), and appears to add a unique element to the WLI assessment as evidenced by the results presented here.

    The response rate of over 70% compares favorably with other studies of this magnitude and exceeds commonly accepted thresholds for survey-based research; however, there may remain some sampling bias, and it is possible that physicians in particular were relatively underrepresented in this sample [75,76]. Similarly, although the survey was confidential, as with any self-reported measures, the responses may be susceptible to recall bias or social desirability bias. The survey respondents reflect a wide variety of health care roles and work settings, with varying interactions with technology. Furthermore, even though many of the direct and indirect patient care roles included in this study heavily feature EHR use, these and other respondents may have referenced other use of technology in their responses, and we did not conduct interviews to further characterize their responses. Prior qualitative research has found that specific drivers of HIT frustration vary among individuals, tasks, and settings [47,77]. Our findings are thus reflective of the overall conceptual relationship between frustration and emotional exhaustion but are unable to provide conclusions regarding any particular piece of technology or source of frustration. Additional research will be necessary to further delineate the specific sources and scope of frustration with technology across health care worker roles, as these data may provide more granular insights of potential interventions to reduce frustrations. Although this survey was administered within a single US state, the Michigan Health and Hospital Association Keystone Center includes all 175 hospitals from 20 health systems within Michigan, making our results likely to be generalizable to community and academic hospitals across the United States.

    It remains unknown whether reducing frustrations with technology through improved training, updated interfaces, or redistributed tasks will be effective in reducing burnout. Longitudinal observational studies may enhance our understanding of the directionality of these relationships, but prospective trials will be needed to fully evaluate the effect of interventions to improve health care worker user experience and well-being.

    In conclusion, frustration with technology and difficulty sleeping were the biggest WLI factors associated with emotional exhaustion across direct and indirect patient care settings. Interventions designed to reduce health care workers’ frustration with technology and improve other aspects of WLI may be effective strategies to reduce burnout among health care workers.

    Acknowledgments

    This study was supported by the Agency for Healthcare Research and Quality (grant K08 HS027837, awarded to DT), the Eunice Kennedy Shriver National Institute of Child Health and Human Development (grant R01 HD084679-01, awarded to JBS and JP), and the Stanford Maternal and Child Health Research Institute (DT). The content is solely the responsibility of the authors and does not necessarily represent the official views of the sponsors. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

    Conflicts of Interest

    TDS is co-inventor of the Well-Being Index instruments and the Participatory Management Leadership Index. Mayo Clinic holds the copyright to these instruments and has licensed them for use outside Mayo Clinic. Mayo Clinic pays TDS a share of royalties received. As an expert in clinician well-being, TDS frequently presents grand round and keynote presentations and advises medical centers on clinician well-being. He receives honorarium for some of these engagements.

    ‎
    Multimedia Appendix 1

    Frustration with technology and work-life integration scores for 818 individual work settings, ordered by mean frustration with technology scores. Lower panel shows mean work-life integration scores for the corresponding units, suggesting a negative unadjusted correlation. Data are shown as mean values and 95% confidence limits.

    DOCX File , 319 KB
    ‎
    Multimedia Appendix 2

    Work-setting frustration with technology scores in relation to emotional exhaustion scores. For ease of interpretation, values are shown as means and 50% confidence limits for each work setting (total of 818 work settings with 5 or more respondents). CLM: confidence limits of the means.

    DOCX File , 215 KB
    ‎
    Multimedia Appendix 3

    Item-level correlations with emotional exhaustion items for 818 work settings.

    DOCX File , 14 KB
    ‎
    Multimedia Appendix 4

    Sensitivity analysis. Multivariable regression with independent variables chosen using lasso.

    DOCX File , 14 KB

    References

    1. Washington V, DeSalvo K, Mostashari F, Blumenthal D. The HITECH era and the path forward. N Engl J Med 2017 Sep 07;377(10):904-906. [CrossRef] [Medline]
    2. Martin SA, Sinsky CA. The map is not the territory: medical records and 21st century practice. Lancet 2016 Oct 22;388(10055):2053-2056. [CrossRef] [Medline]
    3. Berenson RA, Basch P, Sussex A. N Engl J Med 2011 May 19;364(20):1892-1895. [CrossRef] [Medline]
    4. Halamka JD, Tripathi M. The HITECH era in retrospect. N Engl J Med 2017 Sep 07;377(10):907-909. [CrossRef] [Medline]
    5. Downing NL, Bates DW, Longhurst CA. Physician burnout in the electronic health record era: are we ignoring the real cause? Ann Intern Med 2018 Jul 03;169(1):50-51. [CrossRef] [Medline]
    6. Arndt BG, Beasley JW, Watkinson MD, Temte JL, Tuan W, Sinsky CA, et al. Tethered to the EHR: primary care physician workload assessment using EHR event log data and time-motion observations. Ann Fam Med 2017 Sep;15(5):419-426 [FREE Full text] [CrossRef] [Medline]
    7. Spinelli WM, Fernstrom KM, Britt H, Pratt R. "Seeing the patient is the joy:" a focus group analysis of burnout in outpatient providers. Fam Med 2016 Apr;48(4):273-278. [Medline]
    8. Sinsky C, Colligan L, Li L, Prgomet M, Reynolds S, Goeders L, et al. Allocation of physician time in ambulatory practice: a time and motion study in 4 specialties. Ann Intern Med 2016 Dec 06;165(11):753-760. [CrossRef] [Medline]
    9. Tai-Seale M, Olson CW, Li J, Chan AS, Morikawa C, Durbin M, et al. Electronic health record logs indicate that physicians split time evenly between seeing patients and desktop medicine. Health Aff (Millwood) 2017 Apr 01;36(4):655-662 [FREE Full text] [CrossRef] [Medline]
    10. Shanafelt TD, West CP, Sinsky C, Trockel M, Tutty M, Satele DV, et al. Changes in burnout and satisfaction with work-life integration in physicians and the general US working population between 2011 and 2017. Mayo Clin Proc 2019 Sep;94(9):1681-1694 [FREE Full text] [CrossRef] [Medline]
    11. Han S, Shanafelt TD, Sinsky CA, Awad KM, Dyrbye LN, Fiscus LC, et al. Estimating the attributable cost of physician burnout in the United States. Ann Intern Med 2019 Jun 04;170(11):784-790. [CrossRef] [Medline]
    12. Shanafelt T, Goh J, Sinsky C. The business case for investing in physician well-being. JAMA Intern Med 2017 Dec 01;177(12):1826-1832. [CrossRef] [Medline]
    13. Tran B, Lenhart A, Ross R, Dorr DA. Burnout and EHR use among academic primary care physicians with varied clinical workloads. AMIA Jt Summits Transl Sci Proc 2019;2019:136-144 [FREE Full text] [Medline]
    14. Robertson SL, Robinson MD, Reid A. Electronic health record effects on work-life balance and burnout within the I Population Collaborative. J Grad Med Educ 2017 Aug;9(4):479-484 [FREE Full text] [CrossRef] [Medline]
    15. How doctors feel about electronic health records. National Physician Poll by The Harris Poll.: Stanford Medicine; 2018.   URL: https://med.stanford.edu/ehr/electronic-health-records-poll-results.html [accessed 2021-06-15]
    16. Shanafelt TD, Dyrbye LN, Sinsky C, Hasan O, Satele D, Sloan J, et al. Relationship between clerical burden and characteristics of the electronic environment with physician burnout and professional satisfaction. Mayo Clin Proc 2016 Jul;91(7):836-848. [CrossRef] [Medline]
    17. Babbott S, Manwell LB, Brown R, Montague E, Williams E, Schwartz M, et al. Electronic medical records and physician stress in primary care: results from the MEMO Study. J Am Med Inform Assoc 2014 Feb;21(e1):e100-e106 [FREE Full text] [CrossRef] [Medline]
    18. Tawfik DS, Phibbs CS, Sexton JB, Kan P, Sharek PJ, Nisbet CC, et al. Factors associated with provider burnout in the NICU. Pediatrics 2017 May;139(5):e20164134 [FREE Full text] [CrossRef] [Medline]
    19. Tai-Seale M, Dillon EC, Yang Y, Nordgren R, Steinberg RL, Nauenberg T, et al. Physicians' well-being linked to in-basket messages generated by algorithms in electronic health records. Health Aff (Millwood) 2019 Jul;38(7):1073-1078. [CrossRef] [Medline]
    20. Adler-Milstein J, Zhao W, Willard-Grace R, Knox M, Grumbach K. Electronic health records and burnout: time spent on the electronic health record after hours and message volume associated with exhaustion but not with cynicism among primary care clinicians. J Am Med Inform Assoc 2020 Apr 01;27(4):531-538 [FREE Full text] [CrossRef] [Medline]
    21. Hilliard R, Haskell J, Gardner R. Are specific elements of electronic health record use associated with clinician burnout more than others? J Am Med Inform Assoc 2020 Jul 01;27(9):1401-1410. [CrossRef] [Medline]
    22. National Academy of Medicine. Taking Action Against Clinician Burnout: A Systems Approach to Professional Well-Being. In: National Academies of Sciences. Washington, DC: The National Academies Press; 2019.
    23. Lawson R. Frustration: the development of a scientific concept. United Kingdom: Macmillan; 1965.
    24. Opoku-Boateng GA. User frustration in HIT interfaces: exploring past Hci research for a better understanding of clinicians' experiences. AMIA Annu Symp Proc 2015;2015:1008-1017 [FREE Full text] [Medline]
    25. Zayas-Cabán T, White PJ. The national health information technology human factors and ergonomics agenda. Appl Ergon 2020 Jul;86:103109. [CrossRef] [Medline]
    26. Melnick ER, Dyrbye LN, Sinsky CA, Trockel M, West CP, Nedelec L, et al. The association between perceived electronic health record usability and professional burnout among US physicians. Mayo Clin Proc 2020 Mar;95(3):476-487 [FREE Full text] [CrossRef] [Medline]
    27. Friedberg M, Chen P, Van BK, Aunon F, Pham C, Caloyeras J. Factors affecting physician professional satisfaction and their implication for patient care, health systems, and health policy. Santa Monica, CA: RAND Corporation; 2013.
    28. Schwartz SP, Adair KC, Bae J, Rehder KJ, Shanafelt TD, Profit J, et al. Work-life balance behaviours cluster in work settings and relate to burnout and safety culture: a cross-sectional survey analysis. BMJ Qual Saf 2019 Feb;28(2):142-150 [FREE Full text] [CrossRef] [Medline]
    29. Sexton JB, Schwartz SP, Chadwick WA, Rehder KJ, Bae J, Bokovoy J, et al. The associations between work-life balance behaviours, teamwork climate and safety climate: cross-sectional survey introducing the work-life climate scale, psychometric properties, benchmarking data and future directions. BMJ Qual Saf 2017 Aug;26(8):632-640 [FREE Full text] [CrossRef] [Medline]
    30. Sexton JB, Helmreich RL, Neilands TB, Rowan K, Vella K, Boyden J, et al. The Safety Attitudes Questionnaire: psychometric properties, benchmarking data, and emerging research. BMC Health Serv Res 2006 Apr 03;6:44 [FREE Full text] [CrossRef] [Medline]
    31. Profit J, Sharek PJ, Amspoker AB, Kowalkowski MA, Nisbet CC, Thomas EJ, et al. Burnout in the NICU setting and its relation to safety culture. BMJ Qual Saf 2014 Oct;23(10):806-813 [FREE Full text] [CrossRef] [Medline]
    32. Adair KC, Quow K, Frankel A, Mosca PJ, Profit J, Hadley A, et al. The Improvement Readiness scale of the SCORE survey: a metric to assess capacity for quality improvement in healthcare. BMC Health Serv Res 2018 Dec 17;18(1):975 [FREE Full text] [CrossRef] [Medline]
    33. Tawfik DS, Shanafelt TD, Dyrbye LN, Sinsky CA, West CP, Davis AS, et al. Personal and professional factors associated with work-life integration among US physicians. JAMA Netw Open 2021 May 03;4(5):e2111575 [FREE Full text] [CrossRef] [Medline]
    34. Bakker AB, Demerouti E. Job demands-resources theory: taking stock and looking forward. J Occup Health Psychol 2017 Jul;22(3):273-285. [CrossRef] [Medline]
    35. Demerouti E, Bakker AB, Nachreiner F, Schaufeli WB. The job demands-resources model of burnout. J Appl Psychol 2001 Jun;86(3):499-512. [Medline]
    36. Block M, Ehrenworth JF, Cuce VM, Ng'ang'a N, Weinbach J, Saber S, et al. Measuring handoff quality in labor and delivery: development, validation, and application of the Coordination of Handoff Effectiveness Questionnaire (CHEQ). Jt Comm J Qual Patient Saf 2013 May;39(5):213-220. [CrossRef] [Medline]
    37. Sexton JB, Adair KC, Leonard MW, Frankel TC, Proulx J, Watson SR, et al. Providing feedback following Leadership WalkRounds is associated with better patient safety culture, higher employee engagement and lower burnout. BMJ Qual Saf 2018 Dec;27(4):261-270 [FREE Full text] [CrossRef] [Medline]
    38. Sexton JB, Adair KC. Forty-five good things: a prospective pilot study of the Three Good Things well-being intervention in the USA for healthcare worker emotional exhaustion, depression, work-life balance and happiness. BMJ Open 2019 Mar 20;9(3):e022695 [FREE Full text] [CrossRef] [Medline]
    39. Sexton J. SCORE: Assessment of your work setting Safety, Communication, Operational Reliability, and Engagement. Technical Report 16-8. 2017.   URL: https://www.hsq.dukehealth.org/files/2019/05/SCORE_Technical_Report_5.14.19.pdf [accessed 2021-06-16]
    40. Dowding I, Haufe S. Powerful statistical inference for nested data using sufficient summary statistics. Front Hum Neurosci 2018;12:103 [FREE Full text] [CrossRef] [Medline]
    41. Tibshirani R. Regression shrinkage and selection via the lasso. J R Stat Soc Series B Stat Methodol 2018 Dec 05;58(1):267-288. [CrossRef]
    42. Chen SS, Donoho DL, Saunders MA. Atomic decomposition by basis pursuit. SIAM Rev 2001 Jan;43(1):129-159. [CrossRef]
    43. Harris DA, Haskell J, Cooper E, Crouse N, Gardner R. Estimating the association between burnout and electronic health record-related stress among advanced practice registered nurses. Appl Nurs Res 2018 Oct;43:36-41. [CrossRef] [Medline]
    44. Gardner RL, Cooper E, Haskell J, Harris DA, Poplau S, Kroth PJ, et al. Physician stress and burnout: the impact of health information technology. J Am Med Inform Assoc 2019 Feb 01;26(2):106-114. [CrossRef] [Medline]
    45. American Medical Association. Improving care: Priorities to improve electronic health record usability. Chicago, IL: American Medical Association; 2014.
    46. Jamoom EW, Heisey-Grove D, Yang N, Scanlon P. Physician opinions about EHR use by EHR experience and by whether the practice had optimized its EHR use. J Health Med Inform 2016 Jul 30;7(4):1000240 [FREE Full text] [CrossRef] [Medline]
    47. Magsamen-Conrad K, Checton M. Technology and health care: efficiency, frustration, and disconnect in the transition to electronic medical records. GSTF J Media Commun 2014 Feb;1(2):23-27 [FREE Full text] [CrossRef] [Medline]
    48. Mennemeyer ST, Menachemi N, Rahurkar S, Ford EW. Impact of the HITECH Act on physicians' adoption of electronic health records. J Am Med Inform Assoc 2016 Mar;23(2):375-379. [CrossRef] [Medline]
    49. Cantwell E, McDermott K. making technology talk: how interoperability can improve care, drive effidcincy, and reduce waste. Healthc Financ Manage 2016 May;70(5):70-76. [Medline]
    50. Pronovost PJ, Johns MME, Palmer S, Bono RC, Fridsma DB, Gettinger A, et al, editors. Procuring interoperability: Achieving high-quality, connected, and person-centered care. Washington, DC: National Academy of Sciences; 2018.
    51. Maslach CJ, Leiter M. Maslach Burnout Inventory Manual. 3rd ed. Palo Alto, CA: Consulting Psychologists Press; 1996.
    52. Profit J, Etchegaray J, Petersen LA, Sexton JB, Hysong SJ, Mei M, et al. The Safety Attitudes Questionnaire as a tool for benchmarking safety culture in the NICU. Arch Dis Child Fetal Neonatal Ed 2012 Mar;97(2):F127-F132 [FREE Full text] [CrossRef] [Medline]
    53. Lee RT, Ashforth BE. A meta-analytic examination of the correlates of the three dimensions of job burnout. J Appl Psychol 1996 Apr;81(2):123-133. [CrossRef] [Medline]
    54. Bakker AB, Demerouti E, Euwema MC. Job resources buffer the impact of job demands on burnout. J Occup Health Psychol 2005 Apr;10(2):170-180. [CrossRef] [Medline]
    55. Chueh K, Chen K, Lin Y. Psychological distress and sleep disturbance among female nurses: anxiety or depression? J Transcult Nurs 2021 Jan;32(1):14-20. [CrossRef] [Medline]
    56. Tutty MA, Carlasare LE, Lloyd S, Sinsky CA. The complex case of EHRs: examining the factors impacting the EHR user experience. J Am Med Inform Assoc 2019 Jul 01;26(7):673-677 [FREE Full text] [CrossRef] [Medline]
    57. DiAngi YT, Stevens LA, Halpern-Felsher B, Pageler NM, Lee TC. Electronic health record (EHR) training program identifies a new tool to quantify the EHR time burden and improves providers' perceived control over their workload in the EHR. JAMIA Open 2019 Jul;2(2):222-230 [FREE Full text] [CrossRef] [Medline]
    58. Stevens LA, DiAngi YT, Schremp JD, Martorana MJ, Miller RE, Lee TC, et al. Designing an individualized ehr learning plan for providers. Appl Clin Inform 2017;8(3):924-935. [CrossRef] [Medline]
    59. Contratto E, Romp K, Estrada CA, Agne A, Willett LL. Physician order entry clerical support improves physician satisfaction and productivity. South Med J 2017 Dec;110(5):363-368. [CrossRef] [Medline]
    60. Guo U, Chen L, Mehta PH. Electronic health record innovations: helping physicians - one less click at a time. Health Inf Manag 2017 Sep;46(3):140-144. [CrossRef] [Medline]
    61. Bank AJ, Gage RM. Annual impact of scribes on physician productivity and revenue in a cardiology clinic. Clinicoecon Outcomes Res 2015;7:489-495 [FREE Full text] [CrossRef] [Medline]
    62. Gidwani R, Nguyen C, Kofoed A, Carragee C, Rydel T, Nelligan I, et al. Impact of scribes on physician satisfaction, patient satisfaction, and charting efficiency: a randomized controlled trial. Ann Fam Med 2017 Dec;15(5):427-433 [FREE Full text] [CrossRef] [Medline]
    63. Heaton HA, Castaneda-Guarderas A, Trotter ER, Erwin PJ, Bellolio MF. Effect of scribes on patient throughput, revenue, and patient and provider satisfaction: a systematic review and meta-analysis. Am J Emerg Med 2016 Oct;34(10):2018-2028. [CrossRef] [Medline]
    64. Pozdnyakova A, Laiteerapong N, Volerman A, Feld LD, Wan W, Burnet DL, et al. Impact of medical scribes on physician and patient satisfaction in primary care. J Gen Intern Med 2018 Jul;33(7):1109-1115. [CrossRef] [Medline]
    65. Shanafelt TD, Noseworthy JH. Executive leadership and physician well-being: nine organizational strategies to promote engagement and reduce burnout. Mayo Clin Proc 2016 Nov 18;92(1):129-146. [CrossRef] [Medline]
    66. Linzer M, Poplau S, Babbott S, Collins T, Guzman-Corrales L, Menk J, et al. Worklife and wellness in academic general internal medicine: results from a national survey. J Gen Intern Med 2016 Sep;31(9):1004-1010 [FREE Full text] [CrossRef] [Medline]
    67. Sinsky CA, Willard-Grace R, Schutzbank AM, Sinsky TA, Margolius D, Bodenheimer T. In search of joy in practice: a report of 23 high-functioning primary care practices. Ann Fam Med 2013 May;11(3):272-278 [FREE Full text] [CrossRef] [Medline]
    68. Gregory ME, Russo E, Singh H. Electronic health record alert-related workload as a predictor of burnout in primary care providers. Appl Clin Inform 2017 Jul 05;8(3):686-697 [FREE Full text] [CrossRef] [Medline]
    69. Sinsky CA, Rule A, Cohen G, Arndt BG, Shanafelt TD, Sharp CD, et al. Metrics for assessing physician activity using electronic health record log data. J Am Med Inform Assoc 2020 Apr 01;27(4):639-643 [FREE Full text] [CrossRef] [Medline]
    70. Funk KA, Davis M. Enhancing the role of the nurse in primary care: the RN "co-visit" model. J Gen Intern Med 2015 Dec;30(12):1871-1873 [FREE Full text] [CrossRef] [Medline]
    71. Sinsky CA, Bodenheimer T. Powering-up primary care teams: advanced team care with in-room support. Ann Fam Med 2019 Jul 08;17(4):367-371. [CrossRef]
    72. Lyon C, English AF, Chabot Smith P. A team-based care model that improves job satisfaction. Fam Pract Manag 2018;25(2):6-11 [FREE Full text] [Medline]
    73. Mechanic D. Physician discontent: challenges and opportunities. JAMA 2003 Aug 20;290(7):941-946. [CrossRef] [Medline]
    74. Tawfik DS, Scheid A, Profit J, Shanafelt T, Trockel M, Adair KC, et al. Evidence relating health care provider burnout and quality of care: a systematic review and meta-analysis. Ann Intern Med 2019 Oct 15;171(8):555-567 [FREE Full text] [CrossRef] [Medline]
    75. Gordon N, Davidoff F, Tarnow E, Reidenberg MM, Endriss K. A question of response rate. Science Editor 2002;25(1):25-26 [FREE Full text]
    76. Hendra R, Hill A. Rethinking response rates: new evidence of little relationship between survey response rates and nonresponse bias. Eval Rev 2019 Oct;43(5):307-330. [CrossRef] [Medline]
    77. De Leeuw JA, Woltjer H, Kool RB. Identification of factors influencing the adoption of health information technology by nurses who are digitally lagging: in-depth interview study. J Med Internet Res 2020 Aug 14;22(8):e15630 [FREE Full text] [CrossRef] [Medline]

    ‎
    EHR: electronic health record
    HIT: health information technology
    HITECH: Health Information Technology for Economic and Clinical Health
    SCORE: Safety, Communication, Operational Reliability, and Engagement
    WLI: work-life integration

    Edited by R Kukafka; submitted 28.12.20; peer-reviewed by KC Wong, R Malka; comments to author 27.02.21; revised version received 08.04.21; accepted 06.05.21; published 06.07.21

    Copyright

    ©Daniel S Tawfik, Amrita Sinha, Mohsen Bayati, Kathryn C Adair, Tait D Shanafelt, J Bryan Sexton, Jochen Profit. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 06.07.2021.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.