This paper is in the following e-collection/theme issue:

Mobile Health (mhealth) (2929) Web-based and Mobile Health Interventions (3287) Digital Mental Health Interventions, e-Mental Health and Cyberpsychology (1512) mHealth for Wellness, Behavior Change and Prevention (2865) Design and Formative Evaluation of Mobile Apps (692) Anxiety and Stress Disorders (1049) Occupational Health and Ergonomics/Prevention at the Workplace (237)

Published on in Vol 26 (2024)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/48883, first published .
Assessing the Effectiveness of STAPP@Work, a Self-Management Mobile App, in Reducing Work Stress and Preventing Burnout: Single-Case Experimental Design Study

Assessing the Effectiveness of STAPP@Work, a Self-Management Mobile App, in Reducing Work Stress and Preventing Burnout: Single-Case Experimental Design Study

Assessing the Effectiveness of STAPP@Work, a Self-Management Mobile App, in Reducing Work Stress and Preventing Burnout: Single-Case Experimental Design Study

Authors of this article:

Sevda Demirel1 Author Orcid Image ;   Yvette Roke1 Author Orcid Image ;   Adriaan W Hoogendoorn2, 3 Author Orcid Image ;   Jamie Hoefakker4 Author Orcid Image ;   Kirsten Hoeberichts1 Author Orcid Image ;   Peter N van Harten5, 6 Author Orcid Image
Article Authors Cited by (1) Tweetations (3) Metrics

    Original Paper

    1Expertise Center for Autism Spectrum Disorder, GGz Centraal, Almere, Netherlands

    2Department of Psychiatry, Amsterdam Universitair Medische Centra, Amsterdam, Netherlands

    3Mental Health Program, Amsterdam Public Health Research Institute, Amsterdam, Netherlands

    4Praktijk InTeam, Den Haag, Netherlands

    5Department of Psychiatry, GGz Centraal, Amersfoort, Netherlands

    6Department of Psychiatry and Neuropsychology, School of Mental Health and Neuroscience, Maastricht University, Maastricht, Netherlands

    Corresponding Author:

    Sevda Demirel, MSc

    Expertise Center for Autism Spectrum Disorder

    GGz Centraal

    Boomgaardweg 12

    Almere, 1326 AD

    Netherlands

    Phone: 31 41807329

    Email: S.demirel@ggzcentraal.nl


    Background: Work-related stress and burnout remain common problems among employees, leading to impaired health and higher absenteeism. The use of mobile health apps to promote well-being has grown substantially; however, the impact of such apps on reducing stress and preventing burnout is limited.

    Objective: This study aims to assess the effectiveness of STAPP@Work, a mobile-based stress management intervention, on perceived stress, coping self-efficacy, and the level of burnout among mental health employees.

    Methods: The study used a single-case experimental design to examine the use of STAPP@Work among mental health employees without a known diagnosis of burnout (N=63). Participants used the app for 1 week per month repeatedly for a period of 6 months. Using a reversal design, the participants used the app 6 times to assess replicated immediate (1 week after use) and lasting (3 weeks after use) effects. The Perceived Stress Scale, the Coping Self-Efficacy Scale, and the Burnout Assessment Tool were used to measure the outcomes. Linear mixed models were used to analyze the data.

    Results: After 6 months of app use for 1 week per month, the participants showed a statistically significant decrease in perceived stress (b=–0.38, 95% CI –0.67 to –0.09; P=.01; Cohen d=0.50) and burnout symptoms (b=–0.31, 95% CI –0.51 to –0.12; P=.002; Cohen d=0.63) as well as a statistically significant improvement in problem-focused coping self-efficacy (b=0.42, 95% CI 0-0.85; P=.049; Cohen d=0.42). Long-term use of the app provided consistent reductions in burnout symptoms over time, including in the level of exhaustion and emotional impairment.

    Conclusions: The use of an app-based stress management intervention has been shown to reduce burnout symptoms and enhance coping self-efficacy among mental health workers. Prevention of burnout and minimization of work-related stress are of utmost importance to protect employee health and reduce absenteeism.

    J Med Internet Res 2024;26:e48883

    doi:10.2196/48883

    Keywords

    mental health; stress; coping; burnout; stress management; digital intervention; health promotion; mobile apps; mobile health; mHealth; mental health professionals 


    Background

    Work-related stress remains a persistent challenge for both employees and organizations, adversely affecting health outcomes and reducing job performance [1]. Stress is often defined as a condition in which an environmental demand exceeds the natural regulatory capacity of an organism, particularly in situations that are unpredictable and uncontrollable [2]. Numerous studies over the last several decades have demonstrated that psychosocial stress adversely affects hormonal processes and organ systems in the body [3-6]. Excessive stress has a detrimental effect on mental health by causing emotional distress and increasing the risk of mental illnesses such as depression, anxiety, and burnout [7].

    Burnout is defined as a psychological syndrome primarily caused by a prolonged period of stress and the inability or unwillingness to cope effectively, resulting in both physical and mental exhaustion [8]. The term burnout is often used in the occupational context, where stress originates from the work environment. Such job stressors include heavy workload, long working hours, job insecurity, conflicts, low support, role ambiguity, and a lack of autonomy [9]. When work stress is of a long duration without applying effective coping strategies, job demands exceed capacity, leading to the depletion of one’s emotional, physical, and cognitive resources [9]. Individual consequences further include job dissatisfaction, depersonalization, cynicism, and low fulfillment [10]. Organizational consequences include absenteeism, reduced productivity levels, a higher turnover rate, and an increase in health care costs [9,11]. Organizations may incur financial costs not only from losing and replacing experienced staff but also by providing employee burnout support, implementing employee assistance programs, and facilitating reintegration processes [11-14]. Burnout causes significant distress owing to symptoms such as chronic fatigue, trouble sleeping, depressive symptoms, and difficulties with concentration and memory [9]. In 2021, approximately 17% of Dutch employees reported being mentally fatigued by their work, 16% were emotionally exhausted, and 15% were completely exhausted after work [15]. Moreover, the Dutch health care sector had one of the highest proportions of employees experiencing work-related mental exhaustion, at 20% [16].

    Effective interventions for preventing work-related stress and burnout are of high importance. The use of digital health, including mobile apps, as behavioral interventions to promote positive mental health is rapidly growing. The benefits of mobile mental health interventions include, but are not limited to, high accessibility, low costs, less time consumption, no clinician involvement, timely support, and the promotion of autonomy [17]. A recent systematic review found that mindfulness was the most commonly applied theory-based stress management strategy in apps [18]. They further concluded that such apps implemented stress prevention strategies less often than stress management strategies for stress recovery [18]. However, stressor identification and reduction, as primary prevention, is a crucial element in effective stress management [19]. Moreover, another review found that only a few stress management apps combined both stress monitoring and intervention based on detecting users’ stress levels [20].

    Previous studies examining the effectiveness of web- and app-based interventions among employees provided evidence of their effectiveness in stress reduction [21]. Improvements regarding emotional exhaustion and perceived stress in health care professionals using digital interventions were reported earlier as well [22,23]. However, the few available studies on apps for health care workers yielded mixed findings on stress and were often based on mindfulness practices or psychoeducation. For instance, 2 randomized controlled trials (RCTs) assessing an app among health care workers found no significant improvements in stress [24,25]. Although other mindfulness-based apps did find improvements in stress among physicians, nurses, and therapists [26]; hospital nurses [27]; and employees within the general practitioner practice [28], little is known regarding the effectiveness of app-based interventions aimed at stress management and burnout symptoms among employees, specifically health care professionals. Moreover, there is a lack of evidence for the sustained effects of long-term use of such app-based interventions [29].

    Mobile-based ecological momentary interventions (EMIs) comprise an emerging area of research. These interventions are based on ecological momentary assessment (EMA), a widely used tool in psychiatry to collect repeated daily assessments of individuals’ behaviors, emotions, experiences, or thoughts as they occur in their own “real-world” settings [30]. EMA is seen as a promising method that provides insight into variations in symptoms and behavior over time and their underlying person-environment interactions. EMIs offer interventions in the moment of an individual’s daily life, usually based on the assessment [31]. Previous studies on the effects of EMIs showed a positive overall medium effect on stress in clinical populations [32]. To date, no study has assessed the effectiveness of a mobile stress management intervention based on stress monitoring among health care workers.

    Objectives

    To bridge this gap, our group developed a self-management stress-signaling app using a participatory design with mental health employees called STAPP@Work. The app is developed to monitor daily stress levels and track the daily activities in employees’ natural settings, including work settings, based on the principles of EMA. STAPP@Work is a tool to manage stress at work by providing the employee feedback regarding their tracked stress levels and associated activities and creating a visual overview to make stress patterns during workdays and weeks visible. Furthermore, the app provides real-time suggestions for coping strategies specified for both the work environment and at home. The aim of this study was to evaluate the effectiveness of the STAPP@Work app by examining perceived stress, coping self-efficacy, and burnout symptoms. On the basis of previous literature outlined in the Background section, we hypothesized that after 6 months of using the STAPP@Work app for 1 week per month, perceived stress and coping self-efficacy would be improved and burnout symptoms would be reduced in mental health workers. In addition, we explored the progression of perceived stress, coping self-efficacy, and burnout scores over time to examine their lasting effects.


    Study Design

    This study adopted a single-case experimental design (SCED), a within-person design that measures the effectiveness of interventions [33]. Using an SCED, it became possible to focus on individual changes in outcomes (perceived stress, coping self-efficacy, and burnout symptoms) over time. This design was considered a good fit, as it enabled the assessment of behavior change over time in “real-world” settings using heterogeneous samples [33]. Moreover, SCEDs are often used to capture small but meaningful differences in behavior and to examine behavior that may change gradually over time, which is not feasible to assess in RCTs with a high number of successive measurements [34]. In addition, SCEDs are suitable for being able to evaluate the effectiveness of novel innovative interventions before they are evaluated in time-consuming and costly RCTs [35]. The results of this SCED study will provide an indication of whether it is worthwhile to conduct a large-scale RCT to further assess the effectiveness of the app [36]. Given that STAPP@Work is a novel behavioral intervention, using an SCED constitutes a useful first step to measure its effectiveness.

    Furthermore, an A1-B1-A2-B2 reversal design was used, in which A1 represents the baseline period, B1 the introduction of an intervention, A2 the withdrawal of the intervention, and B2 the reintroduction of the intervention [36]. In this study, this translated to the participants having used the STAPP@Work app for 1 week after the baseline period, followed by 3 weeks of nonuse, after which they used the app again for the second time for a weeklong period. Withdrawing the intervention allowed for an assessment of whether the effects of the app were lasting.

    Sample Size

    Owing to the complexity and practical unfeasibility of the sample size calculation for this SCED, the sample size for this study was based on the traditional paired sample 2-tailed t test design. Similar studies on stress management apps among health care employees are limited. Instead, we relied on a meta-analysis of the effectiveness of web- and app-based interventions on stress among employees, which found moderate-to-large effect sizes for mindfulness-based interventions and small-to-moderate effect sizes for stress management interventions [37]. A medium effect size (Cohen d=0.5) was considered in the sample size calculation for this study. To conduct a paired sample 2-tailed t test and detect a medium effect size (Cohen d=0.5) with a low internal correlation (r=0.1), a sample size of 59 participants was required. Considering the potential dropout rate of 7.5%, the required sample size was set at 63 participants.

    Participants

    Participants were eligible to participate in the study when they were employed within GGz Centraal, the fourth largest mental health care institution in the Netherlands. No further inclusion or exclusion criteria were applied. Recruitment strategies included the promotion of the study through flyers, digital media, and approaching teams or staff in various departments.

    The STAPP@Work App

    App Overview

    STAPP@Work is a self-management app designed to manage stress and improve well-being by (1) measuring daily stress levels, (2) providing personal coping advice, and (3) visualizing stress patterns. The app is based on a stress-signaling plan, a widely used tool in Dutch mental health institutions to monitor clients’ stress levels in relation to their mental health [38]. This allows for recognizing and dealing with early warning signs of distress. A stress-signaling plan often consists of 4 phases: green, yellow, orange, and red. Each phase characterizes how a person is feeling and what symptoms are associated, with the colors increasing from green (no stress) to red (very high stress) [38]. Furthermore, the plan describes both stress-inducing and protective factors as well as personal coping strategies to encourage the clients’ self-management [38]. STAPP@Work includes the concept of a stress-signaling plan and the EMA method by assessing stress levels in daily life and ensuring that these are linked to specific moments and activities in a day. The app was designed together with mental health employees and was based on their needs, ideas, and experiences that were explored during focus group discussions. The ongoing iterative development of the prototypes continued until the desired version of the STAPP@Work app was achieved.

    Measuring Daily Stress

    The app calculated daily stress levels by preparing a questionnaire. The employee could specify whether to complete 2, 3, or 4 questionnaires per day via the app settings. The app required a minimum of 2 questionnaires, as there were 4 hours between each questionnaire, and filling in 2 questionnaires would cover a working day of 8 hours. If the employee chose to fill in 3 or 4 questionnaires a day, the app would cover a 12-hour or 16-hour period out of 1 day. On the home page, the user could see if a questionnaire was available to fill in as well as the results of the previous stress measurement, as shown in Figure 1.

    ‎
    Figure 1. Screenshots of the home page and the visual overview page of the STAPP@Work app (translated from Dutch).

    Each questionnaire became available for the user to complete within a 1-hour time frame, probing activities that were performed in the past 4 hours and how one felt while doing so. Then, it followed with 7 stress-signaling questions (eg, “Did you feel irritable?” and “Were you able to take good care of yourself?”) to derive a stress level. The app calculates stress levels based on the answers to 7 stress questions. The stress questions were formulated and validated by focus group discussions and individual interviews with employees, following the design thinking method. The questions were not derived from an existing stress scale. The answer options for each stress question are as follows: “no”; “yes, but not more than normal”; “yes, more than normal”; and “yes, much more than normal.” A score is assigned to each answer option, and from there, the app computes a total stress score based on the answers to all 7 stress questions. The app then assigns the user’s perceived stress to 1 of 4 categories: no stress, little stress, stress, or a lot of stress, based on the total score from the stress-related questions. The app also asked for the user’s feedback on the assigned stress categories and gave personal coping advice.

    Visual Overview

    The app includes a visual overview of the assigned stress levels at both the day and week levels, helping the users to recognize their own stress patterns (Figure 1). The overview also showed which activities were associated with (a lot of) stress and no stress during the week.

    Personal Coping Advice

    During the exploratory focus groups and interviews, employees were also asked how they cope with stress at work and at home. Together with desk research and practitioner experience, a list of different effective approaches to coping with stress was formed, which was referred to as coping strategies. This list forms the general list of predefined suggestions for coping in the app. Suggestions for coping at work included the following: contact or discuss the situation with a colleague, get moving, go walking, plan or make an overview, do breathing exercises, take a break, do something else, and take care of yourself such as eating and drinking enough. However, the participants could add their own suggestions for coping to the list and remove existing suggestions. After each stress assessment, the app suggests a coping strategy displayed in text, which the app randomly selects from the personalized coping list of the user.

    Personalization

    The app was highly customizable, as the users could choose how many questionnaires they wanted to fill out in a day and at what times, to fit their working days. The app consisted of a list of activities created by mental health employees that fit within a workday, but users could add or remove activities as they wished. The same was true for coping advice, in which the app provided a list of coping strategies that the user could change at any time.

    Measures

    Perceived Stress

    The Perceived Stress Scale (PSS) is a 10-item self-report questionnaire that was used to measure how stressful certain situations were perceived in the past month (eg, “How often have you felt that you were in control of things?”), with items rated on a 5-point scale ranging from 0 (never) to 4 (very often) [39]. The PSS has shown good internal consistency (10 items; Cronbach α=0.84) [40].

    Coping Self-Efficacy

    The 13-item version of the Coping Self-Efficacy Scale (CSES) measures one’s confidence in various coping behaviors [41]. The CSES consists of 3 subscales: problem-focused coping (6 items; Cronbach α=0.91), emotion-focused coping (4 items; Cronbach α=0.91), and social support coping (3 items; Cronbach α=0.80) [41]. Items have a 10-point scale ranging from 0 (cannot do at all) to 10 (certainly can do).

    Burnout Symptoms

    The shorter version of the Burnout Assessment Tool (BAT), which consists of a 12-item self-report questionnaire, determines the extent to which burnout symptoms are experienced (eg, “I do not recognize myself in the way I react emotionally at work”), rated from 0 (never) to 4 (always) [42]. Cronbach α has been shown to be similar to the full version of the BAT (23 items) [43]. The BAT (23 items; Cronbach α=0.90) is subdivided into 4 validated core dimensions: exhaustion (8 items; Cronbach α=0.92), mental distance (5 items; Cronbach α=0.91), cognitive impairment (5 items; Cronbach α=0.92), and emotional impairment (5 items; Cronbach α=0.90) [42].

    Procedure

    A measurement involved filling out the questionnaire that was composed of the PSS, CSES, and BAT items. The researchers (KH and JH) sent the web-based questionnaire to all participants at each time of measurement, regardless of whether they had completed the previous questionnaire or not. A total of 17 measurements were obtained from the participants over a period of 9 months. Of these, 24% (4/17) were conducted during the baseline period and 71% (12/17) during the intervention phase, ending with 6% (1/17) follow-up measurement. We chose to include 4 baseline measurements to account for possible learning effects from merely participating in the study. Being introduced to the study and responding to the questionnaires on stress could have possibly contributed to an increased awareness of one’s own stress. During the 7-week baseline phase, the participants completed the questionnaire every 2 weeks. After the baseline measurement, the research team provided individual explanatory sessions on installing, setting up, and using the STAPP@Work app as well as the proceedings of the experiment. The participants could share their comments and questions about the study, as well as inquire about app functionalities, with the help desk that was available.

    The intervention concerned the daily use of the STAPP@Work app for 1 week, called the intervention week. After the withdrawal of the intervention, a questionnaire was completed in the week after use (1 week after use) and 3 weeks after use to examine the immediate and lasting effects, respectively. The introduction and withdrawal of the intervention were repeated 6 times throughout the intervention phase. A follow-up measurement took place 6 weeks after the last time the app was used. Figure 2 shows the order of the events.

    ‎
    Figure 2. Schematic overview of the study design, including the timeline.

    Statistical Analysis

    All analyses were performed using SPSS (version 28; IBM Corp). For data analysis, mean scores were calculated from the total scores of the PSS, CSES, and BAT, including the subscales. The subscales of the CSES and BAT were included as separate outcome variables to gain more insight into the subdomains. Descriptive statistics were used to describe participant characteristics and determine response rates over time. We categorized participants as early dropouts if they failed to use the app at least 3 times and definitively ceased participation. Data analysis did not include early dropouts. Linear mixed model (LMM) analysis was used to analyze the data [44]. LMMs are well suited for longitudinal intervention studies, as they allow the measurement of change in the response variables across units of time and within participants [45,46] and account for missing values [47]. Change scores between baseline and all consecutive intervention measurements were modeled for all outcome variables. We chose the fourth and last baseline measurement (week 7) as a reference to correct for learning effects during the baseline period [36].

    When conducting the LMM, assumptions of normality and equal variance of residuals were checked using graphical representations. As the data were collected over time, the presence of autocorrelation was evaluated both visually by the partial autocorrelation function plot and by the Durbin-Watson statistic test. Accordingly, the appropriate covariance structure was selected for the models for each outcome variable. Time was represented by (binary) week indicators that were included in the LMM as fixed effects, enabling the assessment of mean scores for each week of app use. For each outcome, Cohen d effect sizes were calculated for the mean change score from baseline to 6 months, standardized by the baseline SD. The LMM results were used to create graphs, visually presenting the estimated mean scores of the outcomes and illustrating the progression over time. The significance level was set to .05.

    Ethical Considerations

    To ensure data confidentiality, consideration has been given to secure data collection and storage. Data were collected through web-based questionnaires and stored in a folder. Only researchers (JH, KH, YR, and SD) within the team with access could view the responses. Data were then anonymized and stored in a protected folder on a secure server of GGz Centraal, where access for a single researcher (SD) was protected by a password and a soft token-based authentication. This study was conducted in accordance with the Declaration of Helsinki and was approved by the Medical Ethics Committee of Isala Zwolle (211006211006) on November 11, 2021. Informed consent was obtained from all participants involved in the study. The informed consent form included extensive information regarding the purpose of the study, procedures, data security, and the participant’s rights. Participation in the study was completely voluntary and without any compensation. Furthermore, the use of the STAPP@Work app is in accordance with the General Data Protection Regulation of the European Union.


    Sample Characteristics

    A total of 63 mental health employees were included in the study, with 10 early dropouts, resulting in 53 (84%) participants. Of these, 74% (39/53) were female participants, and the average age was 41 (SD 11.7) years (Table 1).

    Various mental health professions were included, with the majority represented by nurses (22/53, 42%). The response rates of the participants ranged from 45% (24/53) to 100% (53/53), as presented in Table 2.

    Table 1. Characteristics of the participants (n=53).
    CharacteristicValue
    Age (years), mean (SD)41 (11.7)
    Sex n (%)

    		Female39 (74)

    		Male14 (26)
    Profession, n (%)a

    		Nurse22 (42)

    		Psychologist7 (13)

    		Management function7 (13)

    		Counselor7 (13)

    		Staff member or adviser4 (8)

    		Administration or IT function3 (6)

    		Psychiatrist2 (4)

    		Medical doctor in training1 (2)

    aDue to rounding, percentages may not add up to 100%.

    Table 2. Response rates among participants (n=53).

    		QuestionnaireValue, response rate (%)a
    BaselineQuestionnaire 153 (100)
    BaselineQuestionnaire 252 (98)
    BaselineQuestionnaire 352 (98)
    BaselineQuestionnaire 451 (96)
    InterventionQuestionnaire 546 (87)
    InterventionQuestionnaire 642 (79)
    InterventionQuestionnaire 741 (77)
    InterventionQuestionnaire 841 (77)
    InterventionQuestionnaire 937 (70)
    InterventionQuestionnaire 1038 (72)
    InterventionQuestionnaire 1135 (66)
    InterventionQuestionnaire 1229 (55)
    InterventionQuestionnaire 1324 (45)
    InterventionQuestionnaire 1427 (51)
    InterventionQuestionnaire 1528 (53)
    InterventionQuestionnaire 1626 (49)
    Follow-upQuestionnaire 1730 (57)

    aResponse rate refers to the number of participants who completed the questionnaire.

    Perceived Stress

    The perceived stress score significantly improved after 6 months (b=–0.38, 95% CI –0.67 to –.09; P=.01) compared with baseline, with a medium effect size (Cohen d=0.50), as presented in Table 3.

    Figure 3 shows a lower stress score after each intervention week for 1 week after use, except for the fifth intervention week. A significant decrease in stress score was not found at follow-up.

    Table 3. Results of linear mixed model for all outcome variables: perceived stress, coping self-efficacy with subscales, and burnout symptoms with subscales after 6 months of monthly 1-week app use.
    VariableUnstandardized beta coefficient, b (SE: 95% CI)P valueCohen d
    Perceived stress–0.38 (–0.15; 0.67 to –0.09).010.50
    Total coping self-efficacya0.40 (0.22; –0.03 to 0.83).070.37

    		Problem-focused coping0.42 (0.21; 0 to 0.85).0490.42

    		Emotion-focused coping0.52 (0.30; –0.08 to 1.11).090.17

    		Social support coping0.24 (0.33; –0.41 to 0.89).470.29
    Total burnout symptomsb–0.31 (0.10; –0.51 to –0.12).0020.63

    		Exhaustion–0.38 (0.15; –0.68 to –0.08).020.50

    		Mental distance–0.22 (0.14; –0.48 to 0.05).110.34

    		Emotional impairment–0.42 (0.13; –0.68 to –0.16).0020.69

    		Cognitive impairment–0.24 (0.13; –0.49 to 0.01).060.41

    aTotal score of all 13 items on the Coping Self-Efficacy Scale.

    bTotal scores of all 12 items on the Burnout Assessment Tool.

    ‎
    Figure 3. Graphs showing the progression of the estimated means from linear mixed models of perceived stress, total coping self-efficacy, problem-focused coping self-efficacy, emotion-focused coping self-efficacy, and social support coping self-efficacy over a 36-week period. The baseline, intervention, and follow-up phases are presented. Each intervention week is indicated by the corresponding number. Error bars represent CIs. *P<.05. **P<.01 significant change from baseline (wk 7).

    Coping Self-Efficacy

    Coping self-efficacy did not significantly improve after 6 months of app use for 1 week per month (b=0.40, 95% CI –0.03 to 0.83; P=.07), as shown in Table 3. Focusing on the subscales, problem-focused coping abilities significantly improved after 6 months (b=0.42, 95% CI 0-0.85; P=.049) with a small to medium effect size (Cohen d=0.42), and emotion-focused coping (b=0.52, 95% CI –0.08 to 1.11; P=.09) and social support coping (b=0.24, 95% CI –0.41 to 0.89; P=.47) were not significantly improved. Figure 3 shows significant, consistent improvements in problem-focused coping starting from the second intervention week for both 1 week and 3 weeks after use until follow-up. No significant, consistent improvements were found regarding emotion-focused and social support coping self-efficacy scores (Figure 3).

    Burnout Symptoms

    Table 3 shows a significant improvement in the measure of burnout symptoms after 6 months of app use for 1 week per month (b=–0.31, 95% CI –0.51 to –0.12; P=.002) compared with baseline, with a medium effect size (Cohen d=0.63). Regarding the subscales, a significant decrease was found for exhaustion (b=–0.38, 95% CI –0.68 to –0.08; P=.02) and emotional impairment (b=–0.42, 95% CI –0.68 to –0.16; P=.002). The effect sizes were found to be medium for both exhaustion (Cohen d=0.50) and emotional impairment (Cohen d=0.69).

    When examining the progression over time, a significant decrease in emotional impairment was found after the first week of app use and was sustained for both 1 week post use and 3 weeks after use until follow-up (Figure 4). Consistent improvements following initial app use were found for exhaustion as well, with no significant change at follow-up (Figure 4). No consistent and sustained improvements were found for the level of mental distance and cognitive impairment (Figure 4).

    ‎
    Figure 4. Graphs showing the progression of the estimated means from linear mixed models of total burnout symptoms, exhaustion, mental distance, cognitive impairment, and emotional impairment over a 36-week period. The baseline, intervention, and follow-up phases are presented. Each intervention week is indicated by the corresponding number. Error bars represent CIs. *P<.05, **P<.01 significant change from baseline (wk 7).

    Principal Findings

    This effectiveness study showed that the participants using STAPP@Work were able to consistently reduce their measure of burnout symptoms over time. Specifically, the findings consistently showed improvements in exhaustion and emotional impairment, with sustained improvement observed in emotional impairment. Simultaneously, the participants showed lasting improvements in their confidence in problem-focused coping abilities. Perceived stress levels were reduced after 6 months of monthly 1-week app use. Following the findings, the app provides a means for improving problem-focused coping strategies to cope effectively with job stressors and reduce burnout complaints.

    This study found lasting improvements in burnout symptoms. Although each participant scored a measure of burnout symptoms, it is important to emphasize that they did not have a known burnout diagnosis. All participants were working during the time of this study. From the findings, it became apparent that particularly 2 core dimensions—exhaustion and emotional impairment—were consistently reduced. Improvements in exhaustion indicate that the employees felt less tired and experienced higher physical and mental energy levels at work [42]. A decrease in emotional impairment implies that the participants were able to improve their abilities to adequately control their own emotions at work [42]. The app providing insights into one’s stress patterns, and work-related stressors may have contributed to higher awareness and control over one’s stress levels at work. The results of this study showed reductions in perceived stress among the employees. The employees may have been able to reduce their stress as the app assessed their perceived stress several times a day, allowing them to intervene early when stress levels were rising to prevent worsening, similar to how a stress-signaling plan functions [38].

    Moreover, they improved their problem-focused coping self-efficacy, indicating increased confidence in coping effectively by resolving the stressful situation or altering the source of the stress [41]. This suggests that the participants were more able to cope effectively with work-related stressors and demands, resulting in lower exhaustion and higher control of emotional processes. Stress assessment and surveillance are crucial for determining the causal and contributing factors of stress according to the theory of preventive stress management (PSM) [19]. The PSM framework further encourages individual stress assessment by assuming a transactional relationship between an individual and work-related stress because of individual differences [19]. STAPP@Work calculated daily stress levels and created a personal overview in which one could see the moments and activities that were connected to high stress. This allowed the employee to recognize moments of high stress during the day and week as well as the associated activities to identify stress-inducing factors. This may have empowered employees to lessen, modify, or manage recognized work stressors, aiming to reduce or prevent the stress response as part of primary prevention [19]. Addressing work stressors reduces high job demands, leading to less fatigue and more energy, as the job demands–resources model assumes that high job demands and low resources are important predictors of the exhaustion component of burnout [48].

    The PSM theory argues that chronic stress that remains unaddressed and builds over time is manageable by providing interventions focused on both primary and secondary prevention [19]. The app addresses secondary prevention by providing real-time suggestions for coping strategies to manage work stress. Stressful situations become more manageable and less threatening because of such effective coping strategies, which act as a protective factor for the development of burnout [9]. Improvements in problem-focused coping self-efficacy were both immediate and lasting, suggesting that app use has led to the learning of problem-focused coping skills. In contrast, emotion-focused and social support coping abilities did not significantly improve; therefore, our hypothesis only supports problem-focused coping self-efficacy. The features of the app may have been especially profitable for problem-focused coping. Insights into stress patterns and daily stress levels through app use may have caused participants to be more proactive in identifying and resolving the cause of stress. Previous research has shown that individuals engaged more in problem-focused coping instead of emotion-focused coping when they were clearer and more communicative about their own emotions [49]. Therefore, participants gaining insights and self-reflection regarding their stress level may have little to gain by engaging in emotion-focused coping.

    Indeed, the perceived stress level significantly reduced after 6 months; however, when examining the effects over time, these were often to be found immediate instead of lasting with no significant change at follow-up. This implies that the app is most effective when users actively engage with it, leading to increased awareness and control. Conversely, when the app is not in use and does not provide feedback on stress levels, the effects diminish until the app is used again.

    The few studies examining mental health apps in relation to burnout have shown similar results in the improvement of burnout symptoms among employees [50] and specifically in reducing emotional exhaustion among health care professionals [23,51]. Furthermore, findings from an RCT assessing a self-management app among employees showed medium effects for reduction in perceived stress, lower emotional exhaustion, and higher emotion regulation, which are consistent with our findings [52]. Most of the scientifically studied mental health apps were based on the theories of mindfulness, psychoeducation, or cognitive behavioral therapy and were studied in RCTs [22,53]. The findings of this study add to the existing body of research by providing evidence that a stress management app, aligned to the work environment and assessed in a naturalistic context, can reduce symptoms that indicate burnout and, therefore, has the potential to preventively avert the risk of burnout. Furthermore, this study provides insights into the sustained effects of mental health apps, which were often lacking in previous studies [29].

    Strengths and Limitations

    One of the strengths of this study includes the use of an SCED, which made it possible to offer the intervention to all participants [20] and enabled the measurement of the changes of each participant in “real-life” conditions rather than comparing them between the 2 groups in a highly controlled setting [54]. Furthermore, the relatively long length of the experiment allowed for close observation of how stress, coping, and burnout symptoms progressed separately over a 9-month period and whether these were learned over time, such as coping [33]. In addition, STAPP@Work involves an innovative design in which an app incorporates a stress-signaling intervention and is of potential value in terms of reducing absenteeism.

    There are several limitations to consider when interpreting this study. First, the participants may have induced a social desirability bias. However, it is unlikely that such a bias persisted for several months after the intervention. Second, we observed that the number of missing values increased as the study progressed. Participants may have failed to adhere because of a loss of interest or other external circumstances. However, LMMs can deal with dropout if the missing data mechanism is missing at random, which is the case if dropout depends on observed characteristics such as stress in the baseline period [47].

    Third, each participant received the intervention at the same time following the baseline period, as it was not feasible to measure baseline stability at the individual level. Such stability is valuable to ensure that results are accountable to the intervention rather than to contextual influences [55]. This study did apply a reversal design, contributing to a higher degree of experimental control by repeatedly introducing and withdrawing the intervention, leading to a more robust demonstration of replicated effects and resulting in higher internal validity [56]. Fourth, the small sample size and disproportionate representation of female participants limited the generalizability of the findings to a broader and more diverse population. However, the distribution of sex in this study population corresponds to the distributions reported for Dutch mental health institutions [57]. Moreover, potential knowledge on stress management by mental health professionals restricts the generalizability to other industries or work settings as well. Finally, as this study followed a repeated measures design, it became more difficult to assess if changes occurring over time were a result of repeated use of the app or rather the cumulative effects of previous app use [36]. However, this did not limit the interpretation of whether such previous effects of the app were lasting.

    Implications and Future Research

    The preliminary results provided promising evidence that STAPP@Work is effective in reducing burnout symptoms and improving problem-focused coping. These findings build upon existing research by showing that a mobile health app can lead to sustainable effects and is able to address burnout-related symptoms. This is particularly relevant for preventive implementation in organizations to prevent burnout. This self-management app may have the potential to offer support in occupational contexts to promote less absenteeism and a healthier workforce, which are important for well-functioning and successful organizations. In addition, the app has the potential to integrate into daily life to support employees during their workdays or challenging periods. As the app helps to identify job stressors and provides coping advice suited for work environments, it can be implemented in any workplace to protect the health of employees and enhance job satisfaction.

    Because of their high accessibility and autonomy, mobile interventions can be a more convenient and resource-saving way to prevent the onset of aggravated stress symptoms from within the workplace. This is especially relevant as recent challenges in the health care workforce include high-pressure working conditions and a shortage of workers, although the number of employees has increased in recent years [58]. It is of utmost importance to focus on the health of these employees, who are putting in the effort to provide the best care for clients.

    An SCED study is typically characterized by a within-person design. Hence, this design does not include a control group but instead uses longer baseline periods and multiple measures over time. This impeded the ability to draw firm conclusions on the effects of the app, especially for a larger group, and therefore highlights the need for future experimentally controlled studies investigating larger and more heterogeneous samples. Further qualitative research is necessary to complement the findings of this study, as it remains unclear how the app contributes to perceived stress, coping skills, and burnout symptoms. By exploring this context and examining the underlying mechanisms and motivations for employees’ behavior, a richer understanding of users’ experiences and more insight into the meaning of the app in everyday work life are created [59,60].

    Although the intervention resulted in a significant reduction in perceived stress immediately after using STAPP@Work, this study found no significant change in stress levels at follow-up. This suggests that the effectiveness of the app in reducing stress in the short term did not persist in the long term. This underscores the importance of conducting future research studies to investigate this specific aspect further. In addition, it would be beneficial to conduct further research on longer-term follow-up assessments to gain more insight into the lasting impact of the app. As the app shows promise in reducing burnout symptoms, it would be clinically relevant to explore the effect of the app on specific groups of individuals, for instance, those with a history of burnout or absenteeism. Because of the accessibility of the app, it can offer solutions to those individuals who are in immediate need of support in managing their stress, such as clients who are on the waiting list for psychological help, which prompts further research ideas.

    Conclusions

    This intervention study showed that STAPP@Work, as a stress management app, is effective in reducing burnout symptoms and improving problem-focused coping skills among mental health workers. The preliminary findings specifically showed reductions in exhaustion levels and emotional impairment. Moreover, the effects of the app emerged consistently and sustainably for >6 months. This offers great opportunities for implementation in the workplace, as the app has the potential to prevent burnout and reduce distress from work-related stress as well as decrease absenteeism.

    Acknowledgments

    The authors thank GGz Centraal for making this research possible with their support. The authors thank the software company, Eaglescience, for developing the app software. The authors gratefully thank all the study participants and the members of the SAM-STAPP (Stress Autism Mate-Stresssignaleringsapplicatie) team who were closely involved in the design and conduct of the study.

    Data Availability

    The data sets generated during and analyzed during this study are available from the corresponding author on reasonable request.

    Authors' Contributions

    JH, KH, and YR contributed to conceptualization, investigation, and project administration. AWH and SD contributed to data curation, formal analysis, methodology, validation, and visualization. SD contributed to the writing of the original draft and the reviewing and editing of the manuscript. AWH contributed to the reviewing and editing of the manuscript. YR and PNvH contributed to supervision and the reviewing and editing of the manuscript.

    Conflicts of Interest

    None declared.

    1. Niedhammer I, Bertrais S, Witt K. Psychosocial work exposures and health outcomes: a meta-review of 72 literature reviews with meta-analysis. Scand J Work Environ Health. Oct 01, 2021;47(7):489-508. [FREE Full text] [CrossRef] [Medline]
    2. Koolhaas JM, Bartolomucci A, Buwalda B, de Boer SF, Flügge G, Korte SM, et al. Stress revisited: a critical evaluation of the stress concept. Neurosci Biobehav Rev. Apr 2011;35(5):1291-1301. [CrossRef] [Medline]
    3. Block JP, He Y, Zaslavsky AM, Ding L, Ayanian JZ. Psychosocial stress and change in weight among US adults. Am J Epidemiol. Jul 15, 2009;170(2):181-192. [FREE Full text] [CrossRef] [Medline]
    4. Jansson C, Wallander MA, Johansson S, Johnsen R, Hveem K. Stressful psychosocial factors and symptoms of gastroesophageal reflux disease: a population-based study in Norway. Scand J Gastroenterol. 2010;45(1):21-29. [CrossRef] [Medline]
    5. Kemeny ME, Schedlowski M. Understanding the interaction between psychosocial stress and immune-related diseases: a stepwise progression. Brain Behav Immun. Nov 2007;21(8):1009-1018. [CrossRef] [Medline]
    6. Yonas MA, Lange NE, Celedón JC. Psychosocial stress and asthma morbidity. Curr Opin Allergy Clin Immunol. Apr 2012;12(2):202-210. [FREE Full text] [CrossRef] [Medline]
    7. Tafet GE, Nemeroff CB. The links between stress and depression: psychoneuroendocrinological, genetic, and environmental interactions. J Neuropsychiatry Clin Neurosci. 2016;28(2):77-88. [CrossRef] [Medline]
    8. Guseva Canu I, Marca SC, Dell'Oro F, Balázs Á, Bergamaschi E, Besse C, et al. Harmonized definition of occupational burnout: a systematic review, semantic analysis, and Delphi consensus in 29 countries. Scand J Work Environ Health. Mar 01, 2021;47(2):95-107. [FREE Full text] [CrossRef] [Medline]
    9. Edú-Valsania S, Laguía A, Moriano JA. Burnout: a review of theory and measurement. Int J Environ Res Public Health. Mar 04, 2022;19(3):1780. [FREE Full text] [CrossRef] [Medline]
    10. Salvagioni DA, Melanda FN, Mesas AE, González AD, Gabani FL, Andrade SM. Physical, psychological and occupational consequences of job burnout: a systematic review of prospective studies. PLoS One. Oct 04, 2017;12(10):e0185781. [FREE Full text] [CrossRef] [Medline]
    11. Lutz N, Taeymans J, Ballmer C, Verhaeghe N, Clarys P, Deliens T. Cost-effectiveness and cost-benefit of worksite health promotion programs in Europe: a systematic review. Eur J Public Health. Jun 01, 2019;29(3):540-546. [CrossRef] [Medline]
    12. Song Z, Baicker K. Effect of a workplace wellness program on employee health and economic outcomes: a randomized clinical trial. JAMA. Apr 16, 2019;321(15):1491-1501. [FREE Full text] [CrossRef] [Medline]
    13. Lokman S, Volker D, Zijlstra-Vlasveld MC, Brouwers EM, Boon B, Beekman AT, et al. Return-to-work intervention versus usual care for sick-listed employees: health-economic investment appraisal alongside a cluster randomised trial. BMJ Open. Oct 05, 2017;7(10):e016348. [FREE Full text] [CrossRef] [Medline]
    14. Van der Ploeg K, Van der Pal S, de Vroome E, Van den Bossche S. De kosten van ziekteverzuim voor werkgevers in Nederland. TNO. Leiden. TNO; 2014. URL: http://tinyurl.com/mss9hw2j [accessed 2023-11-18]
    15. Psychosociale arbeidsbelasting (PSA) werknemers; geslacht en leeftijd. CBS. Centraal Bureau voor de Statistiek URL: http://tinyurl.com/arc73k6t [accessed 2023-03-19]
    16. Psycho-sociale arbeidsbelasting werknemers. CBS Statline. Centraal Bureau voor de Statistiek URL: http://tinyurl.com/2z56sjz2 [accessed 2023-03-19]
    17. Koh J, Tng GY, Hartanto A. Potential and pitfalls of mobile mental health apps in traditional treatment: an umbrella review. J Pers Med. Aug 25, 2022;12(9):1376. [FREE Full text] [CrossRef] [Medline]
    18. Paganini S, Meier E, Terhorst Y, Wurst R, Hohberg V, Schultchen D, et al. Stress management apps: systematic search and multidimensional assessment of quality and characteristics. JMIR Mhealth Uhealth. Aug 29, 2023;11:e42415. [FREE Full text] [CrossRef] [Medline]
    19. Quick JC, Wright TA, Adkins JA, Nelson DL, Quick JC. Preventive Stress Management in Organizations. 2nd edition. Washington, DC. American Psychological Association; 2013.
    20. Alhasani M, Mulchandani D, Oyebode O, Baghaei N, Orji R. A systematic and comparative review of behavior change strategies in stress management apps: opportunities for improvement. Front Public Health. Feb 24, 2022;10:777567. [FREE Full text] [CrossRef] [Medline]
    21. Egger SM, Frey S, Sauerzopf L, Meidert U. A literature review to identify effective web- and app-based mHealth interventions for stress management at work. Workplace Health Saf. Oct 30, 2023;71(10):452-463. [FREE Full text] [CrossRef] [Medline]
    22. Pospos S, Young IT, Downs N, Iglewicz A, Depp C, Chen JY, et al. Web-based tools and mobile applications to mitigate burnout, depression, and suicidality among healthcare students and professionals: a systematic review. Acad Psychiatry. Feb 2018;42(1):109-120. [FREE Full text] [CrossRef] [Medline]
    23. Salvado M, Marques DL, Pires IM, Silva NM. Mindfulness-based interventions to reduce burnout in primary healthcare professionals: a systematic review and meta-analysis. Healthcare (Basel). Oct 09, 2021;9(10):1342. [FREE Full text] [CrossRef] [Medline]
    24. Mistretta EG, Davis MC, Temkit M, Lorenz C, Darby B, Stonnington CM. Resilience training for work-related stress among health care workers: results of a randomized clinical trial comparing in-person and smartphone-delivered interventions. J Occup Environ Med. Jun 2018;60(6):559-568. [CrossRef] [Medline]
    25. Fiol-DeRoque MA, Serrano-Ripoll MJ, Jiménez R, Zamanillo-Campos R, Yáñez-Juan AM, Bennasar-Veny M, et al. A mobile phone-based intervention to reduce mental health problems in health care workers during the COVID-19 pandemic (PsyCovidApp): randomized controlled trial. JMIR Mhealth Uhealth. May 18, 2021;9(5):e27039. [FREE Full text] [CrossRef] [Medline]
    26. Prado K, Robinson A, Chao YY. The impact of mindful meditation on health care workers during the COVID-19 pandemic. J Nurse Pract. Mar 2023;19(3):104513. [FREE Full text] [CrossRef] [Medline]
    27. Hwang WJ, Jo HH. Evaluation of the effectiveness of mobile app-based stress-management program: a randomized controlled trial. Int J Environ Res Public Health. Nov 03, 2019;16(21):4270. [FREE Full text] [CrossRef] [Medline]
    28. Taylor H, Cavanagh K, Field AP, Strauss C. Health care workers' need for headspace: findings from a multisite definitive randomized controlled trial of an unguided digital mindfulness-based self-help app to reduce healthcare worker stress. JMIR Mhealth Uhealth. Aug 25, 2022;10(8):e31744. [FREE Full text] [CrossRef] [Medline]
    29. Hwang WJ, Ha JS, Kim MJ. Research trends on mobile mental health application for general population: a scoping review. Int J Environ Res Public Health. Mar 02, 2021;18(5):2459. [FREE Full text] [CrossRef] [Medline]
    30. Shiffman S, Stone AA, Hufford MR. Ecological momentary assessment. Annu Rev Clin Psychol. Apr 2008;4:1-32. [FREE Full text] [CrossRef] [Medline]
    31. Heron KE, Smyth JM. Ecological momentary interventions: incorporating mobile technology into psychosocial and health behaviour treatments. Br J Health Psychol. Feb 2010;15(1):1-39. [FREE Full text] [CrossRef] [Medline]
    32. Versluis A, Verkuil B, Spinhoven P, van der Ploeg MM, Brosschot JF. Changing mental health and positive psychological well-being using ecological momentary interventions: a systematic review and meta-analysis. J Med Internet Res. Jun 27, 2016;18(6):e152. [FREE Full text] [CrossRef] [Medline]
    33. Dallery J, Cassidy RN, Raiff BR. Single-case experimental designs to evaluate novel technology-based health interventions. J Med Internet Res. Feb 08, 2013;15(2):e22. [FREE Full text] [CrossRef] [Medline]
    34. Krasny-Pacini A, Evans J. Single-case experimental designs to assess intervention effectiveness in rehabilitation: a practical guide. Ann Phys Rehabil Med. May 2018;61(3):164-179. [FREE Full text] [CrossRef] [Medline]
    35. Epstein LH, Bickel WK, Czajkowski SM, Paluch RA, Moeyaert M, Davidson KW. Single case designs for early phase behavioral translational research in health psychology. Health Psychol. Dec 2021;40(12):858-874. [FREE Full text] [CrossRef] [Medline]
    36. Epstein LH, Dallery J. The family of single-case experimental designs. Harv Data Sci Rev. Sep 08, 2022;4(SI3). [FREE Full text] [CrossRef] [Medline]
    37. Stratton E, Lampit A, Choi I, Calvo RA, Harvey SB, Glozier N. Effectiveness of eHealth interventions for reducing mental health conditions in employees: a systematic review and meta-analysis. PLoS One. Dec 21, 2017;12(12):e0189904. [FREE Full text] [CrossRef] [Medline]
    38. Olivier-Pijpers V, Vugts B. Signaleringsplan: Zegen of Vloek?. Australiëlaan, The Netherlands. Centrum voor Consultatie en Expertise; Mar 2016.
    39. Cohen S, Kamarck T, Mermelstein R. A global measure of perceived stress. J Health Soc Behav. Dec 1983;24(4):385-396. [Medline]
    40. Klein EM, Brähler E, Dreier M, Reinecke L, Müller KW, Schmutzer G, et al. The German version of the Perceived Stress Scale - psychometric characteristics in a representative German community sample. BMC Psychiatry. May 23, 2016;16:159. [FREE Full text] [CrossRef] [Medline]
    41. Chesney MA, Neilands TB, Chambers DB, Taylor JM, Folkman S. A validity and reliability study of the coping self-efficacy scale. Br J Health Psychol. Sep 2006;11(3):421-437. [FREE Full text] [CrossRef] [Medline]
    42. Schaufeli WB, Desart S, De Witte H. Burnout Assessment Tool (BAT)-development, validity, and reliability. Int J Environ Res Public Health. Dec 18, 2020;17(24):9495. [FREE Full text] [CrossRef] [Medline]
    43. Hadžibajramović E, Schaufeli W, De Witte H. Shortening of the Burnout Assessment Tool (BAT)-from 23 to 12 items using content and Rasch analysis. BMC Public Health. Mar 22, 2022;22:560. [FREE Full text] [CrossRef] [Medline]
    44. Declercq L, Jamshidi L, Fernández-Castilla B, Beretvas SN, Moeyaert M, Ferron JM, et al. Analysis of single-case experimental count data using the linear mixed effects model: a simulation study. Behav Res Methods. Dec 2019;51(6):2477-2497. [CrossRef] [Medline]
    45. DeHart WB, Kaplan BA. Applying mixed-effects modeling to single-subject designs: an introduction. J Exp Anal Behav. Mar 2019;111(2):192-206. [CrossRef] [Medline]
    46. Hoffman L, Rovine MJ. Multilevel models for the experimental psychologist: foundations and illustrative examples. Behav Res Methods. Mar 2007;39(1):101-117. [CrossRef] [Medline]
    47. Gueorguieva R, Krystal JH. Move over ANOVA: progress in analyzing repeated-measures data and its reflection in papers published in the Archives of General Psychiatry. Arch Gen Psychiatry. Mar 2004;61(3):310-317. [FREE Full text] [CrossRef] [Medline]
    48. Demerouti E, Bakker AB, Nachreiner F, Schaufeli WB. The job demands-resources model of burnout. J Appl Psychol. Jun 2001;86(3):499-512. [FREE Full text] [Medline]
    49. Baker JP, Berenbaum H. Emotional approach and problem-focused coping: a comparison of potentially adaptive strategies. Cogn Emot. Feb 3, 2007;21(1):95-118. [CrossRef]
    50. Phillips EA, Gordeev VS, Schreyögg J. Effectiveness of occupational e-mental health interventions: a systematic review and meta-analysis of randomized controlled trials. Scand J Work Environ Health. Nov 01, 2019;45(6):560-576. [FREE Full text] [CrossRef] [Medline]
    51. Osman I, Hamid S, Singaram VS. Efficacy of a brief online mindfulness-based intervention on the psychological well-being of health care professionals and trainees during the COVID-19 pandemic: a mixed method design. Health SA. Sep 30, 2021;26 [FREE Full text] [CrossRef] [Medline]
    52. Ebert DD, Heber E, Berking M, Riper H, Cuijpers P, Funk B, et al. Self-guided internet-based and mobile-based stress management for employees: results of a randomised controlled trial. Occup Environ Med. May 2016;73(5):315-323. [FREE Full text] [CrossRef] [Medline]
    53. López-Del-Hoyo Y, Fernández-Martínez S, Pérez-Aranda A, Barceló-Soler A, Bani M, Russo S, et al. Effects of eHealth interventions on stress reduction and mental health promotion in healthcare professionals: a systematic review. J Clin Nurs. Sep 2023;32(17-18):5514-5533. [FREE Full text] [CrossRef] [Medline]
    54. Lobo MA, Moeyaert M, Baraldi Cunha A, Babik I. Single-case design, analysis, and quality assessment for intervention research. J Neurol Phys Ther. Jul 2017;41(3):187-197. [FREE Full text] [CrossRef] [Medline]
    55. Smith JD. Single-case experimental designs: a systematic review of published research and current standards. Psychol Methods. Dec 2012;17(4):510-550. [FREE Full text] [CrossRef] [Medline]
    56. Dallery J, Raiff BR. Optimizing behavioral health interventions with single-case designs: from development to dissemination. Transl Behav Med. Sep 2014;4(3):290-303. [FREE Full text] [CrossRef] [Medline]
    57. van Wijk M. Arbeidsmarktprofiel van zorg en welzijn. Centraal Bureau voor de Statistiek. CBS; 2020. URL: http://tinyurl.com/upt93ynn [accessed 2023-11-18]
    58. Health and care workforce in Europe: time to act. World Health Organization. Copenhagen. The WHO Regional Office for Europe; 2022. URL: https://www.who.int/europe/publications/i/item/9789289058339 [accessed 2023-03-04]
    59. Burney V, Arnold-Saritepe A, McCann CM. Rethinking the place of qualitative methods in behavior analysis. Perspect Behav Sci. Mar 2023;46(1):185-200. [FREE Full text] [CrossRef] [Medline]
    60. Bonell C, Warren E, Melendez-Torres GJ. Methodological reflections on using qualitative research to explore the causal mechanisms of complex health interventions. Evaluation. Apr 2022;28(2):166-181. [CrossRef]

    ‎
    BAT: Burnout Assessment Tool
    CSES: Coping Self-Efficacy Scale
    EMA: ecological momentary assessment
    EMI: ecological momentary intervention
    LMM: linear mixed model
    PSM: preventive stress management
    PSS: Perceived Stress Scale
    RCT: randomized controlled trial
    SCED: single-case experimental design

    Edited by T de Azevedo Cardoso; submitted 13.05.23; peer-reviewed by E Phillips, A Ferrario; comments to author 18.10.23; revised version received 06.12.23; accepted 17.01.24; published 29.02.24.

    Copyright

    ©Sevda Demirel, Yvette Roke, Adriaan W Hoogendoorn, Jamie Hoefakker, Kirsten Hoeberichts, Peter N van Harten. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 29.02.2024.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.