This paper is in the following e-collection/theme issue:

Web-based and Mobile Health Interventions (3297) Digital Mental Health Interventions, e-Mental Health and Cyberpsychology (1514) Anxiety and Stress Disorders (1053) Occupational Health and Ergonomics/Prevention at the Workplace (238) mHealth for Wellness, Behavior Change and Prevention (2871) Use and User Demographics of mHealth (335)
Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/22107, first published .
Effectiveness and Moderators of an Internet-Based Mobile-Supported Stress Management Intervention as a Universal Prevention Approach: Randomized Controlled Trial

Effectiveness and Moderators of an Internet-Based Mobile-Supported Stress Management Intervention as a Universal Prevention Approach: Randomized Controlled Trial

Effectiveness and Moderators of an Internet-Based Mobile-Supported Stress Management Intervention as a Universal Prevention Approach: Randomized Controlled Trial

Authors of this article:

David Daniel Ebert1 Author Orcid Image ;   Marvin Franke2 Author Orcid Image ;   Anna-Carlotta Zarski2 Author Orcid Image ;   Matthias Berking2 Author Orcid Image ;   Heleen Riper3 Author Orcid Image ;   Pim Cuijpers3 Author Orcid Image ;   Burkhardt Funk4 Author Orcid Image ;   Dirk Lehr4 Author Orcid Image
Article Authors Cited by (13) Tweetations (3) Metrics

    Original Paper

    1Department of Sport and Health Sciences, Technical University of Munich, Munich, Germany

    2Clinical Psychology and Psychotherapy, Friedrich-Alexander University Erlangen-Nuremberg, Erlangen, Germany

    3Department of Clinical, Neuro and Developmental Psychology, Amsterdam Public Health Research Institute, Vrije Universiteit Amsterdam, Amsterdam, Netherlands

    4Department of Health Psychology and Applied Biological Psychology, Leuphana University of Lueneburg, Lueneburg, Germany

    Corresponding Author:

    David Daniel Ebert, BSc, MSc, PhD

    Department of Sport and Health Sciences

    Technical University of Munich

    Georg-Brauchle-Ring 60/62

    TUM Campus D

    Munich, 80992

    Germany

    Phone: 49 15116781057

    Email: david.daniel.ebert@tum.de


    Background: Emerging evidence indicates the effectiveness of internet-based mobile-supported stress management interventions (iSMIs) in highly stressed employees. It is yet unclear, however, whether iSMIs are also effective without a preselection process in a universal prevention approach, which more closely resembles routine occupational health care. Moreover, evidence for whom iSMIs might be suitable and for whom not is scarce.

    Objective: The aim of this study was to evaluate the iSMI GET.ON Stress in a universal prevention approach without baseline inclusion criteria and to examine the moderators of the intervention effects.

    Methods: A total of 396 employees were randomly assigned to the intervention group or the 6-month waiting list control group. The iSMI consisted of 7 sessions and 1 booster session and offered no therapeutic guidance. Self-report data were assessed at baseline, 7 weeks, and at 6 months following randomization. The primary outcome was perceived stress. Several a priori defined moderators were explored as potential effect modifiers.

    Results: Participants in the intervention group reported significantly lower perceived stress at posttreatment (d=0.71, 95% CI 0.51-0.91) and at 6-month follow-up (d=0.61, 95% CI 0.41-0.81) compared to those in the waiting list control group. Significant differences with medium-to-large effect sizes were found for all mental health and most work-related outcomes. Resilience (at 7 weeks, P=.04; at 6 months, P=.01), agreeableness (at 7 weeks, P=.01), psychological strain (at 6 months, P=.04), and self-regulation (at 6 months, P=.04) moderated the intervention effects.

    Conclusions: This study indicates that iSMIs can be effective in a broad range of employees with no need for preselection to achieve substantial effects. The subgroups that might not profit had extreme values on the respective measures and represented only a very small proportion of the investigated sample, thereby indicating the broad applicability of GET.ON Stress.

    Trial Registration: German Clinical Trials Register DRKS00005699; https://www.drks.de/DRKS00005699

    J Med Internet Res 2021;23(12):e22107

    doi:10.2196/22107

    Keywords

    stress management intervention; universal prevention; occupational health; moderators 


    Occupational stress is a major public health problem, with high prevalence in western countries [1-3]. It is associated with a range of adverse consequences, including an increased risk for coronary disease [4,5], mental health problems [6,7], and sleeping problems [8]. Moreover, stress-related productivity loss, absenteeism, presentism, and health care usage lead to substantial economic costs [9]. Although manifold meta-analytic evidence exists for the effectiveness of psychological face-to-face stress management interventions for employees [10,11], the high prevalence of stress demands for highly cost-effective and scalable solutions. Internet-based mobile-supported stress management interventions (iSMIs) can be a great solution as they can offer several benefits such as (1) their accessibility at any time and place, (2) the possibility for participants to work and review materials at their own pace, and (3) their high potential for scalability [12]. In theory, only a small increase in resources is required for reaching a greater proportion of the eligible population. However, the real costs of iSMIs are greatly linked to the amount of guidance delivered by professional support [13], thereby limiting the possible reach of these interventions. Therefore, unguided interventions are an important puzzle piece to combat the high prevalence of stress.

    The most recent meta-analysis on iSMIs found overall significant effectiveness (d=0.43, 95% CI 0.31-0.54) with small-to-medium effect sizes, but substantial heterogeneity between studies [14]. Subgroup analyses found that guided interventions (d=0.64, 95% CI 0.50-0.79; n=7) were significantly more effective than unguided interventions (d=0.33, 95% CI 0.20-0.46; n=18). Yet, the most effective iSMI GET.ON Stress yielded high effect sizes even when delivered in an unguided format [13], with an effect size (d=0.96, 95% CI 0.70-1.21) significantly exceeding that reported in the overall sample. In regard to the often cited replication crisis in Psychology [15] and other fields [16-18], it is crucial to evaluate if these effect sizes can withhold further inspection. Moreover, replicating these studies can help to address their shortcomings and add to the existing literature.

    The first shortcoming relates to the evaluation in selected samples. Evidence for the best evaluated iSMI GET.ON Stress [13,19-25] is based on trials in which participants have been preselected based on the high baseline score in perceived stress (Perceived Stress Scale [PSS-10] ≥22) [26,27]. Using a broad range of eligibility criteria, however, can limit the real-life applicability of the results [28], as in routine preventive occupational health care, iSMIs are usually offered to a wide range of participants. Therefore, it is crucial to investigate treatment effects without preselection in a universal prevention approach [29]. This approach has not only theoretical but also practical advantages: while reaching a wider proportion of the working population, it reaches those that benefit from selected and indicated prevention without the cost for screening them specifically [30,31]. Additionally, it allows the inclusion of adults with lower symptom severity who are still motivated to improve their conditions, which might be a better indicator for their readiness for health behavior change [32].

    The second shortcoming relates to the lack of moderator analyses. Even though internet-based intervention research is rapidly growing, empirical data on the moderators of internet-based mobile-supported interventions [33-39] and, in particular, internet occupational health interventions are scarce [34]. Furthermore, randomized controlled trials (RCTs) are usually powered to detect effects on the primary outcome and hence, underpowered to reliably test moderator hypotheses [40]. With reported numbers needed to treat of 5.43 for unguided iSMIs (and 4.20 in the overall sample) [14], it is clear, however, that a substantial proportion of the participants do not profit yet from taking part in an intervention. Therefore, identifying the moderators of iSMI effects is crucial for at least 3 reasons: (1) knowing who likely profits from the intervention helps to identify relevant populations, (2) knowing which patients are unlikely to profit from the treatment helps to prevent wasting resources and provides valuable information on where custom-tailoring such interventions to subgroups of patients is necessary, and (3) a better knowledge regarding who is (un)likely to profit from these interventions helps to identify mechanisms of change that are relevant for these interventions [41]. Therefore, the aim of this study was to evaluate the effects of the iSMI GET.ON Stress as a universal prevention approach, without preselecting participants based on symptom severity. Moreover, in an exploratory approach, a broad range of a priori selected potential effect modifiers were tested in a sample adequately powered for moderator analyses.


    Design

    A two-armed RCT (N=396) was conducted comparing a self-guided iSMI (GET.ON Stress) condition (intervention group [IG]) and a waiting list control (WLC) condition, with both conditions having full access to treatment as usual. Participants were recruited via the occupational health program of a large health insurance company in Germany (BARMER) in a way that mimics the intended implementation of the intervention in routine practice in the future, that is, by offering the intervention in a public occupational mental health approach to the general working population in a web-based setting. Recruitment was directed at the general working population and not restricted to members of the health care insurance company. Recruitment occurred mainly through reports in the membership magazine of the insurance company, and the insurance company’s occupational health management workers informed human resource departments of collaborating small- and medium-sized companies about the possibility for their employees to participate in the trial. Assessments took place at baseline (T1), at posttreatment (7 weeks, T2), and at 6 months (T3; see Figure 1 for a detailed overview of assessments). This study was approved by the University of Marburg ethics committee.

    ‎
    Figure 1. Flow of the study. T1: baseline; T2: at 7 weeks posttreatment; T3: at 6 months posttreatment.
    View this figure

    Interested individuals received an email with detailed information about the study procedures and an invitation to take part in the study. If they were interested, they were asked to complete a web-based screening questionnaire. Individuals meeting the eligibility criteria completed the baseline assessment and were invited to complete the informed consent form. Once the full written informed consent was received, participants entered the study and were randomly allocated to 1 of the 2 study conditions. Randomization took place at a ratio of 1:1. An independent third party who did not have any information about the participants performed the allocation. Randomization was carried out using an automated computer-based random integer generator (randlist). Other researchers could not bias the randomization process since participants were randomized in the order of the incoming informed consent form. Participants were not blinded to study conditions. During the randomization process, the allocation was concealed from the participants, researchers involved in recruitment, and the study administration team.

    Eligibility Criteria

    Inclusion and exclusion criteria were kept to a minimum to mimic routine conditions as closely as possible. Participants were included who (1) were 18 years and older, (2) were currently employed, (3) had internet access, and (4) provided a valid email address. Individuals were only excluded if they were at risk of suicide (indicated by a score of 2 or higher on the Beck Depression Inventory Suicide Item [42]) or if they self-reported to have been diagnosed with psychosis or dissociative symptoms. Besides that, there were no restraints for participants, for example, concerning low or high stress, depression, or other symptom severity criteria.

    Intervention

    As this study evaluates an intervention that was evaluated in an indicated prevention context before, intervention content and delivery methods mirror those in related studies [20]. The training GET.ON Stress is based on Lazarus’ transactional model of stress [43]. According to this model, there are 2 coping strategies to deal with stress. One is a problem-oriented approach in changing the situation. The other approach focusses on situations where only the evaluation of the situation can be adjusted; this is called emotional regulation. Therefore, the training focuses on strategies for systematic problem-solving as well as emotion regulation. The intervention consists of 7 regular sessions and 1 booster session (provided 4 weeks after the last regular session was completed). The program starts with psychoeducation about stress (session 1), which guides participants to understand and find coping strategies for common problem situations. Based on their personal stressors, participants are asked to record their goals and motivation for the training. This is followed by 2 sessions focusing on problem-solving skills (sessions 2-3) where participants are guided through a 6-step method based on problem-solving therapy [44]. The personal plan will be implemented between the sessions before participants are asked to redo the exercise with the same or a different problem. The next part of the training (sessions 4-6) is based on the Affect Regulation training [45,46]. Each session focusses on 1 emotion regulation strategy: muscle and breathing relaxation, acceptance of emotions, as well as effective self-support with 15-minute audio files guiding users through the different exercises. Participants are encouraged to redo the exercises between sessions on a daily basis. The regular training is completed by a session guiding the creation of a “Plan for the future” (session 7), in which users are asked to review their progress and develop strategies according to their personal stress indicators in the future. Moreover, they are guided to write a letter to their future self. The booster session reviews the essential content of the intervention and allows participants to reassess their goals and plans for the future. Additionally, participants could choose from an array of elective modules in sessions 2 to 6 addressing common stress-related topics such as time management, sleep hygiene, or social support. Participants were advised to complete 1-2 sessions per week.

    Across sessions, the training was composed of interactive education, audio and video files, and additional downloadable material. Additionally, testimonials offered impulses and examples concerning the exercises. The intervention is based on responsive design, meaning that it could be completed on a desktop computer, tablet, or smartphone. Through providing multiple choices among various response options, the content of the intervention was automatically tailored to the specific needs and interests of the individual participants. To integrate the newly acquired knowledge into daily life, homework assignments and behavioral planning were a crucial part of the training. Web-based diaries were also offered. The training was not accompanied by therapeutic guidance. However, the participants could opt for automatic text messages on their mobile phones choosing either a light (1 text message every other day) or intensive support (2-3 text messages per day). Text messages focused on ultrabrief exercises to be carried out in daily life routine, aiming to facilitate transfer from training into real life.

    Outcomes

    All measures were assessed through web-based self-report questionnaires and are described below. The assessment took place on a secured web-based system with 256-bit AES encryption. All outcomes were published a priori in detail in the trial register. The primary outcome was the perceived level of stress as measured by the German version of the PSS-10 [26,27] since it is based on Lazarus’ transactional model of stress. The 10 items were answered on a 5-point Likert scale (ranging from 0=never to 4=very often) referring to the past week. Owing to its sum score, the total scale scores range from 0 to 40. Cronbach alpha was .83 at T1, .91 at T2, and .91 at T3 in this study.

    Secondary outcomes included measurement of mental health and work-related outcomes. Among those concerning mental health, the following outcomes were measured using specified scales: depression, using the German adaptation of the Center for Epidemiological Studies’ Depression Scale (15 items, range 0-45; α=.94) [47,48]; mental health–related quality of life, using the Assessment of Quality of Life (8 items, range 8-41; α=.84) [49]; resilience, using the Connor-Davidson Resilience Scale (10 items, range 10-50; α=.93) [50]; emotional exhaustion, using the subscale of the Maslach-Burnout-Inventory (5 items, range 1-6; α=.93) [51]; mental well-being, using the Warwick-Edinburgh Mental Wellbeing Scale (14 items, range 14-84; α=.96) [52]; and psychological well-being, using the 5-item World Health Organization Well-Being-Index (5 items, range 0-100; α=.91) [53]. The following work-related outcomes were assessed: effort-reward imbalance, using the Effort-Reward-Imbalance Questionnaire-Short Form (10 items, α=.84) [54]; occupational self-efficacy, using the Occupational Self-Efficacy Scale (6 items, range 6-36; α=.93) [55]; work engagement, using the Utrecht Work Engagement Scale (9 items, range 0-6; α=.97) [56]; productivity loss, using the Work-Limitations-Questionnaire (8 items, α=.87) [57]; and absenteeism and presentism days, using the respective questions from the German version of the Trimbos and Institute of Medical Technology Assessment Cost Questionnaire for Psychiatry (2 items) [58].

    In addition, the following variables were assessed to be tested as potential moderators: personality, using the Big-Five-Inventory (10 items, subscale range α=.20-.77) [59]; motivation for treatment, using the Psychotherapy Motivation Questionnaire-Short Form (4 items, α=.36) [60]; self-regulation competencies, using the Self-Regulation scale (10 items, range 10-40; α=.84) [61]; and general self-efficacy, using the General Self-Efficacy Scale (10 items, range 10-40; α=.90) [62].

    Power Analysis

    The sample size of the study was optimized to detect potential moderator effects. Previous studies on the same intervention found effects on the primary outcome of perceived stress at T2 ranging from d=0.84 to d=0.93 [13,23,63] in highly stressed employees. Assuming potentially lower effects, owing to the inclusion of less impaired employees, of approximately d=0.60, we would have needed to include 90 participants. Given that simulation studies showed that the sample size required to detect an interaction effect of the same magnitude needs to be at least threefold to fourfold, compared to the sample sizes needed for the primary analyses [64], we included 396 participants. This sample allowed us not only to detect a small regression effect of f² with a power (1-ß) of 80% and an error probability (α) of .05 but also a small intergroup effect size of d=0.25 in a one-tailed test (calculated with G*Power).

    Statistical Methods

    All analyses are reported following the Consolidated Standards of Reporting Trials guidelines for reporting parallel group randomized trials [65]. Therefore, following the intention-to-treat principle, missing data were handled with multiple imputations [66], using 100 estimations per missing value. Outcome levels of IG and WLC were compared at T2 and T3 by using analysis of covariance (ANCOVA), with baseline levels as covariates. Additionally, Cohen d with 95% CIs were calculated based on the imputed data set by comparing means and SDs of the 2 groups at the respective time points. Furthermore, the difference in means along with its 95% CIs was calculated. To improve interpretability, reliable change was calculated according to the method of Jacobson and Truax [67] as an indicator of the number of participants with treatment response. Participants were accredited a treatment response if their PSS-10 score differed more than SD 5.16 in T1-T2 and T1-T3. Besides, the number needed to treat to achieve 1 additional treatment response compared to the control group was calculated. To evaluate the effect of the intervention on the risk for symptom worsening, we also calculated the number of participants with a reliable deterioration of symptoms by using the Reliable Change Index and calculated the absolute and relative reduction of risk, including the respective 95% CIs.

    Moderator Analysis

    Regression analyses were used to test baseline moderator X intervention condition interaction effects by using the SPSS macro PROCESS [68]. In case of a significant interaction effect, we applied the Johnson-Neyman technique [68,69], which tests the conditional effect of X on Y at different values along the continuum of the moderator and calculates transition points, where the effect changes between statistically significant and not significant (at the α level of significance). This allows to identify a “region of significance” of the effect, giving away at which values of the moderator a significant effect can be found. This warrants essential advantages against the pick-a-point approach in which mostly only 3 arbitrarily chosen values of the moderator are tested since even the widespread use of the mean as well as SD is highly sample-specific and can thus lead to wrong conclusions [68]. Moderation variables were neither standardized nor mean-centered since it makes no difference for the moderation effect [68] and can even harm the interpretation if dichotomous variables are altered before the analysis.


    Participants

    The study flow is illustrated in Figure 1. All 396 participants provided data at T1. In the IG, 176 (88.8%) participants at T2 and 134 (67.7%) participants at T3 and in the WLC group, 187 (94.4%) participants at T2 and 179 (90.4%) participants at T3 completed the assessment. The demographic characteristics of participants can be found in Table 1. The average age of the participants was 41.76 (SD 10.09) years. The sample was predominantly female (302/396, 76.3%), married, or in a relationship (209/396, 52.8%), as well as highly educated (285/396, 72%). Most participants were employed full-time (296/396, 74.7%), in a permanent employment relationship (306/396, 77.3%), and nearly half of them held a management function (169/396, 42.7%). Only a small portion (35/396, 8.8%) was self-employed. The average working experience was 17.58 (SD 10.36) years and the participants’ jobs were in various working sectors, with the majority in the economy (97/396, 24.5%) or social (79/396, 19.9%) sector. Only a small percentage had prior experience with health training (55/396, 13.9%). Having received psychotherapy in the past was stated by 147 (37.1%) participants, whereas only 35 (8.8%) indicated that they were currently receiving psychotherapy. Table 2 and Table 3 summarize the mean (SD) for the IG and WLC. Concerning the primary outcome at T1, participants had an average value of 22.65 (SD 5.63) on the PSS, thereby indicating a moderate stress level.

    Table 1. Baseline characteristics of the study population (N=396).
    CharacteristicsAll participants (N=396)Intervention group (n=198)Waiting list control group (n=198)
    Sociodemographic characteristics

    		Age (years), mean (SD)41.76 (10.09)41.96 (10.34)41.56 (9.87)

    		Gender, female, n (%)302 (76.3)154 (77.8)148 (74.7)

    		Married or in a relationship, n (%)209 (52.8)106 (53.5)103 (52)

    		Having kids, n (%)185 (46.7)91 (46)94 (47.5)

    		West Germany, n (%)351 (88.6)176 (88.9)175 (88.4)
    Ethnicity, n (%)

    		Caucasian/White319 (80.6)161 (81.3)158 (79.8)

    		Asian9 (2.3)4 (2)5 (2.5)

    		Hispanic2 (0.5)2 (1)0 (0)

    		Prefer not to say66 (16.7)31 (15.7)35 (17.7)
    Education, n (%)

    		Low17 (4.3)8 (4)9 (4.5)

    		Middle94 (23.7)52 (26.3)42 (21.2)

    		High285 (72)138 (69.7)147 (74.2)
    Working characteristics

    		Full-time, n (%)296 (74.7)144 (72.7)152 (76.8)

    		Part-time, n (%)93 (23.5)50 (25.3)43 (21.7)

    		On sick leave, n (%)7 (1.8)4 (2)3 (1.5)

    		Management function, n (%)169 (42.7)87 (43.9)82 (41.4)

    		Work experience years, mean (SD)17.58 (10.36)17.62 (10.48)17.54 (10.28)
    Employment status, n (%)

    		Permanent306 (77.3)158 (79.8)148 (74.7)

    		Temporary42 (10.6)17 (8.6)25 (12.6)

    		Self-employed35 (8.8)15 (7.6)20 (10.1)

    		Other13 (3.3)8 (4)5 (2.5)
    Working sectors, n (%)

    		Economy97 (24.5)45 (22.7)52 (26.3)

    		Service59 (14.9)27 (13.6)32 (16.2)

    		Social79 (19.9)46 (23.2)33 (16.7)

    		Health62 (15.7)32 (16.2)30 (15.2)

    		Information technology44 (11.1)21 (10.6)23 (11.6)

    		Other55 (13.9)27 (13.6)28 (14.1)
    Gross annual income (in €), n (%)

    		Low income89 (22.5)37 (18.7)52 (26.3)

    		Middle income134 (33.8)72 (36.4)62 (31.3)

    		High income123 (31.1)61 (30.8)62 (31.3)

    		Not reported50 (12.6)28 (14.1)22 (11.1)
    Experience, n (%)

    		Previous health training55 (13.9)30 (15.2)25 (12.6)

    		Previous psychotherapy147 (37.1)73 (36.9)74 (37.4)

    		Current psychotherapy35 (8.8)16 (8.1)19 (9.6)
    Table 2. Means and standard deviations for the intention-to-treat sample at pretreatment.a
    OutcomeIntervention group (n=198)Waiting list control group (n=198)

    		Mean (SD)RangeMean (SD)Range
    Primary outcome

    		PSSb22.39 (5.49)8-3922.91 (5.77)6-37
    Mental health

    		CES-Dc16.02 (7.87)1-4316.44 (7.55)1-38

    		AQoL8D-MHd,e0.26 (0.11)0.05-0.610.25 (0.12)0.06-0.56

    		CD-RISCe,f22.14 (6.63)1-3821.29 (7.00)2-40

    		MBI-EE-Dg4.44 (0.85)1.8-64.43 (0.91)1.6-6.0

    		WEMWBSe,h44.36 (7.51)22-6342.71 (7.97)24-62

    		WHO-5e,i35.09 (16.16)0-8035.05 (16.94)0-96
    Work-related outcomes

    		ERI-S ratioj1.34 (0.53)0.42-3.621.30 (0.39)0.29-3.09

    		OSESe,k23.60 (6.53)7-3623.84 (6.35)6-36

    		Absenteeism days5.54 (10.75)0-663.62 (7.97)0-54

    		Presentism days11.63 (13.50)0-6612.73 (15.93)0-66

    		UWESe,l3.35 (1.18)0.33-5.893.18 (1.27)0-5.67

    		WLQm productivity loss8.44 (5.38)0-22.828.36 (5.14)0-24.53

    aMissing data imputed by multiple imputation.

    bPSS: Perceived Stress Scale.

    cCES-D: Center for Epidemiological Studies’ Depression Scale.

    dAQoL8D-MH: Assessment Quality of Life 8-Dimensions (mental health component).

    eHigher scores indicate better outcomes.

    fCD-RISC: Connor-Davidson Resilience Scale.

    gMBI-EE-D: Maslach Burnout Inventory (depletion subscale).

    hWEMWBS: Warwick-Edinburgh Mental Wellbeing Scale.

    iWHO-5: 5-item World Health Organization Well-Being Index.

    jERI-S: Effort-Reward-Imbalance Questionnaire-Short form.

    kOSES: Occupational Self-Efficacy Scale.

    lUWES: Utrecht Work Engagement Scale.

    mWLQ: Work Limitations Questionnaire.

    Table 3. Means and standard deviations for the intention-to-treat sample at posttreatment (T2) and 6-month follow-up (T3).
    OutcomePosttreatment at 7 weeksaAt 6-month follow-upa

    		Intervention group (n=198), mean (SD)Waiting list control group (n=198), mean (SD)Intervention group (n=198), mean (SD)Waiting list control group (n=198), mean (SD)
    Primary outcome

    		PSSb16.27 (6.18)21.22 (6.97)15.73 (6.07)19.94 (7.15)
    Mental health

    		CES-Dc11.26 (7.46)15.52 (8.47)11.45 (7.21)15.37 (8.47)

    		AQoL8D-MHd,eN/AfN/A0.36 (0.12)0.29 (0.14)

    		CD-RISCe,g24.41 (6.72)20.79 (7.39)N/AN/A

    		MBI-EE-Dh3.94 (1.05)4.31 (0.99)3.73 (1.05)4.27 (1.01)

    		WEMWBSe,i49.28 (7.42)44.19 (8.34)50.55 (7.24)44.02 (8.24)

    		WHO-5e,j51.25 (18.02)40.82 (19.92)54.80 (15.64)40.79 (20.12)
    Work-related outcomes

    		ERI-S ratiok1.19 (0.51)1.24 (0.45)1.17 (0.40)1.25 (0.42)

    		OSESe,l26.21 (5.96)23.89 (6.76)N/AN/A

    		Absenteeism daysN/AN/A4.03 (7.84)3.22 (6.44)

    		Presentism daysN/AN/A8.88 (9.62)11.75 (12.91)

    		UWESe,m3.40 (1.22)3.02 (1.33)3.50 (1.03)3.06 (1.25)

    		WLQn productivity loss7.48 (5.67)8.47 (5.50)N/AN/A

    aMissing data imputed by multiple imputation.

    bPSS: Perceived Stress Scale.

    cCES-D: Center for Epidemiological Studies’ Depression Scale.

    dAQoL8D-MH: Assessment Quality of Life 8-Dimensions (mental health component).

    eHigher scores indicate better outcomes.

    fN/A: not applicable.

    gCD-RISC: Connor-Davidson Resilience Scale.

    hMBI-EE-D: Maslach Burnout Inventory (depletion subscale).

    iWEMWBS: Warwick-Edinburgh Mental Wellbeing Scale.

    jWHO-5: 5-item World Health Organization Well-Being Index.

    kERI-S: Effort-Reward-Imbalance Questionnaire-Short form.

    lOSES: Occupational Self-Efficacy Scale.

    mUWES: Utrecht Work Engagement Scale.

    nWLQ: Work Limitations Questionnaire.

    Primary and Secondary Outcomes

    Tables 4 and 5 display the results of the intention-to-treat analyses for the primary and secondary outcomes. The ANCOVA showed a significant group effect, indicating that participants in the IG had, compared to the WLC, significantly lower scores on the PSS-10 at T2 (F1,393=64.44, P<.001) and T3 (F1,393=46.91, P<.001) with medium-to-large between-group-effect sizes at T2 (Cohen d=0.71, 95% CI 0.51-0.91) and T3 (d=0.61, 95% CI 0.41-0.81). At posttreatment, significantly more participants in the IG (109/198, 55.1%) showed a reliable improvement on the PSS-10 compared to the WLC (44/198, 22.2%; P<.001). The number needed to treat to achieve 1 additional treatment response (reliable change) at posttreatment was 3.05 (95% CI 2.39-4.20). Significantly less participants experienced a symptom deterioration in the IG (10/198, 5.1%) compared to WLC (19/198, 9.6%), which reflects an absolute risk reduction of 4.55% (95% CI –0.567% to 9.658%) and a relative risk reduction of 47.88% (95% CI –10.307% to 74.888%).

    Table 4. Results of the analysis of covariances and Cohen d for the primary and secondary outcome measures at posttreatment.
    OutcomeAt posttreatment after 7 weeksa between-groups effect
    Difference in means (95% CI)Cohen d (95% CI)ANCOVAb, F(1,393)
    Primary outcome

    		PSSc–4.66 (–5.80 to –3.51)0.71 (0.51 to 0.91)64.44d
    Mental health

    		CES-De–4.00 (–5.27 to –2.74)0.55 (0.35 to 0.75)38.74d

    		AQoL8D-MHf,gN/AhN/AN/A

    		CD-RISCf,i2.97 (2.02 to 3.93)0.54 (0.34 to 0.74)37.27d

    		MBI-EE-Dj–0.38 (–0.54 to –0.23)0.47 (0.27 to 0.67)23.60d

    		WEMWBSf,k4.05 (2.81 to 5.29)0.50 (0.30 to 0.70)41.31d

    		WHO-5f,l10.41 (7.16 to 13.66)0.58 (0.38 to 0.78)39.65d
    Work-related outcomes

    		ERI-Sm ratio–0.07 (–0.15 to 0.002)0.21 (0.01 to 0.41)3.65

    		OSESf,n2.49 (1.58 to 3.39)0.51 (0.31 to 0.71)29.30d

    		Absenteeism daysN/AN/AN/A

    		Presentism daysN/AN/AN/A

    		UWESf,o0.25 (0.08 to 0.43)0.23 (0.03 to 0.42)8.17d

    		WLQp–1.01 (–2.07 to 0.05)0.16 (–0.03 to 0.36)3.51

    aMissing data imputed by multiple imputation.

    bControlling for pretreatment scores (T1).

    cPSS: Perceived Stress Scale.

    dSignificant at P<.001.

    eCES-D: Center for Epidemiological Studies’ Depression Scale.

    fHigher scores indicate better outcomes.

    gAQoL8D-MH: Assessment Quality of Life 8-Dimensions (mental health component).

    hN/A: not applicable.

    iCD-RISC: Connor-Davidson Resilience Scale.

    jMBI-EE-D: Maslach Burnout Inventory (depletion subscale).

    kWEMWBS: Warwick-Edinburgh Mental Wellbeing Scale.

    lWHO-5: 5-item World Health Organization Well-Being Index.

    mERI-S: Effort-Reward-Imbalance Questionnaire-Short form.

    nOSES: Occupational Self-Efficacy Scale.

    oUWES: Utrecht Work Engagement Scale.

    pWLQ: Work Limitations Questionnaire.

    Table 5. Results of the analysis of covariances and Cohen d for the primary and secondary outcome measures at posttest and 6-month follow-up.
    OutcomeT3a Between-groups effect
    Difference in means (95% CI)Cohen d (95% CI)ANCOVAb, F(1,393)
    Primary outcome

    		Perceived Stress Scale–3.89 (–5.01 to –2.77)0.61 (0.41 to 0.81)46.9c
    Mental health

    		Center for Epidemiological Studies’ Depression Scale–3.65 (–4.87 to –2.44)0.52 (0.32 to 0.72)34.88c

    		AQoL8D-MHd,e0.06 (0.04 to 0.08)0.52 (0.31 to 0.72)32.09c

    		CD-RISCd,fN/AN/AN/A

    		MBI-EE-Dg–0.56 (–0.73 to –0.39)0.61 (0.41 to 0.81)41.28c

    		WEMWBSd,h5.63 (4.34 to 6.93)0.66 (0.46 to 0.86)73.25c

    		WHO-5d,i13.99 (10.78 to 17.21)0.76 (0.55 to 0.96)73.32c
    Work-related outcomes

    		ERI-Sj ratio–0.10 (–0.17 to –0.03)0.77 (0.56 to 0.97)7.53k

    		OSESd,lN/AN/AN/A

    		Absenteeism days0.48 (–0.91 to 1.87)0.11 (–0.09 to 0.30)0.46

    		Presentism days–2.44 (–4.40 to –0.49)0.13 (–0.07 to 0.33)6.04m

    		UWESd,n0.33 (0.15 to 0.50)0.25 (0.05 to 0.45)13.37c

    		WLQoN/AN/AN/A

    aMissing data imputed by multiple imputation.

    bControlling for pretreatment scores (T1).

    cSignificant at P<.001.

    dHigher scores indicate better outcomes.

    eAQoL8D-MH: Assessment Quality of Life 8-Dimensions (mental health component).

    fCD-RISC: Connor-Davidson Resilience Scale.

    gMBI-EE-D: Maslach Burnout Inventory (depletion subscale).

    hWEMWBS: Warwick-Edinburgh Mental Wellbeing Scale.

    iWHO-5: 5-item World Health Organization Well-being Index.

    jERI-S: Effort-Reward-Imbalance Questionnaire-Short form.

    kSignificant at P<.01.

    lOSES: Occupational Self-Efficacy Scale.

    mSignificant at P<.05.

    nUWES: Utrecht Work Engagement Scale.

    oWLQ: Work Limitations Questionnaire.

    Comparing reliable changes from baseline to 6 months, 53.5% (106/198) in the IG showed a reliable improvement in contrast to 36.4% (72/198) in the WLC (P<.001). The number needed to treat to achieve one additional reliable improvement at 6 months follow-up was 5.82 (95% CI 3.73-13.30). A reliable deterioration was only present in 2% (4/198) of the IG, which was significantly lesser (P<.05) than that in the WLC (13/198, 6.6%).

    As shown in Tables 4 and 5, the ANCOVAs for mental health outcomes showed significant between-group effects for all outcomes at both assessment points on a P<.001 level in favor of the IG. At T2, all effect sizes were moderate ranging from d=0.47 for emotional exhaustion to d=0.58 for well-being according to the 5-item World Health Organization Well-Being Index. At T3, moderate-to-large effect sizes could be found ranging from d=0.52 for mental health to d=0.76 for well-being.

    Concerning work-related outcomes, the results were less coherent. At T2, the ANCOVAs for work-related self-efficacy (P<.001) and work engagement (P=.004) showed significant between-group effects, with a small effect size for work engagement (d=0.23) and a moderate effect size for occupational self-efficacy (d=0.51). There was no significant difference between groups for effort-reward-ratio (P=.06) and productivity loss (P=.06). However, at T3, apart from absenteeism days (P=.50), all between-group effects were significant (effort-reward-imbalance T3, P=.006; presentism days T3, P=.01; work engagement T3, P<.001), with a large effect size of d=0.77 for effort-reward-imbalance and a small effect size of d=0.25 for work engagement and presentism days (d=0.13).

    Adherence

    Of the 198 participants in the IG, 188 (95%) finished session 1. Session 2 was completed by 171 (86.4%), session 3 by 154 (77.8%), session 4 by 135 (68.2%), session 5 by 121 (61.1%), session 6 by 108 (54.5%), session 7 by 94 (47.5%), and the booster session by 66 (33.3%) individuals. On average, participants worked through 5.23 (SD 2.74) sessions representing 66% (5.23/8) of the intervention. A linear regression model controlling for baseline stress indicated that the number of completed sessions significantly predicted stress levels at T2 (b=–0.67, SE 0.14; P<.001; 95% CI –0.94 to –0.39) and T3 (b=–0.39, SE 0.14; P=.004; 95% CI –0.66 to –0.12). The regression coefficient suggested that the PSS value decreased 0.67 (T2) and respectively 0.39 (T3) points with each additional module completed.

    Moderator Analysis

    Baseline symptom severity was not found to be a significant moderator, indicating that the intervention can be effective, irrespective of the baseline level of perceived stress. Resilience, agreeableness, psychological strain, and self-regulation competencies significantly moderated the treatment outcome. Resilience moderated the intervention effect at T2 (P=.04) as well as at T3 (P=.01). The region of significance at T2 and T3 ranged from 1 to 34.28 and 1 to 30.63, respectively, indicating that there was no significant between-group effect for participants with higher levels of resilience at baseline. In total, 2.8% (11/396) of the sample had resilience scores >34.28; 8.8% (35/396) had scores >30.63. Agreeableness moderated the intervention effect on stress at T2 (P=.01) with a region of significance from 1 to 4.55, indicating no significant intervention effects within participants with a higher agreeableness score (2 out of 396 participants had agreeableness scores above 4.55). The intervention effect on stress at T3 was moderated by psychological strain (P=.04) and self-regulation (P=.04). No significant between-group effect was found for participants with a psychological strain score lower than 7.96 (19/396, 4.8%) or self-regulation scores higher than 33.57 (24/396, 6%). Tables S1-S4 in Multimedia Appendix 1 show the results of the moderation analyses whereas Figures 2-6 show a visual representation of the moderation accompanied by a visual representation of the region of significance.

    ‎
    Figure 2. A visual representation of the moderation of the training effect (X) on perceived stress at T2 (Y) by resilience at T1 (M) accompanied with a visual representation of the area of significance according to the Johnson-Neyman technique. IG: intervention group; T1: baseline; T2: at 7 weeks posttreatment; WLC: waiting list control.
    View this figure
    ‎
    Figure 3. A visual representation of the moderation of the training effect (X) on perceived stress at T3 (Y) by resilience at T1 (M) accompanied with a visual representation of the area of significance according to the Johnson-Neyman technique. IG: intervention group; T1: baseline; T3: at 6 months posttreatment; WLC: waiting list control.
    View this figure
    ‎
    Figure 4. A visual representation of the moderation of the training effect (X) on perceived stress at T2 (Y) by agreeableness at T1 (M) accompanied with a visual representation of the area of significance according to the Johnson-Neyman technique. IG: intervention group; T1: baseline; T2: at 7 weeks posttreatment; WLC: waiting list control.
    View this figure
    ‎
    Figure 5. A visual representation of the moderation of the training effect (X) on perceived stress at T3 (Y) by psychological strain at T1 (M) accompanied with a visual representation of the area of significance according to the Johnson-Neyman technique. IG: intervention group; T1: baseline; T3: at 6 months posttreatment; WLC: waiting list control.
    View this figure
    ‎
    Figure 6. A visual representation of the moderation of the training effect (X) on perceived stress at T3 (Y) by self-regulation at T1 (M) accompanied with a visual representation of the region of significance according to the Johnson-Neyman technique. IG: intervention group; T1: baseline; T3: at 6 months posttreatment; WLC: waiting list control.
    View this figure

    No demographic variable (age, gender, relationship status, having kids), other work-related characteristics (full-time work, having a management function, work experience, employment status, working sector, gross annual income), as well as other symptom severity indicators were significantly associated with the intervention effects. P values of nonsignificant interaction effects ranged from P=.08 (T2) for gender to P=.99 for conscientiousness.


    This study aimed to evaluate the effectiveness and moderators of treatment outcome of a self-guided iSMI as a universal prevention approach without an elaborative inclusion process based on baseline symptom severity. The results of our study indicate that the training was highly effective in reducing stress levels in the short term (d=0.71) and long term (d=0.61). Significant moderate-to-large effects were found for several secondary mental health and work-related outcomes, including outcomes such as occupational self-efficacy for which no evidence of iSMIs was yet available. Moreover, this study is one of the first to investigate moderator effects in iSMIs in an adequately powered sample. The intervention was suitable for a wide range of participants. Only a few moderators of the intervention effect were identified, indicating that employees with very high resilience, very low psychological strain, very high agreeableness, and very high self-regulation might not profit from the iSMI.

    The effects in this study were lower than those in the 4 previously conducted RCTs that examined the same iSMI but in which participants were preselected (PSS>22 or PSS-4 ≥8). With therapeutic guidance, large effect sizes of d=0.83 at posttreatment (95% CI 0.58-1.08) were found compared to that of WLC [63], whereas with adherence-focused guidance [23], the between-group effects were d=0.79 (95% CI 0.54-1.04), and as a purely self-guided intervention in employees [13] or students [25], the effect size was d=0.96 (95% CI 0.70-1.21) or d=0.69 (95% CI 0.36-1.02), respectively. However, the CIs in this trial (95% CI 0.51-0.91) overlapped those of all previously conducted studies.

    When comparing the results with those from unguided iSMIs not necessarily conducted in the work settings, the effect sizes found in this trial were larger than those found in a recent meta-analysis on the topic (d=0.33, 95% CI 0.20-0.46) [70] also when taking 95% CIs into account. Potential reasons for the higher effect sizes compared to other interventions might lie in the strong theoretical basis and the possibility to tailor the intervention to individual need or interest. An alternative explanation might be the strong focus of the intervention on supporting participants to implement health behavior changes in daily life routine. Although the study was powered to test moderation hypotheses adequately, and although we included a broad range of theoretically potential effect modifiers, only few baseline variables significantly moderated the intervention effects. Baseline perceived stress was not associated with intervention outcome, indicating that GET.ON Stress can also be effective in heterogeneous samples that are not preselected based on high baseline symptom severity. That said, it needs to be noted that the sample showed a substantial level of baseline impairment, indicated by a mean of 22.65 (SD 5.63) on the PSS. Despite being lower than that in previous studies that have been conducted on GET.ON Stress (PSS, 25.26-23.90), and although 40.3% (160/396) of the participants in this trial would have been excluded in previous trials, the average baseline symptom severity was still approximately 1 SD above the average perceived stress level in a large working population (mean 15.3 [SD 6.2]) [71]. This might indicate that the intervention might especially be attractive for employees who already experience a substantial stress level. Future studies should hence test whether the utilization of universal preventive approaches in employees can be further increased, for example, by utilizing acceptance-facilitating interventions [72-75] that are designed to reduce the barriers of intervention utilization, such as low perceived risk. Another possibility might be that the format is not adequate for employees with low perceived burden and interventions with even a lower threshold, for example, mobile apps, focusing on less burdensome behavior changes are necessary to reach this target group.

    Resilience, psychological strain, agreeableness, and self-regulation moderated the treatment outcome. Our findings clearly show that the difference in the perceived stress between the IG and the WLC is dependent on the resilience score at baseline. This difference (in the perceived stress between the IG and the WLC) is greater in those that show a lower resilience at baseline, whereas the difference almost becomes nonexistent if the baseline resilience is very high. One potential explanation for this finding might be that very resilient participants manage to cope with perceived stress also in the control group without the help of an additional psychological intervention. Such an interpretation is in line with the theoretical framework around resilience assuming that resilience helps to deal with stressors [76,77]. Several systematic reviews show that high resilience might serve as a protective buffer against the development of psychopathological symptoms [78-82].

    Interestingly, employees with low self-regulation profited to a much greater extent from participating compared to those with high self-regulation. This is in contrast to our a priori expectation as we assumed that self-regulation competencies, the ability of an individual to regulate and control their thoughts and behavior enabling them to adapt to a broad range of demands [83], might be a necessary prerequisite for making effective use of such self-help approaches, which require the self-regulated implementation of behavior changes. However, one explanation for the finding might be that participating in the iSMI helps employees with low-self regulation to effectively counteract the missing necessary competencies required to effectively implement health behavior changes in daily life that are needed in order to reduce perceived stress, whereas employees with high self-regulation might manage to realize these necessary self-regulatory tasks also without an additional intervention. Such an assumption of a compensatory effect is supported by the data in the control group showing very high perceived stress level at follow-up in the control group for those with low self-regulation in comparison to low perceived stress levels in the control group for those with high self-regulation. In the IG, the difference between low and high self-regulation is, however, not reflected in the major differences in perceived stress. Nonetheless, empirical studies are needed to confirm these assumptions.

    The finding that the intervention was found to be ineffective in individuals with very low current psychological strain (<7.96) has relevant implications for the implementation of such approaches. If these results are confirmed in future studies, such interventions should not be used solely as a universal preventive intervention, for example, to increase protective factors that might help to cope with future stressors. Instead, such interventions should only be offered to employees experiencing at least a minimal level of psychological strain, as this might serve as an important source of motivation that is required for self-help approaches. However, our study was limited to a 6-months follow-up, and future studies are needed to investigate the potential protective effects in such populations regarding future stressors. This study also found agreeableness as a moderator, which has not emerged in the previous literature. Future research should investigate if this was a spurious finding or could lead to a new understanding of the interaction between personality traits and iSMIs.

    This study also has some limitations to be considered. First, as usual with RCTs, there was a screening process preceding trial participation, that is, informed consent, which might have caused some applicants with lower motivation to drop out before the study began, potentially resulting in a more homogenous sample. Moreover, the exclusion of suicidal individuals limits the generalizability to such samples. Therefore, although this was a pragmatic trial and eligibility criteria were kept to a minimum to mimic routine care, intervention effects might be overestimated compared to the intervention under routine conditions. Second, one needs to keep in mind that the evidence generated by this study is based on an RCT, which brings a rather high structuring of participants and a high research attention with it, which is usually not the case in routine occupational care. Since the securing of commitment represents an adherence-promoting element in self-help interventions, it can be assumed that the effect sizes for pure self-guided interventions found in RCTs are significantly overestimated for what can be expected in occupational routine care [84] when no additional measures to increase adherence are applied. Hence, a clear concept for ensuring adherence such as through minimal guidance from a professional seems favorable, especially when considering that guided internet-based interventions have found to be superior over pure self-help interventions, both for stress-management [70] as well as other areas [85-87]. Preference should be in routine care, whenever possible, given to self-help approaches with at least some form of adherence-promoting guidance. Moreover, although the study was powered for moderator analyses, we did not stratify for extreme values on assessed moderator variables. Hence, some expressions on the investigated moderators, for example, very low stress levels, were probably not included in sufficient numbers so that a moderator analysis could have detected an existing effect. Further, the power was not sufficient to detect small moderator effects that may nevertheless be of clinical relevance. Since this is one of the first trials on iSMIs as well as SMIs, in general, investigating moderator effects, future empirical studies are needed to confirm our findings.

    This study confirms GET.ON Stress to be effective in a heterogeneous sample of employees and to be applicable for a broad range of participant characteristics. Based on the available evidence, iSMIs should be implemented on a broad scale to reduce the adverse consequences of occupational stress.

    Acknowledgments

    The European Union funded this study (EU EFRE: ZW6-80119999, CCI 2007DE161PR001). BARMER supported the recruitment for this study.

    Authors' Contributions

    DDE and DL designed the study. DDE, DL, HR, and MB developed the intervention content. DDE and MF performed the outcome analyses. MF prepared the first draft of the manuscript, DDE supervised the writing process, and MF integrated the coauthor comments and edits. All the authors contributed to the further writing of the manuscript and approved the final manuscript.

    Conflicts of Interest

    DDE and HR are members of the board of the International Society for Research on Internet Interventions. DDE has served as a consultant on the scientific advisory boards of Sanofi, Novartis, Minddistrict, Lantern, Schoen Kliniken, Ideamed, and German health insurance companies (BARMER, Techniker Krankenkasse) and a number of federal chambers for psychotherapy. DDE, MB, BF, and DL are stakeholders in the Institute for health training online (formerly GET.ON, now HelloBetter), which aims to implement scientific findings related to digital health interventions into routine care. HelloBetter distributes the digital intervention under study.

    ‎
    Multimedia Appendix 1

    Supplementary data.

    DOCX File , 21 KB
    ‎
    Multimedia Appendix 2

    CONSORT-eHEALTH checklist (V 1.6.1).

    PDF File (Adobe PDF File), 660 KB
    1. Stress in America survey. American Psychological Association. 2019.   URL: https://www.apa.org/news/press/releases/stress/2019/stress-america-2019.pdf [accessed 2021-11-28]
    2. Thoits PA. Stress and health: major findings and policy implications. J Health Soc Behav 2010;51 Suppl:S41-S53. [CrossRef] [Medline]
    3. Wiegner L, Hange D, Björkelund C, Ahlborg G. Prevalence of perceived stress and associations to symptoms of exhaustion, depression and anxiety in a working age population seeking primary care--an observational study. BMC Fam Pract 2015 Mar 19;16:38 [FREE Full text] [CrossRef] [Medline]
    4. Richardson S, Shaffer JA, Falzon L, Krupka D, Davidson KW, Edmondson D. Meta-analysis of perceived stress and its association with incident coronary heart disease. Am J Cardiol 2012 Dec 15;110(12):1711-1716 [FREE Full text] [CrossRef] [Medline]
    5. Kivimäki M, Steptoe A. Effects of stress on the development and progression of cardiovascular disease. Nat Rev Cardiol 2018 Apr;15(4):215-229. [CrossRef] [Medline]
    6. Stansfeld S, Candy B. Psychosocial work environment and mental health--a meta-analytic review. Scand J Work Environ Health 2006 Dec;32(6):443-462 [FREE Full text] [CrossRef] [Medline]
    7. Theorell T, Hammarström A, Aronsson G, Träskman Bendz L, Grape T, Hogstedt C, et al. A systematic review including meta-analysis of work environment and depressive symptoms. BMC Public Health 2015 Aug 01;15:738 [FREE Full text] [CrossRef] [Medline]
    8. Åkerstedt T, Orsini N, Petersen H, Axelsson J, Lekander M, Kecklund G. Predicting sleep quality from stress and prior sleep--a study of day-to-day covariation across six weeks. Sleep Med 2012 Jun;13(6):674-679. [CrossRef] [Medline]
    9. Hassard J, Teoh K, Cox T, Cosmar M, Gründler R, Flemming D, et al. Calculating the cost of work-related stress and psychosocial risks: literature review. Publications Office of the European Union. 2014.   URL: https://op.europa.eu/en/publication-detail/-/publication/c8328fa1-519b-4f29-aa7b-fd80cffc18cb [accessed 2021-11-28]
    10. Bhui KS, Dinos S, Stansfeld SA, White PD. A synthesis of the evidence for managing stress at work: a review of the reviews reporting on anxiety, depression, and absenteeism. J Environ Public Health 2012;2012:515874 [FREE Full text] [CrossRef] [Medline]
    11. Richardson KM, Rothstein HR. Effects of occupational stress management intervention programs: a meta-analysis. J Occup Health Psychol 2008 Jan;13(1):69-93. [CrossRef] [Medline]
    12. Ebert DD, Cuijpers P, Muñoz RF, Baumeister H. Prevention of Mental Health Disorders Using Internet- and Mobile-Based Interventions: A Narrative Review and Recommendations for Future Research. Front Psychiatry 2017;8:116 [FREE Full text] [CrossRef] [Medline]
    13. Ebert DD, Heber E, Berking M, Riper H, Cuijpers P, Funk B, et al. Self-guided internet-based and mobile-based stress management for employees: results of a randomised controlled trial. Occup Environ Med 2016 May;73(5):315-323. [CrossRef] [Medline]
    14. Heber E, Ebert DD, Lehr D, Cuijpers P, Berking M, Nobis S, et al. The Benefit of Web- and Computer-Based Interventions for Stress: A Systematic Review and Meta-Analysis. J Med Internet Res 2017 Feb 17;19(2):e32 [FREE Full text] [CrossRef] [Medline]
    15. Pickett JT, Roche SP. Questionable, Objectionable or Criminal? Public Opinion on Data Fraud and Selective Reporting in Science. Sci Eng Ethics 2018 Feb;24(1):151-171. [CrossRef] [Medline]
    16. Camerer CF, Dreber A, Forsell E, Ho T, Huber J, Johannesson M, et al. Evaluating replicability of laboratory experiments in economics. Science 2016 Mar 25;351(6280):1433-1436. [CrossRef] [Medline]
    17. Camerer CF, Dreber A, Holzmeister F, Ho T, Huber J, Johannesson M, et al. Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nat Hum Behav 2018 Sep;2(9):637-644. [CrossRef] [Medline]
    18. Mede NG, Schäfer MS, Ziegler R, Weißkopf M. The "replication crisis" in the public eye: Germans' awareness and perceptions of the (ir)reproducibility of scientific research. Public Underst Sci 2021 Jan;30(1):91-102. [CrossRef] [Medline]
    19. Zarski A, Lehr D, Berking M, Riper H, Cuijpers P, Ebert DD. Adherence to Internet-Based Mobile-Supported Stress Management: A Pooled Analysis of Individual Participant Data From Three Randomized Controlled Trials. J Med Internet Res 2016 Jun 29;18(6):e146 [FREE Full text] [CrossRef] [Medline]
    20. Heber E, Ebert DD, Lehr D, Nobis S, Berking M, Riper H. Efficacy and cost-effectiveness of a web-based and mobile stress-management intervention for employees: design of a randomized controlled trial. BMC Public Health 2013 Jul 15;13:655 [FREE Full text] [CrossRef] [Medline]
    21. Heber E, Lehr D, Ebert DD, Berking M, Riper H. Web-Based and Mobile Stress Management Intervention for Employees: A Randomized Controlled Trial. J Med Internet Res 2016 Jan 27;18(1):e21 [FREE Full text] [CrossRef] [Medline]
    22. Ebert DD, Lehr D, Smit F, Zarski A, Riper H, Heber E, et al. Efficacy and cost-effectiveness of minimal guided and unguided internet-based mobile supported stress-management in employees with occupational stress: a three-armed randomised controlled trial. BMC Public Health 2014 Aug 07;14:807 [FREE Full text] [CrossRef] [Medline]
    23. Ebert DD, Lehr D, Heber E, Riper H, Cuijpers P, Berking M. Internet- and mobile-based stress management for employees with adherence-focused guidance: efficacy and mechanism of change. Scand J Work Environ Health 2016 Sep 01;42(5):382-394 [FREE Full text] [CrossRef] [Medline]
    24. Ebert DD, Kählke F, Buntrock C, Berking M, Smit F, Heber E, et al. A health economic outcome evaluation of an internet-based mobile-supported stress management intervention for employees. Scand J Work Environ Health 2018 Mar 01;44(2):171-182 [FREE Full text] [CrossRef] [Medline]
    25. Harrer M, Adam SH, Fleischmann RJ, Baumeister H, Auerbach R, Bruffaerts R, et al. Effectiveness of an Internet- and App-Based Intervention for College Students With Elevated Stress: Randomized Controlled Trial. J Med Internet Res 2018 Apr 23;20(4):e136 [FREE Full text] [CrossRef] [Medline]
    26. Cohen S, Kamarck T, Mermelstein R. A Global Measure of Perceived Stress. Journal of Health and Social Behavior 1983 Dec;24(4):385. [CrossRef]
    27. Reis D, Lehr D, Heber E, Ebert DD. The German Version of the Perceived Stress Scale (PSS-10): Evaluation of Dimensionality, Validity, and Measurement Invariance With Exploratory and Confirmatory Bifactor Modeling. Assessment 2019 Oct;26(7):1246-1259. [CrossRef] [Medline]
    28. Patsopoulos NA. A pragmatic view on pragmatic trials. Dialogues Clin Neurosci 2011;13(2):217-224 [FREE Full text] [Medline]
    29. Alli BO. Fundamental Principles of Occupational Health and Safety. Second Edition. Geneva: International Labor Office; 2008.
    30. Lynch FL, Hornbrook M, Clarke GN, Perrin N, Polen MR, O'Connor E, et al. Cost-effectiveness of an intervention to prevent depression in at-risk teens. Arch Gen Psychiatry 2005 Nov;62(11):1241-1248. [CrossRef] [Medline]
    31. McLaughlin KA. The public health impact of major depression: a call for interdisciplinary prevention efforts. Prev Sci 2011 Dec;12(4):361-371 [FREE Full text] [CrossRef] [Medline]
    32. Quintiliani L, Sattelmair J, Sorensen G. The workplace as a setting for interventions to improve diet and promote physical activity. 2008 Presented at: WHO/WEF Joint Event on Preventing Noncommunicable Diseases in the Workplace; September 2007; Dalian (China)   URL: https://www.who.int/dietphysicalactivity/Quintiliani-workplace-as-setting.pdf?ua=1
    33. Ebert DD, Donkin L, Andersson G, Andrews G, Berger T, Carlbring P, et al. Does Internet-based guided-self-help for depression cause harm? An individual participant data meta-analysis on deterioration rates and its moderators in randomized controlled trials. Psychol Med 2016 Oct;46(13):2679-2693 [FREE Full text] [CrossRef] [Medline]
    34. Junge MN, Lehr D, Bockting CL, Berking M, Riper H, Cuijpers P, et al. For whom are internet-based occupational mental health interventions effective? Moderators of internet-based problem-solving training outcome. Internet Interventions 2015 Mar;2(1):39-47. [CrossRef]
    35. Karyotaki E, Riper H, Twisk J, Hoogendoorn A, Kleiboer A, Mira A, et al. Efficacy of Self-guided Internet-Based Cognitive Behavioral Therapy in the Treatment of Depressive Symptoms: A Meta-analysis of Individual Participant Data. JAMA Psychiatry 2017 Apr 01;74(4):351-359. [CrossRef] [Medline]
    36. Coudray C, Palmer R, Frazier P. Moderators of the efficacy of a web-based stress management intervention for college students. J Couns Psychol 2019 Nov;66(6):747-754. [CrossRef] [Medline]
    37. Karyotaki E, Efthimiou O, Miguel C, Bermpohl FMG, Furukawa TA, Cuijpers PIM, Individual Patient Data Meta-Analyses for Depression (IPDMA-DE) Collaboration, et al. Internet-Based Cognitive Behavioral Therapy for Depression: A Systematic Review and Individual Patient Data Network Meta-analysis. JAMA Psychiatry 2021 Apr 01;78(4):361-371. [CrossRef] [Medline]
    38. Riper H, Hoogendoorn A, Cuijpers P, Karyotaki E, Boumparis N, Mira A, et al. Effectiveness and treatment moderators of internet interventions for adult problem drinking: An individual patient data meta-analysis of 19 randomised controlled trials. PLoS Med 2018 Dec;15(12):e1002714 [FREE Full text] [CrossRef] [Medline]
    39. Karyotaki E, Ebert DD, Donkin L, Riper H, Twisk J, Burger S, et al. Do guided internet-based interventions result in clinically relevant changes for patients with depression? An individual participant data meta-analysis. Clin Psychol Rev 2018 Jul;63:80-92. [CrossRef] [Medline]
    40. Brookes ST, Whitely E, Egger M, Smith GD, Mulheran PA, Peters TJ. Subgroup analyses in randomized trials: risks of subgroup-specific analyses; power and sample size for the interaction test. J Clin Epidemiol 2004 Mar;57(3):229-236. [CrossRef] [Medline]
    41. Kraemer HC, Wilson GT, Fairburn CG, Agras WS. Mediators and moderators of treatment effects in randomized clinical trials. Arch Gen Psychiatry 2002 Oct;59(10):877-883. [CrossRef] [Medline]
    42. Beck AT, Steer RA, Brown GK. BDI-II: Beck Depression Inventory Manual. Second edition. San Antonio, TX: Psychological Corporation; 1996.
    43. Lazarus RS, Folkman S. Stress, Appraisal, and Coping. New York: Springer Publications; 1984.
    44. D'Zurilla TJ, Nezu AM. Problem-solving therapy. In: Handbook of Cognitive-Behavioral Therapies, Third Edition. New York, USA: Guilford Press; 2010:197-225.
    45. Berking M. Training Emotionaler Kompetenzen. Berlin, Heidelberg: Springer; 2015.
    46. Berking M, Ebert D, Cuijpers P, Hofmann SG. Emotion regulation skills training enhances the efficacy of inpatient cognitive behavioral therapy for major depressive disorder: a randomized controlled trial. Psychother Psychosom 2013;82(4):234-245. [CrossRef] [Medline]
    47. Meyer TD, Hautzinger M. Allgemeine Depressions-Skala (ADS). Diagnostica 2001 Oct;47(4):208-215. [CrossRef]
    48. Radloff LS. The CES-D Scale. Applied Psychological Measurement 2016 Jul 26;1(3):385-401. [CrossRef]
    49. Maxwell A, Özmen M, Iezzi A, Richardson J. Deriving population norms for the AQoL-6D and AQoL-8D multi-attribute utility instruments from web-based data. Qual Life Res 2016 Dec;25(12):3209-3219. [CrossRef] [Medline]
    50. Connor KM, Davidson JR. Development of a new resilience scale: the Connor-Davidson Resilience Scale (CD-RISC). Depress Anxiety 2003;18(2):76-82. [CrossRef] [Medline]
    51. Schaufeli W, Leiter M, Maslach C, Jackson S. Maslach Burnout Inventory-general survey (MBI-GS). American Psychological Association 1996:1-5. [CrossRef]
    52. Tennant R, Hiller L, Fishwick R, Platt S, Joseph S, Weich S, et al. The Warwick-Edinburgh Mental Well-being Scale (WEMWBS): development and UK validation. Health Qual Life Outcomes 2007 Nov 27;5:63 [FREE Full text] [CrossRef] [Medline]
    53. Bech P, Olsen LR, Kjoller M, Rasmussen NK. Measuring well-being rather than the absence of distress symptoms: a comparison of the SF-36 Mental Health subscale and the WHO-Five Well-Being Scale. Int J Methods Psychiatr Res 2003;12(2):85-91 [FREE Full text] [CrossRef] [Medline]
    54. Li J, Loerbroks A, Jarczok MN, Schöllgen I, Bosch JA, Mauss D, et al. Psychometric properties and differential explanation of a short measure of effort-reward imbalance at work: a study of industrial workers in Germany. Am J Ind Med 2012 Sep;55(9):808-815. [CrossRef] [Medline]
    55. Rigotti T, Schyns B, Mohr G. A Short Version of the Occupational Self-Efficacy Scale: Structural and Construct Validity Across Five Countries. Journal of Career Assessment 2008 May 01;16(2):238-255. [CrossRef]
    56. Schaufeli W, Bakker A. UWES-Utrecht Work Engagement Scale Test Manual. Utrecht University: Department of Psychology; 2003.   URL: https://www.wilmarschaufeli.nl/publications/Schaufeli/Test%20Manuals/Test_manual_UWES_English.pdf [accessed 2021-11-29]
    57. Walker TJ, Tullar JM, Diamond PM, Kohl HW, Amick BC. Validity and Reliability of the 8-Item Work Limitations Questionnaire. J Occup Rehabil 2017 Dec;27(4):576-583 [FREE Full text] [CrossRef] [Medline]
    58. Bouwmans C, De Jong K, Timman R, Zijlstra-Vlasveld M, Van der Feltz-Cornelis C, Tan Swan S, et al. Feasibility, reliability and validity of a questionnaire on healthcare consumption and productivity loss in patients with a psychiatric disorder (TiC-P). BMC Health Serv Res 2013 Jun 15;13:217 [FREE Full text] [CrossRef] [Medline]
    59. Rammstedt B, John OP. Measuring personality in one minute or less: A 10-item short version of the Big Five Inventory in English and German. Journal of Research in Personality 2007 Feb;41(1):203-212. [CrossRef]
    60. Schulz H, Lang K, Nübling R, Koch U. Psychometrische Überprüfung einer Kurzform des Fragebogens zur Psychotherapiemotivation - FPTM-23. Diagnostica 2003 Apr;49(2):83-93. [CrossRef]
    61. Schwarzer R, Diehl M, Schmitz GS. Self-Regulation Scale. 1999.   URL: http://userpage.fu-berlin.de/health/selfreg_e.htm [accessed 2021-11-28]
    62. Schwarzer R, Jerusalem M. Generalized self-efficacy scale. American Psychological Association. Windsor, UK: Nfer-Nelson; 1995.   URL: https://doi.apa.org/doiLanding?doi=10.1037%2Ft00393-000 [accessed 2021-11-28]
    63. Heber E, Lehr D, Ebert DD, Berking M, Riper H. Web-Based and Mobile Stress Management Intervention for Employees: A Randomized Controlled Trial. J Med Internet Res 2016 Jan 27;18(1):e21 [FREE Full text] [CrossRef] [Medline]
    64. Leon AC, Heo M. Sample Sizes Required to Detect Interactions between Two Binary Fixed-Effects in a Mixed-Effects Linear Regression Model. Comput Stat Data Anal 2009 Jan 15;53(3):603-608 [FREE Full text] [CrossRef] [Medline]
    65. Moher D, Hopewell S, Schulz KF, Montori V, Gøtzsche PC, Devereaux PJ, et al. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ 2010 Mar 23;340:c869 [FREE Full text] [CrossRef] [Medline]
    66. Schafer JL, Graham JW. Missing data: Our view of the state of the art. Psychological Methods 2002;7(2):147-177. [CrossRef]
    67. Jacobson NS, Truax P. Clinical significance: a statistical approach to defining meaningful change in psychotherapy research. J Consult Clin Psychol 1991 Feb;59(1):12-19. [CrossRef] [Medline]
    68. Hayes AF. Introduction to Mediation, Moderation and Conditional Process Analysis: A Regression-Based Approach. New York, USA: The Guilford Press; 2013.
    69. Johnson PO, Fay LC. The Johnson-Neyman technique, its theory and application. Psychometrika 1950 Dec;15(4):349-367. [CrossRef]
    70. Heber E, Ebert DD, Lehr D, Cuijpers P, Berking M, Nobis S, et al. The Benefit of Web- and Computer-Based Interventions for Stress: A Systematic Review and Meta-Analysis. J Med Internet Res 2017 Feb 17;19(2):e32 [FREE Full text] [CrossRef] [Medline]
    71. Lesage F, Berjot S, Deschamps F. Psychometric properties of the French versions of the Perceived Stress Scale. Int J Occup Med Environ Health 2012 Jun;25(2):178-184 [FREE Full text] [CrossRef] [Medline]
    72. Baumeister H, Nowoczin L, Lin J, Seifferth H, Seufert J, Laubner K, et al. Impact of an acceptance facilitating intervention on diabetes patients' acceptance of Internet-based interventions for depression: a randomized controlled trial. Diabetes Res Clin Pract 2014 Jul;105(1):30-39. [CrossRef] [Medline]
    73. Baumeister H, Seifferth H, Lin J, Nowoczin L, Lüking M, Ebert D. Impact of an Acceptance Facilitating Intervention on Patients’ Acceptance of Internet-based Pain Interventions. The Clinical Journal of Pain 2015;31(6):528-535. [CrossRef]
    74. Ebert DD, Berking M, Cuijpers P, Lehr D, Pörtner M, Baumeister H. Increasing the acceptance of internet-based mental health interventions in primary care patients with depressive symptoms. A randomized controlled trial. J Affect Disord 2015 May 01;176:9-17. [CrossRef] [Medline]
    75. Lin J, Faust B, Ebert DD, Krämer L, Baumeister H. A Web-Based Acceptance-Facilitating Intervention for Identifying Patients' Acceptance, Uptake, and Adherence of Internet- and Mobile-Based Pain Interventions: Randomized Controlled Trial. J Med Internet Res 2018 Aug 21;20(8):e244 [FREE Full text] [CrossRef]
    76. Pęciłło M. The concept of resilience in OSH management: a review of approaches. Int J Occup Saf Ergon 2016;22(2):291-300 [FREE Full text] [CrossRef] [Medline]
    77. Johnson SJ, Willis SM, Evans J. An examination of stressors, strain, and resilience in academic and non-academic U.K. university job roles. International Journal of Stress Management 2019 May;26(2):162-172. [CrossRef]
    78. Davies EB, Morriss R, Glazebrook C. Computer-delivered and web-based interventions to improve depression, anxiety, and psychological well-being of university students: a systematic review and meta-analysis. J Med Internet Res 2014 May 16;16(5):e130 [FREE Full text] [CrossRef] [Medline]
    79. Leppin AL, Bora PR, Tilburt JC, Gionfriddo MR, Zeballos-Palacios C, Dulohery MM, et al. The efficacy of resiliency training programs: a systematic review and meta-analysis of randomized trials. PLoS One 2014;9(10):e111420 [FREE Full text] [CrossRef] [Medline]
    80. Dray J, Bowman J, Campbell E, Freund M, Wolfenden L, Hodder RK, et al. Systematic Review of Universal Resilience-Focused Interventions Targeting Child and Adolescent Mental Health in the School Setting. J Am Acad Child Adolesc Psychiatry 2017 Oct;56(10):813-824 [FREE Full text] [CrossRef] [Medline]
    81. Wermelinger Ávila MP, Lucchetti ALG, Lucchetti G. Association between depression and resilience in older adults: a systematic review and meta-analysis. Int J Geriatr Psychiatry 2017 Mar;32(3):237-246. [CrossRef] [Medline]
    82. Brassington K, Lomas T. Can resilience training improve well-being for people in high-risk occupations? A systematic review through a multidimensional lens. The Journal of Positive Psychology 2020 Apr 27;16(5):573-592. [CrossRef]
    83. Baumeister RF, Vohs KD. Self-Regulation, Ego Depletion, and Motivation. Social Pers Psych Compass 2007 Nov;1(1):115-128. [CrossRef]
    84. Ebert DD, Baumeister H. Internet-Based Self-help Interventions for Depression in Routine Care. JAMA Psychiatry 2017 Aug 01;74(8):852-853. [CrossRef] [Medline]
    85. Baumeister H, Reichler L, Munzinger M, Lin J. The impact of guidance on Internet-based mental health interventions — A systematic review. Internet Interventions 2014 Oct;1(4):205-215. [CrossRef]
    86. Ciuca AM, Berger T, Crişan LG, Miclea M. Internet-based treatment for panic disorder: A three-arm randomized controlled trial comparing guided (via real-time video sessions) with unguided self-help treatment and a waitlist control. PAXPD study results. J Anxiety Disord 2018 May;56:43-55. [CrossRef] [Medline]
    87. Lin J, Paganini S, Sander L, Lüking M, Ebert DD, Buhrman M, et al. An Internet-Based Intervention for Chronic Pain. Dtsch Arztebl Int 2017 Oct 13;114(41):681-688 [FREE Full text] [CrossRef] [Medline]

    ‎
    ANCOVA: analysis of covariance
    IG: intervention group
    iSMI: internet-based mobile-supported stress management intervention
    PSS: perceived stress scale
    RCT: randomized controlled trial
    WLC: waiting list control

    Edited by G Eysenbach; submitted 03.07.20; peer-reviewed by L Genaro, M Adly, A Adly, A Adly; comments to author 24.08.20; revised version received 15.12.20; accepted 17.05.21; published 22.12.21

    Copyright

    ©David Daniel Ebert, Marvin Franke, Anna-Carlotta Zarski, Matthias Berking, Heleen Riper, Pim Cuijpers, Burkhardt Funk, Dirk Lehr. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 22.12.2021.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.