This paper is in the following e-collection/theme issue:

Web-based and Mobile Health Interventions (3092) Mobile Health (mhealth) (2748) e-Mental Health and Cyberpsychology (1386) Medicine 2.0'14 Maui/Malaga (Full Paper of Conference Presentation) (20) Medicine 2.0'14 (Maui/Malaga) (18) Anxiety and Stress Disorders (999) Engagement with and Adherence to Digital Health Interventions, Law of Attrition (457)

Published on in Vol 18, No 6 (2016): Jun

Adherence to Internet-Based Mobile-Supported Stress Management: A Pooled Analysis of Individual Participant Data From Three Randomized Controlled Trials

Adherence to Internet-Based Mobile-Supported Stress Management: A Pooled Analysis of Individual Participant Data From Three Randomized Controlled Trials

Adherence to Internet-Based Mobile-Supported Stress Management: A Pooled Analysis of Individual Participant Data From Three Randomized Controlled Trials

Authors of this article:

Anna-Carlotta Zarski1, 2 Author Orcid Image ;   Dirk Lehr2, 3 Author Orcid Image ;   Matthias Berking1, 2 Author Orcid Image ;   Heleen Riper2, 4, 5 Author Orcid Image ;   Pim Cuijpers2, 4 Author Orcid Image ;   David Daniel Ebert1, 2 Author Orcid Image
Article Authors Cited by (90) Tweetations (10) Metrics

    Original Paper

    1Department of Clinical Psychology and Psychotherapy, Friedrich-Alexander University Erlangen-Nuremberg, Erlangen, Germany

    2Innovation Incubator, Division of Online Health Training, Leuphana University Lueneburg, Luenburg, Germany

    3Institute for Psychology, Department of Health Psychology and Applied Biological Psychology, Leuphana University Luenburg, Lueneburg, Germany

    4Department of Clinical Psychology, VU University Amsterdam, Amsterdam, Netherlands

    5Telepsychiatric Centre, University of Southern Denmark, Odense, Denmark

    Corresponding Author:

    Anna-Carlotta Zarski, MSc

    Department of Clinical Psychology and Psychotherapy

    Friedrich-Alexander University Erlangen-Nuremberg

    Nägelsbachstr. 25a

    Erlangen, 91052

    Germany

    Phone: 49 9131 85 67570

    Fax:49 4131 677 171

    Email: Anna-Carlotta.Zarski@fau.de


    Background: Nonadherence to treatment is a prevalent issue in Internet interventions. Guidance from health care professionals has been found to increase treatment adherence rates in Internet interventions for a range of physical and mental disorders. Evaluating different guidance formats of varying intensity is important, particularly with respect to improvement of effectiveness and cost-effectiveness. Identifying predictors of nonadherence allows for the opportunity to better adapt Internet interventions to the needs of participants especially at risk for discontinuing treatment.

    Objective: The goal of this study was to investigate the influence of different guidance formats (content-focused guidance, adherence-focused guidance, and administrative guidance) on adherence and to identify predictors of nonadherence in an Internet-based mobile-supported stress management intervention (ie, GET.ON Stress) for employees.

    Methods: The data from the groups who received the intervention were pooled from three randomized controlled trials (RCTs) that evaluated the efficacy of the same Internet-based mobile-supported stress management intervention (N=395). The RCTs only differed in terms of the guidance format (content-focused guidance vs waitlist control, adherence-focused guidance vs waitlist control, administrative guidance vs waitlist control). Adherence was defined by the number of completed treatment modules (0-7). An ANOVA was performed to compare the adherence rates from the different guidance formats. Multiple hierarchical linear regression analysis was conducted to evaluate predictors of nonadherence, which included gender, age, education, symptom-related factors, and hope for improvement.

    Results: In all, 70.5% (93/132) of the content-focused guidance sample, 68.9% (91/132) of the adherence-focused guidance sample, and 42.0% (55/131) of the participants in the administrative guidance sample completed all treatment modules. Guidance had a significant effect on treatment adherence (F2,392=11.64, P<.001; ω2=.05). Participants in the content-focused guidance (mean 5.70, SD 2.32) and adherence-focused guidance samples (mean 5.58, SD 2.33) completed significantly more modules than participants in the administrative guidance sample (mean 4.36, SD 2.78; t223=4.53, P<.001; r=.29). Content-focused guidance was not significantly associated with higher adherence compared to adherence-focused guidance (t262=0.42, P=.67; r=.03). The effect size of r=.03 (95% CI –0.09 to 0.15) did not pass the equivalence margin of r=.20 and the upper bound of the 95% CI lay below the predefined margin, indicating equivalence between adherence-focused guidance and content-focused guidance. Beyond the influence of guidance, none of the predictors significantly predicted nonadherence.

    Conclusions: Guidance has been shown to be an influential factor in promoting adherence to an Internet-based mobile-supported stress management intervention. Adherence-focused guidance, which included email reminders and feedback on demand, was equivalent to content-focused guidance with regular feedback while requiring only approximately a quarter of the coaching resources. This could be a promising discovery in terms of cost-effectiveness. However, even after considering guidance, sociodemographic, and symptom-related characteristics, most interindividual differences in nonadherence remain unexplained.

    Clinical Trial: DRKS00004749; http://drks-neu.uniklinik-freiburg.de/drks_web/navigate.do?navigationId=trial.HTML&TRIAL _ID=DRKS00004749 (Archived by WebCite at http://www.webcitation.org/6QiDk9Zn8); DRKS00005112; http://drks-neu.uniklinik-freiburg. de/drks_web/navigate.do?navigationId=trial.HTML&TRIAL_ID=DRKS00005112 (Archived by WebCite at http://www.webcitation.org/6QiDysvev); DRKS00005384; http://drks-neu.uniklinik-freiburg.de/ drks_web/navigate.do?navigationId=trial.HTML&TRIAL_ID=DRKS00005384 (Archived by WebCite at http://www.webcitation.org/6QiE0xcpE)

    J Med Internet Res 2016;18(6):e146

    doi:10.2196/jmir.4493

    Keywords

    guidance; treatment adherence; predictors; Internet intervention; work-related stress; stress management 


    Many participants of Internet-based programs do not begin the interventions after registration or are nonadherent and quit the intervention prematurely against recommendations in both study and routine care settings [1-6]. Treatment adherence can be defined as the extent to which individuals experience the intervention content [7]. Low treatment adherence is a major concern because it has been associated with reduced efficacy of Internet interventions for physical and mental disorders and health promotion programs [8-13].

    Occupational stress is associated with an increased risk for common mental disorders in the long term [14]. Internet-based stress management interventions for employees show promising effects in reducing stress and related health problems such as depression [15-19]. Nevertheless, low treatment adherence is a serious problem that adversely impacts the efficacy of Internet-based antistress programs [20-22].

    In order to improve the clinical effects of Internet interventions, it is important to identify and evaluate factors associated with adherence [23]. Treatment factors, in particular human guidance, may be especially important [24-26]. Guided treatments have been shown to result in higher adherence rates (56% to 81%) than unguided treatments (26% to 69%) [27-31]. Treatment adherence rates of guided interventions have often been found to be comparable to those of face-to-face interventions [32,33]. However, guidance from a health care professional is expensive and a limited resource. Thus, it is crucial to investigate the effects of different guidance formats to identify the level and type of guidance necessary to achieve acceptable treatment adherence.

    Guidance in Internet interventions is often classified according to the amount of required coaching or therapist time [34]. However, guidance also differs regarding objectives and content. Accordingly, at least four different guidance formats can be distinguished: (1) unguided interventions, completely self-administered by the user; (2) administrative guidance, providing technical support in case of computer and Internet platform-related problems and dispensing relevant information throughout the course of the treatment; (3) adherence-focused guidance, with adherence monitoring including reminders by email or telephone; and (4) content-focused guidance, personalized written feedback by coaches for completed treatment modules and adherence monitoring.

    Content-focused guidance in Internet interventions has been found to be associated with higher levels of treatment completion (72%) compared to administrative guidance (62%) or unguided interventions (26%) in a meta-analysis [31]. Adherence-focused guidance also seems promising in fostering treatment completion with the added benefit of keeping coaching demands—in terms of time spent per participant—to a minimum [26,35-39].

    However, whether adherence-focused guidance results in comparable adherence rates to content-focused guidance formats remains unclear. Comparisons of adherence-focused guidance to content-focused guidance are currently limited regarding treatment adherence in Internet interventions.

    Mohr and colleagues [25] introduced the notion of “supportive accountability” to explain the relationship between human guidance and treatment adherence in Internet interventions. This model assumes that human support in Internet interventions enhances adherence by allowing the patient to foster a commitment toward an eCoach, who is perceived as being trustworthy, benevolent, and professionally knowledgeable. The eCoach is responsible for accompanying the participants through the program, showing interest in their training processes, and offering support. In this respect, adherence monitoring in addition to reminders in case of nonadherence must be embedded in a benevolent context, with the aim of supporting adherence instead of surveillance of module completion. The expectations placed on the participant by the therapy program, to be sustained on a regular basis throughout the treatment, should be transparent and reasonable. Hence, the participant knows what is to be expected and can be involved in determining these expectations. Both in adherence- and content-focused guidance, the eCoach communicates the requirements for successful participation in the intervention and the associated expectations placed on the participants. By contrast, in unguided or administrative-guided interventions, participants only receive general recommendations on program use.

    Based on the supportive accountability model [25], we assumed that the introduction of an eCoach who offers support in program completion is important for creating an adherence-promoting relationship with the participant. For this reason, we developed the concept of adherence-focused guidance, which is comprised of not only adherence monitoring, but also the opportunity to receive content feedback on demand [40]. At the beginning of the training, the eCoach invites the participant to contact them in case of content-related questions and any desired feedback for completed treatment modules. Feedback on demand is expected to emphasize the supporting role of the eCoach in the training process and thereby promote the participant’s adherence. Assuming that accountability is the essential factor in coaching that keeps participants involved in training, we hypothesized that both adherence-focused guidance and content-focused guidance are superior to administrative guidance with regard to adherence rates. Adherence-focused guidance was expected to be equivalent to content-focused guidance for adherence rates.

    Apart from treatment factors, knowledge of user characteristics related to nonadherence helps to identify individuals who are at risk of discontinuing treatment and might need additional support. According to the behavior change model for Internet interventions by Ritterband and colleagues [41], user characteristics can be classified into disease-related factors, demographics, traits, cognitive factors, beliefs and attitudes, physiological factors, and skills.

    Demographic and disease-specific user characteristics were of particular interest in past adherence research in the field of Internet interventions. Low education level [1,42-44], male gender [1,30,42,45], or being unmarried or single [43,46-48] have been found to be associated with lower treatment adherence across different mental and physical health interventions. Younger age has also been shown to be related to nonadherence in the majority of studies [37,42,45,49]. Only a small number of studies found older individuals to be at a higher risk for nonadherence [1,2,7]. However, some studies found no significant association between any sociodemographic variables and treatment adherence [7,50,51].

    High symptom severity at baseline is also frequently linked to lower treatment adherence [7,47,52,53]. But, the relationship between baseline depressive symptoms and treatment adherence seems inconsistent. Both higher [7,54,55] and lower [1,3,42] baseline depression scores were found to be associated with lower treatment adherence or no significant association was found [30,50,56,57]. Lower treatment expectations have been found to be related to nonadherence in Internet interventions [56,58,59].

    Only a few studies to date have investigated predictors of adherence in Internet intervention and they showed conflicting results [60]. For this reason, we used an exploratory approach including potential predictors based on (1) results of previous studies investigating adherence predictors in Internet interventions [1,7,42-44] and (2) theoretical assumptions due to intervention characteristics. The final list of potential predictors investigated in the present study included: (1) demographics: gender, education level, and age; (2) symptom-related factors: stress, depression, and emotional exhaustion; and (3) variables concerning beliefs and attitudes: hope for improvement.

    To the best of our knowledge, no research to date has explored the influence of different guidance formats and user characteristics on treatment adherence to an Internet intervention in stressed employees.

    The current study aimed to (1) report adherence rates from a newly developed Internet-based mobile-supported stress management intervention (ie, GET.ON Stress) and (2) investigate the role of different guidance formats (content-focused guidance, adherence-focused guidance, and administrative guidance) on adherence. Further goals of the study were to (3) identify user characteristics predictive of treatment nonadherence over and above the guidance formats and (4) analyze differential predictor effects as a function of guidance formats.

    We hypothesized that (1) treatment adherence rates would be greater for content-focused guidance and adherence-focused guidance compared to administrative guidance, and (2) adherence-focused guidance is equivalent to content-focused guidance in terms of adherence rates. User characteristics that contribute to treatment nonadherence apart from guidance formats and differential effects of predictors as a function of guidance formats were analyzed exploratively. In this study, treatment adherence is operationalized by the number of completed treatment modules.


    Data for this analysis were drawn from three randomized controlled trials (RCTs) evaluating the same Internet-based mobile-supported stress management intervention (GET.ON Stress [61-64]) under varying guidance conditions (study 1: content-focused guidance vs waitlist control; study 2: adherence-focused guidance vs waitlist control; study 3: administrative guidance vs waitlist control). Details of the study design for study 1 have been described in a published study protocol [61]. All three trials employed the same design and procedures apart from the guidance format, allowing for the pooling of the data from the three studies.

    Sample

    The analyses in this study were based solely on the intervention group samples of the N=395 participants who received the same Internet-based mobile-supported stress management intervention (study 1: n=132; study 2: n=132; study 3: n=131). Participants in the waitlist control condition were not included in the analyses because they did not receive access to the training until 6 months after randomization. All three studies included (1) currently employed workers, (2) older than age 18 years, (3) with scores ≥22 on the Perceived Stress Scale (PSS-10) [65], (4) who self-reported having Internet access, (5) sufficient skills in reading and writing German, and (6) who were willing to give informed consent. Participants were excluded when (1) they self-reported having been diagnosed with psychoses or dissociative symptoms in the past or (2) showed a notable suicidal risk as indicated by a score of greater than 1 on item 9 (“I feel I would be better off dead”) on the Beck Depression Inventory (BDI) [66].

    Intervention

    The Internet-based mobile-supported stress management intervention GET.ON Stress is based on two main components: problem solving [16,67-69] and emotion regulation [70,71]. A detailed description can be found elsewhere [61]. The intervention consists of seven modules composed of psychoeducation (module 1), problem solving (modules 2-3), emotion regulation (modules 4-6), planning for the future (module 7), and an optional booster session. Additionally, participants are offered eight units that are integrated in modules 2 to 6 that can be opted for based on individual needs or preference. These units are directed at time management, rumination and worrying, psychological detachment from work, sleep hygiene, rhythm and regularity of sleeping habits, nutrition and exercise, organization of breaks during work, and social support. Each module can be completed in approximately 45 to 60 minutes. Participants were advised to do at least one and a maximum of two modules per week. Consequently, the intervention took approximately 4 to 7 weeks (not including the booster session offered 4 weeks after completion of the intervention). Lessons were in the format of text, exercises, and testimonials, and included interactive elements such as audio and video clips. Participants were encouraged to keep a daily online stress diary. One strong focus of the intervention lay in transfer tasks (homework assignments) to integrate the newly acquired strategies and techniques into daily life. The training was adaptive because the content is tailored to the specific needs of the individual participants by continuously asking them to choose among various response options. Using responsive Web design, participants could use the program on a computer, tablet, or mobile phone. An integrated read-aloud function allowed participants to follow narrated lessons. If desired, participants could receive automatic motivational text messages and small exercises on their mobile phones. These messages had the purpose of supporting the participant in transferring the exercises of the training into their daily lives (eg, short relaxation exercises). The participants had the opportunity to choose between “light coach” (one text message every other day) or “intensive coach” (two to three text messages every day) options.

    Content-Focused Guided Internet-Based Mobile- Supported Stress Management Intervention

    Participants in the content-focused guidance condition received personalized written feedback from an eCoach on the exercises they had completed in each module within 48 hours. The eCoaches were psychologists and trained Master’s-level psychology students who followed guidelines about the feedback process that were defined according to the standardized manual for the intervention. The eCoaches were advised to not spend more than 30 minutes on feedback on a given completed module. The eCoaches sent reminders when the participants did not complete a module within 7 days. In total, the eCoaches sent 365 reminders, corresponding to a mean 2.77 reminders per participant (range 0-11, SD 2.41). The time required for coaching totaled up to 4 hours per participant.

    Adherence-Focused Guided Internet-Based Mobile- Supported Stress Management Intervention

    Participants of the adherence-focused guidance condition were also supported by an eCoach. The guidance manual was based on our developed adherence-focused guidance concept as outlined in the Introduction [40]. The eCoaches were trained psychologists who followed guidelines about the feedback process that were defined according to the standardized manual for the intervention. Adherence-focused guidance consisted of adherence monitoring and feedback on demand. Adherence monitoring included regularly checking module completion and sending reminders in case the participant did not complete at least one module within 7 days. In total, the eCoaches sent 463 reminders, corresponding to a mean 3.51 reminders per participant (range 0-13, SD 1.98). Feedback on demand included offering participants the opportunity to contact the eCoach and receive individual support or feedback within 24 hours. In contrast to the content-focused guidance concept, eCoach guidance only took place at the initiative of the participants to minimize the costs. There were only requests for a mean 8 content feedbacks by all participants (range 0-5, SD 0.46), corresponding to 0.06 feedback demands per participant. Thus, most of the time spent per participant was related to checking adherence to the intervention and providing reminders in case of nonadherence. The time required for coaching, including all reminders and feedback by request, added up to 1 hour per participant.

    Administrative-Guided Internet-Based Mobile- Supported Stress Management Intervention

    Participants in the administrative guidance condition were provided with contact information for the study administration team during the study period, which addressed such things as the completion of questionnaires, but they were not supported by an eCoach. They were provided with an email address to use in case of any technical problems.

    Measures

    Adherence Measures

    The number of completed treatment modules in the Internet-based mobile-supported stress management intervention, which ranged from 0 to 7, was the primary outcome measure in this study and was assessed by the system that provided the intervention. Module completion was defined by completion of the last page of a module. To arrive on the last page, participants were required to complete all the previous writing tasks. A module completion score of 0 could either mean that the participant did not start the intervention or did not finish the first module. Each module took approximately 45 to 60 minutes for completion.

    Predictor Measures

    The following variables, assessed at baseline before the start of the program, were evaluated as potential predictors of nonadherence: sociodemographic factors (gender [male/female], age [years], level of education [low, middle, high]), symptom severity factors (perceived stress, depressive symptoms, emotional exhaustion), and hope for improvement (confidence in treatment efficacy).

    Perceived stress at baseline relating to the past week was examined with the German version of the 10-item Perceived Stress Scale (PSS-10) [65,72]. This self-report instrument uses a 5-point Likert-type scale that ranges from 0=“never” to 4=“very often.” A Cronbach alpha of .77 indicated acceptable internal consistency of the PSS-10 in this study.

    Baseline depression symptom severity was measured with the German version of the Center for Epidemiological Studies Depression Scale (CES-D) [73,74]. This frequently used self-report instrument consists of 15 items that are answered on a 4-point Likert-type scale and refer to the previous week. Total scores range from 0 to 60. In this study, internal consistency was good (Cronbach alpha=.88).

    To measure emotional exhaustion, the basic stress dimension of burnout, the German version of the Maslach Burnout Inventory was utilized (MBI-GS-D) [75,76]. This commonly used self-report instrument consists of five items and uses a 6-point Likert-type scale anchored by 1=“never” and 6=“very often.” In this study, internal consistency was acceptable (Cronbach alpha=.79).

    Hope for improvement (confidence in treatment efficacy) was measured using the homonymous subscale of the German Patient Questionnaire on Therapy Expectation and Evaluation (PATHEV) [77] adapted to the online training context. The items are rated on a 5-point Likert-type scale. The hope for improvement subscale showed acceptable internal consistency (Cronbach alpha=.79) in this study.

    Analysis

    An ANOVA was conducted to compare the treatment adherence rates between the three guidance formats with guidance as the independent variable and adherence as the dependent variable [78]. The power analysis revealed that with the given sample size (N=395), small effects (ω2= 0.03) with a power (1-beta) of 80% and alpha=.05 would have been detected. Omega squared (ω2) was used as the measure for effect size of the ANOVA overall effect with values of 0.01, 0.06, and 0.14 representing small, medium, and large effects, respectively [79].

    In planned contrasts, the superiority of the content-focused guidance and adherence-focused guidance over administrative guidance as well as equivalence of adherence-focused guidance and content-focused guidance was assessed. The effect sizes for the planned comparisons were described by r with values of .1, .3, and .5 indicating small, medium, and large effects, respectively [80]. In order to investigate the hypothesis that adherence-focused guidance is equivalent to content-focused guidance, a confidence interval approach was used on the effect size of the difference between adherence-focused guidance and content-focused guidance with a two-sided .05 level of significance [81]. The equivalence margin was set a priori at r=.20 corresponding to the smallest value that would present a relevant effect [82]. The upper bound of the 95% CI for the effect size was compared with the predefined equivalence margin of r=.20 and had to be below the margin to show equivalence, with a significance level of .05. Hierarchical multiple linear regression analysis was performed on the combined sample to explore potential predictors of nonadherence in addition to guidance formats with gender, age, education level, stress level, depression, emotional exhaustion, and hope for improvement as the independent variables, and adherence as the dependent variable [78]. In this model, sociodemographic variables (gender, age, and education level) were entered in the first step, followed by baseline symptom-related factors (stress level, depression, and emotional exhaustion) and hope for improvement in the second step, and guidance (content-focused guidance vs administrative guidance, adherence-focused guidance vs administrative guidance) in the third step. In the final step, the interactions between the guidance formats and the other predictor variables were added to the model to explore potentially differential predictor effects as a factor of guidance formats.

    All continuous predictors were group mean centered. The power analysis revealed that with the given sample size (N=395), small effects (r=.20) with a power (1–beta) of 80% and Cronbach alpha=.05 would have been detected.

    One participant had missing values in the depression level and the hope of improvement variable for which data were imputed using a Markov chain Monte Carlo multivariate imputation algorithm (missing data module in SPSS version 22) with 100 estimations per missing value.

    In order to test the robustness of our results, we applied sensitivity analyses. We used (1) a more conservative outcome by defining modules as completed only when finished within 12 weeks and (2) Kaplan-Meier survival curves to compare the adherence rates between the three different guidance formats. All analyses were performed using SPSS version 22. Directed hypotheses were tested with a one-tailed test and nondirected hypotheses with a two-tailed test.


    Descriptive Statistics

    In total, 395 participants were included in the analysis. Baseline characteristics of the study population are presented in Table 1. There was a significant difference in the gender ratio and the education level between the three studies. The adherence-focused guidance sample showed a significantly lower percentage of male participants (13.6%, 18/132) compared to the administrative guidance (26.0%, 34/131) and content-focused guidance samples (26.5%, 35/132). The content-focused guidance sample had a significantly higher education level (64.4%, 85/132) compared to the administrative guidance (56.5%, 74/131) and adherence-focused guidance samples (52.3%, 69/132). However, gender and education level were not associated with treatment adherence in the explorative analysis and, thus, not accounted for in subsequent analyses.

    Table 1. Baseline characteristics of the study population (N=395).
    CharacteristicAdministrative guidance (n=131)Adherence-focused guidance (n=132)Content-focused guidance (n=132)P
    Age (years), mean (SD)41.2 (9.4)42.6 (9.5)42.4 (10.7).51
    Gender, n (%)


    		.02

    		Female97 (74.1)113 (85.6)97 (73.5)

    		Male34 (26.0)18 (13.6)35 (26.5)

    		Other
    		1 (0.8)

    Ethnicity, n (%)


    		.98

    		Caucasian107 (81.7)108 (81.8)110 (83.3)

    		Asian1 (0.8)1 (0.8)0

    		Not reported23 (17.6)23 (17.4)22 (16.7)
    Marital status, n (%)


    		.96

    		Unmarried40 (30.5)39 (29.6)43 (32.6)

    		Married65 (49.6)62 (47.0)63 (47.7)

    		Cohabited16 (12.2)18 (13.6)17 (12.9)

    		Separated9 (6.9)13 (9.9)8 (6.1)

    		Widowed1 (0.8)01 (0.8)
    Education, n (%)


    		.01

    		Low01 (0.8)5 (3.8)

    		Middle57 (43.5)62 (47.0)42 (31.8)

    		High74 (56.5)69 (52.3)85 (64.4)
    Gross annual income (in Euro), n (%)

    		.34

    		Low39 (29.8)41 (31.1)35 (26.5)

    		Middle40 (30.5)33 (25.0)26 (19.7)

    		High45 (34.4)49 (37.1)59 (44.7)

    		Not reported7 (5.3)9 (6.8)12 (9.1)
    Employment status, n (%)


    		.42

    		Permanent104 (79.4)107 (81.1)110 (83.3)

    		Temporary19 (14.5)14 (10.6)11 (8.3)

    		Self-employed6 (4.6)9 (6.8)11 (8.3)

    		Other2 (1.5)2 (1.5)0
    Experience with health-related programs, n (%)
    		.82

    		Yes14 (10.7)17 (12.9)17 (12.9)

    		No117 (89.3)115 (87.1)115 (87.1)
    Experience with face-to-face psychotherapy, n (%)
    		.32

    		Yes47 (35.9)46 (34.9)57 (43.2)

    		No84 (64.1)86 (65.2)75 (56.8)
    Stress, mean (SD)25.7 (5.0)25.2 (4.6)25.9 (3.9).44
    Depression, mean (SD)25.1a (9.3)23.2 (9.3)23.3 (8.5).15
    Emotional exhaustion, mean (SD)4.8 (0.8)4.7 (0.8)4.7 (0.7).96
    Hope of improvement, mean (SD)3.7a (0.6)3.6 (0.6)3.7 (0.7).27

    a Due to missing data, the means refer to a subsample with n=130 in this group.

    Adherence Rates Between Guidance Formats

    Figures 1 and 2 depict the treatment adherence rates for all three samples. Figure 1 shows the total number of completed modules per participant and Figure 2 depicts the number of completed modules by module. In the administrative guidance sample, 13.7% (18/131) of the participants did not start the intervention, and 42.0% (55/131) completed all seven modules with a mean of 4.4 completed modules (SD 2.8, range 0-7). In the group that received adherence-focused guidance, 5.3% (7/132) of the participants did not start the intervention and 68.9% (91/132) completed all modules with a mean number of 5.6 completed modules (SD 2.3, range 0-7). In the content-focused guidance sample, 7.6% (10/132) of the participants did not start the intervention and 70.5% (93/132) completed all modules with a mean number of 5.7 completed modules (SD 2.3, range 0-7).

    ‎
    Figure 1. Total number of completed modules per participant.
    View this figure
    ‎
    Figure 2. Number of completed modules by module.
    View this figure

    Guidance Formats on Adherence

    As expected, there was a significant effect of guidance on treatment adherence (F2,392=11.64, P<.001; ω2=.05). Planned comparisons revealed that (1) participants in the content-focused guidance (mean 5.7, SD 2.3) and adherence-focused guidance samples (mean 5.6, SD 2.3) completed significantly more modules than the participants in the administrative guidance sample (mean 4.4, SD 2.8; t223=4.53, P<.001; r=.29) and (2) there was no significant difference between the content-focused guidance and adherence-focused guidance groups in treatment adherence (t262=0.42, P=.67; r=.03). The effect size r=.03 (95% CI –0.09 to 0.15) did not cross the equivalence margin of r=.20. Likewise, the upper bound of the two-sided 95% confidence interval lay below the predefined margin, indicating that adherence-focused guidance was equivalent to content-focused guidance. Hence, we conclude that adherence-focused guidance is not associated with relevant lower treatment adherence compared to content-focused guidance.

    Using a more conservative outcome by defining modules as completed only when finished within 12 weeks did not result in any different conclusions (results not shown). Similarly, conducting survival analysis according to Kaplan-Meier, we derived comparable results. The survival distribution for administrative guidance was significantly different from the survival distribution of adherence-focused guidance (χ22=19.0, P<.001) and significantly different from the survival distribution of content-focused guidance (χ22=21.7, P<.001). No significant difference was found between the survival distributions of adherence-focused guidance and content-focused guidance (χ22=0.1, P=.77).

    User Characteristics Predictive of Nonadherence

    Table 2 shows the results of the imputed hierarchical multiple linear regression analysis examining predictors of treatment nonadherence over and above the guidance format. Due to missing values, the depression level variable and the hope of improvement variable of one participant had to be imputed. Entering the sociodemographic variables (gender, age, and education level) as a first step into the model did not significantly explain variance (R2=.02 , P=.09). Entering the symptom-related factors (stress level, depression, emotional exhaustion) and hope for improvement as a second block into the model resulted in significantly more explained variance (∆ R2=.02 , P=.046). Stress level and depression were predictive of treatment nonadherence in this step (t385=2.00, P=.046; t385=–2.10, P=.04). However, none of these factors was predictive of treatment nonadherence over and above the influence of guidance, which was added as dummy-coded variables (content-focused guidance vs administrative guidance; adherence-focused guidance vs administrative guidance) as a third step into the model. The final model explained 9.4% of the variance in treatment nonadherence (∆ R2=.05, P<.001; t385=4.26, P<.001; t385=4.01, P<.001).

    Table 2. Results of the imputed hierarchical multiple linear regression analysis to identify user characteristics predictive of treatment nonadherence over guidance formats.
    VariableR2 or ∆ R2aB (SE)bBetacP
    Step 1.02

    		.09

    		Constant
    		3.22 (1.08)


    		Gender
    		0.53 (0.31).09.09

    		Education
    		0.47 (0.25).10.06

    		Age
    		–0.00 (0.01)–.01.79
    Step 2.02

    		.046

    		Constant
    		0.28 (1.56)
    		.86

    		Gender
    		0.42 (0.31).07.17

    		Education
    		0.44 (0.24).09.07

    		Age
    		–0.01 (0.01)–.03.58

    		Stress
    		0.08 (0.04).14.046

    		Depression
    		–0.04 (0.02)–.15.04

    		Emotional exhaustion
    		0.31 (0.21).09.14

    		Hope for improvement
    		0.26 (0.20).07.20
    Step 3.05

    		<.001

    		Constant
    		–0.49 (1.53)


    		Gender
    		0.32 (0.30).05.29

    		Education
    		0.44 (0.24).09.06

    		Age
    		–0.01 (0.01)–.04.39

    		Stress
    		0.07 (0.04).12.08

    		Depression
    		–0.03 (0.02)–.11.13

    		Emotional exhaustion
    		0.28 (0.20).08.16

    		Hope for improvement
    		0.36 (0.20).09.07

    		Guidance format (content-focused guidance vs administrative guidance)
    		1.31 (0.31).24<.001

    		Guidance format (adherence-focused guidance vs administrative guidance)
    		1.24 (0.31).23<.001

    aR2 is for step 1 and ∆ R2 for steps 2 and 3.

    bUnstandardized regression coefficient and unstandardized standard error.

    cStandardized regression coefficient.

    Interactions Between Guidance Formats and Predictors

    Adding the interactions between the guidance formats and the other variables to the model did not significantly change explained variance (∆ R2=.02 , P=.95).


    Principal Results

    The first aim of this study was to identify the adherence rates for an Internet-based mobile-supported stress management intervention. The content-focused guidance adherence rate was 71%, which is comparable to the rates found in other guided Internet-based stress management interventions (46%-88%) [21,22,83,84]. For the adherence-focused guidance and the administrative guidance conditions, we found adherence rates of 69% and 42%, respectively. Comparing adherence rates for adherence-focused guidance and administrative guidance to previous trials is difficult because most of those trials conducted in the field either used other guidance formats or did not report adherence rates. However, in a study with adherence reminders [85], a module completion rate of 35% was found, which is below the rate for adherence-focused guidance in this study (69%). Nevertheless, varying definitions and operationalization of module completion limit the comparability of treatment adherence rates between different studies. Despite the limited comparability, this study’s results suggest relatively high adherence rates for content-focused guidance and adherence-focused guidance and lower rates for administrative guidance.

    Our second research goal was to investigate the influence of different guidance formats on adherence in an Internet-based mobile-supported stress management intervention. Similarly to studies on other target conditions, such as depression [31] and anxiety [38,56], this study suggests that participants show better adherence with guided treatments in comparison to unguided treatments. As predicted, content-focused guidance and adherence-focused guidance resulted in higher treatment adherence rates compared to unguided treatment with only administrative support (administrative guidance).

    As hypothesized, both content-focused guidance and adherence-focused guidance have high adherence rates; therefore, the next step was to analyze their equivalence in terms of treatment adherence. For both guidance formats, adherence was equivalent. Despite the equivalence in adherence rates, both guidance formats differ in the amount of eCoaching each requires. Content-focused guidance included both reminders and written feedback from an eCoach on every completed module and required up to 4 hours of coaching time. In contrast, adherence-focused guidance consisted of adherence monitoring and feedback on demand and only required up to 1 hour of coaching time per participant during the intervention. Therefore, by choosing adherence-focused guidance regarding the costs of treatment, substantial savings may be made without a significant reduction in patient adherence. This finding is in line with the assumption that the active factor responsible for improving adherence in guided versus unguided self-help interventions is that the participant is accompanied through the intervention. Providing instructions or detailed feedback on the content the participants worked on within the modules seems less critical for continued participant engagement.

    However, the incremental value of offering feedback on demand compared to only adherence monitoring from an eCoach remains yet unclear. Within the adherence-focused guidance concept, it is hypothesized that feedback on demand is an important component so that the eCoach is seen as having the participant’s best interests at heart. Offering support may be an antecedent for creating an adherence-promoting relationship and, according to the supportive accountability model [25], assumed to be necessary to maximize the effects of adherence monitoring. But, it is possible that the superiority of adherence-focused guidance compared to administrative guidance can be purely explained by the effect of personalized reminders. Thus, future studies should investigate whether having the possibility to contact an eCoach has an incremental influence on treatment adherence beyond adherence monitoring.

    In the adherence-focused guidance study arm, monitoring adherence and sending a personalized reminder took up almost all the resources associated with this guidance format (approximately 1 hour per participant). In contrast, feedback on demand required much less resources. Hence, the question arises whether automatic reminders sent from the system on behalf of the eCoach have, in combination with feedback on demand, a similar effect on adherence, while requiring even less resources. Other studies have already shown positive effects on treatment adherence through automatic reminders [86].

    If feedback on demand from a health professional is not a necessary component to achieve sufficient treatment adherence, adherence monitoring may also be performed by nonprofessionals. This would improve cost-effectiveness and dissemination. The eCoaches’ qualification level was not found to significantly influence treatment efficacy in Internet interventions for a range of conditions [27]. A study by Titov and colleagues [87] did not show significant differences in clinical efficacy between layperson- or clinician-assisted Internet interventions. Hence, the characteristics of an eCoach keeping participants adherent to an intervention should be further investigated. It also remains unclear whether a personal coach is necessary at all.

    However, reducing human contact in Internet interventions can also entail potential risks for participants. Without content-related feedback, the eCoach may not become aware of problems participants may experience during the training. Thus, the risk for negative effects with Internet interventions for some individuals has the potential to be higher when receiving adherence-focused guidance instead of content-focused guidance. For this reason, negative effects should be investigated in future studies that compare different guidance formats [88,89].

    Likewise, the guidance format could also influence the acceptance and attractiveness of Internet interventions, and thereby be important for dissemination. Therefore, varying guidance formats in Internet interventions should also be evaluated in terms of attractiveness and general acceptance [90,91].

    The preceding discussion alludes to the many factors that can influence guidance in Internet interventions. Our third research question moves the focus from intervention characteristics to participant characteristics. It aimed to identify participant characteristics as predictors of nonadherence in an Internet-based mobile-supported stress management intervention. Although the guidance formats significantly predicted nonadherence in this study, the predictive value of the variables targeted in this analysis is small. The final model only explained 9.4% of the interindividual differences in nonadherence rates. Other variables, purported in previous literature as predictive of treatment nonadherence in Internet interventions, may prove to be incrementally important and should be investigated further. These include, for example, primary motivation, intention to adhere, and self-efficacy, as well as computer savviness, Internet affinity, and usability in addition to system- and program-related variables [25,44,92,93].

    Limitations

    This study has several limitations. First, the participants were not randomized to the three study arms. Hence, the differences between the guidance formats may be confounded by other differences between the studies.

    Second, participants in the adherence-focused guidance group showed a significantly higher percentage of female users at baseline compared to participants in the administrative guidance group. Furthermore, the content-focused guidance sample had a significantly higher education level compared to the administrative guidance and adherence-focused guidance samples. These unsystematic variations may have contributed to better adherence rates in the adherence-focused guidance and content-focused guidance groups compared to the administrative guidance group. But gender and education level were not significantly associated with treatment nonadherence. However, the total sample was found to be highly educated, which may limit the generalization to the population.

    Third, the adherence rates identified in this study refer to an Internet-based mobile-supported stress management intervention administered in RCTs. Several studies indicate that adherence rates to Internet interventions in the context of RCTs are higher than those available from open-access websites [7,9,94]. Improved usage rates in trial participants may indicate a preference for formal structures (eg, assessments, contact with research staff), which are thought to promote adherence. In addition, the elaborated study inclusion process (ie, completion of two self-report assessments, sending of informed consent) may have led to the inclusion of more motivated individuals through self-selection. As a result, adherence rates may not apply to other Internet interventions or Internet interventions applied to a different context. Hence, treatment adherence rates in this study might not be generalizable to an Internet-based mobile-supported stress management intervention in an open-access practice. However, Internet interventions in open-access contexts lack the structure-giving elements of RCTs, such as contact with the study administration team. Hence, guidance could play an even more important role in open-access Internet interventions by structuring the course of treatment. Thus, the differences between content-focused guidance, adherence-focused guidance, and administrative guidance may have been underestimated in this study.

    Fourth, generalizability of the treatment adherence rates in an Internet-based mobile-supported stress management intervention are further limited by only including employees with elevated stress levels in the studies. Thus, the findings of this study may not be relevant for settings with participants that were not preselected based on their stress level. However, the stress level was not significantly associated with lower adherence rates beyond the guidance formats.

    Fifth, only one adherence measure was included in the analysis. Different adherence measures need to be used which capture the quality of engagement with an intervention to a greater extent, such as time on website, number of completed homework assignments, and diary entries [95].

    Sixth, as in most predictor studies, the analyses in this study were exploratory without any presumptions about the relationship between the predictors and adherence in order to generate hypotheses [96].

    Conclusion and Future Research

    This study has important implications for research and practice. Guidance with focus on treatment adherence has the potential to be helpful in keeping participants involved in the training and, at the same time, keeping coaching costs low. Evaluating the cost-effectiveness as well as the comparative efficacy of the different guidance formats should be a next step to further complement the findings of this study [40]. Moreover, low treatment adherence is not negative per se because participants might quit the intervention due to improvement or because it does not meet their needs. Therefore, it is important to identify the reasons for nonadherence qualitatively to assess the impact of nonadherence for the efficacy of an intervention. Future research should focus on developing a theoretical model to explain the mechanisms behind treatment nonadherence in Internet interventions and derive variables from existing theories. This information can be used as a framework for the construction of an algorithm that generates risk profiles for participants that may be more susceptible to nonadherence.

    Acknowledgments

    This study was funded by the European Union (EFRE: ZW6-80119999, CCI 2007DE161PR001) and the BARMER GEK. Furthermore, we would like to acknowledge Elena Heber, Angelina Scheel, and Laura Iffländer for the study administration.

    Conflicts of Interest

    Dirk Lehr, Matthias Berking and David Ebert are stakeholders of the “Institute for Online Health Trainings“, that aims to transfer scientific knowledge related to the present research into routine healthcare.

    1. Batterham PJ, Neil AL, Bennett K, Griffiths KM, Christensen H. Predictors of adherence among community users of a cognitive behavior therapy website. Patient Prefer Adherence 2008;2:97-105 [FREE Full text] [Medline]
    2. Bossen D, Buskermolen M, Veenhof C, de Bakker BD, Dekker J. Adherence to a web-based physical activity intervention for patients with knee and/or hip osteoarthritis: a mixed method study. J Med Internet Res 2013;15(10):e223 [FREE Full text] [CrossRef] [Medline]
    3. Calear AL, Christensen H, Mackinnon A, Griffiths KM. Adherence to the MoodGYM program: outcomes and predictors for an adolescent school-based population. J Affect Disord 2013 May;147(1-3):338-344. [CrossRef] [Medline]
    4. Farvolden P, Denisoff E, Selby P, Bagby RM, Rudy L. Usage and longitudinal effectiveness of a Web-based self-help cognitive behavioral therapy program for panic disorder. J Med Internet Res 2005;7(1):e7 [FREE Full text] [CrossRef] [Medline]
    5. Newby JM, Mackenzie A, Williams AD, McIntyre K, Watts S, Wong N, et al. Internet cognitive behavioural therapy for mixed anxiety and depression: a randomized controlled trial and evidence of effectiveness in primary care. Psychol Med 2013 Dec;43(12):2635-2648. [CrossRef] [Medline]
    6. Vandelanotte C, Spathonis KM, Eakin EG, Owen N. Website-delivered physical activity interventions a review of the literature. Am J Prev Med 2007 Jul;33(1):54-64. [CrossRef] [Medline]
    7. Christensen H, Griffiths KM, Farrer L. Adherence in Internet interventions for anxiety and depression. J Med Internet Res 2009;11(2):e13 [FREE Full text] [CrossRef] [Medline]
    8. Beintner I, Jacobi C, Taylor CB. Participant adherence to the Internet-based prevention program StudentBodies™ for eating disorders — A review. Internet Interventions 2014 Mar;1(1):26-32. [CrossRef]
    9. Christensen H, Griffiths KM, Korten AE, Brittliffe K, Groves C. A comparison of changes in anxiety and depression symptoms of spontaneous users and trial participants of a cognitive behavior therapy website. J Med Internet Res 2004 Dec 22;6(4):e46 [FREE Full text] [CrossRef] [Medline]
    10. Cugelman B, Thelwall M, Dawes P. Online interventions for social marketing health behavior change campaigns: a meta-analysis of psychological architectures and adherence factors. J Med Internet Res 2011;13(1):e17 [FREE Full text] [CrossRef] [Medline]
    11. Donkin L, Christensen H, Naismith SL, Neal B, Hickie IB, Glozier N. A systematic review of the impact of adherence on the effectiveness of e-therapies. J Med Internet Res 2011;13(3):e52 [FREE Full text] [CrossRef] [Medline]
    12. Manwaring JL, Bryson SW, Goldschmidt AB, Winzelberg AJ, Luce KH, Cunning D, et al. Do adherence variables predict outcome in an online program for the prevention of eating disorders? J Consult Clin Psychol 2008 Apr;76(2):341-346. [CrossRef] [Medline]
    13. Meyer B, Berger T, Caspar F, Beevers CG, Andersson G, Weiss M. Effectiveness of a novel integrative online treatment for depression (Deprexis): randomized controlled trial. J Med Internet Res 2009;11(2):e15 [FREE Full text] [CrossRef] [Medline]
    14. Stansfeld S, Candy B. Psychosocial work environment and mental health--a meta-analytic review. Scand J Work Environ Health 2006 Dec;32(6):443-462 [FREE Full text] [Medline]
    15. Billings DW, Cook RF, Hendrickson A, Dove DC. A web-based approach to managing stress and mood disorders in the workforce. J Occup Environ Med 2008 Aug;50(8):960-968. [CrossRef] [Medline]
    16. Ebert DD, Lehr D, Boß L, Riper H, Cuijpers P, Andersson G, et al. Efficacy of an internet-based problem-solving training for teachers: results of a randomized controlled trial. Scand J Work Environ Health 2014 Nov;40(6):582-596. [CrossRef] [Medline]
    17. Eisen KP, Allen GJ, Bollash M, Pescatello LS. Stress management in the workplace: a comparison of a computer-based and an in-person stress-management intervention. Comput Hum Behav 2008 Mar;24(2):486-496. [CrossRef]
    18. Grime PR. Computerized cognitive behavioural therapy at work: a randomized controlled trial in employees with recent stress-related absenteeism. Occup Med (Lond) 2004 Aug;54(5):353-359 [FREE Full text] [CrossRef] [Medline]
    19. Hasson D, Anderberg UM, Theorell T, Arnetz BB. Psychophysiological effects of a web-based stress management system: a prospective, randomized controlled intervention study of IT and media workers [ISRCTN54254861]. BMC Public Health 2005;5:78 [FREE Full text] [CrossRef] [Medline]
    20. Drozd F, Raeder S, Kraft P, Bjørkli CA. Multilevel growth curve analyses of treatment effects of a Web-based intervention for stress reduction: randomized controlled trial. J Med Internet Res 2013;15(4):e84 [FREE Full text] [CrossRef] [Medline]
    21. Hänggi Y. Kann web-basierte Stressprävention erfolgreich sein? Zeitschrift für Klinische Psychologie und Psychotherapie 2006 Jul 1;35(3):169-177. [CrossRef]
    22. Zetterqvist K, Maanmies J, Ström L, Andersson G. Randomized controlled trial of internet-based stress management. Cogn Behav Ther 2003;32(3):151-160. [CrossRef] [Medline]
    23. Eysenbach G. The law of attrition. J Med Internet Res 2005;7(1):e11 [FREE Full text] [CrossRef] [Medline]
    24. Beintner I, Jacobi C, Schmidt UH. Participation and outcome in manualized self-help for bulimia nervosa and binge eating disorder - a systematic review and metaregression analysis. Clin Psychol Rev 2014 Mar;34(2):158-176. [CrossRef] [Medline]
    25. Mohr DC, Cuijpers P, Lehman K. Supportive accountability: a model for providing human support to enhance adherence to eHealth interventions. J Med Internet Res 2011;13(1):e30 [FREE Full text] [CrossRef] [Medline]
    26. Mohr DC, Duffecy J, Ho J, Kwasny M, Cai X, Burns MN, et al. A randomized controlled trial evaluating a manualized TeleCoaching protocol for improving adherence to a web-based intervention for the treatment of depression. PLoS One 2013 Aug;8(8):e70086 [FREE Full text] [CrossRef] [Medline]
    27. Baumeister H, Reichler L, Munzinger M, Lin J. The impact of guidance on Internet-based mental health interventions — a systematic review. Internet Interventions 2014 Oct;1(4):205-215. [CrossRef]
    28. Berger T, Hämmerli K, Gubser N, Andersson G, Caspar F. Internet-based treatment of depression: a randomized controlled trial comparing guided with unguided self-help. Cogn Behav Ther 2011 Dec;40(4):251-266. [CrossRef] [Medline]
    29. Nicholas J, Proudfoot J, Parker G, Gillis I, Burckhardt R, Manicavasagar V, et al. The ins and outs of an online bipolar education program: a study of program attrition. J Med Internet Res 2010;12(5):e57 [FREE Full text] [CrossRef] [Medline]
    30. Proudfoot J, Parker G, Manicavasagar V, Hadzi-Pavlovic D, Whitton A, Nicholas J, et al. Effects of adjunctive peer support on perceptions of illness control and understanding in an online psychoeducation program for bipolar disorder: a randomised controlled trial. J Affect Disord 2012 Dec 15;142(1-3):98-105. [CrossRef] [Medline]
    31. Richards D, Richardson T. Computer-based psychological treatments for depression: a systematic review and meta-analysis. Clin Psychol Rev 2012 Jun;32(4):329-342. [CrossRef] [Medline]
    32. van Ballegooijen W, Cuijpers P, van Straten A, Karyotaki E, Andersson G, Smit JH, et al. Adherence to Internet-based and face-to-face cognitive behavioural therapy for depression: a meta-analysis. PLoS One 2014;9(7):e100674 [FREE Full text] [CrossRef] [Medline]
    33. Steele RM, Mummery WK, Dwyer T. Examination of program exposure across intervention delivery modes: face-to-face versus internet. Int J Behav Nutr Phys Act 2007;4:7 [FREE Full text] [CrossRef] [Medline]
    34. Newman MG, Szkodny LE, Llera SJ, Przeworski A. A review of technology-assisted self-help and minimal contact therapies for anxiety and depression: is human contact necessary for therapeutic efficacy? Clin Psychol Rev 2011 Feb;31(1):89-103. [CrossRef] [Medline]
    35. Clarke G, Reid E, Eubanks D, O'Connor E, DeBar LL, Kelleher C, et al. Overcoming depression on the Internet (ODIN): a randomized controlled trial of an Internet depression skills intervention program. J Med Internet Res 2002 Dec;4(3):E14 [FREE Full text] [CrossRef] [Medline]
    36. Clarke G, Eubanks D, Reid E, Kelleher C, O'Connor E, DeBar LL, et al. Overcoming Depression on the Internet (ODIN) (2): a randomized trial of a self-help depression skills program with reminders. J Med Internet Res 2005;7(2):e16 [FREE Full text] [CrossRef] [Medline]
    37. Schneider F, van Osch L, Schulz DN, Kremers SP, de Vries H. The influence of user characteristics and a periodic email prompt on exposure to an internet-delivered computer-tailored lifestyle program. J Med Internet Res 2012;14(2):e40 [FREE Full text] [CrossRef] [Medline]
    38. Titov N, Andrews G, Choi I, Schwencke G, Johnston L. Randomized controlled trial of web-based treatment of social phobia without clinician guidance. Aust NZ J Psychiatry 2009 Jan;43(10):913-919. [CrossRef]
    39. Titov N, Dear BF, Johnston L, Lorian C, Zou J, Wootton B, et al. Improving adherence and clinical outcomes in self-guided internet treatment for anxiety and depression: randomised controlled trial. PLoS One 2013;8(7):e62873 [FREE Full text] [CrossRef] [Medline]
    40. Ebert DD, Lehr D, Smit F, Zarski A, Riper H, Heber E, et al. Efficacy and cost-effectiveness of minimal guided and unguided internet-based mobile supported stress-management in employees with occupational stress: a three-armed randomised controlled trial. BMC Public Health 2014;14:807 [FREE Full text] [CrossRef] [Medline]
    41. Ritterband LM, Thorndike FP, Cox DJ, Kovatchev BP, Gonder-Frederick LA. A behavior change model for Internet interventions. Ann Behav Med 2009 Aug;38(1):18-27 [FREE Full text] [CrossRef] [Medline]
    42. Murray E, White IR, Varagunam M, Godfrey C, Khadjesari Z, McCambridge J. Attrition revisited: adherence and retention in a web-based alcohol trial. J Med Internet Res 2013;15(8):e162 [FREE Full text] [CrossRef] [Medline]
    43. Strecher VJ, McClure J, Alexander G, Chakraborty B, Nair V, Konkel J, et al. The role of engagement in a tailored web-based smoking cessation program: randomized controlled trial. J Med Internet Res 2008;10(5):e36 [FREE Full text] [CrossRef] [Medline]
    44. Wangberg SC, Bergmo TS, Johnsen JK. Adherence in Internet-based interventions. Patient Prefer Adherence 2008;2:57-65 [FREE Full text] [Medline]
    45. Zbikowski SM, Hapgood J, Smucker BS, McAfee T. Phone and web-based tobacco cessation treatment: real-world utilization patterns and outcomes for 11,000 tobacco users. J Med Internet Res 2008;10(5):e41 [FREE Full text] [CrossRef] [Medline]
    46. Lange A, Rietdijk D, Hudcovicova M, van de Ven JP, Schrieken B, Emmelkamp PM. Interapy: a controlled randomized trial of the standardized treatment of posttraumatic stress through the internet. J Consult Clin Psychol 2003 Oct;71(5):901-909. [CrossRef] [Medline]
    47. Linke S, Murray E, Butler C, Wallace P. Internet-based interactive health intervention for the promotion of sensible drinking: patterns of use and potential impact on members of the general public. J Med Internet Res 2007;9(2):e10 [FREE Full text] [CrossRef] [Medline]
    48. van Straten A, Cuijpers P, Smits N. Effectiveness of a web-based self-help intervention for symptoms of depression, anxiety, and stress: randomized controlled trial. J Med Internet Res 2008;10(1):e7 [FREE Full text] [CrossRef] [Medline]
    49. Verheijden MW, Jans MP, Hildebrandt VH, Hopman-Rock M. Rates and determinants of repeated participation in a web-based behavior change program for healthy body weight and healthy lifestyle. J Med Internet Res 2007;9(1):e1 [FREE Full text] [CrossRef] [Medline]
    50. Spek V, Nyklícek I, Smits N, Cuijpers P, Riper H, Keyzer J, et al. Internet-based cognitive behavioural therapy for subthreshold depression in people over 50 years old: a randomized controlled clinical trial. Psychol Med 2007 Dec;37(12):1797-1806. [CrossRef] [Medline]
    51. Spek V, Nyklícek I, Cuijpers P, Pop V. Predictors of outcome of group and Internet-based cognitive behavior therapy. J Affect Disord 2008 Jan;105(1-3):137-145. [CrossRef] [Medline]
    52. Neil AL, Batterham P, Christensen H, Bennett K, Griffiths KM. Predictors of adherence by adolescents to a cognitive behavior therapy website in school and community-based settings. J Med Internet Res 2009;11(1):e6 [FREE Full text] [CrossRef] [Medline]
    53. Postel MG, de Haan HA, ter Huurne ED, van der Palen J, Becker ES, de Jong CA. Attrition in web-based treatment for problem drinkers. J Med Internet Res 2011;13(4):e117 [FREE Full text] [CrossRef] [Medline]
    54. Farrer LM, Griffiths KM, Christensen H, Mackinnon AJ, Batterham PJ. Predictors of adherence and outcome in Internet-based cognitive behavior therapy delivered in a telephone counseling Setting. Cogn Ther Res 2013 Oct 27;38(3):358-367. [CrossRef]
    55. von Brachel R, Hötzel K, Hirschfeld G, Rieger E, Schmidt U, Kosfelder J, et al. Internet-based motivation program for women with eating disorders: eating disorder pathology and depressive mood predict dropout. J Med Internet Res 2014;16(3):e92 [FREE Full text] [CrossRef] [Medline]
    56. Nordgreen T, Havik OE, Ost LG, Furmark T, Carlbring P, Andersson G. Outcome predictors in guided and unguided self-help for social anxiety disorder. Behav Res Ther 2012 Jan;50(1):13-21. [CrossRef] [Medline]
    57. Wojtowicz M, Day V, McGrath PJ. Predictors of participant retention in a guided online self-help program for university students: prospective cohort study. J Med Internet Res 2013;15(5):e96 [FREE Full text] [CrossRef] [Medline]
    58. Boettcher J, Renneberg B, Berger T. Patient expectations in internet-based self-help for social anxiety. Cogn Behav Ther 2013;42(3):203-214. [CrossRef] [Medline]
    59. Cavanagh K, Shapiro DA, Van Den Berg S, Swain S, Barkham M, Proudfoot J. The acceptability of computer-aided cognitive behavioural therapy: a pragmatic study. Cogn Behav Ther 2009;38(4):235-246. [CrossRef] [Medline]
    60. Melville KM, Casey LM, Kavanagh DJ. Dropout from Internet-based treatment for psychological disorders. Br J Clin Psychol 2010 Nov;49(Pt 4):455-471. [CrossRef] [Medline]
    61. Heber E, Ebert DD, Lehr D, Nobis S, Berking M, Riper H. Efficacy and cost-effectiveness of a web-based and mobile stress-management intervention for employees: design of a randomized controlled trial. BMC Public Health 2013;13:655 [FREE Full text] [CrossRef] [Medline]
    62. Heber E, Lehr D, Ebert DD, Berking M, Riper H. Web-based and mobile stress management intervention for employees: a randomized controlled trial. J Med Internet Res 2016;18(1):e21 [FREE Full text] [CrossRef] [Medline]
    63. Ebert DD, Heber E, Berking M, Riper H, Cuijpers P, Funk B, et al. Self-guided internet-based and mobile-based stress management for employees: results of a randomised controlled trial. Occup Environ Med 2016 May;73(5):315-323. [CrossRef] [Medline]
    64. Ebert DD, Heber E, Riper H, Cuijpers P, Berking M, Lehr D. Efficacy and mechanism of change of Internet- and mobile based stress management for employees with adherence focused guidance: a randomized controlled trial. Scand Journ Work Env Health 2016:tbd (forthcoming).
    65. Cohen S, Kamarck T, Mermelstein R. A global measure of perceived stress. J Health Soc Behav 1983 Dec;24(4):385-396. [Medline]
    66. Beck AT, Steer AS, Brown GK. BDI-II: Beck Depression Inventory Manual. 2nd ed. San Antonio, TX: Psychological Corporation; 1996.
    67. Junge MN, Lehr D, Bockting CL, Berking M, Riper H, Cuijpers P, et al. For whom are internet-based occupational mental health interventions effective? Moderators of internet-based problem-solving training outcome. Internet Interventions 2015 Mar;2(1):39-47. [CrossRef]
    68. Nezu AM, Perri MG. Social problem-solving therapy for unipolar depression: an initial dismantling investigation. J Consult Clin Psychol 1989 Jun;57(3):408-413. [Medline]
    69. Warmerdam L, van Straten A, Twisk J, Riper H, Cuijpers P. Internet-based treatment for adults with depressive symptoms: randomized controlled trial. J Med Internet Res 2008;10(4):e44 [FREE Full text] [CrossRef] [Medline]
    70. Berking M, Whitley B. Affect Regulation Training. New York: Springer New York; 2014.
    71. Berking M, Ebert D, Cuijpers P, Hofmann SG. Emotion regulation skills training enhances the efficacy of inpatient cognitive behavioral therapy for major depressive disorder: a randomized controlled trial. Psychother Psychosom 2013;82(4):234-245. [CrossRef] [Medline]
    72. Michalsen A, Jeitler M, Brunnhuber S, Lüdtke R, Büssing A, Musial F, et al. Iyengar yoga for distressed women: a 3-armed randomized controlled trial. Evid Based Complement Alternat Med 2012;2012:408727 [FREE Full text] [CrossRef] [Medline]
    73. Hautzinger M, Bailer M, Hofmeister D, Keller F. Allgemeine Depressions Skala (ADS) Manual [Center for Epidemiological Studies Depression Scale (CES-D; Radloff LS, 1977)]. Boston, MA: Hogrefe Publishers; 2012.
    74. Radloff LS. The CES-D scale: a self-report depression scale for research in the general population. Appl Psych Meas 1977 Jun 01;1(3):385-401. [CrossRef]
    75. Neubach B, Schmidt K. Gütekriterien einer deutschen Fassung des Mastach Burnout Inventory (MBI-D) - Eine Replikationsstudie bei Altenpflegekräften. Zeitschrift für Arbeits- und Organisationspsychologie A&O 2000 Jul 1;44(3):140-156. [CrossRef]
    76. Schaufeli WB, Leiter MP, Maslach C, Jackson SE. The MBI-general survey. In: Maslach C, Jackson SE, Leiter MP, editors. Maslach Burnout Inventory Manual. 3rd ed. Palo Alto, CA: Consulting Psychologists Press; 1996:19-26.
    77. Schulte D. Messung der Therapieerwartung und Therapieevaluationvon Patienten (PATHEV). Zeitschrift für Klinische Psychologie und Psychotherapie 2005 Jul 1;34(3):176-187. [CrossRef]
    78. Mertler CA, Vannatta RA. Advanced and Multivariate Statistical Methods: Practical Application and Interpretation (5th ed). Glendale, CA: Pyrczak Publishing; 2000.
    79. Kirk RE. Practical significance: a concept whose time has come. Educ Psychol Meas 1996 Oct 01;56(5):746-759. [CrossRef]
    80. Cohen J. A power primer. Psychol Bull 1992 Jul;112(1):155-159. [Medline]
    81. Piaggio G, Elbourne DR, Pocock SJ, Evans SJ, Altman DG. Reporting of noninferiority and equivalence randomized trials: extension of the CONSORT 2010 statement. JAMA 2012 Dec 26;308(24):2594-2604. [CrossRef] [Medline]
    82. Wiens BL. Choosing an equivalence limit for noninferiority or equivalence studies. Control Clin Trials 2002 Feb;23(1):2-14. [Medline]
    83. Rose RD, Buckey JC, Zbozinek TD, Motivala SJ, Glenn DE, Cartreine JA, et al. A randomized controlled trial of a self-guided, multimedia, stress management and resilience training program. Behav Res Ther 2013 Feb;51(2):106-112. [CrossRef] [Medline]
    84. Ruwaard J, Lange A, Bouwman M, Broeksteeg J, Schrieken B. E-mailed standardized cognitive behavioural treatment of work-related stress: a randomized controlled trial. Cogn Behav Ther 2007;36(3):179-192. [CrossRef] [Medline]
    85. Morledge TJ, Allexandre D, Fox E, Fu AZ, Higashi MK, Kruzikas DT, et al. Feasibility of an online mindfulness program for stress management: a randomized, controlled trial. Ann Behav Med 2013 Oct;46(2):137-148 [FREE Full text] [CrossRef] [Medline]
    86. Robertson L, Smith M, Castle D, Tannenbaum D. Using the Internet to enhance the treatment of depression. R Aust New Zeal Coll Psychiatr 2006;14(4):413-417. [Medline]
    87. Titov N, Andrews G, Davies M, McIntyre K, Robinson E, Solley K. Internet treatment for depression: a randomized controlled trial comparing clinician vs. technician assistance. PLoS One 2010;5(6):e10939 [FREE Full text] [CrossRef] [Medline]
    88. Rozental A, Andersson G, Boettcher J, Ebert DD, Cuijpers P, Knaevelsrud C, et al. Consensus statement on defining and measuring negative effects of Internet interventions. Internet Interventions 2014 Mar;1(1):12-19. [CrossRef]
    89. Ebert DD, Lehr D, Baumeister H, Boß L, Riper H, Cuijpers P, et al. GET.ON Mood Enhancer: efficacy of Internet-based guided self-help compared to psychoeducation for depression: an investigator-blinded randomised controlled trial. Trials 2014;15:39 [FREE Full text] [CrossRef] [Medline]
    90. Baumeister H, Nowoczin L, Lin J, Seifferth H, Seufert J, Laubner K, et al. Impact of an acceptance facilitating intervention on diabetes patients' acceptance of Internet-based interventions for depression: a randomized controlled trial. Diabetes Res Clin Pract 2014 Jul;105(1):30-39. [CrossRef] [Medline]
    91. Baumeister H, Seifferth H, Lin J, Nowoczin L, Lüking M, Ebert D. Impact of an acceptance facilitating intervention on patients' acceptance of Internet-based pain interventions: a randomized controlled trial. Clin J Pain 2015 Jun;31(6):528-535. [CrossRef] [Medline]
    92. Hebert EA, Vincent N, Lewycky S, Walsh K. Attrition and adherence in the online treatment of chronic insomnia. Behav Sleep Med 2010;8(3):141-150. [CrossRef] [Medline]
    93. Kelders SM, Kok RN, Ossebaard HC, Van Gemert-Pijnen JE. Persuasive system design does matter: a systematic review of adherence to web-based interventions. J Med Internet Res 2012;14(6):e152 [FREE Full text] [CrossRef] [Medline]
    94. Poston JM, Hanson WE. Meta-analysis of psychological assessment as a therapeutic intervention. Psychol Assess 2010 Jun;22(2):203-212. [CrossRef] [Medline]
    95. Mohr DC, Burns MN, Schueller SM, Clarke G, Klinkman M. Behavioral intervention technologies: evidence review and recommendations for future research in mental health. Gen Hosp Psychiatry 2013 Aug;35(4):332-338 [FREE Full text] [CrossRef] [Medline]
    96. Kraemer HC, Wilson GT, Fairburn CG, Agras WS. Mediators and moderators of treatment effects in randomized clinical trials. Arch Gen Psychiatry 2002;59(10):877-883. [Medline]

    ‎
    RCT: randomized controlled trial

    Edited by G Eysenbach; submitted 01.04.15; peer-reviewed by D Mohr, S Lippke; comments to author 30.04.15; revised version received 25.06.15; accepted 23.01.16; published 29.06.16

    Copyright

    ©Anna-Carlotta Zarski, Dirk Lehr, Matthias Berking, Heleen Riper, Pim Cuijpers, David Daniel Ebert. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 29.06.2016.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.