This paper is in the following e-collection/theme issue:

Web-based and Mobile Health Interventions (3297) Digital Mental Health Interventions, e-Mental Health and Cyberpsychology (1514) Anxiety and Stress Disorders (1053) Panic Disorder (18) Phobias (e.g. Agoraphobia) other than SAD (12)

Published on in Vol 16, No 9 (2014): September

Short-Term Effectiveness of Web-Based Guided Self-Help for Phobic Outpatients: Randomized Controlled Trial

Short-Term Effectiveness of Web-Based Guided Self-Help for Phobic Outpatients: Randomized Controlled Trial

Short-Term Effectiveness of Web-Based Guided Self-Help for Phobic Outpatients: Randomized Controlled Trial

Authors of this article:

Robin N Kok1, 2 Author Orcid Image ;   Annemieke van Straten1 Author Orcid Image ;   Aartjan T F Beekman3 Author Orcid Image ;   Pim Cuijpers1, 4 Author Orcid Image
Article Authors Cited by (29) Tweetations (5) Metrics

    Original Paper

    1Department of Clinical Psychology and the EMGO institute for Health and Care Research, VU University Amsterdam, Amsterdam, Netherlands

    2National Institute for Mental Health Research, The Australian National University, Canberra, Australia

    3Department of Psychiatry and the EMGO institute for Health and Care Research, VU University Medical Centre, Amsterdam, Netherlands

    4Innovation Incubator, Division Health-Training.Online, Leuphana University, Lüneburg, Germany

    Corresponding Author:

    Robin N Kok, MSc

    Department of Clinical Psychology and the EMGO institute for Health and Care Research

    VU University Amsterdam

    van der Boechorststraat 1

    Amsterdam, 1081 BT

    Netherlands

    Phone: 31 205983833

    Fax:31 205988756

    Email: r.n.kok@vu.nl


    Background: Internet-based guided self-help has been successfully used in the general population, but it is unknown whether this method can be effectively used in outpatient clinics for patients waiting for face-to-face psychotherapy for phobias.

    Objective: The aim was to assess the clinical effectiveness of Phobias Under Control, an Internet-based intervention based on exposure therapy with weekly guidance.

    Methods: We conducted a randomized controlled trial, recruiting 212 outpatients scheduled to receive face-to-face psychotherapy for any type of phobia at an outpatient clinic. Participants suffering from at least 1 DSM-IV or ICD-10 classified phobia (social phobia, agoraphobia with or without panic disorder, and/or specific phobia as ascertained by a telephone interview at baseline) were randomly allocated to either a 5-week Internet-based guided self-help program based on exposure therapy with weekly student support followed by face-to-face psychotherapy (n=105) or a wait-list control group followed by face-to-face psychotherapy (n=107). Primary outcome was the Fear Questionnaire (FQ). Secondary outcomes were the Beck Anxiety Inventory (BAI) and Center of Epidemiological Studies-Depression scale (CES-D). Assessments took place by telephone at baseline (T0) and on the Internet at posttest (T1, self-assessment at 5 weeks after baseline). Missing data at T1 were imputed.

    Results: At posttest, analysis of covariance on the intention-to-treat sample showed significant but small effect sizes between intervention and control groups on the FQ (d=0.35, P=.02), CES-D (d=0.34, P=.03), and a nonsignificant effect size on the BAI (d=0.28. P=.05). Although initial acceptance was good, high nonresponse was observed, with 86 of 212 participants (40.5%) lost to follow-up at T1 and only 14 of 105 (13.3%) intervention participants finishing all 5 weeks.

    Conclusions: Phobias Under Control is modestly effective in lowering phobic and depressive symptoms in a relatively short period and may be clinically beneficial when implemented in routine outpatient practice.

    Trial Registration: Netherlands Trial Register NTR2233; http://www.trialregister.nl/trialreg/admin/rctview.asp?TC=2233 (Archived by WebCite at http://www.webcitation.org/6O2ioOQSs).

    J Med Internet Res 2014;16(9):e226

    doi:10.2196/jmir.3429

    Keywords

    phobias; phobic disorders; anxiety disorders; Web-based intervention; Internet therapy; randomized controlled trial; outpatients 


    Phobias are among the most common mental disorders and the most common type of anxiety disorders [1]. Specific phobias are the most common form of anxiety disorders for both genders, with a total 12-month prevalence of 7.1%, followed by social phobia (4.8%) and agoraphobia without panic disorder (1.2%). All phobias have a negative impact on quality of life and psychosocial functioning [2], and the societal burden of phobias is considerable [3,4]. Despite detrimental effects on quality of life, research has shown a substantial delay of more than 10 years between onset of symptoms and first therapy attendance [5]. In a recent study, social phobia was found not only to have the earliest onset age, but also an even longer delay—on average 28 years—in seeking treatment [6]. Notwithstanding the impact of a phobia on a patient’s quality of life [2,7], phobias are often not the primary reason for seeking treatment from an outpatient clinic [8,9] and it has been argued that commonly occurring comorbid disorders, such as depression, mask underlying social phobia leading to underdiagnosis in primary care [10]. This suggests widespread undertreatment [11] for these disorders, even though there is robust evidence of efficacious psychological treatments for agoraphobia [12], social phobia [13], and specific phobias [14], most notably exposure therapy and cognitive behavioral therapy (CBT).

    Internet-based interventions are increasingly popular adaptations of evidence-based psychotherapies as a replacement of, or adjunct to, traditional face-to-face therapies. Starting with computer-based, offline interventions (eg, [15,16]), existing therapies such as CBT, exposure therapy, systematic desensitization, and relaxation were found to be efficacious [17] and were rewritten to suit delivery on the Internet [18,19]. In past years, Internet interventions have been found efficacious for a number of anxiety disorders [17,18,20,21] and phobias, including agoraphobia [22], specific phobias [14,23,24], and social anxiety disorder [25-27]. Thus, Internet-delivered psychological treatments for anxiety and phobias are feasible, acceptable, and effective.

    Typically, outpatients exhibit higher levels of anxiety and a greater number of comorbid and more complex diagnoses, as well as greater psychosocial impairment when compared with general and primary care populations [5]. Previous research has primarily focused on self-referred participants from primary care settings or from the general population [28], and although some evidence exists on the effectiveness of routine psychological interventions in outpatients [29], only a limited number of trials have specifically evaluated Internet-based treatments in outpatient clinics and secondary care for common mental disorders [30-33]. To the best of our knowledge, there appear to be no large-scale high-quality trials evaluating the efficacy of Internet-based exposure therapy in phobic outpatients.

    Because waiting lists are commonplace in outpatient clinics, time spent waiting for face-to-face treatment could be spent effectively by offering a (guided) self-help intervention to patients. Delegating the routine, basic elements of exposure treatment to a guided Internet-based situation could shorten face-to-face therapy and limit therapist involvement, making the treatment more cost-effective [19,34]. Previous research has indicated that Internet-based therapy for social phobia might be cost-effective relative to face-to-face therapy [35,36]. Furthermore, because pretreatment dropout is common in outpatient clinics [37], a second postulated benefit may be that continually engaging the patients in their treatments throughout the wait-list period will result in lower pretreatment attrition or “no shows.”

    The objective of the current trial was to assess the short-term clinical effectiveness of offering Internet-based guided self-help to outpatients compared to a wait-list control. To our knowledge, this is the first large-scale randomized controlled trial of Internet-based treatment for phobias in outpatients. As such, it will also provide valuable information on the acceptability and feasibility of such an intervention in outpatient clinics. This paper describes the principal short-term outcomes of this multifaceted trial.


    Trial Design

    A full trial protocol is available elsewhere [38]. This trial was approved by the Medical Ethics Committee of the VU Medical Centre, Amsterdam (registration number 2010/77) and registered with the Dutch Trial Registry (NTR2233). A total of 481 participants who recently applied for psychological treatment at an outpatient clinic consented to be contacted by our research group and were referred to the researchers from August 2010 to December 2013. After briefing the participants about the aims of the study, screening, and obtaining informed consent in writing, eligible participants (n=212) were administered a telephone baseline questionnaire and participants were randomized to either the intervention group (n=105) or treatment as usual (n=107). Patients who were ineligible (n=111), declined participation, or could not be contacted (n=153) remained on the waiting list for face-to-face treatment. The research did not interfere with the outpatient clinics’ wait-list duration or start of treatment and participants could start face-to-face psychotherapy after the intervention or control group period.

    Participants

    Recruitment Procedure

    A total of 8 specialized anxiety disorder outpatient clinics in medium-to-large cities in the west of the Netherlands participated. Clinics were selected for a high monthly volume of patients for practical reasons. Participants were referred to the outpatient clinics by their general practitioners (GPs), briefly screened, and placed on a waiting list. Recruitment commenced in August 2010 and was stopped in December 2013 to allow for sufficient follow-up time. Waiting lists for outpatient psychotherapy are common in the Netherlands, and time spent on a waiting list is usually at least 6 weeks from first referral to first treatment session. At the start of the wait-list period, participants presenting with a phobia as a primary or secondary disorder were referred to the researchers and screened by telephone using the Composite International Diagnostic Interview (CIDI) [39] for presence of any phobia by master’s level students. Consequently, exclusion criteria were checked and baseline measures were administered. During this wait-list period, a nontherapeutic meeting with a health care professional from the outpatient clinic took place to ascertain treatment needs and to determine optimal face-to-face treatment for all participants. Additional details on recruitment are available elsewhere [38].

    Eligibility Criteria for Participants

    All computer-literate patients with a possible phobia (social phobia, agoraphobia with or without panic disorder, specific phobia) were referred to the researchers by the outpatient clinic even if a phobia was not the primary reason for seeking treatment at an outpatient clinic. Participants had to (1) be 18 years or older, (2) be currently enrolled to receive face-to-face psychotherapy at 1 of the participating outpatient clinics, and (3) have a Diagnostic and Statistical Manual of Mental Disorders (Fourth Edition, Text Revision; DSM-IV-TR) or International Classification of Diseases, Tenth Revision (ICD-10) diagnosis of any phobia as established by the CIDI. Psychotropic medication use was allowed if stable for at least the duration of the intervention or control group period. Patients presenting with psychotic disorders or at elevated risk for suicide were excluded from the trial, but remained on the waiting list for face-to-face psychotherapy at their outpatient clinics.

    Interventions

    Internet-Based Guided Self-Help

    The Internet-based intervention is an adaptation of an existing self-help book on phobias [40]. The intervention is offered at no cost to the participant, takes 5 weeks to complete, and is based on psychoeducation and exposure therapy. The broad and nonspecific focus of the intervention is on identifying and correcting avoidance behavior by using exposure, a common and evidence-based therapeutic component of most phobia therapies [41]. This broad focus facilitates using the intervention for the entire range of phobias. The intervention was presented to the prospective participants as a free-of-charge voluntary course to start reducing their phobic symptoms during the wait-list time. They were told that the intervention was based on evidence-based principles and that the elements they would encounter during the intervention would essentially be the same as in their upcoming face-to-face psychotherapy, allowing for a head start in their treatment. Participants were informed that face-to-face treatment would commence at the scheduled time, regardless of whether they enrolled in the study, and that their decision to participate or not would neither postpone nor advance their face-to-face treatment.

    During the intervention, participants build a hierarchy (see Figure 1) of fear-inducing situations or stimuli and expose themselves to these situations or stimuli gradually. The participant completes exposure exercises as homework assignments and reports on his or her accomplishments to the coach each week. In the first weeks, the participant makes an inventory of his or her avoidance and safety behaviors and defines a focal point for exposure situations and a desired behavioral goal. The participant then plans a number of gradual exposure exercises to be executed for the upcoming week, with exposure exercises becoming gradually more challenging each week (see Figure 2). The coach monitors the fear hierarchy and planning and replies with a supportive message once a week for 5 weeks, relevant to the participant’s homework experiences through the secure online platform. All coaching was supervised by an experienced psychotherapist. The intervention is tunneled (ie, no new material is available to the participant until the participant has reported on that week’s achievements and the coach has provided feedback on these achievements). If applicable, the coach sends a standardized reminder message through the secure online platform if the participant did not use the website that week. All actions on the platform (eg, new feedback received, new exercise available) prompted an immediate automated email to the participant. Material from previous weeks remains accessible to the participant. Online coaching messages were delivered through a secured message system on the intervention website by trained and supervised master’s level students of clinical psychology. The participant completes exposure exercises alone and reports on completed exercises weekly. Throughout the intervention period, the participants were kept on the waiting list for face-to-face psychotherapy.

    The website platform was migrated to an updated version during the recruitment period. This migration was performed to ensure continuing safety of participant data in accordance with Dutch law and to resolve or mitigate critical bugs and shortcomings in website functionality. Website content, however, remained unaltered throughout recruitment. No substantial website downtime was observed during recruitment.

    ‎
    Figure 1. Screenshot of Phobias Under Control: fear hierarchy.
    View this figure
    ‎
    Figure 2. Screenshot of Phobias Under Control: intervention content.
    View this figure
    Waiting List Condition

    Participants in the waiting list group remained on the waiting list for face-to-face psychotherapy. Additionally, to comply with ethics committee regulations and to provide an incentive for enrolling in the trial, control group participants received a self-help book [40] based on exposure therapy, the de facto standard treatment in phobias. This book was sent to the control group participants free of charge with no instructions or support.

    Assessments

    All outcome measures were administered by phone at baseline (T0) and as self-assessment on the Internet at posttest (5 weeks from randomization, T1), using Web-based questionnaire software visibly associated to VU University Amsterdam. This relatively short period was selected to minimize the posttest assessments taking placing during face-to-face psychotherapy if face-to-face psychotherapy should incidentally take place earlier than 6 weeks after inclusion into the trial. To reduce study dropout, intensive reminder emails and telephone reminder calls were used for T1 assessments. Despite several email and telephone reminders, there was considerable variability in follow-up time (mean 50, SD 15.3 days), yet there was no significant difference in follow-up time between the intervention and wait-list control groups. All trial data were stored on a secured network complying with Dutch safety and privacy standards at the time of inclusion and accessible only to research staff. Data were anonymized as soon as possible.

    Outcomes

    Outcome measures are described in more detail elsewhere [38].

    Primary Outcome Measures

    Fear Questionnaire

    The primary outcome measure was the Fear Questionnaire (FQ) [42]. This instrument measures severity of fear and avoidance of phobic stimuli. The psychometric validity of the FQ has been established for the Dutch version [43]. Internal consistency was good, with Cronbach alpha ranging from.78 (blood-injection-injury subscale) to 0.84 (total score).

    Secondary Outcome Measures

    Anxiety

    The Beck Anxiety Inventory (BAI) [44] is a 21-item self-report questionnaire that focuses primarily on physiological manifestations of anxiety. The BAI has been validated for patients with agoraphobia [45] and other anxiety disorders [46]. Internal consistency was excellent (Cronbach α=.92).

    Depressive Symptoms

    The Center for Epidemiological Studies-Depression Scale (CES-D) [47] was administered as a self-rated questionnaire on the Internet. A Dutch version of the CES-D has been validated in an Internet-administrated form [48]. Internal consistency in this sample was good (Cronbach α=.70).

    Process Outcome Measures

    Adherence

    Following a recent definition of intended usage [49], we defined intervention adherence as “the extent to which individuals should experience the content (of the intervention) to derive maximum benefit.” Because some exercises were deemed to have a larger impact on lowering symptom severity (eg, reporting on performing exposure exercises is more beneficial than filling in a readiness to change questionnaire), different weights were assigned to different exercises accordingly, to a total of 20% for each of the 5 weeks. The intended usage was defined as 100% (ie, finishing the 8 exercises the participants were supposed to finish in 5 weeks). The main use metric was having finished an exercise as verified by the coach.

    Treatment Satisfaction

    Satisfaction with the Internet intervention or the self-help book was evaluated using the Client Satisfaction Questionnaire (CSQ-8) [50] which has been validated for use in a Dutch population [51]. Internal consistency was good (Cronbach α=.85). A few free-text items on participant satisfaction and experiences specific to the Web-based intervention were added.

    Sample Size

    To obtain 90% statistical power with a 2-sided alpha equal to .05 and assuming a mean standardized effect size (Cohen’s d) of 0.7 in the intervention group and 0.2 in the control group, we calculated that 170 participants were needed to establish a clinical effect of the Internet intervention compared to wait-list controls. Assuming a dropout rate of 30% at 1-year follow-up, 244 participants should be included.

    Randomization

    A computer-generated randomization table was prepared by a researcher not involved in the data collection (AvS). Randomization was stratified at clinic level and performed at a 1:1 ratio. To ensure approximately equal randomization ratios per clinic, blocks of 8 were used. An external researcher not involved in the project supervised a list of sequentially numbered allocations and assigned participants to the conditions. All project members involved in data collection were unaware of allocation status until randomization was definitive. Participants were enrolled by a master’s level research assistant.

    Blinding

    Due to the nature of this trial, neither participants nor researchers could be blinded to treatment allocation. All outcome measures are self-report questionnaires, which makes blinding unnecessary.

    Statistical Methods

    Data were analyzed with SPSS for Windows, version 20 (IBM Corp, Armonk, NY, USA) according to the intention-to-treat (ITT) principle. Using the multiple imputation function implemented in SPSS 20, we imputed missing data at posttest yielding 50 imputed datasets with 50 iterations each using the multiple imputation option with predictive mean matching. Predictors for the imputing procedure were pretest and (nonmissing) posttest scores, as well as age, clinic, education level, gender, randomization status, and quality of life at pretest. Because SPSS does not automatically calculate pooled statistics for imputed datasets when using ANCOVA, we calculated these statistics by pooling the saved residuals from each imputed dataset and reported mean values and 95% confidence intervals. Between-group effects on the primary outcome measure (FQ) at posttest were calculated with an ANCOVA, with baseline scores of the FQ, BAI, and CES-D entered as a covariate. Within- and between-group effect sizes were reported as Cohen’s d. Effect sizes of d=0.2 are interpreted as small, effect sizes between 0.2 and 0.5 are interpreted as moderate, and effect sizes of 0.8 and upwards are interpreted as large. Due to the large amount of missing data at posttest, data are also presented separately for participants with full follow-up information. No interim analyses were performed. No stopping guidelines were postulated.

    Changes to Protocol

    There were no changes from the published study protocol [38].


    Sample

    See Figure 3 for a flowchart and overview of participants in this trial. Of 481 participants assessed for eligibility, 212 were randomized to either intervention (n=105) or control (n=107). Baseline data are presented in Table 1. Apart from psychotropic medication use, there were no significant differences between intervention and control groups at baseline. Participants were mainly Dutch, female, and highly educated.

    Table 1. Baseline characteristics of participants.
    CharacteristicsTotal sample (N=212)Intervention (n=105)Control (n=107)Pa
    Demographics




    		Age, mean (SD)34.6 (11.7)35.7 (11.7)33.4 (11.6).09

    		Female, n (%)130 (61.0)58 (55)72 (67).10

    		Higher education,b n (%)120 (57.1)58 (56)62 (58).69

    		Disposable income (€),c mean (SD)1524 (761)1545 (727)1503 (796).70

    		Dutch parents, n (%)





    		Both parents Dutch144 (68.6)72 (69)72 (689).98


    		One parent Dutch23 (11)11 (11)12 (11)


    		Neither parent Dutch43 (21)21 (20)22 (21)

    		Psychotropic medication, n (%)43 (20)14 (13%)29 (27).01
    Baseline scores, mean (SD)




    		FQ40.28 (22.71)42.43 (23.41)38.19 (21.93).24

    		BAI44.81 (13.41)45.15 (13.76)44.48 (13.13).81

    		CES-D24.82 (8.47)24.96 (8.61)24.69 (8.37).84
    CIDI phobia diagnosesd




    		Specific phobia, n (%)





    		Animal-type19 (9)9 (9)10 (9).84


    		Nature-type35 (17)21 (20)14 (13).18


    		Blood-injection-injury51 (24)25 (24)26 (24).93


    		Situational-type72 (34)38 (36)34 (32).50


    		Agoraphobia without panic disorder36 (17)18 (17)18 (17).95


    		Agoraphobia with panic disorder87 (41)48 (45)40 (37).28


    		Social phobia113 (53.3)51 (49)62 (58).17

    		Number of phobias, mean (SD)1.95 (1.08)1.99 (1.11)1.91 (1.06).57

    		Number of phobias, n (%)





    		1 phobia91 (43)43 (41)48 (45)


    		2 phobias61 (29)32 (31)29 (27)


    		≥3 phobias60 (28)30 (29)30 (28)

    a Tested with t test or chi-square test as appropriate.

    b Equivalent to a Bachelor’s degree or higher.

    c N=180.

    d Percentages add up to over 100% due to multiple possible diagnoses per participant.

    ‎
    Figure 3. Participant flowchart.
    View this figure

    Study Dropout

    Posttreatment assessments were completed by 126 of 212 (59.2%) participants. Due to database corruption, 2 pretreatment assessments were unavailable. We tested whether there were significant differences in all baseline characteristics described in Table 1 between those who completed the follow-up assessments and those who did not, with dropouts scoring significantly higher on the BAI (t210=2.275, P=.02) and the CES-D (t210=2.489, P=.01), and dropouts being younger in age (t210=2.022, P=.04), less often highly educated (χ2210=8.1, P=.004), and taking psychotropic medication less often at the time of assessment (χ2210=35.4, P<.001). Intervention group participants were also more likely to be nonresponders (67%, 84/126 of completers were control group participants; χ2210=4.3, P=.04). Table 2 shows the statistically significant differences between nonresponders and study completers. The differences of other characteristics described in Table 1 were not statistically significant.

    Table 2. Differences between dropouts and study completers.
    CharacteristicsCompleter (n=126)Dropout (n=84)Pa
    Baseline CES-D score, mean (SD)23.70 (8.34)26.63 (8.39).01
    Baseline BAI score, mean (SD)43.08 (13.54)47.35 (12.96).02
    Age, mean (SD)35.83 (11.82)32.52 (11.35).04
    Higher education, n (%)82 (65%)38 (45%).004
    On psychotropic medication, n (%)43 (34%)0 (0%).<001

    a Tested with t test or chi-square test as appropriate.

    Intervention Adherence and Satisfaction

    Of the 105 participants, 78 (76.4%) started using the intervention, 14 (13.3%) finished week 5, and 9 (8.8%) met the intended usage of 100% (all 8 exercises) in 5 weeks. Average adherence, as expressed as percentage of intended usage, was 37.5% (SD 30.7%); median number of exercises completed (out of a possible 8) was 3 (IQR 4.0). As found in a previous meta-analysis [52], higher education in this sample was associated with higher adherence (F2,103=8.132, P=.005). Intervention participants were moderately positive about their coach (average grade 6.8 of 10, SD 1.06) and indicated that the quality of the feedback messages was satisfactory (10/43, 23%), good (18/43, 42%), or very good to excellent (13/43, 30%). The number of messages received was also evaluated as being balanced (not too many, not too few) by most participants (32/43, 74%). Mean scores for all 8 CSQ-8 items were acceptable (mean 2.78, SD 0.58-0.81; possible item range 1-4).

    Completers and Intention-to-Treat Analyses

    After imputing missing values at posttest and correcting for baseline scores of the FQ, BAI, and CES-D, ANCOVA showed a significant difference in FQ scores between intervention and control groups at posttest (F2,208=6.327, 95% CI 5.977-6.686; P=.02, 95% CI .01-.02; partial η2=.030, 95% CI .03-.03). ANCOVAs also showed a significant difference in CES-D scores between intervention and control groups at posttest (F2,208=6.121, 95% CI 5.550-6.669; P=.03, 95% CI .02-.03; partial η2=.029, 95% CI .03-.03), but no significant difference in BAI scores between intervention and control groups at posttest (F2,208=4.097, 95% CI 3.818-4.376; P=.05, 95% CI .04-.06; partial η2=.020, 95% CI .02-.02). Changes in scores are presented in Table 3 and presented graphically in Figure 4.

    Table 3. Main results, imputed intention-to-treat sample (N=210).
    ResultsPretest, mean (SD)Posttest ITT, mean (SD)ANCOVAaEffect size, d (95% CI)NNTb



    		F2,208PWithin groupBetween group
    Intervention







    		FQ42.02 (23.39)32.52 (18.48)6.33.160.42 (0.31, 0.52)0.35 (0.07, 0.62)5.10

    		CES-D24.99 (8.58)19.40 (5.97)6.12.260.75 (0.51, 0.97)0.34 (0.07, 0.61)5.26

    		BAI45.01 (13.78)42.30 (10.68)4.10.530.22 (0.03, 0.40)0.28 (0.01, 0.55)6.41
    Wait-list control







    		FQ38.34 (21.97)35.56 (18.41)——0.13 (0.03, 0.23)——

    		CES-D24.75 (8.39)21.19 (7.06)——0.46 (0.27, 0.65)——

    		BAI44.57 (13.16)44.52 (10.79)——0.00 (-0.14, 0.15)——

    a Controlled for baseline scores.

    b Number needed to treat.

    After correcting for baseline scores and age (full follow-ups only), we found no significant differences in posttest scores between intervention and control groups for FQ (F2,208=.137, P=.71, partial η2=.001), CES-D (F2,208=2.086, P=.15, partial η2=0.017), or BAI (F2,208=0.333, P=.57, partial η2=0.003).

    ‎
    Figure 4. Change in scores between baseline and imputed posttest scores.
    View this figure

    Adverse Events

    No adverse events were reported or observed during the trial. In a recent trial, a few participants reported temporary adverse effects in an Internet-based intervention for social anxiety disorder [53]; in particular, exacerbation of anxiety symptoms and negative well-being. One control group participant (1/107, 0.5%) reported a worsening of complaints after having started the intervention. However, it is unknown whether this was due to the intervention or to other circumstances not related to this study.


    Principal Findings

    Results show that an Internet-based guided intervention for phobic outpatients can be effective, with modest but significant effect sizes (d=0.28-0.35). The between-group effect sizes were small but significant in the imputed sample. Interestingly, effect sizes were similar for the FQ and CES-D. Because spontaneous recovery is common in depression [54] but not in phobia [5,6], it is unlikely that the decrease in phobic complaints can be attributed to spontaneous recovery rather than to the intervention. The intervention was not targeted specifically at depression; therefore, the significant decrease in depressive symptoms as measured by the CES-D might be attributable to spontaneous recovery rather than the intervention, although due to randomization this spontaneous recovery should probably have occurred in the control condition as well. Another possible explanation is the decrease in phobic complaints in the intervention group led to a commensurate decrease in depressive symptoms. Because the control group participants received a self-help book as compensation for their time invested in completing the baseline and follow-up assessments, there is a possibility that the between-group effect sizes are a more conservative estimate of the real effect size. Additionally, effect sizes for the BAI were lowest of all 3 primary outcome measures. This may be a result of the BAI being concerned mostly with those physical sensations often associated with panic attacks rather than phobias.

    Comparison to Earlier Literature

    Compared with earlier research, recruitment of patients in the outpatient clinics seemed to be reasonably successful. Of the 481 patients assessed for eligibility, of 370 eligible patients we randomized 212 of the 244 participants intended. In all, 153 patients (31.8%) declined to participate in the study (before being screened) and 111 (23.1%) did not meet inclusion criteria. In total, we were able to include 44.1% of all referrals with approximately 2 patients needed to be screened to include a single participant and approximately 3 hours required for each included participant. This does, however, not include the amount of time needed at the outpatient clinic for administrative tasks. In contrast, a recent study of computerized CBT for anxiety in secondary care [55] managed to randomize only 8% (88 of 1141) of referrals. The recruitment percentage also compares rather favorably to trial recruitment in primary care, which can be problematic in terms of duration and numbers of participants recruited as compared to the planned recruitment time and numbers of participants [56]. For example, 1 study focusing on Internet-delivered CBT for depression managed to recruit only 7 patients from 11 general practices in 8 months [57], a marked contrast to our trial. This may be a result of keeping close contact with outpatient staff, although it is difficult to draw comparisons because similar studies with tight integration in routine outpatient clinics are scarce and health care systems differ on national levels. When comparing effect sizes of the current trial with earlier trials of Internet-based anxiety and phobia interventions, we found that the effect sizes of the current trial are low overall [19,58]. In face-to-face psychotherapy, effect sizes of psychological treatment for participants meeting diagnostic criteria were found to be lower than those not meeting the criteria (eg, [13]), but another study found large and sustained effects of Internet-based CBT for panic disorder with and without agoraphobia in routine psychiatric care [33]. Although these previous trials used CBT compared to exposure therapy, it should be noted that the intervention of the current trial included a number of CBT components and that there is no unambiguous evidence that CBT should outperform exposure therapy for phobias per se, with studies finding either no difference or a negligible advantage for either CBT of exposure therapy in phobias (eg, [14,19,59,60]). There is an exception for social anxiety disorder, where cognitive therapies appear to outperform exposure therapy [13,61,62]. It should be noted that just over half of the participants in the current trial were diagnosed with social anxiety disorder, indicating that perhaps greater attention should have been paid to the cognitive therapy elements in the intervention. However, comorbidity of different phobias was large in the current trial, and participants suffering from multiple phobias were encouraged to focus on a single, well-circumscribed area of phobic avoidance at the start of the intervention. Many participants explicitly chose to work on specific phobias or agoraphobia rather than social anxiety, indicating that exposure therapy was indeed the right choice for these participants.

    Overall, few (n=9) participants completed all 8 exercises, and 14 completed all 5 weeks, indicating that some exercises were skipped by the participants. Ending a treatment early may not necessarily be a negative finding [63]. Although not many participants stated a reason for not finishing the intervention, the primary reasons given were a lack of time (8 of 27) and the intervention not being suited to the needs of the participant (5 of 27), both commonly cited barriers to the uptake of online interventions [64]. Despite the high acceptability, as expressed by the low percentage of participants refusing to enter the trial outright, low adherence to this intervention remains a cause for concern because it seems to be lower than generally found in other Internet interventions for anxiety disorders [64]. Possible causes for the lower adherence in this intervention may be that the intervention was too broad, targeting all types of phobias within 1 intervention. Additionally, participants were aware that they would receive face-to-face psychotherapy regardless of whether they finished the online intervention, which may have lowered their motivation to persist. Furthermore, the intervention itself may not have been persuasive enough to encourage repeated use [49]. Earlier efforts to identify predictors of treatment dropout in social anxiety disorder, for example, have not yielded consistent results [65], and future research into the causes of premature treatment termination is needed.

    A particular strength of this study is the high acceptance rate among outpatients as compared with other studies in outpatients and specialized health care centers, which indicates that this sample is clinically relevant and that the results may generalize well across other outpatient samples using similar recruitment strategies. Implementation in routine practice would perhaps facilitate better uptake due to dropping the constraints surrounding research-oriented RCT setting (eg, randomization, filling in extra questionnaires).

    Limitations

    A number of limitations should be taken into account when interpreting the results from this trial. Firstly, the number of participants fell slightly short from the target number of participants (212 randomized versus 244 targeted). Far-reaching cutbacks in Dutch mental health care during the recruitment into this trial resulted in a dwindling number of patients seeking help with outpatient clinics, effectively shrinking the overall participant pool. Secondly, recruitment through outpatient clinics depends on outpatient clinic staff and may be liable to selection bias. Although the included sample of participants seems relatively representative of a clinical sample, selection bias may have occurred during the outpatient clinics’ own selection procedures over which we had no influence. Thirdly, although we corrected for missing values at follow-up by using multiple imputation, the results should be interpreted with caution due to the large amount of missing data. Imputation is an approximation based on a combination of chosen predictors, techniques, and imputing algorithms, which may yield varying results in different datasets [66], making extensive sensitivity analyses time consuming and inconclusive at best. Although some argue that using covariates yields similar results to imputing [67], this may depend heavily on the dataset, and multiple imputation remains the solution of choice for missing data [68]. Regardless of the method for accounting for missing data, the large amount of missing data in this trial is a limitation and means that results should be interpreted with caution. Thirdly, there was considerable variability in the time between baseline assessment and follow-up (median 48 days, range 29-138 days). However, there were no significant correlations between baseline and posttest scores and the time in days between baseline assessment and posttest assessment, which indicates that the posttest scores as used are a representative assessment. Finally, offering the control group participants a self-help book may have influenced the between-group results because wait-list participants using the self-help book may have improved. However, because an improvement in control group participants would mean a smaller contrast between groups, this would lead to a more conservative estimate of treatment effects. Furthermore, it has been proposed that the use of a wait-list group design may not be a representative control group in that it is functionally different from a no-treatment group. It has also been put forward as actually being a “nocebo” group in that waitlisted participants actually do worse than no-treatment participants [69], which would theoretically lead to an inflation of between-group effect sizes. In the light of the current trial, however, both intervention and control groups were scheduled to receive face-to-face psychotherapy; as such, the arguments pertaining to possibly higher (or lower) effect sizes when using a wait-list control group do not necessarily apply to the current trial.

    Implications and Future Research

    In summary, adding an online guided intervention to routine face-to-face treatment may prove beneficial for outpatients, regardless of type of phobia diagnosis. However, effect sizes were markedly lower (d=0.28-0.35) than those found in research on psychological treatment for phobias in the general population (d=0.70-1.84) [12-14] and for anxiety in primary care (d=0.57) [70], and were only found for the imputed ITT sample. Because there was systematic attrition in this trial, the significant differences between completers and dropouts may provide valuable information to identify focal points for targeted attrition reduction, although previous efforts to identify nontechnological factors influencing attrition in online interventions have yielded inconclusive results [71-75]. Given the combination of high acceptability and low adherence, future research should focus on optimizing the usability and persuasive design of this intervention to improve retention and adherence [49] to maximize potential benefits of an intervention that efficiently uses the time spent waiting for face-to-face psychotherapy. Independent replication of the current results in different outpatient settings and countries is needed to verify the findings before robust inferences can be made, but the current results are promising.

    Acknowledgments

    We thank Manja de Neef, MA for coauthoring the intervention and providing professional supervision and additional input and feedback during the trial. We are very grateful for the cooperation of the trial participants and the participating outpatient clinics: PsyQ, GGZ inGeest, Prezens, GGZ Noord-Holland-Noord, and Altrecht. Furthermore, we thank Marleen Bakker, MSc; Julia van Oenen, MSc; and Kim Setkowski, BSc for their invaluable support in participant coaching and assessments, data collection, and administration. This study is funded by ZonMw (project number 80-82310-97-10015). The funding body had no influence in the design, execution, analysis, or interpretation of the results of this study.

    Authors' Contributions

    PC coauthored the intervention. PC and AvS drafted the study protocol. RK authored this paper. AvS, PC, and AB provided feedback and suggestions for this manuscript. All authors read and approved the final manuscript.

    Conflicts of Interest

    AB has received grants for research from Eli Lilly, AstraZeneca, Janssen, and Shire and as a speaker from Lundbeck and Eli Lilly. The other authors have no conflicts of interest to declare.

    ‎

    Multimedia Appendix 1

    CONSORT-EHEALTH checklist V1.6.2 [76].

    PDF File (Adobe PDF File), 982KB
    1. Bijl RV, Ravelli A, van Zessen G. Prevalence of psychiatric disorder in the general population: results of The Netherlands Mental Health Survey and Incidence Study (NEMESIS). Soc Psychiatry Psychiatr Epidemiol 1998 Dec;33(12):587-595. [Medline]
    2. Mendlowicz MV, Stein MB. Quality of life in individuals with anxiety disorders. Am J Psychiatry 2000 May;157(5):669-682. [Medline]
    3. Konnopka A, Leichsenring F, Leibing E, König H. Cost-of-illness studies and cost-effectiveness analyses in anxiety disorders: a systematic review. J Affect Disord 2009 Apr;114(1-3):14-31. [CrossRef] [Medline]
    4. Smit F, Cuijpers P, Oostenbrink J, Batelaan N, de Graaf R, Beekman A. Costs of nine common mental disorders: implications for curative and preventive psychiatry. J Ment Health Policy Econ 2006 Dec;9(4):193-200. [Medline]
    5. Dingemans AE, van Vliet IM, Couvée J, Westenberg HG. Characteristics of patients with social phobia and their treatment in specialized clinics for anxiety disorders in the Netherlands. J Affect Disord 2001 Jul;65(2):123-129. [Medline]
    6. Green AC, Hunt C, Stain HJ. The delay between symptom onset and seeking professional treatment for anxiety and depressive disorders in a rural Australian sample. Soc Psychiatry Psychiatr Epidemiol 2012 Sep;47(9):1475-1487. [CrossRef] [Medline]
    7. Wittchen HU, Beloch E. The impact of social phobia on quality of life. Int Clin Psychopharmacol 1996 Jun;11 Suppl 3:15-23. [Medline]
    8. Zimmerman M, Mattia JI. Principal and additional DSM-IV disorders for which outpatients seek treatment. Psychiatr Serv 2000 Oct;51(10):1299-1304. [Medline]
    9. Dalrymple KL, Zimmerman M. Treatment-seeking for social anxiety disorder in a general outpatient psychiatry setting. Psychiatry Res 2011 May 30;187(3):375-381. [CrossRef] [Medline]
    10. Weiller E, Bisserbe JC, Boyer P, Lepine JP, Lecrubier Y. Social phobia in general health care: an unrecognised undertreated disabling disorder. Br J Psychiatry 1996 Feb;168(2):169-174. [Medline]
    11. Wittchen H. Social phobia: still neglected, poorly recognized and rarely treated? Acta Psychiatr Scand 2003 Sep;108(s417):3-3. [CrossRef]
    12. Sánchez-Meca J, Rosa-Alcázar A, Marín-Martínez F, Gómez-Conesa A. Psychological treatment of panic disorder with or without agoraphobia: a meta-analysis. Clin Psychol Rev 2010 Feb;30(1):37-50. [CrossRef] [Medline]
    13. Acarturk C, Cuijpers P, van Straten A, de Graaf R. Psychological treatment of social anxiety disorder: a meta-analysis. Psychol Med 2009 Feb;39(2):241-254. [CrossRef] [Medline]
    14. Kok RN, Donker T, Batelaan N, Beekman ATF, van Straten A, Cuijpers P. Psychological treatment of specific phobias: a meta-analysis. Clinical Psychology Review 2015:1-1 (forthcoming)(forthcoming).
    15. Marks IM, Kenwright M, McDonough M, Whittaker M, Mataix-Cols D. Saving clinicians' time by delegating routine aspects of therapy to a computer: a randomized controlled trial in phobia/panic disorder. Psychol Med 2004 Jan;34(1):9-17. [Medline]
    16. Carr AC, Ghosh A, Marks IM. Computer-supervised exposure treatment for phobias. Can J Psychiatry 1988 Mar;33(2):112-117. [Medline]
    17. Cuijpers P, Marks IM, van Straten A, Cavanagh K, Gega L, Andersson G. Computer-aided psychotherapy for anxiety disorders: a meta-analytic review. Cogn Behav Ther 2009;38(2):66-82. [CrossRef] [Medline]
    18. Andrews G, Cuijpers P, Craske MG, McEvoy P, Titov N. Computer therapy for the anxiety and depressive disorders is effective, acceptable and practical health care: a meta-analysis. PLoS One 2010;5(10):e13196 [FREE Full text] [CrossRef] [Medline]
    19. Schneider AJ, Mataix-Cols D, Marks IM, Bachofen M. Internet-guided self-help with or without exposure therapy for phobic and panic disorders. Psychother Psychosom 2005;74(3):154-164. [CrossRef] [Medline]
    20. Andersson G. Guided internet treatment for anxiety disorders. As effective as face-to-face therapies? Stud Health Technol Inform 2012;181:3-7. [Medline]
    21. Van't Hof E, Cuijpers P, Stein DJ. Self-help and Internet-guided interventions in depression and anxiety disorders: a systematic review of meta-analyses. CNS Spectr 2009 Feb;14(2 Suppl 3):34-40. [Medline]
    22. Kiropoulos LA, Klein B, Austin DW, Gilson K, Pier C, Mitchell J, et al. Is internet-based CBT for panic disorder and agoraphobia as effective as face-to-face CBT? J Anxiety Disord 2008 Dec;22(8):1273-1284. [CrossRef] [Medline]
    23. Andersson G, Waara J, Jonsson U, Malmaeus F, Carlbring P, Ost LG. Internet-based self-help versus one-session exposure in the treatment of spider phobia: a randomized controlled trial. Cogn Behav Ther 2009;38(2):114-120. [CrossRef] [Medline]
    24. Müller B, Kull S, Wilhelm FH, Michael T. One-session computer-based exposure treatment for spider-fearful individuals--efficacy of a minimal self-help intervention in a randomised controlled trial. J Behav Ther Exp Psychiatry 2011 Jun;42(2):179-184. [CrossRef] [Medline]
    25. Berger T, Caspar F, Richardson R, Kneubühler B, Sutter D, Andersson G. Internet-based treatment of social phobia: a randomized controlled trial comparing unguided with two types of guided self-help. Behav Res Ther 2011 Mar;49(3):158-169. [CrossRef] [Medline]
    26. Carlbring P, Nordgren LB, Furmark T, Andersson G. Long-term outcome of Internet-delivered cognitive-behavioural therapy for social phobia: a 30-month follow-up. Behav Res Ther 2009 Oct;47(10):848-850. [CrossRef] [Medline]
    27. Hedman E, El Alaoui S, Lindefors N, Andersson E, Rück C, Ghaderi A, et al. Clinical effectiveness and cost-effectiveness of Internet- vs. group-based cognitive behavior therapy for social anxiety disorder: 4-year follow-up of a randomized trial. Behav Res Ther 2014 Aug;59:20-29. [CrossRef] [Medline]
    28. Proudfoot J, Ryden C, Everitt B, Shapiro DA, Goldberg D, Mann A, et al. Clinical efficacy of computerised cognitive-behavioural therapy for anxiety and depression in primary care: randomised controlled trial. Br J Psychiatry 2004 Jul;185:46-54 [FREE Full text] [Medline]
    29. Hans E, Hiller W. A meta-analysis of nonrandomized effectiveness studies on outpatient cognitive behavioral therapy for adult anxiety disorders. Clin Psychol Rev 2013 Dec;33(8):954-964. [CrossRef] [Medline]
    30. Kenter RM, van Straten A, Hobbel SH, Smit F, Bosmans J, Beekman A, et al. Effectiveness and cost effectiveness of guided online treatment for patients with major depressive disorder on a waiting list for psychotherapy: study protocol of a randomized controlled trial. Trials 2013;14:412 [FREE Full text] [CrossRef] [Medline]
    31. Bell CJ, Colhoun HC, Carter FA, Frampton CM. Effectiveness of computerised cognitive behaviour therapy for anxiety disorders in secondary care. Aust N Z J Psychiatry 2012 Jul;46(7):630-640. [CrossRef] [Medline]
    32. Moessner M, Bauer S, Ozer F, Wolf M, Zimmer B, Kordy H. Cost-effectiveness of an internet-based aftercare intervention after inpatient treatment in a psychosomatic hospital. Psychother Res 2014 Jul;24(4):496-503. [CrossRef] [Medline]
    33. Hedman E, Ljótsson B, Rück C, Bergström J, Andersson G, Kaldo V, et al. Effectiveness of internet-based cognitive behaviour therapy for panic disorder in routine psychiatric care. Acta Psychiatr Scand 2013 Dec;128(6):457-467. [CrossRef] [Medline]
    34. Marks IM, Kenwright M, McDonough M, Whittaker M, Mataix-Cols D. Saving clinicians' time by delegating routine aspects of therapy to a computer: a randomized controlled trial in phobia/panic disorder. Psychol Med 2004 Jan;34(1):9-17. [Medline]
    35. Hedman E, Andersson E, Ljótsson B, Andersson G, Rück C, Lindefors N. Cost-effectiveness of Internet-based cognitive behavior therapy vs. cognitive behavioral group therapy for social anxiety disorder: results from a randomized controlled trial. Behav Res Ther 2011 Nov;49(11):729-736. [CrossRef] [Medline]
    36. Titov N, Andrews G, Johnston L, Schwencke G, Choi I. Shyness programme: longer term benefits, cost-effectiveness, and acceptability. Aust N Z J Psychiatry 2009 Jan;43(1):36-44. [CrossRef] [Medline]
    37. Issakidis C, Andrews G. Pretreatment attrition and dropout in an outpatient clinic for anxiety disorders. Acta Psychiatr Scand 2004 Jun;109(6):426-433. [CrossRef] [Medline]
    38. Kok RN, van Straten A, Beekman A, Bosmans J, de Neef M, Cuijpers P. Effectiveness and cost-effectiveness of web-based treatment for phobic outpatients on a waiting list for psychotherapy: protocol of a randomised controlled trial. BMC Psychiatry 2012;12:131 [FREE Full text] [CrossRef] [Medline]
    39. World Health Organization. Composite International Diagnostic Interview (CIDI). Geneva: World Health Organization; 1990.
    40. De Neef M, Cuijpers P. Fobieën. Amsterdam: Boom Uitgeverij; 2007.
    41. Deacon BJ, Abramowitz JS. Cognitive and behavioral treatments for anxiety disorders: a review of meta-analytic findings. J Clin Psychol 2004 Apr;60(4):429-441. [CrossRef] [Medline]
    42. Marks IM, Mathews AM. Brief standard self-rating for phobic patients. Behav Res Ther 1979;17(3):263-267. [Medline]
    43. Van Zuuren FJ. The fear questionnaire. Some data on validity, reliability and layout. Br J Psychiatry 1988 Nov;153:659-662. [Medline]
    44. Beck AT, Steer RA. Beck Anxiety Inventory Manual. San Antonio, TX: Psychological Corperation; 1993.
    45. de Beurs E, Wilson KA, Chambless DL, Goldstein AJ, Feske U. Convergent and divergent validity of the Beck Anxiety Inventory for patients with panic disorder and agoraphobia. Depress Anxiety 1997;6(4):140-146. [Medline]
    46. Leyfer OT, Ruberg JL, Woodruff-Borden J. Examination of the utility of the Beck Anxiety Inventory and its factors as a screener for anxiety disorders. J Anxiety Disord 2006;20(4):444-458. [CrossRef] [Medline]
    47. Radloff LS. The CES-D Scale: A Self-Report Depression Scale for Research in the General Population. Applied Psychological Measurement 1977 Jun 01;1(3):385-401. [CrossRef]
    48. Donker T, van Straten A, Marks I, Cuijpers P. Brief self-rated screening for depression on the Internet. J Affect Disord 2010 May;122(3):253-259. [CrossRef] [Medline]
    49. Kelders SM, Kok RN, Ossebaard HC, Van Gemert-Pijnen JE. Persuasive system design does matter: a systematic review of adherence to web-based interventions. J Med Internet Res 2012;14(6):e152 [FREE Full text] [CrossRef] [Medline]
    50. Nguyen TD, Attkisson CC, Stegner BL. Assessment of patient satisfaction: development and refinement of a service evaluation questionnaire. Eval Program Plann 1983;6(3-4):299-313. [Medline]
    51. de Brey H. A cross-national validation of the client satisfaction questionnaire: the Dutch experience. Eval Program Plann 1983;6(3-4):395-400. [Medline]
    52. Swift JK, Greenberg RP. Premature discontinuation in adult psychotherapy: a meta-analysis. J Consult Clin Psychol 2012 Aug;80(4):547-559. [CrossRef] [Medline]
    53. Boettcher J, Rozental A, Andersson G, Carlbring P. Side effects in Internet-based interventions for Social Anxiety Disorder. Internet Interventions 2014 Mar;1(1):3-11. [CrossRef]
    54. Spijker J, de Graaf R, Bijl RV, Beekman AT, Ormel J, Nolen WA. Duration of major depressive episodes in the general population: results from The Netherlands Mental Health Survey and Incidence Study (NEMESIS). Br J Psychiatry 2002 Sep;181:208-213 [FREE Full text] [Medline]
    55. Carter FA, Bell CJ, Colhoun HC. Suitability and acceptability of computerised cognitive behaviour therapy for anxiety disorders in secondary care. Aust N Z J Psychiatry 2013 Feb;47(2):142-152. [CrossRef] [Medline]
    56. Bower P, Wilson S, Mathers N. Short report: how often do UK primary care trials face recruitment delays? Fam Pract 2007 Dec;24(6):601-603 [FREE Full text] [CrossRef] [Medline]
    57. Woodford J, Farrand P, Bessant M, Williams C. Recruitment into a guided internet based CBT (iCBT) intervention for depression: lesson learnt from the failure of a prevalence recruitment strategy. Contemp Clin Trials 2011 Sep;32(5):641-648. [CrossRef] [Medline]
    58. Reger MA, Gahm GA. A meta-analysis of the effects of internet- and computer-based cognitive-behavioral treatments for anxiety. J Clin Psychol 2009 Jan;65(1):53-75. [CrossRef] [Medline]
    59. Ost LG, Alm T, Brandberg M, Breitholtz E. One vs five sessions of exposure and five sessions of cognitive therapy in the treatment of claustrophobia. Behav Res Ther 2001 Feb;39(2):167-183. [Medline]
    60. Bornas X, Tortella-Feliu M, Llabrés J. Do all treatments work for flight phobia? Computer-assisted exposure versus a brief multicomponent nonexposure treatment. Psychotherapy Research 2006 Jan;16(1):41-50. [CrossRef]
    61. Clark DM, Ehlers A, Hackmann A, McManus F, Fennell M, Grey N, et al. Cognitive therapy versus exposure and applied relaxation in social phobia: A randomized controlled trial. J Consult Clin Psychol 2006 Jun;74(3):568-578. [CrossRef] [Medline]
    62. Ougrin D. Efficacy of exposure versus cognitive therapy in anxiety disorders: systematic review and meta-analysis. BMC Psychiatry 2011;11:200 [FREE Full text] [CrossRef] [Medline]
    63. Hilvert-Bruce Z, Rossouw PJ, Wong N, Sunderland M, Andrews G. Adherence as a determinant of effectiveness of internet cognitive behavioural therapy for anxiety and depressive disorders. Behav Res Ther 2012 Aug;50(7-8):463-468. [CrossRef] [Medline]
    64. Christensen H, Griffiths KM, Farrer L. Adherence in internet interventions for anxiety and depression. J Med Internet Res 2009;11(2):e13 [FREE Full text] [CrossRef] [Medline]
    65. Eskildsen A, Hougaard E, Rosenberg NK. Pre-treatment patient variables as predictors of drop-out and treatment outcome in cognitive behavioural therapy for social phobia: A systematic review. Nord J Psychiatry 2010 Apr;64(2):94-105. [CrossRef] [Medline]
    66. Blankers M, Koeter MW, Schippers GM. Missing data approaches in eHealth research: simulation study and a tutorial for nonmathematically inclined researchers. J Med Internet Res 2010;12(5):e54 [FREE Full text] [CrossRef] [Medline]
    67. Groenwold RH, Donders AR, Roes KC, Harrell FE, Moons KG. Dealing with missing outcome data in randomized trials and observational studies. Am J Epidemiol 2012 Feb 1;175(3):210-217 [FREE Full text] [CrossRef] [Medline]
    68. Liublinska V, Rubin DB. Re: "dealing with missing outcome data in randomized trials and observational studies". Am J Epidemiol 2012 Aug 15;176(4):357-8; author reply 358 [FREE Full text] [CrossRef] [Medline]
    69. Furukawa TA, Noma H, Caldwell DM, Honyashiki M, Shinohara K, Imai H, et al. Waiting list may be a nocebo condition in psychotherapy trials: a contribution from network meta-analysis. Acta Psychiatr Scand 2014 Sep;130(3):181-192. [CrossRef] [Medline]
    70. Seekles W, Cuijpers P, Kok R, Beekman A, van Marwijk H, van Straten A. Psychological treatment of anxiety in primary care: a meta-analysis. Psychol Med 2013 Feb;43(2):351-361. [CrossRef] [Medline]
    71. Blankers M, Koeter MW, Schippers GM. Baseline predictors of treatment outcome in Internet-based alcohol interventions: a recursive partitioning analysis alongside a randomized trial. BMC Public Health 2013;13:455 [FREE Full text] [CrossRef] [Medline]
    72. Wojtowicz M, Day V, McGrath PJ. Predictors of participant retention in a guided online self-help program for university students: prospective cohort study. J Med Internet Res 2013;15(5):e96 [FREE Full text] [CrossRef] [Medline]
    73. Donkin L, Glozier N. Motivators and motivations to persist with online psychological interventions: a qualitative study of treatment completers. J Med Internet Res 2012;14(3):e91 [FREE Full text] [CrossRef] [Medline]
    74. Karyotaki E, Smit F, Kleiboer A, Mira Pastor A, Berger T, Cristina B. Predictors of treatment dropout in self-guided web-based interventions for depression: An individual patient data meta-analysis. Psychol Med 2015:1-1 (forthcoming)(forthcoming).
    75. Al-Asadi AM, Klein B, Meyer D. Pretreatment attrition and formal withdrawal during treatment and their predictors: an exploratory study of the anxiety online data. J Med Internet Res 2014;16(6):e152 [FREE Full text] [CrossRef] [Medline]
    76. Eysenbach G, CONSORT-EHEALTH Group. CONSORT-EHEALTH: improving and standardizing evaluation reports of Web-based and mobile health interventions. J Med Internet Res 2011;13(4):e126 [FREE Full text] [CrossRef] [Medline]

    ‎
    BAI: Beck Anxiety Inventory
    CBT: cognitive behavioral therapy
    CIDI: Composite International Diagnostic Interview
    FQ: Fear Questionnaire
    GP: general practitioner
    IQR: Interquartile range
    ITT: intention-to-treat
    NNT: number needed to treat
    RCT: randomized controlled trial

    Edited by G Eysenbach; submitted 26.03.14; peer-reviewed by T Berger, C Lewis; comments to author 24.06.14; revised version received 02.07.14; accepted 17.07.14; published 29.09.14

    Copyright

    ©Robin N Kok, Annemieke van Straten, Aartjan T F Beekman, Pim Cuijpers. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 29.09.2014.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.