This paper is in the following e-collection/theme issue:

Digital Mental Health Interventions, e-Mental Health and Cyberpsychology (1514) Anxiety and Stress Disorders (1053) Depression and Mood Disorders; Suicide Prevention (1363)

Published on in Vol 21, No 4 (2019): April

Internet-Delivered Cognitive Behavioral Therapy for Anxiety Disorders in Open Community Versus Clinical Service Recruitment: Meta-Analysis

Internet-Delivered Cognitive Behavioral Therapy for Anxiety Disorders in Open Community Versus Clinical Service Recruitment: Meta-Analysis

Internet-Delivered Cognitive Behavioral Therapy for Anxiety Disorders in Open Community Versus Clinical Service Recruitment: Meta-Analysis

Authors of this article:

Geke Romijn1, 2 Author Orcid Image ;   Neeltje Batelaan3 Author Orcid Image ;   Robin Kok1, 4, 5 Author Orcid Image ;   Jeroen Koning6 Author Orcid Image ;   Anton van Balkom3 Author Orcid Image ;   Nickolai Titov7 Author Orcid Image ;   Heleen Riper1, 8 Author Orcid Image
Article Authors Cited by (34) Tweetations (25) Metrics

    Review

    1Department of Clinical, Neuro and Developmental Psychology, VU University, Amsterdam Public Health Research Institute, Amsterdam, Netherlands

    2Altrecht Academic Anxiety Centre, Utrecht, Netherlands

    3Department of Psychiatry, VU University Medical Centre, Amsterdam Public Health Research Institute, Amsterdam, Netherlands

    4Department of Psychology, University of Southern Denmark, Odense, Denmark

    5Centre for Innovative Medical Technology, Odense University Hospital, Odense, Denmark

    6Pro Persona Psychiatric Centre, Tiel, Netherlands

    7Mindspot, Department of Psychology, Macquarie University, Sydney, Australia

    8Centre for Telepsychiatry, Mental Health Services of Southern Denmark, Odense, Denmark

    Corresponding Author:

    Geke Romijn, MSc

    Department of Clinical, Neuro and Developmental Psychology

    VU University

    Amsterdam Public Health Research Institute

    Van der Boechorstraat 7

    Amsterdam,

    Netherlands

    Phone: 31 20 59 89009

    Fax:31 20 59 88758

    Email: g.a.romijn@vu.nl


    Background: Ample studies have shown the effectiveness of internet-delivered cognitive behavioral therapy (iCBT) for anxiety disorders. These studies recruited their participants mainly from the community and, to a lesser extent, from within routine care services. Little is known about whether different recruitment strategies lead to different treatment effects.

    Objective: This meta-analysis compared clinical results obtained in trials with recruitment from the community versus results obtained in trials with clinical service recruitment and explored factors that may mediate differences in treatment outcome.

    Methods: We included randomized controlled trials in which the clinical effects of iCBT for anxiety disorders were compared with a control condition (waitlist controls or face-to-face cognitive behavioral therapy). We classified trials as open recruitment trials (recruitment from the community) or clinical service recruitment trials (recruitment through outpatient clinics). Pooled effect sizes based on measures examining anxiety symptoms, depressive symptoms, and quality of life were computed for each type of trial. Subgroup analyses examined whether clinical results from open recruitment trials differed from those obtained in clinical service recruitment trials. Additional analyses explored which demographic, clinical, and treatment-related factors contributed to differences in effect sizes of open recruitment versus clinical service recruitment trials.

    Results: We included 42 studies with 53 comparisons (43 open recruitment comparisons and 10 clinical recruitment comparisons). Analyses of anxiety measures revealed, first, that iCBT open recruitment studies with waitlist control comparators showed a significantly higher effect size for decrease in anxiety symptoms than did those with clinical recruitment (Q=10.09; P=.001). This association between recruitment method and effect size was no longer significant in a multivariate metaregression with treatment adherence and exclusion of patients with depressive symptoms entered as additional predictors of effect size. Second, effect size for decrease in anxiety symptoms did not differ significantly between clinical recruitment and open recruitment studies with face-to-face cognitive behavioral therapy comparators. The effects of open recruitment trials and clinical recruitment trials did not differ significantly for the secondary outcomes, compared with face-to-face cognitive behavioral therapy and waitlist controls.

    Conclusions: iCBT was effective in samples recruited in clinical practice, but effect sizes were smaller than those found in trials with an open recruitment method for studies with waitlist control comparators. Hence, for patients with anxiety disorders in routine care, the impact of iCBT may not be as positive as for study participants recruited from the community. The difference between open recruitment trials and clinical service recruitment trials might be partly explained by patients’ greater therapy adherence in open recruitment trials and the stricter exclusion of patients with severe depressive symptoms in these studies. Since most trials in this meta-analysis applied an open recruitment method, more studies with routine care populations are needed to further validate these findings.

    J Med Internet Res 2019;21(4):e11706

    doi:10.2196/11706

    Keywords

    anxiety disorders; cognitive behavioral therapy; internet; recruitment method; efficacy; effectiveness 


    Background

    Internet-delivered cognitive behavioral therapy (iCBT) for anxiety disorders has been tested in ample randomized controlled trials and several meta-analyses. These studies show the potential of iCBT to reduce anxiety symptoms among patients and general populations, indicating that iCBT is effective when compared with a waitlist control (WLC), with effect sizes in the moderate to large range. Studies also suggest that it is as effective as face-to-face cognitive behavioral therapy (CBT) in improving symptoms of anxiety [1-5], although these studies are limited in number. Furthermore, iCBT may minimize treatment barriers such as high costs due to reduced time needed by therapists to provide therapy [6,7] and scalability.

    The majority of trials on iCBT apply an open recruitment (OR) strategy, inviting individuals with anxiety symptoms from within the community to directly partake in the research study. These participants refer themselves to such a study. Often these studies apply strict inclusion and exclusion criteria; for example, they may exclude patients taking psychoactive medication, patients with comorbid disorders, or severely depressed patients [8,9]. Clinical service recruitment (CSR) trials, on the other hand, invite patients already seeking treatment in clinical practices to participate. Trials with an OR method provide evidence more related to efficacy (investigating whether a treatment works under ideal circumstances, with high internal validity), as opposed to CSR trials that are more related to effectiveness and provide information on whether a treatment works in clinically representative conditions [10].

    Only a minority of the patient samples in trials with an OR method correspond to patient populations in a regular clinical setting in terms of sociodemographic characteristics, motivation for treatment, level of suffering, and clinical characteristics such as severity of anxiety, comorbidity, or medical history [11-13]. Furthermore, the use of extensive exclusion criteria in OR trials can reduce the degree to which these study samples resemble clinical populations in routine care settings. A meta-analysis [14] found a strong and positive relationship (r=.70) between the number of exclusion criteria and the rate of clinically improved participants for studies on iCBT for anxiety disorders. These results suggest a lower clinical effectiveness in clinically representative studies than in highly controlled studies. This raises the question whether results from OR trials can be extrapolated to routine clinical practice.

    On the other hand, uncontrolled effectiveness studies show large clinical effects [15-20], thereby suggesting that iCBT for anxiety disorders may be as effective in routine care settings as demonstrated in efficacy trials. One review investigated controlled research of iCBT in routine clinical practice [21]. Results showed that effect sizes obtained from effectiveness studies (ranging from 0.75 to 1.73) were in the same range as those obtained in efficacy trials, though only 3 randomized controlled trials were included.

    Objective

    Although several meta-analyses for (internet-based treatment of) anxiety disorders have been conducted in recent years, to our knowledge, none of these studies have compared the potential differences in clinical effectiveness between OR and CSR trials. In this study, we aimed to (1) assess whether OR trials produced clinical effectiveness for anxiety symptoms similar to that of CSR trials and (2) explore predictors of potential effect differences, such as demographic, clinical, and treatment-related characteristics. We based these predictors on differences between OR and CSR trials in patient samples and methods found in previous studies [11,13,14].


    Study Retrieval

    We report this meta-analysis in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines [22]. We retrieved studies through systematic literature searches in PubMed, PsycINFO, and EMBASE databases. Searches were conducted with keywords and text words, in which words indicative of internet treatment were combined with words indicative of anxiety disorder, CBT, and randomized controlled trial (see Multimedia Appendix 1 for the full search string). Furthermore, we checked reference lists of retrieved articles and of earlier reviews on iCBT for anxiety disorders [1-4].

    Inclusion and Exclusion Criteria

    We included randomized controlled trials published up to and including December 2017 on guided and unguided iCBT and blended CBT for adults. Blended CBT combines face-to-face treatment with internet components into a single integrated treatment protocol [23]. We included only randomized controlled trials that assessed a primary diagnosis of an anxiety disorder according to the Diagnostic and Statistical Manual of Mental Disorders (Third Edition Revised [DSM-III-R], DSM (Fourth Edition [DSM-IV]), or DSM (Fifth Edition [DSM-5]) established by a structured diagnostic interview. We excluded studies on obsessive compulsive disorder [24] and posttraumatic stress disorder [25], since they are not classified as anxiety disorders in DSM-5. We included only randomized controlled trials published in English or Dutch.

    Interventions and Comparators

    We considered interventions to be CBT if they were based on cognitive behavioral principles [26] and consisted at least of cognitive restructuring or exposure (interoceptive exposure or exposure in vivo), or a combination of both. To be considered iCBT, the intervention must have been delivered (partly) via a computer or the internet through the use of webpages or email, or both. We included studies on iCBT targeting anxiety disorders and studies on transdiagnostic iCBT [27-31], addressing multiple anxiety disorders or addressing both anxiety and mood disorders, but only if participants had a diagnosis of an anxiety disorder and measures of anxiety were reported. We did not include interventions when the Web-based part of the treatment was limited to exposure scenes on a screen (eg, Heading et al [32]) because we considered this to be in virtuo exposure treatment, which is beyond the scope of this review.

    iCBT was compared with WLC or regular face-to-face CBT treatment (including individual or group CBT delivered in a face-to-face format). We excluded studies with other comparisons such as transdiagnostic iCBT compared with disorder-specific iCBT, or guided iCBT versus self-help [24,25,33,34].

    Outcome Measures

    Our primary outcome was anxiety symptom severity based on the score on a rating scale used to measure general symptoms of anxiety. We applied a hierarchy of preferred outcomes for all measures based on frequency of use in the included trials. For general measures of anxiety, the preferred order was as follows: Beck Anxiety Inventory (BAI [35]), anxiety scale of the Depression Anxiety Stress Scales [36], State-Trait Anxiety Inventory [37], and Anxiety Sensitivity Index [38]. When a general measure of anxiety was not available, we used a measure for specific anxiety symptoms (see Multimedia Appendix 2 [39-49] for the order of rating scales and Multimedia Appendix 3 [7,27-31,50-85] for the outcome measures we used for all studies).

    Since anxiety disorders are frequently accompanied by symptoms of depression and a reduced quality of life [86], secondary outcome measures were effects on depression severity and on quality of life (see Multimedia Appendix 2 for the order of rating scales [36,87-94]).

    Criteria for Open Recruitment and Clinical Service Recruitment

    We classified trials as OR trials if participants were recruited from the community and referred themselves to be interested in the study in response to the invitation from a research team, by means of advertisements in newspapers or magazines, banners on websites, or large-scale mailings. In CSR trials, recruitment was carried out among patients already seeking treatment in outpatient clinical mental health practices. In the case of mixed recruitment strategies, we classified trials according to the most prominent recruitment strategy.

    Study Selection and Data Extraction

    Two of the authors (GR and NB) independently screened the list of titles and abstracts that resulted from the literature search. Reference lists were screened for additional studies of relevance. We obtained full articles for potentially relevant abstracts according to the inclusion criteria. If included trials did not provide complete information, we contacted the primary investigator by email to attempt to obtain unreported data. We sent a second email when we received no response. Two researchers (GR and NB) extracted the data using Excel (2013) spreadsheets (Microsoft Corporation) and differences in such data were resolved by discussion. Extracted data (see Multimedia Appendix 3) included the study characteristics outlined in Textbox 1.

    Study characteristics extracted from the articles.
    • Year of publication
    • Number of participants
    • Recruitment setting (open or clinical service recruitment)
    • Demographic characteristics of participants included in the study (sex, age, employment status [total rate of employed participants and rate of full-time employed participants], education level [rate of participants with college degree or higher])
    • Anxiety severity at baseline
    • Axis I comorbidity rate
    • Exclusion criteria with regard to medication use (benzodiazepines and other psychoactive medication) and depressive symptoms (indicated by a score above a cutoff level on an outcome measure for depressive symptoms)
    • Details of treatment conditions (duration and type of support provided by professionals)
    • Outcome data
    • Number of therapy sessions according to protocol, treatment dose (number of completed therapy sessions), and treatment adherence (number of completed sessions divided by the total number of sessions according to protocol)
    Textbox 1. Study characteristics extracted from the articles.

    Risk-of-Bias Assessment

    Two authors (GR and RK) independently assessed the risk of bias in the included studies based on 6 areas according to the Cochrane tool for assessing risk of bias [95]: (1) adequate generation of allocation sequence, (2) concealment of allocation to conditions (concealing allocation sequence from participants and investigators), (3) blinding of participants and personnel, (4) blinding of outcome assessors, (5) dealing with incomplete outcome data, and (6) selective outcome reporting (reported results give reason to suspect differences between reported and unreported findings). Because RK was an author of one of the included studies [50], this study was independently assessed by a third reviewer (NB). Discrepancies in scoring were resolved through discussion.

    We assessed all areas as low, high, or unclear (ie, not enough information) risk of bias (see Multimedia Appendix 4). We assessed selective outcome reporting by comparing trial registrations with published articles, if available. When primary or secondary outcomes were missing, inserted, or changed in the article compared with the trial registration, or if secondary and primary outcomes had been switched, we deemed a study to be at high risk of selective outcome reporting. If no trial registration was available for a study, we coded the study as being at unknown risk for selective outcome reporting.

    Statistical Analyses

    We used descriptive statistics to summarize demographic characteristics, and clinical and treatment-related characteristics of OR and CSR trials. We compared categorical variables using chi-square tests and continuous variables using t tests. We also compared the percentage of at-risk OR and CSR trials for all risk-of-bias indicators with chi-square tests.

    We then calculated the pooled overall effect sizes (Hedges g) indicating the difference between the conditions at posttest and their 95% confidence intervals using the random-effects model with Comprehensive Meta-Analysis software version 3.0 (Biostat). Hedges g is an effect size that corrects for biases due to small sample sizes [96]. Effect sizes of 0.2, 0.5, and 0.8 indicate a small, moderate, and large effect size, respectively [97]. We used the effect sizes based on intent-to-treat analysis when available (in 51 comparisons); otherwise, we used complete-sample analysis results (in 2 comparisons [51,52]).

    We examined heterogeneity among studies using Higgins I2 statistic. I2=0% reflects no heterogeneity; 25%, 50%, and 75% indicate a low, medium, and high level of heterogeneity, respectively [98]. A higher observed statistical heterogeneity indicates a higher proportion of observed variance, which can point to underlying differences between the pooled studies. This makes interpreting the pooled effect size difficult, as it is hard to distinguish the observed effect size from the true population effect size [99]. We also calculated 95% confidence intervals around I2 with the noncentral chi-square approach in the heterogi module for Stata 13.0SE (StataCorp LLC) [100].

    We first calculated overall effect sizes on anxiety, depression, and quality-of-life treatment outcomes of iCBT compared with WLC and compared with face-to-face CBT. We then carried out subgroup analyses to assess whether clinical results from OR trials differed from those obtained in CSR trials. We performed these subgroup analyses according to the mixed-effects model, in which studies within subgroups are pooled with the random-effects model, and the fixed-effects model is used to test for significant differences among them by the between-subgroups Q-statistic [101].

    We tested publication bias by inspecting the funnel plot and Egger test [102] on our primary outcome measure and by the Duval and Tweedie trim-and-fill procedure [103].

    To more fully understand differences in effect size between OR and CSR trials, we conducted additional exploratory analyses. By means of subgroup and metaregression analyses, we examined which demographic, clinical, and treatment-related factors differed between OR and CSR trials and were associated with effect size on the primary outcome. Next, to examine whether these predictors contributed to the difference of effect size between OR and CSR trials, we tested recruitment method and the significant predictors in a multivariate model, except in case of collinearity. We assessed possible collinearity problems between predictors with the variance inflation factors. We considered variance inflation factor scores higher than 2.5 to indicate multicollinearity [104].

    We also calculated the number needed to treat (NNT), according to Kraemer and Kupfer [105], and rounded upward to the next higher whole number [106]. The NNT gives some clinical context to statistical information, as it translates the magnitude of a statistical effect size into clinical implications—that is, the number of patients who must be treated to generate one more positive outcome than the same number of patients in the control group.


    Study Inclusion

    The literature searches retrieved a total of 3954 abstracts. Checking references of earlier reviews resulted in 7 more citations for consideration. After we removed duplicates, we screened 2808 abstracts. After screening abstracts, we retrieved 134 full-text articles for a more detailed evaluation of eligibility. Subsequently, we excluded 92 articles because they did not meet the inclusion criteria (Figure 1). We did not include 1 study because means and standard deviations for anxiety measures were not reported [107] and we received no response from the addressed researchers to our email questions regarding these issues.

    ‎
    Figure 1. Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flowchart of the selection and inclusion process. RCT: randomized controlled trial.
    View this figure

    Description of Included Studies

    We included a total of 42 trials (Multimedia Appendix 3). These trials entailed 53 comparisons of anxiety outcomes between iCBT and a control condition (WLC n=41, face-to-face CBT n=12) and included 3714 participants. A total of 45 comparisons entailed outcomes on depression and 21, on quality of life. Most studies were OR trials (31 trials with 41 comparisons), versus 8 CSR trials with 9 comparisons. In 3 studies both recruitment strategies were used [31,51,53]. In 2 of these studies, most participants were self-referred (n=129, 92.8% [31] and n=70, 77% [53]) and therefore we classified these studies as OR trials. In the third study, most participants were recruited through a clinical procedure (n=76, 67%) and therefore we classified this study as a CSR trial [51]. Thus, we classified a total of 33 trials with 43 comparisons as OR trials and 9 trials with 10 comparisons as CSR trials.

    iCBT typically consisted of weekly sessions (ranging from 4 to 12 sessions for studies comparing iCBT with WLC and from 4 to 23 sessions for studies comparing iCBT with face-to-race CBT), with durations ranging from 4 weeks to longer than 3 months [54]. In 4 trials a self-help iCBT intervention (unguided) was offered; in the other trials iCBT was guided, meaning that online professional support was provided.

    Table 1 displays demographic, clinical, and treatment-related characteristics of OR and CSR trials. We found significant differences between OR trials and CSR trials with regard to age of participants, sex, baseline severity, exclusion of severely depressed patients, treatment dose, and treatment adherence. We could compare baseline severity only for scores based on the BAI and the Social Phobia Scale, since these were the only outcome measures that were reported in both OR trials and CSR trials. We did not classify 3 OR trials [55-57] and 1 CSR trial [51] that mentioned exclusion of patients with depressive symptoms as such, because no definition in terms of a score on a measurement for depressive symptoms was provided.

    Table 1. Comparison of demographic, clinical, and treatment-related characteristics of included open recruitment studies and clinical service recruitment studiesa.
    CharacteristicsOpen recruitmentClinical recruitmentTest statisticP value
    Demographic characteristics

    		Age (years), mean (SD)37.5 (5.4)34.1 (1.7)t3657=29.1<.001

    		Female sex, n (%)1871 (67)541 (58.7)χ21=21.2<.001

    		Education: college degree, n (%)670 (53.4)171 (49.4)χ21=1.7.19

    		Employed: full-time or part-time, n (%)390 (58.7)279 (59.7)χ21=0.1.71
    Clinical characteristics

    		Primary diagnosis: trials that applied this diagnosis as inclusion criterion, n (%)
    		χ24=4.3.37


    		Panic disorder13 (30.2)3 (30)



    		Social anxiety disorder14 (32.6)3 (30)



    		Generalized anxiety disorder7 (16.3)0 (0)



    		Specific phobia2 (4.7)0 (0)



    		Multiple anxiety disorders7 (16.3)4 (40)


    		Baseline Beck Anxiety Inventory score, mean (SD)25.4 (11.8)29.0 (11.1)t1208=–6.2<.001

    		Baseline Social Phobia Scale score, mean (SD)36.0 (1.7)43.0 (0.4)t606=–82.7<.001

    		Comorbidity: comorbid Axis I diagnosis, %57.658.1χ21=0.0.87

    		Exclusion benzodiazepines: trials that applied this criterion, n (%)10 (23.3)0 (0)χ21=2.2.14

    		Exclusion psychoactive medication other than benzodiazepines: trials that applied this criterion, n (%)0 (0)0 (0)N/AbN/A

    		Exclusion severe depression: trials that applied this criterion, n (%)25 (58.1)2 (20)χ21=4.7.03
    Treatment-related characteristics

    		Treatment dose: number of completed sessions, mean (SD)6.1 (1.4)5.1 (2.4)t835=9.45<.001

    		Treatment adherence: treatment completed %77.953.1t1599=44.73<.001

    aMeans and percentages are based on studies these data were available for. All available data are reported in Multimedia Appendix 3.

    bN/A: not applicable.

    No other variables differed (see Table 1). None of the included studies excluded patients who used psychoactive medication other than benzodiazepines, although a stable dose for the past 1 to 3 months was a criterion for inclusion in general.

    Risk-of-Bias Assessment

    For both OR trials and CSR trials, most of the studies scored a low risk on sequence allocation (OR: 27/33, 82%; CSR: 8/9, 89%), blinding of outcome assessors (OR: 33/33, 100%; CSR: 8/9, 89%), and completeness of outcome data (OR: 27/33, 82%; CSR: 6/9, 67%) (see Multimedia Appendix 4). On allocation concealment most CSR trials scored a low risk (7/9, 78%) compared with 12 of 33 (36%) OR trials. Only 4 of 33 (12%) OR and 2 of 9 (22%) CSR trials scored a low risk on selective outcome reporting, A total of 10 (30%) OR trials and 5 (56%) CSR trials scored a high risk because preregistered outcome measurements were not reported, or other outcome measurements that were not preregistered were inserted in the article. Additionally, 19 (58%) OR trials and 2 (22%) CSR trials were not registered in a trial database and we therefore scored them as having an unclear risk. We rated all included studies as having a high risk of bias on blinding of participants and personnel, because it is not possible to blind participants or therapists to the characteristics of the treatment that is offered.

    We found no significant difference between the percentage of OR trials and the percentage of CSR trials with a high risk for any of the risk-of-bias indicators (P values ranged from P=.08 for sequence generation to P=.49 for allocation concealment).

    Overall Effect

    Primary Outcome

    The overall mean between-groups effect size of iCBT on anxiety symptom reduction when compared with WLC at posttest was g=0.72 (95% CI 0.60-0.83; P<.001) with moderate heterogeneity of I²=53% (95% CI 31-66) and NNT=3.

    The difference in overall effect size for the decrease in anxiety symptoms between iCBT and face-to-face CBT at posttest was nonsignificant (g=0.12, 95% CI –0.02 to 0.26; P=.11; I²=0%, 95% CI 0-75; NNT=15).

    Secondary Outcomes

    Effect sizes of iCBT compared with WLC on depressive symptoms (g=0.61, 95% CI 0.46-0.75; P<.001; I²=70%, 95% CI 57-78; NNT=3) and quality-of-life measurements (g=0.44, 95% CI 0.33-0.55; P<.001; I²=5%, 95% CI 0-54; NNT=5) were moderate.

    For iCBT compared with face-to-face CBT, effect sizes on depression measurements (g=0.04, 95% CI –0.13 to 0.21; P=.65; I²=19%. 95% CI 0-61; NNT=45) and quality-of-life outcomes (g=0.18, 95% CI –0.05 to 0.41; P=.12; I²=0%, 95% CI 0-85; NNT=10) were both nonsignificant.

    Open Recruitment Versus Clinical Service Recruitment

    Primary Outcome

    For studies with WLC comparators, we found a significant difference between OR and CSR trials) in favor of OR trials (Q=10.09; P=.001) (Table 2 and Figure 2). The effect size on anxiety symptom reduction for OR trials was significant and large (g=0.79; P<.001) in favor of iCBT, whereas CSR trials obtained a small effect size (g=0.28; P=.003) in favor of iCBT.

    We found no difference in anxiety symptom reduction between OR (n=6) and CSR trials (n=6) comparing iCBT with face-to-face CBT (Q=0.82; P=.37) (Table 2 and Figure 3). Both OR trials (g=0.19; P=.09) and CSR trials (g=0.06; P=.51) reported a nonsignificant difference between iCBT and face-to-face CBT on decrease in anxiety symptoms.

    Secondary Outcomes

    With regard to depressive symptoms, we found no significant difference between OR trials and CSR trials with WLC comparators (Q=1.43; P=.23) or face-to-face comparators (Q=0.85; P=.36).

    For quality-of-life measurements, we found no significant difference between OR trials and CSR trials for studies comparing iCBT with WLC (Q=0.05; P=.83) or for studies comparing iCBT with face-to-face CBT (Q=0.48; P=.49).

    Multimedia Appendix 5 presents a complete overview of results of OR and CSR subgroup analyses of secondary outcomes.

    Publication Bias

    Neither visual inspection of the funnel plots (see Multimedia Appendix 6) and Egger test (WLC studies: intercept=0.83; 95% CI –0.90 to 2.56; P=.34; face-to-face CBT studies: intercept=1.12, 95% CI –0.93 to 3.17; P=.25) nor the Duval and Tweedie trim-and-fill procedure showed evidence of publication bias.

    Additional Exploratory Analyses

    As Table 1 shows, several demographic (age, sex), clinical (baseline severity, exclusion of severely depressive patients), and treatment-related (treatment dose, treatment adherence) variables differed significantly between OR trials and CSR trials. Of these variables, only exclusion of severely depressed patients (Q=8.06; P=.005), treatment dose (slope=0.10; P=.003), and treatment adherence (slope=0.01; P<.001) appeared to be significantly associated with effect size for WLC comparators in separate subgroup (exclusion of severely depressed patients) and metaregression (treatment dose, treatment adherence) analyses (see Multimedia Appendix 7), meaning that the effect size was higher when severely depressed patients were excluded, when the treatment dose was higher, and when the adherence rate was higher.

    In a multivariate analysis we explored whether the association between recruitment method and effect size for studies with WLC comparators was mediated by these variables. As the variance inflation factors between treatment dose and treatment adherence was 2.7, and treatment adherence was more significantly associated with effect size than treatment dose, we did not include treatment dose in the multivariate model.

    Results showed that recruitment type (slope=0.30; P=.14) was no longer significantly associated with the effect size in the multivariate metaregression analysis, nor was treatment adherence (slope=0.01; P=.23) or exclusion of severely depressed patients (slope=0.13; P=.27).

    Table 2. Main effects of open recruitment trials and clinical service recruitment trials comparing internet-delivered cognitive behavioral therapy versus waitlist control and versus face-to-face cognitive behavioral therapy at posttest, primary outcome.
    GroupComparisons, nRespondents, nHedges g (95% CI)P valueI2 (95% CI)NNTaBetween-groups Q (P value)
    Waitlist control10.09 (.001)

    		Open recruitment3724740.79 (0.71 to 0.87)<.00144 (6 to 58)3

    		Clinical recruitment44460.28 (0.10 to 0.47).00320 (0 to 85)7
    Face-to-face cognitive behavioral therapy0.82 (.37)

    		Open recruitment63360.19 (–0.03 to 0.40).090 (0 to 75)10

    		Clinical recruitment64520.06 (–0.12 to 0.24).530 (0 to 75)30

    aNNT: number needed to treat.

    ‎
    Figure 2. Forest plot of effects on anxiety symptoms of open recruitment trials and clinical service recruitment trials comparing internet-delivered cognitive behavioral therapy (iCBT) with waitlist control (WLC). GAD: generalized anxiety disorder; iCBGT: clinician-guided group iCBT; PD: panic disorder; SAD: social anxiety disorder.
    View this figure
    ‎
    Figure 3. Forest plot of effects on anxiety symptoms of open recruitment trials and clinical service recruitment trials comparing internet-delivered cognitive behavioral therapy (iCBT) with face-to-face cognitive behavioral therapy (CBT). PD: panic disorder; SAD: social anxiety disorder.
    View this figure

    Principal Findings

    This meta-analysis showed that iCBT is more effective than WLC in reducing anxiety symptoms at posttreatment. We found no indication for differences in effect sizes between iCBT and face-to-face CBT. These outcomes confirm the results of previous meta-analyses, which found moderate to large effect sizes for WLC comparator studies and small and nonsignificant effect sizes when comparing iCBT versus face-to-face CBT [1-4].

    Our main research question was whether OR trials produce effects for anxiety symptoms similar to those of CSR trials. For studies with WLC comparators, recruitment method was significantly associated with anxiety treatment outcomes (Q=10.09; P=.001), indicating that effect sizes are higher in OR trials than in CSR trials.

    We explored whether differences between characteristics of samples in OR trials and CSR trials might explain the gap between effects we found in favor of OR trials. Multivariate metaregression analysis revealed that the association between recruitment method and effect size may be partly explained by greater treatment adherence and the exclusion of severely depressed patients in OR trials.

    For studies with face-to-face CBT comparators, we observed no difference in anxiety outcomes between OR trials and CSR trials (Q=0.82; P=.37). Possibly, the number of studies with face-to-face CBT comparators was too low, making these analyses underpowered to detect differences. Another explanation could be that studies with face-to-face CBT comparators resembled each other more on other criteria for clinical representativeness than studies with WLC comparators, as these studies are conducted in routine care. For example, in studies with face-to-face comparators, treatments are generally delivered by skilled clinicians and in clinically representative settings. These study characteristics are more varied in studies with WLC comparators, where treatments can also be delivered by researchers or graduate students and in a research setting such as a university laboratory [108].

    Regarding depressive symptoms and quality of life, we observed no differences between OR trials and CSR trials for either comparator group.

    The difference in results we found between OR and CSR trials for studies with WLC comparators is in line with a previous meta-analysis on effectiveness of face-to-face CBT for anxiety disorders by Stewart and Chambless [9]. The small but significant effect size (d=–0.08; P<.05) they found indicated smaller improvements in more clinically representative patient studies than in less clinically representative studies.

    Our findings are partly in keeping with Andersson and Hedman’s review on the effectiveness of iCBT for anxiety [21]. Results of that review suggested that effectiveness studies obtain similar effects to efficacy trials. Considering they only included studies comparing iCBT with face-to-face CBT, that conclusion corresponds to our results for iCBT compared with face-to-face CBT. However, it needs to be noted that Andersson and Hedman based their distinction between efficacy and effectiveness on the setting in which iCBT was delivered and not on recruitment strategy.

    Strengths and Limitations

    A major strength of this study is that it is, to our knowledge, the first meta-analysis of iCBT for anxiety disorders comparing treatment outcomes between OR and CSR for both WLC and face-to-face CBT comparators. Furthermore, the studies comparing iCBT versus face-to-face CBT were head-to-head comparisons, generating direct evidence.

    Some limitations in this study warrant caution in interpretation. First, the number of trials was relatively low for studies with face-to-face CBT comparators. Hence, finding no difference may have been caused by underpowered analyses.

    Second, clinical representativeness of studies is often rated based on a multitude of criteria, besides recruitment type, such as setting of treatment delivery, experience of therapists, and flexibility in treatment manuals [9,109]. This means that the differences we found may have been caused by predictors not assessed in this study. In future research, considering multiple criteria of efficacy and effectiveness would be helpful to more thoroughly determine clinical representativeness of the studies and the association between clinical representativeness and treatment outcomes.

    Third, in 3 included trials a mixed recruitment strategy was applied. This contamination may have led to some bias. Any such bias will have decreased the difference between OR and CSR trials found in the meta-analysis. We decided to include these trials because they reported clearly on their recruitment method and also the portion of participants recruited through an OR method versus a CSR method.

    Fourth, when interpreting the subgroup analyses and metaregression analyses, it is important to bear in mind that the results were only observational. Direct comparisons are required to verify the findings presented here.

    Conclusions

    This meta-analysis indicated that the effects of iCBT for anxiety disorders compared with WLC in CSR trials were smaller than effects found in OR trials. Hence, for patients with anxiety disorders in routine care, the impact of iCBT may not be as positive as for self-referred study participants recruited from the community. The difference between OR and CSR might be partly caused by a greater treatment adherence of self-referred patients and stricter exclusion criteria for severe depressive symptoms in studies with an OR method. A future challenge is to build a more robust body of evidence supporting the effectiveness of iCBT for anxiety disorders in routine care populations.

    Conflicts of Interest

    NT is Executive Director of MindSpot, which is funded by the Australian Government to deliver iCBT to adults with anxiety and depression across Australia.

    ‎

    Multimedia Appendix 1

    Search string.

    PDF File (Adobe PDF File), 95KB
    ‎

    Multimedia Appendix 2

    Order of rating scales.

    PDF File (Adobe PDF File), 71KB
    ‎

    Multimedia Appendix 3

    Characteristics of included studies.

    PDF File (Adobe PDF File), 151KB
    ‎

    Multimedia Appendix 4

    Risk-of-bias assessment.

    PDF File (Adobe PDF File), 327KB
    ‎

    Multimedia Appendix 5

    Secondary outcomes.

    PDF File (Adobe PDF File), 16KB
    ‎

    Multimedia Appendix 6

    Funnel plot publication bias.

    PDF File (Adobe PDF File), 97KB
    ‎

    Multimedia Appendix 7

    Metaregression and subgroup analyses.

    PDF File (Adobe PDF File), 14KB
    1. Olthuis JV, Watt MC, Bailey K, Hayden JA, Stewart SH. Therapist-supported Internet cognitive behavioural therapy for anxiety disorders in adults. Cochrane Database Syst Rev 2016 Mar 12;3:CD011565. [CrossRef] [Medline]
    2. Adelman CB, Panza KE, Bartley CA, Bontempo A, Bloch MH. A meta-analysis of computerized cognitive-behavioral therapy for the treatment of DSM-5 anxiety disorders. J Clin Psychiatry 2014 Jul;75(7):e695-e704. [CrossRef] [Medline]
    3. Cuijpers P, Marks IM, van Straten A, Cavanagh K, Gega L, Andersson G. Computer-aided psychotherapy for anxiety disorders: a meta-analytic review. Cogn Behav Ther 2009;38(2):66-82. [CrossRef] [Medline]
    4. Reger MA, Gahm GA. A meta-analysis of the effects of internet- and computer-based cognitive-behavioral treatments for anxiety. J Clin Psychol 2009 Jan;65(1):53-75. [CrossRef] [Medline]
    5. Andersson G, Rozental A, Shafran R, Carlbring P. Long-term effects of internet-supported cognitive behaviour therapy. Expert Rev Neurother 2018 Jan;18(1):21-28. [CrossRef] [Medline]
    6. Hedman E, El Alaoui S, Lindefors N, Andersson E, Rück C, Ghaderi A, et al. Clinical effectiveness and cost-effectiveness of internet- vs. group-based cognitive behavior therapy for social anxiety disorder: 4-year follow-up of a randomized trial. Behav Res Ther 2014 Aug;59:20-29. [CrossRef] [Medline]
    7. Nordgren LB, Hedman E, Etienne J, Bodin J, Kadowaki A, Eriksson S, et al. Effectiveness and cost-effectiveness of individually tailored internet-delivered cognitive behavior therapy for anxiety disorders in a primary care population: a randomized controlled trial. Behav Res Ther 2014 Aug;59:1-11 [FREE Full text] [CrossRef] [Medline]
    8. Westen D, Morrison K. A multidimensional meta-analysis of treatments for depression, panic, and generalized anxiety disorder: an empirical examination of the status of empirically supported therapies. J Consult Clin Psychol 2001 Dec;69(6):875-899. [Medline]
    9. Stewart RE, Chambless DL. Cognitive-behavioral therapy for adult anxiety disorders in clinical practice: a meta-analysis of effectiveness studies. J Consult Clin Psychol 2009 Aug;77(4):595-606. [CrossRef] [Medline]
    10. Streiner DL. The 2 “Es” of research: efficacy and effectiveness trials. Can J Psychiatry 2002 Aug;47(6):552-556. [CrossRef] [Medline]
    11. Persons JB, Silberschatz G. Are results of randomized controlled trials useful to psychotherapists? J Consult Clin Psychol 1998 Feb;66(1):126-135. [Medline]
    12. Stirman SW, DeRubeis RJ, Crits-Christoph P, Brody PE. Are samples in randomized controlled trials of psychotherapy representative of community outpatients? A new methodology and initial findings. J Consult Clin Psychol 2003 Dec;71(6):963-972. [CrossRef] [Medline]
    13. Titov N, Andrews G, Kemp A, Robinson E. Characteristics of adults with anxiety or depression treated at an internet clinic: comparison with a national survey and an outpatient clinic. PLoS One 2010;5(5):e10885 [FREE Full text] [CrossRef] [Medline]
    14. Wilks CR, Zieve GG, Lessing HK. Are trials of computerized therapy generalizable? A multidimensional meta-analysis. Telemed J E Health 2016 May;22(5):450-457. [CrossRef] [Medline]
    15. Dear BF, Zou JB, Ali S, Lorian CN, Johnston L, Terides MD, et al. Examining self-guided internet-delivered cognitive behavior therapy for older adults with symptoms of anxiety and depression: Two feasibility open trials. Internet Interventions 2015 Mar;2(1):17-23. [CrossRef]
    16. Titov N, Dear BF, Staples LG, Bennett-Levy J, Klein B, Rapee RM, et al. The first 30 months of the MindSpot Clinic: evaluation of a national e-mental health service against project objectives. Aust N Z J Psychiatry 2017 Dec;51(12):1227-1239. [CrossRef] [Medline]
    17. Hadjistavropoulos HD, Pugh NE, Hesser H, Andersson G. Predicting response to therapist-assisted internet-delivered cognitive behavior therapy for depression or anxiety within an open dissemination trial. Behav Ther 2016 Mar;47(2):155-165. [CrossRef] [Medline]
    18. Ruwaard J, Lange A, Schrieken B, Dolan CV, Emmelkamp P. The effectiveness of online cognitive behavioral treatment in routine clinical practice. PLoS One 2012;7(7):e40089 [FREE Full text] [CrossRef] [Medline]
    19. Williams AD, O'Moore K, Mason E, Andrews G. The effectiveness of internet cognitive behaviour therapy (iCBT) for social anxiety disorder across two routine practice pathways. Internet Interv 2014 Oct;1(4):225-229. [CrossRef]
    20. Mathiasen K, Riper H, Andersen TE, Roessler KK. Guided internet-based cognitive behavioral therapy for adult depression and anxiety in routine secondary care: observational study. J Med Internet Res 2018 Nov 28;20(11):e10927 [FREE Full text] [CrossRef] [Medline]
    21. Andersson G, Hedman E. Effectiveness of guided Internet-based cognitive behavior therapy in regular clinical settings. Verhaltenstherapie 2013;23(3):140-148. [CrossRef]
    22. Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med 2009 Aug 18;151(4):264-9, W64. [Medline]
    23. Riper H, Ballegooijen WV, Kooistra L. Preventie & eMental-health. Onderzoek dat leidt, technologie die verleidt, preventie die bereikt en beklijft. Kennissynthese 2013 in opdracht van ZonMw. Amsterdam, the Netherlands: ZonMw; 2013.
    24. Mahoney AEJ, Mackenzie A, Williams AD, Smith J, Andrews G. Internet cognitive behavioural treatment for obsessive compulsive disorder: a randomised controlled trial. Behav Res Ther 2014 Dec;63:99-106. [CrossRef] [Medline]
    25. Engel CC, Litz B, Magruder KM, Harper E, Gore K, Stein N, et al. Delivery of self training and education for stressful situations (DESTRESS-PC): a randomized trial of nurse assisted online self-management for PTSD in primary care. Gen Hosp Psychiatry 2015;37(4):323-328 [FREE Full text] [CrossRef] [Medline]
    26. Beck AT, Rush AJ, Shaw BF, Emery G. Cognitive Therapy of Depression. New York, NY: Guilford Press; 1979.
    27. Oromendia P, Orrego J, Bonillo A, Molinuevo B. Internet-based self-help treatment for panic disorder: a randomized controlled trial comparing mandatory versus optional complementary psychological support. Cogn Behav Ther 2016 Jun;45(4):270-286. [CrossRef] [Medline]
    28. Johnston L, Titov N, Andrews G, Spence J, Dear BF. A RCT of a transdiagnostic internet-delivered treatment for three anxiety disorders: examination of support roles and disorder-specific outcomes. PLoS One 2011;6(11):e28079 [FREE Full text] [CrossRef] [Medline]
    29. Titov N, Dear BF, Schwencke G, Andrews G, Johnston L, Craske MG, et al. Transdiagnostic internet treatment for anxiety and depression: a randomised controlled trial. Behav Res Ther 2011 Aug;49(8):441-452. [CrossRef] [Medline]
    30. Titov N, Andrews G, Johnston L, Robinson E, Spence J. Transdiagnostic Internet treatment for anxiety disorders: a randomized controlled trial. Behav Res Ther 2010 Sep;48(9):890-899. [CrossRef] [Medline]
    31. Berger T, Urech A, Krieger T, Stolz T, Schulz A, Vincent A, et al. Effects of a transdiagnostic unguided Internet intervention ('velibra') for anxiety disorders in primary care: results of a randomized controlled trial. Psychol Med 2017 Jan;47(1):67-80. [CrossRef] [Medline]
    32. Heading K, Kirkby KC, Martin F, Daniels BA, Gilroy LJ, Menzies RG. Controlled comparison of single-session treatments for spider phobia: live graded exposure alone versus computer-aided vicarious exposure. Behav Change 2012 Feb 22;18(02):103-113. [CrossRef]
    33. Dear BF, Staples LG, Terides MD, Karin E, Zou J, Johnston L, et al. Transdiagnostic versus disorder-specific and clinician-guided versus self-guided internet-delivered treatment for generalized anxiety disorder and comorbid disorders: a randomized controlled trial. J Anxiety Disord 2015 Dec;36:63-77 [FREE Full text] [CrossRef] [Medline]
    34. Titov N, Dear BF, Staples LG, Terides MD, Karin E, Sheehan J, et al. Disorder-specific versus transdiagnostic and clinician-guided versus self-guided treatment for major depressive disorder and comorbid anxiety disorders: a randomized controlled trial. J Anxiety Disord 2015 Oct;35:88-102 [FREE Full text] [CrossRef] [Medline]
    35. Beck AT, Epstein N, Brown G, Steer RA. An inventory for measuring clinical anxiety: psychometric properties. J Consult Clin Psychol 1988 Dec;56(6):893-897. [Medline]
    36. Lovibond SH, Lovibond PF. Manual for the Depression Anxiety Stress Scales. Sydney, Australia: Psychology Foundation of Australia; 1995.
    37. Spielberger CD, Gorsuch RL, Lushene RE. Manual for the State-Trait Anxiety Inventory. Palo Alto, CA: Consulting Psychologists Press; 1970:1-23.
    38. Taylor S, Cox B. Anxiety sensitivity: multiple dimensions and hierarchic structure. Behav Res Ther 1998 Jan;36(1):37-51. [Medline]
    39. Liebowitz MR. Social phobia. Mod Probl Pharmacopsychiatry 1987;22:141-173. [Medline]
    40. Heimberg R, Horner K, Juster H, Safren S, Brown E, Schneier F, et al. Psychometric properties of the Liebowitz Social Anxiety Scale. Psychol Med 1999 Jan;29(1):199-212. [Medline]
    41. Mattick RP, Clarke JC. Development and validation of measures of social phobia scrutiny fear and social interaction anxiety. Behav Res Ther 1998 Apr;36(4):455-470. [Medline]
    42. Furmark T, Tillfors M, Everz P, Marteinsdottir I, Gefvert O, Fredrikson M. Social phobia in the general population: prevalence and sociodemographic profile. Soc Psychiatry Psychiatr Epidemiol 1999 Aug;34(8):416-424. [Medline]
    43. Leary M. A brief version of the Fear of Negative Evaluation Scale. Personal Soc Psychol Bull 1983;9(3):371-375. [Medline]
    44. Spitzer RL, Kroenke K, Williams JBW, Löwe B. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med 2006 May 22;166(10):1092-1097. [CrossRef] [Medline]
    45. Meyer TJ, Miller ML, Metzger RL, Borkovec TD. Development and validation of the Penn State Worry Questionnaire. Behav Res Ther 1990;28(6):487-495. [Medline]
    46. Shear MK, Brown T, Barlow D, Money R, Sholomskas D, Woods S, et al. Multicenter collaborative panic disorder severity scale. Am J Psychiatry 1997 Nov;154(11):1571-1575. [CrossRef] [Medline]
    47. Chambless DL, Caputo GC, Bright P, Gallagher R. Assessment of fear of fear in agoraphobics: the body sensations questionnaire and the agoraphobic cognitions questionnaire. J Consult Clin Psychol 1984 Dec;52(6):1090-1097. [Medline]
    48. Bandelow B. Assessing the efficacy of treatments for panic disorder and agoraphobia. II. The Panic and Agoraphobia Scale. Int Clin Psychopharmacol 1995 Jun;10(2):73-81. [Medline]
    49. Van Zuuren FJ. The fear questionnaire. Some data on validity, reliability and layout. Br J Psychiatry 1988 Nov;153:659-662. [Medline]
    50. Kok R, van Straten A, Beekman A, Cuijpers P. Short-term effectiveness of web-based guided self-help for phobic outpatients: randomized controlled trial. J Med Internet Res 2014 Sep 29;16(9):e226 [FREE Full text] [CrossRef] [Medline]
    51. Bergström J, Andersson G, Ljótsson B, Rück C, Andréewitch S, Karlsson A, et al. Internet-versus group-administered cognitive behaviour therapy for panic disorder in a psychiatric setting: a randomised trial. BMC Psychiatry 2010;10:54 [FREE Full text] [CrossRef] [Medline]
    52. Gallego M, Gerardus EP, van der Kooij M, Mees H. The effects of a Dutch version of an internet-based treatment program for fear of public speaking: a controlled study. Int J Clin Heal Psychol 2011;11(3):459-472 [FREE Full text]
    53. Marks IM, Kenwright M, McDonough M, Whittaker M, Mataix-Cols D. Saving clinicians' time by delegating routine aspects of therapy to a computer: a randomized controlled trial in phobia/panic disorder. Psychol Med 2004 Jan;34(1):9-17. [Medline]
    54. Nordgreen T, Haug T, Öst L, Andersson G, Carlbring P, Kvale G, et al. Stepped care versus direct face-to-face cognitive behavior therapy for social anxiety disorder and panic disorder: a randomized effectiveness trial. Behav Ther 2016 Mar;47(2):166-183. [CrossRef] [Medline]
    55. Andersson G, Waara J, Jonsson U, Malmaeus F, Carlbring P, Ost L. Internet-based self-help versus one-session exposure in the treatment of spider phobia: a randomized controlled trial. Cogn Behav Ther 2009;38(2):114-120. [CrossRef] [Medline]
    56. Andersson G, Paxling B, Roch-Norlund P, Östman G, Norgren A, Almlöv J, et al. Internet-based psychodynamic versus cognitive behavioral guided self-help for generalized anxiety disorder: a randomized controlled trial. Psychother Psychosom 2012;81(6):344-355. [CrossRef] [Medline]
    57. Andersson G, Waara J, Jonsson U, Malmaeus F, Carlbring P, Ost LG. Internet-based exposure treatment versus one-session exposure treatment of snake phobia: a randomized controlled trial. Cogn Behav Ther 2013;42(4):284-291. [CrossRef] [Medline]
    58. Andersson G, Carlbring P, Holmström A, Sparthan E, Furmark T, Nilsson-Ihrfelt E, et al. Internet-based self-help with therapist feedback and in vivo group exposure for social phobia: a randomized controlled trial. J Consult Clin Psychol 2006 Aug;74(4):677-686. [CrossRef] [Medline]
    59. Andersson G, Carlbring P, Furmark T, S. O. F. I. E. Research Group. Therapist experience and knowledge acquisition in internet-delivered CBT for social anxiety disorder: a randomized controlled trial. PLoS One 2012;7(5):e37411 [FREE Full text] [CrossRef] [Medline]
    60. Berger T, Hohl E, Caspar F. Internet-based treatment for social phobia: a randomized controlled trial. J Clin Psychol 2009 Oct;65(10):1021-1035. [CrossRef] [Medline]
    61. Berger T, Boettcher J, Caspar F. Internet-based guided self-help for several anxiety disorders: a randomized controlled trial comparing a tailored with a standardized disorder-specific approach. Psychotherapy (Chic) 2014 Jun;51(2):207-219. [CrossRef] [Medline]
    62. Botella C, Gallego MJ, Garcia-Palacios A, Guillen V, Baños RM, Quero S, et al. An Internet-based self-help treatment for fear of public speaking: a controlled trial. Cyberpsychol Behav Soc Netw 2010 Aug;13(4):407-421. [CrossRef] [Medline]
    63. Carlbring P, Westling BE, Ljungstrand P, Ekselius L, Andersson G. Treatment of panic disorder via the internet: a randomized trial of a self-help program. Behav Ther 2001;32(4):751-764. [CrossRef]
    64. Carlbring P, Nilsson-Ihrfelt E, Waara J, Kollenstam C, Buhrman M, Kaldo V, et al. Treatment of panic disorder: live therapy vs. self-help via the Internet. Behav Res Ther 2005 Oct;43(10):1321-1333. [CrossRef] [Medline]
    65. Carlbring P, Bohman S, Brunt S, Buhrman M, Westling BE, Ekselius L, et al. Remote treatment of panic disorder: a randomized trial of internet-based cognitive behavior therapy supplemented with telephone calls. Am J Psychiatry 2006 Dec;163(12):2119-2125. [CrossRef] [Medline]
    66. Carlbring P, Gunnarsdóttir M, Hedensjö L, Andersson G, Ekselius L, Furmark T. Treatment of social phobia: randomised trial of internet-delivered cognitive-behavioural therapy with telephone support. Br J Psychiatry 2007 Feb;190:123-128. [CrossRef] [Medline]
    67. Carlbring P, Maurin L, Törngren C, Linna E, Eriksson T, Sparthan E, et al. Individually-tailored, Internet-based treatment for anxiety disorders: a randomized controlled trial. Behav Res Ther 2011 Jan;49(1):18-24. [CrossRef] [Medline]
    68. Christensen H, Mackinnon A, Batterham P, O'Dea B, Guastella A, Griffiths K, et al. The effectiveness of an online e-health application compared to attention placebo or Sertraline in the treatment of Generalised Anxiety Disorder. Internet Interventions 2014 Oct;1(4):169-174. [CrossRef]
    69. Furmark T, Carlbring P, Hedman E, Sonnenstein A, Clevberger P, Bohman B, et al. Guided and unguided self-help for social anxiety disorder: randomised controlled trial. Br J Psychiatry 2009 Nov;195(5):440-447. [CrossRef] [Medline]
    70. Kiropoulos LA, Klein B, Austin DW, Gilson K, Pier C, Mitchell J, et al. Is internet-based CBT for panic disorder and agoraphobia as effective as face-to-face CBT? J Anxiety Disord 2008 Dec;22(8):1273-1284. [CrossRef] [Medline]
    71. Klein B, Richards JC, Austin DW. Efficacy of internet therapy for panic disorder. J Behav Ther Exp Psychiatry 2006 Sep;37(3):213-238. [CrossRef] [Medline]
    72. Paxling B, Almlöv J, Dahlin M, Carlbring P, Breitholtz E, Eriksson T, et al. Guided internet-delivered cognitive behavior therapy for generalized anxiety disorder: a randomized controlled trial. Cogn Behav Ther 2011;40(3):159-173. [CrossRef] [Medline]
    73. Richards JC, Klein B, Austin DW. Internet cognitive behavioural therapy for panic disorder: does the inclusion of stress management information improve end-state functioning? Clin Psychol 2006 Mar;10(1):2-15. [CrossRef]
    74. Robinson E, Titov N, Andrews G, McIntyre K, Schwencke G, Solley K. Internet treatment for generalized anxiety disorder: a randomized controlled trial comparing clinician vs. technician assistance. PLoS One 2010;5(6):e10942 [FREE Full text] [CrossRef] [Medline]
    75. Schulz A, Stolz T, Vincent A, Krieger T, Andersson G, Berger T. A sorrow shared is a sorrow halved? A three-arm randomized controlled trial comparing internet-based clinician-guided individual versus group treatment for social anxiety disorder. Behav Res Ther 2016 Sep;84:14-26. [CrossRef] [Medline]
    76. Titov N, Andrews G, Schwencke G, Drobny J, Einstein D. Shyness 1: distance treatment of social phobia over the Internet. Aust N Z J Psychiatry 2008 Jul;42(7):585-594. [CrossRef] [Medline]
    77. Titov N, Andrews G, Schwencke G. Shyness 2: treating social phobia online: replication and extension. Aust N Z J Psychiatry 2008 Jul;42(7):595-605. [CrossRef] [Medline]
    78. Titov N, Andrews G, Choi I, Schwencke G, Mahoney A. Shyness 3: randomized controlled trial of guided versus unguided internet-based CBT for social phobia. Aust N Z J Psychiatry 2008 Dec;42(12):1030-1040. [CrossRef] [Medline]
    79. Titov N, Andrews G, Robinson E, Schwencke G, Johnston L, Solley K, et al. Clinician-Assisted Internet-Based Treatment is Effective for Generalized Anxiety Disorder: Randomized Controlled Trial. Aust N Z J Psychiatry 2009 Jan;43(10):905-912. [CrossRef]
    80. Wims E, Titov N, Andrews G, Choi I. Clinician-assisted Internet-based treatment is effective for panic: a randomized controlled trial. Aust N Z J Psychiatry 2010 Jul;44(7):599-607. [CrossRef] [Medline]
    81. Andrews G, Davies M, Titov N. Effectiveness randomized controlled trial of face to face versus Internet cognitive behaviour therapy for social phobia. Aust N Z J Psychiatry 2011 Apr;45(4):337-340. [CrossRef] [Medline]
    82. Bell C, Colhoun H, Carter F, Frampton C. Effectiveness of computerised cognitive behaviour therapy for anxiety disorders in secondary care. Aust N Z J Psychiatry 2012 Jul;46(7):630-640. [CrossRef] [Medline]
    83. Bruinsma A, Kampman M, Exterkate CC, Hendriks GJ. [An exploratory study of 'blended' cognitive behavioural therapy (CBT) for patients with a panic disorder: results and patients' experiences]. Tijdschr Psychiatr 2016;58(5):361-370 [FREE Full text] [Medline]
    84. Hedman E, Andersson G, Ljótsson B, Andersson E, Rück C, Mörtberg E, et al. Internet-based cognitive behavior therapy vs. cognitive behavioral group therapy for social anxiety disorder: a randomized controlled non-inferiority trial. PLoS One 2011;6(3):e18001 [FREE Full text] [CrossRef] [Medline]
    85. Mathiasen K, Riper H, Ehlers L, Valentin J, Rosenberg N. Internet-based CBT for social phobia and panic disorder in a specialised anxiety clinic in routine care: results of a pilot randomised controlled trial. Internet Interv 2016 May;4:92-98 [FREE Full text] [CrossRef] [Medline]
    86. Norberg M, Diefenbach G, Tolin D. Quality of life and anxiety and depressive disorder comorbidity. J Anxiety Disord 2008 Dec;22(8):1516-1522. [CrossRef] [Medline]
    87. Beck AT, Ward CH, Mendelson M, Mock J, Erbaugh J. An inventory for measuring depression. Arch Gen Psychiatry 1961 Jun;4:561-571. [Medline]
    88. Svanborg P, Asberg M. A new self-rating scale for depression and anxiety states based on the Comprehensive Psychopathological Rating Scale. Acta Psychiatr Scand 1994 Jan;89(1):21-28. [Medline]
    89. Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med 2001 Sep;16(9):606-613 [FREE Full text] [Medline]
    90. Radloff LS. The CES-D Scale. Applied Psychological Measurement 2016 Jul 26;1(3):385-401. [CrossRef]
    91. Frisch M, Cornell J, Villanueva M, Retzlaff P. Clinical validation of the Quality of Life Inventory. A measure of life satisfaction for use in treatment planning and outcome assessment. Psychological Assessment 1992;4(1):92-101. [CrossRef]
    92. The WHOQOL Group. Development of the World Health Organization WHOQOL-BREF quality of life assessment. Psychol Med 1998 May;28(3):551-558. [Medline]
    93. EuroQol Group. EuroQol--a new facility for the measurement of health-related quality of life. Health Policy 1990 Dec;16(3):199-208. [Medline]
    94. Ware J, Kosinski M, Keller SD. A 12-Item Short-Form Health Survey: construction of scales and preliminary tests of reliability and validity. Med Care 1996 Mar;34(3):220-233. [Medline]
    95. Higgins JP, Green S. Cochrane Handbook for Systematic Reviews of Intervention. Chichester, UK: Wiley-Blackwell Publishing Ltd; 2008.
    96. Hedges LV, Olkin I. Statistical Methods For Meta-analysis. Orlando, FL: Academic Press; 1995.
    97. Cohen J. Statistical Power Analysis for the Behavioral Sciences. 2nd edition. Hillsdale, NJ: Lawrence Erlbaum; 1988.
    98. Higgins JPT, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ 2003 Sep 6;327(7414):557-560 [FREE Full text] [CrossRef] [Medline]
    99. Borenstein M, Higgins J, Hedges LV, Rothstein HR. Basics of meta-analysis: I is not an absolute measure of heterogeneity. Res Synth Methods 2017 Mar;8(1):5-18. [CrossRef] [Medline]
    100. Orsini N, Bottai M, Higgins J, Buchan I. Heterogi: Stata Module to Quantify Heterogeneity in a Meta-Analysis. Chestnut Hill, MA: Department of Economics, Boston College; 2006.
    101. Borenstein M, Hedges L, Higgins J, Rothstein H. Introduction to Meta-Analysis. Oxford, UK: Wiley; 2009.
    102. Egger M, Davey SG, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ 1997 Sep 13;315(7109):629-634 [FREE Full text] [Medline]
    103. Duval S, Tweedie R. Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics 2000 Jun;56(2):455-463. [Medline]
    104. Allison P. Logistic Regression Using SAS: Theory And Application. 1st edition. Cary, NC: SAS Institute; 1999.
    105. Kraemer HC, Kupfer DJ. Size of treatment effects and their importance to clinical research and practice. Biol Psychiatry 2006 Jun 1;59(11):990-996. [CrossRef] [Medline]
    106. Citrome L, Ketter T. When does a difference make a difference? Interpretation of number needed to treat, number needed to harm, and likelihood to be helped or harmed. Int J Clin Pract 2013 May;67(5):407-411. [CrossRef] [Medline]
    107. Klein B, Richards JC. A brief internet-based treatment for panic disorder. Behav Cogn Psychother 2001 Mar 6;29(01):113-117. [CrossRef]
    108. Weisz J, Donenberg G, Han S, Weiss B. Bridging the gap between laboratory and clinic in child and adolescent psychotherapy. J Consult Clin Psychol 1995 Oct;63(5):688-701. [Medline]
    109. Shadish WR, Matt GE, Navarro AM, Siegle G, Crits-Christoph P, Hazelrigg MD, et al. Evidence that therapy works in clinically representative conditions. J Consult Clin Psychol 1997 Jun;65(3):355-365. [Medline]

    ‎
    BAI: Beck Anxiety Inventory
    CBT: cognitive behavioral therapy
    CSR: clinical service recruitment
    DSM: Diagnostic and Statistical Manual of Mental Disorders
    iCBT: internet-delivered cognitive behavioral therapy
    NNT: number needed to treat
    OR: open recruitment
    PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses
    WLC: waitlist control

    Edited by G Eysenbach; submitted 26.07.18; peer-reviewed by S Bennett, G Zieve; comments to author 16.08.18; revised version received 31.01.19; accepted 24.03.19; published 17.04.19

    Copyright

    ©Geke Romijn, Neeltje Batelaan, Robin Kok, Jeroen Koning, Anton van Balkom, Nickolai Titov, Heleen Riper. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 17.04.2019.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.