This paper is in the following e-collection/theme issue:

Digital Health Reviews (1174) Depression and Mood Disorders; Suicide Prevention (1271)

Published on in Vol 20, No 11 (2018): November

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/10113, first published .
The Generalizability of Randomized Controlled Trials of Self-Guided Internet-Based Cognitive Behavioral Therapy for Depressive Symptoms: Systematic Review and Meta-Regression Analysis

The Generalizability of Randomized Controlled Trials of Self-Guided Internet-Based Cognitive Behavioral Therapy for Depressive Symptoms: Systematic Review and Meta-Regression Analysis

The Generalizability of Randomized Controlled Trials of Self-Guided Internet-Based Cognitive Behavioral Therapy for Depressive Symptoms: Systematic Review and Meta-Regression Analysis

Authors of this article:

Lorenzo Lorenzo-Luaces1 Author Orcid Image ;   Emily Johns1 Author Orcid Image ;   John R Keefe2 Author Orcid Image
Article Authors Cited by (25) Tweetations (68) Metrics

    Review

    1Department of Psychological and Brain Sciences, Indiana University Bloomington, Bloomington, IN, United States

    2Department of Psychology, University of Pennsylvania, Philadelphia, PA, United States

    Corresponding Author:

    Lorenzo Lorenzo-Luaces, PhD

    Department of Psychological and Brain Sciences

    Indiana University Bloomington

    1101 East 10th Street, Room 158

    Bloomington, IN, 47401

    United States

    Phone: 1 787 222 2493

    Email: lolorenz@indiana.edu


    Background: Self-guided internet-based cognitive behavioral therapies (iCBTs) for depressive symptoms may substantially increase accessibility to mental health treatment. Despite this, questions remain as to the generalizability of the research on self-guided iCBT.

    Objective: We sought to describe the clinical entry criteria used in studies of self-guided iCBT, explore the criteria’s effects on study outcomes, and compare the frequency of use of these criteria with their use in studies of face-to-face psychotherapy and antidepressant medications. We hypothesized that self-guided iCBT studies would use more stringent criteria that would bias the sample toward those with a less complex clinical profile, thus inflating treatment outcomes.

    Methods: We updated a recently published meta-analysis by conducting a systematic literature search in PubMed, MEDLINE, PsycINFO, and EMBASE. We conducted a meta-regression analysis to test the effect of the different commonly used psychiatric entry criteria on the treatment-control differences. We also compared the frequency with which exclusion criteria were used in the self-guided iCBT studies versus studies of face-to-face psychotherapy and antidepressants from a recently published review.

    Results: Our search yielded 5 additional studies, which we added to the 16 studies identified by Karyotaki and colleagues in 2017. Few self-guided iCBT studies excluded patients with severe depressive symptoms (6/21, 29%), but self-guided iCBT studies were more likely than antidepressant (14/170, 8.2%) studies to use this criterion. However, self-guided iCBT studies did not use this criterion more frequently than face-to-face psychotherapy studies (6/16, 38%). Beyond this, we found no evidence that self-guided iCBTs used more stringent entry criteria. Strong evidence suggested that they were actually less likely to use most entry criteria, especially exclusions on the basis of substance use or personality pathology. None of the entry criteria used had an effect on outcomes.

    Conclusions: A conservative interpretation of our findings is that the patient population sampled in the literature on self-guided iCBT is relatively comparable with that of studies of antidepressants or face-to-face psychotherapy. Alternatively, studies of unguided cognitive behavioral therapy may sample from a more heterogeneous and representative patient population. Until evidence emerges to suggest otherwise, the patient population sampled in self-guided iCBT studies cannot be considered as less complex than the patient population from face-to-face psychotherapy or antidepressant studies.

    J Med Internet Res 2018;20(11):e10113

    doi:10.2196/10113

    Keywords

    depression; psychotherapy; CBT; internet-based therapy; pharmacotherapy; generalizability; exclusion criteria; cognitive therapy; telemedicine; drug therapy; patient selection 


    Background

    Depression is one of the leading causes, if not the leading cause, of disability worldwide [1,2]. A well-recognized barrier to reducing the level of disability associated with depression and other common mental disorders is the scarcity of available treatment providers [3]. Although there is a scarcity of providers, the research data support a wide variety of treatments for depression that vary in their modality (eg, self-management, psychotherapy, medications), as well as in how time and resource intensive they are for patients and providers. The combination of antidepressants and psychotherapy is widely seen as providing the best cost-benefit ratio for severe depression [4]. However, it is nearly impossible to advocate this or any other treatment approach as a first-line intervention because depression is highly heterogeneous in its prognosis [5-7]. For example, 50% to 75% of patients with first-onset major depressive episodes recover within a 3- to 6-month period, but 15% to 20% of cases are chronic [8-11]. Similarly, although 50% of patients recover and do not experience a relapse in 20- to 30-year follow-ups, approximately 35% of individuals experience a recurrent course [9,12,13]. Moreover, the presentation of depressive symptoms is highly heterogeneous. Approximately half of cases are classified as mild to moderate [14], but a proportion of cases involve psychotic symptoms [15] or suicidal risk (eg, 27%-53% [16]). As well, there is a substantial degree of comorbidity with other disorders [14].

    A major advancement in the study of treatments for depression is the discovery that self-guided internet-based cognitive behavioral therapy (iCBT) can be more effective than control conditions [17]. In these interventions, patients complete interactive Web-based programs based on the principles of cognitive behavioral therapy with no therapeutic support, although sometimes technical support is available. The efficacy of these interventions can be enhanced by therapist support, although this effect may be smaller than previously thought (eg, standardized mean difference symptoms = 0.27 [18-20]). The high prevalence of depression and the ubiquity of internet access and mobile phone ownership make self-guided iCBTs hugely promising in reducing the burden of disability associated with depressive symptoms, even if they were somewhat less effective than guided iCBTs [21]. Kazdin and Blase [3] have called attention to the fact that the number of individuals with mental disorders such as depression far outnumbers the number of mental health providers available to deliver treatment. Thus, self-guided iCBT has the potential for a greater public health impact than guided iCBT or other forms of treatment that require contact with a trained professional [21]. However, a key concern in evaluating the self-guided iCBT treatment literature is the degree to which the patient population in randomized controlled trials (RCTs) is representative of the heterogeneous nature of the prognosis, severity, and comorbidity found in depressed patients. More than 20 years ago, Seligman [22] pointed out that the literature attesting to the efficacy of face-to-face psychotherapy was limited by virtue of the inclusion and exclusion criteria used, which excluded patients with subthreshold symptoms as well as comorbid conditions. Westen and Morrison [23] quantified the nature of this problem, reporting that most (68%) patients with depression were excluded from a typical psychotherapy study because of suicidality, comorbid disorders, especially substance use, and subclinical symptoms. Furthermore, these authors reported that the number of participants excluded from psychotherapy trials was related to the study-level effect size, such that studies that excluded more patients tended to find larger effect sizes for psychotherapy (see also van der Lem et al [24]). Zimmerman et al [25] made similar observations in the literature on antidepressants. More recently, Zimmerman and colleagues [26-29] reported on the inclusion and exclusion (henceforth entry) criteria for RCTs of antidepressant medications. They stated that most studies excluded patients on the basis of a minimum symptom threshold, suicidality, psychotic features, and substance use disorders.

    Objective

    We sought to review the entry criteria used in trials of self-guided iCBT. We did this by updating and coding the studies from the latest meta-analysis of self-guided iCBT, which was published by Karyotaki et al [17]. In addition to reporting the overall frequency with which specific criteria were used, we sought to explore the relationship between the specific entry criteria and the study-level outcome. Finally, to provide some context for evaluating the frequency with which self-guided iCBT studies used different exclusion criteria, we compared the frequency of use of entry criteria in self-guided iCBT trials versus the rate of inclusion criteria used in face-to-face psychotherapy trials and trials of antidepressant medications. The aim of this analysis was for face-to-face psychotherapy and antidepressant studies to serve as a kind of benchmark against which to compare the self-guided iCBT literature. A prior study suggested that face-to-face psychotherapy trials were less likely than antidepressant trials to use most of the specific entry criteria coded by the study authors [30]. Other studies similarly suggested that psychotherapy studies may be somewhat more generalizable than pharmacotherapy studies for adolescent depression [31], borderline personality disorder [32], social anxiety [33], and posttraumatic stress disorder [34], although not generalized anxiety disorder [35]. Based on prior data [23], we hypothesized that the use of more stringent entry criteria would be associated with better outcomes. However, because studies of self-guided iCBT are often fully remote and lack human support, we hypothesized that studies of self-guided iCBT would use more stringent psychiatric entry criteria than would face-to-face psychotherapy and pharmacotherapy trials.


    Identification and Rating of Studies

    We obtained RCTs exploring the efficacy of self-guided iCBT for depression in adults by referencing and updating the latest meta-analysis of self-guided iCBT for depression [17]. Those authors consulted a broad database of psychological treatments for depression that was constructed from a systematic literature search of free terms combining “psychotherapy” and “depression” in PubMed, EMBASE, PsycINFO, and the Cochrane Library published up to January 1, 2016. We updated the review by applying the same search criteria (see Multimedia Appendix 1), narrowing our search for internet-based interventions. We included studies if they included adults (>18 years of age) with a diagnosis of major depressive disorder on a psychiatric semistructured interview, with elevated symptoms of depression (ie, any specific cutoff score on a depression questionnaire), or who were seeking or undergoing treatment for depression. Two of the authors (LLL, EJ) rated all articles for the presence of common psychiatric inclusion and exclusion criteria used in treatment studies [26], resolving discrepancies by consensus. Descriptive analyses summarizing the specific entry criteria of the iCBT trials are presented. We rated risk of bias according to the Cochrane Collaboration risk-of-bias assessment tool [36]. We rated the primary outcome studies. When we could not determine a specific domain category from the main outcome study, we rated protocol articles if they were available or any publicly available study registry (eg, ClinicalTrials.gov).

    Meta-Regression

    We calculated Hedges g to quantify the difference between self-guided iCBT and control conditions on symptoms of depression. Hedges g is derived from the difference between the average posttreatment scores on self-reported measures of depressive symptoms in the 2 groups (ie, self-guided iCBT group vs the control group), divided by the pooled standard deviation while adjusting for small-sample bias. For entry criteria that had sufficient representation in the dataset (ie, ≥3 studies used them), we conducted meta-regressions, using the R statistical computing language version R-3.5.0. (R Foundation) package metafor [37], to examine the relationship between using specific entry criteria and outcomes. First, we conducted individual meta-regressions in which we regressed the study-level effect size on each of the exclusion criteria used. Then, we regressed the number of exclusion criteria used on the study-level outcomes. Finally, we conducted a simultaneous meta-regression in which we regressed outcomes on all entry criteria.

    Meta-regressions were run in a random effects meta-analytic framework, using exclusion criteria either as dummy-coded categorical variables (0/1) or as an ordinal variable (ie, the number of exclusions). In line with current best practices for calculating confidence intervals with more accurate coverage and less-inflated type I error, we used the Sidik-Jonkman random effects estimator and the Hartung-Knapp adjustment [38,39].

    Benchmarking Against Antidepressant and Face-to-Face Psychotherapy

    To provide a rough index of how entry criteria for guided iCBT studies compare with those of other treatments, we drew on recent reviews of the entry criteria of trials of adult depression with antidepressants [26-29] and face-to-face psychotherapy [30]. These reviews employed searches in PubMed, EMBASE, and PsycINFO, as well as individual meta-analyses and specific journals, to identify acute treatment outcomes studies for depression. The search was limited to studies in which antidepressants or face-to-face psychotherapy was compared with a control condition, so studies of multiple treatments (eg, 2 psychotherapies) were only included if a control condition was used. There are systematic differences between antidepressant and face-to-face pharmacotherapy studies regarding the type of controls employed [40]. Virtually all pharmacotherapy studies use a pill placebo, whereas the efficacy of face-to-face psychotherapy, as well as self-guided iCBT, is tested with a more diverse mix of controls, including a waiting list, treatment as usual, pill placebos, and other conditions (eg, relaxation) that are intended as a control for nonspecific effects (eg, attention). Thus, we searched for face-to-face psychotherapy trials in which a waiting list, treatment as usual, or placebo control was used. We excluded trials if they focused exclusively on comorbidities (eg, only depression and alcohol use), whether psychiatric or general medical, as they are, by definition, less inclusive. As well, we did not include trials focused on subtypes of depression, inpatients, or patients with specific symptom profiles (eg, cognitive symptoms). The application of these criteria yielded 170 studies of antidepressants and 16 studies of face-to-face psychotherapy. To compare the differences between the self-guided iCBT studies and studies of antidepressants and face-to-face psychotherapies, we used a chi-square test, or Fisher exact test when we expected any cell to have a frequency lower than 5.


    Study Characteristics

    We identified 5 new studies (see Multimedia Appendix 1) since the publication of the review by Karyotaki et al [17] that met our inclusion criteria (see Figure 1). Thus, we coded 21 RCTs comparing self-guided iCBTs versus a control in our updated review. These studies analyzed data for 4781 participants. Like Karyotaki et al, we found risk of bias to be low across most studies (see Multimedia Appendix 1).

    There were differences between the self-guided iCBT studies, the antidepressant studies, and the face-to-face psychotherapy studies in terms of sample size per study arm (F2,204=6.45, P=.002) and country of origin (χ21=48.3, P<.001), but no differences in the percentage of studies that excluded participants based on an upper age limit (χ21=1.5, P=.48). Antidepressant (mean 120.69, SD 74.10) and self-guided iCBT (mean 140.61, SD 109.16) studies tended to have more participants in each study arm than face-to-face psychotherapy studies did (mean 55.12, SD 39.48). Most self-guided iCBT studies originated from single sites within European countries (14/21, 67%). By way of contrast, only 5 of the psychotherapy studies were conducted in Europe (5/16, 31%) and, instead, the face-to-face psychotherapy studies were more likely to be published in the United States (8/16, 50%). Very few antidepressants studies were published from within single European countries (15/170, 8.8%), as most were published in the United States (101/171, 59.4%).

    Table 1 lists the psychiatric inclusion and exclusion criteria used in the self-guided iCBT trials.

    ‎
    Figure 1. Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flowchart for self-guided internet-based cognitive behavioral therapy (iCBT) studies included in the systematic review and meta-regression analysis. CBT: cognitive behavioral therapy.
    View this figure
    Table 1. Psychiatric exclusion criteria of randomized controlled trials of self-guided internet-based cognitive behavioral therapy for depressive symptoms (N=21).
    Exclusion criterian (%)
    Severity scale score below the cutoff (minimum score)15 (71)
    Severity scale score above the cutoff (maximum score)6 (29)
    Psychosis12 (57)
    Substance abuse or dependence5 (24)
    Significant suicidal ideation or risk13 (62)
    History of suicide attempt(s)1 (5)
    Episode length too long1 (5)
    Episode length too short1 (5)
    Any axis II disorder0 (0)
    Any axis I disorder1 (5)
    Borderline personality disorder0 (0)
    Antisocial personality disorder0 (0)
    Schizotypal personality disorder0 (0)

    At least half of the iCBT studies used the following 3 criteria: psychotic disorder or current psychotic symptoms (12/21, 57%), a minimum symptom severity on a depression scale (15/21, 71%), and significant suicidal ideation (13/21, 62%). The other criteria were used relatively infrequently. As has been noted of studies of antidepressants and face-to-face psychotherapy, there was considerable variability in the operationalization of these criteria. For example, of the 15 studies that excluded patients on the basis of a minimum score on a depression severity measure, 6 used the Patient Health Questionnaire-8 or -9 as an exclusion criterion, 4 used the Beck Depression Inventory (BDI), and 2 used the Kessler Psychological Distress Scale. The remaining 3 studies used other scales (eg, the Center for Epidemiologic Studies Depression Scale [CES-D]).

    Effects of Entry Criteria on Outcomes

    Only 5 of the exclusionary criteria were used with enough frequency (ie, >3 uses) that we could explore whether their use was associated with treatment outcomes: minimum symptom severity, maximum symptom severity, psychosis, substance misuse, and suicidality. Table 2 lists the results of the meta-regressions in which we regressed the study-level effect sizes in the 5 exclusionary criteria, individually as well as considered simultaneously. The updated meta-analysis found results similar to those by Karyotaki et al [17]. At the study level, self-guided iCBT was associated with more improvement in depression than were the control conditions (g=0.33, 95% CI 0.20-0.46, SE 0.06; P<.001).

    We detected a high degree of between-study heterogeneity of effect sizes (I2=76%), indicating that exclusion criteria could potentially explain between-studies differences in effects. Despite this, the meta-regressions did not find any significant effects of specific exclusionary criteria (P values >.39) or the total number of criteria used on study-level treatment outcomes (B=–0.01, 95% CI –0.09 to 0.10, SE 0.05; P=.92). The study-level exclusion criterion that appeared to have the largest effect on study-level differences was the use of a minimum of symptom severity as an exclusion (B=–0.12, 95% CI –0.41 to 0.17, SE 0.14; P=.39). A simultaneous regression considering all of the exclusion criteria in tandem likewise did not find any significant effect of the use of specific entry criteria on outcomes (P values >.28).

    Comparison of Entry Criteria in Internet-Based Cognitive Behavioral Therapies, Antidepressants, and Psychotherapy

    Table 3 shows the comparisons of the frequency with which different entry criteria were used in studies of self-guided iCBT versus studies of face-to-face psychotherapy versus studies of antidepressant medications. Contrary to our hypothesis, virtually all the exclusion criteria coded were used less frequently in the iCBT trials than in face-to-face psychotherapy and antidepressant trials, though not all these differences were statistically significant.

    When specifically compared with studies of antidepressant medications, studies of self-guided iCBT were less likely to use almost all the exclusion criteria coded. For example, compared with studies of antidepressants (143/170, 84.1%), self-guided iCBT studies were less likely (12/21, 57%) to exclude patients on the basis of psychotic symptoms or a diagnosis (χ21=7.2, P=.01). Similarly, 41.2% (70/170) of studies on antidepressants excluded patients on the basis of borderline personality disorder, although no self-guided iCBT study used this criterion (Fisher P<.001). The only exception to this general pattern of self-guided iCBT studies being more, rather than less, inclusive was on the basis of a maximum symptom severity exclusion. Only 29% (6/21) of the self-guided iCBT studies excluded patients on the basis of high symptom severity. However, this exclusion occurred more frequently in the iCBT studies than in the antidepressant studies (14/170, 8.2%; χ21=6.2, P=.01). The self-guided iCBT studies were no more likely than face-to-face psychotherapy studies (6/16, 38%; χ21=.05, P=.83) to use this criterion.

    Table 2. Meta-regression coefficients for the relationship between individual exclusion criteria and study-level internet-based cognitive behavioral therapy controlled outcomes. No findings were significant at P<.05.
    Exclusion criteriaSingle-predictor models, g (95% CI)Simultaneous-predictor model, g (95% CI)
    Minimum symptom severity–0.12 (–0.41 to 0.17)–0.18 (–0.52 to 0.16)
    Maximum symptom severity–0.04 (–0.33 to 0.25)–0.10 (–0.45 to 0.26)
    Psychosis0.07 (–0.20 to 0.34)0.04 (–0.38 to 0.46)
    Substance problems0.02 (–0.29 to 0.33)0.05 (–0.39 to 0.49)
    Suicidality0.09 (–0.18 to 0.36)0.16 (–0.24 to 0.55)
    Table 3. Comparison of the frequency of use of different inclusion and exclusion criteria across self-guided internet-based cognitive behavioral therapy (iCBT), face-to-face psychotherapy (F2F), and antidepressant medication (AM) trials.
    Exclusion criteriaiCBT (n=21)F2F (n=16)AM (n=170)P value
    iCBT vs F2F vs AMiCBT vs F2FiCBT vs AM
    Minimum symptom severity15 (71)13 (81)170 (100.0)<.001.70<.001
    Maximum symptom severity6 (29)6 (38)14 (8.2)<.001.83.01
    Psychosis12 (57)14 (88)143 (84.1).02.07.01
    Substance abuse or dependence5 (24)12 (75)137 (80.6)<.001<.001<.001
    Suicidal risk13 (62)9 (56)128 (75.3).14.99.29
    Prior suicide attempt(s)1 (5)2 (12)35 (20.6).21.57.13
    Episode length too long1 (5)2 (12)34 (20.0).21.57.13
    Episode length too short1 (5)2 (12)81 (47.6)<.001.57<.001
    Any axis II disorder0 (0)3 (19)60 (35.3)<.001.07<.001
    Any axis I disorder1 (5)1 (6)46 (27.1).01>.99.03
    Borderline personality disorder0 (0)5 (31)70 (41.2)<.001.01<.001
    Antisocial personality disorder0 (0)5 (31)68 (40.0)<.001.01<.001
    Schizotypal personality disorder0 (0)5 (31)63 (37.1)<.001.01<.001

    Most of the differences between self-guided iCBT and face-to-face psychotherapy studies did not meet the statistical significance threshold of P<.05, although the arithmetic differences were often in favor of the self-guided iCBT studies. Face-to-face therapy studies were more likely to specifically exclude participants based on the diagnoses of borderline, schizotypal, or antisocial personality disorder (each 5/16, 31%) than were the iCBT studies, which did not use this exclusion (each 0/21, 0%; Fisher P<.001).

    By far the largest observed difference between the iCBT studies and the face-to-face and antidepressants studies was that iCBT studies infrequently excluded patients on the basis of a substance use disorder (5/21, 24%), though this exclusion was typical in antidepressant (137/170, 80.6%; Fisher P<.001) and face-to-face psychotherapy studies (12/16, 75%; Fisher P=.003).


    Principal Findings

    We summarized the exclusion criteria used in self-guided iCBT studies, explored the relationship of the use of these criteria and the treatment effect size reported in the trial, and compared the frequencies with which such criteria were used in antidepressant trials or trials of face-to-face psychotherapy. Overall, self-guided iCBT studies infrequently used exclusion criteria that are the norm in studies of depression. Contrary to our hypotheses, we did not find the type and number of exclusion criteria to be related to outcomes. Also contrary to our hypothesis, self-guided iCBT trials were either equally likely or less likely to use specific psychiatric entry criteria.

    Limitations

    A noteworthy limitation of this review is the small number of self-guided iCBT studies. It is possible that the entry criteria used had a small effect on outcomes, but a larger number of studies is needed to detect this effect. Our review was limited to studies of self-guided iCBTs. While these interventions may have a broader public health impact than guided iCBT [21], there is evidence that they have somewhat smaller effects and lower rates of treatment completion [18-20]. Future research should study the entry criteria in studies of guided iCBT.

    The fact that studies did not use a specific entry criterion (eg, substance use disorder) does not imply that the trial actually had patients representative of that feature, which is another limitation of our study. Because studies do not uniformly report on all characteristics of their patient sample (eg, substance use, duration, suicide risk), it is impossible to test for the presence of these features of the sample across all the studies. It is also possible that studies used specific entry criteria but did not disclose them in the published report. Moreover, the psychiatric inclusion and exclusion criteria are only one aspect of the generalizability of the sample. Self-guided iCBT studies may have more selective patient samples by virtue of other criteria that they use explicitly (eg, access to the internet) or implicitly (eg, willingness to participate in an iCBT study).

    While we updated the review by Karyotaki et al [17], we drew on published reviews by Lorenzo-Luaces et al [30], instead of updating this review as well. A difference between the 2 reviews is that the review by Lorenzo-Luaces and colleagues was limited to studies in which patients had a diagnosis of major depressive disorder. This difference specifically limited the number of face-to-face therapy studies eligible for coding. Our aim, however, was not to provide an exact estimate of the difference in frequency of use of exclusion criteria, but to provide a benchmarking context to evaluate the frequency with which self-guided iCBT studies excluded patients. The studies also differed by year of publications, as per the search strategy, country of origin, and size. These differences, along with the differences we identified as per our objectives, suggest that the studies are not directly or indirectly comparable (eg, as in a network meta-analysis). Our aim, however, was not to claim that the evidence base behind self-guided iCBTs is equivalent to the evidence base behind antidepressants or face-to-face psychotherapy but to provide a frame of reference against which to compare the entry criteria of self-guided iCBTs.

    Implications

    Only 29% of the self-guided iCBT studies excluded patients on the basis of severe depression, but it is still worth noting that self-guided iCBT studies appeared more likely than antidepressant studies to exclude participants on this basis. The perception that self-guided iCBT will not be effective for cases of more severely symptomatic depression aligns with common sense but is not supported by research data. For example, Bower et al [41] reported that the effects of self-guided internet-based therapies were more rather than less pronounced among patients high in symptom severity. In our analyses, we found no evidence that the effects of iCBT varied strongly according to exclusions by high or low symptom severity. It is possible that self-guided iCBTs are less effective for patients who have more complex presentations, with symptom severity being only one index of case complexity [5]. For example, the presence of anxiety and chronic depression duration have all been implicated in treatment outcomes in depression and may relate to lower response to iCBT [42]. Although we did not find the use of psychosis as an exclusion criterion to be related to outcomes, psychotic depression is relatively rare, and even studies allowing these patients into the trial may have had a low representation of psychotic depression. In contrast, approximately 50% of cases of depression are rated as severe or very severe. It is probable that patients self-select into treatment trials in a way that patients with severe depression avoid iCBT studies, but the data do not strongly support this conclusion. For example, Karyotaki et al [17] reported a mean score of 28 on the BDI for patients in their dataset along with a score of 26 on the CES-D, both of which are in the moderate to severe range for each measure.

    Self-guided iCBT studies often excluded patients on the basis of suicidal risk. Our findings suggest that this exclusion criterion was not associated with outcomes, a finding that mirrors the results of van der Lem et al [24], as well as the fact that suicidal risk is not consistently reported as a predictor of treatment outcomes with face-to-face psychotherapy or antidepressants. A common exclusion criterion in antidepressant and face-to-face psychotherapy studies is substance use disorder, although this was missing from most self-guided iCBT studies. The findings of van der Lem et al [24] suggested that alcohol use disorder is not a strong predictor of outcomes in treatment for depression, and other studies have also not found an association between substance use disorder and outcomes in treatment for depression [43]. Given that there is no evidence to suggest negative outcomes for suicidal patients and those with substance use disorders, as well as the low rate of treatment utilization in these and other patient groups, more self-guided iCBT studies specifically recruiting from these patient groups are needed. Hoertel et al [44] have argued that using, or not using, specific criteria may influence not only the external validity and generalizability of a study but also its internal validity, by virtue of influencing outcomes. This runs counter to the prevailing logic that there is a trade-off between external and internal validity. Overall, our results suggest that the inclusion criteria commonly used in the field sacrifice external validity but provide no gains in internal validity.

    Conclusions

    To our knowledge, this is the first exploration of the effects of inclusion and exclusion criteria in self-guided iCBT studies, as well as the first comparison of the specific inclusion and exclusion criteria used in RCTs of iCBTs versus studies of face-to-face psychotherapy or antidepressants. Our findings can be taken to suggest that self-guided iCBT studies are more inclusive by design than studies of antidepressants or face-to-face psychotherapy. It is possible that, by using remote designs in which no individual face-to-face interviews can be conducted, self-guided iCBT trials limit their ability to exclude participants on specific features. For example, no self-guided iCBT trial excluded participants due to personality pathology broadly construed or borderline, schizotypal, or antisocial personality diagnosis, which are usually measured in psychiatric interviews, not self-reported questionnaires. By comparison, around a third of psychotherapy and around 40% of antidepressant trials used these exclusions. Given the differences in the entry criteria used, it stands to reason that, by design, studies of self-guided iCBT may be characterized by a more heterogeneous group of patients than studies of antidepressant or face-to-face psychotherapy. This level of heterogeneity in the underlying patient population increases the external validity of the research. Moreover, this variability can facilitate the discovery of process-outcome correlations [45], as well as effects of individual patient differences on outcomes [5]. This heterogeneity in the patient population may also contribute to heterogeneity in the overall effect size reported in treatment-control comparisons. Until evidence emerges to the contrary, however, it cannot be said that iCBT studies apply more stringent inclusion and exclusion criteria than studies of other treatments for depression or that the efficacy of the treatments is inflated by using a less complex or severe patient population.

    Acknowledgments

    We would like to thank Eirini Karyotaki, Pim Cuijpers, and the coauthors of the meta-analysis from which we drew the studies for this work, as well as the primary authors of the studies we analyzed.

    Conflicts of Interest

    None declared.

    ‎

    Multimedia Appendix 1

    List of studies, risk-of-bias assessments, and example search terms.

    PDF File (Adobe PDF File), 49KB
    1. World Health Organization. Depression and Other Common Mental Disorders: Global Health Estimates. Geneva, Switzerland: WHO; 2017.   URL: http:/​/apps.​who.int/​iris/​bitstream/​handle/​10665/​254610/​WHO-MSD-MER-2017.​2-eng.​pdf;jsessionid=D11B5ED6C69A736C263041085DAEFD55?sequence=1 [accessed 2018-10-29] [WebCite Cache]
    2. Murray CJL, Vos T, Lozano R, Naghavi M, Flaxman AD, Michaud C, et al. Disability-adjusted life years (DALYs) for 291 diseases and injuries in 21 regions, 1990-2010: a systematic analysis for the Global Burden of Disease Study 2010. Lancet 2012 Dec 15;380(9859):2197-2223. [CrossRef] [Medline]
    3. Kazdin AE, Blase SL. Rebooting psychotherapy research and practice to reduce the burden of mental illness. Perspect Psychol Sci 2011 Jan;6(1):21-37. [CrossRef] [Medline]
    4. Hollon SD, DeRubeis RJ, Fawcett J, Amsterdam JD, Shelton RC, Zajecka J, et al. Notice of Retraction and Replacement. Hollon et al. Effect of cognitive therapy with antidepressant medications vs antidepressants alone on the rate of recovery in major depressive disorder: a randomized clinical trial. JAMA Psychiatry. 2014;71(10):1157-1164. JAMA Psychiatry 2016 Dec 01;73(6):639-640 [FREE Full text] [CrossRef] [Medline]
    5. Lorenzo-Luaces L, DeRubeis RJ, van Straten A, Tiemens B. A prognostic index (PI) as a moderator of outcomes in the treatment of depression: a proof of concept combining multiple variables to inform risk-stratified stepped care models. J Affect Disord 2017 Apr 15;213:78-85. [CrossRef] [Medline]
    6. Parker G. Beyond major depression. Psychol Med 2005 Apr;35(4):467-474. [Medline]
    7. Lorenzo-Luaces L. Heterogeneity in the prognosis of major depression: from the common cold to a highly debilitating and recurrent illness. Epidemiol Psychiatr Sci 2015 Dec;24(6):466-472. [CrossRef] [Medline]
    8. Eaton WW, Anthony JC, Gallo J, Cai G, Tien A, Romanoski A, et al. Natural history of Diagnostic Interview Schedule/DSM-IV major depression. The Baltimore Epidemiologic Catchment Area follow-up. Arch Gen Psychiatry 1997 Nov;54(11):993-999. [Medline]
    9. Eaton WW, Shao H, Nestadt G, Lee HB, Lee BH, Bienvenu OJ, et al. Population-based study of first onset and chronicity in major depressive disorder. Arch Gen Psychiatry 2008 May;65(5):513-520 [FREE Full text] [CrossRef] [Medline]
    10. Patten SB, Wang JL, Williams JVA, Lavorato DH, Khaled SM, Bulloch AGM. Predictors of the longitudinal course of major depression in a Canadian population sample. Can J Psychiatry 2010 Oct;55(10):669-676. [CrossRef] [Medline]
    11. Spijker J, De Graaf R, Bijl RV, Beekman ATF, Ormel J, Nolen WA. Duration of major depressive episodes in the general population: Results from the Netherlands Mental Health Survey and Incidence Study (NEMESIS). Br J Psychiatry 2018 Jan 2;181(03):208-213. [CrossRef]
    12. Mattisson C, Bogren M, Horstmann V, Munk-Jörgensen P, Nettelbladt P. The long-term course of depressive disorders in the Lundby Study. Psychol Med 2007 Jun;37(6):883-891. [CrossRef] [Medline]
    13. Steinert C, Hofmann M, Kruse J, Leichsenring F. The prospective long-term course of adult depression in general practice and the community. A systematic literature review. J Affect Disord 2014 Jan;152-154:65-75. [CrossRef] [Medline]
    14. Kessler RC, Berglund P, Demler O, Jin R, Koretz D, Merikangas KR, et al. The epidemiology of major depressive disorder: results from the National Comorbidity Survey Replication (NCS-R). JAMA 2003 Jun 18;289(23):3095-3105. [CrossRef] [Medline]
    15. Ohayon MM, Schatzberg AF. Prevalence of depressive episodes with psychotic features in the general population. Am J Psychiatry 2002 Nov;159(11):1855-1861. [CrossRef] [Medline]
    16. Office of Applied Studies. Suicidal Thoughts, Suicide Attempts, Major Depressive Episode, and Substance Use among Adults. Rockville, MD: Substance Abuse and Mental Health Services Administration; 2006:31.
    17. Karyotaki E, Riper H, Twisk J, Hoogendoorn A, Kleiboer A, Mira A, et al. Efficacy of self-guided internet-based cognitive behavioral therapy in the treatment of depressive symptoms: a meta-analysis of individual participant data. JAMA Psychiatry 2017 Apr 01;74(4):351-359. [CrossRef] [Medline]
    18. Baumeister H, Reichler L, Munzinger M, Lin J. The impact of guidance on Internet-based mental health interventions — a systematic review. Internet Interv 2014 Oct;1(4):205-215. [CrossRef]
    19. Cowpertwait L, Clarke D. Effectiveness of web-based psychological interventions for depression: a meta-analysis. Int J Ment Health Addict 2013 Jan 18;11(2):247-268. [CrossRef]
    20. Königbauer J, Letsch J, Doebler P, Ebert D, Baumeister H. Internet- and mobile-based depression interventions for people with diagnosed depression: a systematic review and meta-analysis. J Affect Disord 2017 Dec 01;223:28-40. [CrossRef] [Medline]
    21. Weisel KK, Zarski A, Berger T, Krieger T, Schaub MP, Moser CT, et al. Efficacy and cost-effectiveness of guided and unguided internet- and mobile-based indicated transdiagnostic prevention of depression and anxiety (ICare Prevent): a three-armed randomized controlled trial in four European countries. Internet Interv 2018 Apr. [CrossRef]
    22. Seligman ME. The effectiveness of psychotherapy. The Consumer Reports study. Am Psychol 1995 Dec;50(12):965-974. [Medline]
    23. Westen D, Morrison K. A multidimensional meta-analysis of treatments for depression, panic, and generalized anxiety disorder: an empirical examination of the status of empirically supported therapies. J Consult Clin Psychol 2001 Dec;69(6):875-899. [Medline]
    24. van der Lem R, de Wever WWH, van der Wee NJA, van Veen T, Cuijpers P, Zitman FG. The generalizability of psychotherapy efficacy trials in major depressive disorder: an analysis of the influence of patient selection in efficacy trials on symptom outcome in daily practice. BMC Psychiatry 2012 Nov 08;12:192 [FREE Full text] [CrossRef] [Medline]
    25. Zimmerman M, Mattia JI, Posternak MA. Are subjects in pharmacological treatment trials of depression representative of patients in routine clinical practice? Am J Psychiatry 2002 Mar;159(3):469-473. [CrossRef] [Medline]
    26. Zimmerman M, Clark HL, Multach MD, Walsh E, Rosenstein LK, Gazarian D. Have treatment studies of depression become even less generalizable? A review of the inclusion and exclusion criteria used in placebo-controlled antidepressant efficacy trials published during the past 20 years. Mayo Clin Proc 2015 Sep;90(9):1180-1186. [CrossRef] [Medline]
    27. Zimmerman M, Clark HL, Multach MD, Walsh E, Rosenstein LK, Gazarian D. Variability in the substance use disorder exclusion criterion in antidepressant efficacy trials. J Affect Disord 2016 Dec 01;198:39-42. [CrossRef] [Medline]
    28. Zimmerman M, Clark HL, Multach MD, Walsh E, Rosenstein LK, Gazarian D. Symptom severity and the generalizability of antidepressant efficacy trials: changes during the past 20 years. J Clin Psychopharmacol 2016 Apr;36(2):153-156. [CrossRef] [Medline]
    29. Zimmerman M, Multach M, Walsh E, Rosenstein LK, Gazarian D, Clark HL. Problems in the descriptions of the psychiatric inclusion and exclusion criteria in publications of antidepressant efficacy trials: a qualitative review and recommendations for improved clarity. CNS Drugs 2016 Mar;30(3):185-191. [CrossRef] [Medline]
    30. Lorenzo-Luaces L, Zimmerman M, Cuijpers P. Are studies of psychotherapies for depression more or less generalizable than studies of antidepressants? J Affect Disord 2018 Dec;234:8-13. [CrossRef] [Medline]
    31. Blanco C, Hoertel N, Franco S, Olfson M, He J, López S, et al. Generalizability of clinical trial results for adolescent major depressive disorder. Pediatrics 2017 Dec;140(6):e20161701. [CrossRef] [Medline]
    32. Hoertel N, López S, Wang S, González-Pinto A, Limosin F, Blanco C. Generalizability of pharmacological and psychotherapy clinical trial results for borderline personality disorder to community samples. Personal Disord 2015 Jan;6(1):81-87 [FREE Full text] [CrossRef] [Medline]
    33. Hoertel N, de Maricourt P, Katz J, Doukhan R, Lavaud P, Peyre H, et al. Are participants in pharmacological and psychotherapy treatment trials for social anxiety disorder representative of patients in real-life settings? J Clin Psychopharmacol 2014 Dec;34(6):697-703. [CrossRef] [Medline]
    34. Franco S, Hoertel N, McMahon K, Wang S, Rodríguez-Fernández JM, Peyre H, et al. Generalizability of pharmacologic and psychotherapy clinical trial results for posttraumatic stress disorder to community samples. J Clin Psychiatry 2016 Dec;77(8):e975-e981. [CrossRef] [Medline]
    35. Hoertel N, Le Strat Y, Blanco C, Lavaud P, Dubertret C. Generalizability of clinical trial results for generalized anxiety disorder to community samples. Depress Anxiety 2012 Jul;29(7):614-620. [CrossRef] [Medline]
    36. Higgins JP, Altman DG. Assessing risk of bias in included studies. In: Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series. Chichester, England: Wiley-Blackwell; 2008:187-241.
    37. Viechtbauer W. Conducting meta-analyses in R with the metafor package. J Stat Softw 2010;36(3). [CrossRef]
    38. IntHout J, Ioannidis JPA, Borm GF. The Hartung-Knapp-Sidik-Jonkman method for random effects meta-analysis is straightforward and considerably outperforms the standard DerSimonian-Laird method. BMC Med Res Methodol 2014 Feb 18;14:25 [FREE Full text] [CrossRef] [Medline]
    39. Röver C, Knapp G, Friede T. Hartung-Knapp-Sidik-Jonkman approach and its modification for random-effects meta-analysis with few studies. BMC Med Res Methodol 2015 Nov 14;15:99 [FREE Full text] [CrossRef] [Medline]
    40. Huhn M, Tardy M, Spineli LM, Kissling W, Förstl H, Pitschel-Walz G, et al. Efficacy of pharmacotherapy and psychotherapy for adult psychiatric disorders: a systematic overview of meta-analyses. JAMA Psychiatry 2014 Jun;71(6):706-715. [CrossRef] [Medline]
    41. Bower P, Kontopantelis E, Sutton A, Kendrick T, Richards DA, Gilbody S, et al. Influence of initial severity of depression on effectiveness of low intensity interventions: meta-analysis of individual patient data. BMJ 2013;346:f540 [FREE Full text] [Medline]
    42. Kessler RC, van Loo H, Wardenaar KJ, Bossarte RM, Brenner LA, Ebert DD, et al. Using patient self-reports to study heterogeneity of treatment effects in major depressive disorder. Epidemiol Psychiatr Sci 2017 Feb;26(1):22-36 [FREE Full text] [CrossRef] [Medline]
    43. Nunes EV, Levin FR. Treatment of depression in patients with alcohol or other drug dependence: a meta-analysis. JAMA 2004 Apr 21;291(15):1887-1896. [CrossRef] [Medline]
    44. Hoertel N, Le Strat Y, Limosin F, Dubertret C, Gorwood P. Prevalence of subthreshold hypomania and impact on internal validity of RCTs for major depressive disorder: results from a national epidemiological sample. PLoS One 2013;8(2):e55448 [FREE Full text] [CrossRef] [Medline]
    45. DeRubeis RJ, Gelfand LA, German RE, Fournier JC, Forand NR. Understanding processes of change: how some patients reveal more than others-and some groups of therapists less-about what matters in psychotherapy. Psychother Res 2014;24(3):419-428 [FREE Full text] [CrossRef] [Medline]

    ‎
    BDI: Beck Depression Inventory
    CES-D: Center for Epidemiologic Studies Depression Scale
    iCBT: internet-based cognitive behavioral therapy
    RCT: randomized controlled trial

    Edited by G Eysenbach; submitted 14.02.18; peer-reviewed by N Hoertel, P Bower, N Forand, T Berger; comments to author 05.03.18; revised version received 11.07.18; accepted 19.07.18; published 09.11.18

    Copyright

    ©Lorenzo Lorenzo-Luaces, Emily Johns, John R Keefe. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 09.11.2018.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.