Anxiety Online—A Virtual Clinic: Preliminary Outcomes Following Completion of Five Fully Automated Treatment Programs for Anxiety Disorders and Symptoms

Introduction

Anxiety disorder is a generic term given to a group of specific disorders that are typically characterized by fear, worry, and phobic responses. The main anxiety disorder types are generalized anxiety disorder (GAD), panic disorder with or without agoraphobia (PD/A), obsessive–compulsive disorder (OCD), posttraumatic stress disorder (PTSD), and social anxiety disorder (SAD). These disorders are highly prevalent mental health conditions that have deleterious effects on a person’s life, including substantial personal, social, and occupational impairment, and are often associated with considerable comorbidity [1-4] resulting in significant economic costs for the individual and society. In the Australian National Mental Health Survey [2], only around one-third of those with an anxiety disorder (37.8%) reported making use of services over the previous 12 months for their mental health problems [5].

Cognitive behavior therapy (CBT) has been shown to be an effective treatment for GAD, PD/A, OCD, PSTD, and SAD. Face-to-face CBT for these anxiety disorders typically involves 60 to 90 minutes of treatment per week over 9–12 weeks, including psychoeducation, anxiety management (eg, relaxation techniques), cognitive and exposure therapy, and relapse prevention [6]. Nevertheless, this form of specialized treatment is unavailable to many of those affected due to a shortage of suitably qualified health care professionals (especially in regional and rural areas), fee-for-service costs, and the stigma attached to seeing a mental health professional [7,8].

The development of e-therapy or e-mental health interventions (delivery of mental health interventions and services via information and communication technologies) has grown exponentially over the past decade, and is one way of delivering CBT that overcomes the commonly cited obstacles to treatment provision [9]. There are now hundreds of e-mental health interventions designed to treat or prevent mental illness and to enhance well-being. A helpful practitioner and consumer resource that provides information and quality ratings for over 180 e-physical health and e-mental health interventions can be accessed via Beacon [10], an online portal to eHealth interventions [11].

The most common type of e-mental health intervention is the Internet- or Web-based intervention or e-therapy (see [12]). e-Therapy programs can be broadly categorized as being self-help or therapist-assisted, and hundreds have been evaluated across a range of mental health disorders and symptoms, including panic disorder (eg, [13,14], SAD (eg, [15]), PTSD and symptoms (eg, [8,16,17]), anxiety prevention (eg, [18,19]), depression and depressive symptoms (eg, [20-24]), insomnia (eg, [25]), and alcohol issues (eg, [26]). Additionally, therapist-assisted e-therapy treatment programs have been found to be as effective as best-practice face-to-face therapy [13,27]. Numerous reviews [28-30] and meta-analyses (eg, [31,32]) attest to the general effectiveness of e-therapies based on validated therapeutic models such as CBT.

Although hundreds of e-therapy programs have been developed, the vast majority are generally accessible only via participation in research trials. In addition, most of the programs developed are singular offerings rather than broad-based virtual clinics offering multiple services. However, several groups offer an array of e-therapy programs contained within a single platform, such as e-hub [33], eCentreClinic [34], and Anxiety Online [35]. e-hub, operating through the Australian National University, Canberra, Australia, provides a variety of open-access self-help programs for mental health and well-being, such as MoodGYM, BluePages, BlueBoard, and e-couch [36], to the worldwide public. However, these e-mental health programs were largely designed to prevent ill health, rather than to treat clinical populations, and online therapist assistance is not offered (although BlueBoard, an Internet support group facility, includes human moderators who oversee consumer postings and appropriate online behaviors). On the other hand, the eCentreClinic, operating through Macquarie University, Sydney, Australia, offers a range of self-help and therapist-assisted e-therapy treatment programs for the anxiety disorders and depression. However, access is restricted to participation in research trials, opened only to the Australian public and at different times during the year.

Anxiety Online, operating through the National eTherapy Centre at Swinburne University of Technology, and funded by the Australian Government Department of Health and Ageing, provides to the international public a full-service, open-access, virtual psychology clinic for anxiety disorders. More specifically, Anxiety Online comprises four major components: (1) an open-access psychoeducational website that provides information about Anxiety Online, anxiety disorders (symptoms, prevalence, how and where they are treated), links to useful resources, and an entry/registration point for consumers, health care practitioners, and administrators, (2) a freely available online psychological assessment and referral system (e-PASS) that assesses the person for symptoms associated with 21 disorders in the Diagnostic and Statistical Manual of Mental Disorders: DSM-IV-TR, 4th edition, text revision (DSM-IV-TR) [37] (ie, PD/A, agoraphobia without history of panic disorder, SAD, specific phobia, GAD, PTSD, OCD, depression, anorexia nervosa, bulimia nervosa, binge eating disorder, somatization, body dysmorphic disorder, pathological gambling, insomnia, hypersomnia, alcohol dependence, and substance dependence—cannabis, opioids, sedatives, and stimulants), (3) five interactive, fully automated, 12-module self-help or therapist-assisted (via email) e-therapy treatment programs for GAD, PD/A, PTSD, OCD, and SAD, and (4) online e-therapist/CBT training programs and a health care practitioner portal.

Before commencing work as an e-therapist at the National eTherapy Centre, all prospective therapists are provided online e-therapy and CBT training and must pass a competency-based assessment (see [38] for more details). Many of the Anxiety Online e-therapists are postgraduate psychology students from various Australian universities who are undertaking an e-therapy psychological internship or placement. At a minimum, all have provisional registration as a psychologist. Anxiety Online also provides health care professionals worldwide (eg, general practitioners, psychologists, social workers, mental health nurses, aboriginal health workers, and psychiatrists) free access to the Anxiety Online programs (visit [39] to register).

Anxiety Online was launched for the international public in October 2009. This paper reports on the pre- to posttreatment outcome results for the completers of the five fully automated self-help treatment programs from October 2009 to April 2011. The goal of the Anxiety Online service is to increase access to mental health services by reducing the common obstacles and to provide consumers with choice in regard to treatment, as is the case for real-world settings. Therefore, we used a naturalistic design to evaluate the mental health treatment outcomes. The primary treatment outcome measure was the anxiety disorder severity ratings, with secondary outcome measures relating to general psychological distress levels, total number of DSM-IV-TR [37] diagnoses, confidence in managing one’s own mental health care, and quality of life. We expected that after completing one of the five Anxiety Online treatment programs, participants would show decreases in their anxiety disorder severity rating, general psychological distress levels, and the total number of mental health diagnoses at posttreatment, as well as improvements in confidence in managing their own mental health care and quality of life.

Methods

Participants and Flow

Anxiety Online is an open-access website platform. We recruit participants via periodic Facebook advertisements, referral links on other mental health websites, use of local and national media, and presentations and brochure mail-outs to health care practitioners and consumer groups.

When visiting Anxiety Online, participants wanting to undertake e-PASS are first required to register and consent to the Anxiety Online terms and conditions [40]. After providing consent, participants are taken to e-PASS, which is the gateway to the fully automated self-help and therapist-assisted treatment programs. e-PASS was designed to ensure that all participants were offered an appropriate e-therapy treatment program based on their reported symptoms, as well as a way to help them identify whether they are experiencing difficulties within a range of psychological symptoms and disorders. In addition to addressing 21 DSM-IV-TR [37] disorders, a variety of demographic and personal information (eg, whether they are currently accessing mental health treatment) is recorded.

The key inclusion criteria for access to the treatment programs are being 18 years of age or older, completing e-PASS, and having either a subclinical or clinical diagnosis of at least one of GAD, PD/A, OCD, PTSD, or SAD. From October 2009 until April 2011, there were 7140 legitimate e-PASS pretreatment completions. The Anxiety Online data file initially contained 7245 completed pretreatment e-PASS administrations; however, 105 were removed (ie, 81 reported being under 18 years of age; and 24 were identified as health care professionals or researchers not interested in using Anxiety Online for their own personal purposes). From the 7140 e-PASS pretreatment completers, 168 did not have any disorder or symptoms and an additional 593 did not have a clinical or subclinical diagnosis of GAD, PD/A, OCD, PTSD, or SAD, for which Anxiety Online has treatment programs. This left 6379 consumers being offered an Anxiety Online treatment program. From this, 2660 elected to start a program and 3719 elected not to. The overall Anxiety Online e-therapy program acceptance rate was therefore 42% (2660/6379). It is important to note, however, that only 2986 of 6379 participants had an anxiety disorder as their primary diagnosis and, therefore, 89% (2660/2986) of participants with a primary anxiety disorder elected to commence one of the e-therapy programs. This is important because e-PASS strongly encourages participants, via their e-PASS feedback report, to seek treatment for their primary condition first.

Of the 2660 who started an e-therapy program, 75 elected to take the therapist-assisted version (due to the small numbers, these data are not presented in this paper). Of the remaining 2585, at time of data analyses 350 of the participants in the fully automated self-help program were still in progress; thus, the total number of participants who had completed their 12-week treatment period was 2235. From this sample, 832 commenced GAD Online, 406 commenced Panic Stop!, 168 commenced OCD Stop!, 227 commenced PSTD Online, and 602 commenced SAD Online. Anxiety Online also collects e-PASS data from registered program users every year for 5 years, and this follow-up data will be reported in due course. The procedures for reporting of the Anxiety Online data were approved by the Swinburne University Human Research Ethics Committee. Trial registration was obtained retrospectively because Anxiety Online is an ongoing open-access mental health service rather than a pure research trial. The program automatically collects consumer data regarding treatment outcomes via e-PASS and therefore differs from the traditional trial study design that has a clear start and end date.

Measures

Assessment included an online automated self-report clinical interview (e-PASS) assessing for 21 DSM-IV-TR [37] disorders, plus several other online questionnaires at pre- or posttreatment, or both.

Online Psychological Assessment and Referral System

e-PASS (B Klein, DPsych (Clinical), unpublished data, September 2010) is an online self-report diagnostic tool that assesses for 21 DSM-IV-TR [37] disorders and serves as the gateway into the fully automated self-help and therapist-assisted treatment programs. Although e-PASS can be completed over several sittings, it must be completed within a 24-hour period. e-PASS is automated and consists of over 540 items directly using the criteria specified in the DSM-IV-TR [37]. In addition to addressing 21 DSM-IV-TR [37] disorders, a variety of demographic (and personal) questions are asked, as well as several screening items (eg, suicide, distorted thinking). It also checks for whether medical conditions and substance affects may better account for reported symptoms. For those who report suicidal ideation or distorted thinking patterns, e-PASS strongly recommends that the test taker cease completing e-PASS and contact a more appropriate service (referral sources are provided).

e-PASS is a complicated system, using both a categorical and dimensional approach to diagnosis, as well as branching logic and algorithmic scoring rules to minimize the number of irrelevant items presented, and is sensitive to other possible causes for symptoms being reported (eg, medical conditions). As a result, the number and types of items presented differ depending on the symptoms being endorsed and this, in turn, affects the feedback provided to the participant via the comprehensive personalized report generated on completion of e-PASS.

The e-PASS feedback report includes likely primary diagnosis, any likely secondary diagnoses (ranked according to number of symptoms and self-reported severity), and whether each likely diagnosis is within a clinical or subclinical range. Disorders in the subclinical range refer to those individuals who report most, but not all, of the required DSM-IV-TR criteria or, alternatively, report all of the necessary DSM-IV-TR criteria but provide low distress and interference ratings regarding their specific disorder symptoms. People with subclinical disorders are symptomatic (or subthreshold) but do not meet full DSM-IV-TR criteria for a clinical disorder. Disorders at the clinical level are further defined as mild, moderate, or severe, and participants are given hyperlinked or pop-up information explaining in plain language what each of these terms means. Likely clinical disorder severity ratings range from 0 (absence of any symptoms) to 8 (very severe clinical disorder) and constitute one of the main outcome variables for this study. Likely clinical disorder severity scores below 3.50 are given a subclinical label and rating. Feedback reports strongly recommend that consumers address their primary condition first, but ultimately what course of action or treatment participants undertake remains their choice (ie, Anxiety Online enables access to treatment programs for each consumer’s primary diagnosis and any secondary diagnoses). Participants are also provided with a summary of the symptoms for each condition they have, and qualifiers are provided where appropriate (eg, chronicity of PTSD). Consumers are also provided a recommended course of action and multiple referral options. As individuals remain completely free to choose whatever course of action they desire, within the confines of the symptoms reported, Anxiety Online is a participant choice-based system. Nevertheless, it also provides the participant with detailed and evidence-based guidance and recommendations.

e-PASS is undergoing psychometric validation and qualitative evaluation, and the pilot and preliminary data suggest it is an acceptable and valid diagnostic tool (B Klein, DPsych (Clinical), unpublished data, September 2010, [41]), although caution is still warranted until the full and detailed study is published. Basic community-based validation results also attest to its validity. Using the results from the current study, at posttreatment 64 participants reported that they sought confirmation of their e-PASS diagnoses with an external source (n = 33 with a psychologist, n = 16 with a medical doctor, n = 6 with a website, n = 4 with a counselor, n = 4 with a friend, and n = 1 with a book). The overall agreement rate was 95% (61/64). The three sources where agreement was not reached were a medical doctor (n = 1), a website (n = 1), and a friend (n = 1).

Online Questions/Questionnaire: Self-Report

Kessler-6 (K6) [42] is a brief 6-item self-report measure, using a Likert-type scale from 1 to 5, measuring nonspecific psychological distress over the last 30 days [42]. Scores range from 6 to 30, with higher scores indicating greater psychological distress. Normative data (G Andrews, MD, written communication, August 2010) suggest that scores between 6 and 11 indicate low distress levels (71.7% of the population); scores between 12 and 15 indicate moderate distress levels (16.6% of the population); scores between 16 and 19 indicate high distress levels (7.16% of the population); and scores between 20 and 30 indicate very high distress levels (2.5% of the population). The K6 has demonstrated strong psychometric properties (eg, [42,43])

Number of e-PASS diagnoses is the total number of e-PASS diagnoses (at clinical and subclinical levels) as assessed by e-PASS at pre- and posttreatment assessment.

Confidence in managing mental health is a single-question self-report item asking participants to rate their overall level of self-confidence in managing their own mental health. Scores are anchored (1 = very poor, 2 = poor, 3 = neither poor nor good, 4 = good, 5 = very good), with higher scores indicating greater self-confidence.

Quality of life is a single-item self-report question asking participants to rate their overall quality of life. Scores are anchored (1 = very poor, 2 = poor, 3 = neither poor nor good, 4 = good, 5 = very good), with higher scores indicating a higher quality of life.

The two e-Therapy treatment satisfaction questions ask participants to rate (1) how satisfied they were with the e-therapy program, using a scale from 1 (not at all) to 5 (very highly), and (2) how much they liked the e-therapy treatment program, using a scale of 0 (not at all), 2 (a little), 4 (somewhat), 6 (quite a lot), and 8 (very much so) at posttreatment.

Design

The five fully automated self-help e-therapy treatment programs were trialed using a pre- to posttreatment quasi-experimental (participant choice) naturalistic design. The five programs all have a similar structure and look. Each program addresses a particular anxiety disorder (ie, GAD Online treats GAD, Panic Stop! treats PD/A, OCD Stop! treats OCD, PTSD Online treats PTSD, and SAD Online treats SAD). Each program is based on well-established CBT principles and protocols and was reviewed by national and international experts. In addition, all programs were subjected to rigorous technical and consumer usability testing prior to launch.

All programs consist of 12 modules, delivered over 12 weeks, that include a variety of text-based and multimedia materials (audio, video, and animated graphics) and online activities—for example, video (expert speaking, patient speaking, examples of therapy techniques or sessions, etc), audio (breathing control, visual imagery, progressive relaxation therapy instructions, etc), online activities (weekly self-monitoring, quizzes, journal writing, etc), downloadable PDFs (worksheets, transcripts of the audio, monitoring forms, etc), and online interactive animations (flash animations to convey key concepts) (see Multimedia Appendix 1 for several screenshot examples). In addition, there were numerous automated emails welcoming participants to the program, reminding and encouraging them to log on and complete their assessments, as well as various “alert” emails that are triggered depending on participants’ online behavior (eg, alert automated emails are triggered when participants’ self-monitoring of their anxiety and depression remain static for 4 weeks, or remains in the upper extreme range for 2 weeks in a row). These automated emails alert the person to a particular issue and provide recommendations (eg, to consider seeking more intensive assistance). All e-therapy programs contain standard CBT content teachings with regard to psychoeducation, anxiety management, and physiological, cognitive, and behavioral change strategies specific to each anxiety disorder, as well as weekly online and offline homework activities.

Procedures

Participants self-register to use the Anxiety Online virtual clinic. All participants are required to read and agree to the terms and conditions of the Anxiety Online service before being able to proceed to e-PASS. If participants meet criteria (18 years of age and over, and receiving a diagnosis of GAD, PD/A, OCD, PTSD, or SAD), they are offered the e-therapy program(s) that treat their specific disorder(s).

For this study, once participants chose a fully automated self-help e-therapy treatment program, they gained immediate access to the program and their 12-week e-therapy treatment cycle commenced. During this 12-week period, participants could not undertake another e-therapy program that they may have been offered in their e-PASS report (participants are unable to do more than one e-therapy program concurrently). If participants no longer wanted to continue, they were required to opt out of their e-therapy program via an opt-out option provided within each e-therapy program.

After completing the e-therapy program (at the end of week 12), participants who had not opted out were sent automated emails asking them to complete their posttreatment assessment questions and e-PASS. Several reminder emails were sent out over a 3-week period to those who had not completed the posttreatment assessment in a timely manner.

Statistical Procedures, Analyses, and Evaluation of Treatment Effects

After multivariate analysis of variance (MANOVA) tests showed no significant attrition bias, treatment effects from pre- to postassessment were evaluated separately for each of the five fully automated self-help e-therapy treatment programs, using a repeated measures MANOVA for the five treatment outcome variables (ie, clinical disorder severity rating, K6 scores, number of diagnoses, confidence in managing one’s own mental health care, and quality of life). Follow-up analysis of variance (ANOVA) tests were then conducted for the programs with significant results. Normality and homogeneity assumptions were supported by the data and effect sizes were established using Cohen’s [44] classification scheme (small effect = 0.20, medium effect = 0.50, and large effect = 0.80). The 95% confidence intervals for the expected program changes are also presented. In addition, the e-therapy treatment satisfaction results are presented. We used SPSS version 19 for Windows (IBM Corporation, Somers, NY, USA) to analyze all data.

Attrition

The overall completion rate, for the purposes of this study, was defined as the number of participants who started a program and completed the 12-week posttreatment assessment (postassessment completion rate of 10.1% [225/2235], or 89.9% attrition rate). However, true attrition, as defined by those participants who opted out or dropped out of a program after commencement, was much lower at just 4% (97/2235). The bulk of participants (n = 1913) were those who commenced a program, did not opt or drop out, and did not complete their posttreatment assessment. In this situation the most reliable measure of attrition probably ignores this group, producing an attrition rate of 30.1% (225/322 = 69.9% completed). As Figure 1 shows, the attrition rates for the five treatment programs were fairly similar when this attrition measure was used (31% for GAD, 23% for Panic Stop!, 37% for OCD Stop!, 21% for PTSD and, 35% for SAD).

These data clearly illustrate the inherent completion difficulties facing e-mental health evaluation research using open-access research designs. It is therefore necessary to check for attrition bias using robust statistical techniques (eg, [45,46]). In cases where attrition bias is not found to be significant, the use of completer analysis is considered a legitimate and accurate means to analyze the data. However, in cases where significant attrition bias is found, the more conservative intention-to-treat method should be applied. Below we present two methods of checking for attrition bias, although either one is sufficient.

‎View this figure

Methods for Checking for Attrition Bias

We used two methods to check for attrition bias. The first method was initially proposed by Heckman [45]. In this approach, nominal logistic regression is used to predict the probability that each participant will complete the posttreatment assessment using a variety of pretreatment assessment and demographic measures. The Mills ratio is then produced using the ratio of the normal probability and cumulative distribution function for the residuals (1 minus the predicted probability for posttreatment assessment completion). This Mills ratio is then included as a covariate in a multivariate general linear model to determine the effect of the program on the posttreatment minus pretreatment changes in the outcome measures. If the Mills ratio is not significant, it indicates that the responses for those who did complete the posttreatment assessment are indicative of what could be expected for those who did not complete the posttreatment assessment; in other words, there is no attrition bias.

The second approach has been used by authors such as Rubin [46] to allow for propensity subclassification. In this study, propensity is the estimated probability of attrition developed using the nominal logistic regression procedure. In this study, we used the quartiles for this estimated probability to create subclassifications on which a multivariate general linear model is used to test for main effects and an interaction effect between the programs and the attrition propensity quartile effects. If no significant interaction effect and no significant quartile effect are found, it means that the program effects are similar across quartiles, suggesting that there is no significant attrition bias.

Check For Attrition Bias

We found nine pretreatment demographic variables to be significantly associated with attrition. As indicated in Table 1, both types of attrition (e-therapy program completed but no posttreatment assessment, and genuine attrition as defined by formally dropping our or opting out during treatment) were considered in this analysis. The results suggest that those who completed the posttreatment e-PASS tended to differ from the participants who did not in the following ways. It was more likely that on average the completers sought online assistance with the prime objective of finding a self-help program; were married or cohabiting with their partner; were not a homemaker, on a disability pension, or unemployed; were living in a regional area; were more likely to say that they had adequate support; were more likely to say that they learned by reading; had a lower pretreatment K6 score; had a higher age; and had fewer disorders diagnosed at pretreatment assessment. These differences made it necessary for special tests for attrition bias to be performed. These tests showed that none of the attrition-linked variables was associated with changes in the outcome variables, thereby confirming that there was no attrition bias.

We used a nominal logistic regression analysis to predict the attrition category for all participants on the basis of the above nine variables. The estimated probability of completion for the posttreatment assessment was saved for each person who actually completed the posttreatment assessment.

Using the Heckman [45] approach, the Mills ratio for the 225 people who completed the posttreatment assessment was calculated as described above, and a MANOVA was run for the change in all the metric outcome variables to test for differences in the program effects while controlling for the Mills ratio. We found that the Mills ratio had no significant effect (F_9,207 = .686, P = .72) and that there was no significant interaction between the program and the Mills ratio (F_36,840 = .854, P =.71). This confirms that there is unlikely to be any attrition bias for any of the e-therapy programs.

Next, we used the propensity subclassification approach [46] to split the sample of the 225 participants who completed the posttreatment assessment into four groups. The groups were differentiated in terms of the likelihood of attrition using the quartiles for this estimated probability and were of similar size. Effectively this differentiation controls for the likelihood of attrition, allowing us to determine whether there is attrition bias. A 2-way MANOVA was run for the change in all the outcome variables allowing for an interaction effect between the attrition propensity quartiles and the programs. Neither the interaction effect nor the main effect for the attrition propensity quartiles was found to be significant (F_108,1845 = 1.078, P = .28; F_27,597 = .810, P = .74), confirming that there is unlikely to be any attrition bias for any of the programs.

In summary, both techniques for assessing attrition bias delivered nonsignificant findings and demonstrate that attrition bias was highly unlikely for all five of the e-therapy programs. Given this result, we analyzed the data for each of the five fully automated e-therapy anxiety disorder treatment programs using a completer analysis.

Power Analysis

Target sample size required was determined by GPower [47]. To achieve power of 80% (alpha =.05), 34 participants per e-therapy treatment group were required to detect a moderate effect size on the primary outcome measure (clinical severity rating). Three of the five e-therapy treatment program groups had >34 participants (GAD Online, n = 88; SAD Online, n = 50; Panic Stop!, n = 40); however, PTSD Online and OCD Stop! reached only 30 and 17, respectively, so these results should be interpreted with greater caution.

Results

Participant Characteristics

A total of 225 people met the inclusion criteria and completed both pre- and posttreatment assessments. Across the five e-therapy programs, 69 men participated (69/225 = 31%) with the average age of all participants being 42.1 (SD 12.5) years (men, mean 44.5, SD 13.4; women, mean 41.0, SD 12.0 years). The overwhelming majority of participants were Australian residents (215/225 = 95.6%).

Table 2 presents demographic information for each of the five e-therapy program groups.

Treatment Outcomes

The number, means, standard deviations, F scores, P values, Cohen d, and confidence intervals for the five key dependent variables (per e-therapy program group) at the two assessment periods are shown in Table 3.

GAD Online Program

For the GAD Online program a repeated measures MANOVA revealed a significant multivariate time effect for the e-PASS severity rating, K6, e-PASS total, quality-of-life, and confidence outcome measures (F_5,83 = 19.92, P < .001). Follow-up repeated measures ANOVAs revealed significant improvements on all five variables. Three of the five treatment outcome variables produced large effect sizes, with one medium effect size and one small effect size (see Table 3).

Panic Stop! Program

For the Panic Stop! program a repeated measures MANOVA with these variables revealed a significant multivariate time effect (F_5,35 = 8.87, P < .001). Follow-up repeated measures ANOVAs on three of the five variables revealed significant improvements on three variables. Two of these variables produced large effect sizes, with one medium effect size, one small effect size, and one very small (under .20) (see Table 3).

OCD Stop! Program

For the OCD Stop! program a repeated measures MANOVA with these variables revealed a significant multivariate time effect (F_5,12 = 4.21, P = .02). Follow-up repeated measures ANOVAs revealed significant improvements on four of the five variables. Four of these variables produced large effect sizes with one small effect size (see Table 3).

PTSD Online Program

For the PTSD Online program a repeated measures MANOVA with these variables revealed a significant multivariate time effect (F_5,25 = 4.89, P = .003). Follow-up repeated measures ANOVAs revealed significant improvements on all five variables. Four of these outcome variables produced large effect sizes with one medium effect size (see Table 3).

SAD Online Program

For the SAD Online program, a repeated measures MANOVA revealed a significant multivariate time effect for these variables (F_5,45 = 5.14, P = .001). Follow-up repeated measures ANOVAs revealed significant improvements on four of the five variables; however, only one of these variables produced a large effect size, with three medium effect sizes and one small effect size (see Table 3).

e-Therapy Program Treatment Satisfaction and Time Spent using the e-Therapy Program

Satisfaction with the e-therapy programs was rated as moderately high on average within all five groups (see Table 4), with the PTSD Online group obtaining the highest average score (3.73/5.00 = 74.6%). In terms of how much the participants liked their e-therapy program, average scores fell into the “somewhat” to “quite a lot” range, with the PTSD Online group likeability score the highest (5.67/8.00 = 70.9%). Chi-square tests indicated no significant differences between the five e-therapy program groups on the two e-therapy program satisfaction questions (χ²₁₆ = 12.8, P = .69; χ²₁₆ = 16.3, P = .43). Posttreatment assessment completers were also asked how much time they spent using their respective e-therapy programs over the 12 weeks. GAD Online participants reported the most amount of time and the OCD Stop! participants the least amount of time (see Table 4). An ANOVA indicated no significant differences between the five e-therapy program groups with respect to the amount of time in minutes spent using their program over the 12-week treatment period (F_4,220 = 0.176, P = .95). The average total time in minutes across the five different e-therapy programs was 395.60 (SD 277.2) minutes or 6.59 hours over 12 weeks.

Reasons for Opting Out

Participants who opted out of the Anxiety Online program (n = 97) were asked to check one or more items regarding what barrier(s) prevented them from completing their Anxiety Online program when they opted out. Table 5 presents the item(s) endorsed.

Basic Professional Labor Time Cost Analysis

The Australian Psychological Society [48] schedule of recommended fees for psychological services assumes the following costs: consultation session 45–60 minutes = AU $212; clinical assessment session 76–90 minutes = AU $308. If one were to use these fees to calculate the professional human labor time costs associated with all the e-PASS pretreatment assessments undertaken (n = 7140) and uptake of all the Anxiety Online e-therapy treatment programs since launch (n = 2563; 2660 – 97 opt-outs), the human labor time costing would equate to AU $8.7 million. The cost of developing Anxiety Online and ongoing maintenance over this period has been close to AU $2.0 million, with the bulk of this amount being a one-off development cost (AU $1.66 million). The labor time cost saving resulting from the Anxiety Online service in the first 18 months of operation is therefore estimated at AU $6.7 million, and into the future the cost savings should be greater, given that the start-up costs will not be recurring expenses.

Discussion

We observed significant reductions in the GAD, PD/A, OCD, PTSD, and SAD e-PASS diagnostic severity ratings specific to each e-therapy program group and increased self-confidence ratings in managing ones’ own mental health care for all five e-therapy program groups. Cohen d within-group treatment effect sizes were in the high-medium to large categories. Quality-of-life ratings significantly increased for four of the five e-therapy program groupings, with Panic Stop! participants showing little improvement based on the mean scores. Total number of diagnoses was significantly reduced for four of the five programs, and the K6 scores significantly reduced for three of the five e-therapy programs. When looking at the treatment effect sizes for the five program groups over the five measures, 14 were large, six were medium, four were small, and one was under 0.20. These results compare very favorably with other self-help e-therapy treatment programs, which typically have effect sizes between 0.40 and 0.70 [15,31].

Overall, e-therapy treatment satisfaction ratings were good. The total average across all five e-therapy treatment program groups was 72% (3.60/5) for satisfaction and likeability was 67% (5.39/8). Interestingly, these percentages are comparable with those we see for our e-therapy programs provided with therapist assistance [8,14,16,49]. Taking treatment outcome and satisfaction results into account, PTSD Online and GAD Online appear to be the strongest performers of the five e-therapy programs. The total time spent using the fully automated e-therapy programs was under 7 hours over the 12-week treatment period (or just under 33 minutes per week on average).

For those participants who opted out, the main barriers endorsed were none (got what they needed; 29%), time pressures (21%), lack of motivation (14%), and feeling anxious about the content (11%). The least endorsed barriers related to computer anxiety (1%); too much text to read (2%); that the program would take too long to complete (2%); that the material was too hard to understand (3%); that the program was hard to navigate around (3%), unhelpful (3%), boring (3%), or perceived as being not useful (5%);, Internet or computer problems (5%), or preferring (7%) or being able to access to face-to-face therapy (8%). These figures generally support the idea that the Anxiety Online programs themselves are not harmful or detrimental to those participants who opted out. Rather, close to a third of those who opted out prematurely endorsed that they “got what they needed” before the 12-week treatment completion time, and just over a third endorsed time pressures and lack of motivation as reasons for noncompletion.

Although very crude, the basic cost comparison between the development and maintenance costs of Anxiety Online to date relative to the cost of professional human labor using traditional delivery models suggests a 77% saving of AU $6.7 million. However, Anxiety Online and other e-mental health platforms could easily sustain a 10-fold increase in usage without substantially affecting maintenance costs; therefore, the cost savings could be far higher than this into the future. When considered in light of other advantages of e-therapy such as increased consumer access to care due to the removal of traditional barriers such as time, cost, and geographic location restraints, the online-delivery model seems likely to play an increasingly prominent role in modern mental health systems.

Implications

The major implications of being able to effectively deliver psychological treatment via the Internet relate largely to accessibility. The availability of e-mental health services means that anyone in any location with Internet access can access treatment immediately and at the times and intensities they choose (ie, not limited to scheduled appointments). It also means greater access for those with no or limited access to treatment programs or mental health specialists (eg, living in rural areas, incarcerated), or those who move residences frequently (eg, itinerant workers, armed forces personnel) or have limited mobility (eg, with chronic physical illness, older, disabled). e-Mental health interventions may also help those with mental health problems who are reticent to present to services for reasons such as perceived stigma [50], although recent research supports that some stigmatizing attitudes actually lead to increased likelihood to seek professional help [51].

Limitations

This study has some limitations that should be taken into consideration. This study was conducted as an open-access, participant choice, naturalistic trial, so the lack of a control group makes it impossible to conclude whether the improvements are a result of the active e-therapy programs or merely a result of other effects. Nevertheless, this design limitation does represent “real-world” mental health services, as consumers are allowed to choose their treatment program rather than being randomly assigned to a treatment program in a randomized controlled trial. Power analyses indicted that the numbers required for the OCD Stop! and PTSD Online e-therapy programs were suboptimal and so some extra caution in interpretation needs to be taken. In previous studies we have used intention-to-treat analysis to address the issue of missing data. Intention-to-treat analysis is an overly conservative approach [52] that potentially underestimates the effectiveness of open-access e-therapy programs, whereas completer analysis may overestimate the effectiveness of these programs if there is attrition bias. Attrition bias was found to be nonsignificant for these data, so there are strong grounds for accepting that the completer analysis results obtained accurately reflect the true effectiveness of the Anxiety Online fully automated self-help programs.

Finally, four of the five dependent variables used in this study lack strong psychometric validation, and this needs to be taken into account when interpreting the results. Further replication studies are required that involve the use of several validated measures before we can definitively confirm that the Anxiety Online fully automated self-help e-therapy programs are effective. Future analyses with the individual Anxiety Online programs will be able to validate the present results with ideal standardized measures, as well as the aforementioned current formal e-PASS psychometric evaluation.

Future Directions

While results of this open trial are encouraging, it is important that they be followed by randomized controlled trials comparing all five e-therapy programs with both a waitlist control and current best-practice face-to-face treatment in order to unequivocally establish treatment effectiveness. We are running a randomized controlled trial of the OCD Stop! e-therapy program and will be subjecting the other four Anxiety Online e-therapy programs to the same rigorous testing. Furthermore, while e-therapy experts are developing guidelines for working with specific clinical populations (see [53]) as well as Internet intervention research guidelines [54], we also need more research to address the truly important questions about who this modality works best for because, like all forms of treatment delivery, it is unlikely to be universally appealing or effective. We are preparing another paper discussing our preliminary investigations regarding what variables predict attrition for the fully automated self-help e-therapy treatment programs. In addition, we will also be conducting qualitative studies that will include interviewing a random selection of those participants who commenced one of our treatment programs, did not opt out, and did not complete their postassessment. At this point it would be highly speculative to state why so many participants did not complete their postassessment after selecting one of the Anxiety Online programs, apart from pointing to the fully automated nature of the Anxiety Online system (ie, the complete absence of human-based screening, assessment, and therapeutic assistance) to prompt, encourage, and at times enforce adherence behaviors to the treatment protocol.

The Anxiety Online platform will soon be subsumed under the name Mental Health Online, given the addition of new e-therapy treatment programs for nonanxiety conditions (eg, depression, bulimia, insomnia, multidisorder) and several more to follow over the coming months and years (eg, problem gambling, drugs and alcohol, hoarding). We are also integrating other communication modalities into the therapist-assisted programs (ie, instant messaging, audio-only chat, video chat) and the use of 3-dimensional virtual reality platforms and collaborative work spaces. We will offer these new modes of communication and e-therapy training in 2012 and also plan to open up the Anxiety Online/Mental Health Online infrastructure to practitioners in Australia and, potentially, around the world. In the future it will also be possible to access the Anxiety Online/Mental Health Online programs through a national e-mental health portal instigated by the Australian Government Department of Health and Ageing [55]. This portal will bring together many of the evidence-based, yet fragmented, e-mental health interventions currently operating in Australia, thus making it far easier for mental health consumers to find and receive the most appropriate course of treatment with the associated level of assistance that is best for them.

Conclusions

The results of this open-access participant choice evaluation trial suggest that the Anxiety Online e-therapy programs are promising and effective treatments for people with subclinical and clinical diagnoses of GAD, PD/A, OCD, PTSD, and SAD. e-Mental health treatment-delivery formats are increasing accessibility to mental health care and appear to provide a highly cost-effective and sustainable treatment-delivery model. It is envisaged that e-mental health treatment programs will soon become a common feature of modern mental health systems, and that such a development will bring with it unprecedented levels of service provision to those in need of specialist mental health treatment.