This paper is in the following e-collection/theme issue:

Web-based and Mobile Health Interventions (3297) Transdiagnostic Mental Health Interventions (160) Posttraumatic Stress Disorder (PTSD) (116) Depression and Mood Disorders; Suicide Prevention (1363) Anxiety and Stress Disorders (1053) Behavior Change (712)

Published on in Vol 23, No 5 (2021): May

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/27610, first published .
Initial Outcomes of Transdiagnostic Internet-Delivered Cognitive Behavioral Therapy Tailored to Public Safety Personnel: Longitudinal Observational Study

Initial Outcomes of Transdiagnostic Internet-Delivered Cognitive Behavioral Therapy Tailored to Public Safety Personnel: Longitudinal Observational Study

Initial Outcomes of Transdiagnostic Internet-Delivered Cognitive Behavioral Therapy Tailored to Public Safety Personnel: Longitudinal Observational Study

Authors of this article:

Heather D Hadjistavropoulos1 Author Orcid Image ;   Hugh C McCall1 Author Orcid Image ;   David L Thiessen2 Author Orcid Image ;   Ziyin Huang3 Author Orcid Image ;   R Nicholas Carleton1, 3 Author Orcid Image ;   Blake F Dear4 Author Orcid Image ;   Nickolai Titov4 Author Orcid Image
Article Authors Cited by (21) Tweetations (4) Metrics

    Original Paper

    1Department of Psychology, University of Regina, Regina, SK, Canada

    2Department of Mathematics and Statistics, University of Regina, Regina, SK, Canada

    3PSPNET, University of Regina, Regina, SK, Canada

    4eCentreClinic, Department of Psychology, Macquarie University, Sydney, Australia

    Corresponding Author:

    Heather D Hadjistavropoulos, PhD

    Department of Psychology

    University of Regina

    Administration-Humanities Building, AH 345

    3737 Wascana Parkway

    Regina, SK, S4S 0A2

    Canada

    Phone: 1 306 585 5133

    Email: Heather.Hadjistavropoulos@uregina.ca


    Background: Canadian public safety personnel (PSP) experience high rates of mental health disorders and face many barriers to treatment. Internet-delivered cognitive behavioral therapy (ICBT) overcomes many such barriers, and is effective for treating depression, anxiety, and posttraumatic stress disorder (PTSD) symptoms.

    Objective: This study was designed to fill a gap in the literature regarding the use of ICBT tailored specifically for PSP. We examined the effectiveness of a tailored ICBT program for treating depression, anxiety, and PTSD symptoms among PSP in the province of Saskatchewan.

    Methods: We employed a longitudinal single-group open-trial design (N=83) with outcome measures administered at screening and at 8 weeks posttreatment. Data were collected between December 5, 2019 and September 11, 2020. Primary outcomes included changes in depression, anxiety, and PTSD symptoms. Secondary outcomes included changes in functional impairment; symptoms of panic, social anxiety, and anger; as well as treatment satisfaction, working alliance, and program usage patterns.

    Results: Clients reported large symptom reductions on measures of depression and anxiety, as well as moderate reductions on measures of PTSD and secondary symptoms, except for social anxiety. Most clients who reported symptoms above clinical cut-offs on measures of depression, anxiety, and PTSD during screening experienced clinically significant symptom reductions. Results suggested good engagement, treatment satisfaction, and working alliance.

    Conclusions: Tailored, transdiagnostic ICBT demonstrated promising outcomes as a treatment for depression, anxiety, and PTSD among Saskatchewan PSP and warrants further investigation.

    Trial Registration: Clinicaltrials.gov NCT04127032; https://www.clinicaltrials.gov/ct2/show/NCT04127032

    J Med Internet Res 2021;23(5):e27610

    doi:10.2196/27610

    Keywords

    internet; cognitive behavior therapy; anxiety; depression; posttraumatic stress disorder; public safety personnel; CBT; internet-based cognitive behavioral therapy; ICBT; PTSD; outcome; diagnosis; longitudinal; observational; literature; effectiveness 


    Public safety personnel (PSP) include border security personnel, communications officials (eg, emergency response dispatchers), correctional workers, firefighters, paramedics, police officers, and others whose work keeps communities safe [1]. Through their work, PSP frequently experience or witness potentially traumatic events such as motor vehicle collisions and acts of violence [2-4]. Research shows that such events put individuals at risk of developing several emotional disorders, including posttraumatic stress disorder (PTSD), major depressive disorder, and panic disorder [5-10]. In Canada, a large survey study indicated that 44.5% of PSP report clinically significant symptoms of one or more mental health disorders [11]. PSP have also reported high rates of mental health problems in other countries [12-14]. PSP face many barriers to treatment, such as stigma, distance from services, and time constraints [15]. Stigma appears to be a particularly prohibitive barrier to treatment among PSP [15,16].

    Internet-delivered cognitive behavioral therapy (ICBT) is a treatment wherein traditional cognitive behavioral therapy skills are taught via online learning platforms and is poised to help overcome these barriers to treatment for PSP. ICBT can be accessed privately and conveniently, and it is cost-effective because it requires less therapist time per client than traditional face-to-face treatments [17,18]. Meta-analyses have shown that ICBT is effective for treating symptoms of a range of mental disorders, including depression, anxiety, and PTSD [19,20]. There have been several recent calls for internet interventions to be tailored to the unique needs of their target users to facilitate greater treatment engagement [21,22]. As one group stated, this process begins with establishing “a deep understanding of user needs and preferences, and actively involving users in design processes from the outset” [21]. Tailoring ICBT may be particularly important for PSP, who have reported several unique needs such as the need for flexible treatment timelines, diverse preferences for frequency of therapist support, specialized therapists with knowledge of unique PSP issues and culture, and a focus on exposures to potentially psychologically traumatic events [15]. Canadian PSP appear receptive to therapist-supported ICBT, which was ranked as their most preferred mental health treatment after face-to-face psychological services [23].

    The Government of Canada recently announced a $10 million investment into developing, delivering, and evaluating ICBT tailored specifically for PSP [24]. Our research unit, PSPNET, was contracted to carry out this work. We conducted extensive interviews with PSP stakeholders to better understand their needs and preferences [15] and, drawing upon their feedback, tailored a transdiagnostic ICBT program that previously demonstrated positive outcomes for a range of conditions to meet the needs of PSP. The tailored course of treatment is called the PSP Wellbeing Course, which is described in detail below.

    This paper presents the initial outcomes of the PSP Wellbeing Course among the first 83 PSP enrolled in the course during the first 7 months of availability in Saskatchewan. The study objectives were to evaluate the initial (a) effectiveness, (b) treatment satisfaction and working alliance, and (c) program usage patterns of the PSP Wellbeing Course. We hypothesized high completion rates, high satisfaction, and moderate to large effect sizes for symptom change. This study is the first, as far as we are aware, to evaluate the tailored ICBT approach for treating mental health problems among PSP.

    This study’s sample represents only the first third of the sample described in our trial protocol (N=250; Clinicaltrials.gov NCT04127032). This early evaluation of outcomes is important for three reasons. First, it is important to ensure that the development and delivery of the PSP Wellbeing Course is a fruitful use of Canadian tax dollars. Second, evaluation of early outcomes is consistent with the learning health system approach, which is characterized by a continuous cycle of gathering data from practice, transforming data into knowledge, and implementing knowledge into practice [25]. Third, our preliminary findings may be helpful for others involved in the development, provision, evaluation, and funding of mental health services for PSP.


    Study Design

    We employed a longitudinal single-group open-trial design. This study was approved by the local institutional research ethics board (2019-157) and registered on Clinicaltrials.gov (NCT04127032). This article presents the initial study outcomes for the first 83 clients who enrolled in the PSP Wellbeing Course (one third of the total expected sample of 250). We have followed the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) statement in preparing the article [26].

    Setting

    This study was performed in the Canadian province of Saskatchewan. There was no specialized ICBT for PSP in Saskatchewan prior to this study. ICBT has been available to Saskatchewan residents through the Online Therapy Unit [27] since 2010, but the numbers of PSP using this service have not been systematically tracked.

    Clients

    All clients who enrolled in the PSP Wellbeing Course between December 5, 2019 and July 2, 2020 consented to our use of their data for research and were included in this study. Enrollment in the PSP Wellbeing Course required clients to (a) be 18 years or older, (b) currently or formerly employed as a PSP (as defined above), (c) have access to a computer and internet service, (d) provide an emergency medical contact, and (e) reside in the province of Saskatchewan (although we later began offering the PSP Wellbeing Course in English and French to residents of Quebec as well). Prospective clients were excluded and referred to appropriate services if they (a) reported a high suicide risk; (b) had attempted suicide or had been hospitalized for a high suicide risk within the past year; (c) reported a primary problem with psychosis, mania, or alcohol and drugs; (d) reported receiving psychological treatment more than twice per month; or (e) reported concerns about ICBT and requested referral to other services in the community.

    Intervention

    The PSP Wellbeing Course was adapted from a transdiagnostic ICBT program called the Wellbeing Course, which was developed for use in the general population by eCentre Clinic at Macquarie University, Australia [28-32]. The PSP Wellbeing Course is designed to treat symptoms of depression, anxiety, and PTSD. The course consists of five sequential lessons, which are presented in a slideshow format and focus on (1) introducing the cognitive behavioral model and symptom identification, (2) monitoring and challenging maladaptive thoughts, (3) managing physical symptoms, (4) graded exposure, and (5) relapse prevention. Lesson content is relayed through instructive text, diagrams, and illustrative case stories about PSP. Clients are also given access to downloadable materials, weekly homework assignments, and supplementary resources related to issues not addressed in the lessons (eg, assertiveness, communication, sleep problems). Therapists hold graduate degrees in psychology or social work and provide support to clients by phone or secure email up to twice per week, depending on each client’s preference. Each week, clients complete symptom questionnaires and a reflection questionnaire including both open-ended and closed-ended questions designed to gauge response to the treatment program (eg, what clients worked on; any challenges they encountered; and ratings of helpfulness, understanding, and difficulty). The course is designed to be completed in 8 weeks, but clients are provided with therapist support for up to 16 weeks based on symptoms and preference, and are given access to the course materials for 1 year. We tailored the PSP Wellbeing Course using extensive feedback from PSP stakeholders gleaned through 112 interviews [15] and 132 survey responses [23] (eg, replacing generic case stories and examples used to illustrate treatment concepts with case stories and examples about PSP).

    Procedure

    Prospective clients completed a two-stage eligibility screening process consisting of an online screening questionnaire (demographics, clinical history, and the measures described below) and a screening interview by phone. Clients then began the PSP Wellbeing Course. Clients were automatically administered brief weekly questionnaires and a longer battery of questionnaires at 8 weeks, as detailed below. Questionnaires were embedded in the course. Members of our research team followed up with clients by phone or email to encourage them to complete posttreatment measures.

    Measures

    Primary Outcome Measures
    Patient Health Questionnaire-9

    The Patient Health Questionnaire-9 (PHQ-9) measures symptoms of depression and includes one item to assess suicidality [33]. It includes nine items rated from 0 (not at all) to 3 (nearly every day), with total scores ranging from 0 to 27. A score greater than 9 is typically used to identify a likely diagnosis of major depressive disorder [34]. The PHQ-9 has demonstrated excellent psychometric properties [33,35,36]. Cronbach α for the PHQ-9 in this study ranged from .84 to .88.

    Generalized Anxiety Disorder-7

    Generalized Anxiety Disorder-7 (GAD-7) is a brief measure of general anxiety symptoms [37]. It includes seven items rated from 0 (not at all) to 3 (nearly every day) for a total score ranging from 0 to 21. The GAD-7 has demonstrated strong psychometric properties, and a score of 10 or greater is typically used to identify a likely diagnosis of generalized anxiety disorder [36,37]. Cronbach α for the GAD-7 in this study ranged from .86 to .92.

    Posttraumatic Stress Disorder Checklist for DSM-5

    The Posttraumatic Stress Disorder Checklist for the DSM-5 (PCL-5) measures symptoms of PTSD and includes 20 items rated from 0 (not at all) to 4 (extremely) [38]. A cut-off of 33 was recommended by several groups to indicate a likely diagnosis of PTSD [39,40]. The PCL-5 has shown good psychometric properties [39]. Cronbach α for the PCL-5 in this study ranged from .95 to .96.

    Secondary Outcome Measures
    Panic Disorder Severity Scale Self-Report

    The Panic Disorder Severity Scale Self-Report (PDSS-SR) is a psychometrically sound 7-item measure assessing symptoms of panic disorder [41]. Items are rated from 0 to 4 for a total score of 0 to 28. It is common practice to employ a cut-off score of 8 on the PDSS-SR [42-44], based on the cut-off initially established for the nonself-report PDSS [45]. Cronbach α for the PDSS-SR in this study was .89-.91.

    Six-Item Social Phobia Scale and Six-Item Social Interaction Anxiety Scale

    The 6-item Social Phobia Scale (SPS-6) and 6-item Social Interaction Anxiety Scale (SIAS-6) are brief measures of social anxiety that are psychometrically sound when administered together or separately [46]. Items are rated from 0 to 4 for total scores of 0 to 24 on each measure or a combined total of 0 to 48, with a clinical cut-off of 2 on the SPS-6 and of 7 on the SIAS-6 [46]. Cronbach α for the measures combined in the current study was .91 and .94.

    Dimensions of Anger Reactions Scale

    The Dimensions of Anger Reactions Scale (DAR-5) is a 5-item measure of anger that has demonstrated good psychometric properties [47]. Scores range from 5 to 25 with a cut-off of 12 [48]. Cronbach α for the DAR-5 in this study was .88 and .89.

    Sheehan Disability Scale

    The Sheehan Disability Scale (SDS) is a psychometrically sound 3-item measure assessing respondents’ functioning in their home lives, social lives, and work/school lives [49].

    Treatment Satisfaction Questionnaire

    We administered a bespoke questionnaire to explore treatment satisfaction.

    Working Alliance Inventory-Short Revised

    The Working Alliance Inventory-Short Revised (WAI-SR) is a 12-item measure of working alliance with goal, task, and bond subscales that has demonstrated good psychometric properties [50,51]. The Cronbach α for the WAI-SR in this study was .94 for the goal subscale, .93 for the task subscale, and .95 for the bond subscale.

    Administration of Measures

    At screening and posttreatment, clients were administered the PHQ-9, GAD-7, PLC-5, PDSS-SR, SPS-6, SIAS-6, DAR-5, and SDS. Additionally, at posttreatment, clients were administered the Treatment Satisfaction Questionnaire and WAI-SR. Certain measures were administered weekly during treatment, and at 6 months and 1 year postenrollment. These time points are not included in the present analyses but will be reported in the future once more data are available.

    Analyses

    Following intention-to-treat principles, we first created multiple imputations for missing data using predictive mean matching. Imputations for missing data were modeled controlling for lesson completion rates, pretreatment symptom measures, and, if available, weekly symptom measures. We used generalized estimating equations (GEEs) to model the change in symptoms from pretreatment to posttreatment [52]. A gamma distribution with a log-link function was used to model changes between pretreatment and posttreatment as proportional and to accommodate the skewed response distribution [53]. We ran the imputations and GEE models in R version 4.0.3 using the mice [54] and geepack [55] packages.

    The effectiveness of the PSP Wellbeing Course was evaluated by examining Hedges g and proportional reductions in symptoms. Hedges g is calculated as the difference in means between pretreatment and posttreatment divided by the pooled SD and multiplied by a small-sample bias correction. It is interpreted similarly to Cohen d but was considered to be a more appropriate measure of effect size for this study given the small size of our clinical subsamples. Some researchers have recommended calculating Hedges g using the SD of the control or pretreatment group instead of the pooled SD when pretreatment and posttreatment SDs are unequal [56]. Indeed, we noted that in the subsets of clients scoring above clinical cut-offs at pretreatment, the pretreatment SDs were much lower than the posttreatment SDs. However, because these subsets represented a restricted range of pretreatment scores, a greater SD at posttreatment would be expected—even in the absence of a treatment effect—due to regression toward the mean [57]. Therefore, we used pooled SDs in calculating Hedges g in the clinical subsets to avoid overestimating effect sizes.

    We also examined clinically significant deterioration and recovery for the PHQ-9, GAD-7, and PCL-5. Following the UK National Health Service guidelines, deterioration was defined as an increase of at least 6 points on the PHQ-9 [58]. Following precedents set in previous research using the same outcome measures, deterioration was defined as an increase of 5 points on the GAD-7 [59,60] or 10 points on the PCL-5 [61]. Recovery was defined as a decrease of at least 6 points on the PHQ-9, 5 points on the GAD-7, or 10 points on the PCL-5 that resulted in the patient moving from the clinical range to the nonclinical cut-off scores described in the Measures section above. We examined the uptake, treatment satisfaction and acceptability, and program usage patterns using descriptive statistics in SPSS version 23.


    Client Characteristics

    There were 83 clients included in this study. The flow of clients through the study process is shown in Figure 1. Client characteristics are summarized in Table 1.

    ‎
    Figure 1. Flowchart displaying client enrollment, program usage, and completion of posttreatment measures. GAD-7: Generalized Anxiety Disorder-7; PCL-5: Posttraumatic Stress Disorder Checklist for the DSM-5; PHQ-9: Patient Health Questionnaire-9; PSP: public safety personnel.
    View this figure
    Table 1. Client demographics (N=83).
    CharacteristicValue
    Age (years), mean (SD)40.25 (10.95)
    Gender, n (%)

    		Woman45 (54)

    		Man37 (45)

    		Nonbinary1 (1)
    Ethnicity, n (%)

    		White73 (88)

    		First Nations, Inuit, or Metis7 (8)

    		Other3 (4)
    Married or common law, n (%)

    		Yes53 (64)

    		No30 (36)
    Children, n (%)

    		Yes44 (53)

    		No39 (47)
    Community size, n (%)

    		Population of 100,000 or greater43 (52)

    		Population under 100,00040 (48)
    PSPa sector, n (%)

    		Police24 (29)

    		Corrections9 (11)

    		Emergency medical service29 (35)

    		Fire6 (7)

    		Dispatch/communications5 (6)

    		Other10 (12)

    aPSP: public safety personnel.

    Effectiveness

    The effect sizes for each scale, along with the percentage changes and reliable change rates for primary symptom measures are detailed in Table 2. The results indicated large effect sizes on the PHQ-9 and GAD-7, and a moderate effect size on the PCL-5. When examining effect sizes for each primary measure only among clients reporting clinically significant symptoms on that measure at screening, we found large effect sizes on all three measures. In general, we found moderate effect sizes on secondary symptom measures in the entire sample and moderate to large effect sizes when examining only clients reporting clinically significant symptoms at screening; however, the results indicated small effect sizes on the combined SIAS-6 and SPS-6 among all clients and for those with clinically significant SIAS-6 and SPS-6 scores at screening.

    Table 2. Effectiveness of the PSP Wellbeing Course.
    QuestionnairePretreatment, mean (SD)Posttreatment, mean (SD)Hedges g effect size (95% CI) from pretreatmentPercentage change (95% CI) in estimated mean from pretreatmentReliable deterioration (%)Reliable recovery (%)
    Total sample (N=83)






    		PHQ-9a11.09 (5.59)6.81 (5.33)0.78 (0.46 to 1.09)39 (26 to 49)4.429.7

    		GAD-7b10.32 (5.44)6.01 (4.82)0.83 (0.52 to 1.15)42 (30 to 52)1.737.6

    		PCL-5c29.82 (19.18)20.39 (17.60)0.51 (0.20 to 0.82)32 (15 to 45)6.825.1

    		PDSS-SRd5.20 (5.09)3.17 (4.52)0.42 (0.11 to 0.73)39 (15 to 56)N/AeN/A

    		SIAS-6/SPS-6f9.76 (9.22)9.19 (9.58)0.06 (–0.24 to 0.36)6 (–20 to 26)N/AN/A

    		DAR-5g10.31 (4.39)8.59 (3.79)0.42 (0.11 to 0.72)17 (8 to 25)N/AN/A

    		SDSh Family4.98 (2.73)3.69 (2.84)0.46 (0.15 to 0.77)26 (10 to 39)N/AN/A

    		SDS Social5.03 (2.94)3.82 (2.92)0.41 (0.10 to 0.72)24 (9 to 37)N/AN/A

    		SDS Work4.54 (3.02)3.40 (3.05)0.37 (0.07 to 0.68)25 (7 to 40)N/AN/A

    		SDS Total14.54 (7.81)10.80 (7.76)0.48 (0.17 to 0.79)26 (11 to 38)N/AN/A
    Clinical samplei






    		PHQ-9 (≥10, n=48)14.90 (3.82)8.91 (5.46)1.25 (0.82 to 1.70)40 (28 to 51)2.951.3

    		GAD-7 (≥10, n=46)14.22 (3.24)8.08 (5.11)1.41 (0.97 to 1.87)43 (31 to 53)2.556.8

    		PCL-5 (≥33, n=35)48.63 (10.66)32.96 (17.63)1.05 (0.56 to 1.56)32 (17 to 45)7.859.5

    		PDSS-SR (≥8, n=23)12.14 (2.94)7.00 (6.18)1.03 (0.42 to 1.66)42 (15 to 61)N/AN/A

    		SIAS-6/SPS-6 (SIAS≥7 SPS≥2, n=27)20.52 (7.54)18.48 (10.23)0.22 (–0.31 to 0.76)10 (–13 to 28)N/AN/A

    		DAR-5 (≥12, n=25)15.84 (3.21)11.76 (4.80)0.97 (0.39 to 1.57)26 (12 to 37)N/AN/A

    		SDS Family (≥3, n=66)6.03 (1.93)4.23 (2.84)0.73 (0.38 to 1.09)30 (15 to 42)N/AN/A

    		SDS Social (≥3, n=62)6.34 (2.11)4.45 (2.88)0.74 (0.38 to 1.11)30 (16 to 42)N/AN/A

    		SDS Work (≥3, n=58)5.98 (2.40)4.23 (3.00)0.64 (0.27 to 1.01)29 (13 to 43)N/AN/A

    		SDS Total (≥9, n=62)17.95 (5.74)12.35 (7.75)0.81 (0.45 to 1.18)31 (17 to 43)N/AN/A

    aPHQ-9: Patient Health Questionnaire-9.

    bGAD-7: Generalized Anxiety Disorder Questionnaire-7.

    cPCL-5: Posttraumatic Stress Disorder Checklist for DSM-5.

    dPDSS-SR: Panic Disorder Severity Scale Self-Report.

    eN/A: not applicable; reliable recovery and deterioration were only calculated for primary measures because the magnitude of the score change required to determine reliable score changes for several secondary measures was unknown.

    fSIAS-6/SPS-6: 6-item Social Interaction Anxiety Scale/6-item Social Phobia Scale.

    gDAR-5: Dimensions of Anger Reactions-5.

    hSDS: Sheehan Disability Scale.

    iFor each measure, estimated means, effects sizes, and percentage change are calculated based on those who scored in the clinical range on the measure at pretreatment.

    Treatment Satisfaction and Working Alliance

    Of the 62 clients who completed the WAI-SR and questions related to treatment satisfaction, 54/62 (86%) indicated that the PSP Wellbeing Course had increased or greatly increased their confidence in their ability to manage their symptoms, and 61/62 (98%) indicated that the PSP Wellbeing Course was worth their time. Mean scores on the bond, goal, and task subscales of the WAI-SR were 16.11 (SD 4.64), 14.50 (SD 4.72), and 14.63 (SD 4.07), respectively. To our knowledge, there are no formal guidelines for interpreting WAI-SR scores, but these participant scores fall in the “high/positive” range according to the interpretation of one group of ICBT researchers [62].

    Program Usage Patterns

    All 83 clients included in the analyses accessed Lesson 1 of the PSP Wellbeing Course. Most clients also accessed Lesson 2 (79/83, 95%), Lesson 3 (71/83, 86%), Lesson 4 (64/83, 77%), and Lesson 5 (64/83, 77%). Of the 60 clients who completed 8-week measures, at the time they did so, all had accessed Lesson 1 and Lesson 2, and most had accessed Lesson 3 (58/60, 97%), Lesson 4 (54/60, 90%), and Lesson 5 (45/60, 75%). Across the entire sample, most clients who accessed each lesson also accessed the accompanying DIY guides (92% or more for each lesson), FAQs (84% or more), and stories about PSP (84% or more). Many clients accessed additional resources; the most frequently accessed resources were on relationships with significant others (46/83, 55%), anger (42/83, 51%), communication skills (40/83, 48%), and problem solving and worry time (40/83, 48%). Most clients opted to receive therapist support once per week (74/83, 89%), but some opted to receive therapist support twice per week (6/83, 7%) or only on an as-needed basis (3/83, 4%). On average, clients sent 4.98 messages (SD 5.53) to their therapists and received 9.80 (SD 4.71) messages.

    Some clients reported using the PSP Wellbeing Course concurrently with other treatments. At prescreen, some clients reported currently taking one or more medications for their mental health (18/83, 22%) or receiving mental health care from one or more providers (19/83, 23%), the most common of which were family doctors (13/83, 16%), psychologists (6/83, 7%), psychiatrists (4/83, 5%), and counsellors (3/83, 4%). Per the eligibility criteria described above, none of these clients were receiving mental health treatments from these providers more than twice per month. Several clients (4/83, 5%) indicated that they were on a waitlist to see a mental health care provider.


    Principal Findings

    Canadian PSP experience high rates of mental health problems and face several barriers to mental health care [15,16]. ICBT is a promising solution for effectively managing symptoms of common mental disorders [19,20] in a private and convenient fashion that allows clients to overcome key barriers for care [15,17]. PSPNET has developed a tailored ICBT program for PSP called the PSP Wellbeing Course. This article presents the initial program outcomes.

    The results indicated that the PSP Wellbeing Course is effective for treating symptoms of depression, anxiety, PTSD, panic disorder, and anger, and also improves clients’ functioning across three domains of life. The course appeared to be only slightly effective for treating symptoms of social anxiety, but this should be interpreted with caution because the subsample reporting clinically significant social anxiety was small (n=27). Nonetheless, in keeping with the learning health system approach [25], PSPNET will explore the possibility of providing additional resources to help clients manage social anxiety symptoms. This is important because approximately 15% of Canadian PSP—and 33% in this study’s treatment-seeking sample—struggle with clinically significant social anxiety [11]. Results concerning treatment satisfaction, working alliance, and engagement were promising. Although in our interviews with PSP stakeholders [15], most had expressed a preference for a flexible treatment duration, it was encouraging that most clients completed all lessons within 8 weeks. It was also very encouraging that working alliance was high because many PSP we interviewed previously [15] reported negative attitudes toward mental health care providers (eg, distrust, discomfort, beliefs that service providers do not understand PSP). Most clients accessed multiple additional resources, suggesting that they found them helpful and lending some support to recommendations for modularized, optional treatment elements in internet interventions [21].

    Descriptively, these outcomes are comparable to the most recent published outcomes of the Wellbeing Course offered by the Online Therapy Unit to the general population of Saskatchewan [63]. Effect sizes, percentage changes, and reliable change rates were comparable for all available symptom measures with the possible exception of the SIAS-6/SPS-6, which appeared to show better outcomes for the Wellbeing Course. However, mean pretreatment social anxiety symptoms were considerably less severe in the present sample than in the Online Therapy Unit’s clients [63], and the smaller change in symptoms found in this study may be due, in part, to a floor effect. Of note, the present findings show that the PSP Wellbeing Course was moderately effective for treating anger, which has not been measured in research on the Wellbeing Course. PSP participating in this study reported concurrent use of psychotropic medication less frequently than clients of the Wellbeing Course [63]. Results concerning treatment satisfaction, working alliance, and engagement were similar across the two courses.

    Limitations

    This study has several limitations. First, the data were collected during the COVID-19 pandemic, and the impact of the pandemic on outcomes is unclear. Second, there was no control condition and therefore no direct evidence that symptom change can be attributed to the PSP Wellbeing Course. However, it has been argued that the literature on digital mental health interventions would benefit from more research evaluating such interventions in real-world settings [64,65], as we did in this study. Third, at 8 weeks, we were missing PHQ-9 and GAD-7 data for 23% of clients and PCL-5 data for 25% of clients. A conservative approach was used to manage missing data, but it is possible that clients for whom data were missing may represent a unique subset of clients (eg, clients who experienced less positive outcomes).

    Future Directions

    Further research can extend these findings in several ways. This article describes the results for only the first third of the sample (83 of 250) specified in our trial protocol; full results will be reported when available, including outcomes at 1-year follow-up and for PSPNET clients in Quebec. Further research will also be required to explore demographic and clinical predictors of treatment response for the PSP Wellbeing Course and to identify new ways to tailor and improve ICBT for PSP. Finally, although this study suggests that the PSP Wellbeing Course benefits the PSP who access it, economic evaluation will be required in the future.

    Conclusions

    Preliminary results indicate that the PSP Wellbeing Course is a promising and effective method for treating symptoms of depression, anxiety, and PTSD among Saskatchewan PSP. Our results also suggest that the course improved clients’ functioning and reduced their symptoms of panic and anger, but not social anxiety. Clients reported good working alliance and treatment satisfaction, and they demonstrated good engagement. Consistent with the concept of learning health systems, PSPNET will continue applying research results to iteratively improve tailored ICBT services, such as by investigating ways to better treat symptoms of social anxiety among PSP. Our findings may also be helpful for other groups involved in researching, providing, funding, or otherwise supporting mental health care for this unique and underserved population.

    Acknowledgments

    We would like to acknowledge the contributions of all PSP who have participated and the members of the PSPNET team for their support with this project. We would also like to acknowledge the many stakeholders and organizations who have assisted us in tailoring the PSP Wellbeing Course and distributing information about PSPNET to PSP. Acknowledgements are due to the Online Therapy Unit for their support in setting up PSPNET. The authors are indebted to Information Services at the University of Regina, especially Max Ivanov, for his pivotal role in developing the Online Therapy Unit platform that is used to host PSPNET. This research was performed by PSPNET, which is funded by the Canadian Government’s Ministry of Public Safety and Emergency Preparedness. The funders had no involvement in the design of the study, or in the collection, analysis, and interpretation of the data. NT and BD are funded by the Australian Government to operate the MindSpot Clinic. NT and BF developed the Wellbeing Course but derive no financial benefit from it.

    Conflicts of Interest

    HH is the Director of PSPNET and Executive Director of the Online Therapy Unit, and declares funding as noted above. BFD and NT are authors and developers of the Wellbeing Course, but do not derive any personal or financial benefit from it. The other authors have no conflicts of interest to declare.

    1. Oliphant R. Healthy minds, safe communities: supporting our public safety officers through a national strategy for operational stress injuries. Canadian House of Commons. 2016.   URL: https://www.ourcommons.ca/Content/Committee/421/SECU/Reports/RP8457704/securp05/securp05-e.pdf [accessed 2021-04-28]
    2. Carleton RN, Afifi TO, Taillieu T, Turner S, Krakauer R, Anderson GS, et al. Exposures to potentially traumatic events among public safety personnel in Canada. Can J of Behav Sci 2019 Jan;51(1):37-52. [CrossRef]
    3. Galatzer-Levy IR, Madan A, Neylan TC, Henn-Haase C, Marmar CR. Peritraumatic and trait dissociation differentiate police officers with resilient versus symptomatic trajectories of posttraumatic stress symptoms. J Trauma Stress 2011 Oct 06;24(5):557-565 [FREE Full text] [CrossRef] [Medline]
    4. Komarovskaya I, Maguen S, McCaslin SE, Metzler TJ, Madan A, Brown AD, et al. The impact of killing and injuring others on mental health symptoms among police officers. J Psychiatr Res 2011 Oct;45(10):1332-1336 [FREE Full text] [CrossRef] [Medline]
    5. American Psychiatric Association. Diagnostic and statistical manual of mental disorders. Washington DC: American Psychiatric Association; 2013.
    6. Kilpatrick DG, Resnick HS, Milanak ME, Miller MW, Keyes KM, Friedman MJ. National estimates of exposure to traumatic events and PTSD prevalence using DSM-IV and DSM-5 criteria. J Trauma Stress 2013 Oct;26(5):537-547 [FREE Full text] [CrossRef] [Medline]
    7. Nixon RD, Resick PA, Griffin MG. Panic following trauma: the etiology of acute posttraumatic arousal. J Anxiety Disord 2004 Jan;18(2):193-210. [CrossRef]
    8. O'Donnell ML, Creamer M, Pattison P. Posttraumatic stress disorder and depression following trauma: understanding comorbidity. Am J Psychiatry 2004 Aug;161(8):1390-1396. [CrossRef] [Medline]
    9. Perkonigg A, Kessler R, Storz S, Wittchen HU. Traumatic events and post-traumatic stress disorder in the community: prevalence, risk factors and comorbidity. Acta Psychiatr Scand 2000 Jan;101(1):46-59. [CrossRef] [Medline]
    10. Shalev AY, Freedman S, Peri T, Brandes D, Sahar T, Orr SP, et al. Prospective study of posttraumatic stress disorder and depression following trauma. Am J Psychiatry 1998 May;155(5):630-637. [CrossRef] [Medline]
    11. Carleton RN, Afifi TO, Turner S, Taillieu T, Duranceau S, LeBouthillier DM, et al. Mental disorder symptoms among public safety personnel in Canada. Can J Psychiatry 2018 Jan 28;63(1):54-64 [FREE Full text] [CrossRef] [Medline]
    12. Courtney JA, Francis AJP, Paxton SJ. Caring for the country: fatigue, sleep and mental health in Australian rural paramedic shiftworkers. J Community Health 2013 Mar 29;38(1):178-186. [CrossRef] [Medline]
    13. Maia DB, Marmar CR, Metzler T, Nóbrega A, Berger W, Mendlowicz MV, et al. Post-traumatic stress symptoms in an elite unit of Brazilian police officers: prevalence and impact on psychosocial functioning and on physical and mental health. J Affect Disord 2007 Jan;97(1-3):241-245. [CrossRef] [Medline]
    14. Stanley IH, Boffa JW, Hom MA, Kimbrel NA, Joiner TE. Differences in psychiatric symptoms and barriers to mental health care between volunteer and career firefighters. Psychiatry Res 2017 Jan;247:236-242. [CrossRef] [Medline]
    15. McCall HC, Beahm JD, Fournier AK, Burnett JL, Carleton RN, Hadjistavropoulos HD. Stakeholder perspectives on internet-delivered cognitive behavioural therapy for public safety personnel: A qualitative analysis. Can J Behav Sci 2020 Aug 27:online ahead of print. [CrossRef]
    16. Ricciardelli R, Carleton RN, Mooney T, Cramm H. "Playing the system": Structural factors potentiating mental health stigma, challenging awareness, and creating barriers to care for Canadian public safety personnel. Health (London) 2020 May 16;24(3):259-278. [CrossRef] [Medline]
    17. Andersson G. Internet-delivered psychological treatments. Annu Rev Clin Psychol 2016;12:157-179. [CrossRef] [Medline]
    18. Donker T, Blankers M, Hedman E, Ljótsson B, Petrie K, Christensen H. Economic evaluations of internet interventions for mental health: a systematic review. Psychol Med 2015 Dec;45(16):3357-3376. [CrossRef] [Medline]
    19. Andersson G, Carlbring P, Titov N, Lindefors N. Internet interventions for adults with anxiety and mood disorders: a narrative umbrella review of recent meta-analyses. Can J Psychiatry 2019 Jul 16;64(7):465-470. [CrossRef] [Medline]
    20. Sijbrandij M, Kunovski I, Cuijpers P. Effectiveness of internet-delivered cognitive behavioral therapy for posttraumatic stress disorder: a systematic review and meta-analysis. Depress Anxiety 2016 Sep;33(9):783-791. [CrossRef] [Medline]
    21. Fleming TM, de Beurs D, Khazaal Y, Gaggioli A, Riva G, Botella C, et al. Maximizing the impact of e-therapy and serious gaming: time for a paradigm shift. Front Psychiatry 2016 Apr 18;7:65. [CrossRef] [Medline]
    22. Ludden GD, van Rompay TJ, Kelders SM, van Gemert-Pijnen JE. How to increase reach and adherence of web-based interventions: a design research viewpoint. J Med Internet Res 2015 Jul 10;17(7):e172 [FREE Full text] [CrossRef] [Medline]
    23. McCall HC, Sison AP, Burnett JL, Beahm JD, Hadjistavropoulos HD. Exploring perceptions of internet-delivered cognitive behaviour therapy among public safety personnel: informing dissemination efforts. Int J Environ Res Public Health 2020 Aug 19;17(17):6026 [FREE Full text] [CrossRef] [Medline]
    24. Supporting Canada’s public safety personnel: an action plan on post-traumatic stress injuries. Public Safety Canada. 2019.   URL: https://www.publicsafety.gc.ca/cnt/rsrcs/pblctns/2019-ctn-pln-ptsi/index-en.aspx [accessed 2021-04-27]
    25. Menear M, Blanchette M, Demers-Payette O, Roy D. A framework for value-creating learning health systems. Health Res Policy Syst 2019 Aug 09;17(1):79 [FREE Full text] [CrossRef] [Medline]
    26. von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP, STROBE Initiative. Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. BMJ 2007 Oct 20;335(7624):806-808 [FREE Full text] [CrossRef] [Medline]
    27. Online Therapy Unit.   URL: https://www.onlinetherapyuser.ca/ [accessed 2020-04-28]
    28. Titov N, Dear BF, Staples LG, Terides MD, Karin E, Sheehan J, et al. Disorder-specific versus transdiagnostic and clinician-guided versus self-guided treatment for major depressive disorder and comorbid anxiety disorders: A randomized controlled trial. J Anxiety Disord 2015 Oct;35:88-102 [FREE Full text] [CrossRef] [Medline]
    29. Titov N, Dear BF, Staples LG, Bennett-Levy J, Klein B, Rapee RM, et al. MindSpot Clinic: an accessible, efficient, and effective online treatment service for anxiety and depression. Psychiatr Serv 2015 Oct;66(10):1043-1050. [CrossRef] [Medline]
    30. Fogliati V, Dear B, Staples L, Terides M, Sheehan J, Johnston L, et al. Disorder-specific versus transdiagnostic and clinician-guided versus self-guided internet-delivered treatment for panic disorder and comorbid disorders: A randomized controlled trial. J Anxiety Disord 2016 Apr;39:88-102 [FREE Full text] [CrossRef] [Medline]
    31. Dear BF, Staples LG, Terides MD, Karin E, Zou J, Johnston L, et al. Transdiagnostic versus disorder-specific and clinician-guided versus self-guided internet-delivered treatment for generalized anxiety disorder and comorbid disorders: A randomized controlled trial. J Anxiety Disord 2015 Dec;36:63-77 [FREE Full text] [CrossRef] [Medline]
    32. Dear B, Staples L, Terides M, Fogliati V, Sheehan J, Johnston L, et al. Transdiagnostic versus disorder-specific and clinician-guided versus self-guided internet-delivered treatment for Social Anxiety Disorder and comorbid disorders: A randomized controlled trial. J Anxiety Disord 2016 Aug;42:30-44 [FREE Full text] [CrossRef] [Medline]
    33. Kroenke K, Spitzer RL, Williams JBW. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med 2001 Sep;16(9):606-613 [FREE Full text] [CrossRef] [Medline]
    34. Manea L, Gilbody S, McMillan D. Optimal cut-off score for diagnosing depression with the Patient Health Questionnaire (PHQ-9): a meta-analysis. CMAJ 2012 Mar 21;184(3):E191-E196 [FREE Full text] [CrossRef] [Medline]
    35. Cameron IM, Crawford JR, Lawton K, Reid IC. Psychometric comparison of PHQ-9 and HADS for measuring depression severity in primary care. Br J Gen Pract 2008 Jan;58(546):32-36 [FREE Full text] [CrossRef] [Medline]
    36. Kroenke K, Spitzer RL, Williams JBW, Löwe B. The Patient Health Questionnaire Somatic, Anxiety, and Depressive Symptom Scales: a systematic review. Gen Hosp Psychiatry 2010;32(4):345-359. [CrossRef] [Medline]
    37. Spitzer RL, Kroenke K, Williams JBW, Löwe B. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med 2006 May 22;166(10):1092-1097. [CrossRef] [Medline]
    38. Blevins CA, Weathers FW, Davis MT, Witte TK, Domino JL. The Posttraumatic Stress Disorder Checklist for DSM-5 (PCL-5): Development and Initial Psychometric Evaluation. J Trauma Stress 2015 Dec;28(6):489-498. [CrossRef] [Medline]
    39. Bovin MJ, Marx BP, Weathers FW, Gallagher MW, Rodriguez P, Schnurr PP, et al. Psychometric properties of the PTSD Checklist for Diagnostic and Statistical Manual of Mental Disorders-Fifth Edition (PCL-5) in veterans. Psychol Assess 2016 Nov;28(11):1379-1391. [CrossRef] [Medline]
    40. Wortmann JH, Jordan AH, Weathers FW, Resick PA, Dondanville KA, Hall-Clark B, et al. Psychometric analysis of the PTSD Checklist-5 (PCL-5) among treatment-seeking military service members. Psychol Assess 2016 Nov;28(11):1392-1403. [CrossRef] [Medline]
    41. Houck PR, Spiegel DA, Shear MK, Rucci P. Reliability of the self-report version of the panic disorder severity scale. Depress Anxiety 2002 Jun 05;15(4):183-185. [CrossRef] [Medline]
    42. Christensen H, Batterham PJ, Grant JB, Griffiths KM, Mackinnon AJ. A population study comparing screening performance of prototypes for depression and anxiety with standard scales. BMC Med Res Methodol 2011 Nov 22;11(1):154 [FREE Full text] [CrossRef] [Medline]
    43. Ruwaard J, Broeksteeg J, Schrieken B, Emmelkamp P, Lange A. Web-based therapist-assisted cognitive behavioral treatment of panic symptoms: a randomized controlled trial with a three-year follow-up. J Anxiety Disord 2010 May;24(4):387-396. [CrossRef] [Medline]
    44. Allen AR, Newby JM, Mackenzie A, Smith J, Boulton M, Loughnan SA, et al. Internet cognitive-behavioural treatment for panic disorder: randomised controlled trial and evidence of effectiveness in primary care. BJPsych Open 2016 Mar;2(2):154-162 [FREE Full text] [CrossRef] [Medline]
    45. Shear M, Rucci P, Williams J, Frank E, Grochocinski V, Vander Bilt J, et al. Reliability and validity of the Panic Disorder Severity Scale: replication and extension. J Psychiatr Res 2001 Sep;35(5):293-296. [CrossRef]
    46. Peters L, Sunderland M, Andrews G, Rapee RM, Mattick RP. Development of a short form Social Interaction Anxiety (SIAS) and Social Phobia Scale (SPS) using nonparametric item response theory: the SIAS-6 and the SPS-6. Psychol Assess 2012 Mar;24(1):66-76. [CrossRef] [Medline]
    47. Hawthorne G, Mouthaan J, Forbes D, Novaco RW. Response categories and anger measurement: do fewer categories result in poorer measurement?: development of the DAR5. Soc Psychiatry Psychiatr Epidemiol 2006 Mar;41(2):164-172. [CrossRef] [Medline]
    48. Forbes D, Alkemade N, Mitchell D, Elhai JD, McHugh T, Bates G, et al. Utility of the Dimensions of Anger Reactions-5 (DAR-5) scale as a brief anger measure. Depress Anxiety 2014 Mar 25;31(2):166-173. [CrossRef] [Medline]
    49. Sheehan D. The Sheehan Disability Scales. In: The anxiety disease and how to overcome it. New York: Charles Scribner and Sons; 1983.
    50. Hatcher RL, Gillaspy JA. Development and validation of a revised short version of the working alliance inventory. Psychother Res 2006 Jan;16(1):12-25. [CrossRef]
    51. Munder T, Wilmers F, Leonhart R, Linster HW, Barth J. Working Alliance Inventory-Short Revised (WAI-SR): psychometric properties in outpatients and inpatients. Clin Psychol Psychother 2010;17(3):231-239. [CrossRef] [Medline]
    52. Liang K, Zeger SL. Longitudinal data analysis using generalized linear models. Biometrika 1986;73(1):13-22. [CrossRef]
    53. Karin E, Dear BF, Heller GZ, Gandy M, Titov N. Measurement of symptom change following web-based psychotherapy: statistical characteristics and analytical methods for measuring and interpreting change. JMIR Ment Health 2018 Jul 12;5(3):e10200 [FREE Full text] [CrossRef] [Medline]
    54. Buuren SV, Groothuis-Oudshoorn K. Multivariate Imputation by Chained Equations in R. J Stat Soft 2011;45(3):1-67 [FREE Full text] [CrossRef]
    55. Halekoh U, Højsgaard S, Yan J. The R Package geepack for Generalized Estimating Equations. J Stat Soft 2006;15(2):1-11 [FREE Full text] [CrossRef]
    56. Glass GV. Primary, secondary, and meta-analysis of research. Educ Res 2016 Nov 30;5(10):3-8. [CrossRef]
    57. Nesselroade JR, Stigler SM, Baltes PB. Regression toward the mean and the study of change. Psychol Bull 1980;88(3):622-637. [CrossRef]
    58. The improving access to psychological therapies manual: Appendices and helpful resources. National Collaborating Centre for Mental Health. 2019.   URL: https:/​/www.​england.nhs.uk/​wp-content/​uploads/​2018/​06/​iapt-manual-appendices-and-helpful-resources-v3.​pdf [accessed 2021-04-28]
    59. Titov N, Dear BF, Staples LG, Bennett-Levy J, Klein B, Rapee RM, et al. The first 30 months of the MindSpot Clinic: Evaluation of a national e-mental health service against project objectives. Aust N Z J Psychiatry 2017 Dec;51(12):1227-1239. [CrossRef] [Medline]
    60. Titov N, Dear BF, Nielssen O, Wootton B, Kayrouz R, Karin E, et al. User characteristics and outcomes from a national digital mental health service: an observational study of registrants of the Australian MindSpot Clinic. Lancet Digit Health 2020 Nov;2(11):e582-e593 [FREE Full text] [CrossRef] [Medline]
    61. Byllesby BM, Dickstein BD, Chard KM. The probability of change versus dropout in veterans receiving cognitive processing therapy for posttraumatic stress disorder. Behav Res Ther 2019 Dec;123:103483. [CrossRef] [Medline]
    62. Jasper K, Weise C, Conrad I, Andersson G, Hiller W, Kleinstäuber M. The working alliance in a randomized controlled trial comparing Internet-based self-help and face-to-face cognitive behavior therapy for chronic tinnitus. Internet Interv 2014 Apr;1(2):49-57. [CrossRef]
    63. Hadjistavropoulos H, Peynenburg V, Thiessen D, Nugent M, Adlam K, Owens K, et al. A pragmatic factorial randomized controlled trial of transdiagnostic internet-delivered cognitive behavioural therapy: Exploring benefits of homework reflection questionnaires and twice-weekly therapist support. Internet Interv 2020 Dec;22:100357 [FREE Full text] [CrossRef] [Medline]
    64. Mohr DC, Lyon AR, Lattie EG, Reddy M, Schueller SM. Accelerating digital mental health research from early design and creation to successful implementation and sustainment. J Med Internet Res 2017 May 10;19(5):e153 [FREE Full text] [CrossRef] [Medline]
    65. Mohr DC, Riper H, Schueller SM. A solution-focused research approach to achieve an implementable revolution in digital mental health. JAMA Psychiatry 2018 Feb 01;75(2):113-114. [CrossRef] [Medline]

    ‎
    DAR-5: Dimensions of Anger Reactions Scale
    GAD-7: Generalized Anxiety Disorder-7
    GEE: generalized estimating equations
    ICBT: internet-delivered cognitive behavioral therapy
    PCL-5: Posttraumatic Stress Disorder Checklist for the DSM-5
    PDSS-SR: Panic Disorder Severity Scale Self-Report
    PHQ-9: Patient Health Questionnaire-9
    PSP: public safety personnel
    PTSD: posttraumatic stress disorder
    SDS: Sheehan Disability Scale
    SIAS-6: 6-item Social Interaction Anxiety Scale
    SPS-6: 6-item Social Phobia Scale
    WAI-SR: Working Alliance Inventory-Short Revised

    Edited by G Eysenbach; submitted 30.01.21; peer-reviewed by K Mathiasen; comments to author 25.02.21; revised version received 01.03.21; accepted 11.04.21; published 05.05.21

    Copyright

    ©Heather D Hadjistavropoulos, Hugh C McCall, David L Thiessen, Ziyin Huang, R Nicholas Carleton, Blake F Dear, Nickolai Titov. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 05.05.2021.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.