This paper is in the following e-collection/theme issue:

Research Instruments, Questionnaires, and Tools (576) Engagement with and Adherence to Digital Health Interventions, Law of Attrition (491)

Published on in Vol 21, No 10 (2019): October

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/13602, first published .
The Internet Intervention Patient Adherence Scale for Guided Internet-Delivered Behavioral Interventions: Development and Psychometric Evaluation

The Internet Intervention Patient Adherence Scale for Guided Internet-Delivered Behavioral Interventions: Development and Psychometric Evaluation

The Internet Intervention Patient Adherence Scale for Guided Internet-Delivered Behavioral Interventions: Development and Psychometric Evaluation

Authors of this article:

Fabian Lenhard1 Author Orcid Image ;   Kajsa Mitsell2 Author Orcid Image ;   Maral Jolstedt1 Author Orcid Image ;   Sarah Vigerland1 Author Orcid Image ;   Tove Wahlund1 Author Orcid Image ;   Martina Nord1 Author Orcid Image ;   Johan Bjureberg1 Author Orcid Image ;   Hanna Sahlin1 Author Orcid Image ;   Per Andrén1 Author Orcid Image ;   Kristina Aspvall1 Author Orcid Image ;   Karin Melin3 Author Orcid Image ;   David Mataix-Cols1 Author Orcid Image ;   Eva Serlachius1 Author Orcid Image ;   Jens Högström1 Author Orcid Image
Article Authors Cited by (17) Tweetations (8) Metrics

    Original Paper

    1Centre for Psychiatry Research, Department of Clinical Neuroscience, Karolinska Institutet, & Stockholm Health Care Services, Stockholm County Council, Stockholm, Sweden

    2Stockholm Health Care Services, Stockholm County Council, Stockholm, Sweden

    3Institute of Neuroscience and Physiology, The Sahlgrenska Academy, University of Gothenburg, Gothenburg, Sweden

    Corresponding Author:

    Fabian Lenhard, PhD

    Centre for Psychiatry Research

    Department of Clinical Neuroscience

    Karolinska Institutet, & Stockholm Health Care Services, Stockholm County Council

    CAP Research Centre, Gävlegatan 22

    SE-113 30

    Stockholm,

    Sweden

    Phone: 46 8 514 522 12

    Email: fabian.lenhard@ki.se


    Background: Patient adherence is defined as the extent to which a patient complies with medical or health advice. At present, there is a lack of reliable and valid measures specifically designed to measure adherence to internet-delivered behavioral interventions.

    Objective: The objective of this study was to develop and psychometrically evaluate a novel measure of adherence to guided internet-delivered behavioral interventions.

    Methods: In collaboration with experienced clinicians and researchers in the field, a 5-item, clinician-rated internet intervention Patient Adherence Scale (iiPAS) was developed. The initial scale was tested in a sample of children and adolescents (N=50) participating in internet-delivered cognitive behavioral therapy (ICBT) studies. A revised version of the iiPAS was then administered to a larger sample of children and adolescents (N=148) with various behavioral problems participating in ICBT trials. The scale was evaluated according to a classical test theory framework.

    Results: The iiPAS demonstrated excellent internal consistency. Factor analyses revealed one underlying factor, explaining about 80% of the variance, suggesting that the scale captures a homogeneous adherence construct. The iiPAS was strongly associated with objective measures of patient activity in ICBT (number of logins, number of written characters, and completed modules). Furthermore, mid- and posttreatment ratings of the iiPAS were significantly correlated with treatment outcomes. By contrast, objective measures of patient activity in the Web-based platform did not correlate with treatment outcomes.

    Conclusions: The iiPAS could be a useful tool to measure adherence in a broad range of internet-delivered behavioral interventions.

    J Med Internet Res 2019;21(10):e13602

    doi:10.2196/13602

    Keywords

    patient compliance; eHealth; measure; internet; cognitive behavioral therapy 


    Background

    Patient adherence, sometimes also termed compliance, is most broadly defined as “the extent to which the patient’s behavior (in terms of taking medications, following diets, or executing other lifestyle changes) coincides with medical or health advice” [1]. Within the field of psychotherapy, adherence is usually conceptualized within the therapeutic tradition or according to the target condition that is being studied [2]. Accordingly, adherence is measured in various ways, for example, as qualitative or quantitative aspects of homework compliance [3], in-session engagement [4], or patient-therapist alliance [5]. Absence of patient adherence, on the other hand, is often referred to as resistance or ambivalence [6,7], and there are various therapeutic strategies that are specifically designed to target such resistance or ambivalence, such as motivational interviewing [8].

    One important aspect of patient adherence is the logical assumption that adherent patients will have better treatment outcomes than nonadherent patients, though the evidence has not always supported this assumption. There is, for example, evidence that adherence to homework assignments in cognitive behavioral therapy (CBT) is associated with better outcomes, with a small to large effect of homework quality and quantity, but with substantial heterogeneity between studies [3,9]. Similarly, adherence to exposure and response prevention homework has been shown to be a strong predictor of short- and long-term outcome in CBT for obsessive-compulsive disorder (OCD) [10,11]. However, results from the field of pediatric anxiety have failed to establish a clear relationship between homework adherence and treatment outcome [12]. For example, in 1 study, neither the number of attended CBT sessions nor completed homework assignments predicted treatment outcome but a 1-item clinician-rated adherence measure did [13]. One study of CBT for anxiety disorders applied a combined measure of adherence, consisting of a weekly clinician rating of homework completion and a rating of engagement in the therapy session. The authors found that a combination of high treatment dose (high session attendance and completed exposure exercises) and high engagement predicted better treatment outcomes [14]. Importantly, it has been suggested that the adherence-outcome relationship could be affected by the quality of the adherence measure, and thus, reliable and valid measures are essential as research and clinical tools [15].

    Internet-Delivered Interventions and Adherence

    Internet-delivered behavioral interventions have emerged as an interdisciplinary development of psychotherapy and modern information technology. Internet-delivered behavioral interventions share common features as they mirror face-to-face psychotherapy regarding content and therapeutic techniques but are presented online via a personal login to a website or internet portal. The format usually resembles an online course, with instructive texts, pictures, videos, and written exercises, with the content structured similar to a book with chapters or modules. When therapist contact and feedback are involved, the treatment is referred to as guided or clinician-supported and otherwise as unguided or self-help [16]. Internet-delivered CBT (ICBT) has been one of the most prominent formats of this novel branch of interventions, with over 100 randomized controlled trials (RCTs) conducted in adults [17] and currently 19 RCTs in children and adolescents [18].

    Adherence may be particularly difficult to measure in internet-delivered behavioral interventions [19]. In face-to-face treatment, many of the proposed active treatment components, such as behavioral or cognitive strategies, are conducted in-session together with a clinician. In internet-delivered behavioral interventions, the key therapy components are delivered online and practiced without the physical presence of the clinician. Depending on the intervention, the engagement in the active treatment components of the intervention may have low correlation with the number of completed chapters. For instance, patients may go on practicing their homework assignments without necessarily needing to log into the system periodically. Therefore, the number of completed chapters in internet-delivered behavioral interventions is not comparable with the number of completed sessions in face-to-face interventions and, thus, may not be a reliable measure of treatment adherence. As pointed out by Sieverink et al [20], the concept of “intended usage” could be a helpful extension of the traditional definition of adherence. Intended usage is defined as “the extent to which individuals  should experience the content (of the intervention) to derive maximum benefit from the intervention, as defined or implied by its creators” [21].

    Thus, defining and measuring the core therapeutic concepts and minimal effective use of those strategies may be a vital aspect of any measure of adherence to internet-delivered interventions. However, less than half of the studies in the field apply such a definition, and only few provide a justifiable definition of minimal required usage [20]. Currently, the most prevalent operationalizations of adherence are number of treatment modules completed by patients and number of logins [20,22]. The relationship between adherence and clinical outcome has thus far been limited by inconsistent reporting of adherence and a lack of agreement on measures [18,22].

    In summary, there is an important research gap and a need for reliable and valid measures of patient adherence to internet-delivered interventions. We take the perspective that a useful measure of adherence to internet-delivered behavioral interventions should (1) not only capture the structural aspects of the internet format (such as completed chapters or modules) but also consider the intended usage of its central therapeutic aspects (such as use of behavioral strategies) and (2) include relevant, observable behaviors that represent the patients’ adherence in a way that guides clinically meaningful decisions. Thus, the aim of this project was to construct and validate an adherence measure, evaluate its psychometric properties, and establish whether this new measure correlates better with treatment outcome than other traditional measures of adherence (eg, number of logins and completed chapters).


    Scale Construction

    A novel, clinician-rated measure called internet intervention Patient Adherence Scale (iiPAS) was constructed in a 4-step process following recommendations from the psychometric literature [23].

    Step 1: Definition of Construct and Context

    The first author (FL) conceptualized the scale and drafted a first version with the aim to measure patient adherence as one homogeneous construct. To ensure feasibility, usability, and precision, the scale was intended to be a short and time-efficient measure and assess patient behaviors that would be observable by clinicians. The context of the scale was defined as a clinician-rated measure to be used within clinician-supported, internet-delivered behavioral interventions. The advantage of a clinician-rated scale, compared with a self-rated measure, was justified by the ability of the clinician to assess the adherence of the patient relative to the intended, optimal use of the intervention, which was assumed crucial for the reliability of the instrument [20]. Building on the available evidence presented in the introduction, the scale should include items that capture structural aspects of internet-delivered interventions, equivalent to number of completed chapters, as well as content-related aspects, equivalent to the patients’ engagement in the therapeutic content of the intervention .

    Step 2: Response Format and Initial Item Formulation

    To ensure optimal content validity, experienced clinicians and researchers with relevant competence (see “Expert Sample”) were involved in choosing the response format and initial item formulation. The initial draft was circulated among the experts in 2 review rounds and edited iteratively by the first author according to suggestions and comments. The preliminary version of the iiPAS included 5 clinician-rated items rated on a 3-point Likert scale, including anchor labels for the end points of the scale (see description under “Measures”).

    Step 3: Preliminary Data Collection

    The preliminary iiPAS was tested in a small sample (“Patient Sample 1”) to provide basic psychometric information as well as to give clinicians the possibility to test the scale and give feedback regarding usability and applicability.

    Step 4: Preliminary Examination of Psychometric Properties and Scale Revision

    On the basis of additional feedback from step 3, the wording of the items was minimally revised for improved clarity. In addition, the 3-point Likert scale was extended to a 5-point scale to increase response variance. A preliminary psychometric analysis was carried out; results are presented below. For the full psychometric evaluation, the revised iiPAS was then administered to participants in several clinical trials of internet-delivered behavioral interventions (“Patient Sample 2”).

    Participants

    Expert Sample

    The expert sample consisted of 8 experienced researchers and clinicians who had been involved in clinical trials of internet-delivered behavioral interventions (coauthors FL, SV, MJ, MN, PA, KA, TW, and JH) within a Swedish research network called the Child Internet Project or Barninternetprojektet (BiP). The experts were involved in the initial scale development.

    Patient Sample 1

    Initial data were collected from 50 patients, of whom 33 (66%) were children participating in a study of ICBT for pediatric anxiety [24], and 17 (34%) were adolescents participating in a study of ICBT for pediatric OCD [25]. Main inclusion criteria were a primary diagnosis of separation anxiety disorder, social anxiety disorder, generalized anxiety disorder, panic disorder, or specific phobia according to Diagnostic and Statistical Manual of Mental Disorders, 4th edition (DSM-IV) and a clinical severity score above 3 on the Clinician Severity Rating (CSR) [26] or a primary diagnosis of OCD with a score of 16 or above on Children’s Yale Brown Compulsive Scale [27]. Clinicians scoring the iiPAS were the same licensed clinical psychologists (N=13) who provided online support as a part of the ICBT interventions. For more detailed information, see the original publications of the respective trials [24,25]. The regional ethical review board in Stockholm, Sweden, approved both studies.

    Patient Sample 2

    Patient sample 2 consisted of 148 children and adolescents with various mental disorders, who together with their primary caretakers participated in different clinical trials. Common inclusion criteria within all trials were (1) age within the age interval for the respective trial (see “Interventions”), (2) internet access, (3) absence of suicidality or another severe condition that required immediate attention or higher priority, and (4) the clinical presentation of a mental disorder that was targeted in the specific trial (tic disorder, social anxiety disorder, generalized anxiety disorder, agoraphobia, panic disorder, specific phobia, separation anxiety disorder, OCD, and nonsuicidal self-injury). All conditions were diagnosed by experienced psychologists using semistructured diagnostic interviews based on DSM-5 criteria [28]. The iiPAS was scored by the treating clinicians (N=21), of whom 20 were licensed psychologists and a licensed nurse. For more detailed information, see the original publications of each trial [24,29-32]. The regional ethical review board in Stockholm, Sweden, approved all trials.

    Interventions

    All interventions were provided by researchers and clinicians associated to the BiP network, a collaboration between the Karolinska Institutet and the Child and Adolescent Mental Health Service in Stockholm, with a main research focus on development and evaluation of internet-delivered interventions for childhood mental disorders. All interventions in the current evaluation were clinician-guided ICBT interventions that also included parent support. Clinician advice, support, and feedback were given predominantly via written messages within the secure internet platform or via occasional telephone calls. For a short summary of the interventions, see Table 1. More detailed descriptions can be found in the original publications [29-33].

    Table 1. Summary of the included internet-delivered cognitive behavior therapy interventions.
    InterventionTarget age (years)Target conditionNumber of modules/weeksMain features
    BiPa TIC [29]7-17Tourette syndrome/chronic tic disorder10/10Habit reversal training or exposure with response prevention
    BiP SoFTc [30]13-17Social anxiety disorder9/12Exposure training and cognitive strategies
    BiP OCDb [25,31]7-17OCD12/12Exposure with response prevention and parental coping strategies
    BiP Anxiety [33]8-12Mixed anxiety disorders12/12Exposure training and child coping strategies
    BiP ERITAd[32]13-17Self-harm11/12Emotion regulation strategies and adaptive parental behaviors

    aBiP: Barninternetprojektet.

    bOCD: obsessive-compulsive disorder.

    cSoFT: Social Fobi Tonåring.

    dERITA: Emotion Regulation Individual Therapy for Adolescent.

    Procedure

    The included clinical trials applied similar but not identical procedures. A description of the common aspects of the procedures is provided here; for a detailed description, see the respective trial publications. Participants in patient samples 1 and 2 were recruited via clinician- or self-referrals. Selection of eligible participants included both telephone screenings with a clinician and an in-person assessment with the child or adolescent and at least one caregiver. Fulfillment of diagnostic criteria was established by a clinician (most often a clinical psychologist) using the Mini-International Neuropsychiatric Interview child version (MINI Kid [34]) or the Anxiety Disorder Interviews Schedule Child and Parent versions (ADIS C/P [26]). Decision about inclusion or exclusion of participants was made after the in-person assessment, followed by treatment allocation.

    The BiP TIC and BiP Anxiety trials had a randomized controlled design [24,29]. In BiP TIC, participants were randomized to either habit reversal training or exposure and response prevention. In BiP Anxiety, participants were randomized to either ICBT or an active control condition. The remaining trials had an open, within-group design.

    The clinical outcome measures were assessed at pre- and posttreatment. Activity data from the use of the internet platform (number of logins, written characters, login time duration, and completed treatment chapters) were automatically collected during the treatments. The iiPAS was administered by the clinicians participating in the trials, on 2 separate occasions: halfway through the treatment (iiPAS-mid) and at posttreatment (iiPAS-post).

    Measures

    Internet Intervention Patient Adherence Scale

    The iiPAS is a 5-item, clinician-rated measure of patient adherence to internet-delivered behavioral interventions. The 5 items are rated by treating clinicians and cover 5 central aspects of adherence: the patient’s work pace, engagement, communication with the clinician, motivation for change, and login frequency. Each item is rated on a 0 to 4 Likert scale. Thus, the iiPAS-total score ranges from 0 to 20, with 0 indicating no adherence and 20 perfect adherence. The items are designed to generically suit different types of internet-delivered behavioral interventions. A prerequisite, however, is that the intervention is guided, that is, clinician-supported, as opposed to unguided or entirely self-help. The scale is constructed to be applicable to children and adolescents as well as adults. In the case where caregivers are involved in the patient’s treatment, the iiPAS is primarily intended to evaluate the patient’s adherence to treatment rather than the parent’s or carer’s involvement. See Multimedia Appendices 1 and 2 for a full copy of iiPAS in Swedish and English and scoring instructions. For permission requests to use or translate the scale, please contact the corresponding author.

    Clinical Global Impression-Severity

    Clinical Global Impression-Severity (CGI-S [35]) is a commonly used clinician-rated measure of global illness severity. Clinical severity is rated on a 7-point Likert scale, ranging from “normal, not ill” to “severely ill.” The CGI-S is validated as a measure of clinical effectiveness within clinical trials as well as within routine clinical practice [36].

    Clinician Severity Rating

    CSR is rated in conjunction with the clinician-administered ADIS [26], which is a semistructured interview and is considered the gold standard in the assessment of pediatric anxiety disorders. The CSR is a clinician rating of clinical severity between 0 and 8, with 0 indicating absence of symptoms (or no disturbance in functioning and/or disability), 8 maximal severity, and 4 the cut-off for clinical severity (ie, meeting diagnostic criteria for the disorder). ADIS has previously shown excellent interrater reliability [37] and test-retest reliability [38].

    Platform Usage Variables

    The internet treatment platform that the patients logged in to for accessing treatment material (texts, videos, and exercises) provided user data on number of logins to the platform, duration of logged in time, and number of completed chapters and written characters by the patients in communication and exercises. The number of completed chapters was expressed in percentage relative to total number of chapters because of the different number of chapters in different interventions.

    Statistical Analyses

    Reliability Analyses

    Cronbach alpha was calculated as a measure of internal consistency. An exploratory factor analysis (EFA) was conducted regarding the factor structure of the iiPAS, including the Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy to determine the adequacy of the data for factor analysis as well as a Bartlett Test of Sphericity to determine the factorability of the five items.

    Validity Analyses

    Pearson correlations were used to explore convergent validity, defined as the association of the iiPAS with platform usage variables. In variables where outliers with large leverage were identified, Spearman correlation coefficient was used. The intercorrelations of iiPAS between the halfway treatment time point (iiPAS-mid) and at the end of treatment (iiPAS-post) were calculated. For some calculations, an iiPAS total score was used, which is the sum of iiPAS-mid and iiPAS-post, to obtain a measure of adherence that represents the whole treatment period from start to end of treatment. Regarding criterion validity, the association between iiPAS and symptom severity change from pre- to posttreatment was calculated. Because the available measures, CSR (in the BiP Anxiety group) and CGI-S (in the remaining groups), are similar clinician-rated, 1-item Likert scales, but not identical, the CSR and CGI-S change scores were z-transformed and then combined into a single new variable (“symptom severity change”), which was then further used in the analyses. There were only a few missing values in the dataset (2.7%), and missingness was, therefore, not further handled but listwise excluded from the respective analyses. Analyses were conducted in SPSS Statistics version 24 (IBM Corp) and R (R Core Group [39]).


    Study 1: Initial Evaluation

    An initial version of the iiPAS was administered to 50 children and adolescents, aged between 7 and 17 years, receiving ICBT for OCD or anxiety disorders. Preliminary analyses of Cronbach alpha indicated excellent internal consistency (alpha=.93).

    Study 2: Full Psychometric Evaluation

    Sample Characteristics

    The majority of the 148 participants were female (n=89, 60%), with a mean age of 12.7 years. Table 2 displays a detailed description of the characteristics of the participants in each trial.

    Table 3 presents descriptive statistics for iiPAS at the 2 time points midtreatment and posttreatment and for the summary value, iiPAS-total, as well as platform usage variables.

    Table 2. Sample characteristics (N=148).
    CharacteristicBiP TIC (n=23)BiP SoFT (n=30)BiP OCD (n=24)BiP Anxiety (n=46)BiP ERITA (n=25)Total
    Age, years:months

    		Mean (SD)12:3 (2.56)15:0 (1.22)11:5 (2.75)9:11 (1.36)15:9 (1.31)12:7 (2.94)

    		Min-max8-1713-178-178-1213-178-17
    Sex,n (%)

    		Female8 (35)25 (83)14 (58)23 (50)19 (76)89 (60.1)

    		Male15 (65)5 (17)10 (42)23 (50)1 (4)55 (37.1)

    		Other/nonbinary sexN/AaN/AN/AN/A5 (20)4 (2.7)
    Principal diagnosis, n (%)

    		Tourette syndrome/chronic tic disorder23 (100)—b———23 (15.5)

    		Social anxiety disorder—30 (100)—5 (11)—35 (23.6)

    		OCDc——24 (100)——24 (16.2)

    		Separation anxiety disorder———17 (37)—17 (11.4)

    		Generalized anxiety disorder———12 (26)—12 (8.1)

    		Specific phobia———9 (20)—9 (6.0)

    		Panic disorder———3 (7)—3 (2.0)

    		Nonsuicidal self-injury————25 (100)25 (16.8)

    aN/A: not applicable.

    bNo occurrence.

    cOCD: obsessive-compulsive disorder.

    Table 3. Descriptive statistics for internet intervention Patient Adherence Scale and platform usage variables.
    VariableMean (SD)Total score percentiles


    		5th25th50th75th95th
    iiPASa-mid13.43 (4.98)510151720
    iiPAS-post11.27 (6.01)07121619
    iiPAS-total24.74 (10.39)717263439
    Number of logins32 (16.49)—b————
    Logged in time (min)2277c (5673.92)—————
    Number of written characters18,135d (50,624.17)—————
    Completed chapters (%)74.4 (25.8)—————

    aiiPAS: internet intervention Patient Adherence Scale.

    bPercentiles not calculated.

    cMedian=762.

    dMedian=5510.

    Internal Consistency

    Table 4 displays Cronbach alpha for iiPAS-mid and iiPAS-post across the included clinical studies. The internal consistency of the iiPAS was excellent at midtreatment as well as at posttreatment.

    Table 4. Cronbach alpha for internet intervention Patient Adherence Scale.
    StudyiiPASa-midiiPAS-post
    BiPb TIC.90.95
    BiP SoFTc.96.96
    BiP OCDd.91.92
    BiP Anxiety.96.94
    BiP ERITAe.94.97
    All trials combined.93.95

    aiiPAS: internet intervention Patient Adherence Scale.

    bBiP: Barninternetprojektet

    cSoFT: Social Fobi Tonåring

    dOCD: Obsessive Compulsive Disorder

    eERITA: Emotion Regulation Individual Therapy for Adolescents

    Factor Structure

    For iiPAS-mid, a KMO=0.89 and Bartlett Test of Sphericity χ210=578.5, P<.001 indicated factorability. The results of the EFA indicated a single factor with an eigenvalue of 3.97 that explained 79.3% of the variance. For iiPAS-post, KMO was 0.85 and Bartlett Test of Sphericity χ210=787.0, P=.001, again indicating factorability. The EFA resulted in a single factor with an eigenvalue of 4.18 that explained 83.6% of the variance. Multimedia Appendix 3 presents communalities and factor loadings in both EFAs.

    Convergent Validity

    iiPAS-mid, iiPAS-post, and iiPAS-total scores were positively and significantly correlated with platform usage variables (r=0.25-0.79), apart from nonsignificant correlations between iiPAS and logged in time. There was also a substantial and significant correlation between iiPAS-mid and iiPAS-post (r=0.78). Visual inspection of the scatterplots did not indicate any nonlinear associations. Table 5 presents convergent validity correlations.

    Criterion Validity

    iiPAS-mid, iiPAS-post, and iiPAS-total scores correlated positively and significantly with the standardized symptom severity change score (r=0.17-0.19; Table 5). In contrast, number of logins, logged in time, written characters, and percentage of completed chapters were not significantly correlated with symptom severity change. A post hoc item-per-item analysis of the iiPAS-mid items regarding the symptom severity change score revealed a significant correlation with 2 of the 5 items, item 2 (“engagement with exercises”; r=0.21, P=.01), and item 4 (“motivation for change”; r=0.19, P=.02).

    Table 5. Internet intervention Patient Adherence Scale correlations (P values) with platform usage variables and symptom severity change.
    MeasureiiPASa-mid
    (P value)    		iiPAS-post
    (P value)    		iiPAS-total
    (P value)    		Number of logins
    (P value)    		Logged in time
    (P value)    		Written characters
    (P value)    		Completed chapters
    (P value)
    iiPAS-post.78b (<.001)—c—————
    iiPAS-total.93 (<.001).95 (<.001)—————
    Number of logins.53 (<.001).57 (<.001).59 (<.001)————
    Logged in time.15 (.07).06 (.47).11 (.21).19 (.03)———
    Written characters.25 (.003).31 (<.001).31 (<.001).36 (<.001).10 (.24)——
    Completed chapters.70 (<.001).79 (<.001).79 (<.001).58 (<.001).09 (.29).26 (<.001)—
    Symptom severity change.17 (.04).19 (.02).18 (.04).07 (.40)−.04 (.61)−.01 (.94).10 (.25)

    aiiPAS: internet intervention Patient Adherence Scale.

    bItalicized values indicating significant correlations at the <.05 level.

    cNot applicable.


    Principal Findings

    The objective of this project was to develop and psychometrically evaluate a novel measure of patient adherence in internet-delivered behavioral interventions. The measure was designed in close collaboration with experienced clinicians and researchers within the field and resulted in a clinician-rated, 5-item scale that was tested in a sample of children and adolescents with various behavioral disorders. Descriptive statistics indicated an appropriate range and variability of the iiPAS. The scale demonstrated excellent internal consistency. Consistently, factor analyses showed that the iiPAS measures one underlying factor, in agreement with the theoretical assumption that patient adherence within the context of internet interventions can be understood as one homogeneous construct.

    Objective behavioral data were available through automatically registered internet platform data of number of logins, logged in time, written characters, and percentage of completed chapters. The iiPAS was significantly associated with those objective variables, apart from logged in time, which strongly suggests that the iiPAS indeed measures key aspects of patient adherence. The iiPAS was also significantly associated with symptom severity changes after treatment. Post hoc analyses revealed that 2 of the 5 items at midtreatment, engagement and motivation to change, were particularly associated with a decrease in symptoms. The midtreatment administration of the iiPAS may be especially interesting from a clinical point of view, as half of the internet treatment is still ahead. For example, a low iiPAS midtreatment score could inform the clinician that the adherence of the patient should be addressed for optimal outcome. The strong correlation of the iiPAS-mid and iiPAS-post suggests that little change in adherence is to be expected in the second half of the treatment. This could mean that other interventions such as motivational interventions might be needed to increase adherence, or that the intervention might need to be intensified or augmented with telephone, video calls or in-person therapist support.

    Interestingly, none of the objective platform usage variables were associated with symptom severity change at posttreatment. This could possibly indicate that the mere adherence to the formal structure of an internet intervention (logging in, writing text, and completing chapters) is less important for the clinical outcome than actively engaging with the therapeutic content that is presented in the intervention (such as testing new behavioral or cognitive strategies and delivering homework exercises with high quality). The existing literature on patient adherence supports that quality of homework exercise completion is a better predictor of treatment outcome than number of completed exercises [10,13]. However, studies that are specifically designed to clarify the importance of adhering to the formal structure of a treatment in comparison with adhering to the therapeutic content are needed.

    Limitations and Future Directions

    Our analyses included diverse conditions, ranging from anxiety disorders, OCD, tics/Tourette’s disorder, and self-harming behaviors. Future studies should aim for an even broader evaluation of the iiPAS in additional diagnoses and symptoms that are currently treated with internet interventions, especially in the behavioral medicine field, where about half of the internet treatment trials are conducted [18]. Moreover, although the scale was designed to be used across the life span, the age range of the participants is limited. Further validation work in adult samples is warranted. In addition, the iiPAS was evaluated in clinical trials in Sweden and, albeit, with projects from different research groups, all were located at the same university and used the same internet platform, limiting the generalizability to other contexts. Future studies should evaluate the iiPAS in other languages as well as in different cultural, health care, and academic environments.

    Importantly, the iiPAS was designed to be useful in a wide range of internet-delivered, clinician-supported internet interventions of different therapeutic schools. However, the interventions in this study are all based on a CBT framework. An important next step would be to test the iiPAS in different internet-delivered behavioral interventions. Regarding the reliability analyses, an important aspect that was omitted in this study was the calculation of interrater reliability. Due to the scoring format of the iiPAS, this would require 2 clinicians for each patient, which was not feasible at the current stage of evaluation but would be important to take into account in future studies.

    Finally, the sample size in this study was big enough for a global evaluation but too small to conduct detailed separate analyses for each specific condition or age group. A larger sample would allow analysis of the association between patient adherence and clinical outcomes in greater detail with regards to specific patient characteristics such as age, comorbidity, and symptom severity at baseline.

    Conclusions

    To summarize, the iiPAS demonstrated sound psychometric characteristics in a clinical sample of children and adolescents with a broad range of psychiatric and behavioral problems. The iiPAS appears to be a valuable tool to measure and study different aspects of adherence within the context of clinician-guided, internet-delivered behavioral interventions. Further evaluations within different age groups, types of internet-delivered interventions, and patient populations are warranted.

    Acknowledgments

    The project was funded by the Swedish Research Council for Health, Working Life and Welfare (Forte 2014-4052).

    Authors' Contributions

    FL initiated the conception of the iiPAS. FL, SV, MJ, MN, PA, KA, TW, JH, and DMC revised the iiPAS and contributed to the final version of the scale; FL, MJ, SV, TW, MN, JB, HS, PA, KA, and KM contributed to the data collection; FL, KM, and JH conducted the statistical analyses; and FL, KM, and JH drafted the initial manuscript. All authors contributed to the revisions of the manuscript and approved the final version.

    Conflicts of Interest

    None declared.

    ‎
    Multimedia Appendix 1

    Internet intervention Patient Adherence Scale—Swedish.

    PDF File (Adobe PDF File)69 KB
    ‎
    Multimedia Appendix 2

    Internet intervention Patient Adherence Scale—English.

    PDF File (Adobe PDF File)161 KB
    ‎
    Multimedia Appendix 3

    Supplemental tables—factor analyses: communalities and factor loadings.

    PDF File (Adobe PDF File)30 KB
    1. Haynes BR, Taylor WD, Sackett DL. Compliance in Health Care. Baltimore, Maryland: Johns Hopkins University Press; 1979.
    2. Nock MK, Ferriter C. Parent management of attendance and adherence in child and adolescent therapy: a conceptual and empirical review. Clin Child Fam Psychol Rev 2005 Jun;8(2):149-166. [CrossRef] [Medline]
    3. Mausbach BT, Moore R, Roesch S, Cardenas V, Patterson TL. The relationship between homework compliance and therapy outcomes: an updated meta-analysis. Cognit Ther Res 2010 Oct;34(5):429-438 [FREE Full text] [CrossRef] [Medline]
    4. Jungbluth NJ, Shirk SR. Therapist strategies for building involvement in cognitive-behavioral therapy for adolescent depression. J Consult Clin Psychol 2009 Dec;77(6):1179-1184. [CrossRef] [Medline]
    5. Martin DJ, Garske JP, Davis MK. Relation of the therapeutic alliance with outcome and other variables: a meta-analytic review. J Consult Clin Psychol 2000 Jun;68(3):438-450. [CrossRef] [Medline]
    6. Beutler LE, Harwood TM, Michelson A, Song X, Holman J. Resistance/reactance level. J Clin Psychol 2011 Feb;67(2):133-142. [CrossRef] [Medline]
    7. Button ML, Westra HA, Hara KM, Aviram A. Disentangling the impact of resistance and ambivalence on therapy outcomes in cognitive behavioural therapy for generalized anxiety disorder. Cogn Behav Ther 2015;44(1):44-53. [CrossRef] [Medline]
    8. Rubak S, Sandbaek A, Lauritzen T, Christensen B. Motivational interviewing: a systematic review and meta-analysis. Br J Gen Pract 2005 Apr;55(513):305-312 [FREE Full text] [Medline]
    9. Kazantzis N, Whittington C, Zelencich L, Kyrios M, Norton PJ, Hofmann SG. Quantity and quality of homework compliance: a meta-analysis of relations with outcome in cognitive behavior therapy. Behav Ther 2016 Sep;47(5):755-772. [CrossRef] [Medline]
    10. Simpson HB, Maher MJ, Wang Y, Bao Y, Foa EB, Franklin M. Patient adherence predicts outcome from cognitive behavioral therapy in obsessive-compulsive disorder. J Consult Clin Psychol 2011 Apr;79(2):247-252 [FREE Full text] [CrossRef] [Medline]
    11. Simpson HB, Marcus SM, Zuckoff A, Franklin M, Foa EB. Patient adherence to cognitive-behavioral therapy predicts long-term outcome in obsessive-compulsive disorder. J Clin Psychiatry 2012 Sep;73(9):1265-1266 [FREE Full text] [CrossRef] [Medline]
    12. Hughes AA, Kendall PC. Prediction of cognitive behavior treatment outcome for children with anxiety disorders: therapeutic relationship and homework compliance. Behav Cogn Psychother 2007 May 25;35(4):487-494. [CrossRef]
    13. Lee P, Zehgeer A, Ginsburg GS, McCracken J, Keeton C, Kendall PC, et al. Child and adolescent adherence with cognitive behavioral therapy for anxiety: predictors and associations with outcomes. J Clin Child Adolesc Psychol 2019;48(Sup 1):S215-S226. [CrossRef] [Medline]
    14. Glenn D, Golinelli D, Rose RD, Roy-Byrne P, Stein MB, Sullivan G, et al. Who gets the most out of cognitive behavioral therapy for anxiety disorders? The role of treatment dose and patient engagement. J Consult Clin Psychol 2013 Aug;81(4):639-649 [FREE Full text] [CrossRef] [Medline]
    15. DiMatteo MR, Giordani PJ, Lepper HS, Croghan TW. Patient adherence and medical treatment outcomes: a meta-analysis. Med Care 2002 Sep;40(9):794-811. [CrossRef] [Medline]
    16. Andersson G. Using the internet to provide cognitive behaviour therapy. Behav Res Ther 2009 Mar;47(3):175-180. [CrossRef] [Medline]
    17. Hedman E, Ljótsson B, Lindefors N. Cognitive behavior therapy via the internet: a systematic review of applications, clinical efficacy and cost-effectiveness. Expert Rev Pharmacoecon Outcomes Res 2012 Dec;12(6):745-764. [CrossRef] [Medline]
    18. Vigerland S, Lenhard F, Bonnert M, Lalouni M, Hedman E, Ahlen J, et al. Internet-delivered cognitive behavior therapy for children and adolescents: a systematic review and meta-analysis. Clin Psychol Rev 2016 Dec;50:1-10 [FREE Full text] [CrossRef] [Medline]
    19. Eysenbach G. The law of attrition. J Med Internet Res 2005 Mar 31;7(1):e11 [FREE Full text] [CrossRef] [Medline]
    20. Sieverink F, Kelders SM, van Gemert-Pijnen JE. Clarifying the concept of adherence to ehealth technology: systematic review on when usage becomes adherence. J Med Internet Res 2017 Dec 6;19(12):e402 [FREE Full text] [CrossRef] [Medline]
    21. Kelders SM, Kok RN, Ossebaard HC, van Gemert-Pijnen JE. Persuasive system design does matter: a systematic review of adherence to web-based interventions. J Med Internet Res 2012 Nov 14;14(6):e152 [FREE Full text] [CrossRef] [Medline]
    22. Donkin L, Christensen H, Naismith SL, Neal B, Hickie IB, Glozier N. A systematic review of the impact of adherence on the effectiveness of e-therapies. J Med Internet Res 2011 Aug 5;13(3):e52 [FREE Full text] [CrossRef] [Medline]
    23. Furr RM. Scale Construction and Psychometrics for Social and Personality Psychology. London, UK: Sage Publications; 2011.
    24. Jolstedt M, Wahlund T, Lenhard F, Ljótsson B, Mataix-Cols D, Nord M, et al. Efficacy and cost-effectiveness of therapist-guided internet cognitive behavioural therapy for paediatric anxiety disorders: a single-centre, single-blind, randomised controlled trial. Lancet Child Adolesc Health 2018 Nov;2(11):792-801. [CrossRef] [Medline]
    25. Lenhard F, Andersson E, Mataix-Cols D, Rück C, Vigerland S, Högström J, et al. Therapist-guided, internet-delivered cognitive-behavioral therapy for adolescents with obsessive-compulsive disorder: a randomized controlled trial. J Am Acad Child Adolesc Psychiatry 2017 Jan;56(1):10-9.e2 [FREE Full text] [CrossRef] [Medline]
    26. Silverman WK, Albano AM. Clinical Manual: Anxiety Disorders Interview Schedule for DSM-IV (Child and Parent Versions). New York: Oxford Press; 1996.
    27. Scahill L, Riddle MA, McSwiggin-Hardin M, Ort SI, King RA, Goodman WK, et al. Children's Yale-brown obsessive compulsive scale: reliability and validity. J Am Acad Child Adolesc Psychiatry 1997 Jun;36(6):844-852. [CrossRef] [Medline]
    28. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders: DSM-5. Fifth Edition. Arlington, VA: American Psychiatric Publishing; 2013.
    29. Andrén P, Aspvall K, de la Cruz LF, Wiktor P, Romano S, Andersson E, et al. Therapist-guided and parent-guided internet-delivered behaviour therapy for paediatric Tourette's disorder: a pilot randomised controlled trial with long-term follow-up. BMJ Open 2019 Feb 15;9(2):e024685 [FREE Full text] [CrossRef] [Medline]
    30. Nordh M, Vigerland S, Öst LG, Ljótsson B, Mataix-Cols D, Serlachius E, et al. Therapist-guided internet-delivered cognitive-behavioural therapy supplemented with group exposure sessions for adolescents with social anxiety disorder: a feasibility trial. BMJ Open 2017 Dec 14;7(12):e018345 [FREE Full text] [CrossRef] [Medline]
    31. Aspvall K, Andrén P, Lenhard F, Andersson E, Mataix-Cols D, Serlachius E. Internet-delivered cognitive behavioural therapy for young children with obsessive-compulsive disorder: development and initial evaluation of the BIP OCD junior programme. BJPsych Open 2018 May;4(3):106-112 [FREE Full text] [CrossRef] [Medline]
    32. Bjureberg J, Sahlin H, Hedman-Lagerlöf E, Gratz KL, Tull MT, Jokinen J, et al. Extending research on emotion regulation individual therapy for adolescents (ERITA) with nonsuicidal self-injury disorder: open pilot trial and mediation analysis of a novel online version. BMC Psychiatry 2018 Oct 11;18(1):326 [FREE Full text] [CrossRef] [Medline]
    33. Vigerland S, Ljótsson B, Thulin U, Öst LG, Andersson G, Serlachius E. Internet-delivered cognitive behavioural therapy for children with anxiety disorders: a randomised controlled trial. Behav Res Ther 2016 Jan;76:47-56 [FREE Full text] [CrossRef] [Medline]
    34. Sheehan DV, Sheehan KH, Shytle RD, Janavs J, Bannon Y, Rogers JE, et al. Reliability and validity of the Mini International Neuropsychiatric Interview for Children and Adolescents (MINI-KID). J Clin Psychiatry 2010 Mar;71(3):313-326. [CrossRef] [Medline]
    35. Guy W. Assessment Manual for Psychopharmacology. Washington, DC: US Government Printing Office; 1976.
    36. Berk M, Ng F, Dodd S, Callaly T, Campbell S, Bernardo M, et al. The validity of the CGI severity and improvement scales as measures of clinical effectiveness suitable for routine clinical use. J Eval Clin Pract 2008 Dec;14(6):979-983. [CrossRef] [Medline]
    37. Ost LG, Svensson L, Hellström K, Lindwall R. One-session treatment of specific phobias in youths: a randomized clinical trial. J Consult Clin Psychol 2001 Oct;69(5):814-824. [CrossRef] [Medline]
    38. Silverman WK, Saavedra LM, Pina AA. Test-retest reliability of anxiety symptoms and diagnoses with the anxiety disorders interview schedule for DSM-IV: child and parent versions. J Am Acad Child Adolesc Psychiatry 2001 Aug;40(8):937-944. [CrossRef] [Medline]
    39. R Core Team. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing; 2017.

    ‎
    ADIS: Anxiety Disorders Interview Schedule
    CBT: cognitive behavioral therapy
    CGI-S: Clinical Global Impression-Severity
    CSR: Clinician Severity Rating
    DSM: Diagnostic and Statistical Manual of Mental Disorders
    EFA: exploratory factor analysis
    ICBT: internet-delivered cognitive behavioral therapy
    iiPAS: internet intervention Patient Adherence Scale
    KMO: Kaiser-Meyer-Olkin
    OCD: obsessive-compulsive disorder
    RCT: randomized controlled trial

    Edited by G Eysenbach; submitted 08.02.19; peer-reviewed by E Gustafson, A Radomski, C Young; comments to author 28.03.19; revised version received 06.06.19; accepted 29.06.19; published 01.10.19

    Copyright

    ©Fabian Lenhard, Kajsa Mitsell, Maral Jolstedt, Sarah Vigerland, Tove Wahlund, Martina Nord, Johan Bjureberg, Hanna Sahlin, Per Andrén, Kristina Aspvall, Karin Melin, David Mataix-Cols, Eva Serlachius, Jens Högström. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 01.10.2019

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.