Attrition Within Digital Health Interventions for People With Multiple Sclerosis: Systematic Review and Meta-analysis

Introduction

Multiple sclerosis (MS) is a chronic neurological disease that affects over 2 million people globally [1]. The disease course of MS is highly variable and can be associated with a progressive decline in physical and cognitive function. The current treatment for MS involves the use of disease-modifying treatments and symptom management; however, the delivery of health care for MS is becoming increasingly supported by digital health interventions.

People with MS readily seek out web-based information on their disease and appear to do so at rates higher than those with other neurological conditions [2,3]. Further, many people with MS engage with digital technologies to manage their disease, including exchanging medical information with health care providers or to assist in making and maintaining positive lifestyle changes [4]. Digital health interventions that support learning to self-manage MS and scaffold learning and practicing over a specific period of time are a particular subset of digital health interventions for people with MS. Evidence suggests that there are barriers to remaining engaged for the recommended duration in these interventions [5] and technological elements that could support engagement (such as interactivity, multimedia components, or feedback) can play a pivotal role in ameliorating attrition. Understanding the extent to which these technological elements may impact engagement by people with MS in such digital health interventions remains to be explored and is a necessary next step in designing and evaluating such digital health interventions.

We conducted a meta-analysis to address the following primary questions: (1) is there a difference in attrition for participants that were allocated to the intervention in randomized controlled trials (RCTs) of digital health interventions for people with MS between intervention and control arms? (2) How do study characteristics and technological elements of digital health interventions influence observed degrees of attrition?

Methods

Eligibility Criteria

Studies were eligible for inclusion in the review if they met each of the following predetermined criteria: (1) describe RCTs examining a treatment intervention or interventions for MS; (2) were delivered via a technological platform (ie, a “computer-assisted” or “web/internet-based intervention” or “eHealth/mHealth”), limited to a PC or to Mac, smartphone, and tablet devices. Interactivity and communication over a specified period of time were important components; therefore, episodic teleconferencing or telemonitoring and virtual reality–only interventions were excluded; (3) included study participants with a diagnosis of MS (any type); (4) reported attrition for those allocated to the intervention or control group upon conclusion of the intervention and study period, and the intervention and control arms were randomized from the same subject population; that is, no comparisons with healthy controls.

Search Strategy

A literature search was conducted on April 30, 2021, for published studies in the following databases: IEEE, Medline, Scopus, and CINAHL. The search contained terms related to identifying online interventions: (online OR web-based OR internet OR digital OR virtual OR computer-assisted OR mhealth OR mobile OR smartphone OR ehealth OR telehealth OR telemedicine OR app), a term identifying multiple sclerosis (multiple sclerosis), and a Medical Subject Headings (MeSH) term (telemedicine).

Reference lists and in-text citations were also screened for additional studies, as well as recommendations via correspondence from authors of other included studies.

Selection and Data Collection

Duplicates were first removed within and between databases. The remaining articles were then screened on the basis of their titles and abstract; thereafter, the eligibility of the final papers was confirmed following review of the full-text articles (Figure 1). Extraction of study characteristics and outcome data was conducted by authors WB and TW.

The following study characteristics were extracted: study design, study population, country, intervention type, control and comparison intervention, mean age of participants in both arms, percentage of female participants, years since diagnosis, length of intervention in weeks, number of sessions, length of time to final follow-up assessment in weeks, attrition in the intervention and control arms upon conclusion of the intervention, and primary outcome measures. In studies where there were multiple intervention or control arms, attrition data from intervention arms were combined and attrition data from control arms were combined for meta-analysis purposes [7].

Each study was scored on the basis of its technological elements by WB in accordance with the published manuscript. The scoring system used was adapted from Barakat et al [8], who developed it for use in web-based eTherapy programs targeted to eating disorders. Studies were scored in accordance with four domains:

• multimedia channels (written text=“1,” graphics/images=“1,” audio/voiceover=“2,” video=“3,” simulation/3D virtual reality=“4”);
• degree of user interactivity (questionnaires=“1,” quizzes=“1,” goal setting/to‐do list=“2,” homework tasks=“3,” user dashboard=“3,” forums=“3,” self‐monitoring tools=“4,” interactive exercises=“4,” virtual games=“4,” video coaching with professional=“4”);
• level of automated feedback (motivational pop‐ups=“1,” reminders=“2,” nonpersonalized feedback=“3”), personalized feedback (telephone or email)= “4”);
• technological device through which the program was made accessible (outdated technology; eg, compact disk read‐only memory (CD‐ROM)=“1,” modern technology; eg, tablet, desktop computer, laptop, and mobile phone=“2”).

The reporting of technological features of digital health interventions was often vague or absent in publications; therefore, it was necessary to survey each author regarding their examined web-based intervention to clarify scores in accordance with the system adapted from Barakat et al [8]. An email was sent out to authors who had not completed the survey. These authors were sent a summary of the initial scoring by WB and asked to confirm that all the elements listed were present in their study and to identify any that were missed. In total, 19 of the 32 studies responded and the remaining 11 provided additional elements that were not identified in the initial screen by WB. In total, 3 of the 33 studies [9-11] provided access to the program itself and provided confirmation of elements.

‎View this figure

Quality Appraisal

The methodological quality of the final studies included in the review was assessed by author WB using the Cochrane Risk of Bias tool 2 [12]. The tool uses five domains to assess the bias within study design: allocation concealment and random sequence generation, blinding of participants and study personnel, blinding of outcome assessment, incomplete outcome data, and selective reporting of data. Briefly, each domain is composed of questions, to which the response can be “yes,” “possibly yes,” “possibly no,” “no,” and “no information.” Based on answers, each domain is scored as “low risk,” “some concerns,” or “high risk,” whereby the overall judgement is made by the combination of all domains.

Statistical Analyses

Analyses were conducted using Stata (Stata/SE 16 for Mac, StataCorp).

Meta-analysis

The Stata command meta [13] was used for meta-analysis. The input variables required by meta were contained in a 2×2 table; that is, number of individuals who did and did not experience the “outcome event” in either the treatment or the control group. To compare study attrition rates between the intervention groups and the control group, risk ratios were meta-analyzed using the default restricted maximum likelihood method. Log-transformed relative risk of attrition was computed separately for each study, weighted by the inverse of study variance, and pooled to create a summary effect. Studies with no loss to follow-up in one of the intervention or control arms were adjusted by adding 0.5 to each cell within a study’s 2×2 table, enabled by meta-analysis settings within Stata. Studies with no attrition in both intervention and control arms were excluded from the meta-analysis as is best practice [14]. Heterogeneity was assessed using the I² statistic. Publication bias was assessed by funnel plots. Data were exponentiated and also presented as a Forest plot using the meta forestplot command in Stata.

Meta-regression

Meta-regression was used to examine the relationship between the log-transformed relative attrition rates, and various study characteristics, participant demographics, and intervention technological elements. Study characteristics included the type of control, type of intervention, duration of the study, and duration to the final follow-up. Type of control was dummy-coded as follows: wait-list, active control, usual care or active control computerized (1, 2, 3, 4). The type of intervention was dichotomized owing to small numbers of observations dummy-coded as follows: containing an exercise or physiotherapy condition or other (1, 2). The length of the intervention period was converted to years and evaluated as a continuous variable.

Demographic elements included the type of MS, mean age of the participants, female-to-male ratio, and years since disease onset. The type of MS was dummy-coded as either all MS types or relapsing-remitting MS (1, 2).

The intervention technological elements that could support engagement were as follows: multimedia channels, interactivity, and feedback; overall score was the sum of these 3 elements. This analysis omitted the element that assessed how the intervention was made accessible since all interventions used desktop computers. An attempt was made to further clarify whether interventions were designed to accommodate multiple platforms, but this remained unclear for many studies.

Variables were first examined individually and then jointly as a single meta-regression model informed by a directed acyclic graph.

Results

Results Overview

The database search retrieved 1167 articles, of which 187 were duplicates. After duplicates were removed, titles and abstracts of the remaining 835 were screened. Of these, 127 articles were obtained for full-text review. In total, 95 papers did not meet the inclusion criteria and were excluded after full-text review. Of the excluded papers, 34 were not applicable interventions, 36 were not RCTs, 10 had the wrong subject population, and 15 others were excluded for other reasons (Multimedia Appendix 1). In total, 32 papers that described 32 studies were eligible for meta-analysis.

Study Characteristics

Details of the study characteristics for the 32 included papers are reported in Table 1. Of the 32 studies, 10 were published before 2015, and the earliest paper was published in 2004. A total of 23 of 32 studies included all types of MS, 8 included only people with relapsing-remitting MS, and 1 included primary-progressive MS (PPMS). Two included any type of MS if fatigue was a symptom, 2 others included any type of MS if depression was a symptom, 1 assessed whether participants experienced migraines, and 1 only assessed whether participants experienced severe disability. The mean length of interventions was 14.0 (SD 10.2) weeks, and the mean length to last follow-up was 20.9 (SD 12.1) weeks. In total, 20 of 32 studies reported a high risk of bias upon quality appraisal with none reporting low risk (Figure 2).

‎View this figure

Study Populations

The mean sample size across all 32 studies was 97.7 (SD 121.8). The smallest study contained 18 participants [19] and the largest 682 participants [41]. The median numbers of participants in the intervention and control arms at the final follow-up were 33 and 27, respectively.

Types of Digital Health Intervention and Control

All studies included a program aimed at people with MS delivered via web-based technology, with 2 studies [36,37] that included in-person components. The program ranged from 0.14 week to 52 weeks intended duration. In total, 15 of the 32 studies included a cognitive behavioral training element, 11 included an exercise training component, 2 included web-based physiotherapy, 1 included a fall prevention intervention, 1 included a program to build “returning to work” skills, 1 included a program to build health literacy, and 1 included a mindfulness intervention.

In total, 12 of the 32 studies used a wait-list control, while 7 studies used a usual care control. Eight studies used a noncomputerized active control, which included a physical activity regime [17-19,42], physiotherapy [11], modified version of the intervention [38,39], or non–MS-specific psychoeducation [40]. Five studies used a computerized active control, which were all broadly modified versions of the intervention [26,33,35,41,43].

Digital Health Intervention Characteristics App

Complete scoring information for each individual study is available in Multimedia Appendix 2.

Multimedia

The mean multimedia score across all 32 studies was 3.5 (SD 2.2). In total, 29 of the 32 studies contained written text. Two of the remaining 3 studies that did not include written text were a gamified web-based cognitive training program [36,37], and 1 study described a web-based, asynchronous messaging system that uses electronic personal health records [42]. In total, 15 of 32 studies contained video or animation, half of all studies including images or graphics, and only one study used audio voiceover to augment written material [27].

Interactivity

The mean interactivity score across all 32 studies was 5.2 (SD 3.3). Eight of 32 studies contained homework tasks [9,27,32,34, 36-38,40], which varied from set questions or self-directed learning to diary entries between course sessions. In total, 15 of 33 studies contained a self-monitoring component [10,11,15,16,18,21-25,31,32,38,42], which varied from a validated tool integrated within the program to generic mood-related questions and diary entries. Nine studies included interactive exercises or games [26,28,30,33,35-37,39,43], 5 studies contained goal-setting elements [15,21,22,24,38], 3 contained forums for interaction with other participants [20-22], 3 contained video coaching or interactive meditation by professionals [15,24,40], and 1 contained a questionnaire or quiz [32].

Feedback

The mean feedback score across all 32 studies was 2.8 (SD 2.4). In total, 14 of 32 studies included feedback [9-11,16, 18,19,21,22,27,28,36,37,39,40], which ranged from personalized feedback from a trained staff member or professional to general feedback delivered automatically via the web-based platform. Reminders were included in 6 of 17 studies [15,17,20,25,32,42], which prompted users to complete the sessions, while 2 studies also included reminders on when new content was added or existing content was updated [21,22].

Meta-analysis

The final meta-analysis included 29 studies after 3 exclusions due to no attrition in both the intervention and control arms. Meta-analysis described an overall effect size of =–0.993 (95% CI 0.95-1.04) and a P value of .75, which indicates no statistical difference in the degree of attrition between the intervention and control arms (Figure 3). The test of homogeneity of study-specific effect sizes (Q) was rejected (P<.01). The data also suggested the presence of a medium level of heterogeneity between studies (I²=50.50). There is evidence of publication bias as described by the funnel plot showing some asymmetry (Figure 4).

‎View this figure

‎View this figure

Meta-regression

Univariable Association

Univariable assessment of study characteristics and eHealth elements are described in Table 2. No significant association between length-of-treatment, weeks to last follow-up, type of control, MS type, feedback or interactivity, or attrition was observed. Lower attrition was observed when eHealth elements overall score and multimedia score were higher.

Multivariable Models

Adjustment for other covariates did materially impact the association of overall score with attrition, which was significant (P<.01; Table 3). After adjustment, the multimedia subscore remained a significant association (P=.03; Table 4).

Discussion

Principal Findings

The aim of this systematic review and meta-analysis was to describe the study characteristics and technological components of digital health interventions that supported people with MS through a program of learning and practicing self-management, and then investigate any associations these factors may have with attrition. Our meta-analysis found no difference in attrition between participants allocated to intervention and control arms of the included RCTs. A multivariable association was found between degree of attrition and overall score, which was not observed in the univariable model, and between the degree of attrition and the multimedia subscore in both univariable and multivariable models. This suggests that among included studies, the interactivity subscore was the main driver in the association observed between overall score and degree of attrition; that is, the likelihood of attrition within the intervention arm compared with the control arm was associated with the inclusion of interactive elements such as self-directed tasks or self-monitoring.

Only 8 of the 32 studies described a design process or framework by which the intervention was developed. Development methods included engagement with the MS community [9,10,31,32,38], design with practitioners and allied health professionals [34], both [31], or followed a preset development plan [23]. There was no association observed between the number or type of technological elements used within the intervention and whether the study used or described a development framework. It is possible that a greater degree of consultation with the MS community during design phases does not necessarily lead to a great quantity of or more complex technological features. This highlights that while engagement with communities during the design phase is important, it is crucial to ensure this engagement is constructive and not perfunctory. Population preferences for number and type of features require further exploration; however, our findings suggest that elaborate features are not necessarily what people with MS want in a web-based intervention of this kind.

Interestingly, length of intervention was not associated with attrition in either control or intervention arms. Intuitively, the longer the intervention itself, the greater the potential attrition from these studies, in which a negative relationship between attrition and length of treatment for self-management interventions for people with MS has been reported [45]. It is possible that the role of treatment length in attrition may only be true at longer time frames than in the included studies of both meta-analyses. Another likely variable is the number of sessions or time in contact with participants. It is possible that rather than the length of the intervention being a mediator, it is the time participants are exposed to the intervention (“dose”). Studies infrequently report these data; therefore, this needs to be addressed in future studies.

Another interesting finding is that the mean average attrition rates in the intervention and control arms were 14.7% and 15.6%, respectively, which is a useful figure for calculating sample sizes for future studies. A previous meta-analysis investigating attrition rates based on published studies of existing self-management interventions for people with MS reported pooled attrition rates of 16.8% and 14.4% for the intervention and control groups, respectively [45]. Interestingly, studies that used face-to-face delivery of interventions were described as having higher attrition rates. Our analysis supports their pooled data attrition rates, which indicates that there may not actually be a difference between face-to-face and non–face-to-face digital health interventions of this kind. Further, their meta-analysis described associations between attrition and sex, age, and length of intervention, which was not observed in our data set. As their data was pooled between face-to-face and other modes of delivery, it is possible that the web-based mode of delivery may be mediating that relationship in our analysis.

Importantly, 9 of 32 studies did not report any outcomes related to the use of the intervention or technological elements within their intervention; they only reported primary outcome measures related to health outcomes. In total, 15 of 32 studies reported logins, app use, or sessions completed, while only 4 studies reported any interactions between participants and technological elements that were not directly related to the primary outcome measure. This has implications of measuring the “dose” for digital health interventions, whereby factors not reported may have an impact on the outcome measures reported as well as on attrition. Studies describing digital health interventions should address this in the future.

Limitations

Technological elements were underreported in published papers as evidenced by the authors of 10 of 32 studies needing to be followed up to provide information not accessible within the papers. It is possible that this analysis failed to capture the entirety of the included technological elements present within these studies; however, this is unlikely with the follow-up procedure. The need for better reporting of digital health interventions is crucial in ensuring reproducibility and assessment of web-based interventions, especially as several accessible frameworks exist directed at digital tools [44,46] or more generally [47]. Providing access to explore the intervention itself—as 3 authors did for this study—may be another solution, although it is not feasible for some studies.

Digital health interventions are not new; however, there have been fewer published studies on their use in MS compared with other chronic conditions; a greater number of studies would make the meta-regression more reliable. Further stratification of participants may provide greater insight into the attrition in these interventions by accounting for digital literacy, level of access to technology infrastructure, and better general reporting of the intervention design overall.

While the scoring system used for technological component ratings in this review has previously been used and was created in accordance with guidelines for digital intervention development, this system is not yet validated. Several additions were made to this scoring system to accommodate features that were not included, and it is possible that these additions have compromised internal validity. Analysis was undertaken using the previous, unmodified scoring system and revealed no changes to the results using our modified scoring system; however, caution must be applied to interpreting these results before validation has occurred. Additionally, while the scoring was derived from both publication and authors themselves, it is possible there were inaccuracies due to misinterpretation or inability to recall.

It is possible that using qualitative methods and direct observations to understand the ways in which people with MS engage with technological components of these digital health interventions is a better predictor of attrition than merely quantifying and scoring the components themselves. Unfortunately, no studies provided the data required to carry out these analyses. It also possible that analyzing individual components of the subscores for multimedia, interactivity and feedback, rather than overall subscores themselves, may elucidate further associations; however, the weighting within these subscores of the different components aims to address this.

The inclusion criteria were also strict on the definition of the digital health interventions of interest to us. This excluded studies that were exclusively virtual reality interventions and those that were exclusively telemonitoring or conferencing. These exclusions may have biased our data. It is noteworthy that the category of digital health interventions of interest to us is not well-defined in the major taxonomy of digital health interventions published by the World Health Organization; the category Interventions for Clients, this describes “communication” but overlooks the structuring and scaffolding of interventions that would be better described as “education” [48].

Conclusions

In conclusion, this study describes no difference in the rates of attrition between participants allocated to the intervention and control arms in digital health interventions for people with MS. An association between overall technological elements score and attrition was observed; however, the underlying mechanism is unclear. Future studies should further investigate the roles these elements play in retention in web-based interventions.