Factors Associated With Dropout of Participants in an App-Based Child Injury Prevention Study: Secondary Data Analysis of a Cluster Randomized Controlled Trial

Introduction

Owing to recognized advantages such as cost-effective dissemination, real-time data collection and feedback, reduced burden, flexible customization, self-monitoring capacity, and visually attractive multimedia presentation [1-3], mobile health (mHealth) technology has become increasingly popular in health intervention research and practice over the past decade. A wide range of mHealth interventions have been developed to prevent diseases and injuries, increase the compliance to recommended health interventions, and offer remote access to health services [4-10].

Despite these advantages, mHealth interventions suffer from the challenge of high dropout attrition rates compared to studies adopting traditional interventions. For example, a recent 6-month large intervention study for smoking cessation reported an attrition rate of 57% in the intervention group using an app and of only 52% in a control group using a self-help booklet as usual education [11]. Several app-based intervention studies reported high attrition rates, with estimates ranging from 38% to 84% [12-16].

When attrition is high, occurs unequally between intervention and control groups, or occurs in a nonrandom way, it threatens the validity of evaluation studies [17,18]. When dropout attrition is high in public health practice, the efficacy of an intervention is lower than desired. Therefore, researchers must prioritize exploration and understanding of the factors associated with dropout attrition of study participants. Increased understanding would lead to feasible approaches to prevent and reduce dropout attrition.

Previous research examining predictors of dropout attrition from app-based intervention studies in reducing alcohol intake [19], improving health-related behavior [20], and weight loss [21] focused primarily on the demographic characteristics of individuals who are more likely to fail to complete follow-up surveys and therefore are not retained in the research study. For example, in an evaluation of the efficacy of an app-based child burn prevention program, Burgess et al [22] reported a higher proportion of university degree holders among participants who remained in the study (28.7%, 70/244) than those who were lost to follow up (16.5%, 42/254). Despite the value of this line of research, it is equally important to explore relationships between attrition and other indicators such as engagement with the app-based intervention. Such analyses may offer new clues to reducing attrition in app-based intervention studies.

Therefore, the aim of this study was to examine associations both between attrition and demographic characteristics and between attrition and participants’ engagement in the intervention. We hypothesized that a more active and attractive type of intervention would result in greater engagement and thus a lower attrition rate in the intervention arm compared with the control arm. Replicating Burgess et al [22] and others, we also hypothesized that greater education might be related to higher retention in the intervention study. We used data collected from a cluster randomized controlled trial (RCT) examining an app intervention to prevent unintentional child injury in China to test our hypotheses.

Methods

Design

This research comprises secondary analyses of data from a published single-blinded cluster RCT assessing the effectiveness of an app-based intervention for caregivers to prevent unintentional injury among Chinese preschoolers [8]. The trial recruited 2920 caregivers of 3-6–year-old children from 20 eligible preschools using cluster random sampling. The randomization was performed at the school level and was stratified by type of preschool (public vs private), yielding five public and five private schools in both the intervention group (implementing unintentional child injury and parenting education) and the control group (implementing parenting education only).

Both groups engaged in a 6-month intervention delivered through a smartphone app. The intervention group received a more active and attractive intervention than the control group. Specifically, the control group received essays, games, comics, and videos twice a week, and users conducted thematic discussion activity once a month to learn about common children’s diseases and to practice parenting skills outside unintentional injury prevention. In contrast, the intervention group received all content that the control group received, but the participants were also exposed to additional similar essays, games, comics, and videos twice a week, and engaged in a second set of thematic discussions once a month to learn about parenting skills related to unintentional child injury prevention. The intervention group also had access to an app-based portal that supported communication between users and professionals concerning injury prevention knowledge and skills.

Data were collected through three caregiver surveys that were completed at baseline in December 2017 and then two follow-up surveys in March and June 2018, which corresponded to the 3rd and 6th month after the initiation of the interventions in both groups. To encourage adherence in using the app, reminders about newly released knowledge segments appeared on the scroll screen when users opened the app. Users who did not read the content in a timely manner were reminded again. We also provided financial incentives to participants after completing study of the knowledge segments and asked the preschool teachers to remind caregivers regularly to use the app. In addition, an automatic message to remind participants about using the app was sent if the user failed to log in for 1 month. To encourage compliance in completing the study surveys, we provided financial incentives and sent two automatic reminder messages to users who did not complete the follow-up surveys 3 days after they were released.

Figure 1 illustrates the study design. The protocol was approved by the Ethics Committee of Xiangya School of Public Health, Central South University, Changsha, China (approval number: XYGW-2017-02). Full details of the RCT evaluation are published elsewhere [23].

‎View this figure

Outcome Measure

Attrition from the study was defined as a study participant who failed to complete both follow-up surveys at the 3rd and 6th months after initiation of the interventions.

Independent Variables

Based on our study hypotheses and previous studies [17,22], we considered demographic factors and engagement with the app-based intervention as potential factors that would predict dropout attrition of study participants. We also considered previous training experience on unintentional injury prevention, as we hypothesized that participants might engage in the intervention or study only if they valued parenting education and if they were receiving new information that they had not learned in the past.

Data on demographic characteristics and previous training experience were obtained via a baseline survey. Caregiver demographic factors included sex, age, level of education, and monthly household income per capita. Caregiver’s age was divided into three approximately equally sized groups (less than 31 years, 31 to 34 years, and 35 years and above). Level of education was classified into three categories: junior high school and below, high school (including technical secondary school), and junior college and above. Monthly household income per capita was divided into two levels: below and above 3500 yuan (approximately US $540), according to the average income of Changsha residents in 2015 [24].

Data on engagement with the intervention were collected using the app’s backend system across full implementation of the trial; that is, user engagement information was collected in an automated fashion when users logged into the app. Available data included the webpages on the app that were visited, and on what occasions, how frequently, and for how long. This kind of information offers insight into engagement by participants based on how much time they spend with the app, and with what app features.

For this study, we collected app engagement data using four indicators: monthly login frequency, single login duration, knowledge segments learned per login, and single learning duration. Monthly login frequency was defined as the average number of times each participant used the app per month before quitting the study. Quitting was defined as the date of the last login recorded in the app backend system. Single login duration was the average online duration between opening the app and exiting it in the same session. Knowledge segments learned per login was the average number of short written statements with pictures, cartoon episodes, video recommendations, and interactive games learned in each login. Finally, single learning duration was defined as the average time spent on reading and learning individual knowledge segments during each login. To facilitate interpretation of the results, we divided all participants into three equal groups for each indicator according to the percentiles of P_33.4 and P_66.7 (<P_33.4, P_33.4 to P_66.7, and >P_66.7).

Statistical Analysis

Attribution rates and 95% CIs were estimated based on binomial distributions. The χ² test was used to examine the difference in attrition rates between the intervention and control education groups. Differences in app intervention engagement measures between the two groups were evaluated using the Kruskal-Wallis H test. Generalized linear mixed models were used to test the associations between attrition and all independent variables. The intraclass correlation coefficient was calculated to quantify the clustering of study participants at the preschool level. Adjusted odds ratios (aORs) were calculated to quantify the size of associations after adjusting for other independent variables. All analyses were performed using SAS 9.2 (SAS Institute). Statistical significance was based on two-sided tests at the level of .05.

Results

Sample Characteristics

In total, 2920 caregivers from 20 eligible preschools in Changsha, China participated, with 1510 caregivers assigned to the intervention group and 1410 assigned to the control group (Table 1). Participants in the intervention group were slightly more likely to be male, to have a higher monthly household income per capita, and to have received injury prevention education in the past 3 months. There also were differences in engagement: participants in the intervention group were more likely to engage in the website based on all four metrics than those in the control group (P<.001).

Engagement Indicators Between the Intervention and Control Groups

The intervention group had significantly higher engagement than the control group based on all four engagement indicators. This was true both for participants who completed the study and for those who failed to complete the study (Table 2).

Attrition Rate

The 6-month attrition rate for the intervention group was 28.9% (437/1510; 95% CI 26.6%-31.2%), whereas that for the control group was 35.7% (503/1410; 95% CI: 33.2%-38.2%). The attrition was significantly higher in the control group than in the intervention group (OR 1.36, 95% CI 1.17-1.59) (Multimedia Appendix 1).

Influencing Factors

Given the distinct attrition rates across the two groups, we performed multivariable analysis separately for each group. As shown in Table 3, the attrition rates of caregivers in the intervention group with “sometimes” monthly login frequency was higher than that of those with “often” monthly login frequency (P=.03). The attrition rates of caregivers with short and average single login durations were significantly higher than those for caregivers with long single login durations (P<.001 and P=.03, respectively). The attrition rates of caregivers with few and average number of knowledge segments learned per login were significantly higher than that for caregivers completing many knowledge segments learned per login (P<.001 and P=.003, respectively). Caregivers with short and average single learning durations had higher attrition rates than those with a long single learning duration (P<.001 and P=.002, respectively). The multivariable analysis showed that caregivers who learned fewer knowledge segments per login had a substantially higher attrition risk than those who learned more knowledge segments per login after adjusting for sex, age group, level of education, monthly household income per capita, receiving injury prevention education in the past 3 months or not, and the three other intervention engagement measures (Table 3).

As shown in Table 4, the attrition rate of caregivers in the control group with “seldom” monthly login frequency was higher than that for those with “often” monthly login frequency (P=.003). The attrition rates of caregivers with short and average single login durations were higher than that of caregivers with long single login durations (P<.001 and P=.009, respectively). The attrition rates of caregivers with few and average knowledge segments learned per login were much higher than that for caregivers with many knowledge segments learned per login (P<.001 and P=.001, respectively). The attrition rates of caregivers with short and average single learning durations were higher than that for caregivers with long single learning durations (P<.001 and P=.002, respectively). The multivariable analysis for attrition yielded three significant predictors of attrition. Caregivers with a lower monthly login frequency, who learned fewer knowledge segments per login, and who had shorter single learning durations had significantly higher attrition rates than those using the app more often per month (seldom vs often), learning more knowledge segments per login (average vs many), and having a longer single learning duration (short vs long) (Table 4).

Discussion

Principal Findings

The 6-month attrition rate of study participants was 28.9% in the intervention group and was 35.7% in the control group. As hypothesized, higher attrition risks were associated with low app intervention engagement in both the intervention and control groups after controlling for demographic variables and previous training experience on unintentional injury prevention. Compared to the intervention group, the control group had a higher attrition rate, which was primarily explained by lower app intervention engagement. Demographic factors seemed to not be associated with attrition in either group.

Interpretation of Findings

Attrition in our study was quite similar to that reported at 6-month follow up for an educational app on car seat use [25] (34.3%, 387/1129), but was significantly lower than the rate of 51% (254/498) reported in an app-based study of scald burn prevention [22]. A wide range of factors likely explain participant attrition from engagement in an app-based intervention, including design of the app, assessment measures for participant compliance, individual characteristics of participants, and cultural differences across countries.

Previous studies [13,22,26] generally report that female, younger, and less educated individuals have a higher likelihood of attrition. We could not confirm these findings in either the univariable or the multivariable analyses. One possible explanation for this result is the fact that we conducted our study in China, a middle-income country with a collectivist culture that differs from other countries and regions where the previous studies were conducted, such as the United States and Australia. Another possible explanation is related to our multivariable findings, which indicate that participant engagement was a strong predictor of participant attrition after controlling for participant demographics. In both groups, engagement in the intervention was significantly associated with attrition. These results support the hypothesis that a high proportion of participants lost to follow up in app-based interventions may simply be a result of nonusage of the app [17]. If participants perceive the app as helpful and beneficial, they continue to use it and then engage in research surveys.

We found that the attrition rate in the intervention group was lower than that in the control group, a finding we attribute to the less engaging aspects of the parenting education app for the control group. Specifically, the control group received new material twice a week and thematic discussions once a month, while the intervention group received new material four times a week and engaged in thematic discussions twice a month. The intervention group also had access to experts to ask questions and receive consultation, a feature not provided to the control group. These data concord with the findings of Kelders et al [27] who reviewed 101 web-based intervention studies and found that increased interaction with a counselor, more frequent intended usage, more frequent updates, and more extensive employment of dialogue support better adherence.

Implications

Our findings have two major implications. First, they emphasize the unignorable impact of high dropout attrition in mHealth intervention studies. High attrition must be evaluated strictly before drawing conclusions about the efficacy of an intervention, and research findings should be interpreted cautiously when attrition rates are high, vary substantially across comparison groups, or when missing values (dropouts) occur in a nonrandom fashion. Second, the results underscore the need for careful design and implementation of mHealth interventions. The comparatively high attrition rate among participants with less frequent and engaging exposure to educational materials highlights the need for mHealth interventions to be designed and implemented with appealing features. They must attract users and increase engagement in the intervention.

Study Limitations

This study was primarily limited by the RCT design. We do not have data concerning engagement or attrition rate after 6 months since the study ended at that point. Further, other potentially relevant factors for dropout were not available for analysis, including participant preferences, attitudes, and views about the app-based interventions, all of which may influence attrition [28]. These limitations should be considered in future research.

Conclusions

Dropout attrition was high in an RCT examining app-based interventions for unintentional child injury and parenting education versus only parenting education among caregivers of Chinese preschoolers. Engagement in the interventions differed across the two groups, and attrition was significantly associated with intervention engagement in both groups. Greater engagement with the app led to lower attrition. Attrition must be considered by researchers, policymakers, and practitioners when evaluating and implementing mHealth interventions. Efforts to engage users are critical to reduce attrition. Future research might consider feasible strategies to improve and maintain compliance of study participants to mHealth intervention programs.