An mHealth Intervention (ReZone) to Help Young People Self-Manage Overwhelming Feelings: Cluster-Randomized Controlled Trial

Introduction

Mental health difficulties are increasing among young people. Emotional disorders are the most prevalent and show the most significant increase over time and with age [1,2]. Recent figures show that 1 in 8 (12.8%) people aged 5-19 years had at least one mental disorder when assessed [3]. Mobile health (or mHealth) interventions have the potential to provide mental health support at scale, and there is a need for evidence as to the effectiveness of digital interventions for supporting mental health in young people [4]. However, non-adoption and implementation issues present substantial barriers to effectively doing so [5]. Additionally, questions about whether randomized controlled trials are always the most appropriate method of evaluation need to be considered in the context of practicality and feasibility factors [6-8].

The authors have co-designed an mHealth intervention, ReZone, with young people, parents/carers, and teachers, based on cognitive behavioral therapy (CBT), mindfulness, and attention bias modification training (ABMT). This short paper will report the findings from a cluster randomized controlled trial (RCT) to test the hypothesis that young people in classes receiving ReZone have fewer behavioral difficulties than young people in classes receiving management as usual. Full details on the methods and development are reported in the protocol [9] and implementation paper [10].

Methods

Participants and Procedure

A total of 49 classes were recruited (27 mainstream, 22 alternative provision), resulting in a sample of 409 students (mean age 10.9, SD 1.44 years; 56% male; Table 1). Schools were recruited via newsletters and events published in our Schools in Mind network. Participants were enrolled from 12 mainstream primary schools (307 students) and 11 secondary alternative provision schools (103 students). A total of 66 people were lost to follow-up, resulting in an attrition rate of 16%. All students (aged 10-15 years) in schools taking part in the project were eligible to participate. Classes were randomized using a true random number generator in blocks of two by an independent trials unit, to management as usual or ReZone stratified by school type (alternative provision vs. mainstream) after classes were recruited and baseline measures assessed. Ethical approval was received from University College London (UCL) Research Ethics Committee (number: 7969/001). The present research is reported in line with CONSORT guidelines [11].

Intervention: ReZone App

Full details of the ReZone intervention are described in the protocol [9]. The intervention primarily aims to reduce behavioral difficulties. It is available as a web-based smartphone and tablet application flexibly designed to be used in groups or individually, with or without adult support depending on the user’s capability and confidence. This flexibility also extends to the order and frequency in which elements of the tool are used. However, it is required to have an initial guided session or tutorial with an adult. This flexible approach to implementation aimed to fit within a variety of schools and their differing student characteristics, priorities, and timetables.

The primary features of ReZone (Textbox 1) are presented below.

Measures

All measures were self-reported by young people and administered in school by researchers or school staff in paper form. Baseline demographic data were collected and are reported in Table 1. Emotional (α=.78) and behavioral difficulties (α=.81) were measured using the 16-item Me and My School (M&MS) questionnaire [12]. Mental wellbeing was measured using the 7-item Short Warwick-Edinburgh Mental Wellbeing Scale (α=.83) [13]. Empowerment was measured using the 6-item “self” subscale of the Youth Empowerment Scale-Mental Health (α=.73) [14]. Health-related quality of life was measured using the 6-item (with global health visual analog scale) EQ-5D-Y (α=.63) [15].

Intervention Usage

Intervention usage data were collected in ReZone and exported to the data analysis software. There were 3467 usage sessions, which corresponds to 16.9 times per each of the 205 students in classes allocated to ReZone. The game (n=752), breathing exercises (n=811), and happy faces (n=1169) were used the most; stress bucket (n=404), art (n=225), and time out (n=106) were used the least. Based on teacher consultations, which are discussed elsewhere [10], usage methods and frequency varied across schools and students due to a variety of factors, including student characteristics, teacher priorities, and school environment.

Analytic Strategy

Means and standard deviations, for all measures, across conditions at baseline and follow-up, were calculated. Intention-to-treat and per-protocol analyses with four multilevel regressions were tested for each of the outcome variables with time nested within students within classrooms. Per-protocol analyses included only those that had completed follow-up. Model 0 was computed as the null model to examine the change in outcome over time, without predictors. The intraclass correlation coefficients for each measure were 17% (emotional difficulties), 24% (behavioral difficulties), 7% (empowerment), 9% (wellbeing), and 3% (quality of life). We then entered school type (ie, alternative provision vs mainstream) as a fixed effect predictor to Model 1. In Model 2, patient-level grand mean-centered age, ethnicity, gender, and disability were entered as predictors. Demographics were controlled for in the analyses as there were significant differences in gender between groups at baseline within mainstream primary schools and age between groups within alternative provision schools (Table 1). In the Final Model (Model 3), condition (ReZone vs management as usual) was entered as a predictor. The likelihood ratio test was used to compare the fit of Models 1, 2, and 3. For example, Model 1 was significant for behavioral difficulties and therefore compared to Model 2, which was significant. Models 1 and 2 were therefore retained in the final model, with Model 3 (final model) being compared to Model 2 using the likelihood ratio test. Models 1 and 2 were significant for empowerment and wellbeing and therefore retained in the final model. For emotional difficulties and quality of life, only Model 2 was significant and retained in the final Model.

Results

The means and standard deviations of all study measures at baseline and postintervention are presented in Table 2. The likelihood ratio test for the intention to treat analysis was not significant for any of the final models compared to the previous significant model. There were no significant differences between the ReZone condition and management as usual in emotional difficulties (χ²₁=0, P>.05), behavioural difficulties (χ²₁=0.01, P>.05), empowerment (χ²₁=0.01, P>.05), wellbeing (χ²₁=0.05, P>.05), or quality of life (χ²₁=0.3154, P>.05). The per-protocol results showed similar patterns.

Discussion

This cluster randomized controlled trial examined the effectiveness of ReZone in reducing behavioral and mental health difficulties in young people. The study did not show a significant difference between those using ReZone and management as usual in the reduction of behavioral and emotional difficulties and improving empowerment, wellbeing, and quality of life.

Potential limitations of the study, implementation, and adoption barriers to digital interventions within schools have been discussed in depth [10]. Implementation barriers to new digital interventions in schools include the technology itself, the adopter system, and the organization [5]. The process of embedding ReZone into the school system can sometimes leave minimal time to utilize it successfully. Implementation strategies differed, with varying levels of teacher encouragement and consistency of scheduled ReZone use. Competing teacher priorities, opinions, and seniority of the school advocate could have impacted the time and drive to use the tool [10].

ReZone was used a total of 3467 times throughout the 3-month study period, and current usage data show that ReZone is still being used 10 months poststudy (501 sessions over the last 3 months). Aggregated objective usage data is useful to indicate overall acceptability and use. However, teacher consultations (fully reported in [10]) suggested students either used ReZone frequently or not at all. Analysis of a dose-response effect could be improved by gathering further quantifiable individual school usage information related to implementation.

The usage statistics and teacher consultations indicated happy faces and breathing were the most used activities, while the stress bucket, art, and time out were used the least. Happy faces and breathing can be completed in only 1-2 minutes; they are a fun challenge/game type of activity, and most of the time can be completed without teacher assistance. The stress bucket and time out require more time to work through thought processes, requiring the user to be willing to think and talk about their feelings, needing teacher support (which is not always feasible).

The challenges discussed link to the emerging concept of gamification within digital mental health tools [16,17]. There are some basic elements of gamification within the most popular elements of ReZone. Further work on gamification combined with other crucial processes such as co-design, could help address the broader issue of nonadoption and sustainability, with many users reporting that they stop the use of a health app after only two weeks [18]. This lack of consistency could be particularly applicable when working with young people in such settings as alternative provision schools. Pupil behavior often places higher demands on staff in alternative provision schools than in mainstream schools, and disengagement and loss of concentration are common issues.

Measures used were related to overall mental health, wellbeing, and behavior leading us to question how often a tool such as ReZone would need to be used to effect changes in broad domains and specifically in schools. If tools such as ReZone do not provide large-scale, substantial longer-term changes, we must also ask whether there is any impact on immediate feelings, behavior, and interactions. We did attempt to measure this by asking teachers to complete a few short questions related to how well students refocused and re-engaged after using ReZone. However, due to practical issues such as teacher demand, these were not completed. On reflection, this could have also been something students were asked to complete. It is also possible that effects were delayed, and further long-term data collection, such as one-year post-trial, may have shown differences [19-21].

Due to the complex nature of digital interventions, traditional RCTs may be insufficient to assess efficacy [8]. As this study shows, there are multiple objective and subjective data sources available, including quantitative self-report data, objective usage data, and subjective qualitative data, while digital and school studies present particular challenges in adherence, sustainability, and roll-out [10]. As with psychotherapy research in general and a reliance on self-report measures, participant blinding was not possible, although true and independent randomization was used. Additionally, due to this being a pragmatic trial, there may have been other interventions the school and participants may have been using.

It is essential to publish negative results to avoid publication bias. These studies can still make a substantial contribution to the literature, learning, and informing the direction of future work [22]. The negative results of this study could indicate that ReZone has a limited scope to impact change, that its utility is hampered by issues with implementation and nonadoption, and limitations of mHealth research in a school context. The results may also suggest the limited applicability of the RCT study design for evaluating tools such as ReZone.