Development and Evaluation of a Smartphone-Based Chatbot Coach to Facilitate a Balanced Lifestyle in Individuals With Headaches (BalanceUP App): Randomized Controlled Trial

Introduction

Background

Primary headaches, including tension-type headaches (TTHs) and migraine, are among the most prevalent neurological illnesses [1]. TTH and migraine are ranked as the third and sixth most common diseases, respectively, worldwide in both women and men [2]. The physical, social, and mental burden of headaches, defined as the summation of all negative consequences [3], is substantial. In addition, it is important to consider the quality of life impacted by headaches, defined as the subjective assessment of general well-being, position, and prospects in life. Individuals who have recurrent headaches are often afraid of the next headache attack, which can lead to avoidance behaviors, such as the cancelation of social activities or not even planning them [4].

The economic costs of headaches are substantial, primarily manifesting as indirect and intangible costs. This includes work absences because of headaches (absenteeism), reduced on-the-job performance while experiencing headaches (presenteeism), reduced quality of life, and increased pain outside the workplace [5-7]. In the European Union, the total annual costs associated with headaches among adults are estimated at €173 billion (US $189 billion; 64% migraine, 12% TTH, and 24% other types of headaches) [8]. A more recent study conducted in Canada estimated the total annual cost associated with migraine to be CAD $23,756.04 (US $17,750.50) per patient [9]. Given the high personal and financial costs, there is an urgent need for effective management of headaches.

Headaches are multifactorial, and besides physiological factors, lifestyle factors, such as stress or sleep, play a significant role in the development and retention of a headache [10-12]. Insufficient perception of stress reactions, individual attitudes toward stress (eg, high-performance orientation and anxiety), or coping strategies, such as avoidance versus endurance, are seen as dysfunctional stress-coping in relation to headaches [13,14]. The belief that pain-related factors are outside one’s control and the perceived inability to control these factors (ie, low self-efficacy) are further dysfunctional coping mechanisms associated with poor adjustment to headache and psychological functioning [15,16]. Among people who have headache, stress, stress regulation, and mental tension are perceived as critical triggers of a headache [14,17-19], and headache itself serves as a stressor that negatively affects well-being [20]. However, there are controversial findings regarding the association between lifestyle and headache, and individuals differ in the extent to which these factors interact with headache [21].

Guidelines recommend pharmacological and nonpharmacological interventions as standard therapy [22], including behavioral treatment, which has been shown to be effective in both face-to-face and web-based settings [22-25]. These treatments incorporate psychoeducation, relaxation techniques, physical activity, triggers management, and cognitive behavioral therapy elements, focusing on stress management and coping strategies to modify negative and dysfunctional cognitions, emotions, and behavior related to headaches [26,27]. A person-centered approach that integrates various intervention components is more effective [28,29] and may enhance personal control and efficacy in headache management [14]. However, challenges such as cost, access, motivation [18], and stigma hinder engagement [30]. Despite this, consistent care using nonmedical options has proven beneficial [31].

In addition to traditional evidence-based treatment modalities, mindfulness-based interventions have gained research interest, showing enhanced well-being in various settings [32-34], particularly for coping with chronic conditions, such as chronic pain [35-37]. Studies have demonstrated that mindfulness interventions benefit individuals with headaches by improving psychological functioning [38-40]. Furthermore, they help improve affective conditions, such as anxiety and depression, which are often related to poorer treatment outcomes [41,42]. Improvements in these conditions, even if subclinical, may improve coping skills for headaches and increase treatment adherence. In addition, individuals without a psychiatric diagnosis may experience headache-attributed disability, defined as physical, cognitive, and mental incapacities imposed by headaches [3], such as disabling anxiety related to fear of headaches and perceived triggers [43].

Over the past few years, the adoption of app-based interventions for headache management has increased [44,45]. These interventions primarily take the form of electronic headache diary apps, which offer practical solutions for data monitoring [46]. Furthermore, app-based interventions have improved our understanding of the relationship between lifestyle factors and headaches [21]. Apps are also suitable for providing guideline-compliant therapeutic options [23], such as psychoeducation, relaxation techniques, endurance sports, and other elements of behavioral therapy (eg, stress reduction). However, evidence of the effectiveness of app-based behavioral interventions for managing headaches remains weak [44,47].

Conversational agents (CAs), also known as chatbots or digital assistants, are increasingly being applied in both clinical [48-50] and nonclinical [51-53] health care settings to support disease management and behavioral lifestyle interventions. CAs engage users in humanlike conversations [48,54-56], enabling factual, relational, and emotional communication. This interactive style enhances engagement by establishing a working alliance between the users and the CAs [57,58], reflecting the collaborative relationship between shared treatment goals and tasks [59], which is crucial for treatment success in psychotherapy and counseling [60]. In contrast, conventional mobile health (mHealth) interventions may lack a therapist relationship and suffer from noncommittal timing [61], may not be used as intended [62], and engagement may often be low [63-65]. Nevertheless, CA-based coaching offers the potential to deliver personalized, accessible, and scalable content via web-based or mobile-based apps [56,66].

Objectives

To our knowledge, there is no evidence of mHealth coaching interventions delivered by CAs for the treatment of headaches. Building on a successful pilot study [67], we designed BalanceUP, a smartphone-based and CA-delivered intervention aimed at supporting a healthy lifestyle in people with headache. BalanceUP aims to improve mental well-being by promoting behavior change techniques (BCTs) in behaviors, emotions, thoughts, and beliefs related to headaches while ensuring low-threshold access and scalability. Using smartphone apps’ technical potential, scalable interventions can be beneficial in supporting individuals [68]. Consequently, this study had the following objectives: (1) to develop a smartphone-based and CA-delivered intervention for people with headache and (2) to evaluate its effectiveness, engagement, and acceptance.

Methods

App Development

BalanceUP, developed for iOS (Apple Inc) and Android (Google LLC) platforms using MobileCoach [54,69,70], provides a chat-based interface for communicating with the CA (Figure 1B). The communication between users and the server is encrypted. The chat feature offers predefined answer options and free-text input, guiding conversations along dynamic paths to individualized tasks. The CA also shares videos and pictures elaborating on the psychoeducational content. Using the sidebar, users can access the (1) chat channel; (2) audio library (eg, relaxation, mindfulness, and imagination exercises); (3) illustrations; (4) working materials (eg, energy balance and coping circle); (5) video library (eg, animated psychoeducational videos); and (6) frequently asked questions about the study and BalanceUP app (Figure 1D).

‎

Coaching Intervention

Overview

Drawing on the best practice from behavior therapy, BalanceUP is based on the cognitive behavior change migraine therapy manual (Cognitive-Behavioral Therapy for Migraine Management [Kognitiv-verhaltenstherapeutisches Migränemanagement]; MIMA) [13], which has been demonstrated to be feasible [71] and effective, showing results similar to those of the active control group at a 12-month follow-up [72]. BalanceUP comprises 7 consecutive modules: (1) headaches, (2) relaxation, (3) balance, (4) fear, (5) coping, (6) trigger, and (7) stress (refer to Multimedia Appendix 1 for an overview of the coaching), with a procedure similar to MIMA: (1) feedback on tasks, (2) psychoeducation, (3) reflection of behavior, (4) behavioral intention, (5) action planning, and (6) relaxation and imagination (refer to Figure 2 for further details). Each module contains 3 to 4 units (Figure 3), and users can work through these units during sessions with the CA according to their preferences, allowing completion within 24 to 60 days. Examples of the session flow can be found in Figure 4. After completion, users can still access the materials. A diagram showing the intervention flow is presented in Multimedia Appendix 2.

‎

‎

‎

Behavior Change

To foster health-promoting behaviors, various BCTs [73] have been considered (eg, gamification for reward, action planning, and prompts to perform an exercise). A complete list of BCTs and their specific applications is outlined in Multimedia Appendix 3 [73]. BCTs are intervention ingredients designed to alter or redirect causal processes that regulate behavior [74]. Their specific implementation enables accurate replication, precise specification of the intervention content, and investigation of possible mechanisms of action [74].

Engagement

Tailoring (at a subgroup level) and personalization (at an individual level) are essential for promoting trust, engagement, adherence, and effectiveness in mHealth interventions [63,75]. Both tailoring (eg, psychoeducation material based on headache types) and personalization (eg, coach selection [Figure 1A] personal greeting, personalized goals, and individual appointments with the CA) were applied in BalanceUP. Reminders (Figure 1C) are an effective way of improving engagement, especially when they address specific needs [76,77]; refer to Multimedia Appendix 4 [49,56,68,78-81] for the detailed aspects of the engagements implemented.

Intended Use

The intended use of BalanceUP, which is the extent to which an individual needs to experience the content to derive maximum benefit from the intervention [81], is determined by reaching the outro and completing the postsurvey. Users can skip a module or elements based on their diagnosis (migraine vs TTH) and preferences. For instance, participants can skip psychoeducation within a unit and proceed to behavior reflection, whereas relaxation exercises, videos, and worksheets are optional. This procedure offers flexibility in line with the recommended mHealth intervention strategies [82,83] and is similar to the dose-response rate in pharmaceutical research, where medication dosage can vary depending on the patient’s condition and characteristics. Furthermore, we base this approach on the self-determination theory [84], which emphasizes autonomy, competence, and engagement.

Study Design and Procedure

We conducted an unblinded, 2-arm, randomized controlled trial (RCT). After onboarding, eligible participants were asked to provide electronic informed consent. Participants who provided consent proceeded to complete the baseline survey (T1), while those who were ineligible were directed to a farewell conversation with the CA. Those meeting the inclusion criteria and consenting were randomly assigned with a 1:1 allocation ratio to either the intervention or waitlist control group using random numbers (0-1) generated by the BalanceUP app, with numbers below 0.5 assigned to the intervention group. The intervention group immediately began with the coaching. The postintervention survey (T2) was conducted 24 to 60 days after randomization. The waitlist control group received weekly reminders from the CA during the 42-day waiting period. After this, they completed the postintervention survey (T2) and were given the option to access coaching or proceed to the outro (Figure 5). Throughout the study, self-reported primary and secondary outcomes were collected within the BalanceUP app and via the in-app survey tool, LimeSurvey (version 3.4). Refer to Multimedia Appendix 5 for outcome details and data collection times.

‎

Study Participants

Participants were self-recruited in Switzerland, Germany, and Austria between April and November 2022 (German-speaking parts) via the study website. The link to the website was shared via social media by headache organizations, insurance companies, and health care institutions. This recruitment strategy allowed participation from all German-speaking countries. The study website provided study information and app download links. Inclusion criteria were being an adult (aged ≥18 y) with a smartphone (eg, iOS [Apple Inc] or Android [Google LLC]), fluent German-speaking skills, and experiencing regular headaches for at least 3 months with a minimum of 4 incidents per month. These criteria were assessed using the BalanceUP app during onboarding.

Outcomes

Mental Well-Being

To measure effectiveness, we defined mental well-being as the primary outcome, measured by the Patient Health Questionnaire Anxiety and Depression Scale (PHQ-ADS) [85]. The PHQ-ADS is a composite of the Patient Health Questionnaire-9 (PHQ-9) [86] and General Anxiety Disorder Scale-7 (GAD-7) [87]. Scores can range from 0 to 48, with higher scores indicating more severe depression and anxiety; 3 to 4 points were considered the minimum clinically important difference. Cutoff scores of 10, 20, and 30 denoted mild, moderate, and severe degrees of depression and anxiety, respectively.

To assess psychological functioning, multiple secondary outcome measures were used in accordance with the established guidelines [88].

Secondary Outcomes

Depression

The PHQ-9 [86] consists of 9 items for evaluating depressive symptoms, rated on a 4-point Likert scale ranging from 0 (not at all) to 3 (nearly every day). Higher scores indicate higher symptom severity, with scores ranging from 0 to 4 indicating no symptoms of depression and scores from 5 to 9, 10 to 14, 15 to 19, and 20 to 27 indicating mild, moderate, moderately severe, and severe depression, respectively.

Anxiety

The GAD-7 [87] is used for evaluating symptoms of generalized anxiety disorder. It comprises 7 items. Similar to PHQ-9, answers are rated on a 4-item Likert scale ranging from 0 (not at all) to 3 (nearly every day). Higher scores indicate higher levels of anxiety, and the total score ranges from 0 to 21. Scores from 0 to 4, 5 to 9, 10 to 14, and 15 to 21 denoted minimal, mild, moderate, and severe anxiety, respectively.

Somatic Symptoms

We measured somatic symptoms using the Patient Health Questionnaire-15 (PHQ-15) [89]. It is a 15-item self-report questionnaire that can be scored on a scale from 0 (not impaired) to 2 (severely impaired). A total score of ≥15 on the PHQ-15 indicates a high level of impairment owing to somatic symptoms [90]. For this study, we adopted 2 items from the PHQ-9 because the items were similar. However, the answer scale of the 2 items differed between the PHQ-9 and PHQ-15 and had to be converted according to the manual.

Stress

We measured stress with the German version of the Perceived Stress Scale-10 [91]. The 10-item questionnaire can be rated on a scale from 0 (never) to 4 (very often); higher scores reflect a higher level of perceived stress, and scores can range from 0 to 40.

Self-Efficacy

To assess headache-related self-efficacy, we used the German short form of the Headache Management Self-Efficacy (HMSE-G-SF) Scale [92]. The measurement consists of 6 items assessing self-efficacy beliefs related to headaches. Answer scales range from 1 (do not agree) to 7 (agree). Higher scores implied higher self-efficacy expectations, and summed scores <19 indicated below-average self-efficacy expectations compared with other people experiencing headache.

Intention to Change Behavior

To assess participants’ intention to change behavior, we used the health action process approach model [93], which categorizes behavior change into 3 stages: nonintenders, intenders, and actors. In this study, participants indicated their use of psychological techniques for headache treatment by choosing 1 of the 5 possible answers: (1) no, and I do not intend to do so (nonintender); (2) no, but I am considering it (nonintender); (3) no, but I have the intention to do so (intender); (4) yes, but it is not easy (actor); and (5) yes, and it is easy (actor).

Absenteeism and Presenteeism

To measure work-related impairment due to headaches, we applied 4 out of 5 questions from the Migraine Disability Assessment [94]. These questions assessed days with complete loss and days with at least 50% reduced productivity (eg, work, household, and school) for the past 3 months. Given the study’s runtime and potential recall challenge [94], participants reported headache days for a 1-month period instead of 3 months.

Pain Processing

We applied the Questionnaire for the Assessment of Pain Processing (questionnaire to assess pain management; Fragebogen zur Erfassung der Schmerzverarbeitung) to measure pain coping strategies [95]. This tool assesses coping strategies in individuals with persistent pain and is comprised of 2 parts. In this study, we used the first part to evaluate cognitive and behavioral coping using 24 items. The cognitive coping subscale included the dimensions “action planning skills,” “cognitive restructuring,” and “experience of competence.” The behavioral coping subscale included “mental distraction,” “counteracting activities,” and “rest and relaxation techniques.” Answers were scored from 1 (not at all true) to 6 (always true), with higher scores indicating better pain processing.

Sociodemographics

We collected data on age, sex, level of education, parallel app use for headache tracking, concurrent psychotherapy, commitment to the program, and headache diagnosis at baseline to describe the study population. We further assessed participants’ sensitivity to triggers and tendency to avoid triggers optionally in module 6 (Trigger) using the German short version of the Headache Triggers Sensitivity and Avoidance Questionnaire [96]. Mean scores <2.03 indicate below-average trigger sensitivity and scores >3.19 indicate above-average trigger sensitivity, respectively. For avoidance, mean scores <2.09 indicate below-average trigger avoidance and scores >3.23 indicate above-average trigger avoidance, respectively.

Engagement

Overview

According to a systematic review conceptualizing engagement with digital behavior change interventions [64], engagement is both a multidimensional concept and a dynamic process. Engagement consists of 2 parts: (1) the extent of use (eg, amount, frequency, duration, and depth) and (2) a subjective experience characterized by attention, interest, and affect. By using this multidimensional approach, we aimed to capture the various aspects of engagement, as defined by Perski et al [64].

Extent of Use

The following use data were recorded during the coaching intervention: total minutes spent on in-app relaxation and imagination exercises, total reminders sent to participants in cases of inactivity, and the average number of days taken to complete 1 coaching module. In addition, the percentage of answered conversational turns between the participant and the CA coach was calculated, where a higher number indicated higher engagement with the intervention. We also assessed intended use, that is, the number of participants who completed the outro.

Subjective Experience

To gather the subjective experiences of the participants, we used 4 items from the German Group Therapy Session Evaluation by Patients [97] with statements about personal involvement, active participation, perceived comprehensibility, and perceived benefit on a 5-point Likert scale ranging from 1 (disagree) to 5 (agree). Furthermore, we measured perceived enjoyment by applying a single-item measure from technology acceptance research [98,99] (“Did you enjoy the last unit?”), ranging from 1 (not at all) to 5 (very much).

We assessed participants’ commitment to changing their behavior with 1 question (“How committed are you towards changing your behavior?”) on a scale of 1 to 10.

We used a modified version of the Session Alliance Inventory—Patient Version [100,101] to repeatedly measure the working alliance between the participant and the CA. It consists of 3 items for the Bond Scale and 3 items for the Task and Goal Scale. In this study, we used the items of the validated German version of the Working Alliance Inventory-Short Revised [102], which features a 5-point answer scale ranging from 1 (seldom) to 5 (always). In addition, we contextualized the Session Alliance inventory by replacing the term “therapist” with the CA’s name.

Acceptance

We assessed the acceptance of BalanceUP with a slightly modified and translated version of the Mobile App Rating Scale (uMARS) [103] assessing engagement (eg, entertainment, interest, customization, interactivity, and target group of the app); information (eg, quality of information, quantity of information, visual information, and credibility of source); perceived quality (eg, recommendation, use, payment, and overall rating); and perceived impact (eg, awareness, knowledge, attitudes, behavior change, seeking help, and intention to change). All subscales used a 5-point Likert scale ranging from 1 to 5, with higher scores indicating a more favorable judgment.

In addition to the uMARS, the questions “What did you like most about the BalanceUP app?” and “What would you like to see improved about the BalanceUP app?” could be answered in free text.

Impression of Change and Adverse Events

We assessed adverse events with the Patients Global Impression of Change Scale [104]. It is a 7-point scale depicting the perceived overall improvement in general health, rated from 1 (much improved) to 7 (very much worse). In this study, for scores ≥5, we also assessed whether participants believed these changes occurred because of coaching or whether other circumstances (eg, professional situation and conflicts in the social sphere) caused these changes. Participants were asked to note the adverse changes if coaching was given as a reason.

Sample Size Calculation

We estimated the sample size based on the primary outcome (mental well-being) measured by the PHQ-ADS for a linear mixed model (LMM) and a repeated measure ANOVA (within-between interaction). Consistent with previous headache and chronic pain research [105-107], we assumed a small-to-medium effect size for the primary outcome. Statistical power calculation using G*Power³ software revealed that a sample size of 90 (45 for each group) would be sufficient, with a power of 0.80 to detect a small-to-medium time×group interaction effect size (Cohen d=0.35) with an α of .05 and based on 2 measurements. According to our pilot study, we estimated a dropout rate of 40% and aimed to recruit approximately 150 participants.

Data Analysis

The analysis was performed using SPSS (version 28.0; IBM Corp) and R software (version 4.2.2; R Foundation for Statistical Computing) including the lme4. For the primary outcome (PHQ-ADS) from before the intervention (T1) to after the intervention (T2), the LMMs were used, considering time (T1 and T2), group (intervene and wait), and their interaction as fixed effects, with participants as a random factor. Secondary outcomes (ie, PHQ-9, GAD-7, PHQ-15, Perceived Stress Scale-10, Questionnaire for the Assessment of Pain Processing, HMSE-G-SF, Migraine Disability Assessment, and health action process approach) were analyzed accordingly. Missing data were managed using LMM, which is based on all observed data and accounts for data missing at random [108-110]. According to the CONSORT (Consolidated Standards of Reporting Trials) guidelines, we reported the LMM analysis for the (1) intention-to-treat (ITT) analysis, in which all randomized participants were included, regardless of whether they used the coaching and (2) per-protocol (PP) analysis of complete cases. Calculations of within- and between-group effect sizes (Cohen d) were based on the pooled SD of complete cases only and labeled as small (Cohen d=0.2), medium (Cohen d=0.5), and large (Cohen d=0.8). The influence of predictors on outcomes was explored using the LMM, including the primary outcome (PHQ-ADS), with a focus on the 3-way interaction. Change in engagement over time was analyzed via repeated-measures ANOVA, and the effect of early engagement [64] on treatment outcomes was analyzed using linear regression. Descriptive statistics were used to summarize participant characteristics at baseline, and 2-tailed t tests were used to assess baseline differences. We applied qualitative content analysis [111,112] to answer the open-ended questions.

Ethical Considerations

The Swiss Ethics Committee Zurich reviewed the research project and confirmed (Swiss Ethics BASEC-Nr. Req-2021-01365) that it does not fall within the scope of the Human Research Act. This research project was registered in the World Health Organization–accredited German Clinical Trials Register (DRKS00017422). We performed this trial based on the CONSORT-EHEALTH guidelines.

Results

Participant Flow and Baseline Characteristics

During the recruitment phase, from April to November 2020, 405 individuals downloaded the BalanceUP app. Of these, 223 (55.1%) were assessed for eligibility and 7 (1.7%) were excluded from the study. Of those eligible, 198 (48.9%) individuals completed the baseline survey and were randomized into the intervention (n=110) and control (n=88) groups. The full participant flow is presented in Figure 6. The dropout rate for randomized participants after the treatment was 29.1% (32/110) for the intervention group and 18% (16/88) for the waitlist control group and thus can be considered low, particularly for a fully unguided mHealth intervention [113] and in comparison with our pilot study.

‎

As presented in Table 1, most of the participants were women (180/198, 90.9%) with a mean age of 38.7 (SD 12.14) years, and more than half had a university degree (104/198, 52.5%). Migraine was the most prevalent diagnosis, accounting for 72.2% (143/198) of the sample. Approximately half of the participants (95/198, 48%) reported using headache diaries to track their symptoms. Few participants (32/198, 16.2%) reported attending psychotherapy and using the coaching app. In general, participants reported an average of 6.66 (SD 7.34) days of work per month missed because of headaches and 11.82 (SD 9.77) days per month when their performance was reduced by half or more (including work, school, and household). Compared with other individuals who had headaches [92], the study participants reported average levels of headache-related self-efficacy. On average, participants were classified as “intenders,” indicating that they had the intention to change their behavior (as opposed to “nonintenders” or “actors”). On average, the participants had mild depression (mean 9.06, SD 4.24), mild anxiety (mean 6.76, SD 3.87), and moderate psychosomatic symptoms (mean 10.92, SD 4.34). There was no difference between the groups in any of the outcomes.

Effectiveness

Primary Outcome

Table 2 presents the results of the LMM analyses for the ITT and PP analyses. For both the ITT and PP analyses, we found evidence of a treatment effect (group by time interaction) for the PHQ-ADS after the intervention (t₃₄₂=–3.6; P<.001, 95% CI –5.06 to –1.47 and t₂₉₄=–3.58; P<.001, 95% CI –5.11 to –1.49, respectively). BalanceUP significantly affected mental well-being, as shown by the change in the PHQ-ADS. However, the waitlist control group did not improve with time (Cohen d=–0.07, 95% CI –0.23 to 0.08), the intervention group improved from before the intervention to after the intervention with a medium effect (Cohen d=0.62, 95% CI –0.84 to –10.39).

Changes in the PHQ-ADS score differed significantly between groups with a medium effect (Cohen d=0.66, 95% CI –0.99 to –0.33); refer to Table 3 for observed means and effect sizes (Cohen d) for participants who completed the coaching intervention as intended (PP).

Secondary Outcomes

Regarding secondary outcomes, the ITT LMM analyses demonstrated evidence of treatment effects for depression (t₃₄₂=–3.56; P<.001, 95% CI –2.79 to –0.80), somatic symptoms (t₃₄₈=–4.48; P<.001, 95% CI –3.35 to –1.31), stress (t₃₄₁=–3.07; P=.003, 95% CI –4.25 to –0.94), headache-related self-efficacy (t₃₄₂=4.08; P<.001, 95% CI 2.10-5.99), application of BCTs (t₃₄₂=4.82; P<.001, 95% CI 0.45-1.07), presenteeism and absenteeism (t₃₁₇=–3.00; P=.003, 95% CI –7.96 to –1.68), cognitive pain coping (t₃₄₁=4.96; P<.001, 95% CI 3.38-7.79), behavioral coping (t₃₄₁=5.18; P<.001, 95% CI 3.11-6.89), and suggestive evidence for anxiety (t₃₄₂=–2.73; P=.007, 95% CI –2.50 to –0.40). The PP analyses showed similar results (Table 2). The effect sizes of secondary outcomes between groups after the intervention were medium (eg, depression, anxiety, somatic symptoms, stress, absenteeism and presenteeism, and pain coping) and large (eg, headache-related self-efficacy and application of BCTs). Refer to Table 3 for further details.

Predictors

We also explored whether diagnostic status (participants diagnosed with migraine vs participants with other or no headache-related diagnosis), concurrent psychotherapy, concurrent tracking of headaches, and headache-related self-efficacy influenced the pre- and postintervention effects. We did not find evidence of a 3-way interaction among group, time, and predictors. There was no evidence of a difference in the decrease of the PHQ-ADS score between the intervention and waitlist control groups for participants with a diagnosis of migraine (t₃₃₈=–0.81; P=.42, 95% CI –5.93 to 2.48), concurrent psychotherapy (t₃₃₈=–0.45; P=.65, 95% CI –5.69 to 3.57), concurrent headache tracking (t₃₃₈=–1.92; P=.06, 95% CI –6.99 to 0.09), and self-efficacy at baseline (t₃₃₈=–1.50; P=.14, 95% CI –0.48 to 0.06).

Engagement

Table 4 shows the rate of the intended use of the BalanceUP coaching app among participants who started module 1 (n=182), that is, participants from the intervention group (n=110) and participants who started coaching after the waiting time (n=72). As anticipated, the highest dropout rates occurred during module 1 (34/182, 18.7%), with a subsequent decrease in dropout rates during the subsequent modules. Of the 182 participants who began the coaching program with module 1, 118 (64.8%) completed the coaching and thus used it as intended. A visual inspection of the engagement data related to subjective experience revealed that participants who discontinued using the app did not show differences in active participation, internal engagement, perceived benefit, or comprehensibility compared with those who continued using BalanceUP (refer to Multimedia Appendix 6).

We measured engagement among the participants with intended use based on the extent of use (use data) and subjective experience (self-reported data). The use data analysis showed that 86.3% (101/117) of the participants completed all 7 modules, taking an average of 6.9 (SD 1.5) days to work through a module. Participants replied in an average of 77.8% (SD 3.73%) conversational turns. In the event of inactivity, across the entire study period, participants were sent an average of 11.70 (SD 5.99) push notifications for no activity for a 1-hour period and 8.95 (SD 5.87) notifications for no activity for a 2-hour period, with a subsequent decrease in reminders sent for 1, 3, 5, and 10 days of no activity. Participants spent an average of 113.44 (SD 182.51) minutes on relaxation exercises. Refer to Table 4 for further details.

Participants self-reported a mean commitment to change their behavior of 8.15 (SD 5.46). The reported commitment to change behavior significantly increased with time (F_2,192=8.17; P<.001) with a medium effect (Cohen f=0.29), and evidence of higher commitment toward the end of the coaching (mean 8.17, SD 1.49) than in the middle (mean 7.57, SD 1.55, mean difference 0.60, 95% CI 0.31-0.89; P<.001) or beginning of the intervention (mean 7.61, SD 1.57, mean difference 0.56, 95% CI 0.21-0.91; P=.002). A linear regression (β=.05, SE 0.42; P=.67) with pre- to posttreatment PHQ-ADS changes showed that early reported commitment did not predict before improvement to after improvement of mental well-being. The mean perceived alliance with the CA was 3.94 (SD 0.82). The alliance significantly increased with time (F_2,200=10.66; P<.001) with a medium-large effect (Cohen f=0.32), with the evidence of alliance becoming higher toward the end of the coaching (mean 4.07, SD 0.83; P<.001) than at the beginning (mean 3.83, SD 0.83, mean difference 0.23, 95% CI 0.12-0.34; P<.001); refer to Table 5. A linear regression (β=.11, SE 0.80; P=.36) with pre- to posttreatment PHQ-ADS change showed that early reported alliance with the CA did not predict pre- to posttreatment improvement of mental well-being.

Participants with intended use rated the quality of BalanceUP in terms of engagement (eg, personalization and interactivity), information quality (eg, source and credibility), perceived subjective quality (eg, recommend app and pay for app), and perceived impact (eg, knowledge and awareness). The highest-rated subscale was information quality, with a mean of 4.47 (SD 0.47), followed by perceived impact (mean 4.00, SD 0.62), engagement (mean 3.72, SD 0.65), and subjective quality (mean 3.56, SD 0.77).

Impression of Change and Adverse Events

We found evidence of an improved perceived global impression of change in the Patient Global Impression of Change Scale from module 1 to module 7 (t₁₀₁=8.45; P<.001; Cohen d=–0.62, 95% CI –0.91 to –0.33), indicating an improvement in perceived general health. Of the participants who reported worsening of their general condition in module 1 (6/142, 4.2%), only 1 participant reported that this was because of the current coaching intervention and provided the following explanation:

Dealing with migraine triggers constant self-monitoring, which may cause slight discomfort and prevents me from feeling completely free from them. However, in the long term, it is worth pursuing education and behavior change.

The remaining 5 participants reported that the worsening was because of other reasons, such as job and social conflicts. In module 7, only 2% (2/102) of the participants reported worsening of their general condition; however, all cited other reasons as the cause.

Triggers

Participants rated their sensitivity to trigger with a mean of 2.51 (SD 1.09) and avoidance with a mean of 2.32 (SD 1.07), indicating average sensitivity and avoidance compared with a sample of people who have headache [96].

Qualitative Evaluation

We further assessed the BalanceUP app's positive (Figure 7) and negative (Figure 8) aspects using qualitative content analysis. We used a deductive approach to identify themes based on insights from our pilot study and the existing literature. When examining the positive aspects, participants expressed a strong appreciation for the extensive and comprehensive information provided, which included various types of exercises and the delivery mode of the intervention. When evaluating negative aspects, participants expressed a desire for more flexibility in their interactions with the CA.

‎

‎

Discussion

Principal Findings

This study aimed to describe the development and evaluation of the effectiveness of the BalanceUP app. This is the first RCT of a fully unguided coaching intervention delivered by a rule-based CA to facilitate mental well-being in individuals with headaches. We described the BalanceUP app's evidence-based design and systematic evaluation.

With regard to effectiveness, we found evidence of improved mental well-being, as measured by the PHQ-ADS, in individuals with frequent headaches who received BalanceUP, with a medium to large between-group effect size (Cohen d=–0.66). Participants who interacted with BalanceUP experienced a clinically important improvement, reporting, on average, a 3.9 (SD 5.59) point reduction [85] of perceived depression and anxiety symptoms after the intervention. In contrast, participants in the waitlist control group did not show substantial changes in their mental well-being (0.6-point reduction on the PHQ-ADS score). Moreover, we found evidence of reduced anxiety, somatic symptoms, perceived stress, and absenteeism and presenteeism, as well as improved HMSE-G-SF, application of BCTs, and pain coping skills, with effects ranging from medium (Cohen d=0.43) to large (Cohen d=1.05). Diagnostic status, concurrent psychotherapy, concurrent tracking of headaches, and headache-related self-efficacy did not influence the effects of chatbot coaching.

No notable adverse effects were observed owing to the use of BalanceUP. Among the participants who initiated the coaching by starting the first module, a substantial portion (118/182, 64.8%) successfully adhered to the program. Participants who used BalanceUP as intended established a pronounced working alliance with the CA, which significantly improved with time.

In terms of acceptance, the program’s information content received the highest rating, followed by the perceived impact, engagement, and subjective quality of BalanceUP. Its overall average rating, on a scale of 1 to 5 stars, was 3.91 (SD 0.67), indicating a high level of acceptance. Participants expressed their willingness to recommend the app to individuals who might benefit, with a mean rating of 4.02 (SD 1.05), reflected by the following statement in the uMARS questionnaire: “There are many people I would recommend this app to.”

Comparison With Prior Work

Evidence on the effectiveness of mobile interventions for headaches is limited, despite their wide availability [47]. Minen et al [44] found that, despite a trend toward mHealth, most studies using electronic behavioral interventions to treat headaches did not use mobile devices. Only 1 single-arm study explored mHealth migraine behavioral therapy but with potential bias owing to missing diary entries. A recent study by Grazzi et al [114] demonstrated the feasibility and effectiveness of an mHealth mindfulness program for chronic migraineurs (headaches for ≥15 d in a month), resulting in a 50% retrospective reduction in migraine days and medication intake.

With regard to internet-based headache treatments, 1 pilot study by Day et al [115] showed improvement in self-efficacy (Cohen d=0.82) and pain acceptance (Cohen d=0.82), although there was no evidence of headache reduction. An RCT by Bromberg et al [105] assessed a web-based intervention for migraine self-management and coping, revealing improvements in multiple outcomes, including depression, stress, headache self-efficacy, pain catastrophizing, and coping strategies, although without changes in anxiety and disability. Another study on web-based behavioral training for migraine self-management found no difference in headache attack frequency but found evidence for improved migraine self-efficacy and locus of control [116].

This study’s results align with face-to-face behavior change interventions for headaches. Cognitive behavioral therapy, relaxation, or mindfulness interventions have effectively improved the cognitive, behavioral, anxiety, and stress-related aspects associated with headaches [30,117]. Notably, the evaluation of MIMA [72], the basis of our coaching, found limited improvement in headache-related outcomes, except for headache self-efficacy. Furthermore, BalanceUP was extended to address TTH (eg, onset of headache, course of headache, and medication). Although migraine was predominant among participants, we found no evidence that diagnostic status affected treatment outcomes. Tailoring allowed BalanceUP to effectively address both types of headaches.

There was no evidence of the influence of concurrent psychotherapy on coaching effectiveness in this study, which is noteworthy given prior suggestions of potential benefits of combining face-to-face and digital interventions [118,119]. The hypothesized role of headache-related self-efficacy in treatment outcome [120] remained unconfirmed in this study.

The BalanceUP app’s evidence-based design and systematic evaluation contribute to the growing body of evidence on the acceptance and effectiveness of interventions involving CAs in health care. A recent meta-analysis examined 32 RCTs focusing on mental health and CA use [121]. Short-term effects on outcomes such as depressive and generalized anxiety symptoms, quality of life, well-being, and stress were found, with effect sizes ranging from Hedges g=0.24 to Hedges g=0.62. However, the long-term effects remained unclear. Personalization and empathetic responses emerged as crucial effectiveness facilitators, with longer CA interactions linked to larger effect sizes. These findings align with those of another meta-analysis [122], showing the effectiveness of CAs in outcomes related to lifestyle changes, smoking cessation, substance misuse, and medication adherence. Notably, <50% of the participants reported overall satisfaction with the CA, content likeability, and future use. However, many studies had a pre- and postintervention design or were feasibility trials, indicating the need for further RCTs in this field. Abd-Alrazaq et al [123] evaluated the effectiveness of CAs in 12 clinical and nonclinical RCTs. Weak evidence of reduced depression, stress, or agoraphobia rates, but not improved mental well-being, was found. However, bias, low-quality evidence, small sample size, or contradictory results limited the conclusions, necessitating further high-quality RCTs following the guidelines.

Contextual factors, such as psychological traits, motivation, personal relevance, and attributes of digital behavior change interventions themselves (eg, content, reminders, delivery, support, and personalization) influencing engagement [64]. BalanceUP integrates personalization and tailoring (eg, relevant topic selection and adjustment of interaction length) to empower participants and promote a sense of control and ownership over the coaching process. Emotional support, encouragement, or validation from the CA also expresses empathy, reinforcing participants’ feelings (eg, “This is excellent news, well done, keep on track” and “I am sorry to hear, but setbacks are also part of a change process”). Qualitative feedback confirms program flexibility and suggests that addressing individual preferences can enhance satisfaction and further improve the coaching experience.

In BalanceUP, we observed that 65% of the participants used coaching as intended by completing the outro. In behavioral headache treatment, engagement has yet to be thoroughly assessed in terms of dose and duration; however, earlier studies reported high dropout rates [124].

Consistent with previous research [67,125], participants using BalanceUP established a strong alliance with the CA, which improved significantly with time. This finding aligns with studies conducted in in-person, digital, and group settings [126-128] and with individuals with recurrent headaches [129]. However, contradictory to findings in internet-based therapy studies [130,131], the alliance was unrelated to improvements in participants’ mental well-being. These results suggest that, although the alliance between participants and the CA was established and strengthened throughout the coaching program, other factors may significantly influence treatment effectiveness.

Limitations and Future Work

This study had several limitations. First, guidelines for trials of behavioral headache treatments [88] recommend using headache frequency as the primary outcome. However, they also urge investigators to use standardized disability, functional status, or quality of life measures. A recent Delphi study by Leudtke et al [132] emphasized the need for outcome measures that truly reflect patients’ experiences. Therefore, the inclusion of functional disability, quality of life, and associated symptoms should be considered in nonpharmacological interventions. BalanceUP aimed to capture the biopsychosocial impact of headaches by addressing various lifestyle factors.

Second, the self-selection of the participants limits the generalizability of our findings and introduces potential self-selection bias. Participants’ particular interest in the subject matter might make them nonrepresentative of the broader population. Caution is needed when extending these findings to a broader context because of the possibility of differing preexisting characteristics.

Third, it is important to acknowledge potential improvements for enhanced interactions in BalanceUP. The current rule-based nature of the CA allowed for the implementation of an evidence-based program. However, participants’ desire for more flexible interactions with the ability to input personal responses was evident. Previous research [133] indicates user preference for a combination of predefined answer options and text input to enhance perceived interactivity. To enhance text processing in BalanceUP, integrating artificial intelligence (AI)–based technology, such as large language models or natural language processing, could be considered. Natural language processing and large language models enable the CA to interpret user inputs more dynamically, yet existing AI-based chatbots struggle with unforeseen user responses [133,134]. Furthermore, ethical aspects related to AI technology should be considered, as they could lead to misjudgments and potential risks. Research has highlighted a lack of transparency in describing the handling of input data and algorithms, affecting the reliability and validity of findings [122,135]. One approach might involve using AI technology for specific tasks (eg, providing content that humans will select or tailor) while maintaining a rule-based approach for other tasks (eg, handling sensitive information and following guidelines), ensuring predictability and preventing harm.

Fourth, it is worth noting that most participants in this study were women (86%), in line with the higher prevalence of headache in women [136]. Approximately half of them had university degrees, consistent with our pilot study [67]. However, the trial’s sociographic distribution may not fully represent the general population. Future studies should seek greater diversity using a more representative sample.

Fifth, a potential limitation is the digital placebo effect [137]. In this unblinded trial, participants might attribute improvements to using an mHealth intervention rather than interventionist ingredients. Expectations and engagement could bias outcomes positively. Future research should carefully design control conditions, considering active control groups or sham interventions [138].

Sixth, participants needing a smartphone introduced a limitation [139]. Although mHealth usability is favored by a substantial proportion of middle-aged individuals with headache in German-speaking countries, the exclusion of smartphone-less individuals may impact generalizability [4]. Ownership rates are high, for example, 92% in Switzerland and 91% in Europe [140].

Seventh, a small number of participants (28/198, 14.1%) in this study encountered technical issues, such as missing audio tones in videos or loss of internet connection. Although technical support was provided to address these problems, it is likely that these technical issues may have had a negative impact on participants’ engagement.

Finally, the primary objective of developing this app was to enhance the mental well-being outcomes of individuals with headache. The app was specifically designed for this research project. Currently, it is freely available for public use. However, the future availability of the app is uncertain because of ongoing support costs and a lack of collaboration with potential providers. We recognize the potential value of making such interventions accessible in the future, and discussions regarding their availability are ongoing. It is essential to note that the university’s role is not to provide the app as a service but to contribute through accompanying research. The sustainability of digital health interventions, particularly with limited public health funding, remains challenging [141,142]. Clear cost-benefit outcomes and accountability strategies should be addressed in future research.

Conclusions

This study represents the first empirical evaluation of an evidence-based and CA-delivered coaching intervention (BalanceUP) designed to promote a healthy lifestyle among individuals with headaches. The findings provide evidence of the positive impact of BalanceUP on participants’ mental well-being, as indicated by improvements in depression and anxiety symptoms. The establishment of a strong alliance between participants and the CA, along with their high commitment to the program, further reinforces the potential effectiveness of this intervention. The scalability and accessibility of automated coaching through a CA highlight its potential as an engaging and effective tool for behavior change. Further research is needed to explore the long-term effects, intensity, and duration of delivery of coaching interventions for lifestyle modifications in the health sector. Therefore, the applicability of mHealth interventions in diverse populations should be investigated. Understanding these aspects will contribute to the development of effective and inclusive interventions that promote positive health outcomes.