Engagement With a Behavior Change App for Alcohol Reduction: Data Visualization for Longitudinal Observational Study

Introduction

Background

Maintaining alcohol consumption within recommended guidance is widely known to reduce one’s risk of illness or injuries. Such guidance includes the recommendations of the Chief Medical Officer of the United Kingdom to limit alcohol consumption to 14 units a week and to have frequent alcohol-free days [1]. However, anyone in the general adult population who wants to reduce their hazardous or harmful alcohol consumption may face certain challenges to follow such guidance [2]. Challenges include the ease of access to alcohol and alcohol being an addictive substance. This can lead to individuals developing chronic or cyclical patterns of excessive drinking, with the personal behaviors of drinking influenced by internal or external factors [3-5]. Internal factors refer to feeling states or events in an individual’s recent drinking history, such as previous drinking episodes, moods, motives, or cravings that may modify future patterns of drinking [6]. External factors are influential events that occur independently of an individual’s drinking history; for example, how the risk of hazardous drinking of the general population increases during holiday periods or weekends [7,8].

Behavior change apps, sensors, and wearables offer a way of reducing hazardous alcohol consumption through real-time data capture and interventions [9-12]. Benefits of behavior change apps, that can be synchronized with sensors and wearables, include capturing an individual’s dynamic history of alcohol consumption and state of mind while providing around the clock access to support, particularly in moments when an individual’s vulnerability to hazardous drinking may increase [13].

However, a key challenge for the majority of behavior change apps is that levels of engagement remain low [14-16]. Engagement, often a mediator of effectiveness [14], is considered a multifaceted construct composed of behavioral and experiential aspects [17]. Usage data from a behavior change app provides an understanding of behavioral engagement (hereafter referred to as engagement) with the app [18]. Multiple indicators of engagement are thought to convey important information about how users interact with a given intervention, including the frequency (eg, number of log-ins), depth (eg, proportion of available modules accessed), amount (eg, time spent per log-in), and duration (eg, total number of days) of use [19].

Drink Less is a behavior change app that aims to help its users reduce hazardous and harmful alcohol consumption. The app was developed following the multiphase optimization strategy framework (comprising a preparation phase, an optimization phase, and an evaluation phase) [20-23] and the UK Medical Research Council’s guidance on developing complex interventions [24-26]. The app includes 6 different theory and evidence-informed modules: normative feedback, goal setting, cognitive bias training, self-monitoring and feedback, action planning, and identity change. These modules are described in detail by Garnett et al [27]. The app sends a local daily push notification at 11 AM that asks users to “Please complete your drink diaries,” to encourage self-monitoring of drinking behavior. The default 11 AM timepoint was set so as not to disturb late risers and to allow participants time to complete their morning routine; however, the notification timing could be changed by the user.

Owing to the agile nature of app development, optimization of engagement can be done through cycles of research and implementation [28]. Identifying important patterns of engagement for such optimization purposes presents various analytical challenges that visualizations can address. Visualizations have previously been helpful for analyzing a wide variety of rich data streams within public health research [29-33]. Simple visualizations, especially when complemented with clear textual descriptions, are generally recommended for identifying and comparing trends [32]. In previous digital health research, visualizations have delivered at a glance insight from mass volume and time-varying data, including more sophisticated displays of spatiotemporal, contextual, and event-centric outcomes [34-38]. Importantly, visualizations can provide insights into optimization that include (1) patterns of use that may boost or hinder behavior change, (2) a better understanding of temporal engagement with various components of the intervention, and (3) pathways toward personalization of the intervention.

Objectives

The aim of this paper is to explore the usefulness of simple visualizations in uncovering important temporal patterns of engagement and facilitating decision making for further intervention development. This study presents 2 key contributions to improving engagement with Drink Less. The first contribution, provided in the Results section, is to showcase a number of visualizations that helped us understand temporal patterns of engagement with Drink Less. The second contribution, provided in the Discussion section, explains how insights obtained from these visualizations informed the next stages of intervention optimization.

Methods

Data Transformation

Each visualization involved transformation of the data. Original usage data involved merging, by an anonymous user ID, a data set of baseline characteristics (age, sex, employment type, and Alcohol Use Disorders Identification Test [AUDIT] score) to a data set of time stamps of start time of use, screen views, and length (in microseconds) of use. Along with use, the actions of entering an alcohol-free day or recording units of alcohol consumed were measured.

Data

Data set 1 included 19,233 users who downloaded Drink Less between May 17, 2017, and January 22, 2019 (615 days). The inclusion criteria for users included having a baseline AUDIT score of 8 or above, which is indicative of excessive drinking [39]; being from the United Kingdom; being aged 18 years or above; being interested in reducing their alcohol consumption; using app versions 1.0.11 to 1.0.16; and having consented to the Privacy Notice (Multimedia Appendix 1). Screen views data are recorded automatically and downloaded via Panda scripts from Nodechef (a web-based platform for hosting mobile apps) using a secure https protocol. Sessions were derived from screen views using the Pandas script.

Users who downloaded the app on August 21, 2018 (n=5830), were excluded as an article on BBC News was published on this date, which endorsed the app (Garnett et al, unpublished data, 2020); thus, these users were likely to have different characteristics and engagement behavior.

Data set 2 included time stamps of 829,001 sessions and 8,169,005 screen views of the 19,233 users in data set 1. This includes 122,332 entries of alcohol-free days and 123,704 entries of alcohol drinks consumed. All use was recorded from May 17, 2017, to April 16, 2019 (699 days). As such, users had a minimum of 84 days of use measured.

To explore various engagement aspects, we developed sets of data from data sets 1 and 2 with varying engagement measures.

Set A

All use was measured from May 17, 2017, to January 22, 2019 (615 days), including date of download and time stamps of all use. This period was chosen as it reflects a time in which the content of the app was relatively stable.

Set B

Set B included all users whose use was measured in Set A, with data only over the first 30 days from download, with the measure “Did use occur on this day?” (binary, yes or no) for each user.

Measures

Log-in Sessions and Frequency of Log-Ins

A session was defined as a continuous series of screen views, with a new session defined as a new screen view after 30 min of inactivity [40]. Clearing or swiping away the daily notification did not register as use and was not considered as either a session or a module view. All time stamps were appropriately adjusted from Coordinated Universal Time to British Summer Time. The amount of use per log-in session was operationalized as time spent (in seconds) per session. Daily use was captured by the measure “Did use occur on this day?” (binary, yes or no) for each user for 30 days (Set B).

Drinking Diary Entry

In the self-monitoring and feedback module, users enter an alcohol-free day and the date of its occurrence, the number of alcoholic units consumed, and the date of consumption. The time stamps in which records were made was measured.

Disengagement

We defined disengagement as the first day of 7 or more consecutive days of no use after download [41]. The days between download and disengagement were derived for each user. Users who did not disengage after downloading the app were censored.

Data Visualization Methods and Analytical Techniques

We used heat maps, timeline plots, k-modes clustering, generalized estimation equations, and Kaplan-Meier plots to explore and visualize patterns of engagement with Drink Less. Analyses were carried out in R [42] and Stata [43]. We used the following R library packages to create the visualizations: ggplot2 for heat maps and timeline plots [44], rayshader to create the 3D animations [45], viridis for color palettes sensitive to readers with color blindness [46], Klar to perform the k-modes clustering [47], survminer for the Kaplan-Meier survival curves and number at risk table [48], gganimate to create animated plots of use over time [49], and patchwork to place graphs side by side [50]. The data visualization methods, data set and engagement measures are shown in Textbox 1.

K-modes clustering is an extension of the k-means algorithm for partitioning categorical data, which uses a general dissimilarity measure [51,52]. Within each cluster, we visualized the probability of opening the app during the day over time with 95% CI. The appropriate number of clusters was explored through the elbow method and silhouette method [53]. The elbow method explains the variance of the data in relation to the number of clusters and shows by how much the addition of another cluster would reduce the dissimilarity measure. The silhouette method shows how well each user fits into their respective cluster through 2 distance measures: separation (ie, the average distance to the closest other cluster) and compactness (ie, the average within-cluster distance) [54,55]. Kaplan-Meier plots show the estimated cumulative proportion of users engaged and the time scale is days after download [56,57].

To explore the association between the delivery of the notification and subsequent near-term engagement of opening the app (ie, engagement in the hour after the notification is delivered), we compared opening the app (yes or no) between the exposed time period (11 AM to noon) and an unexposed time period (10 AM to 11 AM). We estimated the association between exposure to the notification and opening of the app, which was quantified using a risk ratio. We fitted a marginal model for the outcome of opening the app by using a generalized estimating equation [58] with robust standard errors and an independent working correlation matrix. We fitted an unadjusted model and a model adjusted for the baseline covariates of the continuous variables age, days after download and baseline AUDIT score, which were all included as linear terms, and the categorical variables employment type and gender. Further models explored effect moderation by adding an interaction between exposure to the notification and (1) days after download and (2) cluster (as identified by the k-modes analysis). In the final model, we additionally allowed the association between cluster and exposure to the notification to vary linearly by day after download. Estimated risk ratios with 95% CIs and Wald test P values are presented. For models with interaction terms, we present risk ratios for exposure to the notification estimated at days 1, 7, and 30 after download, estimated separately for each cluster.

Results

Overview

The user characteristics are reported in Table 1. Approximately half (49.5%) of the sample were male. The mean age of users was 44 (SD 11.2) years, and the majority worked in nonmanual employment (71.7%). Just under half (46.6%) had a baseline AUDIT score indicating hazardous alcohol consumption (8 to 15, inclusive).

Summative tables of use (screen views and time on app) by module are provided in Multimedia Appendix 2. It was observed that 85% of screen views occurred in the module Self-Monitoring and Feedback. The number of users who reported at least one alcohol-free day or at least one alcohol drink record was 61.86% (11,898/19,233) and 49.11% (9445/19,233), respectively. Over the first 30 days of use after download (derived for Set B data), the median number of sessions per user was 9, with an IQR of 2 to 28 sessions, and the median time spent per user was 24 min, with an IQR of 9 to 55 min.

Visualizations

Patterns of Frequency of Use, Length of Use, Entries of Alcohol-Free Days, and Alcohol Units Consumed

In Figure 1, both heat maps show days of the week along the x-axis and hour of the day along the y-axis. Plot A in Figure 1 shows the frequency of opening the app by hour of the day and day of the week. This shows that there is a strong association between delivery of the notification and opening of the app in the following hour, and this is consistent throughout the week. Plot B in Figure 1 shows the amount of use by hour of the day and day of the week. This shows that the notification is also associated with the distribution of the total time spent on the app. In plot B, hotspots are observed across the evenings and on Saturday, Sunday, and Monday mornings, which are not evident in plot A. A heat map of when Drink Less was downloaded (Multimedia Appendix 3) shows hotspots of downloads on Sunday and Monday evenings. Rotating 3D heat map films of Figure 1, which show the variations more clearly, are provided in Multimedia Appendices 4 and 5.

In Figure 2, plot C shows the median time spent on the app along the y-axis and plot D shows the total number of sessions starting in the hour along the y-axis. Timeline plots show the hour of the day on the x-axis. Plot D shows that the frequency of sessions sharply peaks in the hour after the notification is sent at 11 AM. A second natural peak of frequency occurred in the evenings and a third smaller peak in the mornings. Plot C shows that the median length of time drastically dropped from 11 AM onward, with a slow and steady recovery as the day progressed. An animation of plot D over time is provided in Multimedia Appendix 6, showing that the shape of the distribution over 30 days remains consistent.

‎View this figure

‎View this figure

In Figure 3, plot E shows the frequency distribution of entering an alcohol-free day and plot F shows the frequency of entering a drink record. Timeline plots show the hour of the day on the x-axis. There are more alcohol-free days entered between 11 AM to 12 PM than drink records made, which suggests that the notification is more strongly associated with entering alcohol-free days than entering alcohol units consumed. Both outcomes see similar prominent, natural peaks in the evenings, with an additional smaller peak in the mornings.

‎View this figure

Visualization of Engagement Clusters

A total of 3 clusters emerged from the k-modes clustering. This was based on the measure did the user open the app? (binary, yes or no) for the first 30 days after download.

Figure 4 plots the probability of use of the app, stratified by cluster, over time (number of days after download). The 3 ribbons represent the probability of use of the app for each engagement cluster, with 95% CI. On the basis of the observed pattern of engagement, we named the 3 clusters as fast disengagers (10,903/19,233, 56.68%), slow disengagers (3679/19,233, 19.12%), and engagers (4651/19,233, 24.18%). The optimal number of clusters was determined by the elbow method and silhouette method (Multimedia Appendix 7). The silhouette method suggested that the optimal number of clusters was 2, whereas the elbow method suggested 3 clusters. Comparing the results under 2 and 3 clusters showed that the slow disengagers and engagers groups identified under 3 clusters were essentially a subdivision of 1 cluster in the 2-cluster model. We chose to retain 3 clusters based on observed differences in the trajectory of engagement over time between the 2 groups—the engagers and slow disengagers.

The probability of using the app 30 days after download for engagers was 0.69 (95% CI 0.67-0.70), slow disengagers was 0.10 (95% CI 0.10-0.11), and fast disengagers was 0.01 (95% CI 0.01-0.02).

‎View this figure

Table 2 shows the distribution of user characteristics across the engagement clusters. The median number of sessions was lowest for the fast disengagers and much higher for the engagers. Engagers are, on average, more likely to be older, male, working in nonmanual employment, and more likely to report a lower AUDIT risk zone, compared with users within the fast disengagers and slow disengagers clusters.

Table 3 provides the estimated associations between exposure to the notification and app use, based on Set B data. Over the first 30 days after day of download, the probability of using the app in the hour after the delivery of the notification was approximately 4 times higher than the probability of using the app in the hour before. All models of the estimated associations between exposure to the notification and app use are adjusted for the continuous variables of age, days after download, baseline AUDIT score, and the categorical variables of employment type and sex. The cluster-specific effects included an effect moderation of the exposure to the notification by cluster group, and the days after download effects included an effect moderation of the exposure by days after download. The adjusted estimated risk ratio was 4.21 (95% CI 4.07-4.36), and the estimated risk ratio was higher among engagers (Wald test P value: fast disengagers vs engagers P=.001 slow disengagers vs engagers P<.001, slow disengagers vs fast disengagers P=.44).

Table 4 shows the estimated association between exposure to the notification and opening of the app in the 3 clusters at 3 different time points (days 1, 7, and 30).

Visualization of Time to Disengagement

Kaplan-Meier plots, both overall and stratified by clusters, were plotted to show days to disengagement, defined as 7 or more consecutive days of no use, for the first 365 days after downloading Drink Less.

In Figures 5 and 6, the x-axis depicts the number of days after download, ranging from 0 to 365, and the y-axis depicts the survival probability, which is the proportion of users who have not disengaged. The dashed lines at the 0.5 survival probability mark shows the time (days) up to when 50%of each cluster has disengaged. Each hash in the plot represents a right-censored user. The number at risk represents the users in the clusters who remain engaged over the year. In Figure 5, we see that 50.00% (9617/19,233) of users have disengaged at 22 days from download, and Figure 6 shows the divergence of longer-term engagement between clusters. The median number of days to disengagement for engagers was 132 days (95% CI 128-137), slow disengagers was 26 days (95% CI 24-29), and fast disengagers was 3 days (95% CI 2-3).

‎View this figure

‎View this figure

Discussion

Principal Findings

Visualizations provided important insights into how users engage with the behavior change app Drink Less. They revealed a strong association between delivery of the daily push notification (sent at 11 AM) and use in the next hour, suggesting that the push notification strongly influences how users engage with Drink Less in the immediate hour after the notification is sent. Push notifications (sometimes known as ecological momentary interventions) are programmed messages sent to a user by the app and are commonly employed within behavior change apps to both monitor and provide support to people at risk of harmful alcohol consumption [3,10,59]. Push notifications are a time-varying component of Drink Less that can be further optimized to become just-in-time adaptive interventions that rely on decision rules in the provision of real-time support and can learn and adapt to the contextual and psychological circumstances of individuals over time [59]. Previous research has found that notifications are important components that influence engagement with behavior change apps [60-62]. This includes an ecological momentary assessment study with Drink Less, which found that establishing a daily routine is important for maintaining engagement and that the daily push notification supports such routines [63]. This study also found that time-varying, endogenous factors of motivation and perceived usefulness of the app were the most consistent predictors of engagement.

Our analysis suggested an approximate adjusted four-fold increase in the probability of using the app in the immediate hour following the notification (11 AM to noon) compared with the preceding hour (10 AM to 11 AM). For the 1 in 5 users belonging to the slow disengagers group, two interesting findings emerged. Firstly, the association between the notification and opening Drink Less in the subsequent hour decreased over time, and secondly, patterns of engagement for this group show, on average, a high probability of use over the first week but low probability after the second week. A possible reason for this decline in probability of use and association of the notification and use could be habituation to the daily notification, or turning the notification off. Importantly, we hypothesize that optimizing the notification policy may generate higher rates of engagement for this group.

Future Research to Understand and Optimize the Notification Policy

To carefully create decision rules for the policy to evolve from an ecological momentary intervention to a just-in-time adaptive intervention, we will undertake a micro-randomized trial (MRT). The aim of the MRT is to further develop the push notification policy to improve engagement by targeting internal or external contextual circumstances that either influence excessive drinking (states of vulnerability) or events of engagement with the app (states of acceptability and opportunity) [64]. Visualization of engagement data helped inform the design of our MRT.

Table 5 summarizes how the visualizations from this exploratory research informed the design of our forthcoming MRT.

Primarily, this research guided our decision to shift the delivery time from 11 AM to 8 PM to exploit the potential increase in vulnerability to excess drinking, in an opportune and acceptable moment to engage with Drink Less [64]. To avoid the risk of an underpowered MRT, the expected effect size used in the sample size calculation of our MRT is based on a more conservative model, with the control defined as use between 9 AM and 11 AM and the treatment defined as use between 11 AM and 1 PM. This means that the MRT is powered to detect a marginal effect, quantified as a risk ratio, of sending a notification (compared with sending no notification) of 2.16 on user engagement rather than 4.22. We added 2 parallel arms to the MRT to provide an assessment of how engagement with Drink Less evolves over time when no notifications are provided and an exchangeable sample to compare the current policy of delivering a fixed notification daily, to a random notification policy, varying the content and sequence of notifications.

Limitations

This paper details exploratory research. Our estimates of the association between the notification and opening Drink Less do not represent a causal effect on engagement, as we are unable to account for systematic differences in use between the 2 periods that are unrelated to the notification. A randomized trial will allow for the causal effect of the notification to be understood. We also found that simple, accessible visualizations achieved our goal of understanding important patterns of engagement; however, when managing denser streams of data, more complex visualizations may be required.

An additional limitation is that disengagement is defined as a period of no use for 7 or more consecutive days and is considered as a one-time event instead of a repeated event; hence, the Kaplan-Meier plots are interpreted for the survival event disengagement for the first time. However, a proportion of users repeatedly disengage and then re-engage with Drink Less. It is not uncommon that even after disengaging a number of times, users re-engage for long, continuous spells of use with Drink Less. We aim to explore this in future research by visualizing the nature of repeated reengagement with accessible graphical applications and available shared toolsets [34]. An additional limitation is that we did not track whether users subsequently turned off or altered the delivery time of their notifications.

Conclusions

Identifying patterns of engagement from voluminous, temporally dense data presents various challenges for researchers and practitioners. The summarization of such data with heat maps, timeline plots, and Kaplan-Meier plots can provide a clear picture of daily, weekly, and long-term patterns of use over time with a behavior change app. Optimizing engagement is a priority for many behavior change apps, and these visualizations provide a way to identify the key features of how this version of a behavior change app is engaged with.

For Drink Less, we have demonstrated the important role of visualizations by showing how these clearly identified how behavioral engagement varies over the day of the week and hour of the day, along with when users first disengage. The visualizations revealed that the daily notification is likely to strongly influence engagement with Drink Less. Both the average probability of use over 30 days and the association between use and the notification remained high for users in the engagers cluster yet steadily declined over time for users in the slow disengagers cluster. This suggests that a fixed notification policy can be effective for maintaining engagement for some users but ineffective for others. It is now our priority to understand the causal effect of the notification on engagement and to consider further optimizing the push notification policy to contextual circumstances of individuals over time to inform the development of a just-in-time adaptive intervention. The MRT aims to inform the development of decision rules to tailor the notification policy to individuals over time, with details found in our protocol [65].