Trajectories of 12-Month Usage Patterns for Two Smoking Cessation Websites: Exploring How Users Engage Over Time

Introduction

Electronically delivered health interventions (or eHealth interventions), such as websites and mobile apps, have been successful methods of health behavior change [1-4]. In this body of research, people who engage more with eHealth interventions tend to have better treatment outcomes [5]. However, while eHealth intervention engagement is usually measured with simple counts of the number of log-ins and modules completed [5], little is known about how users engage with eHealth interventions over time and whether those temporal patterns predict better treatment outcomes. In the educational literature, a well-documented finding is that learning new material becomes more effective when it occurs over a longer period of time as opposed to over a short period of time [6]. This process, called spaced practice, works by way of increasing variability in learning and remembering new information [7].

Websites and mobile apps for health behavior change are usually available for participants to use at will, which results in high variations of individual usage patterns, or usage trajectories, over time. For example, some users may follow a trajectory of logging in several times within the first few days of starting an intervention and then never return. Others may follow a trajectory where they log in consistently and then gradually taper off. And other users may follow a trajectory where they consistently log in over the course of many months. It is possible that some groups of individuals follow unique usage trajectories over time that are associated with differential health outcomes. For example, people who log in consistently over the course of many months might have positive health outcomes because they have consistently benefited from the information and skills presented in the intervention. Alternatively, consistent log-ins may be a marker of ongoing challenges and struggles to change a health behavior, and thus may indicate poorer treatment outcomes. Since we do not know which trajectories of use predict successful behavior change, studying distinct groups of usage trajectories that people follow can help us identify which usage patterns are beneficial and make recommendations for future program use. This will help inform the design of eHealth interventions to improve successful behavior change.

Within the social and behavioral sciences, identifying usage trajectories has been applied for several decades to understanding behavior patterns over time [8-11]. More recently, a few studies have analyzed usage trajectories for eHealth interventions. One study examined 8-week usage trajectories of a diabetes management mobile app. The study found 3 distinct trajectories of usage and described the clusters of people following these trajectories as minimal users, intermittent waning users, and consistent users [12]. However, the study was limited by a small sample size (N=84), as well as short duration (8 weeks), and whether the trajectories predicted health outcomes was not reported. Other research identified 5 distinct usage trajectories of a short message service (SMS) text-messaging–based smoking cessation program over 5 weeks, namely high engagement, increasing engagement, rapid decrease, delayed decrease, and low engagement [13]. The study found that the high engagement and increasing engagement groups were more likely than the other groups to be abstinent over the course of 5 weeks.

If eHealth intervention usage trajectories that predict health outcomes can be identified, understanding the groups of individuals who tend to follow more or less successful trajectories is an important next step. This would reveal the qualities of individuals who are likely to have engagement patterns that are related to successful and unsuccessful outcomes. Knowing these baseline characteristics might allow researchers and intervention designers to tailor eHealth interventions to users’ unique challenges, needs, and limitations. While studies have found that being a woman, being older, and having a higher education are generally consistent predictors of greater eHealth intervention usage [14-17], very little is known about the user characteristics that are associated with different patterns of use over time. To our knowledge, only 1 study has examined this question [12] and found that being female and having higher baseline motivation were associated with more consistent log-in trajectories.

Using the example of smoking cessation websites, in this study we aimed to determine (1) distinct groups of log-in trajectories, (2) their prediction of the smoking cessation outcome, and (3) baseline user characteristics that are associated with different usage trajectory groups. The overall goal was to advance the study of analytic methods of user engagement and, ultimately, the design of more effective interventions that are tailored to users and their longitudinal patterns of engagement. To accomplish these aims, in this study we analyzed 365 consecutive days of log-in data from both arms of a large (N=2637), 2-arm randomized trial of website interventions for smoking cessation (NCT01812278).

Methods

Participants

As described in the main outcome article for the trial [18], we recruited participants (N=2637) from across the United States to participate in a study comparing 2 Web-delivered smoking cessation programs. Participants were recruited between March 24, 2014 and August 11, 2015. To be eligible for the study, participants had to be adult smokers in the United States (≥18 years of age), smoking at least 5 cigarettes daily, motivated to quit in the next 30 days, and have internet access. The 2637 participants were assigned to 1 of 2 Web-based smoking cessation interventions using stratified black randomization (on smoking frequency, education, and sex): WebQuit (n=1319; experimental arm) [18] or Smokefree (n=1318; control arm) [19].

Smoking Cessation Interventions

Participants accessed their assigned website with a unique username and password. For the first 4 weeks, all participants in both programs could opt to receive up to 4 short daily tips via SMS text messaging or email, which were designed to increase engagement. Participants were free to use their assigned program as they wished for 1 year from the date of enrollment.

The WebQuit program was based on acceptance and commitment therapy (ACT) [20], an approach that teaches skills to smokers to let their urges pass without smoking. The program had 4 parts. Step 1, Make a Plan, enabled users to develop a personalized quit plan, identify smoking triggers, learn about US Food and Drug Administration (FDA)-approved cessation medications, and upload a photo of their inspiration to quit (ACT processes: Values and Committed Action). Step 2, Be Aware, contained 3 exercises to illustrate the problems with trying to control thoughts, feelings, and physical sensations rather than allowing them to come and go (ACT process: Creative Hopelessness). Step 3, Be Willing, contained 8 exercises to help users practice allowing thoughts, feelings, and physical sensations that trigger smoking (ACT processes: Willingness, Being Present, and Cognitive Defusion). Step 4, Be Inspired, contained 15 exercises to help participants identify deeply held values inspiring them to quit smoking and to exercise self-compassion in response to smoking lapses (ACT processes: Values and Self-as-Context). The program also prompted users to track smoking, cessation medications, and practice of ACT skills. Tracking results were displayed graphically along with the user’s inspiration for quitting and badges earned for program use. Participants could log in and use the program as much as they liked.

For the control arm, we hosted a secured private version of the US National Cancer Institute’s Smokefree.gov site. This intervention was also named WebQuit so that participants would be blinded to group assignment. Smokefree follows the US clinical practice guidelines [21] and provides standard treatment that teaches skills to smokers to avoid urges. Users were able to navigate through all pages of the website at any time, and there were no restrictions on the order in which they could view the content. Smokefree had 3 main sections: Quit Today, Preparing to Quit, and Smoking Issues. The Quit Today section had 7 pages of content that provided tips for the quit day, staying smoke-free, and dealing with cravings. The section also provided information on withdrawal, benefits of quitting, and FDA-approved cessation medications. The Prepare to Quit section had 7 content pages providing information on various reasons to quit, what makes quitting difficult, how to make a quit plan, and using social support during a quit attempt. The Smoking Issues section provided 5 pages on health effects of smoking and quitting, depression, stress, secondhand smoke, and coping with the challenges of quitting smoking for the lesbian, gay, bisexual, and transgender community. The section also contained 5 quizzes that provided feedback about level of depression, stress, nicotine dependence, nicotine withdrawal, and secondhand smoke, as well as tips for coping with them.

Measures

Baseline Characteristics

At baseline, participants reported on demographics, alcohol use, smoking history, and whether they had a partner and friends who smoked. We measured nicotine dependence with all 6 items of the Fagerström Test for Nicotine Dependence (FTND) [22]. Participants also filled out the Commitment to Quitting Scale [23], which has 8 items measuring participants’ motivation to stay abstinent (example item, “I’m willing to put up with whatever discomfort I have to in order to quit smoking.”). The scale, which has been used in multiple smoking cessation trials [18,24], has been shown to have good reliability and validity [23]. We screened participants for mental health conditions including depression (Center for Epidemiologic Studies Depression scale) [25], generalized anxiety (Generalized Anxiety Disorder 7-item scale) [26], panic disorder (Autonomic Nervous System Questionnaire) [27], posttraumatic stress disorder (PTSD; PTSD Checklist) [28], and social anxiety (mini-Social Phobia Inventory) [29]. We included the results as covariates and predictors, since prior research has shown that mental health symptoms are a predictor of engagement in eHealth interventions [30,31].

Engagement

For each participant, we recorded time- and date-stamped log file records of each page opening. For this analysis, we used a binary measure indicating whether each participant logged in at least once each day (ie, had at least one page opening recorded in the log file data). Using this method, we obtained for each participant a 0/1 code for each day for 365 days from the date of randomization.

Cessation Outcome

The primary outcome of the study was self-reported 30-day point prevalence abstinence (ie, no smoking at all in the past 30 days) at 12-month follow-up. Self-reported smoking or abstinence is a standard method for assessing the efficacy of Web-delivered interventions [32]. The Society for Research on Nicotine and Tobacco Subcommittee on Biochemical Verification has suggested that biochemical confirmation is not necessary in population-based studies with no face-to-face contact and in studies where data are collected through the Web, telephone, or mail because of low demand characteristics of these studies [33,34].

Statistical Analyses

To determine distinct groups of log-in trajectories for each website, we used a functional clustering approach consisting of 3 steps: (1) presmoothing the binary daily engagement time series; (2) conducting functional principal component analysis [35], a dimension reduction procedure to summarize each participant’s log-in trajectory by low-dimensional functional principal component scores; and (3) applying the clustering large applications algorithm [36] to the derived functional principal component scores. This procedure does not rely on any assumptions on the shapes of trajectories and is capable of handling large datasets and complex missing data patterns. We determined the total number of trajectories for each website using predictive strength [37], which is a statistical criterion to assess how many groups can be predicted from the data and how well. We obtained each study participant’s log-in trajectory by transforming longitudinal sequences of log-in time stamps into a binary time series indicating log-in occurrence each day. Note that we chose not to use latent class growth curve approaches that have been used in other eHealth intervention engagement studies [12,13] because these methods do not handle very densely recorded longitudinal data without substantial data reduction (eg, reducing data into weekly or monthly log-in counts per participant) and often rely on restrictive assumptions on the shapes of trajectories.

After determining distinct trajectory clusters, we applied logistic regression models to investigate the associations between the trajectory clusters and the smoking cessation outcome. Both unadjusted and covariate-adjusted regression models were fitted. For covariate-adjusted models, we selected variables by stepwise Akaike information criterion (AIC) in both backward and forward directions. Covariates considered for adjustment were the baseline characteristics described above in the Measures subsection, including commitment to quit smoking, to control for participant characteristics that may confound any association with cessation outcomes. Finally, to identify baseline user characteristics associated with trajectory membership, we applied multinomial logistic regression models with baseline covariates as predictors and the log-in trajectory clusters as outcome. We selected variables in the final multivariate model via a stepwise AIC procedure from a pool of candidate baseline covariates that had a univariate association with log-in trajectory clusters.

Results

Description of Sample

Table 1 shows the baseline demographics and participant characteristics in both the WebQuit and Smokefree arms. Overall, participants were on average 46 years old, about 80% were female, about 80% were white, about 52% were employed, and about 72% had greater than high school education.

There were no baseline differences between treatment arms on how often participants had used the internet in the last 30 days (χ²_{2, n=2495}=2.3, P=.32). Fewer than half (about 42%) of the participants had made a quit attempt in the last year, and about 80% of the sample had been smoking for more than 10 years, with an average FTND score of 5.6 (moderate nicotine dependence). The data retention rate was 87.56% (2309/2637) and did not differ between arms.

Description of Distinct Groups of Trajectories

The functional clustering analysis of 52 weeks of log-ins revealed 3 distinct groups of trajectories for each of the intervention websites. Figure 1 shows log-in patterns for the first 16 weeks for WebQuit (left) and for Smokefree (right). The trajectories were easiest to visualize for the first 16 weeks of use. However, Multimedia Appendix 1 shows the full 52 weeks for reference. For the WebQuit website (Figure 1, left), the first trajectory group (682/1240, 55.00% of sample) logged at least one day in the first week and then had almost no log-ins after that. They were termed 1-week users. The second trajectory group (399/1240, 32.18% of sample) logged in an average of 1.8 days in the first week, 0.8 days in the second week, once every 3 weeks until week 5, and had very sporadic log-ins in week 6 and beyond. They were termed 5-week users. The third trajectory group (159/1240, 12.82% of sample) logged in an average of 3.7 days in the first week, 3.3 days in the second week, 2.7 days in the third week, 2.4 days in the fourth week, 1.6 days in week 5, once in week 6, and then on average once every month starting in week 7 and continuing in this pattern until week 52. They were termed 52-week users.

For the Smokefree website (Figure 1, right), the first trajectory group (645/1309, 49.27% of sample) logged in less than once on average in the first week and then had almost no log-ins after that. As with WebQuit, they were termed 1-week users. The second trajectory group (395/1309, 30.18% of sample) logged in once in week 1, every other week until week 4, and then had almost no log-ins after that. They were termed 4-week users. The third trajectory group (269/1309, 20.55% of sample) logged in an average of 1.5 days in weeks 1 and 2, once in week 3, every other week over the period of weeks 4 to 5, and then had almost no log-ins after that. They were termed 5-week users. Note also that in both intervention arms, there was a pattern of a spike in log-ins at week 12, corresponding to the invitation to complete the 12-week outcome survey that, while completely independent of the interventions, likely triggered some users to engage with their assigned intervention website.

Trajectory Membership Prediction of Smoking Cessation Outcome

Table 2 shows each intervention arm’s trajectory group membership as a predictor of 30-day point prevalence abstinence at the 12-month follow-up. For WebQuit, abstinence rates for these 3 trajectory groups were 116/562 (20.6%) for 1-week users, 100/370 (27.0%) for 5-week users, and 51/149 (34.2%) for 52-week users. Compared with 1-week users, 5-week users had 57% higher odds (OR 1.57, 95% CI 1.13-2.17; P=.007) of being abstinent at 12 months, and 52-week users had 124% higher odds (OR 2.24, 95% CI 1.45-3.43; P<.001) of being abstinent at 12 months. These models adjusted for the baseline covariates selected as outlined in the Methods section and included smoking half a pack or more, the commitment to quitting score, and screening positive for panic disorder. Descriptively, for Smokefree, abstinence rates for the 3 trajectory groups were 139/562 (24.7%) for 1-week users, 85/349 (24.4%) for 4-week users, and 81/252 (32.1%) for 5-week users.

‎View this figure

Compared with 1-week users, 4-week users were not more likely to be abstinent at 12 months (OR 1.00, 95% CI 0.73-1.37; P=.99), but 5-week users had 48% higher odds of being abstinent (OR 1.48, 95% CI 1.05-2.07; P=.02). This analysis adjusted for selected baseline covariates of education, smoking more than 10 years, smoking within 5 minutes of waking, commitment to quitting, and whether one has a partner who smokes.

Baseline Characteristics Predicting Trajectory Membership

Since the groups of trajectories were different across the 2 arms, we explored the baseline characteristics predicting membership in the groups for the 2 arms separately. For WebQuit, baseline characteristics associated with trajectory membership were age, smoking for at least the past 10 years, screening positive for depression, and screening positive for anxiety (all P<.05; results not shown). Controlling for the impact of related covariates, the adjusted multivariate regression model selected by stepwise AIC procedure showed that smoking for at least the past 10 years and screening negative for anxiety each, respectively, predicted a 90% higher odds (OR 1.90, 95% CI 1.14-3.14) and a 56% higher odds (OR 1.56, 95% CI 1.06-2.33) of being a 52-week user (compared with being a 1-week user) (Table 3). Since smoking history is partly a reflection of one’s age, and the variables age, smoking history, and anxiety were correlated with each other, when we calculated a model containing age (categorized by decade), only age emerged as a significant predictor (see Multimedia Appendix 2).

For Smokefree, the baseline characteristics associated with trajectory membership in univariate analysis were being unemployed, smoking less than half a pack per day, and screening as not having PTSD (all P<.05; results not shown). Controlling for the impact of related covariates, the multivariate regression model showed that smoking less than half a pack per day predicted a 72% higher odds (OR 1.72, 95% CI 1.23-2.44) of being a member of the 5-week group, compared with the 1-week user group (Table 3). Being unemployed predicted a 79% higher odds (OR 1.79, 95% CI 1.33-2.38) of being a member of the 5-week user group relative to the 1-week group. Screening negative for PTSD predicted 43% higher odds (OR 1.43, 95% CI 1.11-1.85) of being a member of the 4-week user group relative to the 1-week user group. There was no evidence in either sample that sex predicted trajectory membership (all P>.05).

Discussion

Principal Findings

To our knowledge, this was one of few studies to analyze usage trajectories of eHealth interventions and examine the association between trajectory group membership and health outcomes [12,13]. The study found (1) 3 distinct groups of log-in trajectories for 2 Web-delivered interventions for smoking cessation, (2) that these trajectory groups differentially predicted smoking outcomes at 12 months, and (3) that certain user characteristics are associated with membership in certain trajectory groups. A 5-week usage of either website, and 52-week usage only of WebQuit, predicted a higher odds of quitting smoking. In general, the WebQuit intervention had a greater number of weekly log-ins within each of the 3 trajectory groups as compared with those of the Smokefree intervention. These major results are synthesized and interpreted in greater detail in this discussion.

Usage Trajectories and Health Outcomes

Regarding the first trajectory group, half the participants in both arms were 1-week users, which is a significant concern because they were the least likely to abstain from smoking at 12 months. Thus, it is imperative to learn why a participant would have almost no log-ins after a single week of use. User-centered design research, including laboratory observations and diary studies, could help elucidate the qualities of the intervention that cause an individual to discontinue use of the website. These individuals might benefit from a more intensive intervention, an eHealth intervention that uses a different treatment model, or one that is not eHealth (eg, individual telephone coaching). Regarding the second trajectory group, 5-week users were more likely to quit smoking in the WebQuit intervention (as well as for Smokefree, which had 5-week users as its third trajectory group). These results suggest that strategies to increase eHealth intervention engagement for 4 more weeks (ie, from 1 week to 5 weeks) could be highly cost effective. Example strategies worth testing include (1) proactive check-ins (via text message or phone calls) from staff about progress with the website, (2) daily automated text messages notifying the user of new content now available on the website, (3) rewards for each day’s use of the website with badges or redeemable prizes, and (4) a 5-week challenge that shows other users’ daily log-in progress toward the goal of 5 weeks of usage.

Regarding the third trajectory group, each intervention website had distinct log-in patterns that are likely explained by differing website structures. For Smokefree, this group was the 5-week users. The fact that they had almost no log-ins at 5 weeks and beyond is likely a reflection of Smokefree’s structure—an informational resource for users, functioning like reference material. Thus, 5 weeks may be sufficient time for a user to glean all needed information from Smokefree and apply it appropriately to quitting smoking, as they had 48% higher odds of quitting smoking (compared with 1-week users). For WebQuit, this group was the 52-week users, who had 124% higher odds of quitting smoking (compared with 1-week users). Their much longer-term engagement is likely a reflection of WebQuit’s structure—a step-by-step skills-based program that includes tracking progress with urges and smoke-free days. This program structure may have encouraged long-term, spaced skills practice [6], which may have contributed to the 34% 12-month quit rates observed in WebQuit’s third trajectory group. In general, the findings for both websites’ third trajectory group suggest that consistent use of each program over time is prognostic of a better health outcome, which is contrary to the notion that consistent log-ins may be a marker of ongoing challenges and struggles to change a health behavior. E-intervention design should thus focus on methods to encourage engagement over time, which may include strategies similar to those suggested above.

Personal Characteristics and Usage Trajectories

The impact of personal characteristics on usage trajectories appeared to vary by intervention. Specifically, WebQuit users who had smoked for at least 10 years were more likely to be 5-week users and nearly twice as likely to be 52-week users than 1-week users. However, smoking history differences may be a reflection of age: users aged 50 years and over were over 8 times more likely to be 52-week users. This finding is consistent with past research showing that being older is a predictor of higher eHealth use [14-17], even though it was found only for WebQuit, not Smokefree, in this analysis. On the other hand, participants who screened positive for a mental health condition in either website (PTSD in Smokefree, and anxiety or depression in WebQuit) were more likely to be 1-week users, which suggests the need develop strategies to promote longer-term engagement for people with mental health disorders. There was no evidence in this study that sex predicted trajectory membership. Nonetheless, we recommend that future research examine many subgroup differences (eg, sex, race, age) in eHealth intervention trajectories as research on this model methodology expands to a wide variety of populations. Overall, these analyses suggest a need for further research on what baseline factors might predict different usage trajectories, and therefore inform the development of tailored interventions that facilitate long-term, consistent engagement, based on an individual’s specific baseline characteristics.

Limitations and Future Directions

The study had several key limitations. First, we tested only 2 websites, and both were focused on smoking cessation; thus, future research should examine the extent to which results generalize to other behaviors and to other types of eHealth interventions. Second, cessation outcome data were self-reported for reasons stated in the Methods. Remote biochemical validation of smoking cessation would have introduced biases, including low response rates, prohibitive cost, challenges with confirming the identity of the person providing the sample, and inability to confirm abstinence beyond 24 hours [33,34].

Conclusions

In general, the WebQuit intervention had a greater number of weekly log-ins within each of the 3 trajectory groups as compared with those of the Smokefree intervention. The 1-, 4-, and 5-week usage of websites may be common patterns of how people engage in eHealth interventions over time. The 5-week usage of either website, and 52-week usage only of WebQuit, predicted a higher odds of quitting smoking. Strategies to increase eHealth intervention engagement for 4 more weeks (ie, from 1 week to 5 weeks) could be highly cost effective.