This paper is in the following e-collection/theme issue:

Web-based and Mobile Health Interventions (3114) Text-messaging (SMS, WeChat etc)-Based Interventions (622) Smoking Cessation (518) Registered Report (518)

Published on in Vol 22, No 4 (2020): April

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/17734, first published .
Optimizing Text Messages to Promote Engagement With Internet Smoking Cessation Treatment: Results From a Factorial Screening Experiment

Optimizing Text Messages to Promote Engagement With Internet Smoking Cessation Treatment: Results From a Factorial Screening Experiment

Optimizing Text Messages to Promote Engagement With Internet Smoking Cessation Treatment: Results From a Factorial Screening Experiment

Authors of this article:

Amanda L Graham1, 2 Author Orcid Image ;   George D Papandonatos3 Author Orcid Image ;   Megan A Jacobs1 Author Orcid Image ;   Michael S Amato1, 2 Author Orcid Image ;   Sarah Cha1 Author Orcid Image ;   Amy M Cohn4 Author Orcid Image ;   Lorien C Abroms5 Author Orcid Image ;   Robyn Whittaker6 Author Orcid Image
Article Authors Cited by (16) Tweetations (9) Metrics

    Original Paper

    1Innovations Center, Truth Initiative, Washington, DC, United States

    2Mayo Clinic College of Medicine and Science, Rochester, MN, United States

    3Center for Statistical Sciences, Brown University, Providence, RI, United States

    4Oklahoma Tobacco Research Center, University of Oklahoma Health Sciences Center, Oklahoma City, OK, United States

    5Department of Prevention and Community Health, Milken Institute School of Public Health, The George Washington University, Washington, DC, United States

    6National Institute for Health Innovation, University of Auckland, Auckland, New Zealand

    Corresponding Author:

    Amanda L Graham, PhD

    Innovations Center

    Truth Initiative

    900 G Street, NW

    4th Floor

    Washington, DC, 20001

    United States

    Phone: 1 12024545938

    Email: agraham@truthinitiative.org


    Related ArticleThis is a corrected version. See correction statement in: https://www.jmir.org/2020/7/e21027/

    Background: Smoking remains a leading cause of preventable death and illness. Internet interventions for smoking cessation have the potential to significantly impact public health, given their broad reach and proven effectiveness. Given the dose-response association between engagement and behavior change, identifying strategies to promote engagement is a priority across digital health interventions. Text messaging is a proven smoking cessation treatment modality and a powerful strategy to increase intervention engagement in other areas of health, but it has not been tested as an engagement strategy for a digital cessation intervention.

    Objective: This study examined the impact of 4 experimental text message design factors on adult smokers’ engagement with an internet smoking cessation program.

    Methods: We conducted a 2×2×2×2 full factorial screening experiment wherein 864 participants were randomized to 1 of 16 experimental conditions after registering with a free internet smoking cessation program and enrolling in its automated text message program. Experimental factors were personalization (on/off), integration between the web and text message platforms (on/off), dynamic tailoring of intervention content based on user engagement (on/off), and message intensity (tapered vs abrupt drop-off). Primary outcomes were 3-month measures of engagement (ie, page views, time on site, and return visits to the website) as well as use of 6 interactive features of the internet program. All metrics were automatically tracked; there were no missing data.

    Results: Main effects were detected for integration and dynamic tailoring. Integration significantly increased interactive feature use by participants, whereas dynamic tailoring increased the number of features used and page views. No main effects were found for message intensity or personalization alone, although several synergistic interactions with other experimental features were observed. Synergistic effects, when all experimental factors were active, resulted in the highest rates of interactive feature use and the greatest proportion of participants at high levels of engagement. Measured in terms of standardized mean differences (SMDs), effects on interactive feature use were highest for Build Support System (SMD 0.56; 95% CI 0.27 to 0.81), Choose Quit Smoking Aid (SMD 0.38; 95% CI 0.10 to 0.66), and Track Smoking Triggers (SMD 0.33; 95% CI 0.05 to 0.61). Among the engagement metrics, the largest effects were on overall feature utilization (SMD 0.33; 95% CI 0.06 to 0.59) and time on site (SMD 0.29; 95% CI 0.01 to 0.57). As no SMD >0.30 was observed for main effects on any outcome, results suggest that for some outcomes, the combined intervention was stronger than individual factors alone.

    Conclusions: This factorial experiment demonstrates the effectiveness of text messaging as a strategy to increase engagement with an internet smoking cessation intervention, resulting in greater overall intervention dose and greater exposure to the core components of tobacco dependence treatment that can promote abstinence.

    Trial Registration: ClinicalTrials.gov NCT02585206; https://clinicaltrials.gov/ct2/show/NCT02585206.

    International Registered Report Identifier (IRRID): RR2-10.1136/bmjopen-2015-010687

    J Med Internet Res 2020;22(4):e17734

    doi:10.2196/17734

    Keywords

    smoking cessation; tobacco dependence; internet; text messaging 


    Background

    Internet-based interventions for smoking cessation have the potential to significantly impact public health, given their broad reach and proven effectiveness. Nine of 10 adults in the United States have internet access [1], more than one-third of all smokers—12.4 million individuals—look online each year to quit smoking [2], and hundreds of thousands enroll in freely available programs [3,4]. Systematic reviews and meta-analyses have demonstrated the effectiveness of tailored and interactive internet interventions for smoking cessation [5]. However, a sizable proportion of smokers disengage early from internet programs without being exposed to the content or features that can promote abstinence. Indeed, low levels of engagement with internet interventions have been documented across a range of health behaviors [6]. Given the evidence of a dose-response association between engagement and behavior change outcomes [7-9], identifying strategies to promote engagement has been noted as a priority across digital health interventions [10-12].

    The relationship of engagement to outcomes is complex and includes both behavioral and cognitive dimensions [11,13]. Engagement with an internet intervention can be usefully conceptualized into 3 phases [14], although users’ progress through the phases is often nonlinear. In the first phase, an individual decides to visit a website to determine its relevance and potential utility. In the second phase, the individual uses a part of the intervention. In the third phase, the individual returns to engage more fully with the intervention. This study aimed to influence engagement at this third phase by using prompts and reminders delivered via text messages. Text messaging is a proven intervention modality to promote smoking cessation [15] and a powerful strategy to increase intervention engagement [16,17]. Across most demographic groups, a majority of individuals own a mobile phone and use text messaging [18], including economically disadvantaged groups among whom tobacco use is more prevalent. However, to date, text messaging has not been tested specifically as an engagement strategy for internet cessation interventions [10], and little is known about how best to design such text messages [19].

    The Multiphase Optimization Strategy (MOST) is a method for systematically building and evaluating interventions to ensure they comprise active components delivered in optimal doses [20]. The screening phase of MOST is designed to determine which intervention components are active (ie, make a difference in the target outcome) and should be retained, and which are inactive and should be discarded. This is accomplished efficiently through a randomized experiment involving a factorial design, which allows for the examination of several design factors simultaneously. As they are scalable and can automate the delivery of many experimental conditions, internet interventions are well suited to conduct such experiments [21]. Guided by the principles of MOST, we conducted a screening experiment to evaluate 4 experimental factors hypothesized to promote engagement with an internet smoking cessation program.

    Personalization incorporates user-specific elements (eg, name) to enhance the personal relevance of messaging. People are more likely to actively process information if they perceive it to be personally relevant [22]. Personalization can increase smokers’ attention to written information and the perceived quality of that information [23,24] and is a desired and expected feature of text messaging [25]. Supported by prior literature [26], we hypothesized that text messages that incorporate personalized content would be more efficacious than generic ones.

    Integration refers to the ability to interact with an intervention platform through the web and text messages, accomplished by sharing data between systems. This approach leverages the unique and combined advantages of these 2 different modalities to create a seamless user experience. Internet interventions can be used to deliver multimedia content but require users to initiate contact, whereas text messages are a powerful form of push notification that have a 98% open rate, with 90% of messages being read within 3 min [27]. A 2010 meta-analysis found that the effectiveness of internet interventions for a variety of health behaviors was enhanced by adding text messaging [28], but these early studies and others [29] most often delivered text messaging in parallel to a web-based intervention with little integration between the 2 modalities. We hypothesized that an intervention that allows smokers to interact with the tools and content of an internet program via text messages would be more effective in promoting treatment engagement than delivering text messages in parallel to an internet intervention.

    Dynamic Tailoring delivers individually tailored feedback that adapts over time to a smoker’s needs. Research over several decades supports the superiority of individually tailored messaging over generic, one-size-fits-all messaging in improving behavior change outcomes [26] and in promoting intervention engagement [30]. However, it typically involves messaging around static, theory-driven psychosocial constructs (eg, readiness to quit and self-efficacy) gathered at the outset of an intervention. Few studies have dynamically tailored communications to deliver feedback based on a user’s pattern of intervention engagement [31]. We hypothesized that messages tailored to a user’s pattern of engagement to encourage the exploration of components they have not yet used and prompt continued engagement with the ones they have would yield higher engagement than messages without this kind of feedback.

    Message intensity refers to the delivery schedule of text messages. One of the largest randomized smoking cessation trials demonstrating the effectiveness of a text message intervention [32] delivered 5 messages a day for the first 5 weeks, followed by an abrupt drop to just 3 messages per week for the next 26 weeks. However, a meta-analysis of health behavior change interventions found that the largest effect size was observed for text message interventions with tapered intensity (ie, gradually decreasing content delivery [26]). We set tapered intensity as the active form of this factor and hypothesized that it would make message delivery more salient and impactful than a fixed schedule of messages followed by an abrupt drop-off.

    Objective

    To summarize, this factorial screening experiment evaluated the impact of personalization, integration, dynamic tailoring, and message intensity on engagement with an internet smoking cessation program. We hypothesized that the active form of each experimental factor would yield higher rates of engagement with one or more features of the program and overall metrics of engagement.


    Experimental Design

    This full factorial experiment had 4 factors, each of which was implemented at 2 levels: personalization (on/off), integration (on/off), dynamic tailoring (on/off), and message intensity (tapered/abrupt). The factors were designed to be compatible with each other but also to deliver a coherent intervention when implemented singularly. All participants had full access to the website to use as they desired. In addition, they were randomized by a computer algorithm to one of the 24=16 experimental conditions, stratified by whether they enrolled on a desktop/mobile because mobile access to the website may influence engagement. The use of the website and text messages was automatically tracked for 3 months to allow sufficient time to examine the impact of the text message intervention on intervention engagement because most nonusage attrition happens within this period [33]. There was no involvement by research staff, and there were no missing data.

    Procedure

    This fully automated experiment was conducted within BecomeAnEX (EX), a free, evidence-based smoking cessation program developed in 2008 by Truth Initiative in collaboration with Mayo Clinic. Since it launched, more than 800,000 tobacco users have registered on the site. Approximately 80% of newly registered users sign up for text messaging. As described previously [34], newly registered users who met study eligibility criteria were randomized to 1 of the 16 experimental arms. Eligibility criteria were current smoking (every day/some days), US residence, aged 18 years or older, and enrollment in the EX text message program during website registration. To register on EX, users must agree to the terms of use and privacy policy, which state that (1) Truth Initiative automatically collects information about use of the site, (2) information is used for research and quality improvement purposes, and (3) personal information is kept confidential. To enroll in the EX text message program, users enter their mobile number and explicitly consent to receive text messages during website registration. The study was conducted as a quality improvement project, meaning that eligible individuals were automatically randomized; no recruitment information was presented, and no study informed consent was solicited. The Chesapeake institutional review board approved the trial protocol (CR00086431).

    Web-Based Cessation Program

    EX, which is accessible on any web-enabled device, was designed around tobacco dependence treatment guidelines [35], Social Cognitive Theory [36], and the Mayo Clinic model for engaging tobacco users in cessation treatment [37]. At the time of this study, users could engage with 6 interactive features: (1) Set Quit Date assists users in selecting a quit date, (2) Track Smoking Triggers allows users to track cigarettes and identify personal smoking triggers associated with smoking, (3) Beat Smoking Triggers encourages identification of strategies to dissociate cigarettes from triggers, (4) Choose Quit Smoking Aid educates users about medication and helps them create a medication plan, (5) Build Support System discusses the importance of social support and encourages users to identify supportive friends/family, and (6) EX Community introduces users to a large online social network of current and former smokers. Higher levels of engagement with the site and its features are associated with greater odds of quitting [8,38].

    Text Messaging Intervention

    The standard EX text message program is a 90-day program that is fully automated. Also designed around tobacco cessation treatment guidelines [35] and Social Cognitive Theory [36], like the EX website, the program is designed to enhance motivation for quitting, educate users about addiction, encourage the use of evidence-based cessation methods (eg, nicotine replacement therapy and peer support), prepare them for quitting by building confidence and coping skills, praise success, reinforce the benefits of quitting, and support users in maintaining abstinence. The program is interactive, with bidirectional messages (eg, true/false, multiple-choice, and yes/no questions) and keywords that allow users to request on-demand support (eg, CRAVE, SLIP, and MOOD), additional detail about specific topics (MORE), or to unenroll (STOP). Approximately 10% of messages contain URLs to the EX website for additional information. The program is tailored to a participant’s quit date, should they choose to set one. Users can set and reset their quit date as often as they like either via the website or text messages. This basic level of integration is expected by users and was common across all experimental conditions.

    Text messages focused on promoting engagement were added to this standard program according to the experimental factors described in the following section. These messages were designed to encourage or facilitate the use of the interactive features of EX and promote sustained engagement.

    Experimental Factors

    Personalization

    Half of the participants received text messages personalized with their EX username, chosen during registration on the website. The remaining participants received generic messages with no personalization.

    Integration

    Participants with this experimental factor turned on were able to engage with interactive features of EX through special keywords and interactive messages. For example, the use of Track Smoking Triggers and Beat Smoking Triggers was facilitated as follows:

    System: Quitting is a journey—not an event. If you’ve had a slip, learn about your triggers by texting back SMOKED or going to http://bit.ly/trackcig and noting your high-risk situations.
    User: SMOKED.
    System: Ok, you smoked—we saved it in your EX Plan. Reply with the trigger that made you want to smoke.
    User: Sitting in traffic.
    System: Got it, we’ve noted it in your EX Plan. To make a plan to separate from this trigger, text back COPE.
    User: COPE.
    System: You said that Sitting in traffic was a trigger for you. What else can you do instead of smoke the next time that trigger comes up? Reply with your plan, the more detail the better!
    User: Listen to music.
    System: Great idea! It’s now in your EX Plan at http://bit.ly/beattrigger so you can always reference it if that trigger pops up.

    As seen in this exchange, interaction with the platform via text messages was reflected on the website at the user’s next visit. This approach was used for the first 5 interactive features described earlier. To facilitate engagement with the EX Community via text message, a slightly different approach was required for technical reasons. Once a participant had accessed the community via the website, a special keyword (TIPS) allowed the user to request advice and guidance from community members that had been manually curated from actual community posts. For participants with the integration feature turned off, there was no integration of the website and text message programs beyond the quit date feature, and special keywords were not available.

    Dynamic Tailoring

    Half of the participants received messages tailored to real-time engagement data from EX. Messages reinforced actions that they had already taken or prompted the exploration of features they had not yet used. The remaining participants received standard messages that were agnostic to prior use of EX.

    Message Intensity

    The intervention duration was 12 weeks for both groups. Both groups received 2 messages per day for the first 3 days of the program to ensure a standardized onboarding experience, and in both program versions, approximately half of all engagement messages solicited a response from users. Participants randomized to tapered intensity received a total of 69 engagement messages delivered as follows: weeks 1 to 2, 14 messages per week; weeks 3 to 4, 7 messages per week; weeks 5 to 8, 4 messages per week; weeks 9 to 11, 3 messages per week; and week 12, 2 messages. Participants randomized to an abrupt intensity drop-off received a total of 28 engagement messages, which were delivered as follows: week 1, 8 messages; week 2, 4 messages; weeks 3 to 8, 2 messages per week; weeks 9 to 12, 1 message per week.

    Sources of Data

    Gender, age, and smoking status (every day or some days) entered during website registration were extracted from the EX database. Website utilization metrics were extracted at 3-month postrandomization and included measures of website engagement (number of website visits, time on site in minutes, and page views) and the use of the 6 interactive features described earlier. Text message data were also extracted at 3-month postrandomization and included the number of messages received and sent by participants, the use of the 6 interactive features described earlier via text messages, the use of keywords, and the date of unenrollment.

    Analytic Plan

    A full factorial design was used in the study design phase [34]. The primary outcome used for sample size calculations was a composite engagement score, with weights given by the regression coefficients of a logistic regression model developed to measure the effects of website engagement on 3-month abstinence rates in the control arm of a previous randomized trial by our group [7]. This composite engagement score had the advantage of being continuously distributed, even if some of the individual engagement metrics were binary or count data. A priori sample size calculations based on a normal approximation to the distribution of the composite score determined that a sample of 864 (n=432 per factor level) participants would allow us to detect small main effects (d=0.25) or moderate second-order interactions (d=0.50) on normalizing transformations of this composite outcome. Power was set at 80% at a 2-sided significance level of alpha of .05 out of 10 (multiplicity adjustment based on 4 main effects and 6 two-way interactions in a factorial model, with third- and fourth-order interactions excluded a priori).

    In carrying out our original analytic plan [34], we made 3 post hoc modifications. First, ongoing enhancements to the EX website led us to question the applicability of the weights of the original composite engagement metric. We decided to analyze our engagement metrics separately and to identify common patterns in standardized factorial effects across the full set of engagement metrics. Our primary outcomes were (1) number of interactive features used through the web or text messages, (2) total time spent on the website, (3) number of page views, and (4) return visits to the website (ie, postregistration). This approach is consistent with the multidimensional nature of engagement and numerous systematic reviews and meta-analyses that have called for more standardized engagement metrics to advance the field [10-12,30,39,40]. Our intent was to ensure that study effect sizes could be included in pooled analyses.

    Second, evidence of synergistic interactions led us to supplement analyses focused on individual experimental factors with between-arm comparisons that capture the joint effect of multiple terms in the full factorial model, with the hope of identifying an arm with superior performance across all engagement metrics. If such an arm could be identified, the need to reestimate weights for a composite metric would become moot: any arm that dominated each available metric would also dominate their weighted average. To facilitate such between-arm comparisons, we presented CIs for each engagement metric for all 16 arm-specific means, with the confidence level adjusted so that an overlap in the respective CIs can be interpreted as lack of significant pairwise differences between the arms being compared at the alpha value of .05 significance level. Unlike cases where an arm-specific mean is being compared with constant, pairwise comparisons of means based on the overlap method involve uncertainty in the centers of both CIs under inspection [41]. In such cases, 2 arms may still be significantly different from one another at an alpha value of .05, even if their 95% CIs overlap. To correctly assess the presence of significant pairwise differences based on the overlap rule, we have employed narrower intervals whose individual confidence levels were set to about 83.5% under normality [42]. CIs were first calculated in the scale of the continuously and normally distributed linear predictor and then back-transformed to the original outcome scale.

    Third, large skewness observed in time on site, page views, and website visits led us to transform the data to reduce the impact of outliers in the final model. Although a logarithmic transformation would have served this purpose, it would have changed the interpretation of the main effect of each factor in the original scale to an average of arm-specific medians. We dichotomized these variables instead at cut points that bifurcated the sample at an approximately 1:2 ratio. The cut points were one or more unique website visits (333/864, 38.5% of the sample), 15 or more minutes of website use (257/864, 29.8% of the sample), and 25 or more page views (278/864, 32.2% of the sample). Additional sensitivity analyses examined the robustness of the findings to variation in these cutoffs.

    Use (yes/no) of specific interactive features and overall engagement metrics were analyzed via logistic regression except for interactive feature utilization, a count variable analyzed via quasi-Poisson regression with a scale parameter, φ, to account for under- or overdispersion. CIs for marginal factor effects were estimated via a parametric percentile bootstrap procedure [43] with 1 million replicates. Effect size calculations were based on standardized mean differences (SMDs) between high and low levels of experimental factors when calculating main effects and between arms 1 and 16 when calculating the full impact of the intervention, including main effects and higher order interactions. Arm 1 was defined with all experimental factors off, whereas Arm 16 was defined by all experimental factors on. SMDs for frequency counts were calculated as (μ1−μ2)/[φ(μ1+μ2)]1/2, where μ1 and μ2 were the sample means of each comparison group. SMDs for binary outcomes were calculated as (p1−p2)/[p1×q1+p2×q2]1/2, where p1=1−q1 and p2=1−q2 were the sample outcome prevalence of each comparison group. All analyses were conducted using the glm function in R version 3.6.1 (R Foundation).


    Participants

    Between March 29 and June 5, 2018, 864 newly registered users on EX who met the study eligibility criteria were randomized. Of those, 83.4% (721/864) enrolled on a mobile device, and 16.6% (143/864) enrolled on a desktop. Most (844/864, 97.7%) participants were every day smokers, and 2.3% (20/864) of the participants were some day smokers. The sample was predominantly female (637/864, 73.7%). Age distribution was as follows: 18 to 30 years (182/864, 21.1%), 31 to 44 years (322/864, 37.3%), 45 to 65 years (313/864, 36.2%), and 65 years and older (47/864, 5.4%). No between-arm differences were observed for any of the abovementioned variables (all values for P>.15).

    Intervention Engagement

    Of the 864 participants randomized, 461 (53.5%) completed the full 90-day text message program. Among participants who unenrolled, the median day of unenrollment was 8 days postrandomization (IQR 3-22). On average, study participants used, on average, 2.40 (SD 1.41) of the 6 targeted interactive features. Use by feature was as follows: Set Quit Date, 85.8% (741/864); Track Smoking Triggers, 55.4% (479/864); Choose Quit Smoking Aid, 37.0% (320/864); Visit Community, 33.4% (289/864); Beat Smoking Triggers, 15.3% (132/864); and Build Support System, 12.7% (110/864). Study participants received a median of 87 text messages (IQR 27-160) during the 3-month intervention period and sent a median of 4 text messages (IQR 3-12). One-fourth (213/864, 24.7%) of the sample used one or more keywords: MOOD (76/864, 8.8%), HELP (74/864, 8.6%), CRAVE (70/864, 8.1%), SLIP (60/864, 6.9%), SOS (34/864, 3.9%). Among participants randomized to the active integration arms, 11.6% used at least one special keyword: SMOKED (61/864, 7.1%), COPE (46/864, 5.3%), TIPS (27/864, 3.1%), TRIGGER (25/864, 2.9%), MEDS (14/864, 1.6%). Participants in the active integration arms used standard keywords at similar rates to other participants (all differences <1 percentage point).

    Engagement Outcomes

    Table 1 (interactive features) and Table 2 (key engagement metrics) reflect the study findings under the original analytic plan. They show average response levels at on and off levels of each experimental factor and raw mean differences that correspond to marginal factor effects. SMDs are also included, as they allow us to calibrate the clinical significance of our nominal P values. In this study, P<.001 corresponds to SMDs ranging from 0.17 to 0.28 (small effects in Cohen nomenclature [44]), whereas P values greater than .001 and less than .05 correspond to SMDs ranging from 0.09 to 0.13.

    As seen in Table 1, integration is the strongest experimental factor affecting interactive feature utilization, raising usage rates of Choose Quit Smoking Aid by 18.7 percentage points (95% CI 12.5 to 24.8) and Build Support System by 11.8 percentage points (95% CI 7.2 to 16.4). Altogether, integration raised the average number of interactive features used by participants by 0.36 (95% CI 0.16 to 0.57) when the study-wide mean did not exceed 2.4 features (median 2, IQR 1-3).

    Table 2 suggests that dynamic tailoring was the experimental factor with broadest impact, in that it raised both the average number of interactive features used by 0.29 (95% CI 0.09 to 0.50) and the probability of higher engagement levels by 7.3 percentage points for page views (95% CI 1.1 to 13.5). Its beneficial effect on interactive feature use appears driven by similar increases in the rates of Build Support System (6.3 points; 95% CI 1.7 to 10.9), Track Smoking Triggers (6.6 points; 95% CI 0.1 to 13.1), and Beat Smoking Triggers (5.4 points, 95% CI 0.5 to 10.3).

    Table 1. Marginal effects of experimental design factors on interactive feature utilization rates (95% CI).
    FactorInteractive feature

    		Set Quit DateChoose Quit
    Smoking Aid    		Build Support
    System    		Track Smoking
    Triggers    		Beat Smoking
    Triggers    		Visit Community
    Personalization

    		On84.5 (81.1 to 87.9)35.2 (30.9 to 39.6)14.6 (11.2 to 17.9)55.0 (50.4 to 59.5)16.8 (13.3 to 20.4)32.5 (28.2 to 36.8)

    		Off85.7 (82.3 to 89.0)39.3 (34.9 to 43.7)12.3 (9.1 to 15.5)55.7 (51.1 to 60.2)15.0 (11.6 to 18.4)35.0 (30.6 to 39.4)

    		Raw difference−1.1 (−5.9 to 3.6)−4.1 (−10.3 to 2.1)2.3 (−2.3 to 6.9)−0.7 (−7.1 to 5.8)1.8 (−3.1 to 6.7)−2.5 (−8.7 to 3.7)

    		SMDa−0.02 (−0.12 to 0.07)−0.06 (−0.15 to 0.03)0.05 (−0.05 to 0.14)−0.01 (−0.10 to 0.08)0.04 (−0.06 to 0.13)−0.04 (−0.13 to 0.06)
    Integration

    		On84.5 (81.1 to 88.0)46.6 (42.0 to 52.2)19.3 (15.6 to 23.0)58.2 (53.6 to 62.7)17.7 (14.1 to 21.3)32.3 (28.0 to 36.6)

    		Off85.7 (82.3 to 89.0)27.9 (23.8 to 32.1)7.5 (4.8 to 10.3)52.5 (47.9 to 57.1)14.1 (10.8 to 17.4)35.2 (30.8 to 39.6)

    		Raw difference−1.1 (−5.9 to 3.7)18.7 (12.5 to 24.8)b11.8 (7.2 to 16.4)b5.7 (−0.8 to 12.1)3.6 (−1.3 to 8.5)−2.9 (−9.1 to 3.2)

    		SMD−0.02 (−0.12 to 0.07)0.28 (0.18 to 0.38)b0.25 (0.15 to 0.34)b0.08 (−0.01 to 0.17)0.07 (−0.03 to 0.17)−0.04 (−0.14 to 0.05)
    Dynamic Tailoring

    		On87.0 (83.8 to 90.3)37.7 (33.3 to 42.1)16.6 (13.1 to 20.1)58.7 (54.1 to 63.1)18.6 (15.0 to 22.3)36.6 (32.1 to 41.0)

    		Off83.1 (79.7 to 86.7)36.8 (32.5 to 41.2)10.3 (7.3 to 13.3)52.1 (47.4 to 56.7)13.2 (9.9 to 16.4)30.9 (26.6 to 35.2)

    		Raw difference3.9 (−0.9 to 8.6)0.9 (−53 to 7.1)6.3 (1.7 to 10.9)c6.6 (0.1 to 13.1)d5.4 (0.5 to 10.3)d5.7 (−0.5 to 11.9)

    		SMD0.08 (−.02 to 0.17)0.01 (−0.08 to 0.10)0.13 (0.04 to 0.23)c.09 (0.00 to 0.19)d.11 (0.01 to 0.20)d.09 (−.01 to 0.18)
    Intensity

    		Tapered83.4 (79.9 to 86.9)38.2 (33.7 to 42.6)14.6 (11.2 to 17.9)56.4 (51.8 to 60.9)18.2 (14.6 to 21.8)31.4 (27.1 to 35.7)

    		Abrupt86.8 (83.6 to 90.1)36.4 (32.0 to 40.7)12.3 (9.1 to 15.5)54.3 (49.7 to 58.9)13.6 (10.3 to 16.9)36.1 (31.7 to 40.6)

    		Raw difference−3.4 (−8.2 to 1.4)1.8 (−4.4 to 8.0)2.3 (−2.3 to 6.8)2.1 (−4.4 to 8.5)4.5 (−0.4 to 9.5)−4.8 (−10.9 to 1.4)

    		SMD−0.07 (−0.16 to 0.03)0.03 (−0.06 to 0.12)0.05 (−0.05 to 0.14)0.03 (−0.06 to 0.12)0.09 (−0.01 to 0.18)−0.07 (−0.16 to 0.02)

    aSMD: standardized mean difference.

    bP<.001.

    cP<.01.

    dP<.05.

    Table 2. Marginal effects of experimental design factors on key engagement metrics (95% CI).
    FactorKey Engagement Metric

    		Feature utilization, meanPage views ≥25, %Time on site ≥15 min, %Returned to website, %
    Personalization

    		On2.38 (2.24 to 2.53)31.6 (27.2 to 36.0)31.3 (27.0 to 35.6)37.9 (33.3 to 42.4)

    		Off2.43 (2.28 to 2.58)32.9 (28.5 to 37.3)28.4 (24.1 to 32.6)39.1 (34.6 to 43.7)

    		Raw difference−0.05 (−0.25 to 0.16)−1.3 (−7.5 to 4.9)2.9 (−3.2 to 9.0)−1.2 (−7.7 to 5.2)

    		Standard mean difference−0.021 (−0.115 to 0.074)−0.017 (−0.111 to 0.077)0.045 (−0.049 to 0.139)−0.019 (−0.113 to 0.075)
    Integration

    		On2.59 (2.44 to 2.74)30.8 (26.5 to 35.2)29.7 (25.4 to 34.0)37.0 (32.4 to 41.5)

    		Off2.22 (2.08 to 2.37)33.7 (29.3 to 38.1)29.9 (25.6 to 34.0)40.0 (35.5 to 44.6)

    		Raw difference0.36 (0.16 to 0.57)a−2.9 (−9.1 to 3.3)−0.2 (−6.3 to 5.9)−3.0 (−9.5 to 3.4)

    		Standard mean difference0.166 (0.071 to 0.260)a−0.043 (−0.137 to 0.051)−0.003 (−0.097 to 0.091)−0.044 (−0.138 to 0.048)
    Dynamic Tailoring

    		On2.55 (2.40 to 2.70)35.9 (31.4 to 40.4)32.3 (28.0 to 36.7)41.6 (36.9 to 46.2)

    		Off2.26 (2.12 to 2.40)28.6 (24.3 to 32.9)27.3 (23.1 to 31.0)35.4 (30.9 to 39.9)

    		Raw difference0.29 (0.09 to 0.50)b7.3 (1.1 to 13.5)c5.0 (−1.0 to 11.1)6.2 (−0.3 to 12.6)

    		Standard mean difference0.134 (0.040 to 0.229)b0.110 (0.016 to 0.205)c0.078 (−0.016 to 0.173)0.089 (−0.004 to 0.184)
    Intensity

    		Tapered2.42 (2.27 to 2.57)33.1 (28.72 to 37.6)31.3 (27.1 to 35.7)40.0 (35.5 to 44.6)

    		Abrupt2.39 (2.25 to 2.54)31.4 (27.0 to 35.8)28.3 (24.1 to 32.6)36.9 (32.5 to 41.5)

    		Raw difference0.03 (−0.18 to 0.23)1.7 (−4.5 to 8.0)3.0 (−3.0 to 9.1)3.1 (−3.3 to 9.5)

    		Standard mean difference0.012 (−0.083 to 0.106)0.028 (−0.066 to 0.122)0.047 (−0.047 to 0.141)0.045 (−0.049 to 0.139)

    aSMD: standardized mean difference.

    bP<.001.

    cP<.01.

    The main effects of intensity and personalization failed to attain even nominal levels of statistical significance on any engagement metrics. However, the impact of these 2 experimental factors was still beneficial as a whole via their synergistic interactions with dynamic tailoring and integration. Examination of the factorial models for each interactive feature in isolation revealed a synergistic interaction between integration and personalization on Set Quit Date (P<.001) and a synergistic interaction between dynamic tailoring, integration, and personalization on Track Smoking Triggers (P=.02). A synergistic interaction of dynamic tailoring × integration × intensity × personalization was also detected for time on site (P=.01), while a synergistic interaction of integration and personalization was detected for time on return visits to the website (P=.04).

    To better understand the joint effect of all 4 experimental factors, Table 3 (interactive features) and Table 4 (key engagement metrics) present point estimates and 95% CIs for arm-specific means based on simulation findings also depicted in Multimedia Appendices 1 and 2. As seen in Table 3, Arm 16 yielded the highest engagement rates for Set Quit Date, Build Support System, Track Smoking Triggers, and Beat Smoking Triggers. It lagged behind other arms in terms of the proportions of users at the high engagement level for Choose Quit Smoking Aid and Visit Community. Overall, Arm 16 had the highest rate of interactive feature use (2.95 out of 6).

    As seen in Table 4, Arm 16 also had the greatest proportion of participants at the high engagement level for page views (45%) and return visits (51%), although it ranked second with regard to the proportion at the high engagement level for time on site (41%). Sensitivity analyses that varied the cutoffs for page views and time spent on site from the 55th to the 75th percentile (ie, from 15 to 32 pages and 8 to 20 min, respectively) confirmed the superiority of Arm 16, suggesting that these findings are robust to the choice of cutoff.

    Table 3. Arm-specific interactive feature utilization rates (95% CI).
    ArmFactoraInteractive Feature

    		PbIGcTdISeSet quit dateChoose quit smoking aidBuild support systemTrack smoking triggersBeat smoking triggersVisit community
    1−−−−95 (86-99)21 (12-33)3 (0-12)48 (35-61)14 (6-25)41 (29-54)
    2−−−+88 (77-95)43 (30-56)6 (2-16)48 (35-61)6 (2-16)32 (21-45)
    3−−+−86 (75-94)28 (17-41)8 (3-18)55 (42-68)12 (5-23)39 (27-52)
    4−−++90 (80-96)30 (19-43)12 (5-23)61 (48-73)21 (12-33)45 (32-58)
    5−+−−85 (73-92)54 (40-66)10 (4-20)57 (44-70)8 (3-18)34 (22-47)
    6−+−+75 (63-85)45 (32-58)7 (9-29)65 (51-76)21 (12-33)23 (13-35)
    7−++−81 (69-90)43 (30-56)23 (13-35)52 (39-65)17 (9-29)37 (25-51)
    8−+++85 (73-92)52 (39-65)19 (10-31)59 (46, 71)21 (12-33)30 (19-43)
    9+−−−83 (71-91)25 (15-37)5 (1-14)52 (39-65)12 (5-23)26 (16-39)
    10+−−+75 (63-85)25 (15-37)5 (1-14)48 (35-61)14 (6-25)30 (19-43)
    11+−+−86 (75-94)23 (13-35)10 (4-20)54 (40-66)14 (6-25)35 (24-49)
    12+−++81 (69-90)30 (19-43)12 (5-23)54 (40-66)21 (12-33)34 (22-47)
    13++−−86 (75-94)46 (34-60)21 (12-33)52 (39-65)14 (6-25)34 (22-47)
    14++−+77 (65-87)37 (25-51)15 (8-27)46 (34-60)17 (9-29)28 (17-41)
    15+++−92 (82-97)52 (39-65)19 (10-31)65 (51-76)19 (10-31)43 (30-56)
    16++++95 (86-99)45 (32-58)30 (19-43)70 (57-81)25 (15-37)30 (19-43)

    aFor P, IG, and T, + implies On and − implies Off. For IS, + implies Tapered and − implies Abrupt.

    bP: Personalization.

    cIG: Integration.

    dT: Dynamic Tailoring.

    eIS: Intensity.

    Table 4. Arm-specific summaries of key engagement metrics (95% CI).
    ArmFactoraKey Engagement Metric

    		PbIGcTdISeFeature utilization (mean)Page views ≥25 (percent)Time on site ≥15 min (percent)Returned to website (percent)
    1−−−−2.21 (1.84-2.64)34 (23-48)23 (13-36)47 (34-60)
    2−−−+2.23 (1.86-2.66)28 (18-42)19 (10-32)36 (24-49)
    3−−+−2.29 (1.91-2.72)35 (23-49)24 (14-37)49 (36-62)
    4−−++2.58 (2.18-3.04)44 (31-58)44 (31-58)44 (31-58)
    5−+−−2.47 (2.08-2.92)28 (18-4230 (19-44)26 (16-40)
    6−+−+2.45 (2.06-2.9)28 (18-4)26 (16-39)37 (25-51)
    7−++−2.53 (2.13-2.98)33 (22-47)29 (19-43)37 (25-51)
    8−+++2.66 (2.25-3.12)29 (18-42)29 (18-42)35 (23-48)
    9+−−−2.01 (1.66-2.41)26 (16-39)26 (16-39)30 (19-43)
    10+−−+1.95 (1.61-2.35)31 (20-44)33 (21-46)37 (25-50)
    11+−+−2.21 (1.84-2.64)32 (21-46)34 (23-48)32 (21-46)
    12+−++2.31 (1.93-2.74)37 (25-50)35 (23-48)44 (31-58)
    13++−−2.53 (2.13-2.98)29 (19-43)37 (25-51)33 (22-47)
    14++−+2.21 (1.84-2.64)22 (12-35)22 (12-35)35 (23-49)
    15+++−2.90 (2.47-3.38)30 (19-43)21 (12-33)39 (27-52)
    16++++2.95 (2.52-3.44)45 (32-59)41 (29-55)51 (38-64)

    aFor P, IG, and T, + implies On and − implies Off. For IS, + implies Tapered and − implies Abrupt.

    bP: Personalization.

    cIG: Integration.

    dT: Dynamic Tailoring.

    eIS: Intensity.

    Significance of pairwise differences at an alpha of .05 can be evaluated using the overlap method applied to the 83.5% CIs shown in Figures 1 and 2. The results confirmed the conclusion that Arm 16 was either the top-ranked arm or did not differ from the top-ranked arm for any of the key engagement metrics and individual interactive features across all 16 experimental conditions. However, this begs the question of whether the intervention as a whole led to a significant improvement in utilization outcomes over no intervention at all. To answer this question, we calculated Arm 1 versus Arm 16 SMDs for all outcomes of interest. We found that the combined effect of our 4 experimental factors on interactive utilization features was highest for Build Support System (SMD 0.56; 95% CI 0.27 to 0.81), followed by Choose Quit Smoking Aid (SMD 0.38; 95% CI 0.10 to 0.66) and Track Smoking Triggers (SMD 0.33; 95% CI 0.05 to 0.61). No significant effect was found for Beat Smoking Triggers (SMD 0.20; 95% CI −0.07 to 0.47), Set Quit Date (SMD 0.00; 95% CI −0.28 to 0.28), or Visit Community (SMD −0.17; 95% CI −0.44 to 0.10). As for our key engagement metrics, the largest effect was on overall feature utilization (SMD 0.33; 95% CI 0.06 to 0.59), followed by time on site (SMD 0.29; 95% CI 0.01 to 0.57), page views (SMD 0.16; 95% CI −0.11 to 0.44), and return visits to the website (SMD 0.05; 95% CI −0.22 to 0.33). As no SMD >.30 was observed for main factor effects on any of the outcomes of interest, these results also suggest that, for at least some outcomes, the combined intervention was stronger than individual factors alone.

    ‎
    Figure 1. Arm-specific interactive feature utilization rates.
    View this figure
    ‎
    Figure 2. Arm-specific summaries of key engagement metrics (95% CIs).
    View this figure

    Principal Findings

    The aim of this factorial screening experiment was to test the effectiveness of text message design factors in increasing treatment engagement among adult smokers who enrolled in an internet smoking cessation intervention. We examined general metrics of engagement (ie, page views, time on site, and return visits) and specific engagement metrics for core intervention components. As hypothesized, synergistic factor effects in Arm 16, in which all 4 experimental factors were active simultaneously, resulted in the highest rates of interactive feature use. Nearly all participants (95%) set a quit date, 70% tracked their triggers, and approximately half (45%) designated a medication plan. Arm 16 also yielded the greatest proportion of participants at high levels of engagement, with 40% to 50% of the sample engaged at the highest levels of page views, time on site, and return visits.

    To our knowledge, this is the first study that has examined the impact of tailoring content based on treatment engagement. Dynamic tailoring, aimed at showcasing intervention features that participants had not yet used and encouraging ongoing utilization of those they had, was the most powerful of the 4 experimental factors tested in increasing engagement. It resulted in more participants engaging with the core components of tobacco dependence treatment, namely identifying and rallying the support of key people in their social network, identifying triggers for smoking, and developing coping strategies for those triggers. These results are consistent with previous research showing the effectiveness of individually tailored content and demonstrate that text messages can be an effective strategy to help shepherd and guide users through an intervention, much like would happen in a face-to-face encounter.

    Although dynamic tailoring encouraged participants to explore the most program features, enabling users’ access to these features via text messages had the most dramatic impact on individual feature use. With a relatively simple mechanism to engage users in an interactive fashion via text messaging, integration was effective at increasing the utilization of interactive features above the study-wide mean. It increased the use of tools related to medication selection and planning for social support by 19 and 12 percentage points, respectively. The fact that we did not observe an impact of integration on setting a quit date likely results from a ceiling effect, given that 85% of all participants set a quit date, often immediately after website registration.

    The lack of significant main effects for personalization and intensity is worth noting. Messages designed to feel individually tailored by using a person’s name, but where the content is clearly generic, did not appear to enhance program utilization. These findings are consistent with previous research that has shown that the use of a person’s name alongside generic information that is not perceived as personally relevant may even have counterproductive effects [45]. The fact that we only detected the synergistic effects of personalization when implemented alongside dynamic tailoring and integration is consistent with this notion. Intensity in this study was operationalized as the schedule of message delivery, and it was hypothesized that messages sent at less frequent and changing intervals over the 12-week intervention period would be more impactful than messages sent at a fixed interval with an abrupt drop-off. This hypothesis was not supported. It is possible that differences between the tapered and abrupt arms yielded variations in the dose of text messages received (eg, number of days enrolled, number of messages received), which we intend to explore in secondary analyses.

    Finally, it is also worth noting that none of the experimental factors we tested increased engagement with the online community. For technical reasons, this was the only interactive feature that required the user to first visit the website to subsequently engage with community content via an SMS text message (ie, TIPS keyword). It may be that a text message approach that did not require a website action may have yielded different findings. Alternatively, it may be that interest in and use of social support resources—whether online or offline—may be a more trait-like characteristic that is not subject to external manipulation [46]. Other research has also failed to increase the use of an online social network in an experimental design [47,48].

    Limitations

    Several limitations should be considered. This study does not allow us to draw conclusions about the impact of an engagement strategy versus none on smoking outcomes [10]. This factorial screening experiment was conducted as the first phase of 2-phase trial. Whereas all 16 arms were compared in terms of their ability to increase engagement with a smoking cessation intervention, the next phase of this study involves a comparative effectiveness trial (currently underway), which will allow us to evaluate the impact of the presence versus absence of a comprehensive engagement strategy (ie, Arm 16) in increasing abstinence rates. In addition, we cannot disentangle the effect of engagement messages alone because they were delivered as part of a broader text message intervention. This was a deliberate design decision because text messages solely focused on promoting engagement without reference to a user’s progress in quitting would likely have been perceived as irrelevant. Finally, we are cognizant of the fact that more does not always equal better when it comes to digital engagement [49,50]. Our classification of high levels of engagement was based on empirical distributions, which may not necessarily correlate with clinically meaningful engagement (ie, capable of promoting abstinence). In previous research, McClure et al [51] found positive effects of prescriptive message tone, dictated content viewing order, and reminder emails in a factorial screening experiment focused on internet engagement, but none of these features enhanced cessation outcomes [52]. Phase 2 of this study will enable us to evaluate the impact of the level of engagement on smoking outcomes, to investigate the role of complex issues such as reverse causality and confounding factors on the causal pathway from engagement to outcomes, and to determine what constitutes clinically meaningful engagement.

    Given the proliferation of mobile apps for smoking cessation, one may question our use of text messaging as an engagement strategy over push notifications via a mobile app. Several factors support our decision. Text messaging is a recommended cessation modality [53], whereas the evidence for smartphone apps is lacking [15]. A majority of apps that are downloaded are either never opened or used only once [54]. In addition, text messaging may feel less intrusive to users and be more widely accepted: 57% of app users uninstall/decline to install apps because of privacy concerns [55]. Finally, smartphone penetration lags behind high rates of cellphone ownership [18].

    Comparison With Prior Work

    Program completion results compare favorably with a large study from SmokefreeTXT [56], a US-based text message program from the National Cancer Institute. Among 25,283 individuals who subscribed to SmokefreeTXT, 38.3% (n=9686) completed the entire 42-day program. In our trial, 53.5% (461/864) of all study participants completed the full 90-day text message program. In both programs, a sizable number of participants disengaged early in treatment. We included all participants randomized to treatment in our analyses, whereas Augustson et al [56] restricted analyses to those that fully initiated treatment (ie, set a quit date and received first full day of treatment). Understanding patterns of early opt out from text message interventions and identifying opportunities for improving program delivery remain important areas of inquiry.

    This study addresses several gaps in the literature on digital interventions. Previous studies on improving engagement have largely focused on the use of email and telephone calls [10]. The use of text messaging as an engagement strategy is novel in this regard. In addition, previous studies have suffered from small sample sizes and lack of statistical significance [10] and provided few insights into the characteristics of digital intervention approaches that make them effective for promoting engagement [19]. This study was conducted as a full factorial, which provided a reasonably powered and efficient opportunity to test for the presence of both main factor effects and pairwise interactions. Other smoking cessation trials involving a full factorial design [57] have also found that higher order interactions can account for more variance than the marginal effects. When interactions are significant, factors should not be examined in isolation, but one should consider their joint effects (ie, the sum of their main effects and multiway interactions). Although the MOST framework has been primarily described as an efficient approach for evaluating the main effects of intervention components, it can be easily adapted to accommodate higher order terms as well. Our results and those of Cook et al [57] suggest that synergistic interactions may often be present in smoking cessation trials and should be taken into account at the study design phase.

    Conclusions

    In conclusion, this factorial screening experiment demonstrates the effectiveness of a theory-driven text message intervention in boosting overall engagement and use of the core features of an internet smoking cessation program among adult smokers. The results suggest that enabling users to engage with the tools and content of an internet intervention via text messages and tailoring the experience based on a user’s pattern of program use can boost the overall levels of engagement. These findings have relevance to improving engagement in internet health behavior change interventions more broadly and for future research into the complex relationship between engagement and outcomes. This study can serve as a model for conducting rigorous, fully powered research on engagement as a first step in understanding how to optimize behavior change outcomes in digital interventions.

    Acknowledgments

    This work was supported by the National Institute on Drug Abuse of the National Institutes of Health (R01DA038139). The sponsor was not involved in the conduct of this study or the preparation of this paper. The authors would like to acknowledge the contributions of Fiona Mortell and Monica-Ann Ortiguera Mendoza.

    Conflicts of Interest

    AG, MJ, MA, and SC are employees of Truth Initiative, a nonprofit public health foundation, which sells enterprise digital tobacco cessation programs to support its mission-driven work. All other authors declare no conflicts of interest.

    ‎
    Multimedia Appendix 1

    Simulation-Based Histograms of Arms-Specific Utilization Rates: Interactive Features.

    PDF File (Adobe PDF File), 44 KB
    ‎
    Multimedia Appendix 2

    Simulation-Based Histograms of Arms-Specific Utilization Rates: Key Engagement Metrics.

    PDF File (Adobe PDF File), 35 KB
    1. Pew Research Center. 2019 Jun 12. Internet/Broadband Fact Sheet, 2019   URL: http://www.pewinternet.org/fact-sheet/internet-broadband/ [accessed 2020-03-09]
    2. Graham AL, Amato MS. Twelve million smokers look online for smoking cessation help annually: health information national trends survey data, 2005-2017. Nicotine Tob Res 2019 Jan 4;21(2):249-252 [FREE Full text] [CrossRef] [Medline]
    3. McCausland K, Curry L, Mushro A, Carothers S, Xiao H, Vallone D. Promoting a Web-Based Smoking Cessation Intervention: Implications for Practice. Cases in Public Health Communication & Marketing. Proc 2011;5:3-26.
    4. Rudie M, Bailey L. North American Quitline Consortium. 2018. North American Quitline Consortium FY2018 Annual Survey: Progress Update on State Quitlines   URL: https://www.naquitline.org/resource/resmgr/2018_survey/NAQC_FY2018_Annual_Survey_Da.pdf [accessed 2020-03-09]
    5. Taylor GM, Dalili MN, Semwal M, Civljak M, Sheikh A, Car J. Internet-based interventions for smoking cessation. Cochrane Database Syst Rev 2017 Sep 4;9:CD007078 [FREE Full text] [CrossRef] [Medline]
    6. Kohl LF, Crutzen R, de Vries NK. Online prevention aimed at lifestyle behaviors: a systematic review of reviews. J Med Internet Res 2013 Jul 16;15(7):e146 [FREE Full text] [CrossRef] [Medline]
    7. Graham A, Papandonatos G, Cha S, Erar B, Amato M. Improving adherence to smoking cessation treatment: smoking outcomes in a web-based randomized trial. Ann Behav Med 2018 Mar 15;52(4):331-341 [FREE Full text] [CrossRef] [Medline]
    8. Graham AL, Zhao K, Papandonatos GD, Erar B, Wang X, Amato MS, et al. A prospective examination of online social network dynamics and smoking cessation. PLoS One 2017;12(8):e0183655 [FREE Full text] [CrossRef] [Medline]
    9. Danaher B, Smolkowski K, Seeley J, Severson H. Mediators of a successful web-based smokeless tobacco cessation program. Addiction 2008 Oct;103(10):1706-1712 [FREE Full text] [CrossRef] [Medline]
    10. Alkhaldi G, Hamilton FL, Lau R, Webster R, Michie S, Murray E. The effectiveness of prompts to promote engagement with digital interventions: a systematic review. J Med Internet Res 2016 Jan 8;18(1):e6 [FREE Full text] [CrossRef] [Medline]
    11. Perski O, Blandford A, West R, Michie S. Conceptualising engagement with digital behaviour change interventions: a systematic review using principles from critical interpretive synthesis. Transl Behav Med 2017 Jun;7(2):254-267 [FREE Full text] [CrossRef] [Medline]
    12. Brouwer W, Kroeze W, Crutzen R, de Nooijer J, de Vries NK, Brug J, et al. Which intervention characteristics are related to more exposure to internet-delivered healthy lifestyle promotion interventions? A systematic review. J Med Internet Res 2011 Jan 6;13(1):e2 [FREE Full text] [CrossRef] [Medline]
    13. Yardley L, Spring BJ, Riper H, Morrison LG, Crane DH, Curtis K, et al. Understanding and promoting effective engagement with digital behavior change interventions. Am J Prev Med 2016 Nov;51(5):833-842. [CrossRef] [Medline]
    14. Brouwer W, Oenema A, Crutzen R, de Nooijer J, de Vries NK, Brug J. An exploration of factors related to dissemination of and exposure to internet-delivered behavior change interventions aimed at adults: a Delphi study approach. J Med Internet Res 2008 Apr 16;10(2):e10 [FREE Full text] [CrossRef] [Medline]
    15. Whittaker R, McRobbie H, Bullen C, Rodgers A, Gu Y, Dobson R. Mobile phone text messaging and app-based interventions for smoking cessation. Cochrane Database Syst Rev 2019 Oct 22;10:CD006611. [CrossRef] [Medline]
    16. Fry JP, Neff RA. Periodic prompts and reminders in health promotion and health behavior interventions: systematic review. J Med Internet Res 2009 May 14;11(2):e16 [FREE Full text] [CrossRef] [Medline]
    17. van Dulmen S, Sluijs E, van Dijk L, de Ridder D, Heerdink R, Bensing J. Patient adherence to medical treatment: a review of reviews. BMC Health Serv Res 2007 Apr 17;7:55 [FREE Full text] [CrossRef] [Medline]
    18. Pew Research Center. 2019 Jun 12. Mobile Fact Sheet   URL: http://www.pewinternet.org/fact-sheet/mobile/ [accessed 2020-03-09]
    19. Hall AK, Cole-Lewis H, Bernhardt JM. Mobile text messaging for health: a systematic review of reviews. Annu Rev Public Health 2015 Mar 18;36:393-415 [FREE Full text] [CrossRef] [Medline]
    20. Collins LM, Murphy SA, Nair VN, Strecher VJ. A strategy for optimizing and evaluating behavioral interventions. Ann Behav Med 2005 Aug;30(1):65-73. [CrossRef] [Medline]
    21. Collins LM, Murphy SA, Strecher V. The multiphase optimization strategy (MOST) and the sequential multiple assignment randomized trial (SMART): new methods for more potent eHealth interventions. Am J Prev Med 2007 May;32(5 Suppl):S112-S118 [FREE Full text] [CrossRef] [Medline]
    22. Petty RE, Cacioppo JT, Schumann D. Central and peripheral routes to advertising effectiveness: the moderating role of involvement. J Comsum Res 1983;10(2):135-146. [CrossRef]
    23. Dijkstra A. Working mechanisms of computer-tailored health education: evidence from smoking cessation. Health Educ Res 2005 Oct;20(5):527-539. [CrossRef] [Medline]
    24. Webb MS, Simmons VN, Brandon TH. Tailored interventions for motivating smoking cessation: using placebo tailoring to examine the influence of expectancies and personalization. Health Psychol 2005 Mar;24(2):179-188. [CrossRef] [Medline]
    25. Jamison J, Naughton F, Gilbert H, Sutton S. Delivering smoking cessation support by mobile phone text message: what information do smokers want? A focus group study. J Appl Behav Res 2013;18(1):1-23. [CrossRef]
    26. Head KJ, Noar SM, Iannarino NT, Harrington NG. Efficacy of text messaging-based interventions for health promotion: a meta-analysis. Soc Sci Med 2013 Nov;97:41-48. [CrossRef] [Medline]
    27. Garcia Z. Chromis Technology. 2018 Feb 8. Text Messaging Stats That Matter   URL: https://www.chromis.com/text-stats-that-matter/ [accessed 2020-03-09]
    28. Webb TL, Joseph J, Yardley L, Michie S. Using the internet to promote health behavior change: a systematic review and meta-analysis of the impact of theoretical basis, use of behavior change techniques, and mode of delivery on efficacy. J Med Internet Res 2010 Feb 17;12(1):e4 [FREE Full text] [CrossRef] [Medline]
    29. Heminger CL, Boal AL, Zumer M, Abroms LC. Text2Quit: an analysis of participant engagement in the mobile smoking cessation program. Am J Drug Alcohol Abuse 2016 Jul;42(4):450-458. [CrossRef] [Medline]
    30. Milward J, Drummond C, Fincham-Campbell S, Deluca P. What makes online substance-use interventions engaging? A systematic review and narrative synthesis. Digit Health 2018;4:2055207617743354 [FREE Full text] [CrossRef] [Medline]
    31. Noar SM, Benac CN, Harris MS. Does tailoring matter? Meta-analytic review of tailored print health behavior change interventions. Psychol Bull 2007 Jul;133(4):673-693. [CrossRef] [Medline]
    32. Free C, Knight R, Robertson S, Whittaker R, Edwards P, Zhou W, et al. Smoking cessation support delivered via mobile phone text messaging (txt2stop): a single-blind, randomised trial. Lancet 2011 Jul 2;378(9785):49-55 [FREE Full text] [CrossRef] [Medline]
    33. Eysenbach G. The law of attrition. J Med Internet Res 2005 Mar 31;7(1):e11 [FREE Full text] [CrossRef] [Medline]
    34. Graham AL, Jacobs MA, Cohn AM, Cha S, Abroms LC, Papandonatos GD, et al. Optimising text messaging to improve adherence to web-based smoking cessation treatment: a randomised control trial protocol. BMJ Open 2016 Mar 30;6(3):e010687 [FREE Full text] [CrossRef] [Medline]
    35. Fiore M, Jaén C, Baker T, Tobacco Use and Dependence Guideline Panel. Treating Tobacco Use and Dependence: 2008 Update. Rockville, MD: US Department of Health and Human Services; 2008.
    36. Bandura A. Social Foundations of Thought and Action: A Social Cognitive Theory. Englewood Cliffs, NJ: Prentice-Hall; 1986.
    37. Mayo Clinic. Mayo clinic-mayo foundation courses and meetings to be held in Rochester, MN, 1984. Mayo Clinic Proceedings 1984 Jul;59(7):516 [FREE Full text] [CrossRef]
    38. Richardson A, Graham AL, Cobb N, Xiao H, Mushro A, Abrams D, et al. Engagement promotes abstinence in a web-based cessation intervention: cohort study. J Med Internet Res 2013 Jan 28;15(1):e14 [FREE Full text] [CrossRef] [Medline]
    39. Schubart JR, Stuckey HL, Ganeshamoorthy A, Sciamanna CN. Chronic health conditions and internet behavioral interventions: a review of factors to enhance user engagement. Comput Inform Nurs 2011 Feb;29(2 Suppl):TC9-T20. [CrossRef] [Medline]
    40. Alkhaldi G, Hamilton FL, Lau R, Webster R, Michie S, Murray E. The effectiveness of technology-based strategies to promote engagement with digital interventions: a systematic review protocol. JMIR Res Protoc 2015 Apr 28;4(2):e47 [FREE Full text] [CrossRef] [Medline]
    41. Cumming G, Finch S. Inference by eye: confidence intervals and how to read pictures of data. Am Psychol 2005;60(2):170-180. [CrossRef] [Medline]
    42. Goldstein H, Healy MJ. The graphical presentation of a collection of means. J R Stat Soc Ser A Stat Soc 1995;158(1):175-177. [CrossRef]
    43. Efron B, Tibshirani R. An Introduction to the Bootstrap. London, UK: Chapman & Hall; 1993.
    44. Cohen J. Statistical Power Analysis for the Behavioral Sciences. Second Edition. Hillsdale, NJ: Erlbaum; 1988.
    45. Dijkstra A, Ballast K. Personalization and perceived personal relevance in computer-tailored persuasion in smoking cessation. Br J Health Psychol 2012 Feb;17(1):60-73. [CrossRef] [Medline]
    46. Graham AL, Papandonatos GD, Zhao K. The failure to increase social support: it just might be time to stop intervening (and start rigorously observing). Transl Behav Med 2017 Dec;7(4):816-820 [FREE Full text] [CrossRef] [Medline]
    47. Graham AL, Papandonatos GD, Cha S, Erar B, Amato MS, Cobb NK, et al. Improving adherence to smoking cessation treatment: intervention effects in a web-based randomized trial. Nicotine Tob Res 2017 Mar 1;19(3):324-332 [FREE Full text] [CrossRef] [Medline]
    48. Cutrona SL, Sadasivam RS, DeLaughter K, Kamberi A, Volkman JE, Cobb N, National Dental PBRN Collaborative Group comprises practitioners‚ faculty and staff who contributed to this activity. A list of these persons is at http://nationaldentalpbrn.org/publication.php. Online tobacco websites and online communities-who uses them and do users quit smoking? The quit-primo and national dental practice-based research network Hi-Quit studies. Transl Behav Med 2016 Dec;6(4):546-557 [FREE Full text] [CrossRef] [Medline]
    49. Christensen H, Griffiths K, Groves C, Korten A. Free range users and one hit wonders: community users of an internet-based cognitive behaviour therapy program. Aust N Z J Psychiatry 2006 Jan;40(1):59-62. [CrossRef] [Medline]
    50. Saul JE, Amato MS, Cha S, Graham AL. Engagement and attrition in internet smoking cessation interventions: Insights from a cross-sectional survey of 'one-hit-wonders'. Internet Interv 2016 Sep;5:23-29 [FREE Full text] [CrossRef] [Medline]
    51. McClure JB, Shortreed SM, Bogart A, Derry H, Riggs K, St John J, et al. The effect of program design on engagement with an internet-based smoking intervention: randomized factorial trial. J Med Internet Res 2013 Mar 25;15(3):e69 [FREE Full text] [CrossRef] [Medline]
    52. McClure J, Peterson D, Derry H, Riggs K, Saint-Johnson J, Nair V, et al. Exploring the 'active ingredients' of an online smoking intervention: a randomized factorial trial. Nicotine Tob Res 2014 Aug;16(8):1129-1139 [FREE Full text] [CrossRef] [Medline]
    53. The Community Guide. 2011. Tobacco Use and Secondhand Smoke Exposure: Mobile Phone-Based Cessation Interventions   URL: https:/​/www.​thecommunityguide.org/​findings/​tobacco-use-and-secondhand-smoke-exposure-mobile-phone-based-cessation-interventions [accessed 2020-03-09]
    54. Perez S. Tech Crunch. 2016 May 31. Nearly 1 in 4 People Abandon Mobile Apps After Only One Use   URL: https://techcrunch.com/2016/05/31/nearly-1-in-4-people-abandon-mobile-apps-after-only-one-use/ [accessed 2020-03-09]
    55. Boyles J, Smith A, Madden M. Pew Research Center. 2012 Sep 5. Privacy and Data Management on Mobile Devices   URL: https://www.pewresearch.org/internet/2012/09/05/privacy-and-data-management-on-mobile-devices/ [accessed 2020-03-09]
    56. Augustson E, Cole-Lewis H, Sanders A, Schwarz M, Geng Y, Coa K, et al. Analysing user-reported data for enhancement of SmokefreeTXT: a national text message smoking cessation intervention. Tob Control 2017 Nov;26(6):683-689. [CrossRef] [Medline]
    57. Cook JW, Collins LM, Fiore MC, Smith SS, Fraser D, Bolt DM, et al. Comparative effectiveness of motivation phase intervention components for use with smokers unwilling to quit: a factorial screening experiment. Addiction 2016 Jan;111(1):117-128 [FREE Full text] [CrossRef] [Medline]

    ‎
    EX: BecomeAnEX
    MOST: Multiphase Optimization Strategy
    SMD: standardized mean difference

    Edited by G Eysenbach; submitted 12.01.20; peer-reviewed by V Do, O Perski; comments to author 31.01.20; revised version received 09.02.20; accepted 22.02.20; published 02.04.20

    Copyright

    ©Amanda L L Graham, George D Papandonatos, Megan A Jacobs, Michael S Amato, Sarah Cha, Amy M Cohn, Lorien C Abroms, Robyn Whittaker. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 02.04.2020.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.