This paper is in the following e-collection/theme issue:

Tutorial (60)

Published on in Vol 8, No 3 (2006):

Defining Participant Exposure Measures in Web-Based Health Behavior Change Programs

Defining Participant Exposure Measures in Web-Based Health Behavior Change Programs

Defining Participant Exposure Measures in Web-Based Health Behavior Change Programs

Authors of this article:

Brian G Danaher ;   Shawn M Boles ;   Laura Akers ;   Judith S Gordon ;   Herbert H Severson
Article Authors Cited by (111) Tweetations Metrics

    Original Paper

    Oregon Research Institute, Eugene, OR, USA

    Corresponding Author:

    Brian G Danaher, PhD

    Oregon Research Institute

    1715 Franklin Blvd

    Eugene, OR 97403

    USA

    Phone: +1 541 484 2123

    Fax:+1 541 484 1108

    Email: briand@ori.org


    Background: Published research on the use of Web-based behavior change programs is growing rapidly. One of the observations characterized as problematic in these studies is that participants often make relatively few website visits and spend only a brief time accessing the program. Properly structured websites permit the unobtrusive measurement of the ways in which participants access (are exposed to) program content. Research on participant exposure to Web-based programs is not merely of interest to technologists, but represents an important opportunity to better understand the broader theme of program engagement and to guide the development of more effective interventions.

    Objectives: The current paper seeks to provide working definitions and describe initial patterns of various measures of participant exposure to ChewFree.com, a large randomized controlled trial of a Web-based program for smokeless tobacco cessation.

    Methods: We examined measures of participant exposure to either an Enhanced condition Web-based program (interactive, tailored, and rich-media program) or a Basic condition control website (static, text-based material). Specific measures focused on email prompting, participant visits (number, duration, and pattern of use over time), and Web page viewing (number of views, types of pages viewed, and Web forum postings).

    Results: Participants in the ChewFree.com Enhanced condition made more visits and spent more time accessing their assigned website than did participants assigned to the Basic condition website. In addition, exposure data demonstrated that Basic condition users thoroughly accessed program content, indicating that the condition provided a meaningful, face-valid control to the Enhanced condition.

    Conclusions: We recommend that researchers conducting evaluations of Web-based interventions consider the collection and analysis of exposure measures in the broader context of program engagement in order to assess whether participants obtain sufficient exposure to relevant program content.

    J Med Internet Res 2006;8(3):e15

    doi:10.2196/jmir.8.3.e15

    Keywords

    Health behavior; Internet; Web; behavioral research; participant exposure; engagement 


    One of the common findings of research on Web-based behavior change programs is that participants spend only a relatively meager amount of time accessing their online intervention [1]. This implies minimal participant exposure to the critical behavior change ingredients of the program, which could potentially reduce program impact. In response to this finding, a number of published reports of Web-based interventions have described website usage statistics, including number and duration of visits as well as the number and type of Web pages viewed [2-11]. Research on Web-viewing behavior is rapidly growing in other domains (eg, advertising [12] and technology [13]).

    This paper describes participant exposure to a two-arm randomized controlled trial of Web-based programs designed to assist adults in quitting smokeless tobacco (either snuff or chewing tobacco). Following a brief program description, we present a set of unobtrusive measures of website exposure [14] and the results of our exposure analyses. We believe that this level of detail will prove helpful to other researchers investigating the design of optimally effective Web-based behavior change programs.


    ChewFree Program for Smokeless Tobacco Cessation

    We designed the ChewFree trial to compare the efficacy of two smokeless tobacco cessation websites: Basic and Enhanced.The Basic condition, which represented a subset of the content presented in the Enhanced condition, offered a printable self-help smokeless tobacco cessation booklet, printable cessation resources (eg, describing the use of herbal snuff products, nicotine replacement products), and annotated links to other recommended websites for tobacco cessation. The Enhanced condition offered a tailored and interactive Web-based program that included text-based information (health and behavioral strategies focused on quitting and preventing relapse), video-based testimonials, printable resources, interactive activities, annotated links to other website resources, and two Web forums (a "Talk with Others" social support forum, and an "Ask an Expert" forum for submitting questions to project staff).

    ChewFree.com intervention components were based largely on Bandura's Social Cognitive Theory [15-17] in which individuals are viewed as proactive agents who can exercise motivational and self-regulatory skills to change their health habits. According to this theory, individuals choose their environments, seek out beneficial social networks, and engage in other self-management behaviors that allow them to achieve both initial change and long-term maintenance. Multi-component smoking cessation and relapse-prevention interventions have successfully incorporated these strategies [eg, 7,18-20], and adaptations of these same approaches have been found well suited to smokeless tobacco cessation programs [21-23].

    The Basic and Enhanced Web-based programs offered smokeless tobacco cessation assistance using markedly different information architectures [24]. The Basic condition (Figure 1) presented text-based content using four navigational Web pages: Home, Enough Snuff—an adaptation of the smokeless tobacco cessation manual used in prior research [25], Resources, and Links. The Enhanced condition (Figure 2) used five navigational Web pages: Home, Personal Quitting Assistant, Resources, Forum, and Links. The Personal Quitting Assistant used a hybrid information architecture design [24] that guided participants in a step-wise manner through eight modules of the Planning to Quit content while offering optional content along the way. In addition, the information architecture prevented users from accessing content in the Staying Quit module until they returned to the website at a later date and reported that they had either quit using smokeless tobacco or had relapsed. Progress was self-paced in that participants in both the Enhanced and Basic conditions determined when they chose to visit the program and how much content they would cover during each visit.

    ‎
    Figure 1. Basic condition (excerpt of Enough Snuff guide)
    View this figure
    ‎
    Figure 2. Enhanced condition (excerpt showing video narration by a smokeless tobacco cessation expert to accompany the Personal Quitting Assistant, or PQA)
    View this figure

    Content was presented using text, graphics, activities, and two types of videos: a video expert guide who narrated key portions of content that was also presented as text, and video testimonials of smokeless tobacco users whose presentations supported the recommendations of the program. The narration videos were automatically launched (ie, they did not require user selection) for users with high bandwidth connections, but they were not displayed automatically to participants with dial-up access [26]; all users could toggle them on or off as desired.

    We designed the Enhanced condition to be attractive by offering a broad spectrum of content tailored to the interests and the smokeless tobacco use/abstinence status of each participant. For example, participants who were preparing to quit were encouraged to review program content focused on Planning to Quit, whereas those participants who indicated that they had quit using smokeless tobacco were encouraged to review content on Staying Quit. In addition, the intervention used multiple methods for delivering content along with engaging activities. Compared to the Basic condition, we predicted that the Enhanced condition would encourage participants to visit more often and for longer periods of time—especially during the first several weeks post-enrollment when attempts to quit and related lapse/relapse experiences would most likely occur.

    Participants

    Participants were recruited using a multifaceted marketing campaign that included (1) thematic promotional "releases" to print and broadcast media, (2) Google ads, (3) placement of a link on other websites, (4) limited purchase of paid advertising, (5) direct mailings to smokeless tobacco users, and (6) targeted mailings to health care and tobacco control professionals. This campaign resulted in more than 23500 visits to the ChewFree.com recruitment website from distinct IP addresses over a 1-year period, which yielded 2523 eligible smokeless tobacco users who completed the registration process and enrolled in the ChewFree.com smokeless tobacco cessation research project [27,28]. Participants were randomly assigned either to the Enhanced condition (n = 1260) or the Basic condition (n = 1263).

    Measures of Program Exposure

    There is no single universally accepted measure for assessing participant exposure to a Web-based program. Computer-delivered content lends itself well to unobtrusive monitoring of usage patterns. As noted by Peterson [29], there are a number of potentially complementary sources of computer-based monitoring data, such as server log files, cookies, Web beacons, and session identifiers. Many commercial products are available that analyze Web server log files [eg, 30,31]. Cookies offer another powerful tool to tailor website content and monitor usage [32]. Web beacons can be inserted into Web pages to enhance the ability to obtain even more detailed tracking [33]. Because we used user authentication (obtaining username and password at the beginning of every session) with an appropriate scripting language (Macromedia ColdFusion) and SQL database to create the ChewFree websites, we were able to use the session identifier approach to measure exposure [29]. We believe that the session identifier approach offered more flexibility to focus on topics that were relevant to our research than did a commercial log analyzer product focused on issues of commercial importance such as pay-for-click analyses, average revenue per order, top products, and customer segment analysis.

    For the present paper, we examined data from participants as of January 10, 2006, which, for most participants, represented approximately 12 months after enrollment (mean = 367.1 days, SD = 116.9; no significant differences between conditions). The minimum number of days since enrollment was 181 days and, in each case, the scheduled date of the 6-month follow-up assessment had elapsed. Textbox 1 summarizes the measures we used, each of which is described below. Detailed measures are provided in the Multimedia Appendix.

    Measures of program exposure
    Table .
    Email promptsPercentage of participants sent treatment-related email prompts
    Participant visitsNumber of visits
    Aggregate duration of visits
    Number of daily visits post-enrollment
    Number of days of program access post-enrollment
    Web page viewsOverall number of Web page views
    Specific Web page views (selected smokeless tobacco cessation content)
    Web forum postings
    Textbox 1. Measures of program exposure

    Treatment-Related Email Prompts

    Participants in the Enhanced condition received a variety of email prompts during the study that were not related to assessments. These prompts fell into three categories:

    1. Intervention: Participants were sent up to three email messages prior to their quit date, tailored to their chosen method of quitting (cold turkey, nicotine fading, brand switching, blending), and one message on their quit date.

    2. Support: We sent three supportive emails timed at fixed intervals after the participant's self-reported quit date.

    3. Re-engagement: Participants who failed to log in on a regular basis were typically sent multiple tailored email messages encouraging them to resume accessing the program.

    Participant Visits

    Typical measures of visit data include number of visits per participant per condition and both average and total visit duration. We programmed the ChewFree website to record the date/time stamp of the start and end of each participant visit (also referred to as a "session") and for each Web page the participant viewed during each visit. These date/time stamps allowed us to examine both the number of unique visits per participant and session duration.

    Because participants were able to abruptly end their use of the program by closing their browser window, there were occasions when we did not capture the date/time data for the end of the session. To analyze these instances, we conservatively approximated the end of the visit by using the date/time of the last Web page that had been accessed before the abrupt end of the session. In addition, we followed the operational definition for visit expiration recommended by Peterson [29]; that is, any Web page viewed for 30 minutes or more was defined as having ended the visit using the ending date/time stamp of the Web page that immediately preceded the hiatus. Moreover, if, after the hiatus, the participant resumed activity, it was considered to be a new visit for measurement purposes.

    Participants in both conditions were required to complete an online baseline assessment prior to accessing the program. In addition, all participants received email reminders to complete online follow-up assessments at 6 weeks, 3 months, and 6 months. The email prompt contained a link that caused the log-in page of the Web-based program to appear, followed by presentation of the online assessment. At the end of each assessment, users were returned to their respective website, at which time they were free to explore the website and review its contents. When counting distinct visits that involved program content review, we excluded those visits associated with online assessments unless the participant also explored website content.

    Website Visit Duration

    We focused our analysis on aggregate duration (collapsed across visits) because we were concerned with the overall amount of participant exposure to the program. Although we did not choose to do so for purposes of this paper, we could also have examined the changing patterns in the duration of individual visits over time.

    Visits Following Enrollment

    We examined the time course for each participant visit by calculating the number of days in which a visit occurred since the date the participant completed the baseline assessment and formally began the study. It is important to note that at the end of the baseline assessment, each participant was automatically presented with the home page of the condition to which he/she was randomly assigned. If, following the end of the assessment, a participant continued to explore the Web-based content, then that event was counted as a unique visit and assigned a value of zero (since zero days had elapsed since the end of the baseline assessment). If a participant had multiple visits on any given day, then this analysis counted each of those visits in the total for that day (ie, participants could have multiple visits per day). We limited our analysis to those visits in which Web-based program content pages were accessed.

    In addition to measuring the number of visits per day, we used Kaplan-Meier survival analyses [34-36] to examine the pattern of reduced program participation, also known as nonusage attrition[1]. For purposes of this analysis, each participant's last visit that involved review of program content was designated as the date that program usage ended. Duration was defined as the number of days that elapsed since program enrollment (the start of the program) and the date of the last visit. More technically, the population survivor function represents time versus the probability that a randomly selected program participant will continue to access the program. Since all participants stopped using their assigned website in the analysis period (the defined terminal event), no cases were censored. In addition to examining the survival curve, we also report on the estimated median lifetime for each condition, which describes how much time passed before 50% of the sample stops accessing the Web-based program [34].

    Viewing Smokeless Tobacco Cessation Content

    Finer grained within-visit analyses focused on participants' viewing of Web pages that presented specific content designed to encourage smokeless tobacco cessation. Because we recorded the date/time of each Web page viewed during each visit, it was possible to calculate the percentage of participants who viewed specific types of Web pages (using the participant sample in each condition as the denominator). For example, we were able to measure the extent to which participants in either condition accessed a ChewFree.com Web page that provided links to other websites offering smokeless tobacco cessation information and assistance (eg, the National Cancer Institute, the National Spit Tobacco Education Program, and the Oral Health America Foundation).

    In the Enhanced condition, we also measured participants' use of ChewFree.com Web pages that offered more interactive features, including whether they viewed pages that automatically played video testimonials, whether they accessed a Web page that offered a print feature (and triggered a print dialog box), and whether they listed the names of people whom they believed could offer useful support for smokeless tobacco cessation. And although this paper focuses on exposure rather than on outcome results, we also report on the extent to which participants in the Enhanced condition used the Web page designed to help them choose a quit date for stopping the use of smokeless tobacco.

    Web Forum Data

    Finally, we captured data on the extent to which participants in the Enhanced condition used the available peer Web forum ("Talk with Others") or expert forum ("Ask an Expert"). Forum use was logged into the database when participants posted messages, either by creating a new message or responding to an existing message. Unfortunately, we did not track passive viewing of the forum messages, nor did we collect data that would allow us to calculate the amount of time spent viewing forum content.


    Treatment-Related Email Prompts

    Analysis revealed that 63.3% of participants (760/1220) in the Enhanced condition set a quit date and were sent a program-generated series of tailored email prompts associated with preparing to quit. After having been sent at least one of these emails, 10.7% of these participants (81/760) requested to opt out of receiving further emails. A total of 40.7% of participants (488/1220) who reported having quit using smokeless tobacco during the course of the program were eligible to be sent a series of emails supportive of continued abstinence. However, the number of participants who were sent these supportive emails was reduced to 34.8% (425/1220) because 63 had opted out of receiving program-generated emails. Enhanced condition participants who had not exercised the opt-out option (90%; 1079/1200) were also scheduled to receive emails at 7, 30, and 60 days since last log-in, encouraging them to re-engage with the site. We plan to conduct future analyses to assess the relation between the automated email prompts, website usage, and outcome results.

    Number and Duration of Unique Visits

    Our initial analysis showed that 0.6% of participants (7/1260) in the Enhanced condition and 0.8% of participants (10/1263) in the Basic condition never visited their assigned website after completing the baseline assessment and becoming enrolled. An additional 3.7% of participants (47/1260) in the Enhanced condition and 5.9% of participants (74/1263) in the Basic condition returned following enrollment but did so only to complete online assessments. These individuals never viewed any Web pages that contained smokeless tobacco cessation content. Removing these participants from our analyses reduced the sample to 2375 participants (1200 in Enhanced condition; 1175 in Basic condition) for whom visit duration could be measured (Table 1).

    Rather than being normally distributed, the observed patterns of website visit frequency and duration displayed a significantly positive- or right-skewed distribution, with most cases having occurred at lower values (more frequent and longer visits occurring soon after enrollment). We used the nonparametric Mann-Whitney U test to compare these results by condition. Participants in the Enhanced condition made significantly more visits than participants in the Basic condition (z = -16.64, P < .001, 2-tailed). We also calculated the length of each visit by summing the length of each page view within each visit. Participants in the Enhanced condition spent significantly more time viewing website content collapsed across all Web pages and visits (z = -17.63, P < .001, 2-tailed).

    Table 1. Visit details by participant
    Visits
    by Participant*    		Overall Visit Duration
    by Participant*(min)
    ConditionNMedianInterquartile
    Range    		MedianInterquartile
    Range
    Enhanced12002.003
    (1-4)    		28.9937.75
    (13.60-51.35)
    Basic11751.001
    (1-2)    		12.5015.83
    (6.60-22.43)

    *P < .001

    Visits Following Enrollment

    Visits by time course for those 2375 participants who viewed smokeless tobacco cessation content are depicted in Figure 3. Note that if a participant only viewed website content on the day of his/her enrollment, then that individual would be listed in this analysis as having 0 days (zero days since the day of enrollment). In this analysis, a participant could have multiple visits in any given day.

    ‎
    Figure 3. Visits following enrollment
    View this figure

    We observed 3783 visits for participants in the Enhanced condition and 2054 visits in the Basic condition. Consistent with usage patterns reported in other research of Web-based interventions [1], participants in our study visited their assigned website more frequently and in greater numbers immediately following enrollment. Thereafter, we observed a steady decrease in visits over time with rapid drops occurring soon after enrollment followed by a slower reduction in visits toward zero asymptote. Even though the analysis did not include all visits that focused only on online follow-up assessments, it nonetheless appears that visits for program content were related to the timing of online assessments and their reminder emails (note vertical markers for the assessments at 6 weeks, 3 months, and 6 months) such that assessment dates appeared to reduce the rate (the steepness of the curve) of declining visits. It is important to note in this regard that upon completion of the online assessment, each participant was returned to the website home page, which would encourage them to review program content.

    We also examined the number of days following enrollment that participants continued to access their assigned website for program content (excluding visits to take online assessment only). For purposes of this survival analysis, the last content-accessing visit for each participant was designated as the final date of program usage. For example, 36.1% of participants (433/1200) in the Enhanced condition and 60.7% (713/1175) in the Basic condition stopped using the program on the day they enrolled in the program. Because Figure 4 depicts the percentage of participants who continued to use the program (the "survivors"), it shows that 63.9% of participants in the Enhanced condition and 39.3% of participants in the Basic condition continued to use the program after Day 0 (enrollment day).

    We assumed that each participant, regardless of condition, would eventually stop using the Web-based program. Thus we examined the differential pattern of program use atrophy. As depicted in Figure 4, website access essentially stopped by 6 months following program enrollment. The estimated median lifetime website usage (date when 50% of participants stopped using the program) was 11 days for the Enhanced condition and 0 days (ie, the enrollment day) for the Basic condition. A Kaplan-Meier survival analysis indicated that, following enrollment, participants in the Enhanced and Basic conditions exhibited significantly different patterns of continued access to the Web-based program. Both log-rank (Mantel-Cox) and Breslow (generalized Wilcoxon) tests were highly significant (P < .001), with the Enhanced condition showing a slower decay (less nonusage attrition) over time than the Basic condition. As noted in the analysis of total visits following enrollment (Figure 3), we observed that reduced program usage was related to the prompting effects of the follow-up assessments at 6 weeks, 3 months, and 6 months.

    ‎
    Figure 4. Website activity following enrollment (survival curve)
    View this figure

    Viewing Web Pages With Smokeless Tobacco Cessation Advice

    In addition to metrics of overall website exposure, we were interested in the extent to which participants accessed content that contained specific information most relevant to smokeless tobacco cessation and tobacco abstinence. Table 2 displays data on the viewing of selected Web pages that contained information on smokeless tobacco cessation. It is interesting to note that when similar pages were available on both websites (those presenting outside links and the opportunity to print content), a higher percentage of participants in the Basic condition accessed that content than did participants in the Enhanced condition. Similarly, almost 88% of participants in the Basic condition compared with 12.2% of participants in the Enhanced condition viewed every page in an 11-page self-help smokeless tobacco quitting guide. This guide, adapted from the Enough Snuff guide [37], was deeply embedded in the Enhanced condition website, thus making it somewhat more difficult to use.

    Table 2. Web page viewing by participants who accessed at least one Web page containing smokeless tobacco cessation content
    ConditionNOutside Links (%)Print Content (%)SmokelessTobaccoQuitting Guide (%)Video Testimonial (%)List of Support People (%)Set Quit Date (%)Staying Quit Content After Quitting (%)
    Enhanced 1200 18.2 78.3 12.2 68.2 24.7 63.3 32.8
    Basic 1175 32.1 96.3 87.5

    Web Forum Usage

    We found that 38.2% participants in the Enhanced condition (481/1260) posted content to the Web forum for peers, with 5.2% (65/1260) posting at least one message in the expert forum (Table 3). Each participant who posted a message to the "Ask an Expert" forum also posted at least one message to the peer forum. The distribution of forum postings was right- or positively-skewed, with most cases clustered at lower values (fewer postings). The median number of postings was 2 in the peer forum and 1 in the expert forum. The interquartile range was 11 postings (1-11.50) for the peer forum and 1 posting (1-2) for the expert forum. Using the nonparametric Spearman rank correlation test, we found that forum postings were significantly correlated with visits (ρ = .512, n = 481, P < .001, 2-tailed) and Web page views (ρ = .340, n = 481, P < .001, 2-tailed) for participants in the Enhanced condition.

    Table 3. Web forum activity in the Enhanced condition (n = 1260 users)
    UsersPostings by User
    Forum Typen%MedianInterquartile
    Range
    Peer48138.2211
    (1-11.50)
    Expert655.211
    (1-2)

    It is important to acknowledge several limitations to the present study. First, we did not design the Enhanced intervention website to track passive Web forum viewing. This limitation prevented us from analyzing the duration of Web forum visits by participants who observed postings but did not post their own comment on the forum posts of others. In addition, although study inclusion criteria required all study participants to be able to access their personal email at least once per week, we did not collect data on participants' previous experience using the Internet or on their computer self-efficacy [38]. As a result, we cannot report directly on whether there were significant differences between the intervention and control conditions for these dimensions. We believe that random assignment and our large sample size (N = 2375) would tend to mitigate the likelihood of this effect.

    It is thought that a key ingredient in determining the impact of any Web-based behavior change program is the extent to which participants are exposed to the program. This assumption is consistent with the finding that the efficacy and intensity of treatment programs tend to be positively related. For example, research on smoking cessation interventions—including self-help approaches—has illustrated the relationship between abstinence rates and program intensity, typically defined as contact time and number of sessions [4,39,40]. Williams et al [41] have coined the term program thickness to refer to the "collective intensity, duration, delivery agent, and intervention modality" of an intervention. However, research has also shown that more is not always better when considering which ingredients to include in an intervention [42] or a website [19,43,44], perhaps because adding features increases the response cost of participation and reduces usage.

    Some reviewers of this burgeoning field have recommended that fuller participation in Web-based interventions might be encouraged through the use of a "warm-up period" during which users can demonstrate their commitment by complying with precursor tasks while they become more familiar with what will be asked of them during the course of the program [1,45]. The use of intensive treatment approaches and preliminary litmus tests of commitment must be tempered by recognizing the continuum between clinic-based and public-health models for intervention. Specifically, it may be not be a practical goal to provide a highly intensive, population-wide intervention. Moreover, the use of preliminary barriers may help to reduce attrition in efficacy trials but reduce our ability to conduct effectiveness trials that have a broader reach and greater potential to achieve public health impact [46,47]. A challenging—and fruitful—line of research lies in identifying the proper program ingredients that provide a balance between sufficient exposure to relevant content and structure on the one hand while encouraging widespread user participation and engagement (both recruitment and follow-through).

    We found that the estimated median lifetime website usage (date when 50% of participants stopped using the program) was 11 days for the Enhanced condition and 0 days (ie, the enrollment day) for the Basic condition. We anticipate that some measures of exposure and outcome will likely share a curvilinear (inverted U-shaped) relationship such that those individuals who are least ready to make a meaningful change may be more likely to visit the Web-based program for a short time, while participants who are most prepared to change their behavior may similarly choose to visit the Web-based program for a relatively short time. Those participants who are interested in quitting and decide to learn more about how to do so will spend relatively more time visiting the program. It remains for future research to differentiate characteristics that illuminate the pattern of this relationship among motivation, readiness to quit, and program usage.

    Measures of participant exposure can help researchers and program developers determine the extent to which content is viewed. These data can point to needed changes in the information architecture and design features of the website. It is reasonable to assume that program content cannot be helpful if it is never viewed. Exposure measures may have utility in that they inform us about whether certain content—or clusters of content—is related to outcome and thus might be considered to be active ingredients in accomplishing the desired behavioral goals. They enable us to better understand idiosyncratic patterns of program use, highlighting ways we can adapt program structure and content to better accommodate (be tailored to) individual differences in participant interests, needs, and learning styles.

    In this regard, we intend to examine a variety of relationships between and among measures of exposure and the smokeless tobacco and tobacco cessation outcome measures in the ChewFree.com research project. For example, we will test whether participants who set a quit date are more successful in quitting, as well as whether, after quitting, there is a relationship between accessing content from the Staying Quit module (number and duration of visits) and lasting abstinence. Similarly, we will examine whether those participants who spend more time reviewing program content after they have lapsed are better able to regain control over their behavior and regain abstinence. We also plan to perform content and text analyses of Web forum postings [eg, 48] to explore whether smokeless tobacco cessation might be related to message types, whether cessation and maintenance strategies shared in postings are consistent with program recommendations, and the extent to which postings convey differing levels of confidence and self-efficacy across participants as well as within participants over time.

    There is a significant risk of collecting so much detailed exposure and engagement data that the task of analyzing and interpreting results becomes difficult. We suggest that this task can become more manageable and, thus, more fruitful, by focusing its scope through the use of a rationale that incorporates both theory and pragmatism. Potentially relevant rationales are not difficult to identify. Consider, for example, a rationale that builds on the Web foraging model [49,50], which posits that Web users guide their review of online content by quickly identifying interesting information scents in website materials. This model suggests that websites should foreshadow content even when it is not immediately accessible in order to engage users. It also points to particular usage patterns—brief initial visits followed by later visits of more duration [49]. The Transtheoretical/Stages of Change model may also hold promise in focusing the analysis of exposure and engagement. Velicer et al [51] suggest that users in action stage will access a program relatively more than users characterized as being in early stages (precontemplation, contemplation) or the later maintenance stage. Similarly, it might be helpful to consider the behavioral self-management model [17,52,53], which suggests that users who become more confident and capable in their self-management skills would tend to access a program less over time.

    We view exposure as representing one of a set of complementary measures of the broader theme of program engagement. Other engagement measures include participant comprehension of program content, practice of that content (especially in the participant's everyday routines outside of interacting with the Web-based program), self-reported satisfaction with the function and content of the website, and measures of self-efficacy. While exposure is obviously important (indeed, it is best viewed as a prerequisite), it represents only one piece of the puzzle in seeking to understand program effectiveness.

    Acknowledgments

    Thanks to Edward Lichtenstein, John Noell, and John Seeley for their careful reviews of this paper. We also acknowledge the important contribution of Steven Christiansen and Tim Woolley of InterVision (Eugene, OR) in the development of the ChewFree.com Web-based programs. This project was funded by a grant from the National Cancer Institute (R01-CA84225).

    Conflicts of Interest

    None declared.

    ‎

    Appendix 1

    Table 4. Table of detailed measures of program exposure
    Email promptsa) Percentage of participants receiving email promptsProgrammatic rules defined the timing of email prompts regarding completion of online assessments that were sent to participants in both conditions. Participants in the Enhanced condition also were sent "treatment-related" email prompts that contained tailored content related to quitting plan, quit date, and support of continued abstinence.
    Participant visitsb) Number of visitsVisits by each participant to access smokeless tobacco cessation content on their assigned website were counted. All visits in which only online assessments were accessed were excluded.
    c) Aggregate duration of visitsDuration of each visit was defined as the sum of Web page durations during that visit. With one exception (noted next), Web page durations were defined as their logged end time minus start time. Because our program logic did not include a session-expiration feature that automatically logged out after a period of inactivity, we conservatively approximated the end of the visit by using the date/time of the last Web page that had been accessed before the abrupt end of the session. In addition, we followed the operational definition for visit expiration recommended by Peterson; that is, any Web page viewed for 30 minutes or more was defined as having ended the visit using the ending date/time stamp of the Web page that immediately preceded the hiatus. Moreover, if, after the hiatus, the participant resumed activity, it was considered to be a new visit for measurement purposes.
    d) Number of daily visits post-enrollmentThe number of daily visits per participant was counted with "days" being defined in terms of their occurrence relative to the participant enrollment date. Participants could have more than one visit per day, and visits were defined using Peterson\'s recommendation (see above). Total visits per post-enrollment date aggregated these data across participants.
    e) Number of days of program access post-enrollmentThe number of days of post-enrollment access to smokeless tobacco cessation content was defined for each participant as the last visit date minus the enrollment date.
    f) Plan to QuitParticipants in the Enhanced condition were given the opportunity to define a personal quitting plan and quit date. Although the program allowed participants to define their quitting plan and date multiple times, for purposes of the analyses in this report, we focused only on the first recorded date when a participant defined his/her quitting plan/date.
    g) Smokeless tobacco quit statusParticipants in the Enhanced condition were able to indicate that they had quit using smokeless tobacco. This report enabled them to access "Staying Quit" content.
    Web page viewsh) Overall number of Web page viewsThe total number of Web pages viewed was logged.
    i) Specific page views (selected smokeless tobacco cessation content)The total number of selected Web pages viewed related to smokeless tobacco cessation was logged, and a subset of these is described in this report.
    j) Web forum postingsIndividual forum postings—in each of the two forums—were logged for each participant (Enhanced condition only).
    1. Eysenbach G. The law of attrition. J Med Internet Res 2005 Mar 31;7(1):e11 [FREE Full text] [Medline] [CrossRef]
    2. Feil EG, Noell J, Lichtenstein E, Boles SM, McKay HG. Evaluation of an Internet-based smoking cessation program: lessons learned from a pilot study. Nicotine Tob Res 2003 Apr;5(2):189-194. [Medline]
    3. Clarke G, Eubanks D, Reid E, Kelleher C, O'Connor E, Debar LL, et al. Overcoming Depression on the Internet (ODIN) (2): a randomized trial of a self-help depression skills program with reminders. J Med Internet Res 2005;7(2):e16 [FREE Full text] [Medline] [CrossRef]
    4. Cobb NK, Graham AL, Bock BC, Papandonatos G, Abrams DB. Initial evaluation of a real-world Internet smoking cessation system. Nicotine Tob Res 2005 Apr;7(2):207-216. [Medline] [CrossRef]
    5. Lenert L, Muñoz RF, Stoddard J, Delucchi K, Bansod A, Skoczen S, et al. Design and pilot evaluation of an internet smoking cessation program. J Am Med Inform Assoc 2003 Jan;10(1):16-20 [FREE Full text] [Medline] [CrossRef]
    6. Lenert L, Muñoz RF, Perez JE, Bansod A. Automated e-mail messaging as a tool for improving quit rates in an internet smoking cessation intervention. J Am Med Inform Assoc 2004 Jul;11(4):235-240 [FREE Full text] [Medline] [CrossRef]
    7. Strecher VJ, Shiffman S, West R. Randomized controlled trial of a web-based computer-tailored smoking cessation program as a supplement to nicotine patch therapy. Addiction 2005 May;100(5):682-688. [Medline] [CrossRef]
    8. Christensen H, Griffiths KM, Korten A. Web-based cognitive behavior therapy: analysis of site usage and changes in depression and anxiety scores. J Med Internet Res 2002 Feb 15;4(1):e3 [FREE Full text] [Medline] [CrossRef]
    9. Etter JF. Comparing the efficacy of two Internet-based, computer-tailored smoking cessation programs: a randomized trial. J Med Internet Res 2005 Mar 8;7(1):e2 [FREE Full text] [Medline] [CrossRef]
    10. McKay HG, Feil EG, Glasgow RE, Brown JE. Feasibility and use of an Internet support service for diabetes self-management. Diabetes Educ 1998 Mar;24(2):174-179. [Medline]
    11. Leslie E, Marshall AL, Owen N, Bauman A. Engagement and retention of participants in a physical activity website. Prev Med 2005 Jan;40(1):54-59. [Medline] [CrossRef]
    12. Danaher PJ, Mullarkey GW, Essegaier S. Factors affecting website visit duration: a cross-domain analysis. London, UK: Centre for Economic Performance, London School of Economics and Political Science; 2004.   URL: http://cep.lse.ac.uk/seminarpapers/07-03-05-ESS.pdf [accessed 2006 Jan 17]
    13. Joachims T, Granka L, Pan B, Gay G. Accurately interpreting clickthrough data as implicit feedback. ACM SIGIR 28th Annual International Conference on Research & Development on Information Retrieval Salvador, Brazil; 2005 Aug 15-19 [FREE Full text]
    14. Webb EJ, Campbell DT, Schwartz RD, Sechrest L. Unobtrusive measures. 2nd edition. Thousand Oaks, CA: Sage Publications; 1999.
    15. Bandura A. Health promotion by social cognitive means. Health Educ Behav 2004 Apr;31(2):143-164. [Medline] [CrossRef]
    16. Bandura A. Self-efficacy: the exercise of control. New York: WH Freeman; 1997.
    17. Bandura A. The primacy of self-regulation in health promotion. Appl Psychol: Int Rev 2005;54(2):245-254. [CrossRef]
    18. Brandon TH, Collins BN, Juliano LM, Lazev AB. Preventing relapse among former smokers: a comparison of minimal interventions through telephone and mail. J Consult Clin Psychol 2000 Feb;68(1):103-113. [Medline] [CrossRef]
    19. Mu~oz RF, Lenert LL, Delucchi K, Stoddard J, Perez JE, Penilla C, et al. Toward evidence-based Internet interventions: A Spanish/English Web site for international smoking cessation trials. Nicotine Tob Res 2006 Feb;8(1):77-87. [Medline]
    20. Anderson CM, Tedeschi GJ, Rosbrook B, Johnson CE, Byrd M, Gutierrez-Terrell E; Zhu S-H. Evidence of real-world effectiveness of a telephone quitline for smokers. N Engl J Med 2002 Oct 3;347(14):1087-1093. [Medline]
    21. Hatsukami DK, Severson HH. Oral spit tobacco: Addiction, prevention and treatment. Nicotine Tob Res 1999 Mar;1(1):21-44. [Medline]
    22. Severson HH. What have we learned from 20 years of research on smokeless tobacco cessation? Am J Med Sci 2003 Oct;326(4):206-211. [Medline]
    23. Severson HH, Andrews JA, Lichtenstein E, Danaher BG, Akers L. Cessation programs for smokeless tobacco users: long-term follow-up. Nicotine Tob Res In press.
    24. Danaher BG, Mckay HG, Seeley JR. The information architecture of behavior change websites. J Med Internet Res 2005;7(2):e12. [Medline] [CrossRef]
    25. Severson HH, Andrews JA, Lichtenstein E, Gordon JS, Barckley M, Akers L. A self-help cessation program for smokeless tobacco users: comparison of two interventions. Nicotine Tob Res 2000 Nov;2(4):363-370. [Medline]
    26. Danaher BG, Jazdzewski SA, McKay HG, Hudson CR. Bandwidth constraints to using video and other rich media in behavior change websites. J Med Internet Res 2005 Jun;7(4):e49 [FREE Full text] [Medline] [CrossRef]
    27. Severson HH, Gordon JS, Boles SB, Danaher BG, Akers L, Christiansen SM. Web-based smokeless tobacco cessation: preliminary outcomes and lessons learned. Paper presented at: Web-Assisted Tobacco Intervention Conference; June Bethesda, Maryland; 2005 [FREE Full text]
    28. Gordon JS, Akers L, Severson HH, Danaher BG, Boles SM. Participant recruitment strategies for an online smokeless tobacco cessation program. Nicotine Tob Res . In press.
    29. Peterson ET. Web site measurement hacks. Sebastopol, CA: O'Reilly Media; 2005.
    30. ; WebTrends. WebTrends 7.   URL: http://www.webtrends.com/ [accessed 2006 Jan 11] [WebCite Cache]
    31. ; Sane Solutions. NetTracker Web Analytic Solutions. 2006.   URL: http://www.sane.com/ [accessed 2006 Jan 11] [WebCite Cache]
    32. ; Wikipedia. HTTP cookie.   URL: http://en.wikipedia.org/wiki/HTTP_cookie [accessed 2006 Feb 23] [WebCite Cache]
    33. ; Wikipedia. Web beacon.   URL: http://en.wikipedia.org/wiki/Web_bug [accessed 2006 Feb 23] [WebCite Cache]
    34. Singer JD, Willett JB. Modeling the days of our lives: using survival analysis when designing and analyzing longitudinal studies of duration and the timing of events. Psychol Bull 1991;110(2):268-290. [CrossRef]
    35. Lee ET, Go OT. Survival analysis in public health research. Annu Rev Public Health 1997;18(1):105-134. [Medline] [CrossRef]
    36. Norusis MJ. SPSS 14.0: advanced statistical procedures companion. Upper Saddle River, NJ: Prentice Hall; 2005.
    37. Severson HH. Enough snuff: pocket guide to quitting smokeless tobacco. 6th edition. Eugene, OR: Applied Behavior Science Press; 2002.
    38. Torkzadeh G, Koufteros X. Factorial validity of a computer self-efficacy scale and the impact of computer training. Educ Psychol Meas 1994;54(3):813-821.
    39. Curry SJ, Ludman EJ, McClure J. Self-administered treatment for smoking cessation. J Clin Psychol 2003 Mar;59(3):305-319. [CrossRef] [Medline]
    40. Fiore MC, Bailey WC, Cohen SJ, et al. Treating tobacco use and dependence clinical practice guideline. US Department of Health and Human Services, Public Health Service. 2000.   URL: http://www.surgeongeneral.gov/tobacco/treating_tobacco_use.pdf [accessed 2006 Feb 14]
    41. Williams GC, McGregor H, Borrelli B, Jordan PJ, Strecher VJ. Measuring tobacco dependence treatment outcomes: A perspective from the behavior change consortium. Ann Behav Med 2005 Apr;29 Suppl(2):11-19. [Medline] [CrossRef]
    42. Ershoff DH, Quinn VP, Boyd NR, Stern J, Gregory M, Wirtschafter D. The Kaiser Permanente prenatal smoking-cessation trial: when more isn't better, what is enough? Am J Prev Med 1999 Oct;17(3):161-168. [Medline] [CrossRef]
    43. Mayer RE, Moreno R. Nine ways to reduce cognitive load in multimedia learning. Educ Psychol 2003;38(1):43-52. [CrossRef]
    44. Feinberg S, Murphy M. Applying cognitive load theory to the design of web-based instruction. Proceedings of the IEEE Professional Communication Society International Professional Communication Conference and Proceedings of the 18th Annual ACM International Conference on Computer Documentation: Technology & Teamwork. Piscataway, NJ: EEE Educational Activities Department; 2000:353-360.
    45. Reips UD. Standards for Internet-based experimenting. Exp Psychol 2002;49(4):243-256. [Medline] [CrossRef]
    46. Lichtenstein E. From rapid smoking to the Internet: five decades of cessation research. Nicotine Tob Res 2002 May;4(2):139-145. [CrossRef] [Medline]
    47. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health 1999 Sep;89(9):1322-1327 [FREE Full text] [Medline]
    48. Kramer ADI, Fussell SR, Setlock LD. Text analysis as a tool for analyzing conversation in online support groups. Conference on Human Factors in Computing Systems, CHI '04 Extended Abstracts on Human Factors in Computing Systems. New York: ACM Press; 2004:1485-1488.
    49. Pirolli P, Fu WT. SNIF-ACT: a model of information foraging on the World Wide Web. In: Ninth International Conference on User Modeling. Johnstown, Pennsylvania; 2003.   URL: http://www2.parc.com/istl/projects/uir/publications/items/UIR-2003-02-Pirolli-UM-SNIFACT.pdf [accessed 2006 Aug 13]
    50. Pirolli P, Card SK. Information foraging. Psychol Rev 1999;106(4):643-675. [CrossRef]
    51. Velicer WF, Prochaska JO, Fava JL, Norman GJ, Redding CA. Smoking cessation and stress management: applications of the Transtheoretical Model of behavior change. Homeostasis 1998;38:216-233.
    52. Mahoney MJ. Self-change: strategies for solving personal problems. New York: WW Norton; 1979.
    53. Rachlin H. The science of self-control. Cambridge, MA: Harvard University Press; 2000.

    Edited by G Eysenbach; submitted 13.03.06; peer-reviewed by E Leslie; comments to author 27.03.06; revised version received 16.05.06; accepted 06.08.06; published 30.08.06

    Copyright

    © Brian G Danaher, Shawn M Boles, Laura Akers, Judith S Gordon, Herbert H Severson. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 30.08.2006. Except where otherwise noted, articles published in the Journal of Medical Internet Research are distributed under the terms of the Creative Commons Attribution License (http://www.creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited, including full bibliographic details and the URL (see "please cite as" above), and this statement is included.