Effectiveness of a Web 2.0 Intervention to Increase Physical Activity in Real-World Settings: Randomized Ecological Trial

Introduction

Given low population levels of physical activity and high associated physical and mental burden of disease caused by inactive lifestyles [1], there is a need for effective physical activity interventions that can reach large populations at low cost [2]. In this context, research into the effectiveness of Web-based interventions has become popular [3], as large and increasingly diverse populations can be reached without geographical limitations by using the Internet [4]. While literature reviews and meta-analyses point to the short-term effectiveness of Web-based physical activity interventions [5,6], they also highlight that there is a lack of evidence for long-term behavior change, as well as a lack of knowledge about what are the most effective intervention components [7]. The lack of evidence for long-term behavior change has often been attributed to low levels of participant engagement and retention, due to examining websites that are static in nature, that lack social support elements, and that provide limited opportunity for interactivity or information exchange [5].

Websites with more dynamic, interactive, user-focused features, also referred to as second generation or Web 2.0 features, are now commonly used and include social networking, blogs, wikis, podcasts, and mash-ups [8]. They provide users with the opportunity to directly generate, modify, and share information [9]. Few physical activity studies have examined the effectiveness of Web 2.0 features [8,10,11], which may enhance engagement with the intervention and, in turn, lead to long-term behavior change. A review by Maher et al, however, indicated that the use of online social networks in behavior change trials was only modestly effective [10]. In this context, it has been argued that randomized controlled trials (RCTs) are not the most appropriate research design to truly examine the effectiveness of Web 2.0 features [12].

It has been asserted that the highly controlled nature of RCTs, which aim to minimize impact of selection bias, confounding factors, and contamination, stifle the dynamic, spontaneous, viral nature of Web 2.0 features [12]. While RCTs are an essential component of the research process, complementary approaches with high external validity and generalizability are also essential. For example, if one is not able to invite friends to join an online social network due to RCT-related restrictions, the social network is unlikely to be as functional and effective as it would be in real-world circumstances [13]. As such, there is a need for alternative and ecologically valid research designs that evaluate Web-based interventions in real-life conditions in order to advance the science in this area [14].

The translation and dissemination of Web-based physical activity intervention research into the real world is lacking and becoming increasingly important. Therefore, the main aim of this study was to compare the physical activity behavior of individuals using a traditional Web 1.0 physical activity website to those using an innovative Web 2.0 physical activity website in real-world settings. The study also aimed to assess the effectiveness of Web 2.0 features to engage and retain individuals to a physical activity promotion website, as well as examine differences in quality of life and body mass index (BMI) between intervention groups. The primary hypothesis was that participants in the Web 2.0 condition would display higher levels of physical activity at 3 and 12 months, compared to those in the Web 1.0 condition. The secondary hypotheses were that, in the Web 2.0 condition, there would be higher website engagement and retention as well as improvements in quality of life and BMI when compared to the Web 1.0 condition at 3 and 12 months.

Methods

Overview

As the detailed protocol for this study has been published elsewhere [13], only summary information will be provided here (see Multimedia Appendix 1 for additional screenshots of the intervention). This study is the second phase of the substantive WALK 2.0 project and builds on an earlier RCT [15,16], which rigorously tested the efficacy of the interventions described here [11]. The Western Sydney University Human Research Ethics Committee granted ethical approval for this study (H8767). This trial has been registered at the Australian New Zealand Clinical Trials Registry (ACTRN12611000253909).

Recruitment, Procedures, Study Design, and Participants

Adults 18 years of age or older spontaneously signing up for the freely available and Web-based 10,000 Steps program [17], which attracts over a 1000 new members per month [18], were asked during the registration process whether they wanted to participate in a research study from November 2012 to June 2014. The 10,000 Steps project has been funded since 2001 through Queensland Health, one of the Australian State Ministries of Health. The project is well known through media and marketing events in Australia, particularly in Queensland, with over 70% program awareness in population-based surveys. If potential participants agreed to participate in research, they received more information about the study, were screened online for eligibility, provided informed consent, and completed a brief baseline survey. Using a computer-generated algorithm, they were randomized to receive access to one of two intervention websites: a Web 1.0 intervention, which was the 10,000 Steps website they were originally signing up for, or a Web 2.0 intervention, which was the newly developed WALK 2.0 website. For technical reasons, participants were randomized before completing the baseline measures, however, they only gained access to intervention materials after completing the baseline assessment. Follow-up outcomes were assessed 3 and 12 months postbaseline using online questionnaires; participants were invited by email and received up to three reminders. All actions, from study invitation to completion, were fully automated with no interaction from the research team at any point. The research team also provided no instructions as to how the interventions should be used and there was no predefined intervention duration. However, even though the aim of reaching 10,000 steps a day was implicit, participants were provided with the Australian Physical Activity Guidelines and the websites were designed to encourage self-monitoring and interaction on a daily basis for as long as possible—participants had an option to receive a daily reminder to use the websites. No pedometers were provided. Exclusion criteria were the following: being under 18 years of age; seeking to participate in a 10,000 Steps Workplace Challenge; having been a participant in the WALK 2.0 RCT; and having a medical condition that prevents them from increasing physical activity, assessed through the Physical Activity Readiness Questionnaire [19].

Interventions

Web 1.0 Intervention

Participants allocated to the Web 1.0 group were given access to the existing 10,000 Steps website. This website was originally developed to promote the community-based 10,000 Steps Australia project [20,21]. The website includes features that support individual self-monitoring (eg, step log) and communication exchange (eg, discussion forums), and provides access to a library with educational resources (eg, benefits of activity). Participants were able to log steps and/or type and duration of other physical activities. Participants also had the ability to share stories, ask questions, or make comments in the discussion forum.

Web 2.0 Intervention

Participants allocated to the Web 2.0 group were given access to a newly developed website, WALK 2.0, that provided content and functionality similar to the Web 1.0 condition (eg, step log and library); however, this website was supplemented with Web 2.0 features that included annotation, messaging, and group-publishing tools implemented in a stand-alone social networking setting. Participants in the Web 2.0 group also had access to self-monitoring features and educational resources, however, these had advanced functionalities that provided greater interactivity and participatory communication between users (eg, status updates, internal emails, inviting “friends,” and personalized profile pages). Participants could upload content to their own profile page, share this information with others, and invite individuals who were not study participants to become their “friends” and use the website. Participants could also connect with Facebook (eg, post their step total for the day on their Facebook wall), but the Web 2.0 website was completely separate from Facebook.

Measures

Demographics

Participants’ gender, age, educational level (school education, trade/diploma, or higher education), employment status (full time, part time/casual, or other), occupation (professional white collar, blue collar, or other), weekly household income (<Aus $1000, Aus $1000-$1999, Aus $2000-$5000, or no response), Internet self-confidence (low or high), height (cm), and weight (kg) were assessed. Self-reported BMI was calculated as weight (kg) over height squared (m²) and categorized as normal weight (≤24.99 kg/m²), overweight (25-29.99 kg/m²), and obese (≥30 kg/m²).

Physical Activity

The Active Australia Survey was used to measure self-reported physical activity [22]. This instrument provides an estimate of total weekly minutes of physical activity by summing total minutes of walking for transport and recreation, moderate-intensity physical activity, and vigorous-intensity physical activity—multiplied by 2 to account for the higher energy expenditure per time unit—during the previous week. The Active Australia Survey has acceptable test-retest reliability and validity in Australian adults [23,24] and has been demonstrated to be sensitive enough to detect change over time [25]. The Active Australia Survey was used to determine the following: total physical activity per week and whether participants were engaging in sufficient physical activity—a minimum of 150 minutes of moderate-to-vigorous physical activity per week accumulated over five or more sessions.

Quality of Life

The RAND 36 Short Form Survey was used to assess quality of life in eight health-related categories: physical functioning, bodily pain, role limitations due to physical health problems, role limitations due to personal or emotional problems, emotional well-being, social functioning, energy/fatigue, and general health perceptions [26,27]. All items were scored from 0 to 100, with a high score representing a more favorable health state. Items in each category were then averaged together to create eight subscale scores. The RAND 36 has been validated in Australian populations [28].

Website Engagement and Retention

Website usage statistics for both websites were continuously measured using Google Analytics (eg, time on site) and data were extracted directly from the website databases (eg, step entry information). These measures were only examined from baseline to 3 months (first 12 weeks), due to the low survey completion rate at 12 months. The total and average number of website visits were assessed, as well as the time between the first and last visit. The total and average number of days with a step entry and step entry comments were assessed, including the time between first and last step entry. Nonusage attrition was defined as not having visited the website and/or logged steps for at least two consecutive weeks [11,18].

Website Usability

The System Usability Scale (SUS) was used to assess website usability [29]. This scale is a 10-item survey, scored on a 5-point scale of strength of agreement, with good reliability and concurrent validity [30]. Final scores can range from 0 to 100, where higher scores indicate better usability. Self-reported use and usefulness for different features on both websites is also reported. Participants were asked about usefulness on a 5-point scale; the proportion of participants who thought the feature was “useful” or “very useful” is reported.

Sample Size

 The trial was powered to detect a 4% between-group difference in the prevalence of sufficient physical activity, as defined by the Australian Physical Activity Guidelines, between the Web 1.0 and Web 2.0 groups. To achieve this aim with 80% power and an alpha level of .05, a minimum of 1034 participants per group were needed [13].

Statistical Analyses

All statistical analyses were conducted using SAS version 9.4 (SAS Institute Inc). Differences between participants with complete and missing data were compared using t tests or Pearson chi-square tests. Analysis of covariance (ANCOVA) was used to test for differences between treatment groups at 3 months in physical activity, quality of life, and BMI; baseline physical activity, BMI, quality of life levels, and confounding variables were included in the models as covariates. Logistic regression was used to estimate between-group differences in the proportion of participants who achieved sufficient physical activity. Results are presented both for those with complete data at baseline and 3 months (completer analyses), as well as those with missing data, following intention-to-treat principles. Multiple imputation was applied to deal with missing data, under the missing-at-random assumption, using the chained equations method. Rubin’s method was used to pool the treatment effects using 25 imputed datasets, as the fraction of missing data was high [31]. To analyze between-group differences in website engagement and retention, t tests were used. A proportional hazards regression model was used to estimate between-group differences in time from randomization to nonusage; Kaplan-Meier estimates of the proportion remaining active (the survival distribution) are also presented [32]. Due to very small participant numbers, 12-month data were not included in any of the analyses. The significance level was set at P<.05.

Results

Participants

A participant flowchart is provided in Figure 1. After automatically screening out ineligible people, 10,673 people were invited and 3480 indicated an interest in participating. After eligibility checks, providing informed consent, and website registration, 1328 people completed all baseline measures. Out of 1328 participants, 224 (16.87%) completed the 3-month assessment and 77 (5.80%) completed the 12-month assessment. Table 1 presents participant demographics. At baseline, the majority of participants were female (1095/1328, 82.45%), were 44 years of age or under (818/1328, 61.60%), were overweight or obese (849/1328, 63.93%), had a higher education (693/1328, 52.18%), were full-time employed (771/1328, 58.06%), had a professional or white-collar job (879/1328, 66.19%), and participated in sufficient physical activity (757/1328, 57.00%). There were no significant between-group differences in participant characteristics at baseline; however, several between-group differences were observed among those who completed the 3-month assessment and those who did not. There was significantly greater retention among those who were Web 1.0 group participants, male, aged 45 years or older, and not obese, as well as those having a higher education, professional occupation, and higher income.

Physical Activity

The physical activity outcomes are presented in Table 2. In the intention-to-treat analysis, a significant difference between groups was found: participants randomized to the Web 2.0 group increased physical activity by 92.8 minutes per week more compared to those in the Web 1.0 group (P=.005; Cohen d effect size=0.29). In the completer analysis, participants in the Web 2.0 group increased physical activity by 56.6 minutes per week more compared to those in the Web 1.0 group, however, this difference was not significant (P=.20; Cohen d effect size=0.24).

At baseline, 57.00% (757/1328) of participants in both groups engaged in sufficient physical activity. At the 3-month time point, 77% (62/80) of Web 2.0 participants and 71.5% (103/144) of Web 1.0 participants engaged in sufficient physical activity. A significant between-group difference in favor of the Web 2.0 group was observed in the intention-to-treat analysis (Relative Risk [RR]=1.11, 95% CI 1.01-1.21; t_64.5=2.19, P=.03), but not in the completer analysis (RR=1.03, 95% CI 0.89-1.18; Z=0.36, P=.71).

Quality of Life

There was no effect of the intervention on most quality of life variables (see Table 2), except for the physical functioning domain: a significant improvement (3.6 units) was observed in the Web 2.0 group compared to the Web 1.0 group (P<.001).

Body Mass Index

BMI reduced over time in both groups (see Table 2) and significant between-group differences were observed for both the intention-to-treat analysis (in favor of the Web 2.0 group: change in BMI=-1.03 kg/m², P=.002) and the completer analysis (in favor of the Web 1.0 group: change in BMI=-0.58 kg/m², P=.002).

‎View this figure

Website Engagement, Retention, and Usability

Differences in terms of website usage are shown in Table 3. Participants used the websites on average 3 minutes per week and logged steps for approximately 10 days. There were no significant between-group differences with regard to time on site, entering steps without comments, and the SUS. However, total number of visits (9.0 [19.3] vs 5.7 [17.9], P=.002) and average number of visits (0.7 [1.6] vs 0.4 [1.4], P=.002) were significantly higher in the Web 2.0 group, as well as the time between first and last visit (3.6 [3.7] vs 2.2 [2.9], P<.001). Total days (0.7 [5.9] vs 0.2 [1.5], P=.03) and average days (0.05 [0.45] vs 0.01 [0.10], P=.02) with a step entry comment were also significantly higher in the Web 2.0 group. A Kaplan-Meier survival plot shows the proportion of participants that remained using the website for each week of the study (see Figure 2). Only 21.99% (292/1328) of participants were still using either website after 2 weeks (n=292 summed for both groups) and 6.55% (87/1328) after 10 weeks. The between-group difference in time-to-nonusage attrition was not statistically significant (Hazard Ratio=0.97, 95% CI 0.86-1.09; P=.57). Self-reported use and usefulness of the features available on both websites are reported in Table 4. More participants in the Web 2.0 group used features that were present, or similar, on both websites compared to the Web 1.0 group. Web 2.0 participants also rated the usefulness of these features more highly compared to Web 1.0 participants. Many participants in both groups did not use some of the interactive features or indicate that they were very useful.

‎View this figure

Discussion

Principal Findings

The aim of this study was to compare physical activity behavior of individuals using a traditional Web 1.0 physical activity website with those using a more interactive and social Web 2.0 physical activity website in real-world settings. The primary hypothesis was confirmed: those in the Web 2.0 group did display significantly higher levels of physical activity at 3 months compared to those in the Web 1.0 group. The secondary hypotheses were partially confirmed: BMI significantly decreased, engagement was significantly higher in some variables (eg, website visits), and quality of life significantly improved in one variable (ie, physical functioning) in the Web 2.0 group compared to the Web 1.0 group at 3 months. However, there were no between-group differences for several other engagement variables (eg, logging steps and nonusage attrition) and most quality-of-life variables. This study was the first to demonstrate the importance of using Web 2.0 features in Web-based physical activity interventions in a real-life setting. These outcomes are strengthened by not finding between-group differences in website usability (ie, SUS score), indicating that outcomes were not influenced by factors such as user friendliness. Interestingly, the use and usability of features that were present, or similar, on both websites were higher in the Web 2.0 website compared to the Web 1.0 website. While, from this research, it is not possible to explain this difference given the similar SUS scores, it may in part explain why the Web 2.0 website performed better on several behavioral and engagement outcomes.

While there is an abundance of studies examining the effectiveness of physical activity promotion websites in controlled conditions [33], few studies have examined the use and effectiveness of physical activity websites in natural and real-life conditions. To our knowledge, only one other study has examined people who spontaneously signed up to an online physical activity intervention and where researchers had no direct contact with study participants: all processes were completed automatically [34]. Wanner et al conducted an RCT with two groups of participants that were actively recruited to be randomized to either a control group or an interactive, computer-tailored, physical activity website; however, they included a third group of spontaneous users from the same website. Significantly larger increases in physical activity were observed in the spontaneous users when compared to the actively recruited groups [34]. That study’s outcomes were similar to ours when comparing findings from our RCT with the ecological trial: physical activity increases reported in this ecological trial (+140 for Web 2.0 group; +40 for Web 1.0 group) are much higher than those observed in the RCT (+45.5 for Web 2.0 group; –1.0 for Web 1.0 group) [11]. The large differences observed in these studies indicate that it is not adequate to rely only on RCT outcomes when preparing Web-based physical activity interventions for dissemination and translation.

Poor engagement with Web-based interventions has often been reported; with regard to that aspect, this study is not unique [33,35]. The ecological trial and the RCT [11], however, were remarkably different in terms of engagement and retention outcomes. While the Web 2.0 group also had better engagement outcomes in the RCT, the overall engagement of participants was much lower in the ecological trial. For example, nonusage attrition in the RCT occurred on average after 35.5 and 25.5 weeks in the Web 2.0 and Web 1.0 groups, respectively, whereas it occurred after a mere 1.5 weeks for both groups in the ecological trial. The RCT website was visited, on average, 3.6 and 1.6 times per week for the Web 2.0 and Web 1.0 groups, respectively, but only 0.7 and 0.5 times per week, respectively, in the ecological trial. While participants in both trials were exposed to identical websites, there are factors that encourage engagement in RCTs, and factors that discourage engagement in ecological trials. In the RCT, participants were required to complete multiple face-to-face visits and they received phone calls, a pedometer, and multiple small financial incentives [11]. These strategies increased participant accountability and resulted in a 60% retention rate at 18 months, which is comparable to other studies in this field [6]. In order to keep the intervention implementation as natural as possible in the ecological trial, no such strategies, other than email reminders, were applied. However, in the ecological trial the process of asking 10,000 Steps visitors to participate in research and obtaining informed consent and baseline assessments may have deterred many participants, given the attrition at each step of the process (shown in Figure 1). These necessary steps may have been detrimental to their ongoing participation, as is demonstrated in a previous analysis of 16,948 spontaneous 10,000 Steps users who did not need to go through multiple screenings to be included in the analyses [18]. Among these 10,000 Steps users, nonusage attrition occurred after 4 weeks and the website was visited 2.4 times per week; these engagement outcomes are remarkably higher than those among ecological trial participants [18]. Finally, in another study, Wanner et al compared spontaneous users with trial participants and also reported large engagement differences: nonusage attrition occurred at 41 weeks in trial participants, but it was zero days in spontaneous users [36]. Collectively, these outcomes indicate that caution should be taken when interpreting engagement outcomes from both RCTs and ecological trials, as neither may be a good reflection of how participants engage with websites in reality. The outcomes further suggest that improved participant retention strategies are required that work well in ecologically valid circumstances [37].

The significant reduction of BMI in favor of the Web 2.0 group was surprising, as the intervention did not focus on weight loss, and the 3-month time frame is reasonably short. However, previous Web-based interventions have demonstrated weight loss, both in interventions focused on either weight loss or physical activity [38,39]. Further, few studies have examined how Web-based physical activity interventions can improve health-related quality of life. A meta-analysis of the impact of non-Web-based physical activity interventions on quality of life observed that the interventions only significantly changed the physical functioning domain of quality of life, and not other domains [40]. This is consistent with the outcomes of this study.

Women dominated participation in this study (82%), however, this was very comparable to other studies. For example, Anderson-Bill et al [41] conducted an online walking program and reported that 83% of participants were female and 75% of spontaneous website users were female in the study by Wanner et al [36]. Lower male participation is also commonly observed in RCTs [5,6] and this study demonstrates that the difficulty in attracting men to health behavior change interventions may even be greater in ecologically valid circumstances. Several significant differences were observed between those who participated at 3 months and those who did not. There was less attrition among those randomized to the Web 1.0 group (10,000 Steps). As all participants were originally signing up to participate in the 10,000 Steps program, it is plausible that many of those who were randomized to the Web 2.0 group (WALK 2.0) were disappointed with their allocation and dropped out for that reason. It is also possible that the increased complexity and interactivity of the Web 2.0 website resulted in higher dropout rates, though this is less likely as there was no significant difference in the SUS score between the websites. Older participants and men were less likely to drop out; this is similar to the study by Wanner et al, who found that among their spontaneous users, men and those of increasing age were more likely to repeatedly use their website [36]. It is also in line with a Web-based physical activity intervention study that found older participants spent more time on the website and changed behavior more than younger participants [42]. Finally, it is not surprising to see lower dropout rates among those with higher education, higher income, and a professional occupation, as it has been widely reported that people with a higher socioeconomic status are more amenable to participating in health behavior change interventions [43,44].

Strengths and Limitations

The major strengths of this study were the innovative ecological randomized nature of the study and the comparison of Web 1.0 and Web 2.0 intervention features. The large nonusage and study attrition, however, was an important limitation, making it problematic to analyze 12-month outcomes. This limitation, though, could be considered as a finding that is of interest, as it is a reflection of how Web-based interventions are being used in ecologically valid circumstances; it is not, per se, a reflection of poor study methodology. That said, several methodological limitations are inherent to ecological trials, such as the lack of a true control group and having to resort to less intrusive (ie, self-report) measures to assess outcomes. These methodological concerns can, however, be alleviated by comparing the study outcomes to those of an RCT that applied more rigorous methods [11]. Furthermore, all engagement and retention measures were assessed objectively through the use of website usage statistics. However, it was a limitation of this study that the interventions were only accessible via websites, as the technology landscape has rapidly changed since the study inception and it is now very common to also use mobile phone apps, which are more convenient to access than websites [18].

Conclusions

Web 2.0 intervention features appear to be more effective in increasing physical activity, decreasing BMI, and improving physical functioning (ie, quality-of-life domain) compared to Web 1.0 features in ecologically valid circumstances. While the social and interactive Web 2.0 features increased engagement compared to a traditional Web 1.0 website, website usage was low regardless, especially when compared to our previous RCT [11]. As such, more research is needed to increase our understanding of how people engage with Web-based interventions, both in controlled and ecologically valid circumstances, and how closely this engagement is related to actual behavior change (ie, what is the dose-response relationship?) [45]. Finally, the findings of this study are remarkable in how different they are from the findings observed in controlled conditions in terms of attrition, website usage, and behavior changes observed. This is important, because information obtained through RCTs may not translate well to real-world conditions. Yet almost all of our knowledge in this field is derived from RCTs and this has a major influence on how Web-based interventions are designed, what interventions are deemed effective, and what interventions are being disseminated and implemented. As such, there is an urgent need for more ecological trials and implementation studies.