Assessing the Internal and External Validity of Mobile Health Physical Activity Promotion Interventions: A Systematic Literature Review Using the RE-AIM Framework

Introduction

The numerous health benefits of physical activity (PA) are well known, but still it is estimated that roughly 31% of the world’s adult population (28% men, 34% women) is classified as insufficiently active [1]. Likewise, it is a concern in the United States where only 6-11% of children [2] and 8.2% of adults meet the national PA guidelines based on objective PA assessments [3]. Given these low PA rates, there is a need for increased attention to the development of effective and scalable PA promotion interventions that can reach a large number of people at a low cost [4].

One such approach is the use of mobile technology, since ownership is on the rise in adults and children [5,6]. By 2012, it was estimated that there were 7 billion mobile-connected devices across the globe and the number of mobile devices outnumbered the human population [7]. In the United States, according to a 2012 nationally representative survey, more than 88% of American adults own mobile phones, which is an 11% increase from 2011 [8]. Fifty-three percent of American mobile phone owners own a smartphone [8]. Furthermore, roughly 75% of 12-17 year olds own mobile phones and this is a drastic surge (ie, up 30%) from 2004 [6].

This growth in mobile technology ownership has led to the development of a number of mobile health (mHealth) intervention reviews [9-12]. Specifically, related to PA, mHealth interventions that deliver information and behavioral strategies through short message service (SMS) via mobile phones have been developed to increase PA [13-17]. In addition, ecological momentary interventions through palmtop computers and mobile phones [14] can enhance interventions and aid in improving health outcomes. The potential utility of mHealth interventions to promote PA is also evident in the large number of commercially available fitness applications that promote behavioral tracking (eg, Nike+Running, Runtastic), link to external technology devices (eg, Fitbit), or directly encourage different intensities of PA (eg, Zombies, Run!). A recent meta-analysis on the use of mobile devices [10] and text messaging review [9] for PA promotion summarized the literature in this area and concluded that interventions delivered through this modality were effective for increasing PA. Similarly, a review of Internet-based PA interventions concluded that interactive technology interventions were effective for PA promotion [18]. However, a recent Cochrane review of mHealth interventions for preventive health care suggested that the availability of studies using randomized controlled trials was insufficient to determine if these approaches could influence PA or other health behaviors [12].

Despite the popularity of commercially available health-related applications, there is little evidence that mobile phone-based interventions with demonstrated efficacy have been translated beyond the research setting and been broadly adopted [19]. Some potential reasons for the lack of translation of these interventions into more widespread use are that the scientific approach typically emphasizes high internal validity at the expense of external validity [20] and that the traditional research pace impedes the flow of disseminating relevant findings [21]. To date, reviews of mHealth interventions have evaluated the quality of studies through the lens of internal validity and emphasized improved reporting on potential confounding factors [22]. As a result, the conclusions are largely limited to factors related to intervention efficacy and the extent to which these mHealth interventions report on or achieve external validity to different settings and populations is unclear [13-17,23]. This issue was recently underscored by the publication of the CONSORT-EHEALTH reporting standards [24]. The standards included eight highly recommended and four essential categories of reporting, which highlight the need for additional attention to external validity. Briefly, the four essential categories include (1) reporting on the context within which participants accessed the intervention, (2) the delivery mode, features, and functionality of the intervention, (3) the use of prompts to interact with the intervention, and (4) any co-interventions that may occur.

To improve the reporting across behavioral interventions, Glasgow and colleagues developed the RE-AIM (reach, effectiveness, adoption, implementation, maintenance) framework to evaluate the degree to which behavioral interventions, including those targeting PA, report on internal and external validity factors [25]. The framework specifies standards related to the reporting of “Reach” into the target population and representativeness of the study sample; “Efficacy/effectiveness” of the intervention on the primary outcome tested under either optimal or real-world conditions, quality of life, and avoidance of unintended or negative consequences; “Adoption” rates of organizations and staff that would ultimately use the intervention and the characteristics of those organizations and staff; the degree to which the intervention is “Implemented” as intended; and the “Maintenance” of effects at the individual level and sustainability of the intervention at an organizational or delivery level (RE-AIM) [26]. The RE-AIM framework has demonstrated utility in summarizing reports of internal and external validity factors across numerous bodies of literature (eg, weight loss maintenance, health literacy, tobacco use, and PA interventions for older adults and for breast cancer survivors) [27-37]. Collectively, these previous reviews have provided recommendations and future directions to enhance the likelihood of research to practice. Many of these recommendations align with those proposed in the CONSORT-EHEALTH standards [24]. In particular, the context within which participants access mHealth interventions is documented within the adoption (ie, description of intervention location and staff) and each of the other three essential standards are captured within an assessment of the implementation dimension (ie, cost, intervention description including frequency, type, and duration of contacts). The primary purpose of this systematic review is to determine the degree to which studies testing mHealth interventions to promote PA report on factors that inform generalizability across settings and populations. Recommendations to improve the likelihood of broad dissemination of effective mHealth interventions are also provided based on the literature [24,27-38].

Methods

Selection of Studies for Review

We replicated the search strategy used in a recently published meta-analysis publication that focused solely on effectiveness of mHealth interventions for PA promotion at the individual level [10]. Our literature search was conducted between August 2011 and July 2012 and included articles published between 2000 and 2012 that met the inclusion criteria indicated in Table 1. Review articles, observational (eg, cross-sectional, descriptive) commentaries, methodological articles, and articles not explicitly related to PA were excluded. Implementation of mobile technologies included data collection or conveyance of intervention information via SMS or native mobile device software or hardware. The search strategies to identify eligible articles included queries using three online databases (PsycINFO, PubMed, and Scopus) and a hand search of reference lists for articles that met inclusion criteria. The search terms included mobile phone, cell phone, PDA, SMS, or text messaging combined with PA or exercise [10]. In addition to comprehensively evaluating the reporting of RE-AIM criteria on a single trial, data was extracted from companion articles (eg, qualitative/quantitative methods measuring implementation) of studies that met inclusion criteria. Figure 1 outlines the identification of the 20 articles representing 15 trials that were included in this systematic review.

‎View this figure

Assessment of Reporting Comprehensiveness Across RE-AIM Dimensions

Comprehensiveness of reporting was determined using a previously developed 21-item validated data extraction tool that included both internal and external validity indicators based on the RE-AIM framework [27,28,31-33]. Table 2 includes details on each of the indicators assessed across the RE-AIM framework. In addition, we examined the degree to which reporting across the implementation dimension of RE-AIM addressed essential CONSORT-EHEALTH standards in terms of application costs, intervention features, theoretical backgrounds, prompts, and co-interventions [24]. Finally, due to the emerging nature of this body of literature, we also documented whether studies reported on adaptations that were made across intervention testing [34]. This was useful in order to determine the extent to which intervention fidelity was maintained during intervention implementation while allowing adaptations to the intervention to be made by delivery agents/organizations/systems to improve feasibility/acceptability/utility [39].

Coding Protocol and Scoring

All studies were coded independently by two members of the research team with the exception of the first three studies which were coded by five members of the research team to promote familiarity with the data extraction tool. For each of the 21 items, coders indicated whether or not the indicator was reported (ie, yes or no), and subsequently extracted specific data. After independently coding, the Kappa statistic [41] was calculated to evaluate inter-rater reliability. The average Kappa statistic for consistency of coding was 0.90, indicating strong inter-rater reliability. For the differences that did arise, researchers met to discuss articles, resolve uncertainty, and gain consensus in the coding by revisiting the specific article.

To calculate the proportion reporting for each item, the number of “yes” codes was summed across the 15 studies and then divided by 15. Then the resulting number became the proportion reporting for that particular item. An overall comprehensiveness of reporting score for each article was calculated based on the number of reported indicators (possible score 0-21). Comprehensiveness of reporting score categories have been published in a past RE-AIM review [28], with articles scoring 15-21, 8-14, and less than 8, considered as high, moderate, and low quality reporting, respectively.

Results

Study Characteristics

All trials were published after 2006 and 13 were conducted in Western countries. Six studies were conducted in the United States [43-52], three trials were conducted in the United Kingdom [53-55], three trials were conducted in Australia [56-59], and one each was conducted in Hong Kong [60], New Zealand [61], and Taiwan [62]. The majority of studies were randomized controlled trials and one was a quasi-experimental trial without a control group [47-49]. Most studies intervened at the individual level, two intervened at the group level [43-45,58,59], and one targeted both levels [52]. The length of the studies ranged from 2 weeks to 2 years, with an average of 19 weeks. The most commonly reported intervention length was 12 weeks.

Five studies measured PA only through self-report [43-45,50,52,56,57], four used objective measures [46,52,55,62], and three used both self-report and objective measures [47-49,53,61]. Of the seven studies that objectively measured PA, half of the studies used a pedometer [47-49,52,60,61]. Each of the following objective PA measures were collected once: both biaxial and triaxial accelerometers [58,59], uniaxial accelerometers [53], biaxial accelerometers [51], the walking distance of the incremental shuttle walking test [62], and a mobile sensing platform [46].

In addition to PA, the majority of studies (n=11) reported on other outcomes. Eight studies reported on body mass index (BMI) [43-49,53,54,58-60,62]; two on BMI-z scores [58,59,61]; five studies reported on physiological outcomes [43-45,51,53,61,62]; four studies on psychological outcomes [47-49,53,58-60]; three studies on weight [43-45,53,54]; two each on sedentary activity/screen time [52,58,59]; diet [43-45,58,59]; and percent body fat [51,60]; and one on each on sugar sweetened beverages intake [50]; upper body muscular endurance and core abdominal isometric muscular endurance [58,59]; waist-to-hip ratio [54]; and waist circumference [43-45].

The types of mobile devices used were similar across studies. Nearly all studies (n=13) used mobile phones while two used personal digital assistants [43-45,53]. Most frequently, mobile technology was implemented as a way to monitor outcomes via self-report [43-45,51-53] or data from an external pedometer/accelerometer was manually entered on the mobile phone [50,53]. Additionally, mobile technology was used to provide prompts [47-49,54,61] to encourage behavior change [55,56] and provide health promotion information sent through SMS [58,59]. Furthermore, in two studies, mobile technology was used as an interactive mobile application [46,57], in one study to deliver an exercise program [62] and in another study as a mobile PA diary [47-49]. Table 3 shows the overall quality of RE-AIM reporting across the 21-item validated extraction tool, which is displayed as the proportion reporting.

Reach

Reach was the second most reported dimension at 53.3% (2.67/5). Approximately half of all studies reported on four of the five items (method used to identify target population, inclusion and exclusion criteria, and participation rate). The least reported component was representativeness, with only four studies reporting [43-45,47-49,56,62]. None of the studies reported on characteristics of dropouts. All trials reported on sample size, which ranged from 17-210 participants with a median of 78. The participation rate ranged from 48-91 with a median of 51. The methods that were used to identify the target population included utilizing existing databases [43-45,54,56], regional diabetes services [61], recruitment coordinators [51], listservs [52], and an invitation letter from stakeholders [55]. In addition to English speaking, the most common inclusion criteria were PA requirement [46,50,53,54,56,58,59], weight requirement [43-45,53], or required possession/access to a mobile phone [46-49,54,56]. The majority of studies described recruitment strategies (n=11), which happened through various ways. Eleven studies focused on child participants, while four focused on adult participants [55,58,59]. Approaches for recruiting children included sending recruitment letters home [52,63], giving a presentation at school assemblies [55], teacher referrals [58,59], and using university listservs [52]. The majority of studies that enrolled adult participants were recruited mainly through local or mass media. Local mass media strategies included distributing flyers [43-45], using voicemail announcement systems [43-45], using mailing lists [43-45], posting ads on city buses [43-45], placing newspaper announcements [47-49], email [54,57], and using local mass media outlets [50,52]. Other studies’ recruitment methods comprised obtaining names and contact information from pulmonary rehabilitation coordinators [51], contacting individuals on a weight research registry via letter or email [43-45], targeting previous trial participants [43-45], and using a market research recruitment agency [46,53].

Efficacy/Effectiveness

Efficacy/effectiveness was the most reported dimension at 60.0% (2.4/4). All studies reported on measures or results for at least one follow-up. Approximately three quarters of the studies reported on percent attrition, which ranged from 0-53%. Four studies reported on intent-to-treat analysis [51,55,58-60], six stated present-at-follow-up analyses were used [50,52,54,55,57,62], and one used present-at-follow-up and intention-to-treat analyses [43-45]. Of the two studies that reported a high attrition rate (ie, 25% or higher) [52,56], only one [56] used intent-to-treat analysis.

The majority of studies (n=12) reported whether the trial was an efficacy or effectiveness trial. Of these studies, eight were efficacy trials [47-52,55-57,62] and four were effectiveness trials [46,57-59,62]. A little over 30% of studies reported quality of life or potential negative outcomes and found that mobile PA interventions generally improved quality and did not have any significant negative outcomes [43-45,51,53,61,62]. In terms of PA outcomes for the 14 controlled trials, six studies found that the intervention group had significant differences compared to the control group, four studies had mixed results, and four had nonsignificant differences between groups. In the only quasi-experimental, single group study included in this study, the post-assessments of PA were significantly higher than pre-assessments [47-49]. Only one study assessed cost effectiveness, which indicated that cost per participant associated with a mobile phone-based exercise program was $580 and coaching was added at a low incremental cost of US$80 [51]. Moreover, there were no significant differences in PA outcomes (ie, six-minute walk distance) between these two groups (ie, mobile-coached versus mobile self-monitored) in the study [51].

Adoption

The average proportion reporting on Adoption items was 11% (0.7/6). Level of expertise of delivery agent was the most reported adoption component (n=5). The descriptions of staff level of expertise included a nutritionist [43-45], a master’s level prepared exercise physiologist [43-45], a research assistant [56], a behavioral counselor [56], a nurse [51], and a psychologist [52]. No studies reported on method to identify staff who delivered the intervention, description of staff who delivered the intervention, inclusion/exclusion criteria of delivery agent, or adoption rate of delivery agent.

Setting-level reporting was similar to staff-level reporting. Only five studies specified the intervention location: a school [58,59,62], a research center physiologist, and an outpatient setting from four regional adolescent diabetes services [61]. Last, only two studies described the intervention location [58,59,62], two studies noted inclusion/exclusion criteria of setting [58,59,62], and one indicated adoption rate of setting [58,59].

Implementation

The average proportion reporting on Implementation indicators was 24% (0.7/3). Intervention duration and frequency were the most frequently reported items (n=6) [43-45,50-52,57-59]. Few studies reported on measures of cost of implementation (n=3) [51,58,59,62] or the degree to which the intervention protocol was carried out as intended (n=2) [53,58,59]. Though it appeared that no study charged participants for the applications, none explicitly stated this. Further, no study reported on adaptations made to intervention during the study. More than half (n=8) of the studies had a theoretical basis [43-46,56,58,59,62], with social cognitive theory used most frequently (n=3) [50,56,58,59]. Almost all studies (n=13) stated the degree to which participants received prompts, co-interventions, and other intervention components, including methods such as self-monitoring of outcomes through mobile technology (eg, mobile phone or PDA), class attendance, application usage, or the completion of intervention.

Maintenance

Maintenance was the dimension that was reported least among the RE-AIM dimensions, with no items (0%, 0/3) reported. The reporting on indicators of individual-level or program-level maintenance were not reported in any trial. 

Comprehensiveness of Reporting on RE-AIM Criteria

The average comprehensiveness of reporting score was 6.9 out of a possible 21-item reporting coding sheet and scores ranged from 3-13. None of the studies were categorized as high reporting quality, six studies were moderate (range 8-11) [43-45,51,52,56,58,59,62], and nine studies were of low reporting quality [46-50,53-55,57,60,61].

Discussion

Principal Findings

Our review highlighted a recent increase in studies conducted to determine the efficacy or effectiveness of mHealth interventions for the promotion of PA. We identified gaps across and within each of the RE-AIM dimensions, potentially as a result of the relative early stages of this area of research. We also understand that there is a need to advance research by utilizing innovative, flexible, and rapid research designs and “rapid-learning research systems” where researchers, funders, health systems, practitioners, and community partners collaborate [21]; however, the lack of internal and external validity reporting identified indicated that few innovative designs are currently being used in this area of investigation.

Still, the comprehensiveness of reporting on RE-AIM criteria across these mHealth articles was relatively low with a number of gaps in reporting on both internal (eg, extent that the protocol was delivered as intended) and external validity factors (eg, description of intervention location and staff). At the individual level (ie, reach, efficacy/effectiveness, and maintenance), the reporting on issues related to reach and maintenance are particularly problematic. At the organizational or delivery level (ie, adoption, implementation, maintenance), there are large gaps in reporting across each of the dimensions. These gaps extend to the reporting across the four CONSORT-EHEALTH standards of access as well as the degree to which intervention features and functionality were addressed. Based on our findings, the results reported on mHealth PA interventions, from both an internal and external validity perspective, should be considered with caution.

Consistent with past research, this body of literature does not typically describe the target population or give indications as to the degree to which the study samples are representative of a larger population [27,28,31,33,35,37,40,42]. Thus, inferences cannot be made regarding who may be likely to benefit from these interventions based on different demographic, economic, or behavioral factors. Similarly, it is unclear which subgroups of the population may be more or less likely to engage in mHealth PA interventions. This is especially important to document given that those studies that did examine the representativeness of the study sample to the target population found that nonparticipants were less educated [43-45,56] and, if they did engage in the study, had greater difficulty in operating technology [47-49,62]. Additionally, almost all of the studies used convenience sampling procedures rather than recruiting from a known target population denominator. It is vital to recruit larger numbers of subgroups of the target population so that individuals that could most benefit from the intervention are actually receiving it. This information, across studies, is critical to ensure that interventions are designed to address broad access to the intervention and the needs of subgroups of a target population that suffer from health disparities (eg, lower education levels).

Similar to other areas of research, efficacy or effectiveness based upon changes to the PA and percent attrition were reported consistently across the majority of studies while the maintenance of those changes were not [27-29,31,32,36,37]. However, the generally positive effects found across studies may be overestimated based on the degree to which attrition was considered in follow-up analyses. That is, only one in every three studies reported using intent-to-treat analyses, with the majority limiting the description of study results to those who were present for follow-up assessments. Given the recidivism related to physical inactivity, it may be surprising that no study examined the maintenance of PA change at least 6 months past completion of the intervention. On one hand, the area is relatively new and it appears that researchers have emphasized determining the degree to which these interventions can initiate change. On the other hand, mHealth interventions may be less likely to encourage PA change maintenance due to advancements in newer technology that could make current interventions obsolete, the potential of technical problems that may reduce motivation, or simply decreased participant engagement over time. Until maintenance is documented in mHealth PA intervention studies, it is left to researcher and participant speculations on how well these interventions can contribute to maintained PA change over a long period of time.

Organizational or delivery level facets of RE-AIM have consistently been underreported across behavior change intervention studies; yet, studies on mHealth PA interventions appear to be even less likely to report on organizational adoption, implementation, and maintenance [29,30,37-39]. To be balanced, the majority of the studies included were reported as efficacy trials and some adoption information like inclusion and exclusion criteria of the staff and locations of intervention delivery may not be relevant. Similarly, efficacy and effectiveness trials do not typically have a goal to achieve and track maintained delivery of an intervention beyond the life of a grant. However, to allow for replication and determination of generalizability, even within highly controlled efficacy trials, it is necessary to provide the description of (1) the intervention costs and location, (2) the characteristics of the intervention and those who delivered it, (3) the degree to which the intervention was delivered as intended, and (4) if any adaptations were made to the intervention during the study period. An additional potential critique of this literature is the tendency for participants to use non-assigned treatments [22] that may contribute to the intervention’s effectiveness. However, this body of literature included reports of co-interventions that, to some degree, address this issue. Still, no articles reported explicit tracking of non-assigned treatments so that possible contributions to effectiveness could be determined [22].

Understanding costs across RE-AIM dimensions is also key for dissemination [38]. In addition to implementation costs, other costs may be accrued both by organizations implementing these interventions as well as by the participants using them. For example, tracking of costs related to recruitment, equipment, technology (eg, batteries/chargers, mobile phone, service plan), and maintenance (eg, continue program once study period or funding is over) can improve the information necessary for decision making. From the perspective of the CONSORT-EHEALTH standards, future costs to the consumer should be considered in relation to the expected reach and effectiveness of mHealth PA interventions [24]. Without information on adoption and implementation, it is difficult to know the resources needed to successfully implement mHealth interventions in diverse locations or with staff of different levels of expertise.

Based upon the growth of research in the area of mHealth PA interventions and the review of this literature to date, there are a number of ways to improve the assessment and reporting on individual and organizational level factors that will improve our understanding of both the internal and external validity of this work. In Table 4, we provide a number of recommendations across RE-AIM dimensions specific to mHealth PA intervention research. In addition to these recommendations, the use of mixed method approaches that blend qualitative and quantitative data collected from participants and from those who implement the intervention could add depth to the data collected in mHealth PA studies and improve subsequent replication and implementation efforts [34]. Further, from a translational science perspective, tracking mHealth intervention costs across RE-AIM dimensions can inform adoption and delivery decisions within community and clinical practice settings.

Limitations

Our review includes some limitations. First, our conclusions and recommendations are based on the degree to which these studies reported on specific RE-AIM dimensions. It is possible that some of these data have been collected, but not reported. To address this, we included all available articles on any given trial. Still, investigator plans and data for maintenance/sustainability or designing for dissemination may exist but go unreported; however, a transparent reporting of any existing plans would provide additional important context for any intervention study. In addition, a lack of reporting on an outcome cannot be equated to a lack of an intervention’s ability to achieve that outcome (eg, lack of reporting on maintenance cannot be equated to a lack of maintenance). Second, because mHealth PA interventions are relatively novel and this is an emergent research area, the goal of the studies included within this review may have been to establish internal validity (eg, effectiveness of study outcomes), and therefore we must be cautious of being overly critical of these studies relative to their reporting of organizational adoption or maintenance factors.

Conclusions

There is an emergent body of literature reporting on mHealth PA interventions. On average, the studies provide initial evidence that these interventions may have promise in helping participants initiate PA. However, few studies report on key internal (eg, delivery as intended) or external (eg, descriptions of participants, settings, and delivery staff) factors. As a result, the degree to which these findings are robust and generalizable cannot be determined. Improved reporting across RE-AIM dimensions and the use of intention-to-treat, tracking of costs, and mixed methods approaches are recommended to ensure mHealth PA interventions are developed that can be broadly applicable across target populations, intervention delivery locations, and staff of differing levels of expertise.