Background: Gay, bisexual, and other men who have sex with men (GBMSM) face the highest burden of HIV in the United States, and there is a paucity of efficacious mobile health (mHealth) HIV prevention and care interventions tailored specifically for GBMSM. We tested a mobile app combining prevention messages and access to core prevention services for GBMSM.
Objective: This study aims to measure the efficacy of the Mobile Messaging for Men (M-cubed) app and related services to increase HIV prevention and care behaviors in diverse US GBMSM.
Methods: We conducted a randomized open-label study with a waitlist control group among GBMSM in 3 groups (low-risk HIV-negative group, high-risk HIV-negative group, and living-with-HIV [LWH] group) recruited online and in venues in Atlanta, Detroit, and New York City. Participants were randomly assigned to receive access to the app immediately or at 9 months after randomization. The app provided prevention messages in 6 domains of sexual health and offered ordering of at-home HIV and sexually transmitted infection test kits, receiving preexposure prophylaxis (PrEP) evaluations and navigation, and service locators. Serostatus- and risk-specific prevention outcomes were evaluated at baseline, at the end of the intervention period, and at 3, 6, and 9 months after the intervention period.
Results: In total, 1226 GBMSM were enrolled and randomized; of these 611 (49.84%) were assigned to the intervention group and 608 (99.51%) were analyzed, while 615 (50.16%) were assigned to the control group and 612 (99.51%) were analyzed. For high-risk GBMSM, allocation to the intervention arm was associated with higher odds of HIV testing during the intervention period (adjusted odds ratio [aOR] 2.02, 95% CI 1.11-3.66) and with higher odds of using PrEP in the 3 months after the intervention period (aOR 2.41, 95% CI 1.00-5.76, P<.05). No changes in HIV prevention or care were associated with allocation to the intervention arm for the low-risk HIV-negative and LWH groups.
Conclusions: Access to the M-cubed app was associated with increased HIV testing and PrEP use among high-risk HIV-negative GBMSM in 3 US cities. The app could be made available through funded HIV prevention providers; additional efforts are needed to understand optimal strategies to implement the app outside of the research setting.
Trial Registration: ClinicalTrials.gov NCT03666247; https://clinicaltrials.gov/ct2/show/NCT03666247
International Registered Report Identifier (IRRID): RR2-10.2196/16439
The ambitious goal of the US Ending the HIV Epidemic Initiative to reduce new HIV diagnoses by 90% by 2030  will not be reached without reducing new HIV infections in gay, bisexual, and other men who have sex with men (GBMSM). GBMSM comprised 54% of annual new HIV diagnoses in 2019 in the United States [ ] and face the highest burden of HIV in the United States [ ]. However, there are few efficacious or promising mobile health (mHealth) HIV prevention and care interventions tailored for subgroups of GBMSM [ , ]. New HIV prevention tools should address the needs of GBMSM, young GBMSM aged 15-24 years (YGBMSM) [ ], and YGBMSM of color, for whom HIV incidence is the highest [ - ]. Currently, HIV prevention services are underused by GBMSM: in 2018, just over half (56%) reported being tested in the past 12 months, high proportions (76% of HIV-positive and 66% of HIV-negative GBMSM) reported recent condomless receptive anal intercourse, and few (<20%) reported preexposure prophylaxis (PrEP) use [ , ]. Statistical models demonstrate that high levels of coverage of prevention services will be required to substantially reduce HIV incidence among GBMSM [ , ] and increased use of routine prevention services (eg, HIV testing) might increase uptake of biomedical interventions, such as PrEP [ ]. A package of HIV prevention interventions is needed to maximally reduce the risks of infection, and uptake of multiple prevention services by a substantial proportion of the GBMSM population is needed to reduce HIV incidence among GBMSM.
A growing body of research has suggested that mobile phone apps are a promising environment to offer tailored and on-demand prevention services for GBMSM [, , - ]. GBMSM are open to receiving prevention information and resources via mobile apps [ ]. Communicating prevention messages through mobile apps (mobile messaging) might enhance intervention uptake, because it allows messaging to a wide audience of GBMSM, including rural GBMSM, who use sexual health services and PrEP at lower rates than GBMSM in urban areas [ , , ]. Younger GBMSM might be especially interested in using mobile technology to receive health information [ ].
HIV prevention and care interventions for GBMSM have most often been provided through in-person sessions with behavior change interventions, are often focused on specific subgroups of GBMSM (eg, high-risk HIV-negative episodic substance-using GBMSM, Black GBMSM with HIV-negative or unknown HIV serostatus, HIV-positive clinic patients)  and often target only 1 element of comprehensive prevention—for example, condom use, medication adherence, or PrEP uptake [ ]. In a serostatus-neutral framework [ ], we must evaluate interventions that address multiple prevention behaviors (eg, HIV testing, sexually transmitted infection [STI] testing, condom use, PrEP uptake, and medication adherence), many of which are relevant to GBMSM with HIV and those at risk for HIV infection.
To address these needs, we adapted an existing HIV prevention app designed for GBMSM, HealthMindr, to add and evaluate tailored prevention messaging [- ]. The methods for the app content development specific to this study have been described elsewhere [ ]. We aimed to develop a mobile app that can address multiple prevention and care needs for GBMSM, both those with HIV and those at risk for HIV acquisition (ie, HIV negative). We developed messaging for HIV-negative GBMSM that would be relevant for both high- and low-risk MSM. Messages were drawn from existing mHealth interventions [ , ] and adapted through a theory-driven approach with stakeholder input, as previously described [ ]. Messages were provided through in-app content and videos. We used the core HealthMindr app [ , ] to offer a suite of prevention services, including self-screening for HIV and STI risk; a scheduling and reminder system for routine HIV and STI testing; a PrEP eligibility screener; a nonoccupational postexposure prophylaxis (nPEP) risk assessment tool; an ordering platform for delivery of at-home HIV- and STI-screening kits and of condoms and lubricants; and service locators for HIV and STI testing, PrEP, nPEP, and HIV treatment and care. The app included branding associating the app with Emory University. Content was frozen during the trial. Participants accessed the app from Apple App Store or Google Play Store and received a study-specific activation code.
We aimed to evaluate the use and efficacy of the combined prevention messaging and app service components as a public health intervention strategy to reduce risks of HIV acquisition and improve HIV care outcomes among GBMSM. We hypothesized that exposure to the message delivery platform within a comprehensive HIV prevention app would result in improvements in participants’ self-reported sexual health, prevention, and care behaviors, compared to GBMSM without access to the app.
Data were collected from January 24, 2018, to October 31, 2019. The methods for the study have been previously described  and are summarized here. Eligibility criteria are summarized in . Men were classified as (1) HIV seropositive, (2) HIV seronegative at high risk (any anal sex not protected by condoms or not taking PrEP, as prescribed, in the past 3 months), and (3) HIV seronegative at low risk (anal sex protected by condoms in the past 3 months or taking PrEP, as prescribed, in the past 3 months, ie, adherent PrEP users). Owning a cell phone was also required, and smartphone literacy was thus a de facto eligibility criterion. This research was reviewed and approved by the Emory University institutional review board (protocol IRB00087684) on June 13, 2016. We collected individual informed consent in person at the study enrollment visit. Consenting participants were randomly assigned to either the immediate intervention group or the waitlist control group in a 1:1 ratio within 36 strata based on the 3 serostatus/risk groups, 3 cities, race/ethnicity (non-Hispanic White vs other race/ethnicity), and phone type (Android vs iOS) [ ]. Participants in the intervention group received one 2-sentence message every other day and a 1-minute video weekly, in addition to access to HIV and STI kit ordering and informational resources.
At in-person baseline visits, participants assigned to the intervention group downloaded and received orientation to the study app and received access to the mobile app and prevention messages based on their risk profiles. Participant contact and retention activities were supported through the Emory Study Management and Retention Tool (SMART). The intervention was provided over 3 months, after which the app was deactivated. Waitlist control participants were not provided access to the intervention app at the baseline visit but continued to receive quarterly surveys. An overview of the study timeline and complete survey instruments have been published and are available online .
There were multiple prevention outcomes, which varied among the 3 risk groups, as previously reported . The specific outcomes and measures used are summarized in .
|Sex at birth|
|Current gender identity|
|Sexual risk behaviors|
|City of residence|
|Other eligibility criteria|
|STIb testing||STI testc in the 3 months before the survey|
|HIV testing||Self-reportedd HIV test in the 3 months before the survey|
|Current PrEPe use||Self-reportedd being on PrEP as of the day of the survey|
|PrEP adherence||Self-reportedd taking at least 25/30 daily pills in the prior 30 days|
|Current ARTf use||Self-reportedd being on ART as of the day of the survey|
|ART adherence||Self-reportedd taking ART, as prescribed, on ≥25 of the past 30 days|
|Anal sex not protected by PrEP or condoms||Self-reported anal sex with a main or casual partner when the participant was not on PrEP AND the insertive partner did not use a condom from start to finishd|
|Anal sex not protected by PrEP or condoms||Self-reported anal sex with a main or casual partner when the insertive partner did not use a condom from start to finishd|
aGBMSM: gay, bisexual, and other men who have sex with men.
bSTI: sexually transmitted infection.
cIncludes Mobile Messaging for Men care kit orders with results.
dSelf-report measures were previously validated as part of the American Men’s Internet Survey .
ePrEP: preexposure prophylaxis.
fART: antiretroviral therapy.
gLWH: living with HIV.
Analyses were stratified by the baseline HIV status/risk group: high-risk HIV-negative, low-risk HIV-negative, and living with HIV (LWH) groups. Counts and relative frequencies were used to describe sociodemographics and sexual history at baseline by intervention status (intervention group vs control group) and HIV status/risk group, and chi-square tests assessed for differences in these characteristics by intervention status (). To account for within-person repeated measures and randomization strata, separate mixed-effect logistic models assessed for association over time of the intervention with each outcome.
A post hoc analysis was conducted to compare control participants with intervention participants who had at least 30 minutes of intervention use in order to assess the potentially mediating effect of the time spent on the app; 30 minutes was the estimated time required to read and rate all written messages and view the video messages. A second post hoc analysis was conducted because not all participants who ordered and received at-home STI test kits returned them. In this second analysis, we defined the intention to test for STIs as either ordering an STI test kit through the mobile app or reporting having taken an STI test. Result reporting was prepared in accordance with Consolidated Standards of Reporting Trials (CONSORT) guidelines .
|Characteristic||Overall (N=1220)||High-riskb HIV-negative (n=427)||Low-riskc HIV-negative (n=410)||HIV-positive (n=383)|
|n (%)||n (%)||P valued (intervention vs control)||n (%)||P value (intervention vs control)||n (%)||P value (intervention vs control)|
|MSM of color||709 (58.11)||205 (48.0)||.89||204 (49.8)||.92||300 (78.3)||.84|
|White||510 (41.80)||222 (52.0)||—f||206 (50.2)||—||82 (21.4)||—|
|18-29||448 (36.72)||210 (49.2)||.13||180 (43.9)||.89||58 (15.1)||.58|
|≥30||772 (63.28)||217 (50.8)||—||230 (56.1)||—||325 (84.9)||—|
|≤High school diploma/General Education Development (GED) test||176 (14.42)||54 (12.6)||.96||31 (7.6)||.46||91 (23.8)||.76|
|Some post–high school education||387 (31.72)||128 (29.9)||—||98 (23.9)||—||161 (42.0)||—|
|4-year college degree||377 (30.90)||141 (33.0)||—||158 (38.5)||—||78 (20.4)||—|
|Some graduate education||277 (22.70)||102 (23.9)||—||123 (30.0)||—||52 (13.6)||—|
|City/metropolitan statistical area|
|Atlanta||473 (38.77)||140 (32.8)||.98||145 (35.4)||.99||188 (49.1)||.98|
|Detroit||333 (27.29)||155 (36.3)||—||121 (29.5)||—||57 (14.9)||—|
|New York City||414 (33.93)||132 (30.9)||—||144 (35.1)||—||138 (36.0)||—|
|Baseline outcome variables|
|STIh testing (3 months)||—||165 (38.4)||.18||254 (61.9)||.27||271 (71)||.09|
|HIV testing (3 months)||—||162 (37.9)||.40||259 (63.2)||.74||—||—|
|Current PrEPi use||—||22 (5.1)||.08||222 (54.1)||.21||—||—|
|PrEP adherence (≥25/30 days)j||—||11 of 22 (50.0)||.36||212 of 222 (95.5)||.99||—||—|
|Current ARTk use||—||—||—||—||—||368 (96.1)||.99|
|ART adherence (≥25/30 days)j||—||—||—||—||—||335 of 368 (91.0)||.06|
|Unprotected anal sex (3 months)l||—||392 (91.8)||.37||51 (12.4)||.73||98 (25.6)||.01|
|Engagement in care (3 months)m||—||45 (10.5)||.10||196 (47.8)||.42||251 (65.5)||.06|
aMSM: men who have sex with men.
bHigh risk (condomless/PrEP-less anal sex, past 3 months).
cLow risk (no condomless/PrEP-less anal sex, past 3 months).
dP value for chi-square testing (difference between intervention and control arms).
eOne HIV-positive participant did not report race/ethnicity.
gTwo high-risk HIV-negative and one HIV-positive participants did not report education level.
hSTI: sexually transmitted infection.
iPrEP: preexposure prophylaxis.
jAmong current users.
kART: antiretroviral therapy.
lCondomless and PrEP-less anal sex for HIV-negative and condomless/detectable viral load for HIV-positive users.
mEngagement in PrEP or ART care.
Recruitment and Randomization
From January 24, 2018, to November 2, 2018, a total of 9966 GBMSM were assessed for eligibility for the randomized controlled trial; reasons for lack of eligibility are shown in. Specifically, 2841 (28.51%) of 9966 GBMSM screened were eligible and provided contact information, of which 1236 (43.51%) attended a baseline study visit, 1226 (99.19%) consented and were randomized to intervention (n=611, 49.84%) and control (n=615, 50.16%) arms ( ).
The analytic sample comprised 1220 GBMSM enrolled in the study, including 608 (49.84%) randomized to the intervention arm and 612 (50.16%) randomized to the control arm (antiretroviral therapy (ART; 368/383, 96.1%) and high ART adherence (335/368, 91%). About 1 in 4 reported anal sex not protected by condoms in the past 3 months (98/383, 25.6%), and most (251/383, 65.5%) reported engaging in HIV care during the prior quarter.). There was no failure of randomization within the 3 risk groups, except for unprotected anal intercourse among LWH GBMSM ( ). Most participants were GBMSM of color (n=378 [30.98%] Black/African American, n=183 [15%] Hispanic/Latino, n=148 [12.13%] other/mixed), were aged ≥30 years (772/1220 [63.28%]), and had at least a 4-year college education (654/1220 [53.61%]). Other indicators of risk and prevention services did not differ between control and intervention arms (data not shown). Self-reported levels of use of prevention and care services were often less than recommended by the Centers for Disease Control and Prevention (CDC). High-risk HIV-negative GBMSM reported moderate levels of STI (165/427, 38.6%) and HIV (162/427, 37.9%) testing in the 3 months before baseline; only 22 of 427 (5.1%) reported using PrEP, and adherence to PrEP was low (11/22, 50%, reported ≥25/30 pills in the prior 30 days). A high proportion (392/427, 91.8%) reported sex not protected by PrEP or condoms in the past 3 months. Among low-risk HIV-negative GBMSM, baseline use of STI (254/410, 61.9%) and HIV (259/410, 63.2%) testing was higher; in addition, 222 of 410 (54.1%) reported being on PrEP, with high (212/222, 95.5%) adherence. Many HIV-negative GBMSM were designated as low risk because they were on PrEP; routine PrEP care includes HIV and STI testing. Among LWH GBMSM, most (271/383, 70.8%) reported recent STI testing, being on
Retention was 1065 of 1226 (86.87%) at 3 months, 1046 of 1226 (85.32%) at 6 months, and 985 of 1226 (80.34%) at 9 months. Factors associated with retention were consistent across time points: retention was higher among participants who were White non-Hispanic, had more education, were employed full-time, and were not homeless. At the 9-month follow-up, higher retention was also associated with gay/homosexual identity (vs bisexual or other identities) and online recruitment (vs venue-based recruitment).
Intervention efficacy was analyzed by HIV status/risk group (). Among high-risk HIV-negative GBMSM, the odds of HIV testing ( ) were higher at the immediate postintervention time point and the odds of current PrEP use ( ) were higher at the 3-month postintervention time point, but not at 6 or 9 months postintervention ( ). Of the high-risk GBMSM in the intervention arm who tested for HIV in the immediate postintervention period (n=126), 16 (12.7%) used an at-home HIV self-test kit provided through the app. As shown in , although the initial prevalence of PrEP use was nonsignificantly lower in the intervention group at baseline, PrEP use rose from 3% at baseline to 15% by the 3-month follow-up assessment and remained at 15% through the 6-month follow-up assessment. Although odds ratio (OR) estimates comparing intervention to control for PrEP adherence were >1.0 at all postintervention time points for high-risk GBMSM, they were not statistically significant, given the small number of high-risk HIV-negative GBMSM taking PrEP.
For low-risk HIV-negative GBMSM, there were no changes in study outcomes associated with assignment to the intervention. For LWH GBMSM, STI testing was significantly lower in the immediate posttest period in the intervention group compared to the control group.
|High-risk HIV-negative MSM (n=427)||Low-risk HIV-negative MSM (n=410)||HIV-positive MSM (n=383)|
|Intervention group, M%b (95% CI)||Control group, M% (95% CI)||aORc|
|Intervention group, M% (95% CI)||Control group, M% (95% CI)||aOR |
|Intervention group, M% (95% CI)||Control group, M% (95% CI)||aOR |
|STId testing (3 months)|
|Baseline||35 (28-43)||41 (34-49)||—e||58 (50-66)||64 (55-71)||—||76 (68-82)||68 (59-75)||—|
|Immediate posttest||42 (35-50)||47 (39-55)||1.07 |
|61 (53-69)||56 (48-65)||1.53 |
|65 (56-73)||72 (63-79)||0.49|
|3 months postintervention||39 (31-47)||45 (37-53)||1.01 |
|63 (54-71)||56 (48-64)||1.65 |
|72 (63-79)||59 (50-68)||1.19 |
|6 months postintervention||37 (30-45)||42 (34-50)||1.06 |
|57 (48-66)||58 (49-66)||1.22 |
|71 (62-78)||63 (54-72)||0.96 |
|HIV testing (3 months)|
|Baseline||48 (39-57)||52 (44-60)||—||73 (65-80)||71 (63-78)||—||—||—||—|
|Immediate posttest||63 (55-71)||51 (42-59)||2.02|
|75 (67-81)||65 (57-73)||1.41 |
|3 months postintervention||47 (39-55)||52 (44-60)||0.98 |
|69 (61-77)||66 (58-73)||1.03 |
|6 months postintervention||53 (45-61)||50 (42-58)||1.34 |
|68 (60-76)||68 (59-75)||0.90 |
|Current PrEPg use|
|Baseline||3 (2-7)||7 (4-11)||—||55 (45-64)||49 (39-58)||—||—||—||—|
|Immediate posttest||9 (6-14)||15 (11-21)||1.26 |
|52 (42-62)||49 (39-58)||0.89 |
|3 months postintervention||15 (10-21)||14 (10-20)||2.41|
|52 (42-61)||48 (39-59)||0.88 |
|6 months postintervention||15 (11-21)||19 (14-26)||1.67 |
|51 (42-61)||49 (39-58)||0.85 |
|PrEP adherence (≥25/30 days)|
|Baseline||35 (10-71)||62 (37-82)||—||95 (89-98)||95 (88-98)||—||—||—||—|
|Immediate posttest||78 (54-92)||80 (61-91)||2.72 |
|91 (82-95)||84 (74-91)||1.66 |
|3 months postintervention||84 (64-94)||74 (55-87)||5.47 |
|92 (84-96)||85 (75-91)||1.84 |
|6 months postintervention||85 (64-94)||92 (77-98)||1.38 |
|91 (82-95)||92 (83-96)||0.88 |
|Current ARTh use|
|Baseline||—||—||—||—||—||—||98 (94-99)||98 (95-99)||—|
|Immediate posttest||—||—||—||—||—||—||97 (92-77)||98 (94-99)||0.83 |
|3 months postintervention||—||—||—||—||—||—||98 (94-99)||96 (92-98)||2.43 |
|6 months postintervention||—||—||—||—||—||—||96 (91-98)||96 (91-98)||1.11 |
|ART adherence (≥25/30 days)|
|Baseline||—||—||—||—||—||—||96 (92-98)||91 (85-95)||—|
|Immediate posttest||—||—||—||—||—||—||93 (87-96)||91 (85-95)||0.58 |
|3 months postintervention||—||—||—||—||—||—||93 (87-96)||92 (85-95)||0.40 |
|6 months postintervention||—||—||—||—||—||—||92 (86-96)||92 (86-96)||0.43 |
|Unprotected anal sex/all partners (3 months)i|
|Baseline||94 (90-97)||92 (87-95)||—||14 (9-20)||15 (10-21)||—||21 (16-28)||34 (27-42)||—|
|Immediate posttest||78 (71-84)||73 (65-80)||0.93 |
|23 (16-31)||28 (21-37)||0.79 |
|20 (14-27)||20 (14-28)||1.93 |
|3 months postintervention||72 (64-79)||76 (67-82)||0.58 |
|22 (15-30)||29 (22-38)||0.75 |
|15 (10-22)||19 (14-27)||1.46 |
|6 months postintervention||70 (61-77)||66 (58-74)||0.82 |
|27 (20-35)||26 (19-35)||1.11 |
|18 (12-25)||18 (12-25)||1.98 |
|Engagement in PrEP/ART care (3 months)|
|Baseline||8 (5-13)||13 (9-19)||—||49 (40-59)||46 (37-55)||—||70 (63-77)||61 (53-68)||—|
|Immediate posttest||16 (11-22)||18 (13-25)||1.38 |
|46 (37-56)||45 (36-55)||0.91 |
|61 (53-69)||58 (50-66)||0.75 |
|3 months postintervention||17 (12-24)||15 (10-22)||1.90 |
|47 (37-57)||44 (35-54)||0.97 |
|68 (60-76)||56 (48-64)||1.11 |
|6 months postintervention||17 (12-24)||20 (14-27)||1.37 |
|42 (33-52)||43 (34-53)||0.82 |
|64 (55-72)||55 (47-63)||0.95 |
aMSM: men who have sex with men.
bM%, model-based mean probability of reporting behavior.
caOR: adjusted odds ratio (for intervention vs control change from baseline; regression models include independent variables of the intervention group [intervention, control], survey time point [baseline, immediate posttest, 3- and 6-month postintervention follow-up], and their interaction).
dSTI: sexually transmitted infection.
gPrEP: preexposure prophylaxis.
hART: antiretroviral therapy.
iCondomless/PrEP-less anal sex for HIV-negative MSM and condomless/detectable viral load for HIV-positive MSM.
Posthoc Analyses of Intervention Effects by Key Demographic Variables and Time Spent on App and STI Testing
Given the potential importance of race/ethnicity, age group, and education level as potential moderators of intervention effects, we conducted stratified analyses of the outcome variables for each of the 3 HIV status/risk groups for the following subgroups: non-Hispanic White versus others; age 18-29 versus ≥30 years; and <4-year college degree versus 4-year degree or more. No meaningful difference in outcomes was identified among stratum-specific estimates. Further, a sensitivity analysis assessed the time spent on the mobile app over the 3-month intervention (control vs ≥30 minutes of intervention use) for the 3 HIV status/risk groups. No meaningful or significant difference in outcomes was identified.
In a posthoc analysis, the intention to test for STIs (ie, ordering an STI test kit or reporting having had an STI test) was higher in all HIV-negative participants (low-risk HIV-negative: aOR 3.1, 95% CI 1.7-5.6; high-risk HIV-negative: aOR 3.5, 95% CI 2.0-6.2; all HIV-negative regardless of risk group: aOR 3.4, 95% CI 2.3-5.1).
We tested an app-based intervention to provide messages to promote HIV prevention and care and provide access to core HIV and STI prevention services to GBMSM. The results showed significant increases in HIV screening during the period of app use among high-risk HIV-negative GBMSM. The results also indicated increased prevalence of PrEP use among high-risk HIV-negative GBMSM in the 3-month period after app use. We identified no protective changes in study outcomes for low-risk HIV-negative GBMSM or LWH GBMSM. Given the high baseline PrEP use and adherence among low-risk HIV-negative GBMSM who did not report recent anal sex not protected by condoms and high levels of ART use and the high baseline adherence among LWH GBMSM, there was little room for improvement statistically among these groups on key outcome variables. We recognize, however, that the vulnerability of GBMSM may change over time (ie, seasons of risk ), so the utility of the app and its messaging might change over time for individual men.
We developed our intervention in light of 3 principles. First, we worked from the paradigm of integrated HIV prevention and care continua. Although the HIV status was determined at the beginning of the trial, when the app is used in practice, HIV testing will be the starting point for both continua. Thus, the intervention could serve GBMSM who do not know their HIV status . Second, we sought to develop an intervention that could be offered to all GBMSM rather than to demographic or risk subsets of men. Third, we sought to develop an intervention that could be scaled to provide intervention content to large numbers of men without the need for a large staff of interventionists. A free, downloadable app that disseminates one 2-sentence message every other day and a 1-minute video weekly might be easily incorporated into a user’s life without requiring the time and travel often required of more traditional, in-person interventions. Further, mobile app interventions are a potentially low-cost strategy from the perspective of health departments and community-based organizations (CBOs) that may choose to implement them in their jurisdictions.
We view our mixed efficacy outcomes from several perspectives. The fact that exposure to the app was associated with a doubling of HIV-testing behaviors and of prevalence of PrEP use among high-risk HIV-negative GBMSM suggests that the messages and app services could be a scalable, effective resource for GBMSM who engage in anal intercourse that is not protected by condoms or PrEP. Large national surveys of GBMSM suggest that 47%-53% of HIV-negative GBMSM would fall into this category [, ]; considering the number of high-risk HIV-negative GBMSM in the United States [ ], the potential user base for these services is about 1,800,000-2,000,000 US GBMSM.
The effect sizes for uptake of HIV screening and PrEP use were both about 2. However, the baseline levels of HIV screening and PrEP use were quite different. For high-risk GBMSM, the CDC recommends HIV testing at least annually, and in the intervention arm, nearly two-thirds reported HIV testing in the immediate posttest assessment period (tested in the prior 3 months). However, despite the same relative increase in PrEP use, only 1 in 6 (17%) high-risk GBMSM was taking PrEP at the 3-month postintervention time point. This suggests that additional and more intensive intervention services will likely be required for high-risk PrEP-eligible men who do not initiate PrEP within 6 months of starting the use of the Mobile Messaging for Men (M-cubed) app.
We also recognize that increases in PrEP usage and HIV testing associated with allocation to the app were short lived. The timing of the effects makes sense: the support for HIV testing provided during the period of app use included ordering at-home kits and navigating to testing locations; the significant increase in testing was observed during the period when men had access to the app, and about 1 in 7 high-risk men who tested for HIV used a mailed-out HIV self-test kit provided through the study. For PrEP, after the decision to start PrEP, there is a long process of making an appointment, undergoing lab tests, filling a prescription, and starting the medication; this outcome was higher in the 3 months after the period of app use. In practice, exposure to the app would not be limited to 3 months, and it is a separate question as to whether changes in behavior would be sustained over time with ongoing exposure to the app. Such a question could be answered in an implementation study.
Our approach was developed with an eye toward being able to reach large numbers of GBMSM with a basic package of prevention services. Given that there was efficacy for some outcomes, there are important questions about how the intervention might be made available to GBMSM who would be likely to benefit from it . Most HIV prevention services are currently provided and funded through CBOs and health departments. Although some CBOs and health departments might have the technical capacity to support app-based prevention services, this capacity is likely quite limited. A currently underway randomized study of an efficacious eHealth intervention (Keep It Up [ ]) will examine the outcomes of centralized versus decentralized implementation approaches, and the results of the study may inform how the M-cubed intervention could be most effectively implemented [ ].
Our study was subject to several limitations. Although we used recruitment quotas to guide recruitment and achieved a balanced sample with respect to the 3 HIV status/risk groups, race/ethnicity, and recruitment venues, our sample was skewed to men with higher education. These selection biases threaten the external generalizability of the study; it is possible that the intervention might have had salutary effects for low-risk HIV-negative GBMSM or LWH GBMSM with lower education. However, although not powered for secondary comparisons, our post hoc stratified analyses did not suggest differential effects by educational attainment. Second, PrEP use in our sample was high at baseline (about 29% among all HIV-negative GBMSM), which is unique in large samples of GBMSM to date. For example, current PrEP use was reported by less than 20% of respondents to the American Men’s Internet Survey (a large, national sample of US GBMSM) in 2017  and by 25% of GBMSM who participated in 23 cities in the 2017 National HIV Behavioral Surveillance study [ ]. Over half of GBMSM in our study who were not on PrEP reported HIV testing in the 3 months before enrollment; this is similar to the proportion who tested for HIV in the 12 months before interviews in the 2016 AMIS survey [ ]. Third, outcomes were self-reported and so might be subject to misclassification because of social desirability bias [ ]. Fourth, our sample was restricted to GBMSM recruited online; this might introduce selection bias, in that men recruited through physical venues, such as sex venues, might have higher or lower levels of risk. Lastly, our sample was limited to GBMSM in 3 urban areas of the United States who reported sex with a man in the past year; patterns of risk behaviors and use of prevention services are known to be different in other urban areas or among rural GBMSM [ , ]. The level of ART use in our LWH MSM was high, offering limited opportunity for improvement. Future studies of the app could condition enrollment on the need for support for ART initiation or maintenance for LWH MSM.
The next steps for the study include secondary analysis of extensive app usage data or paradata  in terms of understanding the ratings of each message and the time spent using various optional aspects of the app descriptively [ ]. We will also conduct further assessments of the potential mediation of outcome effects by app usage to help direct future prevention research and program implementation of mobile apps. Based on our data, the app offered benefits to GBMSM who reported recent anal sex not protected by condoms or PrEP but not to HIV-negative men who did not report sex not protected by condoms of PrEP or LWH men. Therefore, we will consider how the app can be updated and implemented [ , ] for GBMSM engaging in sex not protected by condoms or PrEP and for LWH MSM. Because we found that the app is a successful way to increase the intention to test for STIs and to distribute at-home STI self-test kits, the kits were often not returned and additional research is needed to understand the barriers to returning STI specimen collection kits. If this barrier can be overcome, the M-cubed app might also effectively support increased STI testing for some groups of GBMSM.
We sought to produce and evaluate a mobile app with prevention messaging and services to support comprehensive HIV prevention and care with low levels of prevention staff interaction for all GBMSM, regardless of HIV status or current risk behaviors. Such a serostatus-neutral approach, if implemented successfully, would offer advantages in terms of cost efficiency and minimization of segmentation of GBMSM by serostatus. Future work with the M-cubed app should focus on implementing the app for GBMSM who are likely to benefit from it based on their current behaviors and evaluating methods of implementation of the app through public or publicly supported prevention providers .
This study was funded through the US Centers for Disease Control and Prevention (CDC) to Emory University (cooperative agreement no. U01PS004977) and the US National Institutes of Health (P30AI050409).
CDC disclaimer: The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the CDC.
Conceptualization, formal analysis, funding acquisition, investigation, methodology, supervision, writing (original draft), and writing (review and editing) were performed by PSS. Conceptualization, funding acquisition, investigation, supervision, and writing (review and editing) were performed by RS and SH. Formal analysis, methodology, supervision, writing (review and editing) were conducted by CCM. RZ performed funding acquisition, investigation, supervision, and writing (review and editing); JAB and KH performed funding acquisition, methodology, and writing (review and editing); MAC conducted conceptualization, funding acquisition, investigation, writing (review and editing); DG performed conceptualization, methodology, supervision, and writing (review and editing); SM performed formal analysis, methodology, supervision, and writing (review and editing); and MJD, EJO, and SW conducted investigation, methodology, and writing (review and editing). EQR performed investigation, methodology, supervision, and writing (review and editing); ER and AJS performed conceptualization, funding acquisition, methodology, and writing (review and editing); and GM conducted conceptualization, formal analysis, investigation, methodology, supervision, writing (original draft), and writing (review and editing). All authors read and approved the final manuscript.
The authors have no conflicts of interest to disclose. In this cooperative agreement, the US Centers for Disease Control and Prevention (CDC) scientists participated in the design of this study and will play collaborative roles during its execution, analyses, and interpretation and dissemination of the data.
Conflicts of Interest
CONSORT-eHEALTH checklist (V 1.6.1).PDF File (Adobe PDF File), 736 KB
- Fauci AS, Redfield RR, Sigounas G, Weahkee MD, Giroir BP. Ending the HIV epidemic: a plan for the United States. JAMA 2019 Mar 05;321(9):844-845. [CrossRef] [Medline]
- Sullivan P, Woodyatt C, Koski C, Pembleton E, McGuinness P, Taussig J, et al. A data visualization and dissemination resource to support HIV prevention and care at the local level: analysis and uses of the AIDSVu public data resource. J Med Internet Res 2020 Oct 23;22(10):e23173 [FREE Full text] [CrossRef] [Medline]
- Centers for Disease Control and Prevention. Diagnoses of HIV Infection in the United States and Dependent Areas, 2018. URL: https://www.cdc.gov/hiv/library/reports/hiv-surveillance/vol-31/index.html [accessed 2022-01-19]
- Centers for Disease Control and Prevention. Complete Listing of Risk Reduction Evidence-based Behavioral Interventions Internet. URL: https://www.cdc.gov/hiv/research/interventionresearch/compendium/rr/complete.html [accessed 2022-01-19]
- Maloney KM, Bratcher A, Wilkerson R, Sullivan PS. Electronic and other new media technology interventions for HIV care and prevention: a systematic review. J Int AIDS Soc 2020 Jan;23(1):e25439 [FREE Full text] [CrossRef] [Medline]
- Hightow-Weidman LB, Muessig K, Rosenberg E, Sanchez T, LeGrand S, Gravens L, et al. University of North Carolina/Emory Center for Innovative Technology (iTech) for addressing the HIV epidemic among adolescents and young adults in the United States: protocol and rationale for center development. JMIR Res Protoc 2018 Aug 03;7(8):e10365 [FREE Full text] [CrossRef] [Medline]
- Sullivan PS, Peterson J, Rosenberg ES, Kelley CF, Cooper H, Vaughan A, et al. Understanding racial HIV/STI disparities in black and white men who have sex with men: a multilevel approach. PLoS One 2014 Mar 7;9(3):e90514 [FREE Full text] [CrossRef] [Medline]
- Hightow-Weidman LB, Magnus M, Beauchamp G, Hurt CB, Shoptaw S, Emel L, et al. Incidence and correlates of sexually transmitted infections among black men who have sex with men participating in the HIV Prevention Trials Network 073 Preexposure Prophylaxis Study. Clin Infect Dis 2019 Oct 15;69(9):1597-1604 [FREE Full text] [CrossRef] [Medline]
- Sullivan PS, Satcher Johnson A, Pembleton ES, Stephenson R, Justice AC, Althoff KN, et al. Epidemiology of HIV in the USA: epidemic burden, inequities, contexts, and responses. Lancet 2021 Mar;397(10279):1095-1106. [CrossRef]
- Sullivan PS, Sanchez TH, Zlotorzynska M, Chandler CJ, Sineath R, Kahle E, et al. National trends in HIV pre-exposure prophylaxis awareness, willingness and use among United States men who have sex with men recruited online, 2013 through 2017. J Int AIDS Soc 2020 Mar;23(3):e25461 [FREE Full text] [CrossRef] [Medline]
- Sanchez TH, Zlotorzynska M, Sineath RC, Kahle E, Tregear S, Sullivan PS. National trends in sexual behavior, substance use and HIV testing among United States men who have sex with men recruited online, 2013 through 2017. AIDS Behav 2018 Aug 12;22(8):2413-2425. [CrossRef] [Medline]
- Sullivan PS, Carballo-Diéguez A, Coates T, Goodreau SM, McGowan I, Sanders EJ, et al. Successes and challenges of HIV prevention in men who have sex with men. Lancet 2012 Jul;380(9839):388-399. [CrossRef]
- Jenness SM, Goodreau SM, Rosenberg E, Beylerian EN, Hoover KW, Smith DK, et al. Impact of the Centers for Disease Control's HIV preexposure prophylaxis guidelines for men who have sex with men in the United States. J Infect Dis 2016 Dec 15;214(12):1800-1807 [FREE Full text] [CrossRef] [Medline]
- Brookmeyer R, Boren D, Baral SD, Bekker LG, Phaswana-Mafuya N, Beyrer C, et al. Combination HIV prevention among MSM in South Africa: results from agent-based modeling. PLoS One 2014 Nov 14;9(11):e112668 [FREE Full text] [CrossRef] [Medline]
- Hightow-Weidman LB, Muessig KE, Bauermeister J, Zhang C, LeGrand S. Youth, technology, and HIV: recent advances and future directions. Curr HIV/AIDS Rep 2015 Dec;12(4):500-515 [FREE Full text] [CrossRef] [Medline]
- Muessig KE, Nekkanti M, Bauermeister J, Bull S, Hightow-Weidman LB. A systematic review of recent smartphone, internet and Web 2.0 interventions to address the HIV continuum of care. Curr HIV/AIDS Rep 2015 Mar 28;12(1):173-190 [FREE Full text] [CrossRef] [Medline]
- Muessig KE, Pike EC, Legrand S, Hightow-Weidman LB. Mobile phone applications for the care and prevention of HIV and other sexually transmitted diseases: a review. J Med Internet Res 2013 Jan 04;15(1):e1 [FREE Full text] [CrossRef] [Medline]
- Sullivan P, Grey J, Simon Rosser BR. Emerging technologies for HIV prevention for MSM: what we have learned, and ways forward. J Acquir Immune Defic Syndr 2013 Jun 01;63 Suppl 1:S102-S107 [FREE Full text] [CrossRef] [Medline]
- Sullivan PS, Jones J, Kishore N, Stephenson R. The roles of technology in primary HIV prevention for men who have sex with men. Curr HIV/AIDS Rep 2015 Dec 30;12(4):481-488. [CrossRef] [Medline]
- Muessig KE, Pike EC, Fowler B, LeGrand S, Parsons JT, Bull SS, et al. Putting prevention in their pockets: developing mobile phone-based HIV interventions for black men who have sex with men. AIDS Patient Care STDS 2013 Apr;27(4):211-222 [FREE Full text] [CrossRef] [Medline]
- McKenney J, Sullivan PS, Bowles KE, Oraka E, Sanchez TH, DiNenno E. HIV risk behaviors and utilization of prevention services, urban and rural men who have sex with men in the United States: results from a national online survey. AIDS Behav 2018 Jul 6;22(7):2127-2136. [CrossRef] [Medline]
- Khosropour CM, Lake JG, Sullivan PS. Are MSM willing to SMS for HIV prevention? J Health Commun 2014 Aug 26;19(1):57-66. [CrossRef] [Medline]
- Compendium of Evidence-Based Interventions and Best Practices for HIV Prevention Internet. URL: http://www.cdc.gov/hiv/prevention/research/compendium/index.html [accessed 2022-01-19]
- Fernandez MI, Harper GW, Hightow-Weidman LB, Kapogiannis BG, Mayer KH, Parsons JT, et al. Research priorities to end the adolescent HIV epidemic in the United States: viewpoint. JMIR Res Protoc 2021 Jan 04;10(1):e22279 [FREE Full text] [CrossRef] [Medline]
- Sullivan PS, Driggers R, Stekler JD, Siegler A, Goldenberg T, McDougal SJ, et al. Usability and acceptability of a mobile comprehensive HIV prevention app for men who have sex with men: a pilot study. JMIR Mhealth Uhealth 2017 Mar 09;5(3):e26 [FREE Full text] [CrossRef] [Medline]
- Goldenberg T, McDougal SJ, Sullivan PS, Stekler JD, Stephenson R. Building a mobile HIV prevention app for men who have sex with men: an iterative and community-driven process. JMIR Public Health Surveill 2015 Nov 16;1(2):e18 [FREE Full text] [CrossRef] [Medline]
- Goldenberg T, McDougal SJ, Sullivan PS, Stekler JD, Stephenson R. Preferences for a mobile HIV prevention app for men who have sex with men. JMIR Mhealth Uhealth 2014 Oct 29;2(4):e47 [FREE Full text] [CrossRef] [Medline]
- Sullivan PS, Zahn RJ, Wiatrek S, Chandler CJ, Hirshfield S, Stephenson R, et al. HIV prevention via mobile messaging for men who have sex with men (M-cubed): protocol for a randomized controlled trial. JMIR Res Protoc 2019 Nov 15;8(11):e16439. [CrossRef]
- Bauermeister JA, Golinkoff JM, Horvath KJ, Hightow-Weidman LB, Sullivan PS, Stephenson R. A multilevel tailored web app-based intervention for linking young men who have sex with men to quality care (get connected): protocol for a randomized controlled trial. JMIR Res Protoc 2018 Aug 02;7(8):e10444 [FREE Full text] [CrossRef] [Medline]
- Horvath K, Amico K, Erickson D, Ecklund A, Martinka A, DeWitt J, et al. Thrive with me: protocol for a randomized controlled trial to test a peer support intervention to improve antiretroviral therapy adherence among men who have sex with men. JMIR Res Protoc 2018 May 31;7(5):e10182 [FREE Full text] [CrossRef] [Medline]
- Kernan W, Viscoli C, Makuch R, Brass L, Horwitz R. Stratified randomization for clinical trials. J Clin Epidemiol 1999 Jan;52(1):19-26. [CrossRef] [Medline]
- Sanchez TH, Sineath RC, Kahle EM, Tregear SJ, Sullivan PS. The Annual American Men's Internet Survey of behaviors of men who have sex with men in the United States: protocol and key indicators report 2013. JMIR Public Health Surveill 2015 Apr 17;1(1):e3 [FREE Full text] [CrossRef] [Medline]
- Schulz KF, Altman DG, Moher D, CONSORT Group. CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials. Ann Intern Med 2010 Jun 01;152(11):726-732 [FREE Full text] [CrossRef] [Medline]
- Elsesser SA, Oldenburg CE, Biello KB, Mimiaga MJ, Safren SA, Egan JE, et al. Seasons of risk: anticipated behavior on vacation and interest in episodic antiretroviral pre-exposure prophylaxis (PrEP) among a large national sample of U.S. men who have sex with men (MSM). AIDS Behav 2016 Jul 4;20(7):1400-1407 [FREE Full text] [CrossRef] [Medline]
- Myers JE, Braunstein SL, Xia Q, Scanlin K, Edelstein Z, Harriman G, et al. Redefining prevention and care: a status-neutral approach to HIV. Open Forum Infect Dis 2018 Jun;5(6):ofy097 [FREE Full text] [CrossRef] [Medline]
- Zlotorzynska M, Cantu C, Rai R, Sullivan P, Sanchez T. The Annual American Men's Internet Survey of behaviors of men who have sex with men in the United States: 2017 key indicators report. JMIR Public Health Surveill 2020 Apr 13;6(2):e16847 [FREE Full text] [CrossRef] [Medline]
- National HIV Behavioral Surveillance (NHBS) Study Group. HIV Infection Risk, Prevention, and Testing Behaviors Among Men Who Have Sex With Men? National HIV Behavioral Surveillance, 23 U.S. Cities, 2017. 2019 Feb. URL: https://www.cdc.gov/hiv/pdf/library/reports/surveillance/cdc-hiv-surveillance-special-report-number-22.pdf [accessed 2022-01-19]
- Rosenberg ES, Grey JA, Sanchez TH, Sullivan PS. Rates of prevalent HIV infection, prevalent diagnoses, and new diagnoses among men who have sex with men in US states, metropolitan statistical areas, and counties, 2012-2013. JMIR Public Health Surveill 2016 May 17;2(1):e22 [FREE Full text] [CrossRef] [Medline]
- Li DH, Brown CH, Gallo C, Morgan E, Sullivan PS, Young SD, et al. Design considerations for implementing eHealth behavioral interventions for HIV prevention in evolving sociotechnical landscapes. Curr HIV/AIDS Rep 2019 Aug 27;16(4):335-348 [FREE Full text] [CrossRef] [Medline]
- Mustanski B, Parsons JT, Sullivan PS, Madkins K, Rosenberg E, Swann G. Biomedical and behavioral outcomes of Keep It Up! An eHealth HIV prevention program RCT. Am J Prev Med 2018 Aug;55(2):151-158 [FREE Full text] [CrossRef] [Medline]
- Mustanski B, Jones J, Macapagal K, Benbow N, Li D, Hendricks BC, et al. Keep It Up! 3.0: Study Protocol for a Type III Hybrid Implementation-Effectiveness Cluster-Randomized Trial Internet. 202 Nov 13. URL: https://www.researchsquare.com/article/rs-104488/v1 [accessed 2022-01-19]
- Sackett DL. Bias in analytic research. J Chronic Dis 1979 Jan;32(1-2):51-63. [CrossRef]
- Bauermeister J, Golinkoff J, Muessig K, Horvath K, Hightow-Weidman L. Addressing engagement in technology-based behavioural HIV interventions through paradata metrics. Curr Opin HIV AIDS 2017;12(5):446. [CrossRef]
- Hightow-Weidman LB, Bauermeister JA. Engagement in mHealth behavioral interventions for HIV prevention and care: making sense of the metrics. Mhealth 2020;6:7 [FREE Full text] [CrossRef] [Medline]
- Sullivan PS, Hightow-Weidman L. Mobile apps for HIV prevention: how do they contribute to our epidemic response for adolescents and young adults? Mhealth 2021;7:36 [FREE Full text] [CrossRef] [Medline]
|aOR: adjusted odds ratio|
|ART: antiretroviral therapy|
|CBO: community-based organization|
|CDC: Centers for Disease Control and Prevention|
|CONSORT: Consolidated Standards of Reporting Trials|
|GBMSM: gay, bisexual, and other men who have sex with men|
|LWH: living with HIV|
|M-cubed: Mobile Messaging for Men|
|mHealth: mobile health|
|MSM: men who have sex with men|
|nPEP: nonoccupational postexposure prophylaxis|
|OR: odds ratio|
|PrEP: preexposure prophylaxis|
|SMART: Study Management and Retention Tool|
|STI: sexually transmitted infection|
|YGBMSM: young gay, bisexual, and other men who have sex with men|
Edited by A Mavragani; submitted 29.11.21; peer-reviewed by P Serrano; comments to author 20.12.21; revised version received 20.12.21; accepted 21.12.21; published 02.02.22Copyright
©Patrick Sean Sullivan, Rob Stephenson, Sabina Hirshfield, Cyra Christina Mehta, Ryan Zahn, Jose A Bauermeister, Keith Horvath, Mary Ann Chiasson, Deborah Gelaude, Shelby Mullin, Martin J Downing Jr, Evelyn Jolene Olansky, Sarah Wiatrek, Erin Q Rogers, Eli Rosenberg, Aaron J Siegler, Gordon Mansergh. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 02.02.2022.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.