Published on in Vol 27 (2025)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/57619, first published .
Studying the Digital Intervention Engagement–Mediated Relationship Between Intrapersonal Measures and Pre-Exposure Prophylaxis Adherence in Sexual and Gender Minority Youth: Secondary Analysis of a Randomized Controlled Trial

Studying the Digital Intervention Engagement–Mediated Relationship Between Intrapersonal Measures and Pre-Exposure Prophylaxis Adherence in Sexual and Gender Minority Youth: Secondary Analysis of a Randomized Controlled Trial

Studying the Digital Intervention Engagement–Mediated Relationship Between Intrapersonal Measures and Pre-Exposure Prophylaxis Adherence in Sexual and Gender Minority Youth: Secondary Analysis of a Randomized Controlled Trial

Original Paper

1Bouve College of Health Sciences, Northeastern University, Boston, MA, United States

2Roux Institute, Northeastern University, Portland, ME, United States

3College of Nursing, Florida State University, Tallahassee, FL, United States

4Department of Community Health Sciences, Boston University, Boston, MA, United States

Corresponding Author:

Michael P Williams, PhD

Bouve College of Health Sciences

Northeastern University

30 Leon St

Boston, MA, 02115

United States

Phone: 1 617 373 3323

Email: mpw144@gmail.com


Background: Improving adherence to pre-exposure prophylaxis (PrEP) via digital health interventions (DHIs) for young sexual and gender minority men who have sex with men (YSGMMSM) is promising for reducing the HIV burden. Measuring and achieving effective engagement (sufficient to solicit PrEP adherence) in YSGMMSM is challenging.

Objective: This study is a secondary analysis of the primary efficacy randomized controlled trial (RCT) of Prepared, Protected, Empowered (P3), a digital PrEP adherence intervention that used causal mediation to quantify whether and to what extent intrapersonal behavioral, mental health, and sociodemographic measures were related to effective engagement for PrEP adherence in YSGMMSM.

Methods: In May 2019, 264 YSGMMSM were recruited for the primary RCT via social media, community sites, and clinics from 9 study sites across the United States. For this secondary analysis, 140 participants were eligible (retained at follow-up, received DHI condition in primary RCT, and completed trial data). Participants earned US currency for daily use of P3 and lost US currency for nonuse. Dollars accrued at the 3-month follow-up were used to measure engagement. PrEP nonadherence was defined as blood serum concentrations of tenofovir-diphosphate and emtricitabine-triphosphate that correlated with ≤4 doses weekly at the 3-month follow-up. Logistic regression was used to estimate the total effect of baseline intrapersonal measures on PrEP nonadherence, represented as odds ratios (ORs) with a null value of 1. The total OR for each intrapersonal measure was decomposed into direct and indirect effects.

Results: For every US $1 earned above the mean (US $96, SD US $35.1), participants had 2% (OR 0.98, 95% CI 0.97-0.99) lower odds of PrEP nonadherence. Frequently using phone apps to track health information was associated with a 71% (OR 0.29, 95% CI 0.06-0.96) lower odds of PrEP nonadherence. This was overwhelmingly a direct effect, not mediated by engagement, with a percentage mediated (PM) of 1%. Non-Hispanic White participants had 83% lower odds of PrEP nonadherence (OR 0.17, 95% CI 0.05-0.48) and had a direct effect (PM=4%). Participants with depressive symptoms and anxiety symptoms had 3.4 (OR 3.42, 95% CI 0.95-12) and 3.5 (OR 3.51, 95% CI 1.06-11.55) times higher odds of PrEP nonadherence, respectively. Anxious symptoms largely operated through P3 engagement (PM=51%).

Conclusions: P3 engagement (dollars accrued) was strongly related to lower odds of PrEP nonadherence. Intrapersonal measures operating through P3 engagement (indirect effect, eg, anxious symptoms) suggest possible pathways to improve PrEP adherence DHI efficacy in YSGMMSM via effective engagement. Conversely, the direct effects observed in this study may reflect existing structural disparity (eg, race and ethnicity) or behavioral dispositions toward technology (eg, tracking health via phone apps). Evaluating effective engagement in DHIs with causal mediation approaches provides a clarifying and mechanistic view of how DHIs impact health behavior.

Trial Registration: ClinicalTrials.gov; NCT03320512; https://clinicaltrials.gov/study/NCT03320512

J Med Internet Res 2025;27:e57619

doi:10.2196/57619

Keywords



Background

Young sexual and gender minority men who have sex with men (YSGMMSM) are burdened with a disproportionate and growing vulnerability to HIV in the United States [Baral SD, Poteat T, Strömdahl S, Wirtz AL, Guadamuz TE, Beyrer C. Worldwide burden of HIV in transgender women: a systematic review and meta-analysis. Lancet Infect Dis. Mar 2013;13(3):214-222. [CrossRef] [Medline]1-HIV surveillance report, 2020. Centers for Disease Control and Prevention. May 2022. URL: https://tinyurl.com/28s94jhy [accessed 2023-09-05] 8]. Of the 30,692 incident HIV cases in 2020, 71% were among men who have sex with men (MSM) and 24% were among MSM aged 13 to 24 years [HIV surveillance report, 2020. Centers for Disease Control and Prevention. May 2022. URL: https://tinyurl.com/28s94jhy [accessed 2023-09-05] 8]. In addition, a meta-analysis of transgender women aged ≥15 years found that they were 48 times more likely to have HIV infection compared to other adults of reproductive age [Baral SD, Poteat T, Strömdahl S, Wirtz AL, Guadamuz TE, Beyrer C. Worldwide burden of HIV in transgender women: a systematic review and meta-analysis. Lancet Infect Dis. Mar 2013;13(3):214-222. [CrossRef] [Medline]1]. While pre-exposure prophylaxis (PrEP) has demonstrated efficacy in reducing the incidence of HIV infections in YSGMMSM in the US in randomized controlled trials (RCTs) [Buchbinder SP, Glidden DV, Liu AY, McMahan V, Guanira JV, Mayer KH, et al. HIV pre-exposure prophylaxis in men who have sex with men and transgender women: a secondary analysis of a phase 3 randomised controlled efficacy trial. Lancet Infect Dis. Jun 2014;14(6):468-475. [FREE Full text] [CrossRef] [Medline]9,Grant RM, Lama JR, Anderson PL, McMahan V, Liu AY, Vargas L, et al. Preexposure chemoprophylaxis for HIV prevention in men who have sex with men. N Engl J Med. Dec 30, 2010;363(27):2587-2599. [FREE Full text] [CrossRef] [Medline]10], adherence to these medications outside of clinical trial settings has been suboptimal for reducing transmission [Buchbinder SP, Glidden DV, Liu AY, McMahan V, Guanira JV, Mayer KH, et al. HIV pre-exposure prophylaxis in men who have sex with men and transgender women: a secondary analysis of a phase 3 randomised controlled efficacy trial. Lancet Infect Dis. Jun 2014;14(6):468-475. [FREE Full text] [CrossRef] [Medline]9,Grant RM, Anderson PL, McMahan V, Liu A, Amico KR, Mehrotra M, et al. Uptake of pre-exposure prophylaxis, sexual practices, and HIV incidence in men and transgender women who have sex with men: a cohort study. Lancet Infect Dis. Sep 2014;14(9):820-829. [FREE Full text] [CrossRef] [Medline]11]. Furthermore, findings from population and agent-based simulation studies demonstrate that PrEP uptake and adherence are associated with a decrease in incident HIV infections, lowering incidence by as much as 25% [Callander D, McManus H, Gray RT, Grulich AE, Carr A, Hoy J, et al. HIV treatment-as-prevention and its effect on incidence of HIV among cisgender gay, bisexual, and other men who have sex with men in Australia: a 10-year longitudinal cohort study. The Lancet HIV. Jun 2023;10(6):e385-e393. [CrossRef]12,LeVasseur MT, Goldstein ND, Tabb LP, Olivieri-Mui BL, Welles SL. The effect of PrEP on HIV incidence among men who have sex with men in the context of condom use, treatment as prevention, and seroadaptive practices. J Acquir Immune Defic Syndr. Jan 01, 2018;77(1):31-40. [CrossRef] [Medline]13]. Altogether, this suggests that improving PrEP adherence is a fruitful pathway to reducing HIV incidence in YSGMMSM [Buchbinder SP, Glidden DV, Liu AY, McMahan V, Guanira JV, Mayer KH, et al. HIV pre-exposure prophylaxis in men who have sex with men and transgender women: a secondary analysis of a phase 3 randomised controlled efficacy trial. Lancet Infect Dis. Jun 2014;14(6):468-475. [FREE Full text] [CrossRef] [Medline]9,Grant RM, Anderson PL, McMahan V, Liu A, Amico KR, Mehrotra M, et al. Uptake of pre-exposure prophylaxis, sexual practices, and HIV incidence in men and transgender women who have sex with men: a cohort study. Lancet Infect Dis. Sep 2014;14(9):820-829. [FREE Full text] [CrossRef] [Medline]11-LeVasseur MT, Goldstein ND, Tabb LP, Olivieri-Mui BL, Welles SL. The effect of PrEP on HIV incidence among men who have sex with men in the context of condom use, treatment as prevention, and seroadaptive practices. J Acquir Immune Defic Syndr. Jan 01, 2018;77(1):31-40. [CrossRef] [Medline]13].

Digital health interventions (DHIs) are potentially powerful and increasingly prevalent mechanisms for delivering PrEP interventions to YSGMMSM [Bauermeister JA, Golinkoff JM, Horvath KJ, Hightow-Weidman LB, Sullivan PS, Stephenson R. A multilevel tailored web app-based intervention for linking young men who have sex with men to quality care (get connected): protocol for a randomized controlled trial. JMIR Res Protoc. Aug 02, 2018;7(8):e10444. [FREE Full text] [CrossRef] [Medline]14-Songtaweesin WN, LeGrand S, Bandara S, Piccone C, Wongharn P, Moonwong J, et al. Adaptation of a theory-based social networking and gamified app-based intervention to improve pre-exposure prophylaxis adherence among young men who have sex with men in Bangkok, Thailand: qualitative study. J Med Internet Res. Nov 04, 2021;23(11):e23852. [FREE Full text] [CrossRef] [Medline]20]. Due to the pervasiveness of digital communication and entertainment among youth, including YSGMMSM, DHIs are suitable delivery mechanisms for HIV prevention interventions due to their potential to effectively engage participants [Chiasson MA, Hirshfield S, Rietmeijer C. HIV prevention and care in the digital age. J Acquir Immune Defic Syndr. Dec 2010;55 Suppl 2:S94-S97. [CrossRef] [Medline]21-Yardley L, Spring BJ, Riper H, Morrison LG, Crane DH, Curtis K, et al. Understanding and promoting effective engagement with digital behavior change interventions. Am J Prev Med. Nov 2016;51(5):833-842. [CrossRef] [Medline]25]. However, to our knowledge, there is a paucity of clinical trials directly testing the efficacy of digital PrEP adherence interventions. Adjacent digital PrEP adherence interventions, such as automated SMS text-messaging services or digital pill systems, have had mixed results in young populations. For example, digital pill systems have documented barriers to engagement and have not shown that they independently improve PrEP adherence compared to a standard of care [Brothers J, Hosek S, Keckler K, Anderson PL, Xiong D, Liu H, et al. The ATEAM study: advances in technology to enhance PrEP adherence monitoring (ATEAM) among young men who have sex with men. Clin Transl Sci. Dec 2022;15(12):2947-2957. [FREE Full text] [CrossRef] [Medline]26-Serrano VB, Moore DJ, Morris S, Tang B, Liao A, Hoenigl M, et al. Efficacy of daily text messaging to support adherence to HIV pre-exposure prophylaxis (PrEP) among stimulant-using men who have sex with men. Subst Use Misuse. 2023;58(3):465-469. [FREE Full text] [CrossRef] [Medline]29]. Furthermore, DHIs implemented to address other health problems in target youth populations have encountered problems in effectively engaging the participants. For example, 1 study that developed a mobile app for self-management of type 1 diabetes among adolescents found no association between intervention conditions and primary or secondary clinical outcomes. Furthermore, the study found that only 9% of the participants met the criteria for high engagement levels (measured as an individual uploading blood glucose readings for ≥3 d/wk) [Goyal S, Nunn CA, Rotondi M, Couperthwaite AB, Reiser S, Simone A, et al. A mobile app for the self-management of type 1 diabetes among adolescents: a randomized controlled trial. JMIR Mhealth Uhealth. Jun 19, 2017;5(6):e82. [FREE Full text] [CrossRef] [Medline]30]. For these reasons, effectively engaging YSGMMSM in PrEP adherence DHIs is likely an essential element to achieving protective levels of PrEP adherence [Yardley L, Spring BJ, Riper H, Morrison LG, Crane DH, Curtis K, et al. Understanding and promoting effective engagement with digital behavior change interventions. Am J Prev Med. Nov 2016;51(5):833-842. [CrossRef] [Medline]25,Hightow-Weidman LB, Horvath KJ, Scott H, Hill-Rorie J, Bauermeister JA. Engaging youth in mHealth: what works and how can we be sure? Mhealth. 2021;7:23. [FREE Full text] [CrossRef] [Medline]31,Hightow-Weidman LB, Bauermeister JA. Engagement in mHealth behavioral interventions for HIV prevention and care: making sense of the metrics. Mhealth. Jan 2020;6:7. [FREE Full text] [CrossRef] [Medline]32].

The effective engagement framework by Yardley et al [Yardley L, Spring BJ, Riper H, Morrison LG, Crane DH, Curtis K, et al. Understanding and promoting effective engagement with digital behavior change interventions. Am J Prev Med. Nov 2016;51(5):833-842. [CrossRef] [Medline]25] defines effective engagement as “sufficient engagement with the intervention to achieve intended outcomes.” They describe four phases of engagement as follows: (1) initial engagement with the DHI in preparation for behavior change; (2) engagement with the behavior change, mediated by the DHI; (3) DHI use may no longer be required to sustain behavior change; and (4) reengagement with DHI as needed. Furthermore, this framework describes how, in phase 1, effective engagement is largely characterized by a micro form of engagement (defined by moment-to-moment interactions with the DHI). As individuals progress through phases 2, 3, and 4, they are increasingly engaged with a macro form of engagement (defined as engagement that relates to the overarching goal of the intervention).

The effective engagement framework by Yardley et al [Yardley L, Spring BJ, Riper H, Morrison LG, Crane DH, Curtis K, et al. Understanding and promoting effective engagement with digital behavior change interventions. Am J Prev Med. Nov 2016;51(5):833-842. [CrossRef] [Medline]25] also highlights how intrapersonal, social, and environmental characteristics can influence effective engagement and how tailoring DHIs based on those characteristics may improve effective engagement. Tailoring, defined as the use of individuals’ data to customize intervention content based on their psychological, socioecological, and behavioral profile, is a property of DHIs that has demonstrated efficacy in promoting effective engagement with DHIs [Yardley L, Spring BJ, Riper H, Morrison LG, Crane DH, Curtis K, et al. Understanding and promoting effective engagement with digital behavior change interventions. Am J Prev Med. Nov 2016;51(5):833-842. [CrossRef] [Medline]25,Hawkins RP, Kreuter M, Resnicow K, Fishbein M, Dijkstra A. Understanding tailoring in communicating about health. Health Educ Res. Jun 2008;23(3):454-466. [FREE Full text] [CrossRef] [Medline]33-Noar SM, Benac CN, Harris MS. Does tailoring matter? Meta-analytic review of tailored print health behavior change interventions. Psychol Bull. Jul 2007;133(4):673-693. [CrossRef] [Medline]37]. Several barriers and facilitators to engagement have been identified in qualitative research [O'Connor S, Hanlon P, O'Donnell CA, Garcia S, Glanville J, Mair FS. Understanding factors affecting patient and public engagement and recruitment to digital health interventions: a systematic review of qualitative studies. BMC Med Inform Decis Mak. Sep 15, 2016;16(1):120. [FREE Full text] [CrossRef] [Medline]38]. For example, baseline motivation to change [Yardley L, Spring BJ, Riper H, Morrison LG, Crane DH, Curtis K, et al. Understanding and promoting effective engagement with digital behavior change interventions. Am J Prev Med. Nov 2016;51(5):833-842. [CrossRef] [Medline]25,Danaher BG, Seeley JR. Methodological issues in research on web-based behavioral interventions. Ann Behav Med. Aug 2009;38(1):28-39. [FREE Full text] [CrossRef] [Medline]39] or baseline comfort with the intervention modality (eg, phone or app-based intervention) may influence engagement and subsequent behavior change [O'Connor S, Hanlon P, O'Donnell CA, Garcia S, Glanville J, Mair FS. Understanding factors affecting patient and public engagement and recruitment to digital health interventions: a systematic review of qualitative studies. BMC Med Inform Decis Mak. Sep 15, 2016;16(1):120. [FREE Full text] [CrossRef] [Medline]38-Lorimer K, Martin S, McDaid LM. The views of general practitioners and practice nurses towards the barriers and facilitators of proactive, internet-based chlamydia screening for reaching young heterosexual men. BMC Fam Pract. Jun 27, 2014;15:127. [FREE Full text] [CrossRef] [Medline]49]. Furthermore, aspects beyond the individual’s control, such as internet access, may influence engagement and therefore intervention efficacy [O'Connor S, Hanlon P, O'Donnell CA, Garcia S, Glanville J, Mair FS. Understanding factors affecting patient and public engagement and recruitment to digital health interventions: a systematic review of qualitative studies. BMC Med Inform Decis Mak. Sep 15, 2016;16(1):120. [FREE Full text] [CrossRef] [Medline]38,Mac Domhnaill C, Mohan G, McCoy S. Home broadband and student engagement during COVID-19 emergency remote teaching. Distance Educ. Nov 03, 2021;42(4):465-493. [CrossRef]50-Mesch GS, Talmud I. Internet connectivity, community participation, and place attachment: a longitudinal study. Am Behav Sci. Feb 18, 2010;53(8):1095-1110. [CrossRef]52]. However, there is a lack of quantitative research that aims to isolate and quantify the impact that intrapersonal psychological, sociodemographic, and behavioral measures have on engagement and subsequent behavioral outcomes of interest.

Yardley et al [Yardley L, Spring BJ, Riper H, Morrison LG, Crane DH, Curtis K, et al. Understanding and promoting effective engagement with digital behavior change interventions. Am J Prev Med. Nov 2016;51(5):833-842. [CrossRef] [Medline]25] and other review articles using their effective engagement framework describe a plethora of issues with quantitively operationalizing and measuring effective engagement in previous research. Most notably, engagement research to date is largely correlational, relies on the assumption that engagement is intrinsically a precursor to the intended outcome, and does not account for intrapersonal measures, such as motivation to use the intervention or digital literacy [Yardley L, Spring BJ, Riper H, Morrison LG, Crane DH, Curtis K, et al. Understanding and promoting effective engagement with digital behavior change interventions. Am J Prev Med. Nov 2016;51(5):833-842. [CrossRef] [Medline]25,Hightow-Weidman LB, Horvath KJ, Scott H, Hill-Rorie J, Bauermeister JA. Engaging youth in mHealth: what works and how can we be sure? Mhealth. 2021;7:23. [FREE Full text] [CrossRef] [Medline]31,Hightow-Weidman LB, Bauermeister JA. Engagement in mHealth behavioral interventions for HIV prevention and care: making sense of the metrics. Mhealth. Jan 2020;6:7. [FREE Full text] [CrossRef] [Medline]32,Danaher BG, Seeley JR. Methodological issues in research on web-based behavioral interventions. Ann Behav Med. Aug 2009;38(1):28-39. [FREE Full text] [CrossRef] [Medline]39,Nahum-Shani I, Shaw SD, Carpenter SM, Murphy SA, Yoon C. Engagement in digital interventions. Am Psychol. Oct 2022;77(7):836-852. [FREE Full text] [CrossRef] [Medline]53-Van Gemert-Pijnen JE, Kelders SM, Bohlmeijer ET. Understanding the usage of content in a mental health intervention for depression: an analysis of log data. J Med Internet Res. Jan 31, 2014;16(1):e27. [FREE Full text] [CrossRef] [Medline]55]. This highlights the need to empirically test and corroborate the models of engagement, by modeling how intrapersonal measures influence engagement, which subsequently mediates behavioral outcomes of interest (eg, PrEP adherence) [Yardley L, Spring BJ, Riper H, Morrison LG, Crane DH, Curtis K, et al. Understanding and promoting effective engagement with digital behavior change interventions. Am J Prev Med. Nov 2016;51(5):833-842. [CrossRef] [Medline]25,Nahum-Shani I, Shaw SD, Carpenter SM, Murphy SA, Yoon C. Engagement in digital interventions. Am Psychol. Oct 2022;77(7):836-852. [FREE Full text] [CrossRef] [Medline]53,Michie S, Yardley L, West R, Patrick K, Greaves F. Developing and evaluating digital interventions to promote behavior change in health and health care: recommendations resulting from an international workshop. J Med Internet Res. Jun 29, 2017;19(6):e232. [FREE Full text] [CrossRef] [Medline]56]. Causal mediation analysis is a fitting statistical framework for investigating the relationship between intrapersonal measures and effective engagement which addresses the aforementioned methodological issues as well. Causal mediation has been extensively explicated [Vansteelandt S, Bekaert M, Lange T. Imputation strategies for the estimation of natural direct and indirect effects. Epidemiol Methods. 2012;1(1):130. [CrossRef]57-VanderWeele TJ. Explanation in Causal Inference: Methods for Mediation and Interaction. New York, NY. Oxford University Press; 2015. 62], applied in previous research in other domains [Oberg AS, VanderWeele TJ, Almqvist C, Hernandez-Diaz S. Pregnancy complications following fertility treatment-disentangling the role of multiple gestation. Int J Epidemiol. Aug 01, 2018;47(4):1333-1342. [FREE Full text] [CrossRef] [Medline]63], and addresses the aforementioned issues by Yardley et al [Yardley L, Spring BJ, Riper H, Morrison LG, Crane DH, Curtis K, et al. Understanding and promoting effective engagement with digital behavior change interventions. Am J Prev Med. Nov 2016;51(5):833-842. [CrossRef] [Medline]25] by clarifying assumptions needed for causal inference in mediation models, explicitly modeling the relationship between engagement and outcome (as opposed to relying on the assumption that engagement and the outcome are related), and incorporates intrapersonal measures (as exposures or controls) [Vansteelandt S, Bekaert M, Lange T. Imputation strategies for the estimation of natural direct and indirect effects. Epidemiol Methods. 2012;1(1):130. [CrossRef]57,VanderWeele TJ. Mediation analysis: a practitioner's guide. Annu Rev Public Health. 2016;37:17-32. [CrossRef] [Medline]58,Robins JM, Greenland S. Identifiability and exchangeability for direct and indirect effects. Epidemiology. Mar 1992;3(2):143-155. [CrossRef] [Medline]60]. This approach is derived from the counterfactual causal inference framework [Hernán MA. A definition of causal effect for epidemiological research. J Epidemiol Community Health. Apr 2004;58(4):265-271. [FREE Full text] [CrossRef] [Medline]64,Höfler M. Causal inference based on counterfactuals. BMC Med Res Methodol. Sep 13, 2005;5:28. [FREE Full text] [CrossRef] [Medline]65] and allows for the total effect (eg, the effect of baseline digital literacy on PrEP adherence) to be decomposed into a direct and indirect effect. The direct effect models the effect of a given intrapersonal measure on PrEP adherence controlling for engagement with the intervention. Conversely, the indirect effect models how a given intrapersonal measure is related to PrEP adherence operating through engagement, and thus, is an excellent measure of effective engagement. The integration of the effective engagement framework with the casual mediation approach provides a combined theoretical and analytical approach for evaluating effective engagement in DHIs.

This Study

This study collates clinical survey data, biological PrEP adherence measures, and engagement measures from the Prepared, Protected, Empowered (P3) intervention efficacy RCT [LeGrand S, Knudtson K, Benkeser D, Muessig K, Mcgee A, Sullivan PS, et al. Testing the efficacy of a social networking gamification app to improve pre-exposure prophylaxis adherence (P3: Prepared, Protected, emPowered): protocol for a randomized controlled trial. JMIR Res Protoc. Dec 18, 2018;7(12):e10448. [FREE Full text] [CrossRef] [Medline]19,P3 (Prepared, Protected, emPowered): promoting pre-exposure prophylaxis (PrEP) adherence through a social networking, gamification, and adherence support app. National Institutes of Health National Library of Medicine. Oct 25, 2022. URL: https://clinicaltrials.gov/ct2/show/NCT03320512 [accessed 2023-05-31] 66] in a secondary data analysis using the causal mediation framework [Vansteelandt S, Bekaert M, Lange T. Imputation strategies for the estimation of natural direct and indirect effects. Epidemiol Methods. 2012;1(1):130. [CrossRef]57-VanderWeele TJ. Explanation in Causal Inference: Methods for Mediation and Interaction. New York, NY. Oxford University Press; 2015. 62] to quantify whether and to what degree intrapersonal behavioral, mental health, and sociodemographic measures impact effective engagement with respect to PrEP adherence in YSGMMSM.


Study Design

This study is a secondary analysis, which combined clinical survey data, biological PrEP adherence measures, and engagement measures collected from a 3-arm RCT testing the efficacy of P3, a PrEP adherence DHI [LeGrand S, Knudtson K, Benkeser D, Muessig K, Mcgee A, Sullivan PS, et al. Testing the efficacy of a social networking gamification app to improve pre-exposure prophylaxis adherence (P3: Prepared, Protected, emPowered): protocol for a randomized controlled trial. JMIR Res Protoc. Dec 18, 2018;7(12):e10448. [FREE Full text] [CrossRef] [Medline]19,P3 (Prepared, Protected, emPowered): promoting pre-exposure prophylaxis (PrEP) adherence through a social networking, gamification, and adherence support app. National Institutes of Health National Library of Medicine. Oct 25, 2022. URL: https://clinicaltrials.gov/ct2/show/NCT03320512 [accessed 2023-05-31] 66]. This secondary analysis used the causal mediation framework and statistical analysis procedures to characterize whether engagement with the P3 intervention mediated the relationship between baseline intrapersonal measures and PrEP adherence at 3 months.

Ethical Considerations

The parent study was reviewed and approved by the institutional review board of the University of North Carolina at Chapel Hill (17-9551). A certificate of confidentiality was obtained from the Eunice Kennedy Shriver National Institute of Child Health and Human Development. For participants aged between 15 and 17 years, a waiver of parental consent was obtained. This trial is registered at ClinicalTrials.gov (NCT03320512). This secondary analysis was reviewed by the Northeastern University institutional review board and determined to be exempt under category 4 (secondary research for which consent is not required). This secondary analysis used a deidentified analytic dataset curated by the parent study’s staff. The principal investigator had no contact with participants and made no attempts to reidentify participants post hoc.

Parent Study

Intervention

P3 is a user-centered PrEP adherence phone app that incorporates a variety of content in multiple formats to serve the diverse needs, barriers, and motivations of YSGMMSM. This phone app included text, videos, quizzes, and a social wall where participants could share experiences, from success stories to challenges. In addition, P3 incorporated game-like elements, such as daily health-related quests, in-app rewards, unlocking character-driven narratives, and social connection activities. P3 used a financial incentive to encourage daily use in which small monetary incentives are awarded for daily use of P3 (not necessarily PrEP). Participants started with an initial bank of US $90 and then were awarded US $0.50 for each day on which they logged into P3 and completed one of the following tasks: (1) post on the social wall, (2) use the medication tracker, or (3) complete a quest. Each of these tasks corresponds to a putative behavior change mechanism (social support, instrumental support, and gamification, respectively). Conversely, US $1 was deducted for each day on which the participant did not log in and complete 1 of the aforementioned tasks across the 90-day trial period. The maximum a participant could earn was US $135 and the minimum was US $0. P3+ is an extension of P3, in which participants were also connected with an adherence counselor trained on the Next Step Counseling adherence counseling curriculum through the P3 phone app [LeGrand S, Knudtson K, Benkeser D, Muessig K, Mcgee A, Sullivan PS, et al. Testing the efficacy of a social networking gamification app to improve pre-exposure prophylaxis adherence (P3: Prepared, Protected, emPowered): protocol for a randomized controlled trial. JMIR Res Protoc. Dec 18, 2018;7(12):e10448. [FREE Full text] [CrossRef] [Medline]19,P3 (Prepared, Protected, emPowered): promoting pre-exposure prophylaxis (PrEP) adherence through a social networking, gamification, and adherence support app. National Institutes of Health National Library of Medicine. Oct 25, 2022. URL: https://clinicaltrials.gov/ct2/show/NCT03320512 [accessed 2023-05-31] 66-R Amico K, McMahan V, Goicochea P, Vargas L, Marcus JL, Grant RM, et al. Supporting study product use and accuracy in self-report in the iPrEx study: next step counseling and neutral assessment. AIDS Behav. Jul 2012;16(5):1243-1259. [CrossRef] [Medline]68].

Clinical Trial Eligibility and Procedures

Starting in May 2019, participants were recruited from 9 study sites as follows: Tampa, Florida; Boston, Massachusetts; Chicago, Illinois; Houston, Texas; Philadelphia, Pennsylvania; Chapel Hill, North Carolina; Atlanta, Georgia; Bronx, New York; and Charlotte, North Carolina. A mix of in-person, venue-based, and web-based recruitment methods was used. Inclusion criteria were as follows: individuals (1) who were aged between 16 and 24 years, (2) who were assigned male sex at birth, (3) who reported sex with or intentions to have sex with men or transwomen, (4) who had reliable daily access to an Android (Google LLC) or iOS (Apple Inc) smartphone with a data plan, (5) who could speak and read English, (6) who were HIV-uninfected (confirmed by self-report at enrollment visit), and (7) who were not on PrEP but planned to initiate in the next 7 days and had an active PrEP prescription (prescription confirmed by study staff) or those who were currently on PrEP who had an active PrEP prescription (prescription confirmed by study staff). After providing informed consent either in person or electronically, participants were randomized to 1 of the 3 treatment arms (standard of care, P3, or P3+) using a 1:1:1 randomization scheme. Clinical survey assessments and laboratory specimens were collected at baseline and 3 months into the trial period. Engagement measures (in the 2 intervention arms) were collected continuously throughout the intervention period and summarized at 3 months [LeGrand S, Knudtson K, Benkeser D, Muessig K, Mcgee A, Sullivan PS, et al. Testing the efficacy of a social networking gamification app to improve pre-exposure prophylaxis adherence (P3: Prepared, Protected, emPowered): protocol for a randomized controlled trial. JMIR Res Protoc. Dec 18, 2018;7(12):e10448. [FREE Full text] [CrossRef] [Medline]19]. Study visits were initially planned to be conducted in person at the same study site where participants enrolled. All study sites stopped in-person study activities on March 17, 2020, to reduce the transmission of COVID-19. Web-based recruitment and web-based study activities began in June 2020. In addition, some study sites were able to conduct limited in-person activities based on local regulations and COVID-19 restrictions. The trial concluded in September 2021.

Secondary Analysis Eligibility

Participants from the primary study who received the P3 or P3+ intervention were eligible for inclusion in this secondary data analysis (n=163). Participants in the P3 and P3+ conditions who were lost to follow-up (LTFU; defined as participants who did not begin the month 3 survey) were excluded (22/163, 13.5%). Moreover, 1 (0.7%) of the remaining 141 participants with incomplete survey information pertaining to study-relevant exposure measures was also removed. This resulted in a dataset of 140 participants (Figure 1).

Figure 1. Primary Prepared, Protected, Empowered (P3) randomized controlled trial (RCT) participants’ eligibility for inclusion in this study and secondary analysis of effective engagement with respect to pre-exposure prophylaxis adherence in young sexual and gender minority men who have sex with men. LTFU: lost to follow-up; SOC: standard of care; P3+: extension of Prepared, Protected, emPowered.

Outcome: PrEP Nonadherence

PrEP nonadherence at 3 months (binary) was the primary outcome measure used in this analysis. If serum levels of tenofovir-diphosphate and emtricitabine-triphosphate were consistent with ≤4 doses per week, the participant was considered nonadherent. Due to study operation interruptions related to the COVID-19 pandemic, 28.6% (40/140) of eligible participants were unable to provide biological specimens. In cases where tenofovir-diphosphate and emtricitabine-triphosphate values were missing, self-reported doses of PrEP in the last 7 days before the 3-month clinical survey were used. Participants who reported ≤4 self-reported doses of PrEP in the last 7 days were considered nonadherent. There is mixed evidence regarding the accuracy of self-reported measures of PrEP adherence. While 2 studies have found that self-report measures of PrEP adherence correlate with protective serum levels among adults [Agot K, Taylor D, Corneli AL, Wang M, Ambia J, Kashuba AD, et al. Accuracy of self-report and pill-count measures of adherence in the FEM-PrEP clinical trial: implications for future HIV-prevention trials. AIDS Behav. May 7, 2015;19(5):743-751. [FREE Full text] [CrossRef] [Medline]69,Blumenthal J, Pasipanodya EC, Jain S, Sun S, Ellorin E, Morris S, et al. Comparing self-report pre-exposure prophylaxis adherence questions to pharmacologic measures of recent and cumulative pre-exposure prophylaxis exposure. Front Pharmacol. 2019;10:721. [FREE Full text] [CrossRef] [Medline]70], another study examining self-report PrEP adherence measure accuracy among 15 to 23-year-old young MSM found that self-reported measures overreported adherence compared to biological specimens, with the odds of overreporting decreasing by 24% (odds ratio [OR] 0.74, 95% CI 0.65-0.90) with each additional year of age [Baker Z, Javanbakht M, Mierzwa S, Pavel C, Lally M, Zimet G, et al. Predictors of over-reporting HIV pre-exposure prophylaxis (PrEP) adherence among young men who have sex with men (YMSM) in self-reported versus biomarker data. AIDS Behav. Apr 2018;22(4):1174-1183. [FREE Full text] [CrossRef] [Medline]71]. We found that the area under the receiver operating characteristics curve between self-report measures and biological measures among participants with biological and self-report PrEP nonadherence measures was high (≥0.7). Further details are described in the Strengths and Limitations section.

Mediator: Intervention Engagement

Engagement was defined as dollars accrued by 3 months. This measure was mean centered (participant dollars accrued—mean dollars accrued) when used in modeling. As mentioned in the Intervention subsection, participants started with a baseline amount of US $90 in a bank and gained or lost money from this initial bank based on the number of days each participant logged in and completed one of the 3 aforementioned tasks (eg, completing a quest). This measure serves as a quality proxy for engagement because it correlates with the behavior pattern P3 aims to adjust (ie, daily use of P3 mimics the daily dosing pattern of PrEP) and relates to the putative behavior change mechanisms used in the P3 app (eg, completing a quest is related to gamification and posting on the social wall is related social engagement). Engagement measures derived from documenting users’ access, participation, and navigation through a DHI are suitable for constructing measures of engagement, as these measures provide an objective view into patterns of use with high ecological validity [Yardley L, Spring BJ, Riper H, Morrison LG, Crane DH, Curtis K, et al. Understanding and promoting effective engagement with digital behavior change interventions. Am J Prev Med. Nov 2016;51(5):833-842. [CrossRef] [Medline]25,Hightow-Weidman LB, Horvath KJ, Scott H, Hill-Rorie J, Bauermeister JA. Engaging youth in mHealth: what works and how can we be sure? Mhealth. 2021;7:23. [FREE Full text] [CrossRef] [Medline]31,Hightow-Weidman LB, Bauermeister JA. Engagement in mHealth behavioral interventions for HIV prevention and care: making sense of the metrics. Mhealth. Jan 2020;6:7. [FREE Full text] [CrossRef] [Medline]32].

Intrapersonal Measures

Consistent with Yardley et al [Yardley L, Spring BJ, Riper H, Morrison LG, Crane DH, Curtis K, et al. Understanding and promoting effective engagement with digital behavior change interventions. Am J Prev Med. Nov 2016;51(5):833-842. [CrossRef] [Medline]25], several intrapersonal measures capturing experiences with phone and phone apps, mental health, and sociodemographic characteristics were constructed from the baseline survey administered in the primary RCT based on prior research [Yardley L, Spring BJ, Riper H, Morrison LG, Crane DH, Curtis K, et al. Understanding and promoting effective engagement with digital behavior change interventions. Am J Prev Med. Nov 2016;51(5):833-842. [CrossRef] [Medline]25,O'Connor S, Hanlon P, O'Donnell CA, Garcia S, Glanville J, Mair FS. Understanding factors affecting patient and public engagement and recruitment to digital health interventions: a systematic review of qualitative studies. BMC Med Inform Decis Mak. Sep 15, 2016;16(1):120. [FREE Full text] [CrossRef] [Medline]38]. These measures, described subsequently, focused on patterns of phone and phone app use, measures of mental health, and sociodemographic information.

Phone and Phone App Use

Several binary measures describing phone and phone app use were derived from the baseline survey. A participant was considered to have experienced disconnects if they lost access to their phone or phone service at any time in the year leading up to the baseline survey. Prior qualitative research identified poor internet access as a barrier to engagement in DHIs [O'Connor S, Hanlon P, O'Donnell CA, Garcia S, Glanville J, Mair FS. Understanding factors affecting patient and public engagement and recruitment to digital health interventions: a systematic review of qualitative studies. BMC Med Inform Decis Mak. Sep 15, 2016;16(1):120. [FREE Full text] [CrossRef] [Medline]38,Greenhalgh T, Hinder S, Stramer K, Bratan T, Russell J. Adoption, non-adoption, and abandonment of a personal electronic health record: case study of HealthSpace. BMJ. Nov 16, 2010;341:c5814. [FREE Full text] [CrossRef] [Medline]40-Middlemass J, Davy Z, Cavanagh K, Linehan C, Morgan K, Lawson S, et al. Integrating online communities and social networks with computerised treatment for insomnia: a qualitative study. Br J Gen Pract. Dec 2012;62(605):e840-e850. [FREE Full text] [CrossRef] [Medline]42,Greenhalgh T, Wood GW, Bratan T, Stramer K, Hinder S. Patients' attitudes to the summary care record and HealthSpace: qualitative study. BMJ. Jun 07, 2008;336(7656):1290-1295. [FREE Full text] [CrossRef] [Medline]45,Hopp FP, Hogan MM, Woodbridge PA, Lowery JC. The use of telehealth for diabetes management: a qualitative study of telehealth provider perceptions. Implement Sci. May 02, 2007;2:14. [FREE Full text] [CrossRef] [Medline]48,Mac Domhnaill C, Mohan G, McCoy S. Home broadband and student engagement during COVID-19 emergency remote teaching. Distance Educ. Nov 03, 2021;42(4):465-493. [CrossRef]50-Mesch GS, Talmud I. Internet connectivity, community participation, and place attachment: a longitudinal study. Am Behav Sci. Feb 18, 2010;53(8):1095-1110. [CrossRef]52,Horvath KJ, Danilenko GP, Williams ML, Simoni J, Amico KR, Oakes JM, et al. Technology use and reasons to participate in social networking health websites among people living with HIV in the US. AIDS Behav. May 2012;16(4):900-910. [FREE Full text] [CrossRef] [Medline]72]. Those who spent an average of ≥7 hours per day on the internet outside of work or school were considered high internet users. Those who used phone apps ≥2 times per day were considered frequent phone app users. These measures acted as proxy variables for digital literacy and digital familiarity, which have been shown to act as engagement facilitators in qualitative research [Horvath KJ, Bauermeister JA. eHealth literacy and intervention tailoring impacts the acceptability of a HIV/STI testing intervention and sexual decision making among young gay and bisexual men. AIDS Educ Prev. Feb 2017;29(1):14-23. [FREE Full text] [CrossRef] [Medline]34,Danaher BG, Seeley JR. Methodological issues in research on web-based behavioral interventions. Ann Behav Med. Aug 2009;38(1):28-39. [FREE Full text] [CrossRef] [Medline]39-Lorimer K, Martin S, McDaid LM. The views of general practitioners and practice nurses towards the barriers and facilitators of proactive, internet-based chlamydia screening for reaching young heterosexual men. BMC Fam Pract. Jun 27, 2014;15:127. [FREE Full text] [CrossRef] [Medline]49,Payton S, Hague C. Digital literacy across the curriculum. National Foundation for Educational Research. Jan 1, 2010. URL: https://www.nfer.ac.uk/publications/digital-literacy-across-the-curriculum/ [accessed 2023-09-13] 73-Melhem SJ, Nabhani-Gebara S, Kayyali R. Digital trends, digital literacy, and e-health engagement predictors of breast and colorectal cancer survivors: a population-based cross-sectional survey. Int J Environ Res Public Health. Jan 13, 2023;20(2):1472. [FREE Full text] [CrossRef] [Medline]75]. Binary measures describing participants’ propensity to use phone apps for a variety of purposes were derived from questions whose answers follow a Likert scale with the following ranked choices: never, rarely, sometimes, often, and decline to answer. Participants who disclosed that they “often” use phone apps for chatting with friends, chatting with family, looking for romantic dates, looking for casual sex, or tracking their health were considered highly interested in using phone apps for those activities. Previous qualitative research has identified social support in various forms enabled through a DHI as an engagement facilitator [O'Connor S, Hanlon P, O'Donnell CA, Garcia S, Glanville J, Mair FS. Understanding factors affecting patient and public engagement and recruitment to digital health interventions: a systematic review of qualitative studies. BMC Med Inform Decis Mak. Sep 15, 2016;16(1):120. [FREE Full text] [CrossRef] [Medline]38,Winkelman WJ, Leonard KJ, Rossos PG. Patient-perceived usefulness of online electronic medical records: employing grounded theory in the development of information and communication technologies for use by patients living with chronic illness. J Am Med Inform Assoc. 2005;12(3):306-314. [FREE Full text] [CrossRef] [Medline]44,Beattie A, Shaw A, Kaur S, Kessler D. Primary-care patients' expectations and experiences of online cognitive behavioural therapy for depression: a qualitative study. Health Expect. Mar 2009;12(1):45-59. [FREE Full text] [CrossRef] [Medline]46,Fukuoka Y, Kamitani E, Bonnet K, Lindgren T. Real-time social support through a mobile virtual community to improve healthy behavior in overweight and sedentary adults: a focus group analysis. J Med Internet Res. Jul 14, 2011;13(3):e49. [FREE Full text] [CrossRef] [Medline]47,Im EO, Lee B, Chee W. Shielded from the real world: perspectives on internet cancer support groups by Asian Americans. Cancer Nurs. 2010;33(3):E10-E20. [FREE Full text] [CrossRef] [Medline]76,Dasgupta K, Da Costa D, Pillay S, De Civita M, Gougeon R, Leong A, et al. Strategies to optimize participation in diabetes prevention programs following gestational diabetes: a focus group study. PLoS One. 2013;8(7):e67878. [FREE Full text] [CrossRef] [Medline]77]. Finally, a binary measure representing the intervention arm (P3 or P3+) was constructed to be used as a control measure.

Mental Health

The Patient Health Questionnaire-8 (PHQ-8) [Johnson JG, Harris ES, Spitzer RL, Williams JB. The patient health questionnaire for adolescents: validation of an instrument for the assessment of mental disorders among adolescent primary care patients. J Adolesc Health. Mar 2002;30(3):196-204. [CrossRef] [Medline]78,Kroenke K, Strine TW, Spitzer RL, Williams JB, Berry JT, Mokdad AH. The PHQ-8 as a measure of current depression in the general population. J Affect Disord. Apr 2009;114(1-3):163-173. [CrossRef] [Medline]79] and Generalized Anxiety Disorder-7 (GAD-7) [Spitzer RL, Kroenke K, Williams JB, Löwe B. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med. May 22, 2006;166(10):1092-1097. [CrossRef] [Medline]80] questionnaires were used to assess depressive and anxiety symptoms, respectively. Both scales asked participants to rank how frequently they experience symptoms from not at all, several days, more than half the days, and nearly every day. Scores ranged from 0 to 24 in the PHQ-8 and 0 to 21 in the GAD-7, with lower scores representing less frequent experiences and higher scores representing more frequent experiences of depressive and anxious symptoms, respectively. Participants who scored ≥10 on the PHQ-8 and GAD-7 were considered to have depressive or anxious symptoms, respectively [Kroenke K, Strine TW, Spitzer RL, Williams JB, Berry JT, Mokdad AH. The PHQ-8 as a measure of current depression in the general population. J Affect Disord. Apr 2009;114(1-3):163-173. [CrossRef] [Medline]79-Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med. Sep 2001;16(9):606-613. [FREE Full text] [CrossRef] [Medline]81]. Previous research has identified that psychological distress from trauma is associated with lower engagement, which suggests that other stressors on mental health may also act as barriers to engagement [Yeager CM, Shoji K, Luszczynska A, Benight CC. Engagement with a trauma recovery internet intervention explained with the health action process approach (HAPA): longitudinal study. JMIR Ment Health. Apr 10, 2018;5(2):e29. [FREE Full text] [CrossRef] [Medline]82]. Furthermore, symptoms of depression have been consistently linked to lower treatment adherence [DiMatteo MR, Lepper HS, Croghan TW. Depression is a risk factor for noncompliance with medical treatment: meta-analysis of the effects of anxiety and depression on patient adherence. Arch Intern Med. Jul 24, 2000;160(14):2101-2107. [CrossRef] [Medline]83], and symptoms of depression and anxiety have been found to be associated with a higher likelihood of medication nonadherence [Sundbom LT, Bingefors K. The influence of symptoms of anxiety and depression on medication nonadherence and its causes: a population based survey of prescription drug users in Sweden. Patient Prefer Adherence. Aug 19, 2013;7:805-811. [FREE Full text] [CrossRef] [Medline]84].

Sociodemographic Measures

Sociodemographic measures captured from the baseline survey include race, ethnicity, and age. Participants were considered non-Hispanic White if they disclosed non-Hispanic ethnicity and White as their race. In previous research, sociodemographic measures had mixed effects on engagement [Beatty L, Binnion C. A systematic review of predictors of, and reasons for, adherence to online psychological interventions. Int J Behav Med. Dec 2016;23(6):776-794. [CrossRef] [Medline]85-Yeager CM, Benight CC. If we build it, will they come? Issues of engagement with digital health interventions for trauma recovery. Mhealth. Sep 11, 2018;4:37. [FREE Full text] [CrossRef] [Medline]92]. For example, older age has been observed as an engagement facilitator and an engagement barrier in different studies [Beatty L, Binnion C. A systematic review of predictors of, and reasons for, adherence to online psychological interventions. Int J Behav Med. Dec 2016;23(6):776-794. [CrossRef] [Medline]85,Christensen H, Griffiths KM, Farrer L. Adherence in internet interventions for anxiety and depression. J Med Internet Res. Apr 24, 2009;11(2):e13. [FREE Full text] [CrossRef] [Medline]86]. These measures were used to control for confounding (further specifications are given subsequently) and investigated as exposures of interest for their relationship with effective engagement.

Statistical Analysis

Descriptive statistics for participants who were eligible for secondary analysis were generated and are reported in the Results section. There is also a comparison of 140 eligible participants to the 22 participants who were LTFU. To quantify how engagement with P3 mediates the relationship between baseline intrapersonal measures and PrEP nonadherence at 3 months, the causal mediation framework was used. This analytical approach extends the counterfactual causal inference framework to mediation, has been extensively explicated, clarifies several confounding assumptions, accommodates exposure-mediator interaction [Vansteelandt S, Bekaert M, Lange T. Imputation strategies for the estimation of natural direct and indirect effects. Epidemiol Methods. 2012;1(1):130. [CrossRef]57-VanderWeele TJ. Explanation in Causal Inference: Methods for Mediation and Interaction. New York, NY. Oxford University Press; 2015. 62,Valeri L, Vanderweele TJ. Mediation analysis allowing for exposure-mediator interactions and causal interpretation: theoretical assumptions and implementation with SAS and SPSS macros. Psychol Methods. Jun 2013;18(2):137-150. [FREE Full text] [CrossRef] [Medline]93], has been applied in previous research [Oberg AS, VanderWeele TJ, Almqvist C, Hernandez-Diaz S. Pregnancy complications following fertility treatment-disentangling the role of multiple gestation. Int J Epidemiol. Aug 01, 2018;47(4):1333-1342. [FREE Full text] [CrossRef] [Medline]63], and complements the effective engagement theoretical framework by explicitly decomposing the total effect of each intrapersonal measure on PrEP nonadherence into a direct and indirect effect (ie, the effect mediated by P3 engagement). Figure 2 depicts the integration of effective engagement and causal mediation through a theoretical causal diagram. In this study, effect decomposition was accomplished by first fitting a linear regression to assess the effect of each exposure (eg, anxious symptoms) on the mediator (mean-centered dollars accrued at 3 months), adjusting for confounding. Then, a logistic model was fit to examine the relationship between each exposure (eg, anxious symptoms) and PrEP nonadherence, adjusting for the same confounders, dollars accrued at 3 months (ie, the mediator), and the exposure-mediator interaction. Baseline measures, such as age, race, and intervention arm, were used to adjust for confounding. Several other critical measures were controlled through the primary study’s design via the eligibility criteria. Intention to initiate PrEP, PrEP access, sexual orientation, and English literacy were verified at enrollment. The total effects estimated from these 2 models with corresponding CIs and P values are reported in the Results section. The total effect for each intrapersonal measure on PrEP nonadherence at 3 months is presented as an OR (defined as the rate of nonadherence in the exposed divided by the rate of nonadherence in the unexposed). These 2 regression models are then used to derive direct and indirect effects. The direct effect represents the effect of a given intrapersonal measure on PrEP nonadherence independent of P3 engagement. This effect is estimated by comparing the estimated PrEP nonadherence in the exposed relative to the unexposed while setting P3 engagement to the level that would have naturally occurred in the absence of the exposure. The indirect effect represents the effect of a given intrapersonal measure on PrEP nonadherence operating through the mediator. This effect is estimated by comparing the outcome for the exposed for different contrasts of the mediator (eg, between levels of P3 engagement). Mediation results, including direct and indirect effects with corresponding CIs and percentage mediated (PM) are reported in the Results section. Mediation is assessed using a combination of total and indirect effect size, statistical significance, and PM. P values are reported for transparency, but CIs are used as the primary determinant of statistical significance for mediation analysis, determined by if the CI overlaps with the null value. For the first model (linear regression of dollars accrued), the null value is 0, and for the second model (logistic regression of binary PrEP adherence), the null value is 1. All analyses were carried out using R (GNU) and RStudio (Posit, PBC). Mediation models were constructed using the CMAverse R library using the regression-based approach and imputation as the estimation method [Shi B, Choirat C, Coull BA, VanderWeele TJ, Valeri L. CMAverse: a suite of functions for reproducible causal mediation analyses. Epidemiology. Sep 01, 2021;32(5):e20-e22. [CrossRef] [Medline]94].

Figure 2. Integration of effective engagement and causal mediation frameworks in a causal diagram. Direct effects represent relationships between intrapersonal measures and pre-exposure prophylaxis (PrEP) adherence controlling for engagement with Prepared, Protected, Empowered (P3). Indirect effects represent relationships between intrapersonal measures and PrEP adherence operating through P3 engagement (ie, mediated effects).

Participant Characteristics

The median age of participants was 22 years (IQR 20-23). Overall, 17.1% (24/140) were considered nonadherent to PrEP at 3 months (Table 1). The minimum amount the participants earned over the 90-day trial period was US $1.5 (representing 1 day logged in with a task completed). The maximum amount the participants earned over the 90-day trial period was US $135 (corresponding to logging in and completing a task every day for the 90-day trial period). Participants earned a mean of US $96.40 (SD US $35.1) over the 90-day trial period. Participants earned a median of US $112.50 (IQR US $73.50-US $ 123.50) which corresponds to 75 days logged in with a task completed (IQR 49-82) over the 90-day trial period. Participants were largely heterogeneous with respect to phone, technology, and internet use patterns. Notable exceptions to this pattern were as follows: 90.7% (127/140) of the participants used phone apps more than once per day and 90.7% (127/140) of the participants disclosed that they often use phone apps for chatting with friends. No significant differences in selected baseline intrapersonal characteristics were observed between eligible and LTFU participants (Table 1).

Table 1. Comparison of participant characteristics who were eligible for secondary analysis and those who were lost to follow-up (LTFU) in a secondary analysis of engagement with a pre-exposure prophylaxis (PrEP) adherence digital health intervention (DHI) among US young sexual and gender minority men who have sex with men aged between 16 and 24 years.

Eligible participants (n=140)LTFU participants (n=22)P valuea
Nonadherent to PrEP at 3 months, n (%)24 (17.1)b
High internet users, n (%)29 (20.7)3 (13.6).63
Disconnect from phone in past 12 months, n (%)16 (11.4)3 (13.6).99
Frequent phone app users, n (%)127 (90.7)21 (95.5).74
Frequent uses of phone apps, n (%)

Chatting with friends126 (90)22 (100).29

Chatting with family76 (54.3)13 (59.1).85

Finding romantic dates43 (30.7)4 (18.2).34

Looking for casual sex38 (27.1)5 (22.7).86

Tracking health38 (27.1)8 (36.4).53
Depressive symptoms, n (%)18 (12.9)2 (9.1).88
Anxiety symptoms, n (%)21 (15)5 (22.7).55
Non-Hispanic White, n (%)69 (49.3)9 (40.9).62
Male, n (%)128 (91.4)20 (90.9)1.0
Intervention arm: P3+, n (%)71 (50.7)9 (40.9).53
Age (y), median (IQR)22 (20-23)22 (20-23).53c
Site, n (%).12

Tampa23 (16.4)5 (22.7)

Atlanta10 (7.1)5 (22.7)

Boston22 (15.7)2 (9.1)

Philadelphia14 (10)4 (18.2)

Chicago23 (16.4)1 (4.5)

Houston18 (12.9)4 (18.2)

Bronx9 (6.4)1 (4.5)

Chapel Hill15 (10.7)0 (0)

Charlotte6 (4.3)0 (0)

aContinuous measures tested with 2-tailed t tests, categorical measures tested with Fisher exact test.

bLTFU participants did not return for 3-month PrEP adherence data collection.

cFor nonnormal distribution, the Kruskal-Wallis rank sum test was used.

Multivariate Analysis

Table 2 presents multivariate regression results for the effect of intrapersonal measures on engagement (dollars accrued by 3 months). Frequent phone app users earned US $21.49 (95% CI 2.50-40.47) more than infrequent phone app users through the 3-month trial period. Participants who reported anxious symptoms in the past 2 weeks (GAD-7 score≥10) earned US $15.95 (95% CI –31.57 to –0.32) less throughout the 3-month trial period than those with mild or no anxious symptoms in the past 2 weeks. Non-Hispanic White individuals earned US $17.02 (95% CI 5.95-28.10) more on average by 3 months than participants belonging to other racial and ethnic groups. Participants who received the P3+ intervention (P3 with the addition of human adherence counselors accessible through the app) earned US $12.48 (95% CI 1.47-23.50) more on average than those who received the standard P3 app through the 3-month trial period. Finally, for each additional year of age, participants earned on average an additional US $2.55 (95% CI –0.19 to 5.29). The 95% CI for the relationship between age and engagement narrowly overlaps the null value of 0 with a corresponding P value of .07, meaning this relationship is technically statistically insignificant.

Table 2. Multivariate relationships between intrapersonal measures and engagement with Prepared, Protected, Empowered (P3), pre-exposure prophylaxis (PrEP) nonadherence among US young sexual and gender minority men who have sex with men aged between 16 to 24 years (n=140).
Intrapersonal measuresEngagementaPrEP nonadherencea

Estimate (95% CI)bP valueORc,d (95% CI)eP value

Dollars accrued at 3 monthsf0.98 (0.97 to 0.99).02
High internet user–5.51 (–19.65 to 8.63).452.31 (0.80 to 6.46).11
Disconnect from the phone in the past 12 months–6.42 (–24.00 to 11.16).483.84 (1.14 to 12.81).03
Frequent phone app user21.49 (2.50 to 40.47).030.66 (0.17 to 2.88).55
Frequently uses of phone apps

Chatting with friends–1.26 (–20.36 to 17.85).900.82 (0.20 to 4.37).80

Chatting with family9.71 (–1.45 to 20.86).090.99 (0.38 to 2.63).99

Finding romantic dates1.22 (–11.14 to 13.57).850.78 (0.23 to 2.32).67

Looking for casual sex–5.06 (–17.49 to 7.36).430.71 (0.21 to 2.08).55

Tracking health2.00 (–10.42 to 14.41).750.29 (0.06 to 0.96).06
Depressive symptoms–8.16 (–24.82 to 8.51).343.42 (0.95 to 12.00).05
Anxiety symptoms–15.95 (–31.57 to –0.32).053.51 (1.06 to 11.55).04
Non-Hispanic Whiteg17.02 (5.95 to 28.10).0030.17 (0.05 to 0.48).002
Intervention arm: P3+g12.48 (1.47 to 23.50).031.05 (0.41 to 2.71).92
Ageg2.55 (–0.19 to 5.29).070.82 (0.65 to 1.02).08

aMultivariate models are adjusted for age, race/ethnicity, and intervention arm.

bNull value is 0.

cOR: odds ratio.

dDerived by exponentiating estimated regression coefficients.

eNull value is 1.

fNot available.

gIntrapersonal measure is also a control measure. Model constructed using age, race, and intervention arm for engagement and age, race, ethnicity, intervention arm, dollars accrued at 3 months, and dollars accrued at 3 months and the interaction between dollars accrued at 3 months and the focal intrapersonal measure.

Multivariate models are adjusted for age, race, ethnicity, intervention arm, dollars accrued at 3 months, and dollars accrued at 3 months and the interaction between dollars accrued at 3 months and the focal intrapersonal measure.

Total effects for the relationship between baseline intrapersonal measures and PrEP nonadherence at 3 months are reported in Table 2 as ORs. For every dollar earned above the mean throughout the 3-month trial period, participants had 2% (OR 0.98, 95% CI 0.97-0.99) lower odds of PrEP nonadherence at 3 months. Participants who reported using phone apps ≥2 times per day had 34% lower odds of PrEP nonadherence at 3 months (OR 0.66, 95% CI 0.17-2.88). Participants who spent >7 hours on the internet beyond work or school had 2.31 (95% CI 0.80-6.46) times higher odds of PrEP nonadherence at 3 months compared to participants who reported <7 hours on the internet beyond work or school. Both of these measures that aimed to describe broad patterns of phone and internet use had a relatively large effect on the odds of PrEP nonadherence. However, neither was statistically significant as the 95% CI covers the null value of 1. Participants who reported at least 1 disconnect from their internet service or phone in the past year had 3.84 (95% CI 1.14-12.81) times higher odds of PrEP nonadherence at 3 months than participants who reported no disconnects. Participants who reported that they frequently used phone apps to track their personal health information had 71% (OR 0.29, 95% CI 0.06-0.96) lower odds of PrEP nonadherence at 3 months. Participants who reported depressive symptoms (PHQ-8 score ≥10) had 3.42 (95% CI 0.95-12) times higher odds of PrEP nonadherence at 3 months. Participants who reported anxious symptoms also had 3.51 (95% CI 1.06-11.55) times higher odds of PrEP nonadherence at 3 months. Participants who reported their race and ethnicity as non-Hispanic White had 83% (OR 0.17, 95% CI 0.05-0.48) lower odds of PrEP nonadherence at 3 months. For each additional year of age, participants had 18% (OR 0.82, 95% CI 0.65-1.02) less odds of PrEP nonadherence at 3 months. The 95% CI for the relationship between age and PrEP nonadherence narrowly overlaps the null value of 1 with a corresponding P value of .08, meaning this relationship is technically statistically insignificant.

Mediation Analysis

Total effects were decomposed into direct and indirect effects of intrapersonal measures on effective engagement (Table 3). Experiencing disconnects in the past year was primarily directly related to higher odds of PrEP nonadherence at 3 months (PM=5%), with a direct effect of 3.28 (95% CI 0.91-11.42). Despite the statistically insignificant total effect, using phone apps ≥2 times per day was significantly indirectly associated with lower odds of PrEP nonadherence at 3 months (OR 0.76, 95% CI 0.49-1.00). Conversely, spending >7 hours on the internet beyond work or school resulted in primarily a direct relationship with lower odds of nonadherence (1.92, 95% CI=0.80-5.42; PM=18%). However, similar to the total effect on PrEP nonadherence, this relationship did not rise to the level of statistical significance for either the direct or indirect relationship. Frequently using phone apps to track health information was directly associated (0.3, 95% CI 0-0.92; PM=1%) with lower odds of PrEP nonadherence. Symptoms of depression had a statistically significant total effect on PrEP nonadherence and were partially mediated by engagement with P3 with a direct effect of 2.52 (95% CI 0.79-6.81), an indirect effect of 1.25 (95% CI 0.78-2.14), and a PM of 30%. However, neither the direct nor the indirect effect raised to the level of statistical significance on their own. Experiencing anxious symptoms was primarily indirectly related to higher odds of PrEP nonadherence through engagement with P3 (1.55, 95% CI 1-3.34; PM=51%). Being non-Hispanic White was directly related to lower odds of PrEP nonadherence (0.20, 95% CI 0.04-0.59; PM=4%). For each additional year of age, the odds of PrEP nonadherence were decreased, operating through both direct (0.95, 95% CI 0.68-1.02) and indirect (0.92, 95% CI 0.59-1.07; PM=60%) relationships.

Table 3. Direct and indirect effects of intrapersonal measures on effective engagement with Prepared, Protected, Empowered (P3), a pre-exposure prophylaxis adherence digital health intervention, among US young sexual and gender minority men who have sex with men youth aged between 16 and 24 years (n=140)a.

Direct effect (95% CI)bIndirect effect (95% CI)bPercentage mediated (%)c
High internet users1.92 (0.80-5.42)1.11 (0.84-1.55)18
Disconnect from the phone in the past 12 months3.28 (0.91-11.42)1.04 (0.74-1.84)5
Frequent phone app user1.19 (0.37-6.03)0.76 (0.49-1.00)d
Frequently uses of phone apps

Chatting with family1.23 (0.48-3.19)0.83 (0.59-1.03)

Finding romantic dates0.63 (0.19-2.25)1.01 (0.80-1.16)

Looking for casual sex0.74 (0.15-2.02)1.05 (0.81-1.37)

Tracking health0.31 (0-0.92)0.98 (0.71-1.33)1
Depressive symptoms2.52 (0.79-6.81)1.25 (0.78-2.14)30
Anxiety symptoms2.12 (0.58-5.49)1.55 (1-3.34)51
Non-Hispanic Whitee0.20 (0.04-0.59)0.84 (0.46-2.54)4
Intervention arm: P3+e1.34 (0.54-3.62)0.84 (0.58-1.04)
Agee0.95 (0.68-1.02)0.92 (0.59-1.07)60

aModels are adjusted for age, race/ethnicity, intervention arm, dollars accrued at 3 months, and dollars accrued at 3 months.

bNull value is 1.

cPercentage mediated cannot be calculated when direct and indirect effects are in opposite directions.

dNot applicable.

eIntrapersonal measure is also a control measure. age, race, ethnicity, intervention arm, dollars accrued at 3 months, and dollars accrued at 3 months and the interaction between dollars accrued at 3 months and the focal intrapersonal measure.


Principal Findings

Overview

This study leveraged data from the primary RCT testing the efficacy of P3, a digital PrEP adherence intervention, in a secondary data analysis that used the effective engagement framework by Yardley et al [Yardley L, Spring BJ, Riper H, Morrison LG, Crane DH, Curtis K, et al. Understanding and promoting effective engagement with digital behavior change interventions. Am J Prev Med. Nov 2016;51(5):833-842. [CrossRef] [Medline]25] to characterize whether and how engagement with P3 mediated the relationship between baseline intrapersonal measures and PrEP nonadherence at 3 months. This study found that several measures (eg, twice or more daily phone app use) were positively related to engagement, measured as dollars accrued by 3 months. In contrast, measures such as anxious symptoms were negatively related to engagement. Furthermore, this study found that P3 engagement, behavioral patterns of phone and app use, mental health symptoms, and sociodemographic measures were significantly related to PrEP nonadherence. Using causal mediation analysis, this study decomposed these total effects into direct effects to isolate the effect of each intrapersonal measure on PrEP adherence irrespective of P3 engagement and indirect effects to evaluate if and how each measure may be contributing to effective engagement with P3. This process helps to illuminate possible mechanisms that precipitate or protect against susceptibility to PrEP nonadherence.

Phone and Phone App Use

Digital literacy, defined as an understanding of how technology and digital media are used to communicate with others, has been linked to engagement in DHIs in both HIV and non–HIV-related domains [Horvath KJ, Bauermeister JA. eHealth literacy and intervention tailoring impacts the acceptability of a HIV/STI testing intervention and sexual decision making among young gay and bisexual men. AIDS Educ Prev. Feb 2017;29(1):14-23. [FREE Full text] [CrossRef] [Medline]34,Danaher BG, Seeley JR. Methodological issues in research on web-based behavioral interventions. Ann Behav Med. Aug 2009;38(1):28-39. [FREE Full text] [CrossRef] [Medline]39-Lorimer K, Martin S, McDaid LM. The views of general practitioners and practice nurses towards the barriers and facilitators of proactive, internet-based chlamydia screening for reaching young heterosexual men. BMC Fam Pract. Jun 27, 2014;15:127. [FREE Full text] [CrossRef] [Medline]49,Payton S, Hague C. Digital literacy across the curriculum. National Foundation for Educational Research. Jan 1, 2010. URL: https://www.nfer.ac.uk/publications/digital-literacy-across-the-curriculum/ [accessed 2023-09-13] 73-Melhem SJ, Nabhani-Gebara S, Kayyali R. Digital trends, digital literacy, and e-health engagement predictors of breast and colorectal cancer survivors: a population-based cross-sectional survey. Int J Environ Res Public Health. Jan 13, 2023;20(2):1472. [FREE Full text] [CrossRef] [Medline]75,Norman CD, Skinner HA. eHealth literacy: essential skills for consumer health in a networked world. J Med Internet Res. Jun 16, 2006;8(2):e9. [FREE Full text] [CrossRef] [Medline]95]. Participant’s propensity to use phone apps ≥2 times daily may or may not be a direct reflection of their digital literacy. However, this broader representation of their affinity to use phone apps empirically impacted their engagement with P3 (via dollars accrued at 3 months). Furthermore, despite a statistically insignificant total effect on PrEP nonadherence, the moderate to large reduction in odds of PrEP nonadherence combined with the statistically significant indirect effect from the causal mediation analysis suggest that participants’ affinity to use phone apps could be a facilitator of effective engagement in the context of PrEP adherence DHIs. Conversely, participants who were categorized as high internet users had higher odds of PrEP nonadherence at 3 months, but this effect was not statistically significant. Furthermore, these participants did not significantly engage with P3 more or less than the average participant. The combination of these contrasting findings suggests that a minimum affinity for phone apps may be related to effective engagement, but time spent on the internet is likely not related to effective engagement. This aligns with theories of digital literacy which describe literacy as more of a minimum capacity to use and understand technology as opposed to merely time spent using it [Norman CD, Skinner HA. eHealth literacy: essential skills for consumer health in a networked world. J Med Internet Res. Jun 16, 2006;8(2):e9. [FREE Full text] [CrossRef] [Medline]95,Yang K, Hu Y, Qi H. Digital health literacy: bibliometric analysis. J Med Internet Res. Jul 06, 2022;24(7):e35816. [FREE Full text] [CrossRef] [Medline]96]. Furthermore, these results substantiate the recent trend of constructing validated and reliable scales of digital literacy [Nelson LA, Pennings JS, Sommer EC, Popescu F, Barkin SL. A 3-item measure of digital health care literacy: development and validation study. JMIR Form Res. Apr 29, 2022;6(4):e36043. [FREE Full text] [CrossRef] [Medline]97-Ali A, Raza AA, Qazi IA. Validated digital literacy measures for populations with low levels of internet experiences. Dev Eng. Nov 2023;8:100107. [CrossRef]101]. Studies of effective engagement (such as this study) would benefit greatly from a scale that can measure and test relevant core constructs of digital literacy against effective engagement.

While this study did not have the opportunity to implement a reliable and validated scale of digital literacy, several measures captured more specific patterns of behavior with respect to phone apps, including using phone apps for dates, tracking health information, or chatting with one’s family. Of these, frequently using phone apps to track health information significantly reduced the odds of PrEP nonadherence at 3 months. However, this effect was overwhelmingly a direct effect and participants who frequently use phone apps to track health information did not engage with P3 significantly more than average. This suggests that individuals who are prone to using apps for health tracking may be more health-conscious, independent of app use, and therefore more likely to adhere to PrEP. This aligns with the idea of health-specific digital literacy, sometimes referred to as “eHealth literacy,” defined as “the ability to seek, find, understand, and appraise health information from electronic sources and apply the knowledge gained to addressing or solving a health problem” [Norman CD, Skinner HA. eHealth literacy: essential skills for consumer health in a networked world. J Med Internet Res. Jun 16, 2006;8(2):e9. [FREE Full text] [CrossRef] [Medline]95]. Previous work found that YSGMMSM with a high digital health literacy perceived the information of a DHI aimed to promote HIV and sexually transmitted infection testing as more useful when the intervention tailored information to the participants [Horvath KJ, Bauermeister JA. eHealth literacy and intervention tailoring impacts the acceptability of a HIV/STI testing intervention and sexual decision making among young gay and bisexual men. AIDS Educ Prev. Feb 2017;29(1):14-23. [FREE Full text] [CrossRef] [Medline]34]. This study supports the hypothesis that health-related digital literacy is an important measure of DHI efficacy, especially when the content is tailored to the population the DHI is serving. Furthermore, this study suggests that while digital health literacy likely improves health outcomes through DHIs, it does not operate through an increase in engagement with the intervention itself. Instead, the baseline capacity for digital health literacy seems to act as a catalyst for the participant to incorporate the information presented in the DHI into their life.

Participants who experienced disconnects from their phone in the past year had higher odds of being nonadherent to PrEP at 3 months, overwhelmingly operating as a direct effect. Participants who experienced disconnects did not earn significantly more or less money than average throughout the trial period. This suggests that disconnecting from one’s internet service is not a key engagement barrier. Instead, this measure likely reflects the broader social and structural environment in which a given participant exists. This aligns with previous work that has highlighted the difficulties in adapting DHIs to varying infrastructure levels (eg, low internet connectivity) [O'Connor S, Hanlon P, O'Donnell CA, Garcia S, Glanville J, Mair FS. Understanding factors affecting patient and public engagement and recruitment to digital health interventions: a systematic review of qualitative studies. BMC Med Inform Decis Mak. Sep 15, 2016;16(1):120. [FREE Full text] [CrossRef] [Medline]38,Greenhalgh T, Hinder S, Stramer K, Bratan T, Russell J. Adoption, non-adoption, and abandonment of a personal electronic health record: case study of HealthSpace. BMJ. Nov 16, 2010;341:c5814. [FREE Full text] [CrossRef] [Medline]40-Middlemass J, Davy Z, Cavanagh K, Linehan C, Morgan K, Lawson S, et al. Integrating online communities and social networks with computerised treatment for insomnia: a qualitative study. Br J Gen Pract. Dec 2012;62(605):e840-e850. [FREE Full text] [CrossRef] [Medline]42,Greenhalgh T, Wood GW, Bratan T, Stramer K, Hinder S. Patients' attitudes to the summary care record and HealthSpace: qualitative study. BMJ. Jun 07, 2008;336(7656):1290-1295. [FREE Full text] [CrossRef] [Medline]45,Hopp FP, Hogan MM, Woodbridge PA, Lowery JC. The use of telehealth for diabetes management: a qualitative study of telehealth provider perceptions. Implement Sci. May 02, 2007;2:14. [FREE Full text] [CrossRef] [Medline]48,Mac Domhnaill C, Mohan G, McCoy S. Home broadband and student engagement during COVID-19 emergency remote teaching. Distance Educ. Nov 03, 2021;42(4):465-493. [CrossRef]50-Mesch GS, Talmud I. Internet connectivity, community participation, and place attachment: a longitudinal study. Am Behav Sci. Feb 18, 2010;53(8):1095-1110. [CrossRef]52,Horvath KJ, Danilenko GP, Williams ML, Simoni J, Amico KR, Oakes JM, et al. Technology use and reasons to participate in social networking health websites among people living with HIV in the US. AIDS Behav. May 2012;16(4):900-910. [FREE Full text] [CrossRef] [Medline]72,Campbell BR, Ingersoll KS, Flickinger TE, Dillingham R. Bridging the digital health divide: toward equitable global access to mobile health interventions for people living with HIV. Expert Rev Anti Infect Ther. Mar 2019;17(3):141-144. [FREE Full text] [CrossRef] [Medline]102-Butzner M, Cuffee Y. Telehealth interventions and outcomes across rural communities in the United States: narrative review. J Med Internet Res. Aug 26, 2021;23(8):e29575. [FREE Full text] [CrossRef] [Medline]105]. Similarly, non-Hispanic White participants had significantly lower odds of being PrEP nonadherent at 3 months and earned significantly more money than average throughout the trial period. However, the effect this had on PrEP nonadherence was also largely direct, suggesting that despite the increase in superficial engagement did not drive the lower odds of PrEP nonadherence at 3 months. Therefore, it seems more likely that non-Hispanic White individuals are experiencing fewer social and structural barriers in life external to the intervention, which affords an easier adoption of adherence behaviors. Previous work reinforces this hypothesis, as significant adherence disparities have been found among Black patients relative to White patients in non-DHI settings [Whiteley L, Craker L, Sun S, Tarantino N, Hershkowitz D, Moskowitz J, et al. Factors associated with PrEP adherence among MSM living in Jackson, Mississippi. J HIV AIDS Soc Serv. 2021;20(3):246-261. [FREE Full text] [CrossRef] [Medline]106-Ezennia O, Geter A, Smith DK. The PrEP care continuum and black men who have sex with men: a scoping review of published data on awareness, uptake, adherence, and retention in PrEP care. AIDS Behav. Oct 2019;23(10):2654-2673. [CrossRef] [Medline]108]. Furthermore, previous research has also established that HIV disproportionately affects individuals who are economically disadvantaged [Denning P, Dinenno E. Communities in crisis: is there a generalized HIV epidemic in impoverished urban areas of the United States? In: Proceedings of the XVIII International AIDS Conference. 2010. Presented at: AIDS 2010; July 18-23, 2010; Vienna, Austria. URL: https://tinyurl.com/7a6cc8uw109]. Collectively, the results of this study combined with this body of literature suggest that measures of race and phone disconnects in this study represent structural characteristics that impact PrEP adherence DHI efficacy directly (ie, not through engagement). The implications of this finding align with a systematic review of qualitative studies on engagement conducted by O’Connor et al [O'Connor S, Hanlon P, O'Donnell CA, Garcia S, Glanville J, Mair FS. Understanding factors affecting patient and public engagement and recruitment to digital health interventions: a systematic review of qualitative studies. BMC Med Inform Decis Mak. Sep 15, 2016;16(1):120. [FREE Full text] [CrossRef] [Medline]38], which describes several recommendations for future DHI development and implementation. First, the systematic review by O’Connor et al [O'Connor S, Hanlon P, O'Donnell CA, Garcia S, Glanville J, Mair FS. Understanding factors affecting patient and public engagement and recruitment to digital health interventions: a systematic review of qualitative studies. BMC Med Inform Decis Mak. Sep 15, 2016;16(1):120. [FREE Full text] [CrossRef] [Medline]38] highlights the need for DHI developers to lessen the burden of self-care through DHIs. This aligns with our study, where the preponderance of direct effects suggests several mechanisms for tailoring that do not operate through an increase in DHI use. For example, DHIs may be able to adapt intervention elements to low internet connectivity environments (eg, allow participants to download any video content so that it is viewable offline). While this may not increase the engagement levels of those living in low internet connectivity environments to a level significantly above the average, it may allow those participants to interact with the DHI more meaningfully by consuming DHI content uninterrupted during an optimal time for the participant. Second, the systematic review recommends incorporating interpersonal relationships (eg, family, friends, and care providers) and public health institutions in designing, using, and implementing DHIs to mitigate the effects of structural disparities [O'Connor S, Hanlon P, O'Donnell CA, Garcia S, Glanville J, Mair FS. Understanding factors affecting patient and public engagement and recruitment to digital health interventions: a systematic review of qualitative studies. BMC Med Inform Decis Mak. Sep 15, 2016;16(1):120. [FREE Full text] [CrossRef] [Medline]38]. These recommendations align with the effective engagement framework by Yardley et al [Yardley L, Spring BJ, Riper H, Morrison LG, Crane DH, Curtis K, et al. Understanding and promoting effective engagement with digital behavior change interventions. Am J Prev Med. Nov 2016;51(5):833-842. [CrossRef] [Medline]25], which describes the increased need for tailored approaches to consider the “contextual needs” of DHI participants in addition to traditional tailoring approaches centered on individual sociodemographic characteristics.

Mental Health

While past literature has highlighted the impact of DHIs on improving mental health outcomes, to our knowledge, this is the first study that directly measures the impact of baseline mental health symptoms on effective engagement in a study focused on a nonmental health–related outcome using a digital intervention [Gan DZ, McGillivray L, Han J, Christensen H, Torok M. Effect of engagement with digital interventions on mental health outcomes: a systematic review and meta-analysis. Front Digit Health. Nov 04, 2021;3:764079. [FREE Full text] [CrossRef] [Medline]110,Donkin L, Christensen H, Naismith SL, Neal B, Hickie IB, Glozier N. A systematic review of the impact of adherence on the effectiveness of e-therapies. J Med Internet Res. Aug 05, 2011;13(3):e52. [FREE Full text] [CrossRef] [Medline]111]. A meta-analysis by DiMatteo et al [DiMatteo MR, Lepper HS, Croghan TW. Depression is a risk factor for noncompliance with medical treatment: meta-analysis of the effects of anxiety and depression on patient adherence. Arch Intern Med. Jul 24, 2000;160(14):2101-2107. [CrossRef] [Medline]83] examined the link between symptoms of depression and anxiety on treatment adherence in a range of medical conditions (eg, cancer and asthma) and found that symptoms of depression were consistently linked to lower treatment adherence, while the relationship between anxious symptoms and lower treatment adherence were mixed (either small or null). However, Sundbom and Bingefors [Sundbom LT, Bingefors K. The influence of symptoms of anxiety and depression on medication nonadherence and its causes: a population based survey of prescription drug users in Sweden. Patient Prefer Adherence. Aug 19, 2013;7:805-811. [FREE Full text] [CrossRef] [Medline]84] found that symptoms of depression and anxiety were both linked to medication nonadherence in a more recent population study of Swedish adults. In this study, we similarly found that depressive and anxious symptoms increased the odds of PrEP nonadherence at 3 months in multivariate models. Despite both measures having significant total effects, only anxious symptoms were significantly indirectly related to PrEP nonadherence through lowered engagement. Broadly, there is a need for further research exploring the relationship between mental health conditions, DHI engagement, and PrEP adherence.

DiMatteo et al [DiMatteo MR, Lepper HS, Croghan TW. Depression is a risk factor for noncompliance with medical treatment: meta-analysis of the effects of anxiety and depression on patient adherence. Arch Intern Med. Jul 24, 2000;160(14):2101-2107. [CrossRef] [Medline]83] suggest several hypotheses to explain the relationship between depression and lower treatment adherence. They note that symptoms of depression (depressed mood, feelings of hopelessness, diminished interest or pleasure in activities, sleep disturbances, and diminished ability to concentrate) may directly influence treatment adherence [Johnson JG, Harris ES, Spitzer RL, Williams JB. The patient health questionnaire for adolescents: validation of an instrument for the assessment of mental disorders among adolescent primary care patients. J Adolesc Health. Mar 2002;30(3):196-204. [CrossRef] [Medline]78,Kroenke K, Strine TW, Spitzer RL, Williams JB, Berry JT, Mokdad AH. The PHQ-8 as a measure of current depression in the general population. J Affect Disord. Apr 2009;114(1-3):163-173. [CrossRef] [Medline]79,DiMatteo MR, Lepper HS, Croghan TW. Depression is a risk factor for noncompliance with medical treatment: meta-analysis of the effects of anxiety and depression on patient adherence. Arch Intern Med. Jul 24, 2000;160(14):2101-2107. [CrossRef] [Medline]83]. For example, if participants with depression are having difficulties concentrating, they may find it more difficult to remember to take PrEP every day as prescribed. They also suggest that depression may be related to social isolation and that social support may be related to better adherence. While depression has been linked to social isolation in young adults broadly [Elmer T, Stadtfeld C. Depressive symptoms are associated with social isolation in face-to-face interaction networks. Sci Rep. Jan 29, 2020;10(1):1444. [FREE Full text] [CrossRef] [Medline]112-Matthews T, Danese A, Wertz J, Odgers CL, Ambler A, Moffitt TE, et al. Social isolation, loneliness and depression in young adulthood: a behavioural genetic analysis. Soc Psychiatry Psychiatr Epidemiol. Mar 2016;51(3):339-348. [FREE Full text] [CrossRef] [Medline]114], the literature examining the relationship between social support and PrEP adherence has shown mixed results across studies, where varying sources and kinds of social support show differing effects [Nacht CL, Reynolds HE, Jessup O, Amato M, Storholm ED. The association between social support and pre-exposure prophylaxis use among sexual minority men in the United States: a scoping literature review. AIDS Behav. Nov 2024;28(11):3559-3573. [CrossRef] [Medline]115]. Furthermore, the provision of social support was one of the putative mechanisms in the P3 DHI. Future research may consider exploring how participants with symptoms of depression engage with specific features of DHIs to explore mechanisms for how depression impacts PrEP nonadherence. For example, participants with depression may only be engaging with the instrumental support modules (eg, medication tracker) or gamification elements (eg, quests) and avoiding the social elements (eg, social wall). Conversely, the social wall may not be enough or the right kind of social support, despite heavy engagement among participants with depression.

The impact symptoms of anxiety have on DHI engagement and efficacy is unclear. DiMatteo et al [DiMatteo MR, Lepper HS, Croghan TW. Depression is a risk factor for noncompliance with medical treatment: meta-analysis of the effects of anxiety and depression on patient adherence. Arch Intern Med. Jul 24, 2000;160(14):2101-2107. [CrossRef] [Medline]83] note that the range in effect sizes and significance found in their meta-analysis may align with the large degree of heterogeneity in anxiety disorders. For example, symptoms associated with a panic disorder may be quite different from symptoms associated with obsessive-compulsive disorder. In this study, we measured symptoms associated with generalized anxiety disorder using the GAD-7 questionnaire, which measures symptoms of restlessness, feeling on edge, or irritability [Spitzer RL, Kroenke K, Williams JB, Löwe B. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med. May 22, 2006;166(10):1092-1097. [CrossRef] [Medline]80,Khdour HY, Abushalbaq OM, Mughrabi IT, Imam AF, Gluck MA, Herzallah MM, et al. Generalized anxiety disorder and social anxiety disorder, but not panic anxiety disorder, are associated with higher sensitivity to learning from negative feedback: behavioral and computational investigation. Front Integr Neurosci. Jun 29, 2016;10:20. [FREE Full text] [CrossRef] [Medline]116]. It is plausible that a generalized feeling of irritability or restlessness could negatively impact one’s capability to complete daily tasks in P3. Furthermore, Sundbom and Bingefors [Sundbom LT, Bingefors K. The influence of symptoms of anxiety and depression on medication nonadherence and its causes: a population based survey of prescription drug users in Sweden. Patient Prefer Adherence. Aug 19, 2013;7:805-811. [FREE Full text] [CrossRef] [Medline]84] found that among men with anxiety, the stated reason for medication nonadherence with the largest effect size was fear of potential adverse drug reactions (OR 3.07, 95% CI 1.55-6.08). However, fearing an adverse drug reaction does not seem like a likely mechanism that can explain the lowered engagement among participants with anxiety using P3, as using a DHI has few, if any, documented risks or adverse reactions. This represents a largely unexplored dimension of DHI engagement research. Like depression, future work on DHI engagement and efficacy should assess the relationship between symptoms of anxiety disorders and specific DHI modules. This will help elucidate the mechanisms that diminish engagement among those with symptoms consistent with generalized anxiety disorder. Furthermore, while the GAD-7 questionnaire does not explicitly screen for social anxiety disorder, there is significant symptom overlap between these 2 conditions and a high degree of comorbidity, with previous clinical trial findings revealing that 57% to 77% of individuals aged between 7 and 17 years with generalized anxiety disorder also have social anxiety disorder [Khdour HY, Abushalbaq OM, Mughrabi IT, Imam AF, Gluck MA, Herzallah MM, et al. Generalized anxiety disorder and social anxiety disorder, but not panic anxiety disorder, are associated with higher sensitivity to learning from negative feedback: behavioral and computational investigation. Front Integr Neurosci. Jun 29, 2016;10:20. [FREE Full text] [CrossRef] [Medline]116-Hearn CS, Donovan CL, Spence SH, March S, Holmes MC. What's the worry with social anxiety? Comparing cognitive processes in children with generalized anxiety disorder and social anxiety disorder. Child Psychiatry Hum Dev. Oct 2017;48(5):786-795. [CrossRef] [Medline]119]. One possible hypothesis to explore in future work is how social anxiety impacts engagement in DHIs with a heavy emphasis on social interaction (such as P3). A 2020 global study of social anxiety rates found that 58% of US individuals aged between 18 and 24 years had scores on the Social Anxiety Scale consistent with social anxiety (≥29) [Jefferies P, Ungar M. Social anxiety in young people: a prevalence study in seven countries. PLoS One. Sep 17, 2020;15(9):e0239133. [FREE Full text] [CrossRef] [Medline]120]. Due to the preponderance of individuals aged between 18 and 24 years with symptoms consistent with social anxiety, future work that aims to develop DHIs for adolescents and young adults should consider further explicating the role of social anxiety in DHI engagement and efficacy.

Strengths and Limitations

This study has several methodological and analytical strengths. The research design of the primary RCT is complementary to the theoretical constructs of effective engagement and causal mediation analysis. RCTs provide clear temporality, which is simultaneously necessary to establish effective engagement (ie, engagement leading to a downstream health outcome) and causal mediation analysis [Yardley L, Spring BJ, Riper H, Morrison LG, Crane DH, Curtis K, et al. Understanding and promoting effective engagement with digital behavior change interventions. Am J Prev Med. Nov 2016;51(5):833-842. [CrossRef] [Medline]25,Vansteelandt S, Bekaert M, Lange T. Imputation strategies for the estimation of natural direct and indirect effects. Epidemiol Methods. 2012;1(1):130. [CrossRef]57-Robins JM, Greenland S. Identifiability and exchangeability for direct and indirect effects. Epidemiology. Mar 1992;3(2):143-155. [CrossRef] [Medline]60]. This study demonstrates the clarity causal mediation analysis provides to engagement research by disentangling total effects into direct and indirect effects, allowing a causal and mechanistic characterization of how baseline intrapersonal measures relate to effective engagement. This approach helps to avoid mischaracterizations that can occur with traditional regression techniques, as these techniques often model engagement as the outcome and rely on the assumption that an increase in engagement would precipitate an increase in the behavior change of interest. For example, quantitative research into medication adherence disparities has found disparities exist among racial or ethnic groups after adjusting for other socioeconomic confounders [Xie Z, St Clair P, Goldman DP, Joyce G. Racial and ethnic disparities in medication adherence among privately insured patients in the United States. PLoS One. Feb 14, 2019;14(2):e0212117. [FREE Full text] [CrossRef] [Medline]107], and qualitative research into engagement with DHIs has largely lacked a focus on race and ethnicity [O'Connor S, Hanlon P, O'Donnell CA, Garcia S, Glanville J, Mair FS. Understanding factors affecting patient and public engagement and recruitment to digital health interventions: a systematic review of qualitative studies. BMC Med Inform Decis Mak. Sep 15, 2016;16(1):120. [FREE Full text] [CrossRef] [Medline]38], which suggests that race and ethnicity might play a role in effective engagement for adherence DHIs. In this study, non-Hispanic White participants earned significantly more money than other trial participants (US $17) 3 months into the trial period. However, effect decomposition demonstrated that the effect of race and ethnicity on PrEP nonadherence was largely direct, suggesting that race and ethnicity is related to PrEP adherence irrespective of the observed increase in dollars accrued (ie, engagement) among non-Hispanic White participants. This example illustrates how traditional regression and causal mediation approaches would arrive at divergent conclusions based on the same data. Traditional approaches assume engagement leads to a behavior change, whereas causal mediation analysis estimates behavior change based on changes in engagement. In this example, given the results from the causal mediation analysis, it seems substantially more likely that reported non-Hispanic White race and ethnicity is a proxy for the systematic inequalities people of sexual, gender, racial, and ethnic minority face in the United States and that these inequalities propagate difficulties with PrEP adherence in ways P3 cannot or did not address.

One limitation of this study is the small and selective sample size. The small sample size limits the power to detect effects. This means that some of the null results in this study may in fact be more significant in a study with a larger sample. However, this also means that this study is likely a conservative measure of effective engagement. The significant results from this study which have relatively small effect sizes may in truth be much larger. Furthermore, 13.5% (22/163) of the eligible participants were LTFU, which is consistent with the primary P3 efficacy RCT, which observed 13% LTFU across all intervention conditions. Due to this, we believe that participants were not LTFU for reasons specific to study operations or intervention conditions. Similarly, the sample of individuals in the primary RCT is relatively homogenous (small age range, relatively high digital literacy, generally nonrural, and same sexual orientation). Therefore, the results of this study may or may not be generalizable to other populations. However, while this feature of the primary RCT limits generalizability, it also strengthens confounding assumptions necessary to carry out causal mediation analysis. By having a more homogenous sample, many of the measures that may have been conceived as confounders have been controlled for through the primary RCT’s study design, such as sexual orientation and English literacy. Furthermore, while several aspects of the population are homogenous, there is a high degree of geographic diversity as the primary RCT was carried out at 9 study sites. Another limitation is the degree of missingness in the biological measures of adherence. Before the COVID-19 pandemic, participants completed study activities in person at study sites, which included the collection of biological specimens by study site staff. Once COVID-19 restrictions were in place and study activities resumed, most activities were completed remotely to enable the continuation of data collection. Participants were asked to complete the at-home dried blood spot collection. The change from site-directed biological specimen collection to dried blood spot self-collection likely impacted the amount of missing biological specimen data. However, self-report measures are accurate for estimating protective serum levels of PrEP among adults [Agot K, Taylor D, Corneli AL, Wang M, Ambia J, Kashuba AD, et al. Accuracy of self-report and pill-count measures of adherence in the FEM-PrEP clinical trial: implications for future HIV-prevention trials. AIDS Behav. May 7, 2015;19(5):743-751. [FREE Full text] [CrossRef] [Medline]69,Blumenthal J, Pasipanodya EC, Jain S, Sun S, Ellorin E, Morris S, et al. Comparing self-report pre-exposure prophylaxis adherence questions to pharmacologic measures of recent and cumulative pre-exposure prophylaxis exposure. Front Pharmacol. 2019;10:721. [FREE Full text] [CrossRef] [Medline]70] and the likelihood of incorrectly estimating protective serum levels of PrEP decreases significantly with age among adolescents and young adults [Baker Z, Javanbakht M, Mierzwa S, Pavel C, Lally M, Zimet G, et al. Predictors of over-reporting HIV pre-exposure prophylaxis (PrEP) adherence among young men who have sex with men (YMSM) in self-reported versus biomarker data. AIDS Behav. Apr 2018;22(4):1174-1183. [FREE Full text] [CrossRef] [Medline]71]. Given that the average age of participants was 22 years in this study, combined with the relatively high area under the receiver operating characteristics curve (≥0.7) between self-report measures and biological measures among participants without missing biological measures, using self-report PrEP adherence measures where biological measures are missing seems sufficient. Finally, the inclusion of a validated digital literacy scale and social anxiety–specific scale (as opposed to only a scale of generalized anxiety disorder) would have been ideal to measure digital literacy and social anxiety, respectively.

Conclusions

This study used a causal mediation approach using secondary data from an RCT testing the efficacy of P3, a digital PrEP adherence intervention. This study combined data from the primary RCT, including biological PrEP adherence measures, with engagement data, to characterize how baseline intrapersonal measures relate to effective engagement in participants who received P3. Broadly, P3 engagement (dollars accrued) was strongly related to lower odds of PrEP nonadherence. Specifically, this study identified digital literacy as a potential engagement facilitator and measures of structural disparity (eg, disconnection from phone or internet in the past year) and mental health (eg, anxious symptoms) as engagement barriers. Study results suggest tailoring as a critical DHI mechanism to address barriers to engagement and emphasize engagement facilitators in indicated individuals. Furthermore, these findings highlight the suitability of causal mediation analysis for effective engagement research by delineating the total effect of each intrapersonal measure into direct and indirect effects (effective engagement). Future research into effective engagement would benefit from adopting a causal mediation approach. Furthermore, as hypotheses regarding exact mechanisms for fostering engagement arise, future research should measure engagement with measure-specific areas of the DHI as a mediator.

Acknowledgments

The authors would like to collectively acknowledge and thank the Prepared, Protected, Empowered (P3) study team, P3 study participants, iTech, and the Adolescent Medicine Trials Network for HIV Interventions for their continued support and contributions to this project. MPW is supported by the National Institutes of Health: Eunice Kennedy Shriver National Institute of Child Health and Human Development (5F31HD108046) and National Institute on Aging (1T32AG081327). DFH is supported by the National Institutes of Health: National Institute on Drug Abuse (5K01DA046307).

Authors' Contributions

MPW and DFH conceived and designed the study. LBH-W conceived and designed the primary study (Prepared, Protected, Empowered [P3]). MPW and the P3 study team, namely CBR, constructed the analytic dataset. LHS, JM, and MPW built the statistical models. All authors revised the manuscript and interpreted the data.

Conflicts of Interest

None declared.

  1. Baral SD, Poteat T, Strömdahl S, Wirtz AL, Guadamuz TE, Beyrer C. Worldwide burden of HIV in transgender women: a systematic review and meta-analysis. Lancet Infect Dis. Mar 2013;13(3):214-222. [CrossRef] [Medline]
  2. HIV and young men who have sex with men. Centers for Disease Control and Prevention. URL: https://tinyurl.com/2azvbmue [accessed 2023-04-12]
  3. HIV surveillance report, 2018 (updated). Centers for Disease Control and Prevention. 2018. URL: https://tinyurl.com/3968pb23 [accessed 2024-12-17]
  4. Men who have sex with men (MSM). Centers for Disease Control and Prevention. URL: https://tinyurl.com/yhrf6c64 [accessed 2023-04-12]
  5. Transgender and gender diverse persons. Centers for Disease Control and Prevention. URL: https://tinyurl.com/4pe2snn7 [accessed 2023-04-12]
  6. Clark H, Babu AS, Wiewel EW, Opoku J, Crepaz N. Diagnosed HIV infection in transgender adults and adolescents: results from the National HIV Surveillance System, 2009-2014. AIDS Behav. Sep 29, 2017;21(9):2774-2783. [FREE Full text] [CrossRef] [Medline]
  7. Hall HI, Walker F, Shah D, Belle E. Trends in HIV diagnoses and testing among U.S. adolescents and young adults. AIDS Behav. Jan 12, 2012;16(1):36-43. [CrossRef] [Medline]
  8. HIV surveillance report, 2020. Centers for Disease Control and Prevention. May 2022. URL: https://tinyurl.com/28s94jhy [accessed 2023-09-05]
  9. Buchbinder SP, Glidden DV, Liu AY, McMahan V, Guanira JV, Mayer KH, et al. HIV pre-exposure prophylaxis in men who have sex with men and transgender women: a secondary analysis of a phase 3 randomised controlled efficacy trial. Lancet Infect Dis. Jun 2014;14(6):468-475. [FREE Full text] [CrossRef] [Medline]
  10. Grant RM, Lama JR, Anderson PL, McMahan V, Liu AY, Vargas L, et al. Preexposure chemoprophylaxis for HIV prevention in men who have sex with men. N Engl J Med. Dec 30, 2010;363(27):2587-2599. [FREE Full text] [CrossRef] [Medline]
  11. Grant RM, Anderson PL, McMahan V, Liu A, Amico KR, Mehrotra M, et al. Uptake of pre-exposure prophylaxis, sexual practices, and HIV incidence in men and transgender women who have sex with men: a cohort study. Lancet Infect Dis. Sep 2014;14(9):820-829. [FREE Full text] [CrossRef] [Medline]
  12. Callander D, McManus H, Gray RT, Grulich AE, Carr A, Hoy J, et al. HIV treatment-as-prevention and its effect on incidence of HIV among cisgender gay, bisexual, and other men who have sex with men in Australia: a 10-year longitudinal cohort study. The Lancet HIV. Jun 2023;10(6):e385-e393. [CrossRef]
  13. LeVasseur MT, Goldstein ND, Tabb LP, Olivieri-Mui BL, Welles SL. The effect of PrEP on HIV incidence among men who have sex with men in the context of condom use, treatment as prevention, and seroadaptive practices. J Acquir Immune Defic Syndr. Jan 01, 2018;77(1):31-40. [CrossRef] [Medline]
  14. Bauermeister JA, Golinkoff JM, Horvath KJ, Hightow-Weidman LB, Sullivan PS, Stephenson R. A multilevel tailored web app-based intervention for linking young men who have sex with men to quality care (get connected): protocol for a randomized controlled trial. JMIR Res Protoc. Aug 02, 2018;7(8):e10444. [FREE Full text] [CrossRef] [Medline]
  15. Biello KB, Marrow E, Mimiaga MJ, Sullivan P, Hightow-Weidman L, Mayer KH. A mobile-based app (MyChoices) to increase uptake of HIV testing and pre-exposure prophylaxis by young men who have sex with men: protocol for a pilot randomized controlled trial. JMIR Res Protoc. Jan 07, 2019;8(1):e10694. [FREE Full text] [CrossRef] [Medline]
  16. Garcia J, Parker C, Parker RG, Wilson PA, Philbin M, Hirsch JS. Psychosocial implications of homophobia and HIV stigma in social support networks: insights for high-impact HIV prevention among Black men who have sex with men. Health Educ Behav. Apr 2016;43(2):217-225. [FREE Full text] [CrossRef] [Medline]
  17. Harawa NT, Bempong GA, Fields S, Leibowitz A, Lee D, Meyer P, et al. Engaging Black men who have sex with men (BMSM) in Los Angeles in HIV preExposure Prophylaxis (PrEP). California HIV/AIDS Policy. Feb 2016. URL: https://chipts.ucla.edu/wp-content/uploads/2016/02/White-Paper-Black-MSM-and-PrEP.pdf [accessed 2024-12-17]
  18. Liu A, Coleman K, Bojan K, Serrano PA, Oyedele T, Garcia A, et al. Developing a mobile app (LYNX) to support linkage to HIV/sexually transmitted infection testing and pre-exposure prophylaxis for young men who have sex with men: protocol for a randomized controlled trial. JMIR Res Protoc. Jan 25, 2019;8(1):e10659. [FREE Full text] [CrossRef] [Medline]
  19. LeGrand S, Knudtson K, Benkeser D, Muessig K, Mcgee A, Sullivan PS, et al. Testing the efficacy of a social networking gamification app to improve pre-exposure prophylaxis adherence (P3: Prepared, Protected, emPowered): protocol for a randomized controlled trial. JMIR Res Protoc. Dec 18, 2018;7(12):e10448. [FREE Full text] [CrossRef] [Medline]
  20. Songtaweesin WN, LeGrand S, Bandara S, Piccone C, Wongharn P, Moonwong J, et al. Adaptation of a theory-based social networking and gamified app-based intervention to improve pre-exposure prophylaxis adherence among young men who have sex with men in Bangkok, Thailand: qualitative study. J Med Internet Res. Nov 04, 2021;23(11):e23852. [FREE Full text] [CrossRef] [Medline]
  21. Chiasson MA, Hirshfield S, Rietmeijer C. HIV prevention and care in the digital age. J Acquir Immune Defic Syndr. Dec 2010;55 Suppl 2:S94-S97. [CrossRef] [Medline]
  22. Kirby T, Thornber-Dunwell M. Phone apps could help promote sexual health in MSM. Lancet. Oct 18, 2014;384(9952):1415. [CrossRef] [Medline]
  23. Muessig KE, Pike EC, Fowler B, LeGrand S, Parsons JT, Bull SS, et al. Putting prevention in their pockets: developing mobile phone-based HIV interventions for Black men who have sex with men. AIDS Patient Care STDS. Apr 2013;27(4):211-222. [FREE Full text] [CrossRef] [Medline]
  24. Sullivan PS, Grey JA, Simon Rosser BR. Emerging technologies for HIV prevention for MSM: what we have learned, and ways forward. J Acquir Immune Defic Syndr. Jun 01, 2013;63 Suppl 1(0 1):S102-S107. [FREE Full text] [CrossRef] [Medline]
  25. Yardley L, Spring BJ, Riper H, Morrison LG, Crane DH, Curtis K, et al. Understanding and promoting effective engagement with digital behavior change interventions. Am J Prev Med. Nov 2016;51(5):833-842. [CrossRef] [Medline]
  26. Brothers J, Hosek S, Keckler K, Anderson PL, Xiong D, Liu H, et al. The ATEAM study: advances in technology to enhance PrEP adherence monitoring (ATEAM) among young men who have sex with men. Clin Transl Sci. Dec 2022;15(12):2947-2957. [FREE Full text] [CrossRef] [Medline]
  27. Buchbinder SP, Siegler AJ, Coleman K, Vittinghoff E, Wilde G, Lockard A, et al. Randomized controlled trial of automated directly observed therapy for measurement and support of PrEP adherence among young men who have sex with men. AIDS Behav. Feb 2023;27(2):719-732. [FREE Full text] [CrossRef] [Medline]
  28. Chai PR, Mohamed Y, Bustamante MJ, Goodman GR, Najarro J, Castillo-Mancilla J, et al. DigiPrEP: a pilot trial to evaluate the feasibility, acceptability, and accuracy of a digital pill system to measure PrEP adherence in men who have sex with men who use substances. J Acquir Immune Defic Syndr. Feb 01, 2022;89(2):e5-15. [FREE Full text] [CrossRef] [Medline]
  29. Serrano VB, Moore DJ, Morris S, Tang B, Liao A, Hoenigl M, et al. Efficacy of daily text messaging to support adherence to HIV pre-exposure prophylaxis (PrEP) among stimulant-using men who have sex with men. Subst Use Misuse. 2023;58(3):465-469. [FREE Full text] [CrossRef] [Medline]
  30. Goyal S, Nunn CA, Rotondi M, Couperthwaite AB, Reiser S, Simone A, et al. A mobile app for the self-management of type 1 diabetes among adolescents: a randomized controlled trial. JMIR Mhealth Uhealth. Jun 19, 2017;5(6):e82. [FREE Full text] [CrossRef] [Medline]
  31. Hightow-Weidman LB, Horvath KJ, Scott H, Hill-Rorie J, Bauermeister JA. Engaging youth in mHealth: what works and how can we be sure? Mhealth. 2021;7:23. [FREE Full text] [CrossRef] [Medline]
  32. Hightow-Weidman LB, Bauermeister JA. Engagement in mHealth behavioral interventions for HIV prevention and care: making sense of the metrics. Mhealth. Jan 2020;6:7. [FREE Full text] [CrossRef] [Medline]
  33. Hawkins RP, Kreuter M, Resnicow K, Fishbein M, Dijkstra A. Understanding tailoring in communicating about health. Health Educ Res. Jun 2008;23(3):454-466. [FREE Full text] [CrossRef] [Medline]
  34. Horvath KJ, Bauermeister JA. eHealth literacy and intervention tailoring impacts the acceptability of a HIV/STI testing intervention and sexual decision making among young gay and bisexual men. AIDS Educ Prev. Feb 2017;29(1):14-23. [FREE Full text] [CrossRef] [Medline]
  35. Kreuter MW, Skinner CS. Tailoring: what's in a name? Health Educ Res. Feb 2000;15(1):1-4. [CrossRef] [Medline]
  36. Morrison LG, Yardley L, Powell J, Michie S. What design features are used in effective e-health interventions? A review using techniques from Critical Interpretive Synthesis. Telemed J E Health. Mar 2012;18(2):137-144. [FREE Full text] [CrossRef] [Medline]
  37. Noar SM, Benac CN, Harris MS. Does tailoring matter? Meta-analytic review of tailored print health behavior change interventions. Psychol Bull. Jul 2007;133(4):673-693. [CrossRef] [Medline]
  38. O'Connor S, Hanlon P, O'Donnell CA, Garcia S, Glanville J, Mair FS. Understanding factors affecting patient and public engagement and recruitment to digital health interventions: a systematic review of qualitative studies. BMC Med Inform Decis Mak. Sep 15, 2016;16(1):120. [FREE Full text] [CrossRef] [Medline]
  39. Danaher BG, Seeley JR. Methodological issues in research on web-based behavioral interventions. Ann Behav Med. Aug 2009;38(1):28-39. [FREE Full text] [CrossRef] [Medline]
  40. Greenhalgh T, Hinder S, Stramer K, Bratan T, Russell J. Adoption, non-adoption, and abandonment of a personal electronic health record: case study of HealthSpace. BMJ. Nov 16, 2010;341:c5814. [FREE Full text] [CrossRef] [Medline]
  41. Flynn D, Gregory P, Makki H, Gabbay M. Expectations and experiences of eHealth in primary care: a qualitative practice-based investigation. Int J Med Inform. Sep 2009;78(9):588-604. [CrossRef] [Medline]
  42. Middlemass J, Davy Z, Cavanagh K, Linehan C, Morgan K, Lawson S, et al. Integrating online communities and social networks with computerised treatment for insomnia: a qualitative study. Br J Gen Pract. Dec 2012;62(605):e840-e850. [FREE Full text] [CrossRef] [Medline]
  43. Hottes TS, Farrell J, Bondyra M, Haag D, Shoveller J, Gilbert M. Internet-based HIV and sexually transmitted infection testing in British Columbia, Canada: opinions and expectations of prospective clients. J Med Internet Res. Mar 06, 2012;14(2):e41. [FREE Full text] [CrossRef] [Medline]
  44. Winkelman WJ, Leonard KJ, Rossos PG. Patient-perceived usefulness of online electronic medical records: employing grounded theory in the development of information and communication technologies for use by patients living with chronic illness. J Am Med Inform Assoc. 2005;12(3):306-314. [FREE Full text] [CrossRef] [Medline]
  45. Greenhalgh T, Wood GW, Bratan T, Stramer K, Hinder S. Patients' attitudes to the summary care record and HealthSpace: qualitative study. BMJ. Jun 07, 2008;336(7656):1290-1295. [FREE Full text] [CrossRef] [Medline]
  46. Beattie A, Shaw A, Kaur S, Kessler D. Primary-care patients' expectations and experiences of online cognitive behavioural therapy for depression: a qualitative study. Health Expect. Mar 2009;12(1):45-59. [FREE Full text] [CrossRef] [Medline]
  47. Fukuoka Y, Kamitani E, Bonnet K, Lindgren T. Real-time social support through a mobile virtual community to improve healthy behavior in overweight and sedentary adults: a focus group analysis. J Med Internet Res. Jul 14, 2011;13(3):e49. [FREE Full text] [CrossRef] [Medline]
  48. Hopp FP, Hogan MM, Woodbridge PA, Lowery JC. The use of telehealth for diabetes management: a qualitative study of telehealth provider perceptions. Implement Sci. May 02, 2007;2:14. [FREE Full text] [CrossRef] [Medline]
  49. Lorimer K, Martin S, McDaid LM. The views of general practitioners and practice nurses towards the barriers and facilitators of proactive, internet-based chlamydia screening for reaching young heterosexual men. BMC Fam Pract. Jun 27, 2014;15:127. [FREE Full text] [CrossRef] [Medline]
  50. Mac Domhnaill C, Mohan G, McCoy S. Home broadband and student engagement during COVID-19 emergency remote teaching. Distance Educ. Nov 03, 2021;42(4):465-493. [CrossRef]
  51. McCloud RF, Okechukwu CA, Sorensen G, Viswanath K. Beyond access: barriers to internet health information seeking among the urban poor. J Am Med Inform Assoc. Nov 2016;23(6):1053-1059. [FREE Full text] [CrossRef] [Medline]
  52. Mesch GS, Talmud I. Internet connectivity, community participation, and place attachment: a longitudinal study. Am Behav Sci. Feb 18, 2010;53(8):1095-1110. [CrossRef]
  53. Nahum-Shani I, Shaw SD, Carpenter SM, Murphy SA, Yoon C. Engagement in digital interventions. Am Psychol. Oct 2022;77(7):836-852. [FREE Full text] [CrossRef] [Medline]
  54. Strecher VJ, McClure J, Alexander G, Chakraborty B, Nair V, Konkel J, et al. The role of engagement in a tailored web-based smoking cessation program: randomized controlled trial. J Med Internet Res. Nov 04, 2008;10(5):e36. [FREE Full text] [CrossRef] [Medline]
  55. Van Gemert-Pijnen JE, Kelders SM, Bohlmeijer ET. Understanding the usage of content in a mental health intervention for depression: an analysis of log data. J Med Internet Res. Jan 31, 2014;16(1):e27. [FREE Full text] [CrossRef] [Medline]
  56. Michie S, Yardley L, West R, Patrick K, Greaves F. Developing and evaluating digital interventions to promote behavior change in health and health care: recommendations resulting from an international workshop. J Med Internet Res. Jun 29, 2017;19(6):e232. [FREE Full text] [CrossRef] [Medline]
  57. Vansteelandt S, Bekaert M, Lange T. Imputation strategies for the estimation of natural direct and indirect effects. Epidemiol Methods. 2012;1(1):130. [CrossRef]
  58. VanderWeele TJ. Mediation analysis: a practitioner's guide. Annu Rev Public Health. 2016;37:17-32. [CrossRef] [Medline]
  59. Vanderweele TJ, Vansteelandt S. Odds ratios for mediation analysis for a dichotomous outcome. Am J Epidemiol. Dec 15, 2010;172(12):1339-1348. [FREE Full text] [CrossRef] [Medline]
  60. Robins JM, Greenland S. Identifiability and exchangeability for direct and indirect effects. Epidemiology. Mar 1992;3(2):143-155. [CrossRef] [Medline]
  61. VanderWeele TJ. Explanation in causal inference: developments in mediation and interaction. Int J Epidemiol. Dec 01, 2016;45(6):1904-1908. [FREE Full text] [CrossRef] [Medline]
  62. VanderWeele TJ. Explanation in Causal Inference: Methods for Mediation and Interaction. New York, NY. Oxford University Press; 2015.
  63. Oberg AS, VanderWeele TJ, Almqvist C, Hernandez-Diaz S. Pregnancy complications following fertility treatment-disentangling the role of multiple gestation. Int J Epidemiol. Aug 01, 2018;47(4):1333-1342. [FREE Full text] [CrossRef] [Medline]
  64. Hernán MA. A definition of causal effect for epidemiological research. J Epidemiol Community Health. Apr 2004;58(4):265-271. [FREE Full text] [CrossRef] [Medline]
  65. Höfler M. Causal inference based on counterfactuals. BMC Med Res Methodol. Sep 13, 2005;5:28. [FREE Full text] [CrossRef] [Medline]
  66. P3 (Prepared, Protected, emPowered): promoting pre-exposure prophylaxis (PrEP) adherence through a social networking, gamification, and adherence support app. National Institutes of Health National Library of Medicine. Oct 25, 2022. URL: https://clinicaltrials.gov/ct2/show/NCT03320512 [accessed 2023-05-31]
  67. Amico KR, Miller J, Balthazar C, Serrano PA, Brothers J, Zollweg S, et al. Integrated next step counseling (iNSC) for sexual health and PrEPuse among young men who have sex with men: implementation and observations from ATN110/113. AIDS Behav. Jul 2019;23(7):1812-1823. [FREE Full text] [CrossRef] [Medline]
  68. R Amico K, McMahan V, Goicochea P, Vargas L, Marcus JL, Grant RM, et al. Supporting study product use and accuracy in self-report in the iPrEx study: next step counseling and neutral assessment. AIDS Behav. Jul 2012;16(5):1243-1259. [CrossRef] [Medline]
  69. Agot K, Taylor D, Corneli AL, Wang M, Ambia J, Kashuba AD, et al. Accuracy of self-report and pill-count measures of adherence in the FEM-PrEP clinical trial: implications for future HIV-prevention trials. AIDS Behav. May 7, 2015;19(5):743-751. [FREE Full text] [CrossRef] [Medline]
  70. Blumenthal J, Pasipanodya EC, Jain S, Sun S, Ellorin E, Morris S, et al. Comparing self-report pre-exposure prophylaxis adherence questions to pharmacologic measures of recent and cumulative pre-exposure prophylaxis exposure. Front Pharmacol. 2019;10:721. [FREE Full text] [CrossRef] [Medline]
  71. Baker Z, Javanbakht M, Mierzwa S, Pavel C, Lally M, Zimet G, et al. Predictors of over-reporting HIV pre-exposure prophylaxis (PrEP) adherence among young men who have sex with men (YMSM) in self-reported versus biomarker data. AIDS Behav. Apr 2018;22(4):1174-1183. [FREE Full text] [CrossRef] [Medline]
  72. Horvath KJ, Danilenko GP, Williams ML, Simoni J, Amico KR, Oakes JM, et al. Technology use and reasons to participate in social networking health websites among people living with HIV in the US. AIDS Behav. May 2012;16(4):900-910. [FREE Full text] [CrossRef] [Medline]
  73. Payton S, Hague C. Digital literacy across the curriculum. National Foundation for Educational Research. Jan 1, 2010. URL: https://www.nfer.ac.uk/publications/digital-literacy-across-the-curriculum/ [accessed 2023-09-13]
  74. Lepore SJ, Rincon MA, Buzaglo JS, Golant M, Lieberman MA, Bauerle Bass S, et al. Digital literacy linked to engagement and psychological benefits among breast cancer survivors in internet-based peer support groups. Eur J Cancer Care (Engl). Jul 2019;28(4):e13134. [FREE Full text] [CrossRef] [Medline]
  75. Melhem SJ, Nabhani-Gebara S, Kayyali R. Digital trends, digital literacy, and e-health engagement predictors of breast and colorectal cancer survivors: a population-based cross-sectional survey. Int J Environ Res Public Health. Jan 13, 2023;20(2):1472. [FREE Full text] [CrossRef] [Medline]
  76. Im EO, Lee B, Chee W. Shielded from the real world: perspectives on internet cancer support groups by Asian Americans. Cancer Nurs. 2010;33(3):E10-E20. [FREE Full text] [CrossRef] [Medline]
  77. Dasgupta K, Da Costa D, Pillay S, De Civita M, Gougeon R, Leong A, et al. Strategies to optimize participation in diabetes prevention programs following gestational diabetes: a focus group study. PLoS One. 2013;8(7):e67878. [FREE Full text] [CrossRef] [Medline]
  78. Johnson JG, Harris ES, Spitzer RL, Williams JB. The patient health questionnaire for adolescents: validation of an instrument for the assessment of mental disorders among adolescent primary care patients. J Adolesc Health. Mar 2002;30(3):196-204. [CrossRef] [Medline]
  79. Kroenke K, Strine TW, Spitzer RL, Williams JB, Berry JT, Mokdad AH. The PHQ-8 as a measure of current depression in the general population. J Affect Disord. Apr 2009;114(1-3):163-173. [CrossRef] [Medline]
  80. Spitzer RL, Kroenke K, Williams JB, Löwe B. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med. May 22, 2006;166(10):1092-1097. [CrossRef] [Medline]
  81. Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med. Sep 2001;16(9):606-613. [FREE Full text] [CrossRef] [Medline]
  82. Yeager CM, Shoji K, Luszczynska A, Benight CC. Engagement with a trauma recovery internet intervention explained with the health action process approach (HAPA): longitudinal study. JMIR Ment Health. Apr 10, 2018;5(2):e29. [FREE Full text] [CrossRef] [Medline]
  83. DiMatteo MR, Lepper HS, Croghan TW. Depression is a risk factor for noncompliance with medical treatment: meta-analysis of the effects of anxiety and depression on patient adherence. Arch Intern Med. Jul 24, 2000;160(14):2101-2107. [CrossRef] [Medline]
  84. Sundbom LT, Bingefors K. The influence of symptoms of anxiety and depression on medication nonadherence and its causes: a population based survey of prescription drug users in Sweden. Patient Prefer Adherence. Aug 19, 2013;7:805-811. [FREE Full text] [CrossRef] [Medline]
  85. Beatty L, Binnion C. A systematic review of predictors of, and reasons for, adherence to online psychological interventions. Int J Behav Med. Dec 2016;23(6):776-794. [CrossRef] [Medline]
  86. Christensen H, Griffiths KM, Farrer L. Adherence in internet interventions for anxiety and depression. J Med Internet Res. Apr 24, 2009;11(2):e13. [FREE Full text] [CrossRef] [Medline]
  87. Funk KL, Stevens VJ, Appel LJ, Bauck A, Brantley PJ, Champagne CM, et al. Associations of internet website use with weight change in a long-term weight loss maintenance program. J Med Internet Res. Jul 27, 2010;12(3):e29. [FREE Full text] [CrossRef] [Medline]
  88. Glasgow RE, Christiansen SM, Kurz D, King DK, Woolley T, Faber AJ, et al. Engagement in a diabetes self-management website: usage patterns and generalizability of program use. J Med Internet Res. Jan 25, 2011;13(1):e9. [FREE Full text] [CrossRef] [Medline]
  89. Neil AL, Batterham P, Christensen H, Bennett K, Griffiths KM. Predictors of adherence by adolescents to a cognitive behavior therapy website in school and community-based settings. J Med Internet Res. Feb 23, 2009;11(1):e6. [FREE Full text] [CrossRef] [Medline]
  90. Wang Z, Wang J, Maercker A. Program use and outcome change in a web-based trauma intervention: individual and social factors. J Med Internet Res. Sep 09, 2016;18(9):e243. [FREE Full text] [CrossRef] [Medline]
  91. Geraghty AW, Torres LD, Leykin Y, Pérez-Stable EJ, Muñoz RF. Understanding attrition from international internet health interventions: a step towards global eHealth. Health Promot Int. Sep 2013;28(3):442-452. [FREE Full text] [CrossRef] [Medline]
  92. Yeager CM, Benight CC. If we build it, will they come? Issues of engagement with digital health interventions for trauma recovery. Mhealth. Sep 11, 2018;4:37. [FREE Full text] [CrossRef] [Medline]
  93. Valeri L, Vanderweele TJ. Mediation analysis allowing for exposure-mediator interactions and causal interpretation: theoretical assumptions and implementation with SAS and SPSS macros. Psychol Methods. Jun 2013;18(2):137-150. [FREE Full text] [CrossRef] [Medline]
  94. Shi B, Choirat C, Coull BA, VanderWeele TJ, Valeri L. CMAverse: a suite of functions for reproducible causal mediation analyses. Epidemiology. Sep 01, 2021;32(5):e20-e22. [CrossRef] [Medline]
  95. Norman CD, Skinner HA. eHealth literacy: essential skills for consumer health in a networked world. J Med Internet Res. Jun 16, 2006;8(2):e9. [FREE Full text] [CrossRef] [Medline]
  96. Yang K, Hu Y, Qi H. Digital health literacy: bibliometric analysis. J Med Internet Res. Jul 06, 2022;24(7):e35816. [FREE Full text] [CrossRef] [Medline]
  97. Nelson LA, Pennings JS, Sommer EC, Popescu F, Barkin SL. A 3-item measure of digital health care literacy: development and validation study. JMIR Form Res. Apr 29, 2022;6(4):e36043. [FREE Full text] [CrossRef] [Medline]
  98. Amin H, Malik MA, Akkaya B. Development and validation of digital literacy scale (DLS) and its implication for higher education. Int J Distance Educ E Learn. Jan 04, 2022;7(1):24-43. [CrossRef]
  99. Ma S, Wang Y, Shu Z, Duan Z, Sun L. Development and validation of internet literacy scale for high school students. Educ Inf Technol (Dordr). Apr 03, 2023;29:1-28. [FREE Full text] [CrossRef] [Medline]
  100. Reddy P, Chaudhary K, Sharma B, Hussein S. Essaying the design, development and validation processes of a new digital literacy scale. Online Inf Rev. Jul 20, 2022;47(2):371-397. [CrossRef]
  101. Ali A, Raza AA, Qazi IA. Validated digital literacy measures for populations with low levels of internet experiences. Dev Eng. Nov 2023;8:100107. [CrossRef]
  102. Campbell BR, Ingersoll KS, Flickinger TE, Dillingham R. Bridging the digital health divide: toward equitable global access to mobile health interventions for people living with HIV. Expert Rev Anti Infect Ther. Mar 2019;17(3):141-144. [FREE Full text] [CrossRef] [Medline]
  103. Xiong S, Lu H, Peoples N, Duman EK, Najarro A, Ni Z, et al. Digital health interventions for non-communicable disease management in primary health care in low-and middle-income countries. NPJ Digit Med. Feb 01, 2023;6(1):12. [FREE Full text] [CrossRef] [Medline]
  104. Easton P, Entwistle VA, Williams B. Health in the 'hidden population' of people with low literacy. A systematic review of the literature. BMC Public Health. Aug 05, 2010;10:459. [FREE Full text] [CrossRef] [Medline]
  105. Butzner M, Cuffee Y. Telehealth interventions and outcomes across rural communities in the United States: narrative review. J Med Internet Res. Aug 26, 2021;23(8):e29575. [FREE Full text] [CrossRef] [Medline]
  106. Whiteley L, Craker L, Sun S, Tarantino N, Hershkowitz D, Moskowitz J, et al. Factors associated with PrEP adherence among MSM living in Jackson, Mississippi. J HIV AIDS Soc Serv. 2021;20(3):246-261. [FREE Full text] [CrossRef] [Medline]
  107. Xie Z, St Clair P, Goldman DP, Joyce G. Racial and ethnic disparities in medication adherence among privately insured patients in the United States. PLoS One. Feb 14, 2019;14(2):e0212117. [FREE Full text] [CrossRef] [Medline]
  108. Ezennia O, Geter A, Smith DK. The PrEP care continuum and black men who have sex with men: a scoping review of published data on awareness, uptake, adherence, and retention in PrEP care. AIDS Behav. Oct 2019;23(10):2654-2673. [CrossRef] [Medline]
  109. Denning P, Dinenno E. Communities in crisis: is there a generalized HIV epidemic in impoverished urban areas of the United States? In: Proceedings of the XVIII International AIDS Conference. 2010. Presented at: AIDS 2010; July 18-23, 2010; Vienna, Austria. URL: https://tinyurl.com/7a6cc8uw
  110. Gan DZ, McGillivray L, Han J, Christensen H, Torok M. Effect of engagement with digital interventions on mental health outcomes: a systematic review and meta-analysis. Front Digit Health. Nov 04, 2021;3:764079. [FREE Full text] [CrossRef] [Medline]
  111. Donkin L, Christensen H, Naismith SL, Neal B, Hickie IB, Glozier N. A systematic review of the impact of adherence on the effectiveness of e-therapies. J Med Internet Res. Aug 05, 2011;13(3):e52. [FREE Full text] [CrossRef] [Medline]
  112. Elmer T, Stadtfeld C. Depressive symptoms are associated with social isolation in face-to-face interaction networks. Sci Rep. Jan 29, 2020;10(1):1444. [FREE Full text] [CrossRef] [Medline]
  113. Ge L, Yap CW, Ong R, Heng BH. Social isolation, loneliness and their relationships with depressive symptoms: a population-based study. PLoS One. Aug 23, 2017;12(8):e0182145. [FREE Full text] [CrossRef] [Medline]
  114. Matthews T, Danese A, Wertz J, Odgers CL, Ambler A, Moffitt TE, et al. Social isolation, loneliness and depression in young adulthood: a behavioural genetic analysis. Soc Psychiatry Psychiatr Epidemiol. Mar 2016;51(3):339-348. [FREE Full text] [CrossRef] [Medline]
  115. Nacht CL, Reynolds HE, Jessup O, Amato M, Storholm ED. The association between social support and pre-exposure prophylaxis use among sexual minority men in the United States: a scoping literature review. AIDS Behav. Nov 2024;28(11):3559-3573. [CrossRef] [Medline]
  116. Khdour HY, Abushalbaq OM, Mughrabi IT, Imam AF, Gluck MA, Herzallah MM, et al. Generalized anxiety disorder and social anxiety disorder, but not panic anxiety disorder, are associated with higher sensitivity to learning from negative feedback: behavioral and computational investigation. Front Integr Neurosci. Jun 29, 2016;10:20. [FREE Full text] [CrossRef] [Medline]
  117. Walkup JT, Albano AM, Piacentini J, Birmaher B, Compton SN, Sherrill JT, et al. Cognitive behavioral therapy, sertraline, or a combination in childhood anxiety. N Engl J Med. Dec 25, 2008;359(26):2753-2766. [FREE Full text] [CrossRef] [Medline]
  118. Alomari NA, Bedaiwi SK, Ghasib AM, Kabbarah AJ, Alnefaie SA, Hariri N, et al. Social anxiety disorder: associated conditions and therapeutic approaches. Cureus. Dec 19, 2022;14(12):e32687. [FREE Full text] [CrossRef] [Medline]
  119. Hearn CS, Donovan CL, Spence SH, March S, Holmes MC. What's the worry with social anxiety? Comparing cognitive processes in children with generalized anxiety disorder and social anxiety disorder. Child Psychiatry Hum Dev. Oct 2017;48(5):786-795. [CrossRef] [Medline]
  120. Jefferies P, Ungar M. Social anxiety in young people: a prevalence study in seven countries. PLoS One. Sep 17, 2020;15(9):e0239133. [FREE Full text] [CrossRef] [Medline]


DHI: digital health intervention
GAD-7: Generalized Anxiety Disorder-7
LTFU: lost to follow-up
MSM: men who have sex with men
OR: odds ratio
P3: Prepared, Protected, Empowered
PHQ-8: Patient Health Questionnaire-8
PM: percentage mediated
PrEP: pre-exposure prophylaxis
RCT: randomized controlled trial
YSGMMSM: young sexual and gender minority men who have sex with men


Edited by A Mavragani; submitted 28.02.24; peer-reviewed by A Barnett, W Evans, M Wang; comments to author 10.06.24; revised version received 18.10.24; accepted 13.11.24; published 13.01.25.

Copyright

©Michael P Williams, Justin Manjourides, Louisa H Smith, Crissi B Rainer, Lisa B Hightow-Weidman, Danielle F Haley. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 13.01.2025.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research (ISSN 1438-8871), is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.