This paper is in the following e-collection/theme issue:

Randomized trials (Editor: G. Eysenbach)* (33) Vaccination and Immunization in the Digital Age (347) Influenza and Influenza-Like-Illness (ILI) Surveillance (203) Web-based and Mobile Health Interventions (3128)

Published on in Vol 24, No 10 (2022): October

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/38710, first published .
Influence of Digital Intervention Messaging on Influenza Vaccination Rates Among Adults With Cardiovascular Disease in the United States: Decentralized Randomized Controlled Trial

Influence of Digital Intervention Messaging on Influenza Vaccination Rates Among Adults With Cardiovascular Disease in the United States: Decentralized Randomized Controlled Trial

Influence of Digital Intervention Messaging on Influenza Vaccination Rates Among Adults With Cardiovascular Disease in the United States: Decentralized Randomized Controlled Trial

Authors of this article:

Nell J Marshall1 Author Orcid Image ;   Jennifer L Lee1 Author Orcid Image ;   Jessica Schroeder1 Author Orcid Image ;   Wei-Nchih Lee1 Author Orcid Image ;   Jermyn See1 Author Orcid Image ;   Mohammad Madjid2 Author Orcid Image ;   Mrudula R Munagala3 Author Orcid Image ;   John D Piette4 Author Orcid Image ;   Litjen Tan5 Author Orcid Image ;   Orly Vardeny6 Author Orcid Image ;   Michael Greenberg7 Author Orcid Image ;   Jan Liska8 Author Orcid Image ;   Monica Mercer7 Author Orcid Image ;   Sandrine Samson9 Author Orcid Image
Article Authors Cited by (1) Tweetations (10) Metrics

    Original Paper

    1Evidation Health, Inc, San Mateo, CA, United States

    2David Geffen School of Medicine, University of California, Los Angeles, CA, United States

    3Department of Cardiology, University of Miami, Miami, FL, United States

    4Department of Health Behavior and Health Education, School of Public Health, University of Michigan, Ann Arbor, MI, United States

    5Immunize.org, St. Paul, MN, United States

    6Center for Care Delivery and Outcomes Research, Veterans Health Administration, Minneapolis, MN, United States

    7Sanofi, Swiftwater, PA, United States

    8Sanofi, Gentilly, France

    9Sanofi, Lyon, France

    *these authors contributed equally

    Corresponding Author:

    Nell J Marshall, MPH, DrPH

    Evidation Health, Inc

    63 Bovet Rd #146

    San Mateo, CA, 94402

    United States

    Phone: 1 (415) 515 1985

    Email: nmarshall@evidation.com


    Background: Seasonal influenza affects 5% to 15% of Americans annually, resulting in preventable deaths and substantial economic impact. Influenza infection is particularly dangerous for people with cardiovascular disease, who therefore represent a priority group for vaccination campaigns.

    Objective: We aimed to assess the effects of digital intervention messaging on self-reported rates of seasonal influenza vaccination.

    Methods: This was a randomized, controlled, single-blind, and decentralized trial conducted at individual locations throughout the United States over the 2020-2021 influenza season. Adults with self-reported cardiovascular disease who were members of the Achievement mobile platform were randomized to receive or not receive a series of 6 patient-centered digital intervention messages promoting influenza vaccination. The primary end point was the between-group difference in self-reported vaccination rates at 6 months after randomization. Secondary outcomes included the levels of engagement with the messages and the relationship between vaccination rates and engagement with the messages. Subgroup analyses examined variation in intervention effects by race. Controlling for randomization group, we examined the impact of other predictors of vaccination status, including cardiovascular condition type, vaccine drivers or barriers, and vaccine knowledge.

    Results: Of the 49,138 randomized participants, responses on the primary end point were available for 11,237 (22.87%; 5575 in the intervention group and 5662 in the control group) participants. The vaccination rate was significantly higher in the intervention group (3418/5575, 61.31%) than the control group (3355/5662, 59.25%; relative risk 1.03, 95% CI 1.004-1.066; P=.03). Participants who were older, more educated, and White or Asian were more likely to report being vaccinated. The intervention was effective among White participants (P=.004) but not among people of color (P=.42). The vaccination rate was 13 percentage points higher among participants who completed all 6 intervention messages versus none, and at least 2 completed messages appeared to be needed for effectiveness. Participants who reported a diagnosis of COVID-19 were more likely to be vaccinated for influenza regardless of treatment assignment.

    Conclusions: This personalized, evidence-based digital intervention was effective in increasing vaccination rates in this population of high-risk people with cardiovascular disease.

    Trial Registration: ClinicalTrials.gov NCT04584645; https://clinicaltrials.gov/ct2/show/NCT04584645

    J Med Internet Res 2022;24(10):e38710

    doi:10.2196/38710

    Keywords

    influenza; randomized trial; public health; cardiovascular disease; immunization; vaccination; digital messaging; digital intervention; mobile health; mHealth 


    About 5% to 15% of the US population contracts influenza annually [1], resulting in more than 20,000 deaths [1] and substantial economic impact [2]. For people with cardiovascular disease (CVD), influenza can be particularly dangerous. In one study, the risk of myocardial infarction was 6 times higher within a week of influenza infection [3]. A study of more than 80,000 US adults hospitalized with influenza over 8 seasons found that 1 in every 8 patients developed sudden, serious cardiac complications and that having underlying cardiac disease was significantly associated with experiencing an acute cardiac event with influenza [4]. For these reasons, the Centers for Disease Control and Prevention (CDC) consider persons with CVD to be at high risk for influenza complications and therefore a priority group for vaccination [5].

    Vaccination remains the most effective primary prevention method against influenza, with age-adjusted effectiveness rates of up to 68% over the past 5 years [6]. The CDC reported a 51.4% vaccination rate for 2019-2020 for persons aged 18-64 years who have high-risk conditions such as CVD [7], far below the 70% national vaccination rate goal [8]. Given the increased burden of influenza for people with CVD, even small improvements in vaccination rates could substantially reduce the number of patients having major adverse cardiac events [2,9-11].

    Novel, scalable, cost-optimal, and effective solutions are needed to address barriers to influenza vaccination among people with CVD, such as complacency, time and cost constraints, and a lack of confidence [12]. Observational [13] and randomized controlled trials [14,15] have shown the effectiveness of digital messaging to increase vaccine uptake in general adult populations. In a randomized trial of digital messaging in persons with diabetes, a population also at increased risk of influenza-related complications [16], the vaccination rate was 3.1% higher in the intervention group than the control group. Alternatively stated, 33 people would need to receive the intervention for 1 additional person to become vaccinated.

    The primary objective of this study was to examine the efficacy of a digital intervention designed to increase self-reported influenza vaccination rates in individuals with CVD.


    Study Design

    This 8-month, pragmatic randomized controlled trial was conducted remotely in the United States. Participants were blinded to study participation status to minimize observation bias, although all participants agreed that their survey responses and behavioral data would be used for research purposes before completing each survey (see below).

    All participants were members of the free Achievement mobile health and research platform (Evidation Health, Inc), which includes more than 4 million individuals spanning all 50 states and 90% of zip codes [17]. The platform provides personalized insights and tools to motivate and empower people to take evidence-supported actions to manage their health. Members can connect activity trackers and fitness and health apps to the platform and share self-reported health information. Achievement does not have the ability to access clinical or claims data; it relies solely on member-generated data.

    Ethics Approval

    The trial protocol was approved by Solutions IRB, Yarnell, Arizona (Registration: IORG0007116; Federalwide Assurance: IRB00008523), and registered at ClinicalTrials.gov (NCT04584645). Since the digital intervention messages were consistent with publicly available information on influenza vaccination, we obtained a waiver of informed consent from this Institutional Review Board on the basis that participants would face only minimal risk from the study. Participants were informed about how their survey responses and behavioral data would be used through a Data Usage and Permissions Agreement.

    Digital Intervention Design

    The 6 digital intervention messages were developed using a 3-part approach [18], building on a previous study [16] and the Theory of Planned Behavior [19]. Message designs were refined throughout the development process using Rapid Iterative Testing and Evaluation–inspired methods [20]. See Multimedia Appendix 1 [16,18-21] for details of the development process and the content of the intervention messages (Table S1 and Figures S1-7 in Multimedia Appendix 1), which were delivered via the Achievement platform.

    Participants

    Eligible participants were those aged ≥18 years, living in the United States, with any of the following self-reported conditions on the Achievement platform (eg, through past surveys): atrial fibrillation; abnormal or irregular heart rhythm or other arrhythmic heart disease; cardiac arrest or myocardial infarction; coronary artery disease treated with medication, stenting, percutaneous intervention, or bypass surgery; congestive heart failure; or stroke or cerebrovascular accident.

    Recruitment, Screening, and Enrolling

    Members who met the inclusion criteria were identified for study inclusion (“participants”). Participants took no action to enroll and were not informed about their participation status. We used block randomization by cardiovascular condition to randomize participants into either the intervention group, which received the digital intervention messages, or the control group, which received none of the messages.

    Randomization and Blinding

    Evidation Health, Inc generated the random allocations, enrolled participants, and randomized them using block randomization (arrhythmia vs nonarrhythmia) into the intervention or control group before offering the opportunity to complete any study activities.

    Study Procedures

    Participants were asked to complete the web-based surveys at baseline, 3 months (after 4 digital messages had been sent in the intervention group), and 6 months (after 2 more messages had been sent in the intervention group). Reminder messages were used to motivate survey completion.

    Primary and Other Outcomes

    Participants self-reported their vaccination status (yes or no) via the app at baseline, 3 months, and 6 months. Participants also reported the estimated date of vaccination, if any, on the 3- and 6-month surveys.

    To assess engagement with the intervention messages, we examined platform-generated data indicating that the person had completed a given message and created a summary measure indicating the number of messages completed.

    Each survey measured the drivers and barriers to vaccination as well as vaccine knowledge. The vaccine drivers and barriers of interest included the number of visits to a primary care provider in the 3 months before randomization (none, 1-2, or 3 or more), number of visits to a cardiology specialist in the prior 3 months (none, 1-2, or 3 or more), number of hospitalizations in the prior 3 months (none, 1, or 2 or more), whether a health care provider had offered influenza vaccination (yes, no, or unsure), and whether a health care provider had informed the individual that they were in a “high-risk group” (yes, no, or unsure).

    Vaccine knowledge factors were based on responses to the survey question “What sources of information do you use to learn about the flu vaccine?” with possible responses of health care professionals, family member or peers, social media including blog posts, mobile apps, or conventional news media (eg, television and newspapers).

    Sample Size Calculation

    The sample size was determined a priori for a 2-arm interventional statistical superiority study design with self-reported vaccination rates as the primary outcome [22]. Large studies on the impact of messaging and telephone reminders to improve influenza vaccination rates show a range of effect sizes from 2.5% to 3.5% [23,24]. A total of 8000 participants were needed to detect a 3% difference in vaccination rates with a type I error rate of 0.05 and power of 0.80. Since a participation drop-off of 67% was observed for digital interventions aimed at increasing influenza vaccination in people with diabetes [16], we conservatively estimated an engagement rate of about 16%. The targeted enrollment list therefore included approximately 49,000 individuals to yield the analysis population of 8000 participants.

    Statistical Analysis

    We first compared the unadjusted proportions of participants reporting vaccination at follow-up between the intervention and control groups. In predefined subgroup analyses, we examined variations in intervention effects between White and non-White participants. Process analyses included differences in self-reported vaccination rates within the intervention group by the number of intervention messages completed and intervention participants’ levels of engagement with each message. Controlling for randomization group, we examined other predictors of vaccination status, including cardiovascular condition type, vaccine drivers or barriers, and vaccine knowledge.

    An exploratory objective was to describe the impact of the COVID-19 pandemic on influenza vaccination behavior. Another exploratory objective—self-reported complications from influenza, overall and by vaccination status—was not analyzed because the surveys did not ask about influenza complications. Information on safety and adverse events was not collected, given the minimal-risk nature of the intervention and study.

    Variables were compared at the 5% significance level using 2-sided tests or 2-sided 95% CI unless otherwise specified. Comparison of means used 2-sided Student t test for normal distributions or a Mann-Whitney U test for nonnormal distributions. Comparisons of frequencies used chi-square tests. For the logistic regression model, the P values, odds ratios (ORs), and 95% CIs associated with each of the β parameter estimates were reported. To describe the relative importance of each predictor variable, we calculated their Shapley Additive Explanation values [25]. Kaplan-Meier curves were constructed for time to influenza vaccination, using the participant-estimated dates of influenza vaccination from the 3- and 6-month surveys.


    Participants

    Between July and September 2020, we generated a list of 49,138 candidate participants (Figure 1). Of these, 24,570 were randomized to receive digital intervention messages and 24,568 were randomized to the control group. On September 21, 2020, the first baseline and demographic surveys were sent to these 49,138 participants, and 10,402 (21.17%) completed the baseline survey. In all, 11,237 participants (22.87%) completed the midstudy or final survey by April 11, 2021, yielding groups of 5575 intervention and 5662 control participants who reported vaccination status at either 3 or 6 months after randomization.

    Of the 11,237 participants, the average age was 45 (SD 13) years, 81.18% (n=9122) were White, 78.01% (n=8766) were female, and 86.21% (n=9687) had health insurance (Table 1). More than half (n=6891, 61.32%) had a college degree, and a third (n=3770, 33.55%) had a household income of at least US $75,000. The most commonly reported cardiovascular condition was arrhythmia (intervention: 2251/5575, 40.38%; control: 2331/5662, 41.17%). Baseline characteristics did not differ substantially between groups. Despite previous self-reports of CVD from all participants, almost a third (intervention: 1798/5575, 32.35%; control: 1844/5662, 32.57%) in both groups reported not having any of the listed conditions in the baseline survey. Study participants represented all 50 states and the District of Columbia (Figure S8 in Multimedia Appendix 1).

    ‎
    Figure 1. Disposition of Study Participants.
    View this figure
    Table 1. Baseline demographic and clinical characteristics of the participants.
    CharacteristicIntervention (N=5575)Control (N=5662)
    Age (years; intervention: n=5530; control: n=5607), mean (SD)45.0 (13.5)44.9 (13.3)
    Sex, n (%)a

    		Female4339 (77.83)4427 (78.19)

    		Male1071 (19.21)1114 (19.68)

    		Other169 (3.03)135 (2.38)
    Race/ethnicity, n (%)a

    		American Indian or Alaska Native155 (2.78)143 (2.53)

    		Asian250 (4.48)227 (4)

    		Black or African American388 (6.96)354 (6.25)

    		Hispanic, Latino, or Spanish319 (5.72)375 (6.62)

    		Middle Eastern or North African46 (0.83)50 (0.88)

    		Native Hawaiian or Other Pacific Islander45 (0.81)49 (0.87)

    		White4510 (80.9)4612 (81.46)

    		Other60 (1.08)64 (1.13)

    		Prefer not to answer187 (3.35)179 (3.16)
    Had health insurance, n (%)4815 (86.37)4872 (86.05)
    Had a college degree, n (%)3381 (60.65)3510 (61.99)
    Household income ≥US $75,000, n (%)1861 (33.38)1909 (33.72)
    Cardiovascular condition type, n (%)a

    		Arrhythmia2251 (40.37)2331 (41.17)

    		Atrial fibrillation488 (8.75)496 (8.76)

    		Cardiac arrest112 (2)99 (1.75)

    		Myocardial infarction385 (6.91)412 (7.28)

    		Heart failure332 (5.96)293 (5.17)

    		Coronary artery disease366 (6.57)356 (6.29)

    		Stroke or cerebrovascular accident433 (7.77)436 (7.7)

    		Other cardiovascular diseases539 (9.67)545 (9.63)

    		None of the above diagnosesb1798 (32.25)1844 (32.57)

    aParticipants could choose more than 1 option, and percentages may add up to >100%.

    bDespite previous self-reports of cardiovascular disease from all participants, some reported not having any of the included conditions at baseline. Please see the Limitations section for more details.

    Primary Outcome

    By the end of the study period, 3418 (61.31%) of the 5575 participants in the intervention arm had reported obtaining influenza vaccination compared to 3355 (59.25%) of the 5662 participants in the control arm (absolute difference: 2.06%; relative risk 1.03, 95% CI 1.004-1.066; P=.03). Based on this difference, 48.3 persons would have to receive the digital intervention messages for 1 additional person to become vaccinated.

    Secondary Outcomes

    In logistic regression modeling, overall predictors of vaccination status included White or Asian race and being older or a college graduate (Figure 2 and Figure S9 in Multimedia Appendix 1). Being in the intervention group was associated with a significantly increased likelihood of getting the influenza vaccine (OR 1.099, 95% CI 1.012-1.192; P=.02). Participants who had cardiac arrest (OR 3.477, 95% CI 1.85-6.54; P<.001), atrial fibrillation (OR 1.332, 95% CI 1.068-1.66; P=.01), or coronary disease (OR 1.411, 95% CI 1.055-1.885; P=.02) were also more likely to report vaccination (>65%) than participants with other conditions. Digital interventions appeared to be more effective in encouraging vaccinations among White participants (intervention: 2837/4510, 62.9% vs control: 2763/4612, 59.91%; P=.004) than among non-White participants (intervention: 581/1065, 54.55% vs control: 593/1050, 56.48%; P=.42; Figure S10 in Multimedia Appendix 1).

    Kaplan-Meier analysis of the time to vaccination showed that at least 2 digital intervention messages, completed 2 weeks apart, were necessary for a difference to begin to emerge (Figure 3). In the intervention group (N=5575), the most completed messages were the knowledge quiz (n=4248, 76.2%), cost article (n=4276, 76.7%), and CDC article (n=4315, 77.4%; Figure S11 in Multimedia Appendix 1). In all, 44.81% (n=2498) of the intervention group completed all 6 messages, and 7.7% (n=429) completed none of them; the reported vaccination rate for the former group was about 13 percentage points higher than that for the latter group (1626/2498, 65.09% vs 223/429, 51.98%).

    ‎
    Figure 2. Predictors of self-reported influenza vaccination.
    View this figure
    ‎
    Figure 3. Self-reported vaccination rates over time. Dashed vertical gray lines indicate the timings of the 6 digital intervention messages. CDC: Centers for Disease Control and Prevention.
    View this figure

    After controlling for age, sex, race, and education over the entire study population, participants who saw a health care provider, were offered an influenza vaccine, or got their vaccine information from a health care provider were more likely to report getting the vaccine (Figure S12 in Multimedia Appendix 1). Those who were told by a health care provider that they were part of a high-risk group were also more likely to report vaccination (OR 2.369, 95% CI 2.171-2.586; P<.001).

    Participants who reported a diagnosis of COVID-19 were 40% more likely to report influenza vaccination than those who did not, regardless of intervention assignment (Figure S13 in Multimedia Appendix 1). Of the 7457 participants who reported getting the influenza vaccine, 4252 (57.02%) stated that the COVID-19 pandemic did not influence their decision to vaccinate, and 4026 (66.33%) of the 6070 participants who did not report influenza vaccination said their decision was not influenced by the COVID-19 pandemic.


    Principal Findings

    In this cohort of 11,237 adults with CVD, digital intervention was associated with a significantly higher rate of self-reported influenza vaccination at the end of the study period than control participants. Based on epidemiologic estimates, roughly 26 million Americans have CVD [26], and an increase in vaccination rates of 2.06% as shown in this trial would mean another 535,600 persons with CVD being immunized. This increase would likely translate to substantial reductions in morbidity, mortality, and costs to the health care system, as well as potential improvements in the quality of life if applied at scale.

    Our findings add to a growing body of evidence that interventions delivered via digital communication channels can be effective in improving vaccination rates among high-risk patients. Previous randomized studies have generally shown significantly improved influenza vaccination rates with email prompts, app-based messages, SMS text messaging, and web-based interventions in general adult populations [14,15,23,27-31], high-risk patients (some of whom had heart disease) [16,32], and pregnant women [33-35], but a few have not [36-38]. To our knowledge, this is the first randomized study to show promising results with a digital intervention specifically designed for and delivered to a population with CVD.

    The patient-centered digital interventions, developed with evidence-based behavioral theory of vaccine behavior [19], were generally viewed as informative, trustworthy, and engaging. In all, about 45% of the intervention group completed all intervention messages, compared to 27% in our previous study [16] and the industry standard of 22% [39], indicating very strong engagement. These messages also produced results at least as good as other recent app-based digital influenza vaccination interventions in general Canadian [13] and US adult populations [14,15].

    Older age, more education, and White or Asian race were significant predictors of vaccination in this study. The apparent lack of effect in other participants of color might reflect small sample sizes or heterogeneity among non-White participants. Further assessment of racial and ethnic differences in responses to nontailored digital interventions is needed. This study does reinforce the importance of engagement with the health care system, as participants who saw a health care provider, were told they were in a high-risk group, or were offered the influenza vaccine were more likely to report vaccination.

    Vaccination against influenza is a cost-effective method for reducing some of the risk associated with CVD [40,41]. Combined with the cost-effectiveness of digital intervention design and deployment relative to other prevention strategies, the messaging presented here appears to be suitable as a population-health management strategy in the context of limited budgets for health systems, insurers, and public agencies. In addition, half of the 380,000 people [1] hospitalized annually with influenza in the United States have heart disease [5]. Scaling this digital intervention to the larger population of people with CVD could help reduce hospitalizations and emergency department and clinic visits, along with days of productivity lost, particularly in already digitally connected populations.

    The strengths of this study include its decentralized, pragmatic nature, which can provide high-quality evidence of effectiveness in real-world settings. Other strengths imclude its large sample size, nationwide scope, and variety of data collected, including patient-generated health data. The study also reflects real-world data on vaccination rates among persons with variable risk levels from influenza infection conferred by different cardiovascular diagnoses. The design of the study may inspire the design of future vaccination campaigns to assess the drivers of vaccination and their public health impact and investigate vaccination behavior in other patient populations.

    Limitations and Future Work

    Participants reported their CVD diagnoses at different times via different survey sources (eg, historical surveys vs current self-reports). This method resulted in discrepancies from using different data sources in health outcomes, potentially due to question formatting and the time period for recall: almost a third of participants reported having none of the candidate CVD conditions despite previous self-reports of such disease. Future studies could forgo blinding in favor of supplementing self-reports with additional sources of information (eg, health claims and medical records). Participants were blinded to participation, reflecting real-world engagement with health messages outside of a known research-related setting. The potential influence of unknowingly participating in research is unclear. Only about 23% of the sample reported on the primary end point. The generalizability of findings to nonresponders is therefore limited. We also have no knowledge about why participants did not respond.

    This trial was conducted during the COVID-19 pandemic. Due to the pandemic, participants may have had increased awareness of viral diseases and vaccines generally through other sources (eg, governmental sources, television, and social media), possibly limiting the generalizability of our findings, although most participants in both groups stated that the pandemic did not affect their decision about influenza vaccination.

    Participants in the intervention group were compensated in the form of points, which could be redeemed for cash. However, given that the total possible monetary compensation was only US $1.52 regardless of vaccination status, it likely did not influence the motivation to vaccinate enough to impact the outcome.

    All participants were existing members of the Achievement platform, reflecting a population already engaged with digital technology. The baseline (control) vaccination rate (59.25%) was also about 8% higher than the 51% CDC average for individuals with comorbidities [7]. Thus, it might have been more difficult to see an incremental uplift compared to populations with less technology use or a lower baseline vaccination rate.

    Most of the population was female, non-Hispanic, and White. The effects of the intervention in other demographic groups are less certain, although the sample size was sufficient for models adjusting for age, education, sex, and race to confirm that the intervention effect remained significant. Barriers to health equity in accessing digital health interventions and methodologies remain significant [42]. This study should serve as a foundation for future evaluation and tailoring to reach individuals from diverse backgrounds more effectively, as Brewer and colleagues [43] have shown that people from diverse racial and ethnic backgrounds engage with digital health information via the web and digital health research at a high rate.

    Although several evidence-based sources and techniques were leveraged in the development of the intervention messages, their exact mechanisms of action are unknown. The act of prompting, rather than the content, might result in similar improvement. Future studies examining which components or messages would be the most beneficial could help optimize future interventions while minimizing burden.

    Conclusions

    A digital intervention using health condition–relevant information and widely available public health information can be an effective way to increase influenza vaccination rates in persons with CVD. These results may have broader public health implications as an easily scalable intervention to increase vaccination behavior. Future studies should examine the effectiveness and cost-effectiveness of such digital campaigns in diverse populations with other chronic conditions and for other types of vaccination, such as COVID-19 vaccines.

    Acknowledgments

    This study was funded by Sanofi. This sponsor was involved in the intervention development, study design, data interpretation, and writing of the report and was kept informed during data collection and data analysis. Sanofi produces one of the influenza vaccines available in the United States, but participants were not instructed to get a specific type of influenza vaccine. The senior (SS) and corresponding (NJM) authors had full access to all of the data in the study and had final responsibility for the decision to submit for publication. The authors thank Patricia A. French, BS, of Left Lane Communications for assistance in the drafting and editing of the manuscript.

    Data Availability

    Qualified researchers may request access to the aggregate results and related study documents including the study report, study protocol with any amendments, blank case report form, statistical analysis plan, and data set specifications. Further details on Sanofi’s data sharing criteria, eligible studies, and process for requesting access can be found at https://www.vivli.org/.

    Authors' Contributions

    NJM, JLL, J Schroeder, WNL, J See, M Madjid, MRM, LT, OV, MG, JL, M Mercer, and SS conceived the study and contributed to its design. NJM, JLL, J Schroeder, WNL, J See, JDP, and LT were responsible for the acquisition, analysis, or interpretation of data. NJM, JLL, J Schroeder, WNL, J See, and JD contributed to the drafting of the manuscript. J See conducted the statistical analyses. All authors critically revised the manuscript for important intellectual content and approved the final manuscript.

    Conflicts of Interest

    NM, J Schroeder, WNL, and J See are current or former employees of Evidation Health, Inc, the developer of the Achievement health and research platform, and may hold stock options in Evidation Health, Inc. JLL is a former employee of Evidation Health, Inc and a current employee of Lyra Health, Inc; receives income from Lyra Health, Inc; and has been granted equity in Lyra Health, Inc. M Madjid has received consulting fees from Sanofi and Seqirus. MRM has received consultation fees, honoraria, or both from Sanofi, CareDx, Alnylam, and Akcea. OV has received consulting fees from Sanofi. MG, M Mercer, SS, and JL are employees of Sanofi, the study sponsor. JL and SS are also shareholders of Sanofi. No other authors declare a conflict of interest.

    ‎
    Multimedia Appendix 1

    Supplemental information.

    PDF File (Adobe PDF File), 723 KB
    ‎
    Multimedia Appendix 2

    CONSORT (Consolidated Standards of Reporting Trials) eHealth checklist.

    PDF File (Adobe PDF File), 88 KB
    1. Estimated flu-related illnesses, medical visits, hospitalizations, and deaths in the United States — 2019-2020 flu season. Centers for Disease Control and Prevention. 2021 Oct 01.   URL: https://www.cdc.gov/flu/about/burden/2019-2020.html [accessed 2022-04-06]
    2. Putri WCWS, Muscatello DJ, Stockwell MS, Newall AT. Economic burden of seasonal influenza in the United States. Vaccine 2018 Jun 22;36(27):3960-3966. [CrossRef] [Medline]
    3. Kwong JC, Schwartz KL, Campitelli MA, Chung H, Crowcroft NS, Karnauchow T, et al. Acute myocardial infarction after laboratory-confirmed influenza infection. N Engl J Med 2018 Jan 25;378(4):345-353. [CrossRef] [Medline]
    4. Chow EJ, Rolfes MA, O'Halloran A, Anderson EJ, Bennett NM, Billing L, et al. Acute cardiovascular events associated with influenza in hospitalized adults: a cross-sectional study. Ann Intern Med 2020 Oct 20;173(8):605-613 [FREE Full text] [CrossRef] [Medline]
    5. Flu and people with heart disease or history of stroke. Centers for Disease Control and Prevention. 2021 Aug 26.   URL: https://www.cdc.gov/flu/highrisk/heartdisease.htm [accessed 2022-04-06]
    6. US flu vaccine effectiveness data for 2019-2020. Centers for Disease Control and Prevention. 2020 Dec 22.   URL: https://www.cdc.gov/flu/vaccines-work/2019-2020.html [accessed 2022-04-06]
    7. Flu vaccination coverage, by age group and high risk status, adults, United States, 2019-20 season. Centers for Disease Control and Prevention.   URL: https:/​/www.​cdc.gov/​flu/​excel/​fluvaxview/​2020/​general_population/​OnlineReport_2019_​20_AdditionalTable_FluVxByAge_Adults.​xlsx [accessed 2022-04-06]
    8. Increase the proportion of people who get the flu vaccine every year—IID-09. Healthy People 2030.   URL: https:/​/health.​gov/​healthypeople/​objectives-and-data/​browse-objectives/​vaccination/​increase-proportion-people-who-get-flu-vaccine-every-year-iid-09 [accessed 2022-04-06]
    9. MacIntyre CR, Mahimbo A, Moa AM, Barnes M. Influenza vaccine as a coronary intervention for prevention of myocardial infarction. Heart 2016 Dec 15;102(24):1953-1956 [FREE Full text] [CrossRef] [Medline]
    10. Yedlapati SH, Khan SU, Talluri S, Lone AN, Khan MZ, Khan MS, et al. Effects of influenza vaccine on mortality and cardiovascular outcomes in patients with cardiovascular disease: a systematic review and meta-analysis. J Am Heart Assoc 2021 Mar 16;10(6):e019636 [FREE Full text] [CrossRef] [Medline]
    11. Behrouzi B, Udell JA. Moving the needle on atherosclerotic cardiovascular disease and heart failure with influenza vaccination. Curr Atheroscler Rep 2021 Oct 21;23(12):78 [FREE Full text] [CrossRef] [Medline]
    12. Schmid P, Rauber D, Betsch C, Lidolt G, Denker M. Barriers of influenza vaccination intention and behavior - a systematic review of influenza vaccine hesitancy, 2005-2016. PLoS One 2017;12(1):e0170550 [FREE Full text] [CrossRef] [Medline]
    13. Dale LP, White L, Mitchell M, Faulkner G. Smartphone app uses loyalty point incentives and push notifications to encourage influenza vaccine uptake. Vaccine 2019 Jul 26;37(32):4594-4600. [CrossRef] [Medline]
    14. Lee W, Stück D, Konty K, Rivers C, Brown CR, Zbikowski SM, et al. Large-scale influenza vaccination promotion on a mobile app platform: a randomized controlled trial. Vaccine 2020 Apr 16;38(18):3508-3514. [CrossRef] [Medline]
    15. Milkman KL, Gandhi L, Patel MS, Graci HN, Gromet DM, Ho H, et al. A 680,000-person megastudy of nudges to encourage vaccination in pharmacies. Proc Natl Acad Sci U S A 2022 Feb 08;119(6):e2115126119 [FREE Full text] [CrossRef] [Medline]
    16. Lee JL, Foschini L, Kumar S, Juusola J, Liska J, Mercer M, et al. Digital intervention increases influenza vaccination rates for people with diabetes in a decentralized randomized trial. NPJ Digit Med 2021 Sep 17;4(1):138 [FREE Full text] [CrossRef] [Medline]
    17. Evidation Health, Inc.   URL: https://evidation.com/ [accessed 2021-04-06]
    18. Schroeder J, Madjid M, Munagala M, Piette J, Tan LJ, Vardeny O, et al. Developing a patient-centered digital intervention for increasing influenza vaccination rates in people with cardiovascular disease using a remote mixed-methods approach [poster]. 2021 Dec Presented at: Eighth European Scientific Working group on Influenza (ESWI) Influenza Conference; December 6-9, 2021; Valencia, Spain.
    19. Ajzen I. The theory of planned behavior. Organ Behav Hum Decis Process 1991 Dec;50(2):179-211. [CrossRef]
    20. Medlock MC, Wixon D, Terrano M, Romero RL, Fulton B. Using the RITE method to improve products: a definition and a case study. 2002 Presented at: UPA 2002: 11th Annual Usability Professionals' Association Conference; July 8-12, 2002; Orlando, FL   URL: https://www.jpattonassociates.com/wp-content/uploads/2015/04/rite_method.pdf
    21. Frank L, Basch E, Selby JV, Patient-Centered Outcomes Research Institute. The PCORI perspective on patient-centered outcomes research. JAMA 2014 Oct 15;312(15):1513-1514. [CrossRef] [Medline]
    22. Zhong B. How to calculate sample size in randomized controlled trial? J Thorac Dis 2009 Dec;1(1):51-54 [FREE Full text] [Medline]
    23. Nehme EK, Delphia M, Cha EM, Thomas M, Lakey D. Promoting influenza vaccination among an ACA health plan subscriber population: a randomized trial. Am J Health Promot 2019 Jul;33(6):916-920. [CrossRef] [Medline]
    24. Hurley LP, Beaty B, Lockhart S, Gurfinkel D, Breslin K, Dickinson M, et al. RCT of centralized vaccine reminder/recall for adults. Am J Prev Med 2018 Aug;55(2):231-239. [CrossRef] [Medline]
    25. Lundberg S, Lee SI. A unified approach to interpreting model predictions. arXiv 2017 May 22:1-10. [CrossRef]
    26. Virani SS, Alonso A, Aparicio HJ, Benjamin EJ, Bittencourt MS, Callaway CW, American Heart Association Council on Epidemiology and Prevention Statistics Committee and Stroke Statistics Subcommittee. Heart Disease and Stroke Statistics-2021 update: a report from the American Heart Association. Circulation 2021 Feb 23;143(8):e254-e743 [FREE Full text] [CrossRef] [Medline]
    27. Baskin E. Increasing influenza vaccination rates via low cost messaging interventions. PLoS One 2018;13(2):e0192594 [FREE Full text] [CrossRef] [Medline]
    28. Wijesundara JG, Ito Fukunaga M, Ogarek J, Barton B, Fisher L, Preusse P, et al. Electronic health record portal messages and interactive voice response calls to improve rates of early season influenza vaccination: randomized controlled trial. J Med Internet Res 2020 Sep 25;22(9):e16373 [FREE Full text] [CrossRef] [Medline]
    29. Cutrona SL, Golden JG, Goff SL, Ogarek J, Barton B, Fisher L, et al. Improving rates of outpatient influenza vaccination through EHR portal messages and interactive automated calls: a randomized controlled trial. J Gen Intern Med 2018 May;33(5):659-667 [FREE Full text] [CrossRef] [Medline]
    30. Szilagyi PG, Albertin C, Casillas A, Valderrama R, Duru OK, Ong MK, et al. Effect of patient portal reminders sent by a health care system on influenza vaccination rates: a randomized clinical trial. JAMA Intern Med 2020 Jul 01;180(7):962-970 [FREE Full text] [CrossRef] [Medline]
    31. Cheung YA. Improving influenza vaccination rates in nurses through electronic text message reminders: a randomized controlled trial [thesis]. The University of Hong Kong. 2016.   URL: https://hub.hku.hk/handle/10722/241308 [accessed 2022-08-16]
    32. Regan AK, Bloomfield L, Peters I, Effler PV. Randomized controlled trial of text message reminders for increasing influenza vaccination. Ann Fam Med 2017 Nov;15(6):507-514 [FREE Full text] [CrossRef] [Medline]
    33. Bushar JA, Kendrick JS, Ding H, Black CL, Greby SM. Text4baby influenza messaging and influenza vaccination among pregnant women. Am J Prev Med 2017 Dec;53(6):845-853 [FREE Full text] [CrossRef] [Medline]
    34. O'Leary ST, Narwaney KJ, Wagner NM, Kraus CR, Omer SB, Glanz JM. Efficacy of a web-based intervention to increase uptake of maternal vaccines: an RCT. Am J Prev Med 2019 Oct;57(4):e125-e133. [CrossRef] [Medline]
    35. Stockwell MS, Westhoff C, Kharbanda EO, Vargas CY, Camargo S, Vawdrey DK, et al. Influenza vaccine text message reminders for urban, low-income pregnant women: a randomized controlled trial. Am J Public Health 2014 Feb;104 Suppl 1:e7-12. [CrossRef] [Medline]
    36. Herrett E, Williamson E, van Staa T, Ranopa M, Free C, Chadborn T, et al. Text messaging reminders for influenza vaccine in primary care: a cluster randomised controlled trial (TXT4FLUJAB). BMJ Open 2016 Feb 19;6(2):e010069 [FREE Full text] [CrossRef] [Medline]
    37. Yudin MH, Mistry N, De Souza LR, Besel K, Patel V, Blanco Mejia S, et al. Text messages for influenza vaccination among pregnant women: a randomized controlled trial. Vaccine 2017 Feb 01;35(5):842-848. [CrossRef] [Medline]
    38. Moniz MH, Hasley S, Meyn LA, Beigi RH. Improving influenza vaccination rates in pregnancy through text messaging: a randomized controlled trial. Obstet Gynecol 2013 Apr;121(4):734-740. [CrossRef] [Medline]
    39. Email marketing benchmarks and statistics by industry. Mailchimp. 2019 Oct.   URL: https://mailchimp.com/resources/email-marketing-benchmarks/ [accessed 2022-04-06]
    40. Udell JA, Zawi R, Bhatt DL, Keshtkar-Jahromi M, Gaughran F, Phrommintikul A, et al. Association between influenza vaccination and cardiovascular outcomes in high-risk patients: a meta-analysis. JAMA 2013 Oct 23;310(16):1711-1720. [CrossRef] [Medline]
    41. Clar C, Oseni Z, Flowers N, Keshtkar-Jahromi M, Rees K. Influenza vaccines for preventing cardiovascular disease. Cochrane Database Syst Rev 2015 May 05(5):CD005050 [FREE Full text] [CrossRef] [Medline]
    42. Wood BR, Young JD, Abdel-Massih RC, McCurdy L, Vento TJ, Dhanireddy S, et al. Advancing digital health equity: a policy paper of the Infectious Diseases Society of America and the HIV Medicine Association. Clin Infect Dis 2021 Mar 15;72(6):913-919 [FREE Full text] [CrossRef] [Medline]
    43. Brewer LC, Fortuna KL, Jones C, Walker R, Hayes SN, Patten CA, et al. Back to the future: achieving health equity through health informatics and digital health. JMIR mHealth uHealth 2020 Jan 14;8(1):e14512 [FREE Full text] [CrossRef] [Medline]

    ‎
    CDC: Centers for Disease Control and Prevention
    CVD: cardiovascular disease
    OR: odds ratio

    Edited by A Mavragani; submitted 13.04.22; peer-reviewed by R Rahimi, H Veldandi; comments to author 01.06.22; revised version received 21.06.22; accepted 31.07.22; published 07.10.22

    Copyright

    ©Nell J Marshall, Jennifer L Lee, Jessica Schroeder, Wei-Nchih Lee, Jermyn See, Mohammad Madjid, Mrudula R Munagala, John D Piette, Litjen Tan, Orly Vardeny, Michael Greenberg, Jan Liska, Monica Mercer, Sandrine Samson. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 07.10.2022.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.