Use of Random Domain Intercept Technology to Track COVID-19 Vaccination Rates in Real Time Across the United States: Survey Study

Introduction

Accurate and timely COVID-19 vaccination coverage data are vital for informing targeted, effective messaging and outreach and identifying barriers to equitable health service access. The tracking of vaccination rates is needed in conjunction with effective COVID-19 case surveillance and death monitoring, as well as research on predictors of vaccination uptake and vaccine hesitancy, to optimize vaccine coverage in persons at disproportionate risk of COVID-19 and severe outcomes [1-5]. However, gathering such vaccination rate data is challenging, even in high-income countries such as the United States. Bill Gates recently expressed the importance of investment in global pandemic preparedness [6], which should involve assessing and implementing surveillance systems that can be utilized globally.

In the United States, the Centers for Disease Control and Prevention (CDC) rely on multiple external sources for their vaccination tracker [7]. Although considered the gold standard, this approach is labor intensive and limited to the collection of administrative data. In other words, certain demographic, attitudinal, and behavioral constructs are not available. Another approach involves national polling by using traditional recruitment methods (eg, panels and random digit dialing), in which respondents self-report their vaccination status (eg, Kaiser Family Foundation surveys and the Census Household Pulse) [8,9]. These polls have the benefit of collecting extended information, which could allow for deeper insights into the factors that predict vaccine uptake. Although immensely valuable for assessing knowledge, attitudes, and practices, these polls often do not allow for large-scale, rapid data collection, nor do they provide real-time granular results (with data available daily on a variety of population subcategories). Further, neither of these approaches are easily expandable to other countries with differing monitoring systems and access.

To identify targetable risk factors, it is critical to combine the benefits of traditional survey research with the advantages of rapid and continuous data collection by using a method that is scalable and can be incorporated into a global surveillance system. As such, we assessed the validity of Random Domain Intercept Technology (RDIT; RIWI Corp)—a form of online intercept sampling [10]—for tracking self-reported vaccination rates in real-time, with broad reach, and at the national and state levels across the United States.

Methods

Procedure

We used RDIT from June 30 to July 26, 2021, to reach a broad sample of US web users. RDIT can be administered anywhere in the world where there is web access. When web users click on a registered but commercially inactive web link or type in a web address for a site that is dormant, they have a random chance of that link being temporarily managed by the company that owns and administers RDIT—RIWI Corp. In this situation, instead of coming across a “this page does not exist” notification, a survey is delivered. Web users then decide whether to anonymously participate and may exit the survey at any time. No incentives are provided for participation.

Upon encountering the landing page, web users chose their preferred language (English or Spanish), and were told that the survey was anonymous and about COVID-19. Web users who chose to participate first reported their age and gender (those aged under 18 years were immediately exited from the landing page). They then reported their vaccination status (the focus of the validation exercise). Following this, those who were vaccinated were administered questions about the vaccine they received, and those who were unvaccinated were given a series of questions about COVID-19 vaccines (intentions, incentives, barriers, etc). All respondents were asked if they had heard recent accounts of COVID-19 vaccination side effects and were asked to provide information on where they receive their news and entertainment. Except for age, gender, and race and ethnicity, the remaining demographic questions were asked at the conclusion of the survey (education, urban or rural living location, political affiliation, and annual household income).

Ethical Considerations

This project was approved as public health practice to inform message development (not human subjects research) by the institutional review boards of Johns Hopkins University and Emory University.

Measures

With regard to COVID-19 vaccination status, respondents were asked “Have you personally received the COVID-19 vaccine?” They were given the following four response options: “Yes, a single-dose vaccine (J&J)”; “Yes, the first of two doses (Moderna or Pfizer)”; “Yes, both doses of a two-dose vaccine (Moderna or Pfizer)”; and “I have not received a vaccine.” RDIT captures nonpersonally identifiable state location information from a respondent’s IP address, which is instantaneously translated into a unique identifier. For this validation analysis, we compared self-reported adult vaccination rates from RDIT (having received at least 1 dose) to those that were provided in the publicly available CDC COVID-19 Vaccinations in the United States, Jurisdiction data set on July 15, 2021 (midway through RDIT data collection) [11].

Statistical Analysis

RDIT-based vaccination rate estimates (and associated logit transformed 95% CIs) were computed for the full national sample (N=63,853) and by state among those with available state information (n=57,986). A series of correlational analyses and 1-sample t tests (2-tailed) were used to compare RDIT-based estimates to CDC-reported rates.

Results

Respondent Description

By using RDIT, 1,026,850 US web users were exposed to the survey. Of those, 63,853 (6.2%) were aged ≥18 years and reported their vaccination status. Respondents were distributed throughout the sampling window from June 30 to July 26, 2021, with an average of 2365 (SD 1078; range 566-4266; skew=0.04) respondents per day. With regard to gender, 42.4% (27,060/63,853) of respondents were women, 51.6% (32,939/63,853) were men, and 6% (3854/63,853) indicated “other” gender. The median age was 39 (range 18 to ≥85) years. Respondents with available state information were regionally representative (ie, the RDIT-based state sample size strongly and positively correlated with the census-based state population size [12]; r=0.98, 95% CI 0.96-0.99; P<.001).

Of the 46,955 respondents who reported their race and ethnicity (46,955/63,853, 73.5% of those who reported their vaccination status), 23,505 (50.1%) identified as White; 5702 (12.1%) identified as African American or Black; 5094 (10.8%) identified as Hispanic or Latinx; 3046 (6.5%) identified as Asian; 1367 (2.9%) identified as Native American, American Indian, or Alaskan Native; 4954 (10.6%) identified as multiracial; and 3287 (7%) identified as a nonspecified racial and ethnic group. Of the 14,801 respondents who reported their annual household income (14,801/63,853, 23.2% of those who reported their vaccination status), 6700 (45.3%) reported an income of US $50,000 or less, 4914 (33.2%) reported an income of US $50,001 to US $125,000, and 3187 (21.5%) reported an income of US $125,001 or more. The race and ethnicity and economic distributions observed among the respondents are comparable to the most recent (2020) estimates derived from the US Census Current Population Survey [13,14].

National Comparison

The national CDC-estimated vaccine coverage on July 15, 2021, was 67.9%. The RDIT-based estimate among the full sample (N=63,853) from June 30 to July 26, 2021, was slightly higher (44,524/63,853, 69.7%; 95% CI 69.4%-70.1%; t_63,852=10.06; P<.001).

State Comparison

The RDIT-based and CDC-reported state-level estimates were strongly and positively correlated (r=0.90, 95% CI 0.83-0.94; P<.001). RDIT-based estimates were higher than the CDC-reported estimates by a mean of 3% (SD 4.5%, 95% CI 1.7%-4.2%; t₅₀=4.71; P<.001). RDIT-based estimates were higher than the CDC’s estimates for 37 states and were within 2 and 5 percentage points for 12 and 29 states, respectively. When considering the absolute value of the estimate discrepancies, states with more RDIT respondents were associated with smaller discrepancies (r=−0.43, 95% CI −0.64 to −0.18; P=.001). We observed the largest discrepancy for Alaska (percent difference: 14.5%), which was likely due in part to the small sample size (n=85). The states with RDIT estimates within 1 percentage point of the relative CDC estimate were California; Connecticut; Washington, District of Columbia; Florida; Maryland; Maine; New York; Texas; and Utah (Table 1).

Discussion

Principal Findings

RDIT provided similar estimates to those provided via the CDC method (the current standard) for vaccination rates at the national and state levels; however, estimates from RDIT are accessible daily at variable magnitudes and have region-targeting capabilities. Although the July 2021 RDIT-based national estimate was higher than the CDC estimate by 2%, it was more comparable to the CDC estimate than those derived from the Census Household Pulse and Delphi-Facebook surveys, which overestimated vaccination coverage by 17% and 14% in May 2021, respectively [15]. At the state level, RDIT-estimated rates strongly correlated with CDC reports, were slightly higher on average, and were within a 5% margin for 57% (29/51) of the states. These findings provide early evidence of the validity of RDIT as a complementary surveillance mechanism for tracking COVID-19 vaccination coverage across the United States.

Limitations

There are limitations to RDIT that should be considered. First, RDIT only reaches the web-using population; nonetheless, RDIT reaches a diverse set of the web-using population, including respondents who are not habitual survey takers, thereby allowing for subgroup analyses as needed. Second, a repeated measures assessment (ie, a follow-up to assess changes in vaccine status per individual) is not possible because RDIT does not collect identifying information. However, population-level changes can be identified. Similarly, without identifying information, opt-in bias is unknown, but one can record and evaluate trends in retention throughout the survey. Additionally, while associated with analytic limitations, the anonymous nature of RDIT is a strength, as participant privacy is prioritized. Although RDIT enables the collection of additional factors that may provide targets for improving vaccine coverage, it is possible for participants to drop out before they provide this information. An analysis of such drop-off can provide further understanding, and researchers can decide whether it is most appropriate to draw insights from all available data or only from respondents who complete the entire question set (in this case, we chose the former). Finally, administrative data, such as the data that inform the CDC vaccination tracker, could be less susceptible to self-reporting bias; however, we found strong correlations between the self-reported vaccination rate estimates and the CDC administrative metrics.

Conclusion

Access to this broad-reaching data stream is potentially less labor intensive than the alternative approaches that are currently used by the CDC, and RDIT-based estimates demonstrate adequate accuracy when compared to CDC estimates. RDIT’s real-time nature may be a valuable tool for tracking vaccine uptake; pinpointing localities for targeted, timely interventions; and enabling the rapid evaluation of interventions and messaging campaigns globally. Of course, further investigation is needed to assess the accuracy of RDIT for tracking vaccination status globally. Nonetheless, RDIT could be harnessed to rapidly assess demographic, attitudinal, and behavioral constructs that are not available in administrative data, which could allow for deeper insights into the real-time predictors of COVID-19 vaccine uptake. Such data could be translated into effective interventions to strengthen vaccination coverage.