Background: Accurate and timely COVID-19 vaccination coverage data are vital for informing targeted, effective messaging and outreach and identifying barriers to equitable health service access. However, gathering vaccination rate data is challenging, and efforts often result in information that is either limited in scope (eg, limited to administrative data) or delayed (impeding the ability to rapidly respond). The evaluation of innovative technologies and approaches that can assist in addressing these limitations globally are needed.
Objective: The objective of this survey study was to assess the validity of Random Domain Intercept Technology (RDIT; RIWI Corp) for tracking self-reported vaccination rates in real time at the US national and state levels. RDIT—a form of online intercept sampling—has the potential to address the limitations of current vaccination tracking systems by allowing for the measurement of additional data (eg, attitudinal data) and real-time, rapid data collection anywhere there is web access.
Methods: We used RDIT from June 30 to July 26, 2021, to reach a broad sample of US adult (aged ≥18 years) web users and asked questions related to COVID-19 vaccination. Self-reported vaccination status was used as the focus of this validation exercise. National- and state-level RDIT-based vaccination rates were compared to Centers for Disease Control and Prevention (CDC)–reported national and state vaccination rates. Johns Hopkins University’s and Emory University’s institutional review boards designated this project as public health practice to inform message development (not human subjects research).
Results: By using RDIT, 63,853 adult web users reported their vaccination status (6.2% of the entire 1,026,850 American web-using population that was exposed to the survey). At the national level, the RDIT-based estimate of adult COVID-19 vaccine coverage was slightly higher (44,524/63,853, 69.7%; 95% CI 69.4%-70.1%) than the CDC-reported estimate (67.9%) on July 15, 2021 (ie, midway through data collection; t63,852=10.06; P<.001). The RDIT-based and CDC-reported state-level estimates were strongly and positively correlated (r=0.90; P<.001). RDIT-based estimates were within 5 percentage points of the CDC’s estimates for 29 states.
Conclusions: This broad-reaching, real-time data stream may provide unique advantages for tracking the use of a range of vaccines and for the timely evaluation of vaccination interventions. Moreover, RDIT could be harnessed to rapidly assess demographic, attitudinal, and behavioral constructs that are not available in administrative data, which could allow for deeper insights into the real-time predictors of vaccine uptake–enabling targeted and timely interventions.
Accurate and timely COVID-19 vaccination coverage data are vital for informing targeted, effective messaging and outreach and identifying barriers to equitable health service access. The tracking of vaccination rates is needed in conjunction with effective COVID-19 case surveillance and death monitoring, as well as research on predictors of vaccination uptake and vaccine hesitancy, to optimize vaccine coverage in persons at disproportionate risk of COVID-19 and severe outcomes [- ]. However, gathering such vaccination rate data is challenging, even in high-income countries such as the United States. Bill Gates recently expressed the importance of investment in global pandemic preparedness [ ], which should involve assessing and implementing surveillance systems that can be utilized globally.
In the United States, the Centers for Disease Control and Prevention (CDC) rely on multiple external sources for their vaccination tracker . Although considered the gold standard, this approach is labor intensive and limited to the collection of administrative data. In other words, certain demographic, attitudinal, and behavioral constructs are not available. Another approach involves national polling by using traditional recruitment methods (eg, panels and random digit dialing), in which respondents self-report their vaccination status (eg, Kaiser Family Foundation surveys and the Census Household Pulse) [ , ]. These polls have the benefit of collecting extended information, which could allow for deeper insights into the factors that predict vaccine uptake. Although immensely valuable for assessing knowledge, attitudes, and practices, these polls often do not allow for large-scale, rapid data collection, nor do they provide real-time granular results (with data available daily on a variety of population subcategories). Further, neither of these approaches are easily expandable to other countries with differing monitoring systems and access.
To identify targetable risk factors, it is critical to combine the benefits of traditional survey research with the advantages of rapid and continuous data collection by using a method that is scalable and can be incorporated into a global surveillance system. As such, we assessed the validity of Random Domain Intercept Technology (RDIT; RIWI Corp)—a form of online intercept sampling —for tracking self-reported vaccination rates in real-time, with broad reach, and at the national and state levels across the United States.
We used RDIT from June 30 to July 26, 2021, to reach a broad sample of US web users. RDIT can be administered anywhere in the world where there is web access. When web users click on a registered but commercially inactive web link or type in a web address for a site that is dormant, they have a random chance of that link being temporarily managed by the company that owns and administers RDIT—RIWI Corp. In this situation, instead of coming across a “this page does not exist” notification, a survey is delivered. Web users then decide whether to anonymously participate and may exit the survey at any time. No incentives are provided for participation.
Upon encountering the landing page, web users chose their preferred language (English or Spanish), and were told that the survey was anonymous and about COVID-19. Web users who chose to participate first reported their age and gender (those aged under 18 years were immediately exited from the landing page). They then reported their vaccination status (the focus of the validation exercise). Following this, those who were vaccinated were administered questions about the vaccine they received, and those who were unvaccinated were given a series of questions about COVID-19 vaccines (intentions, incentives, barriers, etc). All respondents were asked if they had heard recent accounts of COVID-19 vaccination side effects and were asked to provide information on where they receive their news and entertainment. Except for age, gender, and race and ethnicity, the remaining demographic questions were asked at the conclusion of the survey (education, urban or rural living location, political affiliation, and annual household income).
This project was approved as public health practice to inform message development (not human subjects research) by the institutional review boards of Johns Hopkins University and Emory University.
With regard to COVID-19 vaccination status, respondents were asked “Have you personally received the COVID-19 vaccine?” They were given the following four response options: “Yes, a single-dose vaccine (J&J)”; “Yes, the first of two doses (Moderna or Pfizer)”; “Yes, both doses of a two-dose vaccine (Moderna or Pfizer)”; and “I have not received a vaccine.” RDIT captures nonpersonally identifiable state location information from a respondent’s IP address, which is instantaneously translated into a unique identifier. For this validation analysis, we compared self-reported adult vaccination rates from RDIT (having received at least 1 dose) to those that were provided in the publicly available CDC COVID-19 Vaccinations in the United States, Jurisdiction data set on July 15, 2021 (midway through RDIT data collection) .
RDIT-based vaccination rate estimates (and associated logit transformed 95% CIs) were computed for the full national sample (N=63,853) and by state among those with available state information (n=57,986). A series of correlational analyses and 1-sample t tests (2-tailed) were used to compare RDIT-based estimates to CDC-reported rates.
By using RDIT, 1,026,850 US web users were exposed to the survey. Of those, 63,853 (6.2%) were aged ≥18 years and reported their vaccination status. Respondents were distributed throughout the sampling window from June 30 to July 26, 2021, with an average of 2365 (SD 1078; range 566-4266; skew=0.04) respondents per day. With regard to gender, 42.4% (27,060/63,853) of respondents were women, 51.6% (32,939/63,853) were men, and 6% (3854/63,853) indicated “other” gender. The median age was 39 (range 18 to ≥85) years. Respondents with available state information were regionally representative (ie, the RDIT-based state sample size strongly and positively correlated with the census-based state population size ; r=0.98, 95% CI 0.96-0.99; P<.001).
Of the 46,955 respondents who reported their race and ethnicity (46,955/63,853, 73.5% of those who reported their vaccination status), 23,505 (50.1%) identified as White; 5702 (12.1%) identified as African American or Black; 5094 (10.8%) identified as Hispanic or Latinx; 3046 (6.5%) identified as Asian; 1367 (2.9%) identified as Native American, American Indian, or Alaskan Native; 4954 (10.6%) identified as multiracial; and 3287 (7%) identified as a nonspecified racial and ethnic group. Of the 14,801 respondents who reported their annual household income (14,801/63,853, 23.2% of those who reported their vaccination status), 6700 (45.3%) reported an income of US $50,000 or less, 4914 (33.2%) reported an income of US $50,001 to US $125,000, and 3187 (21.5%) reported an income of US $125,001 or more. The race and ethnicity and economic distributions observed among the respondents are comparable to the most recent (2020) estimates derived from the US Census Current Population Survey [, ].
The national CDC-estimated vaccine coverage on July 15, 2021, was 67.9%. The RDIT-based estimate among the full sample (N=63,853) from June 30 to July 26, 2021, was slightly higher (44,524/63,853, 69.7%; 95% CI 69.4%-70.1%; t63,852=10.06; P<.001).
The RDIT-based and CDC-reported state-level estimates were strongly and positively correlated (r=0.90, 95% CI 0.83-0.94; P<.001). RDIT-based estimates were higher than the CDC-reported estimates by a mean of 3% (SD 4.5%, 95% CI 1.7%-4.2%; t50=4.71; P<.001). RDIT-based estimates were higher than the CDC’s estimates for 37 states and were within 2 and 5 percentage points for 12 and 29 states, respectively. When considering the absolute value of the estimate discrepancies, states with more RDIT respondents were associated with smaller discrepancies (r=−0.43, 95% CI −0.64 to −0.18; P=.001). We observed the largest discrepancy for Alaska (percent difference: 14.5%), which was likely due in part to the small sample size (n=85). The states with RDIT estimates within 1 percentage point of the relative CDC estimate were California; Connecticut; Washington, District of Columbia; Florida; Maryland; Maine; New York; Texas; and Utah ().
rate, % (95% CI)
vaccination rate, %
aDifference = RDIT-based vaccination rate − CDC-reported vaccination rate.
RDIT provided similar estimates to those provided via the CDC method (the current standard) for vaccination rates at the national and state levels; however, estimates from RDIT are accessible daily at variable magnitudes and have region-targeting capabilities. Although the July 2021 RDIT-based national estimate was higher than the CDC estimate by 2%, it was more comparable to the CDC estimate than those derived from the Census Household Pulse and Delphi-Facebook surveys, which overestimated vaccination coverage by 17% and 14% in May 2021, respectively . At the state level, RDIT-estimated rates strongly correlated with CDC reports, were slightly higher on average, and were within a 5% margin for 57% (29/51) of the states. These findings provide early evidence of the validity of RDIT as a complementary surveillance mechanism for tracking COVID-19 vaccination coverage across the United States.
There are limitations to RDIT that should be considered. First, RDIT only reaches the web-using population; nonetheless, RDIT reaches a diverse set of the web-using population, including respondents who are not habitual survey takers, thereby allowing for subgroup analyses as needed. Second, a repeated measures assessment (ie, a follow-up to assess changes in vaccine status per individual) is not possible because RDIT does not collect identifying information. However, population-level changes can be identified. Similarly, without identifying information, opt-in bias is unknown, but one can record and evaluate trends in retention throughout the survey. Additionally, while associated with analytic limitations, the anonymous nature of RDIT is a strength, as participant privacy is prioritized. Although RDIT enables the collection of additional factors that may provide targets for improving vaccine coverage, it is possible for participants to drop out before they provide this information. An analysis of such drop-off can provide further understanding, and researchers can decide whether it is most appropriate to draw insights from all available data or only from respondents who complete the entire question set (in this case, we chose the former). Finally, administrative data, such as the data that inform the CDC vaccination tracker, could be less susceptible to self-reporting bias; however, we found strong correlations between the self-reported vaccination rate estimates and the CDC administrative metrics.
Access to this broad-reaching data stream is potentially less labor intensive than the alternative approaches that are currently used by the CDC, and RDIT-based estimates demonstrate adequate accuracy when compared to CDC estimates. RDIT’s real-time nature may be a valuable tool for tracking vaccine uptake; pinpointing localities for targeted, timely interventions; and enabling the rapid evaluation of interventions and messaging campaigns globally. Of course, further investigation is needed to assess the accuracy of RDIT for tracking vaccination status globally. Nonetheless, RDIT could be harnessed to rapidly assess demographic, attitudinal, and behavioral constructs that are not available in administrative data, which could allow for deeper insights into the real-time predictors of COVID-19 vaccine uptake. Such data could be translated into effective interventions to strengthen vaccination coverage.
This project was made possible by a grant from the US Centers for Disease Control and Prevention, in collaboration with the Task Force for Global Health, under award number 6 NU51IP000873-05-02.
Conflicts of Interest
WAO is on the scientific advisory board for Moderna. JVL is a member of the Bioethics Advisory Council for Pfizer Inc. DAS is a member of the Janssen Policy Board and receives consulting and grant funding from Merck. SL is an employee of, and RHS is a consultant to, RIWI Corp—the company that owns the technology that was used to conduct the surveys. No other authors have conflicts of interest to report.
- Hennessee I, Clennon JA, Waller LA, Kitron U, Bryan JM. Considerations for improving reporting and analysis of date-based COVID-19 surveillance data by public health agencies. Am J Public Health 2021 Dec;111(12):2127-2132. [CrossRef] [Medline]
- Ahmad FB, Anderson RN, Knight K, Rossen LM, Sutton PD. Advancements in the National Vital Statistics System to meet the real-time data needs of a pandemic. Am J Public Health 2021 Dec;111(12):2133-2140. [CrossRef] [Medline]
- Khairat S, Zou B, Adler-Milstein J. Factors and reasons associated with low COVID-19 vaccine uptake among highly hesitant communities in the US. Am J Infect Control 2022 Mar;50(3):262-267 [FREE Full text] [CrossRef] [Medline]
- Ndugga N, Hill L, Artiga S, Haldar S. Latest data on COVID-19 vaccinations by race/ethnicity. Kaiser Family Foundation. 2022 Apr 07. URL: https://www.kff.org/coronavirus-covid-19/issue-brief/latest-data-on-covid-19-vaccinations-by-race-ethnicity/ [accessed 2022-05-31]
- Sargent RH, Laurie S, Moncada L, Weakland LF, Lavery JV, Salmon DA, et al. Masks, money, and mandates: A national survey on efforts to increase COVID-19 vaccination intentions in the United States. PLoS One 2022 Apr 21;17(4):e0267154. [CrossRef] [Medline]
- Neville S, Kuchler H. Bill Gates calls for global surveillance team to spot pandemic threats. Financial Times. 2022 May 01. URL: https://www.ft.com/content/c8896c10-35da-46aa-957f-cf2b4e18cfce [accessed 2022-05-31]
- COVID-19 vaccination data in the United States. Centers for Disease Control and Prevention. URL: https://www.cdc.gov/coronavirus/2019-ncov/vaccines/distributing/about-vaccine-data.html [accessed 2021-10-27]
- KFF COVID-19 vaccine monitor archives. Kaiser Family Foundation. URL: https://www.kff.org/coronavirus-covid-19/kff-covid-19-vaccine-monitor-archives/ [accessed 2021-11-08]
- Measuring household experiences during the coronavirus pandemic. United States Census Bureau. URL: https://www.census.gov/data/experimental-data-products/household-pulse-survey.html [accessed 2021-12-20]
- Schonlau M, Couper MP. Options for conducting web surveys. Stat Sci 2017 May;32(2):279-292 [FREE Full text] [CrossRef]
- COVID-19 vaccinations in the United States, jurisdiction. Centers for Disease Control and Prevention. URL: https://data.cdc.gov/Vaccinations/COVID-19-Vaccinations-in-the-United-States-Jurisdi/unsk-b7fc [accessed 2022-01-04]
- State population totals and components of change: 2020-2021. United States Census Bureau. URL: https://www.census.gov/data/tables/time-series/demo/popest/2020s-state-total.html [accessed 2021-12-22]
- Population distribution by race/ethnicity (CPS). Kaiser Family Foundation. URL: https://tinyurl.com/3h874zkp [accessed 2022-05-31]
- HINC-01. Selected characteristics of households by total money income. United States Census Bureau. URL: https://www.census.gov/data/tables/time-series/demo/income-poverty/cps-hinc/hinc-01.html [accessed 2022-05-31]
- Bradley VC, Kuriwaki S, Isakov M, Sejdinovic D, Meng XL, Flaxman S. Unrepresentative big surveys significantly overestimated US vaccine uptake. Nature 2021 Dec;600(7890):695-700 [FREE Full text] [CrossRef] [Medline]
|CDC: Centers for Disease Control and Prevention|
|RDIT: Random Domain Intercept Technology|
Edited by A Mavragani; submitted 11.03.22; peer-reviewed by R Schechter, N Seeman; comments to author 08.05.22; revised version received 06.06.22; accepted 07.06.22; published 01.07.22Copyright
©Rikki H Sargent, Shaelyn Laurie, Leo F Weakland, James V Lavery, Daniel A Salmon, Walter A Orenstein, Robert F Breiman. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 01.07.2022.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.