Efficacy, Usability, and Acceptability of a Chatbot for Promoting COVID-19 Vaccination in Unvaccinated or Booster-Hesitant Young Adults: Pre-Post Pilot Study

Introduction

COVID-19 vaccines are highly effective in preventing severe disease and death but are underused. By mid-2022, the full vaccination rate has remained suboptimal in many places where COVID-19 vaccines are readily available (eg, 67% in the United States and 75% in the United Kingdom) [1]. COVID-19 booster shots are also being delivered to address waning immunity and viral variants, but studies have shown that some fully vaccinated people were unwilling to take the booster [2-4]. COVID-19 may also become an endemic disease such as seasonal influenza, and regular vaccination may be needed to protect high-risk populations. Interventions to promote the vaccine is crucial to reduce the burden of COVID-19.

Vaccine hesitancy is considered 1 of the 10 major threats of global health according to the World Health Organization (WHO) [5]. Studies have consistently shown higher COVID-19 vaccine hesitancy in women, younger people, ethnic minority populations, and people with lower socioeconomic status [6,7]. Partly due to the fast-tracked development and authorization of the vaccines, the lack of confidence in the vaccine efficacy and safety were among the main drivers for hesitancy [7]. Widespread misinformation against the vaccine further amplified its safety concerns [8,9]. Debunking such misinformation could reduce COVID-19 vaccine hesitancy and promote uptake, especially in subpopulations that are more susceptible to misinformation such as young people [10].

Chatbots or conversational agents are increasingly developed as a scalable and accessible platform for supporting health care delivery. The interface of a chatbot that is familiar to most people with experiences in mobile messaging could promote the usability and user engagement of the chatbot compared to other platforms. Several chatbots have been developed amid the COVID-19 pandemic [11,12], mostly for symptom checking and information dissemination [13,14]. The WHO has also launched chatbots on popular social networking sites such as WhatsApp to provide instant and credible information about COVID-19, including vaccination [15]. Nevertheless, empirical evidence on the utility of chatbots for promoting vaccination has remained scarce.

Mass COVID-19 immunization has begun in February 2021 in Hong Kong. Despite ample supply of both an inactivated vaccine (CoronaVac; Sinovac) and mRNA vaccine (Comirnaty; Fosun-BioNTech), the uptake had been slowed, with only 62% of the population being fully vaccinated by the beginning of 2022 [1]. We conducted a population-based survey on 1501 general adults in Hong Kong (COVID-19 Health Information Survey) and found higher COVID-19 vaccine hesitancy among young adults (aged 18-29 years) than older adults (aged ≥30 years) [16,17]. We also found the low perceived COVID-19 severity and safety concerns of the vaccines to be the main drivers of vaccine hesitancy [16]. Additionally, the COVID-19 Health Information Survey showed that eHealth literacy was associated with adherence to mask wearing, hand washing and social distancing [17], but its role in vaccine hesitancy has remained under-studied. Therefore, the primary aim of the study was to examine the preliminary efficacy, usability, and acceptability of using a chatbot for promoting COVID-19 vaccination. We also examined the feasibility of assessing the long-term effect on COVID-19 vaccination status in a post-hoc 4-month follow-up. The secondary aim was to examine the factors associated with COVID-19 vaccine hesitancy, including eHealth literacy.

Methods

Study Design and Recruitment

We conducted a pilot study using a pretest-posttest design to evaluate “Vac Chat, Fact Check,” a chatbot designed to provide updated information and debunk misinformation about the COVID-19 vaccine. The study was conducted in November 2021, between the end of the fourth wave (June 2021) and the start of the fifth wave (January 2022) of the outbreak in Hong Kong, which had about 12,000 cumulative cases and 200 deaths.

The study targeted adults aged ≥18 years who can read and communicate in Chinese. A mass email with a link to a web-based survey of COVID-19 vaccination was sent to all students at a public university in Hong Kong on November 8, 2021. The survey link was open for 7 days and received 290 valid responses. Of these, 273 (94.1%) respondents indicated their interest in participating in the pre-post evaluation of the chatbot by leaving their contact information at the end of the survey. We identified and invited all 46 respondents who were either unvaccinated (n=22) or fully vaccinated but hesitant to receive boosters if eligible (n=24; response rate: 46/46, 100%). The planned sample size (20-25 each for unvaccinated or booster-hesitant participants) was based on a previous formative study of a chatbot for promoting human papillomavirus vaccination [18]. Figure 1 shows the study flow diagram.

‎View this figure

Ethics Approval

The study was approved by the Institutional Review Board of the University of Hong Kong/Hospital Authority Hong Kong West Cluster (UW 21-449).

Study Procedures

Participants who were invited to participate in the pre-post study received a WhatsApp message describing the study purpose and provided informed consent. The participants then received a URL link to access the web-based chatbot and start a 7-day trial period. The chatbot could be accessed repeatedly. WhatsApp reminders to use the chatbot were sent on day 3, day 5, and day 7. On day 8, we sent a URL link to the postintervention questionnaire. Participants who completed the pre-post study were given HK $300 (US $38.5) for their time and effort.

On March 30, 2022, about 4 months after the completion of the pre-post study, we conducted a post hoc follow-up with a single question about COVID-19 vaccination or booster status via WhatsApp. Participation was voluntary, and consent was obtained from those who responded to the question. The additional follow-up served to examine the feasibility of measuring the long-term effect of the chatbot.

Design of the Chatbot

The “Vac Chat, Fact Check” chatbot was developed by our team. To promote dissemination, the chatbot could be accessed by any internet browser on smartphones, tablets, and personal computers (ie, web-based). The chatbot was available in Chinese only since most Hong Kong residents (>90%) spoke Chinese. Upon entering the chatbot, the user received a message about how to use the chatbot and a menu of options showing the core functionality of the chatbot. The users could navigate the chatbot by typing the corresponding number of the options in the menus or keywords (eg, allergy) to obtain information directly (Figures 2 and 3).

The chatbot conversation generally unfolded by following predefined rules or decision trees. To better simulate human interactions, the chatbot also used natural language processing (NLP) powered by Google Dialogflow for handling small talk (eg, greetings and thank-yous). The chatbot provided responses mainly using texts with emojis, but some messages also included infographics (Figure 3).

The intervention content followed the Confidence, Complacency and Convenience (“3C’s”) model of vaccine hesitancy [19]. Specifically, the information addressed the lack of trust in the effectiveness and safety of the vaccine (confidence), the lack of perceived risk of COVID-19 or the perception that vaccination is not necessary (complacency), and barriers to access the vaccine (convenience). The information provided by the chatbot was categorized into 6 major topics (Table 1). Our population-based survey suggested that inadequate knowledge about COVID-19 could contribute to vaccine hesitancy [16]. Therefore, the chatbot included general information about COVID-19. Since 2 types of vaccine (inactivated and mRNA vaccines) were available in Hong Kong with differing eligibility criteria (age and pregnancy status), the chatbot also included a decision aid for selecting the suitable vaccine.

‎View this figure

‎View this figure

Instruments

COVID-19 Vaccine–Related Outcomes

All COVID-19 vaccination outcomes were measured at preintervention and postintervention. COVID-19 vaccination status was assessed by asking “have you been vaccinated against COVID-19?” with responses options of “yes, 2 doses,” “yes, 1 dose,” and “no.” Intention to receive the COVID-19 vaccine (for those responded “no”) or COVID-19 boosters (for those responded “yes, 2 doses”) were assessed on a scale from 1 (not likely at all) to 5 (very likely) [20]. By adapting an item from the OCEANS study [21], we also asked, “if people around you were thinking of getting a COVID-19 vaccination, you would...” Responses were coded from 1 (suggest that they do not get the vaccination) to 5 (strongly encourage them).

The main efficacy outcome was changes in COVID-19 vaccine hesitancy from preintervention to postintervention. We adapted the Vaccine Hesitancy Scale (VHS) developed by the WHO’s Strategic Advisory Group of Experts on Immunization for assessing COVID-19 vaccine hesitancy [22]. The COVID-19 VHS included 9 Likert-style items, each coded from 1 (strongly disagree) to 5 (strongly agree; Multimedia Appendix 1 [22,23]). After the reverse coding of some items, all items were summed to give a total score ranging from 9 to 45, with higher scores indicating greater COVID-19 vaccine hesitancy. The VHS can also be divided into the “Lack of confidence” subscale (7 items) and the “Risk” (2 items) subscale and analyzed.

In our sample, the COVID-19 VHS had high internal consistency preintervention (Cronbach α=.86) and postintervention (Cronbach α=.88) [24]. Concurrent validity was supported by a higher mean VHS score in unvaccinated participants versus those who received 1 dose and 2 doses of vaccine (28.6 vs 26.4 vs 23.0, respectively; P<.001). The VHS score was also inversely and moderately correlated to intention to receive the vaccine (Spearman ρ=–0.48; P=.01) or boosters (Spearman ρ=–0.55; P<.001) and willingness to encourage others to vaccinate (Spearman ρ=–0.64; P<.001) [25].

In the 4-month follow-up, we assessed the COVID-19 vaccination status in initially unvaccinated participants by their responses of “no” and “yes, [number] dose(s).” For booster-hesitant participants, we asked whether they had received a booster shot (“yes” or “no”).

Usability and Acceptability Outcomes

The postintervention questionnaire included the System Usability Scale (SUS), a widely used instrument in software engineering, to measure the participants’ perceived usability of the chatbot. The 10-item SUS gives a composite score ranging from 0 to 100, with 68 or above indicating above-average usability [26]. Other acceptability measures included the perceived usefulness of the chatbot in (1) getting information about the COVID-19 vaccine, (2) making decisions about vaccination, and (3) increasing the motivation to get vaccinated, each assessed on a scale from 1 (not useful at all) to 5 (very useful). Overall satisfaction with the chatbot was assessed by asking “how likely would you recommend the chatbot to other people” on a 11-point scale from 0 (not likely at all) to 10 (very likely).

Other Measures

The baseline questionnaire included the eHealth Literacy Scale (eHEALS) [27], which has been translated into Chinese and used in our study population [28]. The scale included 8 items, which are summed to give an overall score from 8 to 40. Higher scores indicate greater perceived ability to use health technologies. The eHEALS had high internal consistency in our sample (Cronbach α=.91). To assess the perceived susceptibility to and severity of COVID-19, we also asked, “how likely do you think you will contract COVID-19 in the future?” and “how dangerous do you think COVID-19 is to health?” respectively, each with 11-point response options. Data on sociodemographic characteristics, chronic disease, and previous flu vaccination were also collected.

Statistical Analysis

To evaluate the chatbot efficacy, we used 1-sample, 2-tailed t test and Wilcoxon sign-rank test to examine the change in intention to receive a vaccine or booster and COVID-19 VHS scores from preintervention to postintervention. We also examined changes in the “Lack of confidence” and “Risk” subscales of the COVID-19 VHS. The effect size of the pre-post difference in COVID-19 VHS scores (Cohen d) was calculated as a mean difference divided by the SD of the mean difference. Our sample size of 46 participants could detect a moderate effect size of 0.43 (Cohen d) in the pre-post difference in COVID-19 VHS scores with 80% power at 2-sided 5% level of significance. The corresponding effect sizes detectable were 0.64 for intention to vaccinate (n=21) and 0.60 for intention to receive boosters (n=24). Intervention usability and acceptability were reported descriptively. For the secondary aim, we used bivariate and multivariable linear regression to examine the factors associated with the COVID-19 VHS score in all survey respondents. Factors examined included sociodemographic characteristics, chronic disease status, previous flu vaccination, eHealth literacy, and the perceived susceptibility to and severity of COVID-19.

All statistical analyses were conducted in Stata/MP software (version 15.1; StataCorp). We used complete case analyses because there were no missing data in the web-based survey and postintervention assessment except eHealth literacy (n=2) and the perceived susceptibility (n=4) and severity (n=4) of COVID-19. A 2-sided P<.05 denoted statistical significance.

Results

Participant Characteristics

The mean (SD) age of all survey respondents was 21.4 (6.3) years, and 61% (177/290) of respondents were female (Table 2). Participants of the pre-post study (n=46) had similar characteristics to those of nonparticipants (n=244) except, as expected, having significantly higher COVID-19 vaccine hesitancy (P<.001).

Factors Associated With COVID-19 Vaccine Hesitancy

In all survey respondents, both bivariate and multivariable models showed that lower eHealth literacy and perceived danger of COVID-19 were associated with higher COVID-19 vaccine hesitancy (Table 3). The results were similar after additionally adjusting for COVID-19 vaccination status (data not shown).

Pre-Post Evaluation of the Chatbot

The completion rate of the postintervention assessment was 100% (46/46). Table 4 shows the favorable changes in all measures related to COVID-19 vaccination from preintervention to postintervention (mean duration: 15.0 days). The main efficacy outcome of COVID-19 VHS score significantly decreased from 28.6 (preintervention) to 24.5 (postintervention), with a mean difference of –4.2 (P<.001) and an effect size (Cohen d) of 0.94. Similarly, both the “Lack of confidence” and “Risk” subscale scores significantly decreased. Intention to vaccinate or receive boosters and willingness to encourage others to vaccinate significantly increased from preintervention to postintervention. One unvaccinated participant at preintervention reported having received the first dose of the vaccine at postintervention.

Usability and Acceptability of the Chatbot

On average, the participants used the chatbot for a total of 64 (SD 47) minutes during the 1-week trial period. Longer time spent on the chatbot was correlated with a larger reduction in vaccine hesitancy with marginal significance (Spearman ρ=0.26; P=.08). Among participants who used the chatbot (n=46), the median (IQR) SUS score was 72.5 (65-77.5) out of 100. On a scale from 1 (not agree at all) to 5 (strongly agree), the median (IQR) score on the perceived usefulness of the chatbot was 4 (4-4) for getting information about the COVID-19 vaccine, 3 (2-4) for making decisions about vaccination, and 3 (2-3) for increasing the motivation to get vaccinated. The median (IQR) recommendation score was 7 (6-8) on a scale from 0 to 10.

Vaccination Status at 4-Month Follow-up

Overall, 18 (82%) of the 22 initially unvaccinated participants and 19 (79%) of the 24 booster-hesitant participants responded to the post hoc 4-month follow-up. All 18 unvaccinated participants reported having received COVID-19 vaccination (2 doses: n=16, 89%; and 1 dose: n=2, 11%), whereas 7 (37%) of the 19 booster-hesitant participants reported having received boosters.

Discussion

Principal Findings

This pilot study showed a significant decrease in COVID-19 vaccine hesitancy after using the “Vac Chat, Fact Check” chatbot in young adults who were hesitant to vaccinate or receive boosters. According to the rule of thumb of Cohen [29], the effect size (Cohen d=0.94) was large. Other efficacy outcomes, including intention to vaccinate or receive boosters and willingness to encourage others to vaccinate, consistently showed the benefit of the chatbot. The usability of the chatbot was supported by the median SUS score of 72.5 out of 100, which fell between the cutoffs of “good” (a score of 71.4) and “excellent” (a score of 85.5) adjective ratings [30]. The median recommendation score of 7 on a scale from 0 to 10 indicated the satisfactory acceptability of the chatbot [31].

Our PubMed search using the keywords of vaccine and chatbot and their synonyms only identified 1 peer-reviewed study that provided empirical evidence on the efficacy of a chatbot for promoting COVID-19 vaccination. The study was a web-based experiment on a French sample population, which found that interacting with a chatbot could promote more positive attitudes toward COVID-19 vaccines and intention to vaccinate [32]. A study (preprint) also showed an increase in vaccine acceptance in Japanese adults after using “Corowa-kun,” a chatbot in LINE instant messenger [33]. Direct comparison between our study with these studies were difficult because of differences in the study methods, sample characteristics, and outcome measures. Nevertheless, our findings were consistent with these studies by showing a positive impact of chatbot on COVID-19 vaccine uptake.

To our knowledge, our study was the first to include actual receipt of COVID-19 vaccines or boosters as outcome measures in chatbot evaluation. Assuming (conservatively) that all participants lost to follow-up did not receive any vaccine or booster, 82% (18/22) of the initially unvaccinated participants received at least 1 dose of vaccine, whereas 29% (7/24) of booster-hesitant participants received a booster. As a reference, the corresponding rates were 92% and 28% in Hong Kong residents aged 20 to 29 years on March 30, 2022 (same date as the follow-up) [34]. Note that these figures could not be directly compared because of differences in sample characteristics, and our participants were likely more vaccine- or booster-hesitant than the general population. Nonetheless, the satisfactory response rate of 80% (37/46) provides support for the feasibility of conducting longer-term (>3 months) follow-up in future trials.

Corroborating our previous findings in the general population [16], we found that higher perceived severity of, but not susceptibility to, COVID-19 was associated with lower COVID-19 vaccine hesitancy. Previously studies have found that eHealth literacy was associated with knowledge and adherence to nonpharmacological preventive measures against COVID-19 [17,35]. This study further found eHealth literacy to be associated with COVID-19 vaccine hesitancy. Higher eHealth literacy helps people process and discern the credibility of web-based health information, which may buffer the impact of the infodemic (an overabundance of information, both accurate or otherwise, during a disease outbreak) and misinformation against the vaccine and thus hesitancy. Our findings corroborate the importance of building eHealth literacy to fight the COVID-19 pandemic.

Similar to most chatbots built to support health care amid the COVID-19 pandemic [11,12], our “Vac Chat, Fact Check” chatbot was primarily rule-based. We decided against building a chatbot that is entirely powered by NLP for practical reasons. First, existing and readily available NLP engines remain inadequate in handling free-flow conversations in Cantonese (the local Chinese dialect). Second, rule-based chatbots are relatively inexpensive and could be quickly developed and deployed to mitigate the pandemic when health care resources are stretched. NLP-based chatbots could better simulate human interaction but require extensive training and resources to become adequately usable. Nonetheless, our study has provided proof-of-concept evidence to support chatbots as a mode of delivery to promote vaccination, which provides the impetus for developing more sophisticated and potentially more effective chatbots.

Limitations

The main limitation of the pre-post study is the lack of a control or comparison group, which limited the causal inference of any changes observed after using the chatbot. The possibility that the observed changes were attributable to contextual changes along the course of the outbreak could not be excluded. However, the study was conducted at a time when Hong Kong had been maintaining a low level of local transmission with nearly 0 daily local case (from June to December 2021). This setting, coupled with the short interval between preintervention and postintervention assessments, was unlikely to have had a substantial effect on the vaccination outcomes. Nevertheless, the findings must be considered preliminary and hypothesis-generating. Another limitation is the small sample size, which precludes the examination of the chatbot’s efficacy in sociodemographic subgroups (eg, sex). Third, since all measures were self-reported, social desirability bias could not be excluded. Finally, our study targeted young adults given their greater vaccine hesitancy than older populations and their frequent use of social networking sites—a major source of misinformation. The generalizability of the findings to other populations is unclear. Due to the convenience sampling method, our participants may not be representative of all young adults who are unvaccinated or booster-hesitant.

Conclusions

Promoting the uptake of COVID-19 vaccines is crucial to mitigating the impact of COVID-19. This pilot study provided initial evidence to support the efficacy, usability, and acceptability of a chatbot for promoting COVID-19 vaccination in young adults who were unvaccinated or booster-hesitant. Randomized controlled trials are warranted to test the effectiveness of the chatbot in increasing COVID-19 vaccination. Although our study indicated the benefits of the chatbot in both unvaccinated and booster-hesitant young adults, the drivers for vaccine hesitancy between the 2 groups likely differ. Further research is also needed to understand their differences to provide more tailored information and optimize the chatbot’s efficacy.