Linking Opinions Shared on Social Media About COVID-19 Public Health Measures to Adherence: Repeated Cross-Sectional Surveys of Twitter Use in Canada

Introduction

Background

Vaccination and nonpharmaceutical public health measures (PHMs) are used to reduce the spread of SARS-CoV-2 and minimize the occurrence of severe COVID-19, hospitalizations, and deaths. Vaccination provides some immunity against infection, reduces the severity of symptoms, and prevents death [1-3]. Nonpharmaceutical PHMs aim to minimize and prevent infectious contacts and include case detection and isolation, contact tracing and quarantine, travel restrictions, physical distancing, hand washing, and mask wearing. The effectiveness of vaccination against SARS-CoV-2 transmission was repetitively shown to be high, although heterogeneous for different populations and virus variants [4]. The reported effectiveness of nonpharmaceutical PHMs in lowering SARS-CoV-2 transmission is also variable [5,6]. Whether considering vaccination or nonpharmaceutical PHMs, effectiveness is influenced by the levels of coverage and adherence across the target population. During the pandemic, several methods have been used to measure adherence to PHMs. Public opinion research can measure the knowledge, attitudes, and behavior of respondents from panels of people who have agreed to participate [7]. Limitations of this approach include time- and resource-intensiveness, missing individuals reluctant to share their opinions, and errors from respondents who may not truthfully answer or misinterpret the meaning of survey questions.

Another potential source of adherence data is social media [8,9]. While the use of open-access social media data for public health has grown over the past 20 years, there are challenges. In particular, sociodemographic information of social media users is unavailable. Consequently, it is not possible to obtain a representative sample through sampling strategy or weights. Moreover, it is difficult to validate opinions and behaviors. For example, someone may tweet that they do not value COVID-19 vaccination, even if they are vaccinated.

Objectives

The primary objective of this repeated cross-sectional study was to compare self-reported PHM adherence among people who tweeted about COVID-19 in the past 2 weeks, people who tweeted about something else in the past 2 weeks, and non–Twitter users. In addition, we aimed to compare the sociodemographic characteristics among these 3 groups, as well as among Twitter users, and to analyze reported PHM adherence according to the opinion regarding PHMs shared in their tweets.

Methods

Ethical Considerations

This project was approved by the Health Canada and the Public Health Agency of Canada Research Ethics Board (REB 2021-043P) and Comité d’éthique de la recherche en sciences et en santé at the University of Montreal (CERSES-20-056-D). Survey respondents were invited from the Angus Reid Forum, a large-scale national panel whose members receive small monetary compensation and occasional prize draws for completing surveys [10]. Consent was obtained through the Angus Reid platform, and participation to the panel and surveys was voluntary. Angus Reid reported participants could opt out at any time and refuse to answer any question. The data provided by Angus Reid were anonymized and deidentified. Although some information such as age and gender was provided, the identification of participants by a third party is very unlikely. Moreover, the data were kept in a secure server and were password protected.

Recruitment

The repeated cross-sectional surveys were conducted every fortnight (cycles) from September 2020 to March 2022, except during the Canadian federal election (end of July 2021 to mid-November 2021). Moreover, 5 additional cycles were skipped due to financial restrictions (October 7, 2020; November 4, 2020; February 16, 2021; March 16, 2021; and March 30, 2021). While the Angus Reid Forum is not a probabilistic sample of the Canadian population, it includes enough panelists from basic demographic groups and regions to ensure that survey samples are similar to the adult Canadian population and weights are calculated according to demographic and regional data [10]. To minimize survey fatigue, forum respondents were removed from the pool of potential invitees for the 4 cycles after completing the survey. This means that some participants were repeated over cycles, but because respondents were asked to report behavior and social media use during the past 2 weeks, they were treated as unique respondents. The 27 cycles of survey data were divided into 6 periods representing different phases of the virus transmission and vaccination campaign in Canada (Table 1).

Outcome Variables

Respondents were asked how often they wore a mask in public places in the previous fortnight. Adherence was dichotomized as poor (never, rarely, or sometimes) or good (every time or most of the time). Respondents were also asked whether they would be, or were already, vaccinated. The wording of the question changed according to the availability of vaccines (Multimedia Appendix 1). Adherence to vaccination was categorized into poor (would not be vaccinated) or good (was not sure, would be vaccinated, or had at least 1 dose of vaccine). The inclusion of respondents who were unsure about vaccination in the good adherence category was chosen as the vaccines were not approved at the beginning of the study and did not become available to everyone at the same time (Multimedia Appendix 1).

Exposure Variables of Primary Interest

We were mainly interested in the association of 2 variables (referred to as exposure) with the adherence to mask wearing and vaccination (referred to as outcomes): the reported use of Twitter and related tweeted opinion. We used 2 approaches to measure Twitter use and the tweeted opinion about PHMs because the questions varied over the study period (Multimedia Appendix 2). Until August 2021 (periods 1-4), reported Twitter use in the previous fortnight was categorized as one of following: tweeting about COVID-19, not tweeting about COVID-19, or not using Twitter. In periods 5 and 6, question wording was changed such that no information on Twitter users not tweeting about COVID-19 was collected (Multimedia Appendix 2). For all periods, Twitter users who reported having posted about COVID-19 were asked to select all the specific COVID-19–related topics they had tweeted about in the previous fortnight (Multimedia Appendix 2), including PHMs and vaccination. Those selecting these options were asked whether they posted being in support, neutral, or critical of the PHMs. The 3-category opinion variable was combined with Twitter use to create a 5-category variable for regression analyses over all 6 periods: did not use Twitter or did not tweet about COVID-19, tweeted about COVID-19 but not about PHMs, tweeted in support of PHMs, tweeted about PHMs in a neutral manner, and tweeted critically about PHMs.

Other Variables of Interest

Collected data also included survey date and respondent information, namely the province of residence, forward sortation area, age, gender, education level, occupation, household income, used official language, the country of origin, and ethnicity. We used the population size of the respondents’ residence by linking the forward sortation area to census data [11]. The daily containment and health index (CHI) for each province provided a measure for the implementation intensity of PHMs [5]. These data were obtained from a group at the Public Health Agency of Canada that used the methodology developed by the Oxford COVID-19 Government Response Tracker [12]. It was summarized as the median CHI per cycle per province and categorized into quartiles.

Statistical Analyses

Statistical analyses were performed using R (version 4.2.2; R Foundation for Statistical Computing) [13]. For all analyses, data were considered cross-sectional due to the study design and the fact that Twitter use was reported over the 2 weeks before the survey. Descriptive statistics, including proportions with 95% CIs, were calculated for all periods using the weights provided by Angus Reid (srvyr package) [14]. Weighted proportions are presented in the Results section as proportion of Canadians. We considered the sociodemographic characteristics of participants to be similar to those of the overall Canadian population if the CI included the proportion reported by the National Census of Population 2021 [15]. Sociodemographic descriptive statistics by Twitter use categories were run for periods 1 to 4 only. We considered categories to be different when trends were outside the CI.

While there were no missing values in the survey answers, the option prefer not to say for income and ethnicity variables was considered as a missing value to minimize misclassification and removed from regression analyses (listwise deletion). Associations were estimated using Bayesian logistic regression models (brms package) [16]. Priors for regression coefficients were normal distributions centered on 0 with a precision of 1. Intercepts were normal distributions centered on 0 with a sigma of 1.

The modifying effect of selected covariables (region, period, age, gender, language, education, and CHI) on the association between the exposure of interest (Twitter use or combined tweeted opinions) and the 2 outcomes (vaccination and mask-wearing adherence) was assessed. Additional covariables were added to the model to adjust for confounding as follows: to avoid including collider variables, the variables of the minimal sufficient adjustment set identified in a directed acyclic graph (Multimedia Appendix 3) were added one at a time and kept for the final model if the beta coefficients changed by >10% [17]. Briefly, the directed acyclic graph included 10 covariables (sociodemographic variables, period, and CHI) and their expected association with Twitter use or opinions (exposures), PHMs’ adherence (outcome), and each other. Associations were based on previous findings about social media use [18,19], census information [11,15], and hypotheses from the research team. The final models were generated using 3 chains with a length of 3000, in which the first 1000 iterations were used as burn-in [20]. Convergence was monitored through the visual inspection of trace plots of variance components and density plots and by obtaining effective sample sizes. Goodness of fit was assessed using the correct classification rate and area under the curve [21]. Results are presented as odds ratio (OR) with 95% Bayesian credibility intervals (BCIs).

Results

Respondents

Survey data included 40,230 survey responses. Sociodemographic characteristics of the respondents varied slightly throughout the 6 periods (Multimedia Appendix 4). Table 2 compares the weighted distribution of respondents over the 6 periods to that of the Canadian census [15]. Several differences were noted, for example, for gender, household income, age, and the country of origin (Table 2).

Twitter Use and Tweeted Opinions About PHMs

During periods 1 to 4, 20.6% (95% CI 20.1%-21.2%) of Canadians reported using Twitter in the previous fortnight (Multimedia Appendix 4), of whom 29.9% (95% CI 28.6%-31.3%) tweeted about COVID-19 at least once. Table 2 shows the weighted distribution of respondents by the type of reported Twitter use, compared to that of the whole sampled population and to the census data. For example, compared to the overall study population or Canadian census, Twitter nonusers were overrepresented in Quebec but underrepresented in Ontario. Indeed, the proportion of Twitter users was lower in Quebec (14.3%, 95% CI 13.3%-15.3%) than in other provinces (British Columbia: 19.3%, 95% CI 17.8%-20.7%; Prairies: 20.5%, 95% CI 19.3%-21.8%; Ontario: 25%, 95% CI 24.1%-25.9%; and Atlantic: 21.4%, 95% CI 19.4%-23.4%). There was a greater proportion of Twitter users in large (22%, 95% CI 21.3%-22.6%) than in medium (18%, 95% CI 16.4%-19.5%) and small (16.5%, 95% CI 15.3%-17.8%) population size areas, among younger (aged 18 to 54 years; 23%, 95% CI 22.3%-23.8%) than among older (aged ≥55 years; 16.6%, 95% CI 15.8%-17.3%) age groups, in men (22.6%, 95% CI 21.8%-23.4%) than in women (18.8%, 95% CI 18.1%-19.6%), and in people with university education (23.9%, 95% CI 23.1%-24.7%) than in people with lower levels of education (high school: 18.3%, 95% CI 17.3%-19.4%; college: 19.7%, 95% CI 18.8%-20.5%). Moreover, there was a lower proportion of Twitter users of European ancestry (19.9%, 95% CI 19.4%-20.5%) than Twitter users of Indigenous, First Nation, Inuit, or Metis (22.4%, 95% CI 19.7%-25.1%) or of other ethnic ancestries (25.7%, 95% CI 23.9%-27.5%).

Of the Twitter users who tweeted about PHMs, 39.8% (95% CI 37.4%-42.3%) self-declared that their tweets were in support of PHMs, 49.4% (95% CI 46.9%-51.9%) self-declared that their tweets were critical, and 10.7% (95% CI 9.2%-12.3%) self-declared that their tweets were neutral. Opinions about PHMs varied over the study period, with a greater proportion of tweets supporting PHMs in period 1 and a greater proportion of critical tweets in period 6 (Figure 1). A greater proportion of Quebec, Ontario, and Prairies inhabitants reported tweeting critically about PHMs than those from the Atlantic provinces and British Columbia (Table 3). This was also the case for men, those with an annual household income >CAD $100,000 (according to the Bank of Canada, the annual conversion rate to US $ for the study period were 1.34 in 2020, 1.25 in 2021, and 1.30 in 2022), and those without a university degree (Table 3).

‎

Adherence to PHMs

Overall, 11% (95% CI 10.6%-11.3%) of Canadians reported poor mask-wearing adherence, which varied by period between 5.9% and 16.3%, with the highest value reported in period 4 (Multimedia Appendix 5). Mask-wearing adherence also varied by sociodemographic characteristics, with French-speaking respondents and women wearing a mask more frequently than English-speaking respondents and men, respectively. Mask-wearing frequency increased with age and education level, and decreased with income (Multimedia Appendix 6). Similarly, 10.8% (95% CI 10.4%-11.2%) of Canadians reported poor adherence to vaccination, which also varied by period. The proportion of people who reported poor adherence was 15.1% (95% CI 13.9%-16.3%) in period 1, which decreased to a minimum of 7.4% (95% CI 5.7%-9.2%) in period 4 before increasing again to 9.8% (95% CI 9.0%-10.6%) in period 6 (Multimedia Appendix 5). Vaccination adherence also varied by sociodemographic characteristics, with a greater proportion of Canadians who would not be vaccinated in the Prairies than in other regions, among men than women, among those aged between 18 and 54 years than among the older groups, and in urban areas than in rural areas. The proportion of Canadians in favor of vaccination increased with education level and household income (Multimedia Appendix 7). During periods 2 and 3 (not measured in other periods), 91.1% (95% CI 90.3%-92.0%) of Canadians who were planning to receive vaccination said they would also continue to follow other preventive health measures, including mask wearing.

Association Among Twitter Use, Tweeted Opinion, and Adherence to PHMs

The proportion of Canadians who reported poor adherence to PHMs differed by Twitter use and tweeted opinion. Overall, 19% (95% CI 16.4%-21.6%) of those who tweeted about COVID-19 had poor adherence to vaccination, and the proportion of those who had poor adherence to mask wearing varied by geographical region and age (Tables 4 and 5). The highest proportion of respondents with poor adherence to mask wearing and vaccination was in the subset who tweeted critically about PHMs (29.6% to 33.1%), while the lowest proportion was in the subset who tweeted in support of PHMs (2.2% to 5.2%; Tables 6 and 7).

In comparing non–Twitter users to participants who tweeted about COVID-19, the adjusted OR of poor adherence to mask wearing ranged from 0.79 (95% BCI 0.18-1.69) to 4.83 (95% BCI 3.13-6.86; Table 4). Geographical region and age were effect modifiers: the association was not observed in the Prairies and the Atlantic provinces, and the strength of the association decreased with age. In comparing non–Twitter users to those who tweeted about something else, the adjusted OR of poor adherence to mask wearing ranged from 0.27 (95% BCI 0.04-0.67) to 1.02 (95% BCI 0.59-1.51; Table 4). The adjusted OR of poor vaccination adherence was 1.76 (95% BCI 1.48 to 2.07) for participants who tweeted about COVID-19 and 0.67 (95% BCI 0.57 to 0.78) for those who tweeted about something else compared to non–Twitter users (Table 5).

The adjusted ORs of poor adherence to mask wearing for participants tweeting critically about PHMs were 4.07 (95% BCI 3.38-4.80) for English-speaking participants and 7.31 (95% BCI 4.26-11.03) for French-speaking participants, compared to non–Twitter users and those who did not tweet about COVID-19 (Table 6). For participants tweeting in support of PHMs compared to non–Twitter users and those who did not tweet about COVID-19, the adjusted ORs were 0.47 (95% BCI 0.31-0.64) and 0.96 (95% BCI 0.18-2.33) for English- and French-speaking participants, respectively (Table 6). The adjusted ORs of poor vaccination adherence for those tweeting critically about PHMs and for those tweeting in support of PHMs were 4.10 (95% BCI 3.40-4.85) and 0.20 (95% BCI 0.10-0.32), respectively, compared to non–Twitter users or those not tweeting about COVID-19 (Table 7).

Discussion

Principal Findings

This is the first individual-level repeated cross-sectional study to measure the magnitude of the association, in the context of COVID-19, of self-reported Twitter use and tweeted content with self-reported mask-wearing and vaccination behaviors. Previous studies were all susceptible to ecological bias and confounding linked to the sociodemographic characteristics of social media users [8,9]. Our study lasted nearly 2 years and used web-based survey data from participants weighted to be representative of the Canadian population. Our results underscore the sociodemographic differences among types of Twitter users and non–Twitter users. Similar to other study findings [18,19], more Twitter users were younger men and living in large cities with a higher income or level of education compared to nonusers. Furthermore, our study provides evidence that people who tweet critically about PHMs are also less likely to comply with these measures.

Among all study participants, there was an overall 89% adherence to both mask wearing and vaccination. Among participants tweeting about COVID-19, however, the adherence to mask wearing was between 70% and 96%, and the adherence to vaccination was 81%. This is somewhat comparable to what was found in a worldwide Twitter poll where vaccination intention in February 2021 was 83% [22]. Canadians who tweeted critically about PHMs and those who tweeted in support of PHMs differed drastically in their adherence to mask wearing (67% to 70%, and 95%, respectively) and vaccination (68% and 98%, respectively). These results suggest that sentiments shared on Twitter are linked to adherence to PHMs but that these estimates are not representative of overall adherence. It is possible these discrepancies would be observed for other social media platforms, but this would need to be confirmed. For example, correlations were found between the sentiment of tweets extracted using an algorithm, announcement of nonpharmaceutical interventions from governmental entities, and reported compliance [23]. However, our results highlight the importance of considering sociodemographic and behavioral discrepancies in the general population, given the types of social media use and nonuse, when using health intelligence from social media for health surveillance applications [24].

The association between tweeting about COVID-19 as well as opinions about PHMs and poor adherence to PHMs was confirmed in multivariable analyses adjusting for sociodemographic confounders. Although we did not explore the underlying explanation for the association between Twitter use and PHM adherence, our findings align with a Canadian study that found an association between social media exposure and having more misperceptions and less social distancing adherence, compared to exposure to traditional news media [25]. Other studies have reported a negative association between believing in misinformation and adherence to PHMs [26-28], although not specific to social media exposure. Therefore, our findings may reflect greater misperceptions among some Twitter users [25].

The association between Twitter use and adherence to mask wearing was not uniform across age or geographical regions. The association between Twitter use and adherence to mask wearing weakened with age, except among those aged 65 to 74 years. From the beginning of the pandemic, this age group was a highly susceptible population and generally reported high adherence to PHMs [29,30], which could explain this finding. Regional differences in the association between Twitter use and adherence to mask wearing reflected stronger associations between mask wearing and Twitter use in Quebec and no associations in the Atlantic provinces [31]. There was a greater adherence to mask wearing in both these regions than in the other regions, but the Twitter user population differed. In the Atlantic provinces, the proportion of adherence to PHMs was high across Twitter users and nonusers. This could be due to the aggressive implementation of PHMs early in the pandemic, which allowed a return to economic and social activities [32]. There was a smaller proportion of Twitter users in Quebec, but those who tweeted had generally low support for PHMs. Perhaps the few Twitter users in Quebec were more prone to use this medium to voice their discontent with PHMs than those in other regions. A major difference in Quebec is that French is the official language and is the primary language spoken. While 79% of the Quebec population speaks French at home, this proportion ranged from <1% (British Columbia, Alberta, Saskatchewan, and Newfoundland and Labrador) to 28% (New Brunswick) in other provinces [15]. While geographical region was not an effect modifier for the model assessing the association between adherence to mask wearing and opinions shared on Twitter, language was. There are French-speaking communities across the country, but this interaction is likely driven by differences in Quebec. These results point to the need to account for regional and cultural differences when using social media data for estimating populations’ behaviors.

Social Media and Real-Time Monitoring of PHM Adherence

Open-access social media are among the most timely and voluminous digital information sources for public health surveillance to infer population-level health-related behaviors. Application program interfaces (APIs) enable efficient collection of high volumes of posts. Some APIs allow targeted sample collection, such as sample collection by content (eg, keywords and hashtags), languages, and regions. Associated metadata from collected social media publications can include a user identifier, time stamp of publication, and republication status. Some users make the location of publication or residence public. The publications and associated metadata can then be used to derive geolocated indicators to measure real- or near real–time changes in health-related behaviors. Possible indicators include time-series variables for the volume of publications, volume of publications about a topic, or sentiment of publications or topics. Monitoring social media for trends in PHM adherence could improve agility in adapting PHMs and communication strategies to encourage adherence. Likewise, real-time signals of low adherence could provide advanced warning for areas at a higher risk of disease emergence or re-emergence.

Public health institutes could capitalize on using social media for health intelligence through event-based surveillance systems and other tools that scan open-access data from the internet for evidence of public health threats [33,34]. However, there are ethical and practical challenges to using social media for public health surveillance, which are constantly evolving. People who post publicly may not be aware that their data are being collected or used for purposes beyond their intent, and this raises ethical, legal, and privacy considerations [35]. In addition, the open access to data through APIs can be discontinued at any moment, which happened with Twitter following the ownership change in 2022. Representative sampling from the population of interest is also difficult. Platforms may be used differently over time and according to social demographic factors [36]. Furthermore, social media contain misinformation and disinformation about health, which can influence health-related behaviors [37].

Limitations and Biases

The surveyed population was slightly different from the Canadian population [15], with an overrepresentation of younger people with a high income. This could have driven part of the sociodemographic characteristics we observed for Twitter users and overestimated the proportion of Twitter users in Canada [18,19]. However, it likely had a limited impact on the adjusted associations.

Classification errors were also possible. First, we asked respondents to self-report their adherence to PHMs; answers could have been influenced by social desirability bias [38]. It is possible that Canadians who tend to tweet in support of PHMs would be more susceptible to social desirability bias, while those tweeting critically might overstate their nonadherent behaviors. Hence, we cannot discount the possibility of differential bias when estimating the type of Twitter use. This is less likely to have happened for the measurement of Twitter use. Second, the classification used for good and poor adherence could have resulted in errors: we determined that respondents with poor adherence to vaccination included only respondents who would definitely not get vaccinated and classified respondents who were unsure as having good adherence. We used this approach because of the change in vaccine approval and availability throughout the study period [31] and to allow for the aggregation of answer choices that also changed (Multimedia Appendices 1 and 2). We classified good adherence to mask wearing as wearing a mask in public “most of the time” or “all the time.” The classification bias resulting from these decisions would likely underestimate the proportion of Canadians with poor adherence, but the direction or magnitude of this bias on association measures is difficult to predict.

Conclusions

We found that Twitter users who tweeted about COVID-19 had different sociodemographic characteristics and reported lower adherence to PHMs than users who did not tweet about COVID-19 and nonusers. Moreover, the lowest adherence was among Twitter users who tweeted critically about PHMs. Furthermore, the proportion of people tweeting in support of (or critically about) PHMs was not a good indicator of adherence to PHMs in the general population. Study trends were relatively stable over the 2-year study period, supporting the utility of using indicators of health-related behavior from social media over a limited multiyear period. However, because sociodemographic information from social media users is usually unavailable, our results call for caution in using social media information to estimate support for and adherence to PHMs in the general population. Future research should determine whether adjustments using sociodemographic characteristics could mitigate this, improving the ability to predict behavior and adherence and, ultimately, to characterize or forecast trends in disease transmission.