Using Transformer-Based Topic Modeling to Examine Discussions of Delta-8 Tetrahydrocannabinol: Content Analysis

Introduction

Emergence of Novel Cannabinoids

Over the last 10 years, the use of cannabis-based products in the United States has increased, with possible reasons including increased legalization, expanding types of products, and wider access to cannabis [1,2]. The passage of the Agriculture Improvement Act of 2018 (Farm Bill) effectively legalized hemp; consequently, novel cannabinoids derived from hemp such as delta-8 tetrahydrocannabinol (THC) emerged in the marketplace [3]. The health effects of novel cannabinoids are not well or widely known. Delta-8 THC is a psychoactive cannabinoid found in small amounts naturally in the cannabis plant and can also be manufactured in larger quantities from hemp-derived cannabidiol (CBD). Most states permit the sale of hemp and hemp-derived CBD products, which typically are not psychoactive in nature [4]. Because delta-8 THC products are derived from hemp, they have become widely available in some state hemp marketplaces. Increasing concerns about the potential unanticipated psychoactive effects (in part given by its availability in marketplaces where cannabis is illegal) and health risks from delta-8 THC warrant the need for more research around its use, trends in popularity, and the types of health effects individuals may experience [5].

Delta-8 THC and Health Concerns

Emerging evidence from public health agencies further supports the need to study hemp-derived cannabinoids such as delta-8 THC and determine their adverse health outcomes and psychoactive effects that consumers may or may not anticipate [3]. The Centers for Disease Control and Prevention (CDC) and the Food and Drug Administration (FDA) have issued public notices warning of the potentially harmful effects of delta-8 THC. On September 14, 2021, the CDC published a health advisory through the Health Alert Network reporting an increase in adverse health events (eg, lethargy, difficulty breathing, and decreased blood pressure) associated with delta-8 THC [6]. The FDA reported 104 adverse events between December 1, 2020, and February 28, 2022, including hallucinations, dizziness, confusion, and loss of consciousness [7]. Both notices included data from national poison control centers, which reported 2362 delta-8 THC exposure cases between January 1, 2021, and February 28, 2022, including pediatric inpatient and outpatient cases. Furthermore, delta-8 THC products are unregulated and contain byproducts that are not well-characterized; thus, a better understanding of trends in usage and potential adverse health effects could help to avoid public health problems such as the recent e-cigarette, or vaping, product use associated lung injury (EVALI) outbreak associated with an unregulated chemical additive [8,9].

Social Media and Health Surveillance

Data on cannabis-based product use trends and health outcomes have traditionally come from self-report surveys or interactions with health care providers and clinicians [10,11]. The study of social media data has emerged as a complement to augment traditional public health surveillance [12]. Social media data can help provide real-time information and identify discussions on rare or emerging topics [13]. Social media also allows researchers to see public health concerns from the public’s perspective regarding potential symptomatic and health-related effects that may not be reported in routine medical visits or other traditional health surveillance systems.

This study aims to leverage social media data from Reddit (Reddit Inc) to identify emerging trends and topics discussed about delta-8 THC, including health concerns, adverse symptoms from delta-8 THC use, and other emerging THC products or isomers. We test and use state-of-the-art topic models based on transformer-based neural network architectures used in natural language processing (NLP) [14]. BERTopic, which uses the Bidirectional Encoder Representations From Transformers (BERT) framework, specifically uses transformers, a novel type of neural network model, to detect the coherent context in textual data, such as social media, and cluster-based techniques to determine topics [15]. The analytical approach we use helps inform our current understanding of delta-8 THC and provides a model for future public health efforts to study emerging substances.

Methods

Data Source

Reddit is a major social networking site comprised of multiple topical web-based forums and has referred to itself as “the front page of the Internet [16].” Available estimates indicate that Reddit maintains over 50 million daily active users who post content such as images, links, and questions on various topics, including cannabis and substance use [16]. “Subreddits” or topic-specific communities house the designated topics [17].

Pushshift.io provides an application programming interface (API) that allows users to easily query and access Reddit’s historical data [18]. In this study, we identified 115 subreddits related to cannabis, drawn from prior research on drug subreddits and additional qualitative searching of links on the Reddit platform [13]. We obtained the questions and comments (hereafter referred to as posts) from each subreddit starting in January 2008, the earliest date when Reddit enabled user-generated subreddit, to December 2021, using the Pushshift API. Critical data collected included the raw unstructured text from the post and the date of the post.

Determining Mentions of Delta-8 THC

To first isolate posts about delta-8 THC out of the larger sample of all cannabis and drug-related posts, we identified spelling variants and abbreviations used in discussing delta-8 THC. A Word2Vec model was trained and implemented in PySpark to identify spelling variants [19-21]. Minimal preprocessing of the raw text was performed, as is typical when further downstream use in neural network models is used, with the exception of lowercasing text and removing URL links. This allows neural network models to use all available information, including nonstandard abbreviations or symbols. Word2Vec is a neural network-based machine learning model that converts words to numerical vector representations [20]. We used the resulting word vectors to identify the terms with the highest cosine similarity to delta-8 THC. The authors reviewed these terms to identify the relevant linguistic variations of delta-8 THC (ie, d-8, d8, delta-8-thc, delta-8, d8thc, delta8, d8thc, 8thc, delta8thc, and delta 8). We used this list of terms to filter our data set for posts explicitly mentioning delta-8 THC or synonymous word variations.

Topic Modeling

Recent advances in unsupervised topic modeling have focused on leveraging models based on transformer architectures and large amounts of pretrained language data, such as the well-known BERT language model released by Google [14]. However, most applications of BERT and transformer models have centered on supervised learning tasks—explicitly teaching the model what categories to classify into by tuning the model using human-labeled data. In this work, we use unsupervised learning capabilities of such models to identify topics discussed in the posts without a priori knowledge of the existing themes. The main benefit of using transformer-based models is that they more efficiently identify contextual relationships between words in large data sets than previous NLP models [22].

For the specific implementation of this unsupervised topic modeling, we use the BERTopic framework, which is available as a Python-based library by the same name [15]. BERTopic uses a probabilistic generative topic model, which uses deep learning models to determine the topics in a collection of documents. BERTopic first represents each post as an embedding, leveraging a large pretrained language model. As a next step, the dimensionality of the embeddings is reduced using uniform manifold approximation and projection, an algorithm that outperforms traditional techniques such as principal component analysis in the setting of high-dimensional data [23]. Finally, the reduced embeddings are clustered via Hierarchical Density-Based Spatial Clustering of Applications with Noise (HDBSCAN) and assigned to topics [15]. The parameters selected in our implementation of BERTopic were as follows: sentence transformer model “all-MiniLM-L6-v2” for the embedding model and min_cluster_size=73, metric=euclidean, cluster_selection_method=eom, and min_samples=5 for HDBSCAN.

As the accuracy of any topic modeling approach depends on selecting the ideal number of clusters or topics needed to describe the data optimally, we determined the ideal number of topics through a grid search of possible minimum cluster sizes. We evaluated each iteration’s UMass coherence score, a widely used metric for this purpose which can be calculated via the Python-based Gensim library [24]. Manual inspection of topics or clusters verified the accuracy of the assignment of posts; BERTopic also identifies posts that cannot be reliably assigned to any cluster. This is both a strength and a limitation of the algorithmic approach, as elaborated further in the study. The ability of BERTopic to prevent the inclusion of poorly fitting posts into a given topic helps preserve topic clarity and interpretation of results. However, this also means that some information is unused. Nonetheless, algorithmic approaches are necessary when the volume of information is too large to be reviewed by human annotators. Furthermore, the brevity and informality of social media text can also make it impossible for even human reviewers to accurately assign topics; thus, maintaining a topic categorization scheme that permits a category for content that is unable to be classified is necessary. Finally, we randomly sampled 50 posts from each topic generated and labeled each topic by qualitatively reviewing the posts. The posts were reviewed by 3 reviewers to ensure agreement and accuracy of the assigned topic label. Last, we collapsed topics into significant themes for the presentation of results.

Ethical Considerations

This study used publicly available data from the Pushshift API, available to all researchers and the public. The study was reviewed by the CDC and deemed exempt from CDC institutional review board review (#6871). Nonetheless, there are important ethical considerations for internet-based research. Example posts that are presented are slightly edited to preserve anonymity. Further, manual qualitative inspection and review of the data found no identifiable information and content included in this study therefore includes no such identifying information.

Results

In total, there were 44,719,379 posts retrieved from the 115 cannabis-related subreddits from January 2008 to December 2021. Of these, less than 1% (n=41,191) were discussions of delta-8 THC. Figure 1 [6] illustrates the rate (number of mentions per 100,000 posts) of delta-8 THC mentions in cannabis-related posts from 2008 to 2021. Until 2020, Reddit posts contained fewer than 10 mentions of delta-8-THC for every 100,000 cannabis posts annually. However, in 2020 these rates increased by 13 times the 2019 rate to 99.2 posts per 100,000 cannabis posts and continued to increase into 2021 (349.5 mentions per 100,000 posts).

‎

Topic modeling on the 41,191 delta-8 THC-related Reddit posts yielded 81 discrete topics, including a single category that the algorithm returns for posts that the topic modeling could not classify. This topic contained 17,843 (43.3%) posts. Although, this is a large percentage of the total posts, manual qualitative inspection of these posts confirmed the difficulty in classifying this content as discussed in the Methods section above. For example, some of these posts were brief phrases without a clear topical focus, such as “upvote d8” or “looks like delta-8,” and we omitted the topic encompassing these posts from further analysis. Additionally, we classified 682 posts into a noninformative category that contained only a subreddit title or the word “delta 8” without any additional information; these were also omitted from further analysis, yielding 22,666 relevant posts.

Of the resulting 22,666 posts, the most common topic, comprising 27.6% (n=6260) of posts, centered on product modalities, such as cartridges, edibles, vapes, tinctures, oils, waxes, gummies, and pens (Table 1). The second most common topic (n=4495, 19.8%) compared delta-8 THC to other cannabinoids, including delta-9 THC, CBD, cannabigerol, THC-O acetate, and hexahydrocannabinol in terms of toxicity, costs, symptoms, and product brands. The third most common topic comprised 13.8% (n=3128) of posts. It included warning posts made by bots, a product feature within Reddit that some subreddits or moderators may use to send automated messages to users. For example, posts on this topic warned Reddit users about the potentially harmful illicit drug market for delta-8 THC products or potential misinformation about lab testing of delta-8 THC brands or products. The legality topic comprised 11.2% (n=2548) of posts. It included discussions about delta-8 THC usage in states where cannabis use is illegal or the implementation of the Farm Bill and its possible effects on delta-8 THC availability and usage. Another 6.1% (n=1390) of posts fell under the topic of general interest in delta-8 THC and the discussions included experience with delta-8 THC products among people who use them and questions from Reddit users who were curious about delta-8 THC more generally. The topic of health made up 5.4% (n=1220) of posts and included discussions about the perceived therapeutic effects of delta-8 THC (eg, reduction in anxiety or sleep issues) and the negative health effects (eg, lung and breathing issues). Each of the following topics comprised less than 5% of posts: lab testing for safety, psychoactive and other effects, chemical nature or properties, product mistrust, purchasing, and drug testing.

Table 2 displays example posts from each of the 12 topic areas to further elucidate the type of content being discussed.

Discussion

Principal Findings

This study evaluated delta-8 THC posts in a web-based drug forum. We further identified critical topics of discussion using a state-of-the-art unsupervised machine learning approach based on transformer models. This work constitutes a crucial effort in public health as new, unregulated cannabis-related products are rapidly emerging in the marketplace and gaining wide usage, despite a limited understanding of health effects. Our study additionally identified that posts related to delta-8 THC began rapidly increasing in March 2020, which was well before delta-8 THC–related health concerns were identified through traditional public health data sources, suggesting this method may improve the timeliness of surveillance of novel substances [6]. This study further underscores the relevance and applicability of web-based data to serve as a complimentary, leading indicator of potential emerging health threats.

Examining the topics from our modeling approach informs an understanding of the rapid growth in delta-8 THC interest. First, a variety of product formulations and ingestion methods are being promoted for this substance, from smoking to vaping to edibles to syringe-filled distillates [25]. While historically cannabis has been commonly consumed through smoking (eg, joints, blunts, and bongs), other modalities are on the rise (eg, edibles, tinctures, and oils) [26]. In the current marketplace, delta-8 THC is available in many different forms; this may facilitate uptake, given that people who use delta-8 THC can easily find a product in their preferred consumption modality [27].

Growth in the use of delta-8 THC is also facilitated by the hemp policy in the 2018 Farm Bill [28], which Reddit users discuss in the relatively large topic of legality. This loophole allows the sale of hemp-derived THC isomers and other cannabinoids, including delta-8 THC, to be sold in products across all states [3,28]. Delta-8 THC is known to have psychoactive effects, and this is supported by individuals who report in Reddit posts that they experienced feeling “high,” “baked,” or “stoned” [29]. Thus, delta-8 THC products represent an alternative to illicit consumption of traditional THC products. An emerging understanding of the psychoactive effects of delta-8 THC, including (1) apprehension and anxiety; (2) euphoria; (3) loquaciousness; (4) lowering of inhibitions; (5) hunger and thirst; (6) feeling “high;” (7) uncontrollable bursts of laughter or giggles; and (8) drowsiness, languor, lassitude, and a pleasant feeling of fatigue, indicate that they are similar to psychoactive effects of delta-9 THC [29]. Furthermore, considerable increases in mentions of delta-8 THC occurred during the 2019 COVID-19 pandemic, a time in which consumers discuss the use of delta-8 THC to treat mental health problems brought on by related stress or anxiety:

delta 8 has helped so much...covid hit and i have pretty much been high. it got to the point where i was hitting my pen at work, unable to get through the day unless i was stoned.

Several studies support the idea that stress, anxiety, boredom, and wanting to be high during the COVID-19 pandemic increased a person’s desire to use cannabis and nicotine vaping products [30-32].

Although interest in delta-8 THC is elevated, individuals in web-based forums expressed concern about various uncertainties related to the substance, including contamination and safety of products. Lab testing was a key topic of discussion. A recent study analyzed delta-8 THC products such as vaporizers (vapes) and distillates for unknown impurities and found several substances in these products that were not listed on the certificate of analysis [33]. Indeed, a relatively large proportion of the posts we identified were made by bots deployed by moderators to help encourage careful consideration of products.

Health-related discussions about delta-8 THC exhibited nuance. Some posts about the substances referenced the use of delta-8 THC to treat health concerns, while others discussed the negative health effects of the consumption of delta-8 THC. For example, anxiety was a prevalent health issue discussed and some posts credited delta-8 THC for anxiety relief—which may indicate unmet health care needs—while other posts discussed exacerbation of anxiety from delta-8 THC [34,35].

Finally, it should be noted that one of the topics with the most posts compared delta-8 THC to other cannabinoids and related substances. For example, delta-8 THC products are considered by some people who use it as “less strong” than delta-9 THC; thus, people who use delta-8 THC products reported feeling comparably safer using them. Because people who use delta-8 THC may rely on Reddit to understand the nuances of this product compared to other novel cannabinoids, this finding further underscores the importance of accurate health-related information on social media platforms.

Strengths and Limitations

This study applies a robust, unsupervised topic model, BERTopic, to delta-8 THC–related Reddit posts to identify themes and topics. Unsupervised models perform cluster-based calculations to determine the spatial relationship of words resulting in clusters or topics. We used BERTopic as our topic model for several reasons. First, BERTopic is a topic modeling approach that combines techniques of early probability generative models such as latent Dirichlet allocation (LDA) with the use of semantic embeddings and includes dimension reduction of those embeddings for optimized performance. Additionally, BERTopic has been shown to outperform classic topic models such as LDA, nonnegative matrix factorization (NMF), and correlated topic models (CTM) when measured by topic coherence [15,36]. However, BERTopic may underperform in topic diversity or the percentage of unique words for all topics compared to CTM [15]. Finally, BERTopic is advantageous because of several options for fine-tuning the model with transformer-based state-of-the-art language models.

There are also several limitations to this study. First, Reddit data and user profiles do not contain geographical proxies at the state or local level. Thus, we could not determine if Reddit users resided in states where nonmedical adult use of cannabis is legal, which could give insight into whether adverse health effects related to the use of delta-8 THC products were mentioned more often in legal versus nonlegal states. Similarly, age data were not accessible for Reddit users; thus, inferences about users’ age and public perception of delta-8 THC products and use were not assessed. Nevertheless, studies reveal that individuals selling psychoactive substances do target teenagers and children via social media [37,38]. Furthermore, data from Reddit are not representative of the general population and therefore may not perfectly represent public interest in delta-8 THC. Finally, although BERTopic is a state-of-the-art topic modeling approach, no machine learning–based system is perfect at fully categorizing unstructured text. A reliance on human validation, at least partially, remains an important component of such work and although we manually examined a sample of posts from each topic, even human review is imperfect. It should also be noted that it is possible that some posts may contain more than 1 topical area of discussion. Identifying multiple topics within posts is not currently supported by the BERTopic algorithm and represents an area for future development of NLP models. However, the algorithm did optimize topic categorization by finding the ideal UMass coherence score, indicating that the topic selected was, to the best of our ability, the most suitable one assigned. Nonetheless, ongoing advances in NLP, such as increasingly sophisticated large language models and improved tuning of such models to relevant tasks and data, represent an important area for future work [39].

Conclusions

This study shows the use of web-based data for monitoring emergent public health concerns related to delta-8 THC. Even if emerging substances are detected early, a central question is whether the emerging substance carries the risk of harm to human health and what the nature of the health threat may be. These questions cannot be quickly answered from existing public health surveillance systems and thus novel data sources such as social media offer potential promise in gathering early insights. The fraction of social media posts that are health-related can be potentially used to gauge the magnitude of health-related concerns. Furthermore, examining the types of health concerns mentioned can inform further studies in clinical settings to obtain more definitive assessments of these potential health harms, particularly since the physiologic and psychoactive effects of a novel substance may not be fully known during periods of early emergence in the population. Second, even more mundane topical categories, such as the products and brands being discussed, can be useful for public health purposes. For example, during the response to the national outbreak of EVALI, a key question was whether certain products or brands were of greatest concern for lung injury. Finally, the topics identified can inform the development of public health communication and education materials. Our results revealed that users did have concerns about product safety and legitimacy and were exploring options to verify those factors in products they were considering using or purchasing. Thus, a better understanding of the discussion themes can help public health communication specialists be more responsive to the questions that are emerging. In summary, our study of a novel topic modeling approach highlights its potential for use in understanding evolving public health topics to help inform public health officials and respond to events in a timely manner.