This paper is in the following e-collection/theme issue:

Medicine 2.0: Social Media, Open, Participatory, Collaborative Medicine (1641) Infodemiology and Infoveillance (1349) Infoveillance, Infodemiology, Digital Disease Surveillance, Infodemic Management (1041) Smoking Cessation (557)

Published on in Vol 22, No 7 (2020): July

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/16962, first published .
Characterizing Social Media Messages Related to Underage JUUL E-Cigarette Buying and Selling: Cross-Sectional Analysis of Reddit Subreddits

Characterizing Social Media Messages Related to Underage JUUL E-Cigarette Buying and Selling: Cross-Sectional Analysis of Reddit Subreddits

Characterizing Social Media Messages Related to Underage JUUL E-Cigarette Buying and Selling: Cross-Sectional Analysis of Reddit Subreddits

Authors of this article:

Hejing Liu1, 2 Author Orcid Image ;   Qiudan Li1, 3 Author Orcid Image ;   Yongcheng Zhan4 Author Orcid Image ;   Zhu Zhang1, 3 Author Orcid Image ;   Daniel D Zeng1, 2, 3 Author Orcid Image ;   Scott J Leischow5 Author Orcid Image
Article Authors Cited by (15) Tweetations (5) Metrics

    Original Paper

    1The State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China

    2School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China

    3Shenzhen Artificial Intelligence and Data Science Institute (Longhua), Shenzhen, China

    4Orfalea College of Business, California Polytechnic State University, San Luis Obispo, CA, United States

    5College of Health Solutions, Arizona State University, Phoenix, AZ, United States

    Corresponding Author:

    Qiudan Li, PhD

    The State Key Laboratory of Management and Control for Complex Systems

    Institute of Automation

    Chinese Academy of Sciences

    95 Zhongguancun East Rd

    Beijing, 100190

    China

    Phone: 86 10 82449802

    Email: qiudan.li@ia.ac.cn


    Background: Stopping the epidemic of e-cigarette use among youth has become the common goal of both regulatory authorities and health departments. JUUL is currently the most popular e-cigarette brand on the market. Young people usually obtain and exchange information about JUUL with the help of social media platforms. Along with the rising prevalence of JUUL, posts about underage JUUL buying and selling have appeared on social media platforms such as Reddit, which sharply increase the risk of minors being exposed to JUUL.

    Objective: This study aims to analyze Reddit messages about JUUL buying and selling among the users of the UnderageJuul subreddit, and to further summarize the characteristics of those messages. The findings and insights can contribute to a better understanding of the patterns of underage JUUL use, and help public health officials provide timely education and guidance to minors who have intentions of accessing JUUL.

    Methods: We used a novel cross-subreddit method to analyze the Reddit messages on 2 subreddits. From July 9, 2017, to January 7, 2018, we collected data from the UnderageJuul subreddit, which was created for underage JUUL use discussion. The data set included 716 threads, 2935 comments, and 844 Reddit users (ie, Redditors). We collected our second data set, comprising 23,840 threads and 162,106 comments posted between July 9, 2017, and January 8, 2019, from the JUUL subreddit. We conducted analyses including the following: (1) annotation of users with buying/selling intention, (2) posting patterns discovery and topic comparison, and (3) posting activeness observation of discovered Redditors. Term frequency–inverse document frequency and regular expression-enhanced keyword search methods were applied during the content analysis to extract the posting patterns. The public posting records of the discovered users on the JUUL subreddit during the year after the UnderageJuul subreddit was shut down were analyzed to determine whether they were still active and interested in obtaining JUUL.

    Results: Our study revealed the following: (1) Among the 716 threads on the UnderageJuul subreddit, there were 214 threads related to JUUL sale and 168 threads related to JUUL purchase, which accounted for 53.5% (382/714) of threads. (2) Among the 844 Redditors of the UnderageJuul subreddit, 23.82% (201/844) of users were annotated with buying intention, and 21.10% (178/844) of users were annotated with selling intention. There were 34 users with buying/selling intention that self-reported as being <21 years old. (3) The most common key phrases used in selling threads were “WTS,” “want to sell,” “for sale,” and “selling” (154/214, 72.0%). The most common key phrases used in buying threads were “look for/get JUUL/pods” (58/168, 34.5%) and “WTB” (53/168, 31.5%). (4) The most important concern that UnderageJuul Redditors had in obtaining JUULs was the price (311/1306, 23.81%), followed by the delivery service (68/1306, 5.21%). (5) The most popular flavors among the users with buying/selling intention were mango, cucumber, and mint. The flavor preferences remained consistent on both subreddits. Adverse symptoms related to the mango flavor were reported by 3 users on the JUUL subreddit. (6) In total, 24.4% (49/201) of users wanted to buy JUULs and 46.6% (83/178) of users wanted to sell JUULs, including 11 self-reported underage users, who also participated in the discussions on the JUUL subreddit. (7) Within one year of the UnderageJuul subreddit shutting down, there were 40 users who continued to post 186 threads on the JUUL subreddit, including 10 threads indicating buying/selling willingness that were posted shortly after the UnderageJuul subreddit was closed.

    Conclusions: There were overlapping users active in the JUUL and UnderageJuul subreddits. The buying/selling-related content appeared in multiple venues with certain posting patterns from July 9, 2017, to January 7, 2018. Such content might lead to a high risk of health problems for minors, such as nicotine addiction. Based on these findings, this study provided some insights and suggestions that might contribute to the decision-making processes of regulators and public health officials.

    J Med Internet Res 2020;22(7):e16962

    doi:10.2196/16962

    Keywords

    JUUL; e-cigarette; Reddit; cross-sectional analysis; electronic nicotine delivery system; underage JUUL use 


    The use of Electronic Nicotine Delivery Systems (ENDS; also called “e-cigarettes”) has been increasing rapidly among youth. In 2019, a cross-sectional survey with 19,018 participants showed the prevalence of self-reported current e-cigarette use was 27.5% among US high school students and 10.5% among middle school students [1]. At present, JUUL (by JUUL Labs) is the most popular brand, with more than 70% of the market share of e-cigarettes [2]. The appealing flavors and its relaxing effect cause potential intention and actual use among nonsmoking teenagers [3-5]. In addition, because of the lax regulatory environment in the United States, unregulated constituents in e-liquids as well as high nicotine delivery have led to concerns about addiction potential and pulmonary risks from the use of ENDS products [6-8]. To protect the physical health of young people, in April 2018, the Food and Drug Administration (FDA) issued a statement on preventing teenagers from using and buying JUUL and similar products [9]. As the FDA indicated [9], JUUL was more difficult for parents and teachers to recognize and detect than other e-cigarette products. Therefore, increased efforts to strengthen the regulation of minors' access to JUUL have been implemented.

    JUUL’s marketing heavily relied on social media platforms like Twitter and Instagram [10]. As a result, young people read and spread information about JUUL with the help of social media platforms [11]. Common JUUL discussions on social media included flavor preference [12-14], use experience [12-16], use location [12,14], purchasing methods [17], etc. Allem et al [12] found that JUUL use occurred in school locations, such as classrooms, bathrooms, libraries, and gyms. Brett et al [14] found teenagers used JUUL because the use of this product was part of a popular trend and to feel the “buzz” (ie, a feeling caused by nicotine). Multiple researchers have noted that adolescents prefer the mango and mint flavors [14,17,18]. Li et al [19] found that the fruit flavor was associated with multiple adverse symptoms, including cough, headache, and lung tightness, though the causal relationship is unclear. With the rise of youth exposure to JUUL, Zhan et al [17] analyzed their purchasing methods. They found that peer-to-peer purchase and shipping was the most common method for UnderageJuul subreddit users.

    As an online forum that allows users to create communities, Reddit has multiple sections (also called “subreddits”) related to JUUL discussion. Previous research determined that underage Redditors post about their JUUL use on Reddit [20]. To better understand and surveil youth JUUL use, researchers have conducted analyses on Reddit data focusing on underage JUUL users and their posting content, such as topic discovery [14,15], age and location distribution [17], popular flavor discovery [15,17], and negative symptoms analysis [19].

    Previously, there was a public subreddit on Reddit called UnderageJuul, which was dedicated to discussing underage JUUL use. This subreddit was created in July 2017 and had almost 1000 users at its peak. Right before it was shut down in January 2018, the UnderageJuul subreddit was gaining 8 new members per day [20,21]. Previous studies conducted qualitative analysis on the UnderageJuul subreddit and found that posts in this subreddit included content such as self-reported underage JUUL users seeking sellers, sale advertisements without age verification, and adult users buying for minors to earn money, researchers thought such trade-related content indicated that minors had increased access to JUUL, and there might be other similar communities on social media platforms [17,20,21]. However, the characteristics of the buying/selling-related messages among the users of the UnderageJuul subreddit have not been further identified.

    This study aimed to analyze the content on the UnderageJuul and JUUL subreddits that indicated JUUL buying and selling, which provides a more comprehensive understanding of underage JUUL use. Specifically, this study was designed to address the following questions:

    First, what are the posting patterns of messages related to JUUL buying and selling on the UnderageJuul subreddit? The “posting patterns” refer to the common key phrases, way of expression, variants, or abbreviations the Redditors used when publishing buying/selling-related posts. For instance, Redditors usually use “look for/need + juul/pods” or “WTB” (short for “want to buy”) to express their buying willingness. Discovering users’ posting patterns could enable researchers to extract the core features of this kind of message, and help regulators continue to identify similar content on social media, thus contributing to the prevention of minors’ access to JUULs.

    Second, what are the concerns of the users of the UnderageJuul subreddit when they want to buy or sell JUULs? While UnderageJuul users had a willingness to buy and use JUULs, they also expressed concerns regarding JUULs, such as the choice of flavors. Given this subreddit was created to discuss underage JUUL use, identifying the concerns of its users has the potential to uncover specific characteristics of minors or others using that subreddit who intend to buy or sell JUUL. Understanding the topics and concerns discussed in this subreddit could be useful for health departments or educators responsible for the development and implementation of education programs on potential ENDS risk, as well as for regulatory agencies responsible for developing media and educational programs for young people.

    Third, will users of the UnderageJuul subreddit cross-publish their comments, questions, and concerns on multiple subreddits at the same time? Will they continue to be active after the UnderageJuul subreddit is closed? We adopted the evaluation of “posting activeness” to address these questions, which involves counting the total number of different posts by users [22]. We chose the JUUL subreddit (ie, the general JUUL discussion subreddit) as a contrast. The posting activeness across subreddits will be compared during two time periods: the time period when UnderageJuul was accessible, and the year after UnderageJuul was shut down. This analysis could provide a deeper understanding of the buying and selling behavior of UnderageJuul users, including the extent to which shutting down one social media channel impacts information-sharing on other platforms, which could have significance for both understanding the breadth and scope of social media networks and the implementation of social media–based educational programs.

    By answering the 3 questions above, we hope to provide valuable information to regulators to help them establish practices and policies to minimize adolescent e-cigarette use and safeguard their health. Note that the user analysis in this study was for the users who had public posting records on the UnderageJuul subreddit. The ages of underage users were based on their self-reported age information.


    Overview

    We applied a novel cross-subreddit method to analyze the content related to JUUL buying and selling, with a focus on comparing those using the UnderageJuul subreddit (which was originally designed for young people to discuss the use of JUULs, but soon became a platform to explore and share ways to illegally obtain JUULs) with those using the general JUUL subreddit (which is the main Reddit channel for discussing general JUUL-related topics). Figure 1 shows the framework for our cross-subreddit analyses. It consists of three components: data collection; cross-subreddit content and posting activeness analysis; and results.

    Data Collection

    The data used in this research consisted of two parts: data from the UnderageJuul subreddit and data from the JUUL subreddit.

    UnderageJuul Subreddit Data Collection

    Since the UnderageJuul subreddit was shut down by the time we began our analysis, we collected data from the UnderageJuul subreddit through the application programming interface (API) provided by pushshift.io [23], which is a website that stores all publicly available Reddit threads and comments. The UnderageJuul subreddit was created on July 9, 2017, and was shut down on January 8, 2018. We obtained the complete UnderageJuul subreddit posting and user information data set from that period, including 716 threads, 2935 comments, and 844 Redditors.

    JUUL Subreddit Data Collection

    We collected data from the JUUL subreddit via Reddit’s API [24] from July 9, 2017, to January 8, 2019, which covered two continuous time periods: (1) the same time period of the UnderageJuul data set (from July 9, 2017, to January 7, 2018), and (2) the year after the end of UnderageJuul activities (from January 8, 2018, to January 8, 2019). The first data set (ie, the JUUL-overlap data set) contained 3978 threads, 31,589 comments, and 5785 Redditors. The second data set (ie, the JUUL-afterwards data set) contained 19,862 threads, 130,517 comments, and 18,460 Redditors. Table 1 shows a detailed description of these three data sets.

    Note that the data we used in this research were all publicly available on Reddit, and no personal information (eg, account profile) was included. The usernames contained in these data sets are not presented in this paper, for the privacy protection of the Redditors involved in this study.

    ‎
    Figure 1. Framework for cross-subreddit analyses on Reddit. API: application programming interface.
    View this figure
    Table 1. Data set description.
    Data setTime spanThreads, nComments, nRedditors, nThreads per Redditor, nComments per Redditor, n
    UnderageJuul07/09/2017-01/07/201871629358441.954.11
    JUUL-overlap07/09/2017-01/07/2018397831,58957852.455.95
    JUUL-afterwards01/08/2018-01/08/201919,862130,51721,1862.497.07

    Data Analysis

    Overview

    The data analyses consisted of three steps. The first step was to annotate the users on the UnderageJuul subreddit that indicated buying/selling intention. Second, the text features were extracted to discover the posting patterns in their posting content. Finally, the activeness of the Reddit accounts was analyzed. We used a cross-subreddit analysis method in this study, thus the results of each step were all compared with the content on the JUUL subreddit.

    User Analysis Across Subreddits
    Overview

    We first set the rules for annotation, then manually labeled Redditors who wanted to buy or sell JUUL on the UnderageJuul subreddit. Based on the results of user annotation on the UnderageJuul subreddit, the overlapping users were detected to discover how many of these users also participated in the JUUL subreddit during the same time period.

    Step 1: User Annotation on the UnderageJuul Subreddit

    First, two annotators labeled the users who wanted to buy or sell JUULs according to the descriptions in Table 2. Users interested in buying or selling usually post a specialized thread to attract sellers or to advertise the product to be sold, so we went through all threads on UnderageJuul subreddit. For example, when a Redditor posted a thread saying “[WTB] Looking to buy blue Juul and cucumber pods for under $75,” he/she was labeled as a user who wanted to buy JUULs; when a Redditor posted a thread saying “I am selling Juul warranty codes, …, 25$ per code, 20$ per code (if you buy 3 or more at once),” he/she was labeled as a user who wanted to sell JUULs. The threads were also tagged with corresponding labels. A Redditor could be annotated as a user who wanted to buy JUULs as well as a user who wanted to sell JUULs, if he/she posted multiple threads indicating both buying and selling intentions. Then, we annotated all comments below these labeled threads to discover users who did not post themselves but replied to those threads and showed buying/selling willingness. The Cohen κ coefficient of the annotation from two annotators was 0.83. We next extracted age information from the posts of these users to discover underage JUUL users. Since the US federal minimum age for sale of tobacco products had been raised from 18 to 21 years (ie, the Tobacco 21 legislation [25]), we labeled Redditors who self-reported that they were under 21 years old as the self-reported underage users. We used regular expressions to extract phrases like “# years old” or “I’m #” to detect age information of corresponding users. Some users did not mention a specific age but had expressions like “I’m underage” in their posts. These users were also labeled as self-reported underage users.

    Table 2. Annotation categories and description.
    LabelDescription
    User who wanted to buy JUULsUser who wanted to purchase JUULs peer-to-peer or face-to-face, asked others to buy JUULs for him/her, or wanted to buy a JUUL replacement code to get JUULs.
    User who wanted to sell JUULsUser who wanted to sell JUULs to others through shipping or face-to-face trade, buy JUUL for others for a fee, or sell JUUL replacement codes.
    Self-reported underage userUser who wanted to buy or sell JUULs and had a self-report age under 21 years old or a description indicating underage.
    Step 2: Overlapping User Analysis on the JUUL Subreddit

    The username is the unique identifier for each Redditor [26]. We compared the public posting records of these users on the two subreddits UnderageJuul and JUUL (ie, the JUUL-overlap data set). Redditors with the same usernames were regarded as overlapping users.

    Content Analyses and Comparison
    Overview

    The following content analyses consisted of two steps. First, we extracted the common key phrases and words from the user-generated content to summarize the regular posting patterns. Second, we compared the overlapping users’ posts between the UnderageJuul subreddit and the JUUL subreddit.

    Step 1: Posting Patterns Extraction on the UnderageJuul Subreddit

    First, the textual features needed to be extracted from users’ posts. To achieve this, we used term frequency–inverse document frequency to uncover important words in each post automatically. Based on the frequency of these important words, we adopted regular expression-enhanced keyword search [27] on each post to discover key phrases. The trigrams near the important words were extracted from the buying/selling-related threads. The key phrases were determined according to the frequency rank and whether the expression was complete. For example, the key phrase “need juuls/pods” was determined because it was a complete expression and had a top rank in the search results of the word “need.” We manually classified the important words and key phrases into several posting patterns based on semantics and phrase structure.

    Step 2: Content Comparison on the JUUL Subreddit

    We compared the user-generated contents of overlapping users between the JUUL and UnderageJuul subreddits to discover whether these users also published trade-related content on the JUUL subreddit, and whether their concerns remained consistent. For the first perspective, each thread posted by the overlapping users on the JUUL subreddit was automatically labeled if it implied a willingness to buy or sell JUULs using the posting patterns discovered in Step 1. For the second perspective, we conducted a qualitative content analysis on users’ posts on both subreddits. To acquire more detailed and specific concerns, our analysis was applied to all the comments under trade-related threads on the UnderageJuul subreddit, as well as all the content they posted on the JUUL subreddit (ie, threads and comments). We used the inductive category development method [28] to generate the topics based on the posts’ semantics and JUUL background information. For example, expressions like “v1” and “v2” represented versions of JUUL device. When they were identified in the posts, they were categorized with other words (eg, “leak,” “refill”) that were previously found to be related to JUUL devices, regarded as the “device description” topic. Because flavor is one of the most important factors in ENDS products that appeal to teenagers, we specifically counted the occurrences of JUUL’s seven flavors (ie, mango, mint, cucumber, menthol, fruit, crème, and tobacco) among the discussions of these users to discover their preference.

    Posting Activeness Observation

    Finally, the online posting activeness of these users on the JUUL subreddit was observed during the year after the UnderageJuul subreddit was shut down. We analyzed the posting activeness of users to find whether they continued to express a willingness to buy or sell in other communities after the UnderageJuul subreddit was closed. Specifically, we searched through the JUUL-afterwards data set to count the number of threads and comments posted by these users, recorded their posting dates to get the posting trend changes, and analyzed their posts. Thus, we manually checked their content to determine whether their posts showed the intention to buy/sell JUUL.


    User Analyses Across Subreddits

    Among 844 Redditors on the UnderageJuul subreddit, we discovered 201 users who wanted to buy JUULs, 178 users who wanted to sell JUULs, and 45 users with both intentions. Among the 334 discovered users, there were 34 users that were self-reported underage. Of these, 30 indicated a willingness to purchase and 4 were willing to sell JUUL. Table 3 shows the proportion of each self-reported age group. The “teenager, age unknown” group represents users that did not mention a specific age but used expressions like “I’m underage” in their posts.

    Table 3. Groups of discovered users with self-reported age.
    User age (years)Number of discovered users
    ≤155
    16-1814
    19-213
    Teenager, age unknown5

    After analyzing the users across subreddits, we found 49 of 201 (24.4%) users who wanted to buy JUULs and 83 of 178 (46.6%) users who wanted to sell JUULs also participated in discussions on the JUUL subreddit. Among the overlapping users, 11 underage users (8 users with buying willingness and 3 users with selling willingness) were discovered. Table 4 shows the detailed statistics of the overlapping users.

    We also counted the threads and comments posted by the discovered users on each subreddit. They posted 382 threads in the UnderageJuul subreddit and there were 1306 comments under their threads. In addition, they also posted 123 threads and 1600 comments in the JUUL subreddit. Table 5 shows the detailed statistics of their posts. We ranked the activeness of all overlapping users according to their posting amount. The most active self-reported underage user published 16 threads and 96 comments on the JUUL subreddit, as well as 1 thread and 44 comments on the UnderageJuul subreddit. There were 128 Redditors that posted 182 replies to this user.

    Table 4. Statistics of the discovered users and overlapping users.
    User typesUnderageJuul subreddit, n/N (%)JUUL-overlap subreddit, n/N (%)
    Users who wanted to buy JUULs201/844 (23.8)49/201 (24.4)
    Users who wanted to sell JUULs178/844 (21.1)83/178 (46.6)
    Self-reported underage users34/334 (10.2)11/34 (32.4)
    Table 5. Statistics of posts from discovered users and overlapping users.
    Data setUsers who wanted to buy JUULsUsers who wanted to sell JUULs

    		Threads, n/NComments, n/NTotal, nThreads, n/NComments, n/NTotal, n
    UnderageJuul168/716180/1306348214/716238/1306452
    JUUL-overlap39/3978447/31,58948684/31,5891153/31,5891237

    Content Analyses and Comparison

    Posting Patterns Analysis

    We summarized 10 patterns observed in the threads posted by the users discovered on the UnderageJuul subreddit. Table 6 presents each pattern and its ratios. Since one thread may contain multiple key phrases, the total post numbers of the key phrases groups are greater than the total number of posts mentioned in Table 5.

    The posting patterns in the threads from users who wanted to buy JUULs all expressed buying willingness directly. They usually included “looking for juul/pods” (58/168, 34.5%) and “want to buy (WTB)” (53/168, 31.5%) in the title, then described the buying requests in the body, such as the flavor and the location. An example is “Need Pods In Orlando Area. Just an underaged kid looking for pods in the Orlando area any flavor.” The body of these threads also contained the key phrases like “need juul/pods” or “need help for getting juul/pods” (36/168, 21.4%). Alternative expressions were also used to show buying willingness, such as “anyone selling juul/pods” and “anyone know seller in somewhere” (28/168, 16.7%).

    Table 6. Regular posting patterns of discovered users on the UnderageJuul subreddit.
    User type, topic number, and key phrasesPost, n (%)
    Users who wanted to buy JUULs (n=168)

    		1WTBa, want to buy, buy juul/pods53 (31.5)

    		2Need juul/pods, need help, need seller36 (21.4)

    		3Look for juul/pods, get juul/pods58 (34.5)

    		4Anyone selling, anyone know seller28 (16.7)

    		5Other57 (34.0)
    Users who wanted to sell JUULs (n=214)

    		1WTSb, want to sell, for sale, selling154 (72.0)

    		2Ship, discreet97 (45.3)

    		3Charger, refill, starter55 (25.7)

    		4Sealedc, unboxing, unopen, brand new40 (18.7)

    		5Other39 (18.2)

    aWTB: want to buy.

    bWTS: want to sell.

    c“Sealed” indicates that the product is new and has not been opened.

    The posting patterns in the threads from users who wanted to sell JUULs included multiple kinds of information. They usually contained key phrases such as “WTS,” and “something for sale” (Pattern 1, 154/214, 72.0%) in the title to draw attention. Pattern 2 (97/214, 45.3%) was about shipping services, such as the shipping scope and discretion. Pattern 3 (55/214, 25.7%) involved a description of the product, including the accessories. If the product was brand-new, that was emphasized (40/214, 18.7%). For instance, a user who wanted to sell JUULs posted the following thread: “[WTS] JUUL V3 Starter Kit $40 in BTC. Brand new, sealed starter kit. Will ship quickly and discreetly.”

    Based on the above posting patterns, we further analyzed the threads posted by the overlapping users on the JUUL subreddit and found 17 threads with buying purpose and 18 threads with selling purpose.

    Content Comparison

    We summarized the 15 topics discussed by the discovered users in the UnderageJuul and JUUL subreddits. Table 7 shows the 10 topics discussed in the UnderageJuul subreddit and the 5 topics discussed in the JUUL subreddit.

    Table 7. Topic comparison of the UnderageJuul and JUUL subreddits.
    Subreddit, topic number, and topicKey phrasesPosts, n (%)
    UnderageJuulsubreddit (n=1306)

    		1PricePrice, how much, $number311 (23.81)

    		2ContactPMa, message63 (4.82)

    		3ShipShip, shipping68 (5.21)

    		4ScamScammer, scam, fake83 (6.36)

    		5Product descriptionv1, v2, v3b, refill53 (4.06)

    		6Purchasing methodeBay, website, .com, store20 (1.53)

    		7Paying methodGift card, PayPal, Bitcoin, BTC40 (3.06)

    		8StockStill available, still selling, how many, still have, SOLD15 (1.15)

    		9FlavorMango, mint, fruit, cucumber, crème, menthol, tobacco27 (2.07)

    		10OtherPosts did not contain the words above626 (47.93)
    JUULsubreddit (n=123)

    		1Buying/sellingThis was annotated manually.35 (28.46)

    		2FlavorMango, mint, fruit, cucumber, crème, menthol, tobacco16 (13.01)

    		3Product descriptionLeak, refill, charged, help, real, fake, charger, battery25 (20.33)

    		4Experience sharingBuzz, cough, lung9 (7.32)

    		5OtherPosts did not contain the words above38 (30.90)

    aPM: private message.

    bv1, v2, and v3 represent the different versions of JUUL devices.

    In the UnderageJuul subreddit, discussions were primarily focused on buying and selling JUULs. Topic 1 (311/1306, 23.81%) refers to users raising questions about the price. The replies usually contained specific prices in the form of $number. Topic 2 (63/1306, 4.82%) refers to posts in which authors would ask others to contact them through private messages if they did not want to directly answer questions in public replies. Topic 3 (68/1306, 5.21%) was about shipping services. Topic 4 (83/1306, 6.36%) includes posts in which some users commented under the threads with selling purposes that the post author is a scammer. Topic 5 (53/1306, 4.06%) refers to descriptions of the selling product, including the version of the product and whether it was refilled. Topic 6 (20/1306, 1.53%) and Topic 7 (40/1306, 3.06%) refer to the methods of purchasing and paying. Since some users self-claimed they were vendors from eBay or other websites, they accepted payment methods such as VISA gift cards and PayPal. Some users offered to buy JUULs from stores for underage users for a fee. Bitcoin was sometimes accepted as a form of payment. Topic 8 (15/1306, 1.15%) is related to users asking post authors about their stock. The post authors might update the threads with “SOLD” to indicate that they had sold out. Topic 9 (27/1306, 2.07%) represents the flavors discussed in the threads and comments; this topic is analyzed in detail in the next section.

    In the JUUL subreddit, Topic 1 (35/123, 28.46%) represented the threads about buying and selling JUULs, which suggested the most popular topic discussed by these users did not change on this subreddit. Topic 2 (16/123, 13.01%) was about the flavors discussed by the overlapping users. The flavor topic is also analyzed individually in the next section. However, the focus of this topic was different from that on the UnderageJuul subreddit. Topic 3 (25/123, 20.33%) referred to the product descriptions. Unlike the product description topic (Topic 5) on the UnderageJuul subreddit, the discussions about this topic on the JUUL subreddit were about describing the device’s problems and seeking help. For instance, a post about this topic stated, “… All my official juul pods keep leaking out all over my juul. Idk what's going on.” Topic 4 (3/123, 2.44%) was about negative experience sharing. For example, one user said, “I used the juul to stop smoking but have noticed a chronic cough starting to begin.”

    It can be seen that access to JUUL was a common topic for the discovered users in both subreddits. However, the content in the UnderageJuul subreddit included more details about JUUL transactions. The flavor topic and the product description topic appeared on both subreddits. The focal point in the UnderageJuul subreddit was users describing their buying/selling demands, while in the JUUL subreddit, users commonly shared their JUUL use experience.

    We identified the 10 users who posted the most for each topic as the active users. In the UnderageJuul subreddit, there were 16 users who were active in multiple topics. There were 4 active users who were active in 3 topics at the same time. The most active user was in the top 10 among 6 topics. In the JUUL subreddit, there were 6 users active in multiple topics. The most active user posted about 4 topics at the same time. Notably, there were 7 users who were active in both subreddits, and the topics they promoted were related to JUUL buying and selling.

    Flavor Analysis and Comparison

    We counted the word frequency of the 7 flavors mentioned above and compared their popularity in the two subreddits. Among 382 threads and 1306 comments in the UnderageJuul subreddit, mango (n=50) was the most popular flavor, followed by cucumber (n=39) and mint (n=31). Fruit (n=9), tobacco (n=7), menthol (n=5), and crème (n=5) were not mentioned frequently. These flavors were mentioned when users described the selling products or the buying requests, as shown in the two examples given here:

    Pods Available: I have 8 tobacco, 16 cucumber, 8 mint, 12 menthol, 8 mango, and 4 fruit. …
    … Looking for 2 packs of mango pods let me know if you have any for sale.

    Among the 123 threads in the JUUL subreddit, there were 16 threads that mentioned flavor. The most popular flavor was mango (n=8), followed by cucumber (n=7) and mint (n=7). These results were similar to those found in the UnderageJuul subreddit, which indicates that users’ flavor preferences did not change between subreddits. There was not much discussion of the crème (n=2), menthol (n=2), tobacco (n=2), and fruit (n=1) flavors. One example of a flavor topic post is as follows:

    Just got my hands on a couple packs of cool cucumber. …, it's really good and up there with mango.

    However, we found 3 reports of adverse symptoms among the 8 threads that mentioned the mango flavor in the JUUL subreddit. Two users reported mouth burning and one user reported a stomachache, as shown below:

    …when ever I hit a mango pods my stomach almost immediately starts hurting.

    Posting Activeness Observation

    We observed the posting activeness of all labeled users in the year following the shutdown of the UnderageJuul subreddit (January 8, 2018, to January 8, 2019). Table 8 shows the number of labeled users who had posted during this time period, and Table 9 shows the number of posts they made. For both tables, values were summed across consecutive 2-month periods.

    Table 8. The number of users who posted messages in the year after the UnderageJuul subreddit was removed.
    Time periodUsers who wanted to buyUsers who wanted to sellSelf-reported underage users
    1/8/2018-3/7/201811124
    3/8/2018-5/7/2018580
    5/8/2018-7/7/2018210
    7/8/2018-9/7/2018121
    9/9/2018-11/7/2018060
    11/8/2018-1/9/2019231
    Table 9. The number of posts by the labeled users in the year after the UnderageJuul subreddit was removed.
    Time periodPosts by users who wanted to buyPosts by users who wanted to sellPosts by self-reported underage users
    1/8/2018-3/7/2018344817
    3/8/2018-5/7/201812240
    5/8/2018-7/7/2018630
    7/8/2018-9/7/2018131
    9/9/2018-11/7/20180230
    11/8/2018-1/9/20194282

    In the first 4 months after the UnderageJuul subreddit was shut down, the discovered users were still active in the JUUL subreddit. They posted 10 threads with the purpose of buying or selling. In the next 2 months, these users were less active and did not post threads with buying or selling willingness. One possible reason was that the administrators of the JUUL subreddit started to prohibit posts about trading JUULs. In the second half of the year, the discussions changed to flavor and device-related topics. There were 2 users who expressed the intention of quitting JUULs because of nicotine addiction and the high price.


    Principal Findings

    This paper used a cross-subreddit method to analyze underage JUUL use. Based on our previous study [17], this paper further analyzed the content related to underage JUUL buying and selling on Reddit, and summarized its characteristics and patterns. Additionally, this paper analyzed the posting activeness of the discovered users in the JUUL subreddit after long-term observation.

    In summary, we found 214 threads related to JUUL sale and 168 threads related to JUUL purchase on the UnderageJuul subreddit, which accounted for 53.5% (382/714) of the UnderageJuul subreddit’s threads. In addition, we found that these threads were posted with certain regular word-level patterns. The most common key phrases used in selling-related threads were “WTS,” “want to sell,” “for sale,” and “selling” (154/214, 72.0%). The most common key phrases used in buying-related threads were “look for/get JUUL/pods” (58/168, 34.5%) and “WTB” (53/168, 31.5%). The FDA has already announced policies about preventing youth use of and access to JUUL e-cigarettes, and reducing the marketing and promotion of tobacco products toward minors [29,30]. Though the UnderageJuul subreddit has been removed, there might be other similar communities that support underage JUUL trading. The posting patterns could help discover this type of content among other communities on Reddit and different social media platforms such as Twitter and Instagram. Once such content is discovered, a link to the official educational page about the tobacco products’ adverse health effects could be automatically added below such posts or sent to the content publisher via private message. This could have implications for interventions to reduce youth ENDS use.

    We have demonstrated that discussions about obtaining and using JUUL e-cigarettes occur in multiple venues, and not just the subreddit dedicated to underage JUUL use (UnderageJuul). Among the 844 Redditors of the UnderageJuul subreddit, 23.82% (201/844) of users were annotated with buying intention, and 21.10% (178/844) of users were annotated with selling intention. We found that 24.38% (49/201) of users who wanted to buy JUULs and 46.63% (83/178) of users who wanted to sell JUULs had posting activeness on the JUUL subreddit. The results are consistent with our earlier research [17] regarding the approaches of illegally obtaining JUUL products. The buying/selling topics were common concerns for users on both subreddits during the same time span. This indicates the people who want to access to JUULs may not express their desires on only one platform and may seek multiple resources. When regulatory organizations take action to reduce youth’s access to JUUL and similar products, they might need strategies on multiple social media platforms simultaneously. When selling ENDS products via online shopping sites and other vaping-related websites, necessary warning statements should be clearly presented alongside, including the health risks and nicotine strength [31].

    In our cross-subreddit analysis, we developed a procedure to evaluate the effect that the discovered users had based on their posting activeness and user-generated content. The most active self-reported underage user we discovered had published content that at least 128 Redditors read about, so there was a social network effect that could impact youth use of ENDS products like JUUL. Evaluation of JUUL-related content posted by those active users on social media platforms, especially the underage ones, is highly relevant to educators and those involved in tobacco regulation. In addition, it is not clear from these analyses whether these active users are paid by vendors or companies for advertising tobacco products to minors, and this is an important analysis that should be conducted to understand the motivation of those who post about these products on social media.

    Our findings suggest that price was what the UnderageJuul users most cared about, along with the delivery services. The mango, mint, and cucumber flavors were the favorite flavors among these users. In January 2018, the UnderageJuul subreddit was banned. We found there was a self-reported underage user that posted a message on the JUUL subreddit asking for an alternative to the mango flavor, such as mint. In addition, we discovered negative health symptoms were reported on the JUUL subreddit after the use of the mango flavor. These popular flavors are still available in many countries and young people continue to discuss ways to acquire ENDS products like JUUL on Twitter, Reddit, etc. Despite regulatory and company efforts to reduce access to flavored ENDS products [32], considerable research is still needed on the role of flavors regarding the addiction potential of ENDS products [13,18,19,33]. In particular, since social media can provide timely and rapid information on what is being discussed about a particular ENDS product, continued analysis of social media is an essential early warning surveillance method for identifying addiction and the health risks of JUUL and other products (eg, the NJOY Ace product that has steadily increased in sales).

    In our posting activeness observation, we found that in the year after the UnderageJuul subreddit was removed, 40 users (including 4 self-reported underage users) continued to post 186 threads in the JUUL subreddit. There were 10 threads with buying/selling willingness during the first 2 months. This result indicates that shortly after the UnderageJuul subreddit was closed, users transferred to other subreddits to have discussions, and continued to publish messages about buying/selling JUUL pods through social media platforms.

    In conclusion, this study highlights the need for continued research on how social media can become a more fundamental component of the tobacco and nicotine surveillance system. In particular, rapid and extensive analyses of social media can serve as an early warning system of the rapid rise in the use of new products and health problems that are occurring as a result of using those products.

    Limitations and Future Work

    Since the data we used for analysis was user-generated public information (ie, post, publication date, and username), there were many limitations for us regarding determining the authenticity and legitimacy of users’ claims like self-reported age, their stock of JUUL devices, and the completion of trading. As the UnderageJuul subreddit was built specifically for discussing underage JUUL use, we thought the age distribution analysis was valuable and necessary. In addition, we noticed the scam topic (ie, Topic 4 on the UnderageJuul subreddit on Table 7) during the content analysis, which indicates some Redditors who claimed to have JUULs to sell might post such content to scam other users, rather than to trade JUULs. This might cause discrepancies for regulators when trying to identify actual buyers and sellers.

    In addition, the conclusions and insights of our study were based on content analysis of the two specific subreddits during the specific time span. Our findings might not be generalizable to other subreddits or other time periods, considering the different barrier policies being carried out. However, future studies could be extended to other subreddits that are about JUULs and other social media platforms (eg, Twitter) with the same methodology and mechanism.

    Conclusion

    This is the first study to investigate the patterns of buying and selling among underage JUUL users by utilizing information from multiple connected online forums. We hope these findings can be conducive to timely guidance and education of underage JUUL users, thus protecting the health of young people.

    Acknowledgments

    This work was partially supported by the National Key R&D Program of China under grant number 2016QY02D0305, the National Institutes of Health (NIH) grant 5R01DA037378-05; the National Natural Science Foundation of China under grant numbers 61671450, 71621002, 71902179, and 71702181; the Key Research Program of the Chinese Academy of Sciences under grant number ZDRW-XH-2017-3, and the Early Career Development Award of the State Key Laboratory for Management and Control of Complex Systems (SKL-MCCS) under grants 20190212 and 20190204. Research reported here was supported by the National Center Institute (NCI) and FDA Center for Tobacco Products (CTP). The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH or the FDA.

    Authors' Contributions

    HL, QL, YZ, ZZ, DZ, and SL conceived the idea for this study. HL and QL designed the study. HL, QL, and YZ collected the data, conducted the data analysis, and drafted the manuscript. ZZ, DZ, and SL provided critical feedback and revised the manuscript. All authors read and approved the final manuscript.

    Conflicts of Interest

    SL is the co-owner and editor-in-chief of the journal Tobacco Regulatory Science. All other authors declare no conflicts of interest.

    1. Cullen KA, Gentzke AS, Sawdey MD, Chang JT, Anic GM, Wang TW, et al. e-Cigarette Use Among Youth in the United States, 2019. JAMA 2019 Nov 05;322(21):2095 [FREE Full text] [CrossRef] [Medline]
    2. King BA, Gammon DG, Marynak KL, Rogers T. Electronic Cigarette Sales in the United States, 2013-2017. JAMA 2018 Oct 02;320(13):1379-1380 [FREE Full text] [CrossRef] [Medline]
    3. Vaporized: Youth and young adult exposure to e-cigarette marketing. Truth Initiative. 2019.   URL: https:/​/truthinitiative.​org/​sites/​default/​files/​media/​files/​2019/​03/​Vaporized-Youth-Exposure-To-E-Cigarette-Marketing.​pdf [accessed 2020-01-28]
    4. Alexander JP, Williams P, Lee YO. Youth who use e-cigarettes regularly: A qualitative study of behavior, attitudes, and familial norms. Prev Med Rep 2019 Mar;13:93-97 [FREE Full text] [CrossRef] [Medline]
    5. Pepper JK, Ribisl KM, Brewer NT. Adolescents' interest in trying flavoured e-cigarettes. Tob Control 2016 Nov 15;25(Suppl 2):ii62-ii66 [FREE Full text] [CrossRef] [Medline]
    6. Barrington-Trimis JL, Samet JM, McConnell R. Flavorings in electronic cigarettes: an unrecognized respiratory health hazard? JAMA 2014 Dec 17;312(23):2493-2494 [FREE Full text] [CrossRef] [Medline]
    7. Centers for Disease Control and Prevention. Severe Pulmonary Disease Associated with Using E-Cigarette Products. Health Alert Network. 2019.   URL: https://emergency.cdc.gov/han/han00421.asp [accessed 2019-09-30]
    8. Flower M, Nandakumar L, Singh M, Wyld D, Windsor M, Fielding D. Respiratory bronchiolitis-associated interstitial lung disease secondary to electronic nicotine delivery system use confirmed with open lung biopsy. Respirology Case Reports 2017 Apr 03;5(3):e00230. [CrossRef]
    9. FDA. Statement from FDA Commissioner Scott Gottlieb, M.D., on new enforcement actions and a Youth Tobacco Prevention Plan to stop youth use of, and access to, JUUL and other e-cigarettes. 2018.   URL: https:/​/www.​fda.gov/​news-events/​press-announcements/​statement-fda-commissioner-scott-gottlieb-md-new-enforcement-actions-and-youth-tobacco-prevention [accessed 2019-09-30]
    10. Huang J, Duan Z, Kwok J, Binns S, Vera LE, Kim Y, et al. Vaping versus JUULing: how the extraordinary growth and marketing of JUUL transformed the US retail e-cigarette market. Tob Control 2019 Mar 31;28(2):146-151 [FREE Full text] [CrossRef] [Medline]
    11. Chu K, Colditz JB, Primack BA, Shensa A, Allem J, Miller E, et al. JUUL: Spreading Online and Offline. J Adolesc Health 2018 Nov;63(5):582-586 [FREE Full text] [CrossRef] [Medline]
    12. Allem J, Dharmapuri L, Unger JB, Cruz TB. Characterizing JUUL-related posts on Twitter. Drug Alcohol Depend 2018 Sep 01;190:1-5 [FREE Full text] [CrossRef] [Medline]
    13. Wang L, Zhan Y, Li Q, Zeng D, Leischow S, Okamoto J. An Examination of Electronic Cigarette Content on Social Media: Analysis of E-Cigarette Flavor Content on Reddit. Int J Environ Res Public Health 2015 Nov 20;12(11):14916-14935 [FREE Full text] [CrossRef] [Medline]
    14. Brett EI, Stevens EM, Wagener TL, Leavens EL, Morgan TL, Cotton WD, et al. A content analysis of JUUL discussions on social media: Using Reddit to understand patterns and perceptions of JUUL use. Drug Alcohol Depend 2019 Jan 01;194:358-362. [CrossRef] [Medline]
    15. Liu H, Li Q, Yao R. Analyzing Topics of JUUL Discussions on Social Media Using a Semantics-assisted NMF model. In: 2019 IEEE International Conference on Intelligence and Security Informatics (ISI). 2019 Presented at: 2019 IEEE International Conference on Intelligence and Security Informatics (ISI); July 1-3 2019; Shenzhen, China p. 212-214. [CrossRef]
    16. Zhan Y, Liu R, Li Q, Leischow SJ, Zeng DD. Identifying Topics for E-Cigarette User-Generated Contents: A Case Study From Multiple Social Media Platforms. J Med Internet Res 2017 Jan 20;19(1):e24 [FREE Full text] [CrossRef] [Medline]
    17. Zhan Y, Zhang Z, Okamoto JM, Zeng DD, Leischow SJ. Underage JUUL Use Patterns: Content Analysis of Reddit Messages. J Med Internet Res 2019 Sep 09;21(9):e13038 [FREE Full text] [CrossRef] [Medline]
    18. Leventhal AM, Miech R, Barrington-Trimis J, Johnston LD, O'Malley PM, Patrick ME. Flavors of e-Cigarettes Used by Youths in the United States. JAMA 2019 Nov 05;322(21):2132 [FREE Full text] [CrossRef] [Medline]
    19. Li Q, Zhan Y, Wang L, Leischow SJ, Zeng DD. Analysis of symptoms and their potential associations with e-liquids' components: a social media study. BMC Public Health 2016 Jul 30;16(1):674 [FREE Full text] [CrossRef] [Medline]
    20. Kavuluru R, Han S, Hahn EJ. On the popularity of the USB flash drive-shaped electronic cigarette Juul. Tob Control 2019 Jan 13;28(1):110-112 [FREE Full text] [CrossRef] [Medline]
    21. McKelvey K, Baiocchi M, Halpern-Felsher B. Adolescents' and Young Adults' Use and Perceptions of Pod-Based Electronic Cigarettes. JAMA Netw Open 2018 Oct 05;1(6):e183535 [FREE Full text] [CrossRef] [Medline]
    22. Shi J, Salmon CT. Identifying Opinion Leaders to Promote Organ Donation on Social Media: Network Study. J Med Internet Res 2018 Jan 09;20(1):e7 [FREE Full text] [CrossRef] [Medline]
    23. Reddit Statistics. pushshift.io.   URL: https://pushshift.io/ [accessed 2019-08-18]
    24. Reddit. API documentation. 2019.   URL: https://www.reddit.com/dev/api/ [accessed 2019-09-30]
    25. Food and Drug Administration. Tobacco 21. 2019.   URL: https://www.fda.gov/tobacco-products/retail-sales-tobacco-products/tobacco-21 [accessed 2020-02-12]
    26. Reddit. 2018. Reddit Privacy Policy.   URL: https://www.redditinc.com/policies/privacy-policy [accessed 2019-09-30]
    27. Thompson K. Programming Techniques: Regular expression search algorithm. Commun ACM 1968;11(6):419-422. [CrossRef]
    28. Mayring P. Qualitative content analysis: theoretical foundation, basic procedures and software solution. Social Science Open Access Repository. 2014.   URL: https://www.ssoar.info/ssoar/handle/document/39517 [accessed 2019-09-30]
    29. Food and Drug Administration. Warning Letters and Civil Money Penalties Issued to Retailers for Selling JUUL and Other E-Cigarettes to Minors. 2018.   URL: https:/​/www.​fda.gov/​tobacco-products/​ctp-newsroom/​warning-letters-and-civil-money-penalties-issued-retailers-selling-juul-and-other-e-cigarettes [accessed 2020-02-09]
    30. Food and Drug Administration. FDA finalizes enforcement policy on unauthorized flavored cartridge-based e-cigarettes that appeal to children, including fruit and mint. 2020.   URL: https:/​/www.​fda.gov/​news-events/​press-announcements/​fda-finalizes-enforcement-policy-unauthorized-flavored-cartridge-based-e-cigarettes-appeal-children [accessed 2020-01-04]
    31. Food and Drug Administration. Selling Tobacco Products in Retail Stores. 2020.   URL: https:/​/www.​fda.gov/​tobacco-products/​retail-sales-tobacco-products/​selling-tobacco-products-retail-stores [accessed 2020-06-19]
    32. 2019. JUUL Labs stops the sale of mint JUULpods in the United States.   URL: https://newsroom.juul.com/juul-labs-stops-the-sale-of-mint-juulpods-in-the-united-states/ [accessed 2020-02-01]
    33. Cwalina SN, Leventhal AM, Barrington-Trimis JL. E-cigarette flavour enhancers: Flavoured pod attachments compatible with JUUL and other pod-based devices. Tob Control 2020 Apr 09:2020. [CrossRef] [Medline]

    ‎
    API: application programming interface
    ENDS: Electronic Nicotine Delivery Systems
    FDA: Food and Drug Administration

    Edited by G Eysenbach; submitted 07.11.19; peer-reviewed by J Cappella, A Majmundar, N O'Kane; comments to author 06.01.20; revised version received 26.05.20; accepted 25.06.20; published 20.07.20

    Copyright

    ©Hejing Liu, Qiudan Li, Yongcheng Zhan, Zhu Zhang, Daniel D Zeng, Scott J Leischow. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 20.07.2020.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.