Quality and Audience Engagement of Takotsubo Syndrome–Related Videos on TikTok: Content Analysis

Introduction

Takotsubo syndrome (TTS), also known as the broken heart syndrome or stress cardiomyopathy, is characterized by transient ventricular dysfunction with typical wall motion abnormalities [1,2]. The number of patients diagnosed with TTS has been gradually increasing [3], with hospitalizations for TTS increasing from 5.7 per 100,000 person-years in 2007 to 17.4 per 100,000 person-years in 2012 (P<.001) [4]. The clinical manifestation of TTS is similar to that of acute coronary syndrome (ACS) and frequently presents with chest discomfort or dyspnea, ST segment deviation on electrocardiogram, and cardiac biomarker abnormalities [2,5,6]. Given the overlap with the clinical presentation of ACS, TTS can be easily misdiagnosed. Previous reports [7,8] have shown that about 0.7%-2% of all patients with possible ACS were eventually found to have had TTS. Ongoing research has shown that the prognosis of TTS is not as benign as previously thought, with in-hospital mortality of 4.1% and long-term mortality of 24.7% [6,9].

The internet is a useful platform for effectively communicating new information, and many new technological applications have taken advantage of this to serve as a medium for patient education on health-related topics. Seeking web-based health information has become increasingly popular; many people rely on web-based resources to obtain health information and aid their medical decision-making [10]. Research has shown that health outcomes can be positively impacted by appropriately and effectively utilizing social media platforms [11]. One of the many social media applications that has been used to reach a large audience is TikTok [12]. However, the most important limitation of such platforms is the unreliable quality of the information presented. Anyone can present information on social media platforms, most of which lack formal moderation for authenticity and reliability of the presented material [13]. Recent systematic reviews have suggested that the quality of web-based health information is problematic and perhaps made worse when considering information disseminated on social media platforms [14].

Along with the quality of information, audience engagement is another key component of effective web-based health communication. Audience engagement in health-related topics has been studied in traditional social media platforms such as YouTube and Facebook [15-19]. The influence of many factors such as video content, quality, and information sources on audience engagement has been studied previously [15,16,18-20]. Several studies have shown that video content is associated with audience engagement [15,18,20]. Szmuda et al [15] studied the association between video content and audience engagement in COVID-19 videos on YouTube, and they found that videos showing the causes, management strategies, diagrams, and structure anatomies were associated with a higher “like” ratio. Another study [20] on audience engagement and COVID-19 short videos on TikTok found that content type (news, codebook, etc) influenced the level of audience engagement.

Previous studies have yielded inconsistent conclusions on the impact of video quality on audience engagement. A study on videos of stroke on YouTube by Szmuda et al [16] showed that there was no strong correlation between the DISCERN score (an indicator of video quality) and audience engagement. Huang et al [19] found that nephrolithiasis-related videos on YouTube with inaccurate statements were associated with higher audience engagement (viewer-generated comments, thumbs-up and thumbs-down ratings) than videos without inaccuracies. The source of information was also found to affect audience engagement. Szmuda et al [16] found that higher engagement was noted in stroke-related videos that were uploaded by an educational channel on YouTube. Recently, studies [21,22] have shown that the emerging short-video social apps can satisfy people’s intrinsic motivations and elicit user engagement when disseminating health information.

TikTok (DouYin in Chinese) is a short-video social app with a sizable userbase wherein individual users create and publicly post short videos on various subjects. Initially, when the platform first became popular, the video length was limited to short 60-second clips. However, with growing demand, the length was extended to allow up to 5-minute-long videoclips [23]. TikTok has attained significant popularity since its launch in September 2016 and has since raked up more than 500 million active users and 1 billion downloads [24,25]. Given its extensive reach and better audience engagement than other traditional social media platforms [23], TikTok has the potential to be a great source of health information dissemination and become increasingly popular among general health consumers as an emerging health information source. Several studies evaluating patient education videos regarding COVID-19 [26], skin-related diseases [27], chronic obstructive pulmonary disease [28], and aesthetic surgery procedures [29] posted on TikTok have been published. These studies have revealed that the overall quality of such videos presenting health information on TikTok is low, and some even present overtly false information [27,29]. Very few studies [15,16,18,20] have focused on audience engagement with health care information on TikTok as compared to that with health care information on other social media platforms. We found a considerable number of videos regarding TTS on TikTok; however, the quality and content of the presented information and whether these factors may affect audience engagement remain largely unknown. Therefore, this study aims to evaluate the content and quality of videos related to TTS on TikTok and assess the qualitative metrics that drive audience engagement (in terms of the number of likes) with a video.

Methods

Search Strategy

The search was conducted on August 2, 2021, in China. Six specific hashtags（“#应激性心肌病” “#Takotsubo综合征” “#Takotsubo心肌病” “#心碎综合征” “#章鱼壶心肌病” “#心尖球形综合征”）that refer to TTS in Chinese were used to retrieve TikTok videos related to TTS. TikTok provides 3 ways to filter videos, that is, overall ranking, most recent, and most liked. Considering that most users use the default sorting option, “overall ranking,” we performed the search in TikTok under the discover mode using the “overall ranking” sort option. All the resultant videos for each keyword were retrieved and screened. The initial search returned 2549 videos (441, 400, 433, 412, 439, and 424, respectively), of which 80 videos met our criteria for analysis after screening. The exclusion criteria were as follows: (1) videos not related to TTS or lacking educational information, (2) duplicate videos, (3) videos not in Chinese, and (4) videos that were not original. Figure 1 illustrates the selection process implemented in our study.

‎View this figure

Data Extraction

Baseline characteristics were extracted directly from each video and the video author’s public TikTok profile on the same day of the data search. With respect to the video authors, we collected their account ID, inauguration unit, number of both followers and those they follow, overall number of posted videos, and likes obtained. Additionally, we further ascertained whether the public TikTok profile had photos, live broadcast information, and contact details. For each individual video, we also collected the uniform resource locator, the date posted, the number of likes, comments, reposts, and duration. Based on the abovementioned information and historical videos on their profile page, the TikTok authors were classified into 6 categories: (1) individual science communicators, (2) news agencies, (3) for-profit organizations, (4) health professionals, (5) nonprofit organizations, and, (6) general users. Furthermore, the content of each video was assessed for the following characteristics: use of characters, background music, emoji, flash, and video subtitles.

Coding Schema

We used the DISCERN instrument and Journal of the American Medical Association (JAMA) criteria to evaluate the quality of the selected TikTok videos. These instruments are commonly used standards for health information evaluation [30,31]. The DISCERN instrument (quality criteria for consumer health information) consists of 16 questions, with each question scored from 1 to 5 points. Questions are divided into 3 parts: reliability of the publication (questions 1 to 8), quality of information about treatment options (questions 9 to 15), and the overall score of the publication (question 16) [32]. The JAMA benchmarks were applied to evaluate health-related information’s reliability, plausibility, and usefulness in the internet [33]. The JAMA criteria consist of 4 main sections, which are scored from 0 to 4. Six questions obtained from reports of Goobie et al [34] were used to measure the quality of the video content. The 6 questions assessed the definition, signs or symptoms, risk factors, evaluation, management, and outcomes of the disease discussed in the videos. Each aspect was scored across 3 degrees from 0 to 2: not addressed, partially addressed, and sufficiently addressed.

Video Coding

All video contents evaluated using the DISCERN instrument, JAMA benchmarks, and the 6 questions from the studies of Goobie et al [34] were independently scored and coded by 2 study authors (JL and YX). Prior to coding, a training exercise was conducted, during which 20 videos each were scored and coded independently by the 2 coding authors, and any resulting discrepancies were addressed and resolved to ensure homogeneity in coding. An average of 2 points was assigned by rounding to the nearest unit. Microsoft Excel (2019) was used to extract, code the basic information of each video, and process statistical data.

Statistical Analysis

Descriptive statistics for continuous variables, including the mean, median, range, and standard deviation, were calculated. SPSS version 22.0 (IBM Corp) was used to perform data analysis. An intraclass correlation coefficient was used to assess the interrater agreement. The average agreement by intraclass correlation coefficient was 0.87 for content rating, 0.92 for DISCERN scales rating, and 0.88 for the JAMA benchmark rating. The average intraclass correlation coefficients for interrater agreements all exceeded the recommended value of 0.75, indicating that the ratings had good reliability [35]. The Kruskal-Wallis test was performed to identify differences between the extracted variables. A P value <.05 was deemed statistically significant. In our paper, we use the number of likes as a proxy measure of audience engagement. Multiple linear regression was performed to evaluate the association between audience engagement and other factors such as video quality and author types. The regression analysis was conducted on R (R version 4.2.1; 2022-06-23 ucrt), with a log transformation of the outcome variable audience engagement. This study did not involve human subjects, and hence, the study was not reviewed by the institutional review board.

Results

Video Characteristics

The average length of the evaluated videos was 59 seconds, with a maximum duration of 700 seconds and a minimum duration of 7 seconds. On average, a video received 5543 likes and 635 reposts. The majority of the videos (49/80, 61%) had the presence of people, 90% (72/80) had subtitles, and 70% (56/80) of the videos had background music. Approximately 19% (15/80) and 23% (18/80) of the videos had emojis and animations, respectively (Table 1).

With regard to content creation, users classified as health professionals posted the most videos (22/80, 28%), followed by general users (20/80, 25%), individual science communicators (18/80, 23%), news agencies (11/80, 14%), and for-profit organizations (6/80, 8%). Nonprofit organizations posted the fewest videos (3/80, 4%) (Multimedia Appendix 1). The average number of likes received per video, categorized by video author types, from high to low were as follows: news agencies, health professionals, for-profit organizations, individual science communicators, general users, and nonprofit organizations (Table 2).

Video Content

The average total score of Goobie et al’s [34] 6-question survey by both raters was 5.40 out of 12, indicating that the overall content quality of these videos was average (in our study, the average total score of Goobie et al’s [34] 6-question survey ≥9.0 is considered excellent, ≥6.0-<9.0 is considered good, and <6.0 is considered average). The average scores for the videos given by the 2 raters for each of the 6 criteria described by Goobie et al [34] are shown in Multimedia Appendix 2. Of the 6 items, “symptoms” was the most common, while “diagnosis” was the least common. The average scores (total score for each item is 2), graded for each item and ranked from highest to lowest were as follows: symptoms (1.42), risk factors (1.09), definition (1.06), outcome (0.83), management (0.66), and diagnosis (0.34). We further analyzed video authors by categorizing them into the professional and nonprofessional group. Health professionals comprised the professional group, while the nonprofessional group included authors other than health professionals, such as news agencies and for-profit organizations. Health professionals scored higher in terms of video content compared to the other 5 author categories (Table 3). Videos posted by professionals (6.20) scored higher on average than those posted by nonprofessionals (5.09).

Video Quality

The average score given for the videos on the DISCERN instrument by both raters was 36.93 out of 80, indicating that the overall quality of these videos was poor (average total score of 16-26 is very poor, a score of 27-38 is poor, a score of 39-50 is fair, a score of 51-62 is good, and a score >63 is excellent [36]). Among videos created by different author types, the average score given to videos made by health professionals was the highest (40.59), followed by for-profit organizations (40.33) and non-profit organizations (39.67). Although the video scores of the above 3 resources are relatively higher, the quality rating level is only fair. The lowest average score was for those created by general users (30.80), indicating that videos posted by general users were the poorest in terms of quality. The comparison of scores between the professional and nonprofessional group is shown in Multimedia Appendix 3. Questions 3, 6, and 8 were scored above 3 points, and these questions addressed relevance, bias, and areas of uncertainty with reference to the selected videos, respectively. Questions 5 and 9-14 were found to have scores of less than 2 points. These questions mainly addressed whether the date of the information used or reported in the publication was clear; whether the mechanism, benefits, and risks of each treatment, consequences of not treating, or impact of different treatment options on the overall quality of life were described; and whether the possibility of more than one treatment choice being available was clear (Multimedia Appendix 3).

Both raters agreed that 98% (78/80) of the videos provided the latest information, 43% (34/80) of the videos provided information regarding authorship, but none mentioned the disclosure statement. None of the videos met all the JAMA criteria. The average JAMA score was 1.51. By video author categories, an average score of 1.91 was obtained by health professionals, 1.08 by individual science communicators, 1.08 by general users, 1.77 by news agencies, 1.83 by for-profit organizations, and 2.00 by nonprofit organizations. Based on the Kruskal-Wallis test, there were statistically significant differences in content, DISCERN instrument scores, and the JAMA standard video assessment scores among the 6 author groups (P<.05).

Analysis of Information Quality and Audience Engagement

The results of our multiple linear regression analysis showed that author types that identified as health professionals (exponentiated regression coefficient 17.48, 95% CI 2.29-133.52; P=.006) and individual science communicators (exponentiated regression coefficient 13.38, 95% CI 1.83-97.88; P=.01) were significant and independent determinants of audience engagement (in terms of the number of likes). Other author types of videos, video content, and DISCERN document scores were not associated with higher likes (Table 4).

Discussion

Principal Findings

Growing evidence suggests that TTS is a severe cardiac disorder with a substantial mortality risk. A meta-analysis showed no difference in the in-hospital and long-term mortality between TTS and ACS [37]. Long-term recurrence rates of TTS have been reported to be as high as 11.4% over 4 years [38]. Although regional wall dysfunction is reversible in TTS, patients may continue to experience chest pain, fatigue, and dyspnea even after recovery of wall function [39]. Notably, traditional cardiovascular risk factors are less commonly seen in TTS compared with those seen in ACS [37]. TTS presentations are often instigated by stress-related emotional and physical factors preceding symptoms [1]. Although such factors may not always be preventable, awareness of TTS as an entity by the general population may improve the measures adopted by individuals in its recognition and seeking prompt medical attention. Additionally, it may improve the collateral history provided by affected individuals, which may aid health care providers in its diagnosis. This is especially important as patients who develop TTS are known to be more apathetic with regard to their mortality and their acute presentation, as evidenced by a study [40] evaluating the psychology of patients with TTS and ACS. That study [40] has also shown that patients with TTS were less likely to be concerned about contracting diseases (P<.05) and had fewer thoughts related to the acute cardiac episode that interfered with their life (P<.001). It is still unclear whether awareness of TTS will change these perceptions, but an increase in awareness may certainly encourage individuals to adopt measures to mitigate individual stressors, which, in turn, may reduce the incidence of TTS. Some patients with TTS continue to experience symptoms despite resolution of the acute phase. Cardiac rehabilitation may be beneficial for these patients to improve the quality of life and reduce episodes of ongoing chest pain, but only few patients have been reported to receive cardiac rehabilitation owing to the lack of awareness among patients and even doctors [41]. Improving the awareness of TTS may aid the rate of those seeking cardiac rehabilitation for persistent symptoms as well. There is an increasing trend toward using social media for patient education. Despite there being a lot of relevant information on social media platforms from reliable sources, there also exists a large amount of inaccurate information. This poses a significant challenge to users seeking easily accessible and comprehensible information regarding their health. Moreover, low-quality information weakens the ability of people to make informed decisions and can even lead to harmful consequences.

Video Quality Analysis

To the best of our knowledge, this is the first study to analyze the quality of TTS-related videos on TikTok. As a popular social media platform, TikTok has the potential to become a valuable and influential platform to disseminate health information, especially in the context of the current COVID-19 pandemic [42,43]. In our study, 80 TTS-related videos received 443,469 likes and were commented on and shared thousands of times, which also affirms TikTok’s powerful communication capabilities. However, there is significant concern regarding the quality of these TTS videos on TikTok. Their scores on the DISCERN instrument (36.93) and the JAMA criteria (1.51) were generally low. These findings are similar to the results of Śledzińska et al [44] in their study on YouTube videos (n=61) on meningioma treatment. In their study, the mean total DISCERN score was poor as well at 36.4. Part of the reason for the poor quality may be that short-video platforms lack scientific insight for health information dissemination. Furthermore, there is no restriction on the type of content that video authors may publish nor is there any restriction based on author type to ensure content quality. Addressing these concerns in the context of health information dissemination will be invaluable [43]. Considering this, we further studied the quality of videos based on author type. We found that health professionals are the leading creators of popular science videos (Multimedia Appendix 1). Videos created by health professionals had higher average DISCERN scores than those created by non–health professionals. This is consistent with current literature reports that videos created by professionals are likely to have higher quality [29,45,46]. Given the inconsistency in the quality of videos based on the source, we recommend that patients be cautious when obtaining health-related information through platforms such as TikTok.

Notably, videos created by health professionals also received the highest number of likes and reposts in our study, which is in contrast to the results of other prior studies. In previous studies evaluating videos on psoriasis and nephrolithiasis on YouTube, poor-quality videos received greater attention [19,47]. Our study shows that higher health professional participation in health-related short video productions results in higher popularization ranks, having gained wider user attention. Although the quality of the videos produced by health professionals needs to be improved, it is undeniable that the participation of medical professionals in the creation of high-quality videos plays an essential role in promoting health education [29].

Content and Optimization Analysis

In terms of video content, we found that very few videos introduced the concept of diagnosis of TTS, and the display of reference information sources was not common, which remains an area to be addressed. The comprehensiveness and accuracy of the video content is a necessary prerequisite to ensure reliable transmission of information, especially when most TikTok users do not have the ability to differentiate health information for reliability. Therefore, we recommend that video producers provide sources of reference information. In addition, in the process of screening videos, we noticed that many videos showed the causes of TTS to be mainly emotional factors, and only a few videos emphasized physical factors, which are important and cannot be ignored. A retrospective study on patients with TTS showed that physical factors are considerable risk factors for in-patient mortality [48]. It is appropriate for video producers to ask experts for review before content creation or check authoritative source materials to ensure that the information published is comprehensive and accurate. As put forward by Oh and You [49], it may be essential to form an expert evaluation team to authenticate the reliability of health-related videos prior to dissemination and provide corresponding identification certificates. In addition, organizational formats of health information can affect the health intervention. Therefore, video authors should fully understand individuals’ needs regarding health information and organize information effectively to provide targeted health information [50]. The video monitoring platform should also design a recommendation algorithm that filters higher-quality videos for users as much as possible, especially with reference to medical- and health-related information.

Audience Engagement Analysis

Through multiple linear regression analysis, we found that, compared with videos made by general authors, those made by health professionals and individual science communicators were more likely to obtain likes. Interestingly, no correlation was found between audience engagement and a video’s DISCERN scores. The number of likes is commonly viewed as a collective filter and as an indicator of popularity, which may reflect video quality [19,51,52]. However, our findings suggested that, similar to that in other platforms, the audience on the TikTok platform pay more attention to the identity of the author rather than the content and quality of the video. This finding echoes with those of previous studies [16,19], which show that consumers should remain cautious of using such indictors to judge a video’s credibility and that health professionals and individual science communicators have a significant role in video production in TikTok. By providing high-quality videos through these professional authors, TikTok can allow for accelerated health care information dissemination and even potentially improve outcomes in certain diseases.

Limitations and Future Directions

Our study has some limitations. First, this was a cross-sectional study with a small sample size, despite attempts to include all relevant videos. Second, we only evaluated Chinese videos, which may not be representative in a global context. Future research studies can target characteristics of videos in various languages and regions. In addition, we utilized common standards that are currently used for health information video evaluation, namely, the DISCERN instrument and the JAMA criteria [30,34]. However, some scholars have suggested that these 2 standards were developed relatively early and were initially used to evaluate website information and may have limitations when used to evaluate video information [53]. It is vital that a new video information evaluation tool is developed to adapt in this era of rapidly proliferating video content. Finally, we did not analyze the user comments addressed in each video, and we were unable to track the behavioral and psychological changes of the end user or the information recipient. How these videos affect patient behavior in reality is an area that needs to be studied in the future.

Conclusion

By analyzing the quality of TTS-related videos on TikTok, we found that videos produced by health professionals were found to have the highest DISCERN scores. However, the overall quality of the videos related to TTS on TikTok is poor. The multiple linear regression analysis showed that author type categories of health professionals (P=.006) and individual science communicators (P=.01) were significant and independent determinants of consumer engagement (in terms of the number of likes). Our study indicates that the formulation of a measure to review video quality and reliability, especially with respect to health-related information dissemination on TikTok platform, is imperative and patients should be cautious when obtaining health-related information through TikTok. Medical professionals and individual science communicators should be encouraged to create high-quality health-related videos, which may potentially have higher audience engagement and promote health education.