This paper is in the following e-collection/theme issue:

Digital Mental Health Interventions, e-Mental Health and Cyberpsychology (1514) Infodemiology and Infoveillance (1349) Psychotic Disorders (160) Linguistics, Comprehension, and Learning (52)

Published on in Vol 22, No 4 (2020): April

A Comparison of the Psycholinguistic Styles of Schizophrenia-Related Stigma and Depression-Related Stigma on Social Media: Content Analysis

A Comparison of the Psycholinguistic Styles of Schizophrenia-Related Stigma and Depression-Related Stigma on Social Media: Content Analysis

A Comparison of the Psycholinguistic Styles of Schizophrenia-Related Stigma and Depression-Related Stigma on Social Media: Content Analysis

Authors of this article:

Ang Li1, 2 Author Orcid Image ;   Dongdong Jiao3 Author Orcid Image ;   Xiaoqian Liu2 Author Orcid Image ;   Tingshao Zhu2 Author Orcid Image
Article Authors Cited by (29) Tweetations (2) Metrics

    Original Paper

    1Department of Psychology, Beijing Forestry University, Beijing, China

    2Institute of Psychology, Chinese Academy of Sciences, Beijing, China

    3National Computer System Engineering Research Institute of China, Beijing, China

    Corresponding Author:

    Tingshao Zhu, PhD

    Institute of Psychology

    Chinese Academy of Sciences

    No 16 Lincui Rd

    Chaoyang District

    Beijing, 100101

    China

    Phone: 86 150 1096 5509

    Email: tszhu@psych.ac.cn


    Background: Stigma related to schizophrenia is considered to be the primary focus of antistigma campaigns. Accurate and efficient detection of stigma toward schizophrenia in mass media is essential for the development of targeted antistigma interventions at the population level.

    Objective: The purpose of this study was to examine the psycholinguistic characteristics of schizophrenia-related stigma on social media (ie, Sina Weibo, a Chinese microblogging website), and then to explore whether schizophrenia-related stigma can be distinguished from stigma toward other mental illnesses (ie, depression-related stigma) in terms of psycholinguistic style.

    Methods: A total of 19,224 schizophrenia- and 15,879 depression-related Weibo posts were collected and analyzed. First, a human-based content analysis was performed on collected posts to determine whether they reflected stigma or not. Second, by using Linguistic Inquiry and Word Count software (Simplified Chinese version), a number of psycholinguistic features were automatically extracted from each post. Third, based on selected key features, four groups of classification models were established for different purposes: (a) differentiating schizophrenia-related stigma from nonstigma, (b) differentiating a certain subcategory of schizophrenia-related stigma from other subcategories, (c) differentiating schizophrenia-related stigma from depression-related stigma, and (d) differentiating a certain subcategory of schizophrenia-related stigma from the corresponding subcategory of depression-related stigma.

    Results: In total, 26.22% of schizophrenia-related posts were labeled as stigmatizing posts. The proportion of posts indicating depression-related stigma was significantly lower than that indicating schizophrenia-related stigma (χ21=2484.64, P<.001). The classification performance of the models in the four groups ranged from .71 to .92 (F measure).

    Conclusions: The findings of this study have implications for the detection and reduction of stigma toward schizophrenia on social media.

    J Med Internet Res 2020;22(4):e16470

    doi:10.2196/16470

    Keywords

    stigma; schizophrenia; depression; psycholinguistic analysis; social media 


    Stigma is a destructive phenomenon that undermines efforts to improve mental health and well-being in people with mental illness. A high perception of stigma is associated with reduced self-disclosure to psychotherapists and others, leading to delayed treatment. Schizophrenia is among the most stigmatized mental illnesses; it can be considered a paradigm for mental illness [1,2]. Therefore, it is essential to maintain schizophrenia-related stigma as the main focus of antistigma campaigns [3-5].

    Enhancing public knowledge and beliefs about mental illnesses is foundational for stigma reduction. Because information distributed through mass media is an important contributor to the dissemination of materials that may increase mental illness stigma [6-11], mass media campaigns, especially social media campaigns, are effective in raising public awareness of mental health literacy. Social media enables users to bring personal experience into the public domain with the potential to influence public perceptions of mental illness [12]. It also allows users to create their own social networks, which can be leveraged to facilitate the acceptance of received knowledge and then accelerate changes in individual attitudes and behaviors [13-15]. However, such campaigns should be carefully planned and target knowledge deficits, suggesting a need for monitoring mental illness–related stigma on social media. To address this concern, a number of content analysis studies have been performed by human coders on social media to examine stigma related to different mental illnesses, including schizophrenia, depression, suicide, eating disorder, and obsessive-compulsive disorder [2,16-19]. However, the sheer volume of information available on social media makes it difficult for human coders to keep track of all information. For example, in China, Sina Weibo (a free social media site that is similar to Twitter) has over 500 million registered users and produces more than 100 million microblogs (Weibo posts) per day. Therefore, in order to develop more effective strategies to challenge stigma, there is a need for accurate and efficient detection of mental illness–related stigma on social media.

    The way people use words provides insight into their psychological profiles [20]. By using computerized text analysis tools (eg, Linguistic Inquiry and Word Count [LIWC]), psycholinguistic analysis can be automatically performed on texts to characterize language use patterns in terms of psychologically meaningful categories, making it easier to keep track of social media data. Recent studies have concluded that psycholinguistic analysis methods can be used to discover characteristics of stigmatizing expressions in social media posts (eg, suicide- and depression-related stigma) [21,22]. However, to date, no research has investigated psycholinguistic characteristics of stigma related to schizophrenia. To our knowledge, there is no convincing evidence for the lack of psycholinguistic differences in the expression of stigma across different mental illnesses. Therefore, additional analysis is necessary to examine the ways in which stigma associated with schizophrenia is presented on social media.

    To address this concern, this study aims to investigate psycholinguistic characteristics of schizophrenia-related stigma on social media (ie, Sina Weibo), and then attempts to explore whether schizophrenia-related stigma can be distinguished from stigma related to other mental illnesses (ie, depression-related stigma). Stigma associated with depression was selected for the comparison since it is considered to be one of the most stigmatized mental illnesses, particularly among social media users [2,21].


    Overview

    This protocol was reviewed and approved by the Institutional Review Board at the Institute of Psychology, Chinese Academy of Sciences. Participant consent was not obtained, as it is not required for analyzing publicly available data [17,18,21,22]. To protect the privacy of participants, personally identifiable information (including usernames, real names, and other personal information) was excluded from data analysis.

    The research process included the following three steps: (a) data collection, (b) data preprocessing, and (c) data modeling.

    Data Collection

    First, a participant pool was created for data collection. Similar to Twitter, Sina Weibo is a free social media site that enables users to communicate and interact with others by posting digital messages (Weibo posts). Although some users opt to privatize their accounts, the majority of Weibo posts are publicly available for viewing and downloading. According to a previous study [23], a total of 1,953,485 Sina Weibo users were identified as potential participants. However, to due restrictions on data access imposed by Sina Weibo, of these potential participants, Weibo posts from only 1.06 million users were available for download.

    Second, data from these participants were downloaded to construct a database of Weibo posts. By using an application programming interface (API), on April 2012, Weibo posts were downloaded automatically from 1.06 million users since the beginning of their registration. After that, the database of Weibo posts was maintained and updated regularly, with the latest update occurring in June 2017.

    Third, relevant Weibo posts were identified from the database. To identify posts that are highly relevant to the topics of schizophrenia and depression, all downloaded posts should be searched using schizophrenia- and depression-related keywords, respectively. In this study, in order to make sure that identified posts refer to schizophrenia or depression as mental illnesses, two sets of keywords were selected, including “depressive disorder” () and “schizophrenia” (). Therefore, posts with such keywords were included for further analysis. A total of 15,879 depression-related posts were obtained from 10,130 distinct users (time period: September 21, 2009, to June 9, 2017); a total of 19,224 schizophrenia-related posts were obtained from 15,676 distinct users (time period: September 24, 2009 to June 11, 2017). More details about the gender and location of these users can be found in Table 1.

    Table 1. Demographics of included users.
    CharacteristicUsers, depression-related posts (n=10,130), n (%)Users, schizophrenia-related posts (n=15,676), n (%)
    Gender


    		Male3208 (31.67)7152 (45.62)

    		Female6457 (63.74)8425 (53.74)

    		Not specified465 (4.59)99 (0.63)
    Location (provinces)


    		Anhui97 (0.96)187 (1.19)

    		Aomen13 (0.13)24 (0.15)

    		Beijing1138 (11.23)2039 (13.01)

    		Chongqing143 (1.41)205 (1.31)

    		Fujian283 (2.79)458 (2.92)

    		Gansu27 (0.27)66 (0.42)

    		Guangdong1732 (17.10)2789 (17.79)

    		Guangxi134 (1.32)201 (1.28)

    		Guizhou62 (0.61)115 (0.73)

    		Hainan31 (0.31)78 (0.50)

    		Hebei122 (1.20)217 (1.38)

    		Henan208 (2.05)285 (1.82)

    		Heilongjiang86 (0.85)124 (0.79)

    		Hong Kong46 (0.45)121 (0.77)

    		Hubei206 (2.03)323 (2.06)

    		Hunan157 (1.55)241 (1.54)

    		Inner Mongolia40 (0.39)74 (0.47)

    		Jilin64 (0.63)86 (0.55)

    		Jiangsu421 (4.16)680 (4.34)

    		Jiangxi68 (0.67)111 (0.71)

    		Liaoning185 (1.83)273 (1.74)

    		Ningxia19 (0.19)38 (0.24)

    		Qinghai6 (0.06)8 (0.05)

    		Shandong752 (7.42)1146 (7.31)

    		Shanxi63 (0.62)122 (0.78)

    		Shaanxi134 (1.32)235 (1.50)

    		Shanghai1073 (10.59)1524 (9.72)

    		Sichuan271 (2.68)489 (3.12)

    		Taiwan18 (0.18)55 (0.35)

    		Tianjin125 (1.23)180 (1.15)

    		Tibet11 (0.11)16 (0.10)

    		Xinjiang49 (0.48)61 (0.39)

    		Yunnan80 (0.79)161 (1.03)

    		Zhejiang555 (5.48)740 (4.72)

    		International634 (6.26)1112 (7.09)

    		Not specified1077 (10.63)1092 (6.97)

    Data Preprocessing

    After data collection, data preprocessing was performed on raw data to prepare it for data modeling.

    First, to obtain predicted class labels for data modeling, a content analysis was performed on collected Weibo posts to determine whether they reflected stigma or not. The coding framework was developed on the basis of expert consensus and available evidence. Specifically, a researcher reviewed relevant studies [17,24-27], and conducted an inductive analysis of all collected posts to construct an initial framework. After that, two human coders were recruited and received training on the content of the initial framework and gave suggestions for its amendment. Finally, the initial framework was amended accordingly, and the formal framework was established. As a result, the formal framework of schizophrenia-related stigma included 11 subcategories while the formal framework of depression-related stigma included 9 subcategories [21]. There was a high degree of overlap in stigma subcategories between the two frameworks. Specifically, the framework of schizophrenia-related stigma was quite similar to that of depression-related stigma, but two additional subcategories (ie, weird and stupid) were added to make a set of eleven. By using the formal frameworks, two independent human coders were instructed to analyze all Weibo posts with keywords (15,879 depression-related posts and 19,224 schizophrenia-related posts). Levels of consistency of coding were evaluated by computing Cohen kappa coefficients. If cases of inconsistency, input from a third researcher was used to resolve the issue. The coding results were considered as the ground truth for data modeling validation.

    Second, to obtain predictors for data modeling, LIWC software (Simplified Chinese version) was used to automatically extract psycholinguistic features from each post. This is a reliable and valid text analysis tool for the automatic estimation of word frequency in multiple psychologically meaningful categories, including linguistic processes (eg, personal pronouns), psychological processes (eg, affective processes), personal concerns (eg, achievement), and spoken categories (eg, assent) [28]. For example, values of features indicating personal pronouns, affective processes, achievement, and assent refer to frequencies of words associated with personal pronouns (eg, them), affective processes (eg, abandon), achievement (eg, hero), and assent (eg, agree), respectively. To remove the effects of keywords on data modeling, keywords, including “depressive disorder” and “schizophrenia”, were deleted from each post before feature extraction. Therefore, a number of psycholinguistic features can be obtained for each post. Finally, standardized values of psycholinguistic features were computed as potential predictors.

    Data Modeling

    In this study, the Waikato Environment for Knowledge Analysis software (Weka, version 3.8.1) was used to build four groups of classification models.

    The first group of classification models was built to differentiate schizophrenia-related stigma from nonstigma. To solve the class imbalance problem, a certain number of posts were randomly selected from the majority class to obtain a well-balanced data set. After that, to improve the performance of data modeling, psycholinguistic features that were valid for differentiating between posts with and without stigma toward schizophrenia were selected as key features. Specifically, for each psycholinguistic feature, an independent sample t test was performed to compare values between two groups (schizophrenia-related stigma and nonstigma), and then the effect size value (Cohen d, which is one of the most common ways to measure effect size and can be used to indicate the standardized difference between two means) was calculated using the estimated t value. In this study, features that were statistically significant at .05 and had a Cohen d >0.20 or <–0.20 were considered as key features. Finally, by using four different algorithms (support vector machine [SVM]; naïve Bayes [NB]; multilayer perceptron neural network [MPNN]; logistic model trees [LMT]), four classification models (SVM, NB, MPNN, and LMT models) were established based on selected key features. Each model was tested by 10-fold cross-validation. Specifically, the data set was randomly divided into ten subgroups with the same sample size. Each subgroup was used to test the model that was built on the other nine subgroups. After 10 rounds of model training, the modeling results were integrated into a final model. The classification performance was evaluated by three indicators: precision (number of true positives / number of instances predicted to be positive), recall (number of true positives / number of positive instances), and F measure (a tradeoff between precision and recall).

    The second group of classification models was built to differentiate a certain subcategory of schizophrenia-related stigma from other subcategories. In this study, two major subcategories of stigma related to schizophrenia (unpredictable and dangerous stigma) were examined (unpredictable stigma / other subcategories; dangerous stigma / other subcategories). The balanced classification data sets, key features, and classification models were obtained using the same method outlined above.

    The third group of classification models was built to differentiate schizophrenia-related stigma from depression-related stigma. The balanced classification data set, key features, and classification models were obtained using the same method outlined in the section on the first group of classification models.

    For the third group of classification models, the reason for good classification performance may be attributed to marked differences in amount and distribution of stigma subcategories between schizophrenia and depression rather than actual existence of differences in psycholinguistic style between schizophrenia- and depression-related stigma. To clarify this issue, the fourth group of classification models was built to differentiate a certain subcategory of schizophrenia-related stigma from the corresponding subcategory of depression-related stigma. To obtain enough data for further analysis in this study, two subcategories of stigma (unpredictable and glorified stigma) were examined (unpredictable stigma related to schizophrenia / unpredictable stigma related to depression; glorified stigma related to schizophrenia / glorified stigma related to depression). The balanced classification data sets, key features, and classification models were obtained using the same method outlined in the section on the first group of classification models.


    Coding

    The coding results for stigma related to schizophrenia are shown in Table 2 (see Multimedia Appendix 1 for posts in Chinese). The Cohen kappa coefficients for schizophrenia-related stigma and its subcategories reached .77 and .78, respectively, reflecting a satisfying level of agreement [29]. Of all schizophrenia-related posts, 26.22% (5041/19,224) were labeled as stigmatizing posts. Of these posts, 41.14% (n=2074) and 26.86% (n=1354) reflected the view that “people with schizophrenia are unpredictable” (unpredictable stigma) and “people with schizophrenia are dangerous” (dangerous stigma), respectively.

    Table 2. Coding framework for schizophrenia-related stigma.
    SubcategoryDefinitionRepresentative Weibo post
    (English translation)    		Posts, n (%)
    It is best to avoid people with schizophrenia so that you do not become vulnerable to schizophreniaBeliefs that schizophrenia is an infectious disease“It is said that schizophrenia is highly infectious...”26 (0.52)
    Schizophrenia is a sign of personal weaknessBeliefs that people with schizophrenia show a lack of strength and cannot sustain pressure“...A mentally weak person is so vulnerable to schizophrenia...”39 (0.77)
    People with schizophrenia are dangerousBeliefs that people with schizophrenia are likely to cause harm or injury“When you need to talk to a person with schizophrenia, it is very important to pay attention to your safety!”1354 (26.86)
    People with schizophrenia are unpredictableBeliefs that people with schizophrenia behave in a way that cannot be not easily predicted“People with schizophrenia can suddenly turn crying into laughing (still with snots and tears of sadness)...”2074 (41.14)
    Schizophrenia is not a real medical illnessBeliefs that schizophrenia is a made up, rather than a medical disease“Schizophrenia is really just an excuse to get a lesser sentence or to get out of prison”21 (0.42)
    People with schizophrenia could snap out of it if they wantedBeliefs that people with schizophrenia can recover from their illness at will“As you pray to the Buddha, your schizophrenia will be recovered soon...”13 (0.26)
    People would not tell anyone if they had schizophreniaBeliefs that people should be ashamed of their own schizophrenia“...Schizophrenia is God\'s punishment for family sins...”115 (2.28)
    People with schizophrenia are glorifiedBeliefs that schizophrenia is a sign of noble souls or a quality of being graceful“I think those with schizophrenia are charming...”281 (5.57)
    People with schizophrenia are self-centeredBeliefs that people with schizophrenia only think of their own advantage“...Schizophrenia is the same as narcissism.”48 (0.95)
    People with schizophrenia are weirdBeliefs that people with schizophrenia behave in an unsettling way that is strikingly odd or unusual“...Someone is throwing money like paper towels. Could this person have schizophrenia?”929 (18.43)
    People with schizophrenia are stupidBeliefs that people with schizophrenia are silly or unwise“The schizophrenia patients, the idiot people”141 (2.80)

    The coding results for stigma associated with depression showed that 6.09% (967/15,879) of depression-related posts were labeled as stigmatizing posts. Further details on coding results can be found in a previous study [21].

    The proportion of posts indicating depression-related stigma (967/15,879) was significantly lower than that indicating schizophrenia-related stigma (5041/19,224) (χ21=2484.64, P<.001).

    Differentiating Schizophrenia-Related Stigma From Nonstigma

    A total of 13 key features were selected for data modeling (Multimedia Appendix 2). Within each predicted class (schizophrenia-related stigma and nonstigma), there existed 5041 posts. The LMT model had the best classification performance (precision=.89, recall=.89, F measure=.89) (Table 3).

    Table 3. Performance of classification models.
    ModelsStigma / nonstigmaUnpredictable / other subcategoriesDangerous / other subcategoriesDepression / schizophreniaUnpredictable (depression) / unpredictable (schizophrenia)Glorified (depression) / glorified (schizophrenia)
    Support vector machine

    		Precision.70.72.76.80.93.65
    Recall.70.72.76.79.92.65
    F measure.70.72.76.79.92.65
    Naïve Bayes

    		Precision.65.67.69.67.88.67
    Recall.60.64.66.65.88.65
    F measure.57.62.64.64.88.64
    Multilayer perceptron neural network

    		Precision.67.69.71.75.88.71
    Recall.67.69.71.75.88.71
    F measure.67.69.71.75.88.71
    Logistic model trees

    		Precision.89.70.75.78.91.65
    Recall.89.70.75.77.91.65
    F measure.89.70.75.77.91.65

    Differentiating a Certain Subcategory of Schizophrenia-Related Stigma From Other Subcategories

    A total of 28 and 27 key features were selected for building two subgroups of classification models respectively: one for differentiating between unpredictable stigma and other subcategories, and one for differentiating between dangerous stigma and other subcategories (Multimedia Appendix 2). For the first subgroup, within each predicted class (unpredictable stigma and other subcategories), there existed 2074 posts, while for the second subgroup, within each predicted class (dangerous stigma and other subcategories), there existed 1354 posts. For both of the two subgroups, the SVM model had the best classification performance (subgroup 1: precision=.72, recall=.72, F measure=.72; subgroup 2: precision=.76, recall=.76, F measure=.76) (Table 3).

    Differentiating Schizophrenia-Related Stigma From Depression-Related Stigma

    A total of 30 key features were selected for data modeling (Multimedia Appendix 2). Within each predicted class (depression- and schizophrenia-related stigma), there existed 967 posts. The SVM model had the best classification performance (precision=.80, recall=.79, F measure=.79) (Table 3).

    Differentiating a Certain Subcategory of Schizophrenia-Related Stigma From the Corresponding Subcategory of Depression-Related Stigma

    A total of 52 and 19 key features were selected for building two subgroups of classification models, respectively: one for differentiating the expression of unpredictable stigma between depression and schizophrenia, and one for differentiating the expression of glorified stigma between depression and schizophrenia (Multimedia Appendix 2). For the first subgroup, within each predicted class (unpredictable stigma related to depression and that related to schizophrenia), there existed 380 posts, while for the second subgroup, within each predicted class (glorified stigma related to depression and that related to schizophrenia), there existed 114 posts. Furthermore, for the first subgroup, the SVM model had the best classification performance (precision=.93, recall=.92, F measure=.92), while for the second subgroup, the MPNN model had the best classification performance (precision=.71, recall=.71, F measure=.71) (Table 3).


    Principal Findings

    According to our knowledge, this is the first study to investigate the psycholinguistic characteristics of schizophrenia-related stigma on social media and characterize psycholinguistic differences in the expression of stigma between two different mental illnesses (depression and schizophrenia). The findings of this study have a number of implications for the detection and reduction of stigma associated with schizophrenia on social media.

    First, it is necessary for campaigns to reduce stigma associated with schizophrenia on social media. Results showed that schizophrenia-related stigma was prevalent on Chinese social media. According to this study, 26.22% of relevant Weibo posts indicated stigmatizing attitudes toward schizophrenia, which is higher than that reported by previous studies on Twitter (5%-9.7%) [17,30]. Such inconsistency may be partly due to the role of anonymity in increasing the likelihood of users to express stigmatizing attitudes on Sina Weibo (an anonymous social media site) rather than on Twitter (an open public forum), and partly due to a low level of schizophrenia literacy among people in China [31-33]. In addition, stigmatization of schizophrenia was significantly higher than stigmatization of depression (χ21=2484.64, P<.001), suggesting the essential role of schizophrenia-related stigma in antistigma campaigns on social media. Moreover, people with schizophrenia were more frequently perceived as unpredictable and dangerous (41.14% and 26.86%), which should be the targets of stigma reduction campaigns. It is worth noting that these results are largely consistent with relevant studies [2,24,26,34-38], supporting the use of social media to monitor mental illness–related stigma.

    Second, the use of psycholinguistic analysis methods facilitates automatic detection of stigma toward schizophrenia on social media. Results showed that, by using psycholinguistic analysis methods, the best performance for detecting schizophrenia-related stigma and its subcategories ranged from .72 to .89 (F measure). Compared with the results of other studies (F measure=.66 to .86) [21,22], this performance is satisfactory. It is worth noting that, in this study, no single algorithm achieved the best performance in all six classification tasks (Table 3). Although it is still unclear why the performance of algorithms varied considerably across tasks, the results of this study imply that it is necessary to select appropriate algorithms for solving different tasks.

    More importantly, the use of psycholinguistic analysis methods may provide insight into the ways in which schizophrenia-related stigma is presented on social media. Multimedia Appendix 2 showed that the expression of schizophrenia-related stigma was associated with increased use of words related to social processes (eg, mate), humans (eg, adult), death (eg, kill), and anger (eg, hate). Such language use patterns indicate a preference for more social comparisons and a higher level of negative emotion, which may fit into two elements of stigma processes, including cognitive separation (comparisons between people with and without a stigmatizing label) and emotional reactions (negative emotional reactions to people with a stigmatizing label) [39]. In addition, there existed psycholinguistic differences between subcategories of schizophrenia-related stigma as well. For example, compared with other subcategories, the expression of unpredictable stigma was associated with more frequent use of words related to cognitive processes (eg, ought) and personal pronouns (eg, them), which may be due to sustained confusion about patient behavior and the misbelief that people with schizophrenia have multiple personalities. Unlike unpredictable stigma, the expression of dangerous stigma was associated with an increased use in words related to death (eg, kill) and health (eg, clinic), which may be because of the expectation that patients with schizophrenia are likely to cause harm or injury.

    Third, the development of an accurate tool for measuring stigma should be disease-specific. Results showed that stigma associated with schizophrenia can be distinguished from depression-related stigma in terms of psycholinguistic style. These significant differences existed not only at the level of general stigma (depression- or schizophrenia-related stigma as a whole), but also at the level of stigma subcategories (eg, unpredictable and glorified stigma) (Multimedia Appendix 2). The classification performance of the corresponding models ranged from .71 to .92 (F measure) (Table 3). These results can be explained by the fact that the reason for good classification performance may not be solely attributed to differences in amount and distribution of stigma subcategories between schizophrenia and depression, but also to the actual existence of differences in psycholinguistic style between stigma related to schizophrenia and depression. Therefore, in order to improve stigma detection performance, it is necessary to develop disease-specific measurement tools.

    Limitations

    There are a number of limitations. First, it is unknown whether all relevant posts can be searched by keywords that were used in this study. Therefore, it is unclear whether any additional language use patterns are associated with stigmatizing expressions. In addition, it is uncertain whether the current keywords selection may bias estimations of the number of stigmatizing posts. Second, social media users are not representative of the general population. For example, in China, Sina Weibo users are more likely to be aged between 20 to 29 years old, well educated, and located in urban areas [40]. Therefore, the findings of this study might have limited generalizability. Third, in this study, all analyzed social media posts were written in Chinese. Therefore, it is uncertain whether the findings of this study will be applicable to other languages. Fourth, although all established models were validated using the method of cross-validation, they should be further tested on other data sets in the future.

    Conclusions

    In this study, a nonintrusive method was used to collect and analyze data under nonexperimental conditions. As a result, the current research should have high ecological validity and could investigate actual attitudes toward people with schizophrenia. The results of this study may facilitate the automatic detection of stigma on social media and improve social media campaigns related to stigma reduction.

    Acknowledgments

    The authors thank RY Li for inspiring this study. This study was supported by the National Social Science Fund of China (grant number: 16AZD058) and the National Natural Science Foundation of China (grant number: 31700984). The funders played no role in this research.

    Conflicts of Interest

    None declared.

    ‎
    Multimedia Appendix 1

    Coding framework for schizophrenia-related stigma.

    PDF File (Adobe PDF File), 91 KB
    ‎
    Multimedia Appendix 2

    Key features selected for data modeling.

    PDF File (Adobe PDF File), 240 KB
    1. Stuart H. Fighting the stigma caused by mental disorders: past perspectives, present activities, and future directions. World Psychiatry 2008 Oct;7(3):185-188 [FREE Full text] [Medline]
    2. Robinson P, Turk D, Jilka S, Cella M. Measuring attitudes towards mental health using social media: investigating stigma and trivialisation. Soc Psychiatry Psychiatr Epidemiol 2019 Jan;54(1):51-58. [CrossRef] [Medline]
    3. Sartorius N, Schulze H. Reducing the Stigma of Mental Illness: A Report from a Global Association. Cambridge, England: Cambridge University Press; 2005:152-171.
    4. Gaebel W, Zäske H, Baumann AE, Klosterkötter J, Maier W, Decker P, et al. Evaluation of the German WPA “Program against stigma and discrimination because of schizophrenia — Open the Doors”: results from representative telephone surveys before and after three years of antistigma interventions. Schizophr Res 2008 Jan;98(1-3):184-193. [CrossRef] [Medline]
    5. Razali SM. Prevention of relapse and interventions for enhancing medication adherence in schizophrenia: an East Asian perspective. Asia Pac Psychiatry 2010 May;2(2):68-76. [CrossRef]
    6. Corrigan PW, Watson AC, Gracia G, Slopen N, Rasinski K, Hall LL. Newspaper stories as measures of structural stigma. Psychiatr Serv 2005 May;56(5):551-556. [CrossRef] [Medline]
    7. Dietrich S, Heider D, Matschinger H, Angermeyer MC. Influence of newspaper reporting on adolescents' attitudes toward people with mental illness. Soc Psychiatry Psychiatr Epidemiol 2006 Apr;41(4):318-322. [CrossRef] [Medline]
    8. Morgan AJ, Jorm AF. Recall of news stories about mental illness by Australian youth: associations with help-seeking attitudes and stigma. Aust N Z J Psychiatry 2009 Sep;43(9):866-872. [CrossRef] [Medline]
    9. McGinty EE, Webster DW, Jarlenski M, Barry CL. News media framing of serious mental illness and gun violence in the United States, 1997-2012. Am J Public Health 2014 Mar;104(3):406-413. [CrossRef] [Medline]
    10. Niederkrotenthaler T, Reidenberg DJ, Till B, Gould MS. Increasing help-seeking and referrals for individuals at risk for suicide by decreasing stigma: the role of mass media. Am J Prev Med 2014 Sep;47(3 Suppl 2):S235-S243. [CrossRef] [Medline]
    11. Koike S, Yamaguchi S, Ojio Y, Ohta K, Ando S. Effect of name change of schizophrenia on mass media between 1985 and 2013 in Japan: a text data mining analysis. Schizophr Bull 2016 May;42(3):552-559 [FREE Full text] [CrossRef] [Medline]
    12. Betton V, Borschmann R, Docherty M, Coleman S, Brown M, Henderson C. The role of social media in reducing stigma and discrimination. Br J Psychiatry 2015 Jun;206(6):443-444 [FREE Full text] [CrossRef] [Medline]
    13. Christakis NA, Fowler JH. The spread of obesity in a large social network over 32 years. N Engl J Med 2007 Jul;357(4):370-379. [CrossRef] [Medline]
    14. Bond RM, Fariss CJ, Jones JJ, Kramer ADI, Marlow C, Settle JE, et al. A 61-million-person experiment in social influence and political mobilization. Nature 2012 Sep;489(7415):295-298 [FREE Full text] [CrossRef] [Medline]
    15. Valente TW. Network interventions. Science 2012 Jul;337(6090):49-53. [CrossRef] [Medline]
    16. Fu K, Cheng Q, Wong PWC, Yip PSF. Responses to a self-presented suicide attempt in social media: a social network analysis. Crisis 2013 Jan;34(6):406-412 [FREE Full text] [CrossRef] [Medline]
    17. Reavley NJ, Pilkington PD. Use of Twitter to monitor attitudes toward depression and schizophrenia: an exploratory study. PeerJ 2014 Oct;2:e647 [FREE Full text] [CrossRef] [Medline]
    18. Li A, Huang X, Hao B, O'Dea B, Christensen H, Zhu T. Attitudes towards suicide attempts broadcast on social media: an exploratory study of Chinese microblogs. PeerJ 2015 Sep;3:e1209 [FREE Full text] [CrossRef] [Medline]
    19. O'Dea B, Achilles MR, Larsen ME, Batterham PJ, Calear AL, Christensen H. The rate of reply and nature of responses to suicide-related posts on Twitter. Internet Interv 2018 Sep;13:105-107 [FREE Full text] [CrossRef] [Medline]
    20. Pennebaker JW, Mehl MR, Niederhoffer KG. Psychological aspects of natural language use: our words, our selves. Annu Rev Psychol 2003;54:547-577. [CrossRef] [Medline]
    21. Li A, Jiao D, Zhu T. Detecting depression stigma on social media: a linguistic analysis. J Affect Disord 2018 May;232:358-362. [CrossRef] [Medline]
    22. Li A, Huang X, Jiao D, O'Dea B, Zhu T, Christensen H. An analysis of stigma and suicide literacy in responses to suicides broadcast on social media. Asia Pac Psychiatry 2018 Mar;10(1):e12314. [CrossRef] [Medline]
    23. Li L, Li A, Hao B, Guan Z, Zhu T. Predicting active users' personality based on micro-blogging behaviors. PLoS One 2014 Jan;9(1):e84997 [FREE Full text] [CrossRef] [Medline]
    24. Griffiths KM, Nakane Y, Christensen H, Yoshioka K, Jorm AF, Nakane H. Stigma in response to mental disorders: a comparison of Australia and Japan. BMC Psychiatry 2006 May;6:21 [FREE Full text] [CrossRef] [Medline]
    25. Griffiths KM, Christensen H, Jorm AF, Evans K, Groves C. Effect of web-based depression literacy and cognitive-behavioural therapy interventions on stigmatising attitudes to depression: randomised controlled trial. Br J Psychiatry 2004 Oct;185(4):342-349 [FREE Full text] [CrossRef] [Medline]
    26. Wood L, Birtel M, Alsawy S, Pyle M, Morrison A. Public perceptions of stigma towards people with schizophrenia, depression, and anxiety. Psychiatry Res 2014 Dec;220(1-2):604-608. [CrossRef] [Medline]
    27. Chen M, Lawrie S. Newspaper depictions of mental and physical health. BJPsych Bull 2017 Dec;41(6):308-313 [FREE Full text] [CrossRef] [Medline]
    28. Zhao N, Jiao D, Bai S, Zhu T. Evaluating the validity of Simplified Chinese version of LIWC in detecting psychological expressions in short texts on social network services. PLoS One 2016 Jun;11(6):e0157947 [FREE Full text] [CrossRef] [Medline]
    29. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977 Mar;33(1):159-174. [Medline]
    30. Passerello GL, Hazelwood JE, Lawrie S. Using Twitter to assess attitudes to schizophrenia and psychosis. BJPsych Bull 2019 Aug;43(4):158-166 [FREE Full text] [CrossRef] [Medline]
    31. Wong DFK, He X. Schizophrenia literacy among Chinese in Shanghai, China: a comparison with Chinese-speaking Australians in Melbourne and Chinese in Hong Kong. Aust N Z J Psychiatry 2011 Jul;45(7):524-531. [CrossRef] [Medline]
    32. Gong AT, Furnham A. Mental health literacy: public knowledge and beliefs about mental disorders in mainland China. Psych J 2014 Jun;3(2):144-158. [CrossRef] [Medline]
    33. Chen S, Wu Q, Qi C, Deng H, Wang X, He H, et al. Mental health literacy about schizophrenia and depression: a survey among Chinese caregivers of patients with mental disorder. BMC Psychiatry 2017 Mar;17:89 [FREE Full text] [CrossRef] [Medline]
    34. Angermeyer MC, Matschinger H. Public beliefs about schizophrenia and depression: similarities and differences. Soc Psychiatry Psychiatr Epidemiol 2003 Sep;38(9):526-534. [CrossRef] [Medline]
    35. Marie D, Miles B. Social distance and perceived dangerousness across four diagnostic categories of mental disorder. Aust N Z J Psychiatry 2008 Feb;42(2):126-133. [CrossRef] [Medline]
    36. Björkman T, Angelman T, Jönsson M. Attitudes towards people with mental illness: a cross-sectional study among nursing staff in psychiatric and somatic care. Scand J Caring Sci 2008 Jun;22(2):170-177. [CrossRef] [Medline]
    37. Angermeyer MC, Millier A, Rémuzat C, Refaï T, Toumi M. Attitudes and beliefs of the French public about schizophrenia and major depression: results from a vignette-based population survey. BMC Psychiatry 2013 Nov;13:313 [FREE Full text] [CrossRef] [Medline]
    38. Yilmaz E, Okanli A. The effect of internalized stigma on the adherence to treatment in patients with schizophrenia. Arch Psychiatr Nurs 2015 Oct;29(5):297-301. [CrossRef] [Medline]
    39. Link BG, Yang LH, Phelan JC, Collins PY. Measuring mental illness stigma. Schizophr Bull 2004 Jan;30(3):511-541. [CrossRef] [Medline]
    40. China Internet Network Information Center. 2016 research report on behaviors of social media users in China   URL: http://www.cnnic.cn/hlwfzyj/hlwxzbg/sqbg/201712/P020180103485975797840.pdf [accessed 2019-09-26]

    ‎
    API: application programming interface
    LIWC: Linguistic Inquiry and Word Count
    SVM: support vector machine
    NB: naïve Bayes
    MPNN: multilayer perceptron neural network
    LMT: logistic model trees

    Edited by G Eysenbach; submitted 02.10.19; peer-reviewed by S Lawrie, Y Zhang; comments to author 09.11.19; revised version received 13.11.19; accepted 07.02.20; published 21.04.20

    Copyright

    ©Ang Li, Dongdong Jiao, Xiaoqian Liu, Tingshao Zhu. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 21.04.2020.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.