Predicting Emerging Themes in Rapidly Expanding COVID-19 Literature With Unsupervised Word Embeddings and Machine Learning: Evidence-Based Study

Introduction

The COVID-19 pandemic is a global health threat and has proven to be an enigma, with its diverse clinical presentation, controversial evidence for treatment, fast-tracked vaccine development, and unclear systemic implications. Most countries have been affected by COVID-19, with around 187 million confirmed cases over a short span and more than 4 million deaths recorded until July 13, 2021 [1]. The literature around COVID-19 is growing exponentially, with more than 150,000 COVID-19 articles vetted by the World Health Organization (WHO) [2]. Understanding evolving themes in a context, such as COVID-19, is essential as knowledge synthesis from peer-reviewed literature becomes increasingly difficult for researchers, clinicians, and policymakers alike. Methods, such as topic modeling and sentiment analysis, have been previously carried out comparing preprint with peer-reviewed literature only over a short period. Ebadi et al [3] studied the temporal patterns of sentiments and the similarity between publications from different sources over time, using document embeddings. High-level research topics like oncology, personal protective equipment, analytics, rehabilitation panic, high-risk groups, and genomics were uncovered using structural topic modeling. Although such analyses reflect an abstract overview of the broad areas of research, they do not capture the evolving context between distinct domain-specific entities. The objective of our study was to analyze and track word-level semantic similarity among biomedical entities to uncover emerging themes.

Abstracts of articles hold a substantial amount of information in the literature. Named entities within abstracts play a crucial role in deducing valuable information from large amounts of text and influencing literature trends [4]. Models pretrained on biomedical, scientific, and clinical benchmark data sets have been used to extract various clinical entities, such as diseases, symptoms, chemicals, and adverse drug reactions, from continuous text. The relative context of these entities changes over time, leading to a shift in similarity with other words [5]. Unsupervised word embeddings have previously been used to capture complex science concepts using the semantic relationship signified by cosine similarity [6].

Predicting links between “medical terms” is of high significance to understand the underlying themes within the literature and the phenomenon. Link prediction is the task of predicting the existence of links between 2 nodes in a complex network based on a set of topological features. The problem of link prediction in real-world temporal networks has been explored a lot in recent years [7], primarily in online social media networks where nodes are represented by users and edges are represented by the relationship between them. Supervised learning methods based on topological proximity measures have been vastly used to capture the shifting of links across time within networks [8,9]. Our paper aims to fill these gaps through our proposed framework, EvidenceFlow [10], an interactive web application for tracking literature trends using alluvial diagrams, projection of influential entities, and network analysis across different months. We propose for the first time the use of diachronic word embeddings, link prediction in dynamic networks of entities, and machine learning to predict emerging theme literature and make these publicly available as a web application. This paper also studies the evolution of literature based on changing cosine similarity between extracted entities in weighted temporal networks and predicts future emerging trends using link prediction.

We have primarily focused on the fast emerging COVID-19 literature to train and validate our architecture for this study. We forecasted semantic and topological proximity features of named entity pairs generated from their temporal trends in prior months. Further, we used these forecasted features to predict links between clinical entities extracted from textual data over the forecasted time interval using machine learning algorithms. Furthermore, these links were used to create a network weighted by forecasted cosine similarity for detecting communities of entities that tend to reflect on the themes of the articles published in that month. To assess the efficacy of our predictive modeling, we validated the proximity features of entity pairs forecasted from autoregressive integrated moving average (ARIMA) using mean squared error (MSE). We also evaluated the machine learning algorithm’s performance for predicting the links over a time span of 3 months.

The schematic representation of workflow has been demonstrated (Figure 1). The interactive analysis and results of emerging themes are available publicly on our web application called EvidenceFlow. The details about its working can also be found in Multimedia Appendix 1. This study proposes a framework for capturing and tracking imminent themes formed by medical entities in the temporal space based on networks constructed using word embeddings trained upon the evolving COVID-19 literature.

‎View this figure

Methods

Data Set and Text Preprocessing

The data set was created from abstracts of approximately 150,000 COVID-19 articles published in the publicly available WHO Database [2] from February 2020 to June 2021 (Figure 2A). For every research article, the database contains the corresponding title, authors, source of publication, journal, database, language, type of publication, entry date, country, and full-text URL. We queried the database on all full-text articles in the English language, keeping the rest of the fields unfiltered. The frequency of articles concerning specific categories and keywords has been depicted in Multimedia Appendix 2. Formatting of text and removal of white spaces, punctuations, digits, and stop words were carried out on lower-case converted text using the Natural Language Toolkit (NLTK) package [11]. We list all the software and packages used in further analysis along with the corresponding versions and sources in Multimedia Appendix 3.

‎View this figure

Named Entity Recognition

Named entity recognition (NER) was used to extract 2 types of entities (diseases and chemicals) from the original abstracts of vetted research articles using a model pretrained on the BC5CDR corpus by SciSpacy, an open-source project for biomedical natural language processing [12]. The model identifies entities with an F1 score of 84.49% [13]. The words extracted under the category of diseases also contained symptoms, adverse effects, conditions, disorders, and syndromes. All of these are collectively referred to as diseases in the other sections. Entities were further used to create networks to study the trends through alluvial diagrams and predict links between nodes across past and upcoming months.

Unsupervised Word Embeddings

Word embeddings were trained upon the abstracts obtained from the WHO database updated with new publications and preprints as these become available every month. A low-dimensional representation (d=100) for the words present in the corpus of abstracts was learned using the Word2Vec model with the skip-gram algorithm and a fixed window size of 5, implemented in Gensim [14-16]. Cosine distance between the word vectors of the extracted entities was calculated to analyze the dis(similarity) between entity pairs. Visualization of the word vectors was carried out using TensorFlow Embedding Projector [17] to allow interactive exploration of the relationships between diseases and chemicals. To create each month’s network of entities, separate Word2Vec models were trained to capture shifts in word similarities in the literature published over time.

Longitudinal Entity Networks and Communities

High cosine similarity represents strong relationships between words. We used diachronic word embeddings to capture the evolving contextual similarities between various diseases and studied the evolution over time. Weighted networks were constructed using the similarity between word vectors of extracted entities as edge weights. From each month’s corpus of abstracts, top N (=100) most frequently occurring diseases were extracted, and pairs having greater than the 90th percentile of cosine similarity based on the corresponding month’s word embeddings were used to create a union set of entities across months, preserved as nodes in the temporal networks. Therefore, every month’s network had a fixed set of nodes with varying links, labeled as 0 or 1 based on the threshold of cosine similarity, and varying weights, calculated based on the evolving semantic closeness. The mentioned threshold has been chosen empirically based on experimentation; a high threshold has been selected to depict contextual similarity between 2 words present in the same latent space. For training and evaluation, a fixed set of entity pairs was created from the diseases identified in the abstracts of the papers published from February 2020 to February 2021, using the mentioned procedure. For the subsequent months, the word embedding models were trained on the respective corpora of abstracts, and the links between the fixed set of node pairs were assigned if they appeared in the vocabulary and were weighted by the cosine similarity between their word vectors. Community detection was performed over the monthly networks using the Infomap algorithm [18]. Semantic change in the word embeddings led to the formation of communities, which shifted as emerging themes over months. The importance of each node (entities) was tracked using an alluvial visualization based on PageRank values, which changed across different months [19]. Detailed steps with parameters are available in Multimedia Appendix 1.

Time Series Forecasting of Proximity Scores

In order to predict the existence of links between nodes in the networks of subsequent months, we computed 5 neighborhood proximity scores for the network of each month. Jaccard similarity, common neighbors, preferential attachment [20], and Adamic Adar similarity [21] were used as topology-based features, and cosine similarity between the entities represented by the nodes was used as a semantic feature. These proximity scores based upon network topology were calculated using the NetworkX package [22]. Adamic Adar similarity, common neighbors, and preferential attachment values lie between 0.00 and ∞, while Jaccard similarity and cosine similarity values lie between 0.00 and 1.00. To scale the values, we normalized the former 3 scores in each network to bring them in the range of 0.00 to 1.00.

Every proximity score was modeled as a time series for each node pair, and the value was predicted for the subsequent month using the ARIMA model [23]. Stationarity of the time series was assessed using the augmented Dickey-Fuller test. A first-order autoregressive model (p=1, d=0, q=0) was used for stationary series, and nonstationary time series were passed through the random walk order of the model (p=0, d=1, q=0). For validation, proximity scores for the network at timestamp τ+1 were predicted based on their respective past values in the networks till timestamp τ. The model’s performance was assessed by comparing the predictions with the original proximity scores in the τ+1 time using MSE. MSE is one of the robust indicators to measure the closeness of forecast outputs to actual values in the time-series setting. To assess its sensitivity to outliers, we analyzed the distribution of errors (Multimedia Appendix 4). It was seen that the median of errors was close to zero, with minimal influence from outliers. Detailed steps with parameters are available in Multimedia Appendix 1.

Link Prediction Between Entities

The proximity scores predicted using the ARIMA model were further used to identify the occurrence of a link between entities in network G_𝜏+1 based on the proximity scores and links in all previous networks (G₁, G₂, G₃, …, G_𝜏), using supervised machine learning. We experimented with the proposed link prediction approach using logistic regression [24], random forest [25], support vector machine [26], AdaBoost [27], and XGBoost [28]. For training the models, 4 proximity scores (Jaccard coefficient, preferential attachment, Adamic Adar index, and common neighbors) were used as features of node pairs at each timestamp till 𝜏. For validation, the forecasted proximity scores of the network at timestamp 𝜏+1 were used to predict links between nodes. Due to the high imbalance between the labels, the area under the receiver operating characteristic curve (AUROC) was evaluated to select the optimal threshold for binary classification. While training, validating, and testing the model, we did not use cosine similarity as a feature as it was the identifier variable for the link. Validation of the model was performed on the predicted proximity scores of April 2021 to June 2021. For logistic regression, evaluation of the key assumptions was done using the variance inflation factor for measuring the degree of multicollinearity, the Cook distance for detecting the presence of strongly influential outliers, and the scatter plot of log-odds for checking the linearity of independent variables. These tests were not satisfied for the data of most months; hence, logistic regression was not our preferred model, and we did not consider it further in the results. The Welch t test was performed for comparing the performance of the machine learning models, followed by Bonferroni correction [29]. The full details of the algorithm and features are available in Multimedia Appendix 1. We list the parameters set for all the models in Multimedia Appendix 5.

Community Detection on Predicted Networks

The links between node pairs predicted by the best performing model were used to create networks weighted by cosine similarity scores predicted by the ARIMA model. The Infomap algorithm was applied on the predicted and original test network to cluster the nodes into 10 modules. The modules were compared using intersection over union (IOU) with the following formula:

where A represents a set of nodes in the predicted ith module, i ∊ {1, 2, …, 10}, and B represents a set of nodes in the original jth module, j ∊ {1, 2, …, 10}.

Results

Overall, 46,885 distinct diseases and 53,375 unique chemicals were identified. The top entities are shown in Figure 2C and 2D. Anxiety, depression, and hypertension were found to be present in the top 20 most discussed medical conditions in the research articles. Oxygen and hydroxychloroquine were followed by nucleic acid and angiotensin, a peptide hormone that causes vasoconstriction, among the most discussed chemicals. The latent space of word embeddings around the keyword “post-covid syndrome” visualized using a t-distributed stochastic neighbor embedding plot (Figure 2B) depicted “chronic fatigue,” “debilitating,” “neurodegenerative disorders,” and “vascular complications” among the closest medical entities in terms of cosine distance. Similar visualization for the term “mental disorders” can be found in Multimedia Appendix 6, and the top 10 most similar entities with the selected keywords “vaccine,” “comorbidity,” “adverse effects,” “social,” and “psychological” can be found in Multimedia Appendix 7.

We conducted detailed inference of the alluvial diagram across different months to graphically explore the temporal trends in the literature based on dynamic and homogeneous networks of prevalent medical entities and their associated cosine similarities. Figure 3A represents the flow of themes found in the literature published in 2020. For March 2020, the dominant themes noted were chest pain, acute kidney injury, and lymphocytopenia. While there were lesser traces of “thromboembolic complications” in the literature of early months, it emerged as the most significant theme in August 2020 (Figure 3A). Myocardial injury and cardiovascular diseases surfaced as a crucial cluster of entities in December 2020. Mental health factors, such as depression, loneliness, anxiety, and burnout, gained significance in the literature of the last quarter of 2020. Figure 3B presents the flow of themes found in the literature published in 2021. While thromboembolism, hypoxemia, and myocardial infarction remained major concerns till January 2021, a significant transition toward long COVID symptoms was found as a major theme in March 2021. In June 2021, central modules, including posteffects and neurological complications, stroke, headache, and anosmia, were found to gain importance, along with newer themes around immunocompromised and chronic diseases. Cross-infection–related entities gained focus due to the second wave of COVID-19 cases in multiple countries around the world. The importance of mental health effects transitioned from lesser importance in the first quarter to more emerging and prominent links in the second quarter as highlighted in the alluvial diagram (Figure 3).

We further advanced the analysis of trends to predicting links between entity pairs for the upcoming months. Our proposed framework for temporal link prediction effectively forecasted 5 proximity scores, including semantic and topological measures, between node pairs by modeling the time series using the ARIMA model. The MSE in the prediction of each proximity score for April 2021, May 2021, and June 2021 is shown in Figure 4A (Multimedia Appendix 8). The associations between diseases for the successive month were predicted as links, using supervised learning based on dynamic networks belonging to the previous months. Our results showed that among the 4 classifiers (Multimedia Appendix 9), the AdaBoost model with 50 estimators and a learning rate of 0.1 classified links with a mean AUROC of 0.871 (all P<.001; statistically significant at a Bonferroni-corrected significance level of .02) in the test data of June 2021 (Figure 4B and 4C). Comparisons among other classifiers are shown in Multimedia Appendix 10. The predicted links weighted by forecasted cosine similarity showed a high intersection with the original modules, hence validating the proposed architecture. Multimedia Appendix 11 shows the clusters detected in the original network versus the predicted network. The ARIMA model was used for forecasting proximity scores for subsequent months based on the trends in node pair proximity measures retrieved from the prior months (February 2020 to June 2020). Our findings suggest that the themes of predisposing conditions and risk factors, and studies on cross-infection and neuropsychiatric manifestation will assume a higher centrality in the upcoming quarter of 2021 (Multimedia Appendix 12).

The intersection of nodes between the predicted and original modules was analyzed to prospectively validate the effectiveness of the proposed prediction framework. Table 1 depicts the top nodes in the different modules along with their respective IOU scores for January and June 2021. The collection of intersecting nodes has been interpreted to represent broad themes. Organ damage, like acute kidney injury and pulmonary embolism associated with COVID-19, was the most central theme in the literature from January 2021, followed by cardiovascular diseases, respiratory infections, and psychological effects. Interestingly, major themes in June 2021 shifted toward conditions related to long COVID and neurological symptoms. Headache, encephalitis, and confusion were predicted to be the central nodes, and showed a high IOU score when compared with the original network. The percentages of articles published in June 2021 mentioning entities from each module for the actual and predicted networks are presented in Multimedia Appendix 13. A subset of nodes belonging to different modules from both predicted and true networks has been presented in Multimedia Appendix 11.

Analysis of networks constructed upon chemical entities revealed the evolution of various drugs studied in the COVID-19 literature. During February 2020, the major module contained entities such as paracetamol, tofacitinib, thalidomide, vitamins, zinc, and other linked chemicals. Another relevant module included central entities, such as doxycycline, ruxolitinib, heparin, and ivermectin, which were discussed in the scientific research on the treatment and prevention of COVID-19. In contrast, our recently updated models showed the emergence of evidence for various immunosuppressive drugs, such as tacrolimus, and anti-inflammatory drugs, such as glucocorticoids and colchicine, during November 2021 (Multimedia Appendix 14). These relatively less important entities in earlier months started to become more prominent as the literature expanded. Evidence around “statins” also gained centrality over recent months. Our findings show that the proposed framework captures the dynamic changes in the importance of entities based on their evolving relationship with neighboring entities.

‎View this figure

‎View this figure

Discussion

Principal Findings

In this paper, we demonstrate a computational approach, EvidenceFlow, in which a user interacts with the rapidly expanding COVID-19 literature to derive and predict emerging themes. The proposed framework tracks patterns of changing semantic and topological proximity between entity pairs across months. Further, it predicts links and network communities that may emerge in future months. Hence, users can follow the papers that contribute to emerging communities of themes, for example, literature around thromboembolic complications captured as early as August 2020 and mental health factors during the end of 2020. Interacting with the clusters on the interactive interface of the EvidenceFlow model revealed that symptoms of long COVID, such as fatigue, headache, myalgia, cough, and anosmia, were forming a central cluster during March 2021. This early signal for accumulating evidence was later validated in large prospective and retrospective cohorts of COVID-19 patients [30-32]. Another way in which users can interact with EvidenceFlow is to gain an understanding of the evolution of themes going beyond current approaches such as topic modeling and sentiment tracking [3]. An example is the early finding of imminent themes around neurological complications, such as confusion, psychiatric illness, and stroke, and mental health factors, such as anxiety, depression, posttraumatic stress disorder, burnout, and insomnia, in June 2021. Our violin plot analysis (Multimedia Appendix 4) showed that despite the mean error being centered on zero, there were some outlier node pairs whose predicted associations deviated from the ground truth. The future scope of this work will involve an analysis of such associations and insights gained by an interactive analysis of such pairs on the EvidenceFlow application.

Prediction of the themes represented by rising centrality of entities can assist in the formation of promising research hypotheses. The dynamics of the literature reveal the emergence of central themes as a combination of pre-existing themes in recent times [6]. For example, the alluvial diagram (Figure 3A) demonstrated how entities from multiple modules in March 2020 merged into a major cluster of thromboembolic complications. Similarly, the flow of importance of psychological disorders over the months indicates their contemporary relevance in the COVID-19 literature and their links with other entities in the cluster. Our framework can potentially help researchers in monitoring existing themes and directing their studies based on trends and predictions.

We conducted an analysis on the trends of the PageRank centrality of selected chemical and disease entities. Statins, a class of lipid-lowering medications, were found to be gaining centrality in late 2021 as compared to earlier values (Multimedia Appendix 15). Numerous studies discussed statins for having anti-inflammatory and immunomodulatory effects that may reduce the severity of COVID-19 [33,34]. Glucocorticoids, a class of steroid hormones that reduce inflammation and suppress the immune system, also emerged as a rising entity (Multimedia Appendix 15). Depression and other mental health disorders started becoming a prominent topic of research during the middle of 2020 and gained higher importance in subsequent months (Multimedia Appendix 15). COVID-19 has also been largely discussed in the context of a thromboembolism, and our model captured its emerging evidence as a theme till late 2020. However, the trends showed that its centrality in the literature relatively decreased in 2021 (Multimedia Appendix 15). Discovering such trends from a large corpus is indeed possible using manual curation and analysis by experts. However, our EvidenceFlow pipeline provides an efficient lens to discover, track, and predict emerging trends. This framework will enable faster synthesis of evidence, which then can be validated by experts.

To explore the potential of unsupervised word embeddings and changing cosine similarity among words, we analyzed the trends of terms having maximum similarity with selected keywords. For example, we analyzed the temporal shift in the context of “vaccine” over the months by finding the top 10 terms most similar to vaccine in the latent space of word embeddings trained on the abstracts from each month (Figure 5). From February to August 2020, research on COVID-19 vaccines was underway, and the studies revolved around “therapeutics,” “prophylactics,” “drug repurposing,” and associations with the MMR (measles-mumps-rubella) vaccine and BCG (Bacillus Calmette–Guérin) vaccine. As the clinical trials of certain vaccine candidates became prominent after August 2020, the theme of vaccine hesitancy emerged in October 2020 and gained higher similarity in subsequent months. Additionally, as the literature evolved in 2021, a wide range of COVID-19 vaccines, such as BNT162b1, Pfizer-BioNTech, AstraZeneca, ChAdOx1, mRNA-1273, and Moderna, were found to be majorly discussed in the context of research on vaccines. Terms, such as immunogenicity and efficacy, further suggested high association with vaccine trials and rollouts. Recently updated models showed the emergence of “booster” doses from August 2021 onwards. Such retrospective evaluation of the development of evidence from the literature over time can assist the research community in deriving detailed insights leveraging the applications of word embeddings.

‎View this figure

Limitations

Our study has some limitations. First, although the WHO database has been built using a detailed search strategy for COVID-19 literature, it does not explicitly report the exact purpose or accuracy of the search and decision process. The documentation [35] mentions screening done by expert reviewers and an attempt to remove duplicates, but further details are lacking. For example, the process does not clarify if redundancy across various publishers was taken care of. Further, the frequent use of the “OR” combination of keywords may have led to the inclusion of less relevant articles, while other forms of literature, such as patent applications, which can add value to the study, were not included in this database. Nonetheless, we chose the WHO COVID-19 database as it provides a large collection of articles that are updated regularly from searches of multiple bibliographic databases [2]. This, combined with curated expert-referred scientific articles, which would not be readily accessible on a custom search, was useful for building the EvidenceFlow pipeline. Future work with this framework will include potential extension to databases curated through both generic queries and expert vetting, thus facilitating targeted evidence synthesis from a variety of databases.

Further, we are currently using abstracts of research articles to extract named entities and may be missing on the details contained in the full text of the articles while training word embeddings. Therefore, future work may build upon the framework to include the full text of articles wherever available. The NER model used in our study has been reported to have achieved an F1 score of 84.49% on a benchmark data set [13]. Despite the limitations of the F1 score, such as equal weightage given to precision and recall [36,37], F1 remains one of the most widely reported performance indicators. We chose this metric in the absence of other metrics reported for this NER model. For forecasting, we used a relatively basic model (autoregressive approach), as our goal was to capture robust patterns. However, further research is possible for the use of more complex time-series approaches with higher-order difference and lags. Moreover, as the number of timestamps and data points increase, advanced architectures, such as recurrent neural network and long short-term memory [38,39], can be used for handling complex trends in the time series efficiently. Further experiments with larger networks can reveal themes that were not found with the top 100 entities. Importantly, our model is supporting the early detection of emerging trends, but it cannot capture themes on which no evidence has been accumulated.

Conclusion

Consortia across the globe were formed for the advancement of research related to COVID-19. The global attention has led to a widespread increase in the scientific literature to study and prevent the disease from spreading, resulting in an understanding of the disease from multiple perspectives. We introduced a framework built upon COVID-19–specific literature vetted by the WHO and deployed as a dashboard called EvidenceFlow [10]. The dashboard allows the user to unravel the literature with an interactive map of embeddings based on the visualization provided by Tensorboard. It aims to track literature trends using alluvial diagrams, multilevel community detection, and projection of influential entities through network analysis across different months. This study presented how machine learning–based prediction of emerging links can contribute toward analyzing research by capturing themes represented by groups of medical entities, based on patterns of semantic relationships over time.