Applying AI to Structured Real-World Data for Pharmacovigilance Purposes: Scoping Review

Introduction

Background

Pharmacovigilance is defined by the World Health Organization as “the science and activities relating to the detection, assessment, understanding, and prevention of adverse effects or any other drug-related problem” [1]. Pharmacovigilance plays a crucial role in ensuring the safety of medications and protecting the health of patients because it mostly focuses on the identification of potential adverse drug reactions (ADRs) after medicinal products have been licensed and released to the public.

ADRs can range from mild and tolerable side effects to severe and life-threatening events. They constitute 5% to 7% of emergency department consultations [2]. Their impact in terms of public health is significant because there are estimates concluding that ADRs can cause an increase in the duration of hospitalization stays for outpatient (mean 9.2, SD 0.2 d) and inpatient (mean 6.1, SD 2.3 d) settings [3]. Typically, pharmacovigilance professionals analyze data from individual case safety report (ICSR) databases (such as the Food and Drug Administration Adverse Event Reporting System, the database maintained by the US Food and Drug Administration) to identify potential pharmacovigilance signals, namely potential causal relationships between an ADR and a drug. ICSRs are typically submitted either by patients or by health care or pharmacovigilance professionals, and they are the main data source used today for pharmacovigilance. However, ICSR databases are subject to many biases; in addition, underreporting has been identified as a huge issue [4]. Moreover, such databases frequently lack information that could make a significant difference in the examination of a potential signal (eg, patients’ medical history). Hence, the early detection of potential pharmacovigilance signals by collecting and analyzing data from various sources is critical to prevent serious side effects as soon as possible.

The term “real-world data” (RWD) refers to data collected outside of the controlled environment of clinical trials, such as electronic health records (EHRs), patient registries, insurance claims databases, electronic prescription systems, and so on. There is a growing interest in using RWD for pharmacovigilance signal management to facilitate faster and more efficient postmarketing surveillance [5]. The significance of RWD in pharmacovigilance lies in its potential for representing longitudinal real-world patient experiences and health care practices that can provide insights into drug safety under real-life conditions. Analyzing RWD could also enrich and consolidate the already existing knowledge on ADRs (eg, by detecting new cofounders). Indicatively, a federated RWD network was used recently to validate the value of RWD in terms of pharmacovigilance signal management [6].

To this end, the European Medicines Agency and the US Food and Drug Administration have established infrastructures for the leverage of RWD for drug safety purposes, called Data Analysis and Real World Interrogation Network (DARWIN) [7] and the Sentinel Initiative [8], respectively. RWD are also being actively investigated for purposes beyond drug safety (eg, epidemiology) [9]. It should be noted that although RWD could in principle provide a good overview of patients’ clinical course, two major challenges are preventing their use: (1) these datasets typically come with significant data quality risks and usually contain a high proportion of null values and errors; and (2) because of legal, ethical, and regulatory issues (eg, patient privacy issues), it is difficult to access these data sources.

Rationale

Artificial intelligence (AI) is widely acknowledged as a potentially very useful technical breakthrough that could be used to support decisions in health care (eg, clinical decision support systems) due to its ability to efficiently process big data to seek useful information. AI could be used to identify patterns and associations within large amounts of data (eg, RWD) where traditional statistical methods of data analysis may struggle to extract because of the amount and complexity (eg, nonlinear relationships between variables) of the data. AI has been widely investigated regarding its applications in health care (eg, personalized medicine) with promising results [10,11]; however, it is not yet widely applied in clinical practice. In the context of pharmacovigilance, AI could potentially support multiple aspects (eg, the identification of patient subpopulations who may be more vulnerable to specific ADRs), contributing to the vision of personalized drug safety management.

Objectives

The objective of this scoping review (SR) was to identify and characterize the current research trends regarding the use of AI on structured RWD for pharmacovigilance and identify relevant gaps.

Methods

The PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) [12] methodology or rationale was applied. The PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) statement is a road map for authors to describe more precisely the state of the art and the findings of the literature search, as well as discuss the results.

Eligibility Criteria

Journal and conference articles written in English were selected if they focused on pharmacovigilance and reported the use of symbolic and nonsymbolic AI approaches applied to RWD, specifically EHRs, insurance claims databases, and administrative health data (Textbox 1).

Review and opinion articles were excluded from the final manuscript selection. Furthermore, research articles focusing on image and text data (eg, social media and clinical notes) were also excluded. In addition, AI methods focusing on the use of natural language processing (NLP), natural language understanding, image processing, or object detection were considered beyond the scope of this work.

A key issue that came up during this SR was the lack of a clear distinction between plain statistical methods and machine learning (ML) approaches because these 2 domains frequently overlap, and these 2 terms are sometimes used interchangeably. In this manuscript, we acknowledge that the difference between AI and statistical methods is that AI creates models that can “learn” from data during iterative training processes, while statistical methods deal with finding relationships between variables. Thus, we considered the iterative “learning” part of an algorithm as the key feature to classify the algorithm as AI and ML. We excluded papers that were based on algorithms with no iterative “learning” scheme because we considered them to be part of the “plain statistical methods” approaches. Finally, we excluded papers that focused on adverse drug events related to medical devices.

Information Sources and Search Strategy

A search query was developed and executed on January 31, 2024, to include research articles from 2010 to 2024 exclusively from the MEDLINE scientific library, given that it is the oldest and biggest repository of journal articles in life sciences. Textbox 2 presents the query structure.

Selection Process

The initial phase (phase 1) focused on screening the titles and abstracts of the articles retrieved from the search query (Textbox 2) to map those that potentially met our inclusion criteria and exclude irrelevant studies using the Rayyan tool (Rayyan Systems Inc) [13]. Rayyan is an AI tool designed to facilitate remote collaboration among researchers when conducting systematic literature reviews. The platform gathers the titles and abstracts of all articles selected for the study, and reviewers can evaluate the eligibility (ie, “include,” “exclude,” or “maybe”) of every article based on their review’s objectives in blind mode, that is, each reviewer assesses the articles without prior knowledge of the other reviewers’ decisions. We resolved any conflicts that arose during this process through consensus meetings involving all reviewers.

The second phase focused on the full-text review of the papers selected during phase 1 to decide on the final set for inclusion in this study. In the full-text review of the studies selected based on titles and abstracts, we excluded research papers that did not meet ≥1 of the inclusion criteria (ie, strong focus on AI, RWD, and pharmacovigilance) as well as studies that met the exclusion criteria (eg, studies related to image and text data or those following only statistical approaches).

Data-Charting Process

A standard data extraction form was used to obtain an overview of the 36 selected studies (Tables S1 and S2 in Multimedia Appendix 1). For each study, we extracted information about the authors; journal name (where the study was published); publication year; country of origin (where the study was conducted); the objective of the study; types of organizations that participated in the study (based on the authors’ affiliations); and key findings that relate to the scoping review question, which are described in the next subsection (Data Collection Process and Mapping). Any inconsistencies were discussed and resolved among the reviewers.

Data Collection Process and Mapping

The selected studies were further elaborated and mapped against evaluation criteria using a spreadsheet. The main categories of mapping criteria were as follows: pharmacovigilance objectives (drug safety core activities and drug safety special topics), data provenance (data source categories and data sources), countries of origin, AI algorithm categories, data preprocessing methods, the use of explainable AI (XAI) methods, code availability, the use of models in clinical practice, ethical AI, and so on. Table 1 presents an external description of the mapping criteria.

Studying Risk-of-Bias Assessment

To effectively map the risk of bias in each included study, we considered selection, measurement, temporal, implicit, confounding and automation biases. Furthermore, we translated these categories into more specific categories according to our study (Table 1).

Synthesis Methods

The mapping strategy was designed based on the 3 main pillars of the objective; in addition, we included general information about the research papers (Table 2). Furthermore, we included free-text fields in the mapping Microsoft Excel file to add significant extra details that cannot be easily classified. These fields included “objective,” “methods,” “assessment,” and “interesting results.” The criteria encompassed specific attributes (eg, drug safety core activities) that were defined based on previous experience of conducting an SR in the field [14] and key interest aspects identified during the review.

Furthermore, in terms of ethical AI, the included studies were evaluated based on trustworthy AI guidelines for solutions in medicine and health care from the FUTURE (Fairness, Universality, Traceability, Usability, Robustness, and Explainability)-AI initiative [15]. These guidelines are separated into 7 categories (fairness, universality, traceability, usability, robustness, explainability, and a general category). For our evaluation procedure, we included only the highly recommended subcategories from each of the 7 main categories for proof of concept (low technology readiness levels for ML models) [16]. Table 3 presents the selected criteria and their description.

Reporting Risk-of-Bias Assessment

The selection of only studies written in English and the exclusion of AI studies focused on text mining or NLP, image processing, and statistical analysis could be identified as potential risks for bias. Furthermore, the selection of papers only from the MEDLINE database could be identified as a potential bias risk because it potentially leads to the omission of papers from other databases (eg, AI databases).

Results

Study Selection

The PubMed search query originally returned 4264 studies. During the abstract and title screening process (phase 1), we selected 93 (2.18%) of the 4264 articles for full-text screening (phase 2). During phase 2, based on the inclusion criteria, of these 93 research papers, 36 (39%) were selected. The PRISMA-ScR flowchart (Figure 1) presents a detailed overview of the selection procedure. The PRISMA-ScR checklist is presented in Multimedia Appendix 1.

‎

Study Characteristics

The included studies were published between 2015 and 2023, with a notable increase in the number of studies after 2019 (Table 4).

Of the 36 studies, 19 (53%) originated from the United States, 4 (11%) from Korea, and 4 (11%) from the United Kingdom, while the rest of the studies (n=9, 25%) were distributed across a variety of other countries (Table 5).

Most of the studies (30/36, 83%) were conducted from academia (Table 6).

In terms of AI, of the 36 studies, 34 (94%) applied only nonsymbolic AI, and 1 (3%) used only symbolic AI, while 1 (3%) study combined the symbolic and nonsymbolic AI technical paradigms. Of the 34 nonsymbolic AI articles, 29 (85%) used classification tasks, whereas 3 (9%) selected regression algorithms, 3 (9%) applied causality algorithms (causal inference: n=2, 67%; causal discovery: n=1, 33%), and only 1 (3%) applied an association rule mining technique. The association rule mining study [17] followed a mathematical framework called formal concept analysis to create association rules between drugs and phenotypes to detect possible ADRs. Moreover, of the 29 studies that used classification tasks, 6 (21%) used XAI techniques, of which 4 (67%) used Shapley additive explanations, 1 (17%) used local interpretable model-agnostic explanations, and 1 (17%) tested both approaches.

Regarding RWD (Table 7), of the 36 articles, 28 (78%) focused on the use of EHRs (from local hospital databases), 4 (11%) used data from pharmacy dispensing records, and 3 (8%) used administrative claims data, while 2 (6%) focused on patient registries and 1 (3%) on insurance claims. In addition, a variety of other sources were used, including RWD such as drug information databases (3/36, 8%), spontaneous reports (3/36, 8%), adverse drug event databases (2/36, 6%), electronic prescription data (2/36, 6%), and genetics and biochemical databases (1/36, 3%).

Of the 36 studies, 23 (64%) used AI for ADR detection, 4 (11%) examined ADR assessment, 2 (6%) focused on ADR monitoring, 7 (19%) investigated ADR prevention, and 2 (6%) used AI to collect information about ADRs (Table 8).

The classification studies (29/36, 81%; Table 9) tested several AI techniques, with random forest (RF) being the most frequently used algorithm (17/29, 59%). However, the regression studies (3/36, 8%) developed AI models only with extreme gradient boosting (1/3, 33%) and logistic regression (2/3, 67%).

Finally, for the evaluation of AI models (Table 10), most of the studies (24/36, 67%) reported area under the receiver operating characteristic curve as the primary metric.

Of the 36 studies, 32 (89%) investigated specific drug safety topics: 16 (50%) on specific adverse effects, 14 (44%) on specific class of drugs, 8 (25%) on specific (class of) diseases, 6 (19%) on signal detection, 3 (9%) on drug interactions, 2 (6%) on personalized drug safety, and 1 (3%) on vaccine safety (Table 11).

Table 12 presents the diversity in the data sources used in the included studies. Of the 36 studies, 29 (81%) chose proprietary closed data sources (eg, specific hospital EHRs) for their experiments. Along with EHR data, other data sources were also used (eg, Food and Drug Administration Adverse Event Reporting System and Side Effect Resource). Of the 36 studies, 2 (6%) selected the Stockholm Electronic Patient Record Corpus. The remaining RWD sources (Medical Information Mart for Intensive Care and the Osteoarthritis Initiative dataset) are represented in only 2 (6%) of the 36 studies (n=1, 50% for every database).

In terms of data models, of the 36 studies, 27 (75%) used proprietary data models, 3 (8%) did not mention any data model, 5 (14%) used the Observational Medical Outcomes Partnership Common Data Model (OMOP-CDM), and 1 (3%) used the Sentinel model (Table 13).

Figure 2 presents the case studies examined in the included articles. Notably, an important number of studies (20/36, 55%) did not work in specific ADR case studies. Another significant outcome is the diversity of case studies; the articles do not focus on a specific drug, indication, or reaction. It can be observed that chemotherapy drugs and their associated reactions in various types of cancers emerge as slightly more prominent categories in this review (Figure 3).

‎

‎

Although most of the studies (21/36, 58%) used complex AI algorithms (black boxes), such as RF (an ensemble method) and artificial neural networks (ANNs), to construct their prediction models in all ADR categories, many studies (15/36, 42%) used simple interpretable ML approaches such as logistic regression. Moreover, it is important to highlight that all studies worked on EHR databases, except for the adverse drug event assessment category in which we detected a single study with a vaccine database.

RWD databases were also used alongside other types of data; for example, EHRs were mostly combined with spontaneous reporting systems and drug information databases, vaccine data with adverse drug event databases, and administrative claims data with spontaneous reporting systems. Furthermore, some of the studies (3/36, 8%) integrated different types of observational data to develop AI models, combining pharmacy dispensing records with EHRs and administrative claims data.

Evaluation Results

Only 3 (8%) of the 36 studies included in this SR openly provided their code. In addition, only 16 (44%) of the 36 studies included a detailed description of data preprocessing pipelines for RWD. Moreover, just 4 (11%) of the 36 studies evaluated their methodology within a clinical environment (Table 14).

In terms of trustworthy AI, only 5 (14%) of the 36 studies scored <50% on the Fairness, Universality, Traceability, Usability, Robustness, and Explainability–AI (FUTURE-AI) criteria (Tables 15 and 16). Among the studies that achieved scores of >75% [18-20], 3 (75%) out of 4 used external data to evaluate their models, addressing the Universality criterion (Table S4 in Multimedia Appendix 1).

Risk of Bias in the Included Studies

Table 17 provides an overview of the distribution of biases across the included studies. Notably, of the 36 studies, 21 (58%) included selection biases, and 17 (47%) included confounding biases. Algorithmic biases were identified in 14 (39%) of the 36 studies. Table S5 in Multimedia Appendix 1 presents the detailed categorization of the included studies in the different risk-of-bias categories.

Interesting Results of Individual Studies

It is important to mention that 6% (2/36) of the studies successfully combined AI and a self-controlled case series (SCCS) model for ADR detection. Morel et al [24] introduced the convolutional SCCS (ConvSCCS) model in which the SCCS model is enriched with a convolutional neural network. This allows the ConvSCCS model to consider a few longitudinal data dimensions (eg, drug exposure) from observational data and predict a potential ADR without a prior definition of risk windows, which is mandatory in SCCS models. The ConvSCCS model was tested in glucose-lowering drugs and the risk of bladder cancer case study. Another interesting advantage shown by the results is that the ConvSCCS model is useful for analyzing high-dimensional data while requiring minimal data preprocessing. Zhang et al [22] developed the neural SCCS (NSCCS) model to detect probable drug interactions and control for time-invariant confounders. The NSCCS model was tested in the OMOP-CDM reference dataset [53]. Both the ConvSCCS and NSCCS models outperformed traditional SCCS statistical models in comparative analyses; the ConvSCCS model demonstrated superior precision and computational speed, while the NSCCS model achieved an area under the receiver operating characteristic curve score of 0.779 [22].

Furthermore, Kidwai-Khan et al [28] focused on improving the prediction of preventable adverse events by integrating in an AI decision support tool, EHRs with genetic data (the presence or absence of genes contraindicated with a person’s medication). This is the only study in this review that combined EHRs with genetic data and 1 (25%) of the 4 studies that used XAI methods. In addition, all AI models achieved high evaluation scores (>95%).

A few ADR prediction studies (3/36, 8%) introduced innovative ideas on feature preprocessing. Jeong et al [33] developed an ML prediction model in which the features are calculated from algorithms such as the “comparison of extreme laboratory test” results, “comparison of extreme abnormality ratio”, and “prescription pattern around clinical events” to help determine whether a drug-laboratory event pair is associated. A different approach was proposed by Wang et al [37] who addressed problems with low-quality observational data (eg, missing data) by creating patient embeddings and treating patients “as bags with the various number of feature-value pairs, called instances.” This method led to the development of the final AI model (AMI-Net3), which achieved exceptional performance. Chen et al [54] also proposed an embedding methodology, called “physiological signal embeddings.” This study proved that training deep embedding models on physiological signals could lead to better forecasts of adverse outcomes. In addition, this methodology enables data transferability through the physiological signal embeddings models.

Only 1 (3%) of the 36 studies developed an application for the prediction of ADRs. Mosa et al [41] leveraged the interoperability of a decision tree ML model and, based on their results, designed a rule-based mobile app to assess the risk of specific ADRs and indications.

A completely different approach to ADR prediction was introduced by Liu et al [26]. In this study, the authors applied an ML method to develop a prediction model for osteoarthritis ADRs in analgesic drugs. Afterward, the authors used explainability techniques to identify patients who might be prescribed analgesic drugs without the risk of osteoarthritis ADRs. The diversity of this study is addressed in a different scope: instead of predicting ADRs based on a patient’s medical history, the model focuses on identifying the characteristics that make the patient suitable for a medication, specifically by considering the presence or absence of an ADR.

Recently, the causal ML paradigm was introduced into pharmacovigilance through the studies of Wang et al [48] and Zhang et al [52], who applied causal inference with average treatment effects and causal discovery with directed acyclic graphs, respectively. Wang et al [48] used causal ML models to make a representation of a randomized clinical trial with EHR data. Their results successfully identified both well-known and new medications that could cause the suspected ADR in their case study. Furthermore, Zhang et al [52] created a causal graph for a drug-event combination and compared the results from 2 causal discovery algorithms. Their results showcased the causal discovery algorithms’ abilities to explore the mechanisms of the suspect drug that could lead to a potential ADR, uncovering previously unknown causal links.

Only 1 (3%) study focused on the use of symbolic AI [43] compared to those investigating the use of ML (35/36, 97%) and 1 study combine symbolic and nonsymbolic AI. Notably, Pichardo et al [43] stand out for integrating ontologies and ML, namely combining symbolic and nonsymbolic AI. The objective of this study is to examine the performance of a clinically informed framework for the prediction of short-term ADRs.

Furthermore, very few studies (5/36, 14%) focused on the clinical evaluation of the proposed ML approach. Segal et al [20] presented a clinical decision support system designed to provide medication error alerts to prevent ADRs, demonstrating significant results—40% of the prescriptions were altered based on these alerts. Herrin et al [42] compared the effectiveness of their proposed ML scheme to that of an established clinical practice, specifically the HAS-BLED approach, to evaluate a patient’s risk of gastrointestinal bleeding.

Synthesized Findings

Although the included studies followed widely used approaches in AI and pharmacovigilance to predict potential ADRs, primarily using EHR data, such as predicting ADR outcomes using well-known ML model architectures, the aforementioned studies follow fundamental methodologies. In terms of data, it describes several interesting processes such as the creation of patient embeddings and the application of the “comparison of extreme laboratory test” results, “comparison of extreme abnormality ratio”, and “prescription pattern around clinical event” algorithms for calculating input data for the ML model. Furthermore, it is described a variety of innovative AI algorithms such as SCCS models, causal ML, and symbolic AI. Finally, out-of-the-box pharmacovigilance approaches were followed, such as identifying patients suitable for specific treatments based on their ADR profiles.

Discussion

The findings of this study enabled us to identify innovative ideas, spot existing limitations, and propose potential directions for future work in this field.

Principal Findings

In summary, the number of studies on AI methodologies applied to RWD for pharmacovigilance purposes has significantly increased in the last 5 years—most of the included studies (28/36, 78%) were published after 2019, with the United States contributing the most publications in this field (19/36, 53%).

Comparing this review study with 3 recent reviews from 2022 to 2023 in the same field, we conclude that only the study by Kaas-Hansen et al [55] could be considered a study with the exact focus with our review. Their review included only 7 scientific papers because they restricted their selection to studies published between 2015 and 2021 and involving >1000 patient records. Their findings are similar to ours in terms of dominant AI solutions (classification) and the type of RWD used (EHR). Finally, it is essential to note that their review, like ours, highlights the limited adoption of widely used common data models, such as the OMOP-CDM.

In terms of risk of bias, selection biases were due to the fact that most of the studies (15/36, 42%) did not include patients from >1 data source (regional hospital [46] or insurance claims database [24]) in their models. Regarding confounding, while high-dimensional RWD offer a significant amount of information, they also contain substantial noise and null values. Sequentially, features that could be potential confounders are eliminated from the final dataset.

Pharmacovigilance

The reviewed papers focused on using ML to detect ADRs to confirm whether previously known ADRs could have been identified using RWD. Another major theme identified was the prediction of ≥1 ADRs based on the classification of patients with different characteristics, ultimately aiming to support personalized ADR prevention. However, there was a lack of studies investigating new, potential pharmacovigilance signals. Regarding the investigated ADRs, there was a slightly higher interest in chemotherapy drugs for different types of cancers due to the high incidence of serious reactions associated with this treatment.

Finally, it is important to highlight that only 4 (11%) of the 36 studies in this review were tested in real-world clinical environments, which leads us to conclude that AI models may lack generalizability or that health care professionals may lack trust in AI models. By contrast, the trustworthy AI evaluation based on the FUTURE-AI guidelines proved that only a few studies (4/36, 11%) failed to satisfy half of the criteria, indicating relatively high research quality.

RWD Preprocessing

In terms of RWD, EHRs were the most commonly used data source. EHRs are multidimensional, offering data that could be crucial for detecting postmarketing ADRs. At least in principle, EHRs could serve as an invaluable data source for investigating potential drug synergies or interactions across diverse populations. Furthermore, the variety of information in patient records could be an advantage in the creation of multimodal datasets, for example, by integrating biological, signaling pathway, and drug information databases.

However, the use of EHRs comes with significant burdens because they contain sensitive personal data, leading to limited access. Medical Information Mart for Intensive Care is the only openly available EHR dataset for researchers, but it is not commonly used in pharmacovigilance (only 2/36, 6% studies used it).

Moreover, it is important to mention that RWD preprocessing is challenging due to its complexity and real-world nature (biases, errors, gaps, noise, etc). Consequently, less than half of the articles (17/36, 47%) described in detail the data preprocessing step in their pipelines.

Another noteworthy outcome is that widely adopted data models such as the OMOP-CDM, Informatics for Integrating Biology & the Bedside, and Sentinel appeared sporadically in the studies. This could be attributed to the fact that the use of EHR data and AI models is relatively new. However, it should be noted that initiatives in this direction are emerging (eg, the Assessment of Pre-trained Observational Large Longitudinal models in Observational Health Data Sciences and Informatics initiative [56]).

Finally, it should be noted that RWD have a very substantial longitudinal dimension and questionable quality (due to gaps, errors, etc). As such, leveraging RWD for pharmacovigilance purposes requires the development of new approaches that focus on using time-related sequential information. While several attempts have been made to exploit this temporal aspect of RWD [19,33-35], validating AI and ML algorithms focusing on time-series rationale for pharmacovigilance signal detection remains a critical issue.

AI Models

The detection of potential pharmacovigilance signals is a challenging procedure. As a result, the development of ML models to support the detection of ADR signals could have a significant impact. We can outline 2 major approaches for ADR signal detection. The first focuses on creating an AI tool that could discover unknown relationships between drugs and conditions, highlighting potential causal associations. The second approach emphasizes AI pipelines tailored to specific ADRs, where the input data for the final ML model are preprocessed based on prior medical knowledge of the drug-event combination.

The AI models identified in this review are generally complex, with ensemble methods such as RF being the most commonly used. A significant number of studies also applied ANNs. As RWD contain a substantial amount of diverse information, the relationships between different features may not be linear. Hence, the use of black box models (eg, ensemble methods and ANNs) is essential for discovering more complicated associations in a dataset beyond linear relationships.

On the basis of the review papers, there is a noticeable lack of use of XAI models (ie, local interpretable model-agnostic explanations and Shapley additive explanations models). Health care professionals highlight the necessity to understand the motifs between AI models’ tasks to accept the decisions made by the algorithms. This could not only lead to the biological translation of the results based on existing knowledge but also reveal new information about a disease, a medication, and so on. In terms of pharmacovigilance, XAI models applied to RWD could bring evidence about unknown confounders in an ADR and provide more informative results for pharmacovigilance experts regarding the causes of a potential pharmacovigilance signal. Although XAI methods’ results are tested extensively in the health care domain, we found that only 6 recent studies (4/36, 11% are included in this review) had applied them in the pharmacovigilance domain [21,23,27,28,54,57] (4 in 2021, 2 in 2022). Another novel approach discussed extensively in the explainability field is the newly introduced causal ML or causal deep learning algorithm, which combines AI and causal inference to uncover underlying cause-and-effect relationships between variables. The complexity of RWD presents a challenge that causal ML could potentially solve more efficiently by providing meaningful explanations of the causal relationships between variables [58]. These innovative AI models could serve as a good hypothesis for future work because they seek and present the relationships between different variables in RWD sources with a more informative structure than traditional AI models. They have already been applied efficiently in pharmacological treatment patterns [59]. In this review, only 3 (8%) of the 36 studies applied causal deep learning to EHR data [25,48,52].

Finally, a major problem identified based on the SR findings is the lack of code availability. This issue hinders the reproducibility of the models, preventing further testing on different datasets and raising questions about the developed AI models’ robustness and generalizability.

Strengths and Limitations

The strengths of this review include the use of a considerable number of studies (n=36), providing a thorough knowledge of the specific scientific field. Besides, we compared our findings with those of the most recent review in the same field and analyzed the differences.

Nevertheless, this systematic review has several limitations. First, we only included articles from the MEDLINE database. As such, we may have excluded other existing AI approaches to structural RWD in the field of pharmacovigilance that could be available in AI databases. Second, because of the variation in the articles’ methodologies, we were unable to conduct a meta-analysis of the quantitative results. Finally, it is important to mention the limited number of symbolic AI studies in this review (4/36, 11%) [17,45,60,61]. The construction of knowledge graphs (KGs) usually requires the use of text mining procedures such as NLP and focused on real text such as clinical notes. As we excluded NLP studies from our query, we assume that this contributed to the small number of symbolic AI articles (n=36) included in our review.

Current Gaps and Potential Future Work Paths

Detecting new pharmacovigilance signals using ML approaches requires evidence of a causal association between the suspect drug and the reaction. XAI models can assist pharmacovigilance professionals in this process. To this end, further investigation into causal ML and causal deep learning approaches could be a highly impactful line of research for identifying pharmacovigilance signals from RWD.

Another gap identified in this SR that could indicate future work paths could be the use of multitask learning approaches. Multitask learning is an ML methodology that takes as input 1 dataset to execute multiple prediction tasks. RWD, such as EHR data, are rich data sources that could support >1 task (eg, pharmacovigilance and pharmacoepidemiology); for instance, a multitask learning model could predict an adverse drug event, the severity of an adverse drug event, and the likelihood of the same adverse drug event occurring with other drugs in a patient.

Furthermore, combining ML approaches with symbolic AI is a line of work that offers further potential for exploration. Combining ML with ontologies and automatic reasoning upon KGs could enable new AI approaches (eg, neurosymbolic AI) and provide new insights based on well-established expert knowledge formed as a KG. Moreover, using ontologies and KGs could support integration with other kinds of data sources (eg, data sources containing low-level biochemical or pharmacokinetics and pharmacodynamics information and signaling pathway information).

Finally, exploiting the currently formed federated data networks could also be an interesting area for future research; for example, the European Health Data & Evidence Network is currently setting up a network of >180 data partners across Europe, using the OMOP-CDM as the main data model [62]. The adoption of the OMOP-CDM and the potential exploitation of such data networks would significantly enhance the prospects of potential AI models used for pharmacovigilance.

Conclusions

In this paper, we reviewed scientific papers focusing on AI approaches to structured RWD for pharmacovigilance purposes. It should be noted, as a key finding, that most models are designed not for pharmacovigilance signal detection but for personalized ADR prediction. Furthermore, XAI methods and causal ML and causal deep learning are not investigated in depth. Moreover, there are no identified gold standard methodologies for data preprocessing of structured RWD for pharmacovigilance. Finally, an evaluation of the already developed AI models in external data is difficult because of code unavailability and a lack of data access.

Therefore, there is an essential need for more informative XAI models that can be validated on external datasets and for a more detailed description of RWD preprocessing pipelines and methods to examine potential pharmacovigilance signals in clinical practice. Implementing AI approaches in RWD analysis could tackle the problems of pharmacovigilance signaling underreporting and support the vision of personalized ADR management.