Automatic Detection of Adverse Drug Events in Geriatric Care: Study Proposal

Background One-third of older inpatients experience adverse drug events (ADEs), which increase their mortality, morbidity, and health care use and costs. In particular, antithrombotic drugs are among the most at-risk medications for this population. Reporting systems have been implemented at the national, regional, and provider levels to monitor ADEs and design prevention strategies. Owing to their well-known limitations, automated detection technologies based on electronic medical records (EMRs) are being developed to routinely detect or predict ADEs. Objective This study aims to develop and validate an automated detection tool for monitoring antithrombotic-related ADEs using EMRs from 4 large Swiss hospitals. We aim to assess cumulative incidences of hemorrhages and thromboses in older inpatients associated with the prescription of antithrombotic drugs, identify triggering factors, and propose improvements for clinical practice. Methods This project is a multicenter, cross-sectional study based on 2015 to 2016 EMR data from 4 large hospitals in Switzerland: Lausanne, Geneva, and Zürich university hospitals, and Baden Cantonal Hospital. We have included inpatients aged ≥65 years who stayed at 1 of the 4 hospitals during 2015 or 2016, received at least one antithrombotic drug during their stay, and signed or were not opposed to a general consent for participation in research. First, clinical experts selected a list of relevant antithrombotic drugs along with their side effects, risks, and confounding factors. Second, administrative, clinical, prescription, and laboratory data available in the form of free text and structured data were extracted from study participants’ EMRs. Third, several automated rule-based and machine learning–based algorithms are being developed, allowing for the identification of hemorrhage and thromboembolic events and their triggering factors from the extracted information. Finally, we plan to validate the developed detection tools (one per ADE type) through manual medical record review. Performance metrics for assessing internal validity will comprise the area under the receiver operating characteristic curve, F1-score, sensitivity, specificity, and positive and negative predictive values. Results After accounting for the inclusion and exclusion criteria, we will include 34,522 residents aged ≥65 years. The data will be analyzed in 2022, and the research project will run until the end of 2022 to mid-2023. Conclusions This project will allow for the introduction of measures to improve safety in prescribing antithrombotic drugs, which today remain among the drugs most involved in ADEs. The findings will be implemented in clinical practice using indicators of adverse events for risk management and training for health care professionals; the tools and methodologies developed will be disseminated for new research in this field. The increased performance of natural language processing as an important complement to structured data will bring existing tools to another level of efficiency in the detection of ADEs. Currently, such systems are unavailable in Switzerland. International Registered Report Identifier (IRRID) DERR1-10.2196/40456

The Delphi method adopted for the selection of relevant characteristics is appropriate and scientifically sound.
However, the computational algorithms proposed for the structured and unstructured data mining are absolutely unclear. In particular, the referee could not assess whether the predictive models would be based on regression modelling, or any other approach.
The sample size calculation method does not seem sufficiently clear, stable and objective. The final estimates are based on comparisons extracted from a table that presents different values, based on the arguable target of confidence intervals with a 20% width. Firstly, such confidence intervals seem too wide to guarantee the required stability required to validate the text based algorithm as opposed to the structured format. Secondly, and most importantly, the method takes into account only one aspect of reliability, i.e. sensitivity, while more complete assessment might be performed using approaches eg likelihood ratio or ROC curve, which take simultaneously into both sensitivity and specificity. Finally, traditional techniques embedding the calculation of adequate power would lead to more objective selection of minimal sample size.
Use of de-identified data may not necessarily imply no risks for privacy and data protection, given the level of detail required by the collection of individual characteristics that would allow re-identification. The challenges involved with these problems are not sufficiently addressed in the proposal. The ethical committee should consider whether patient consent would be required for the conduction of this study.

Inter-and transdisciplinarity
The project team involves experts with background in pharmacology, clinical research and medical informatics, which seems quite appropriate for the scope. However, given the methodological challenges involved in this project, it would have been really beneficial to have a co-applicant with strong statistical background who does not show up in the list. The lack of specialistic expertise in this field is reflected by the limitations highlighted in the methodology of this proposal.

Application and implementation
The implementation plan is adequate as it involves aspects of relevant interest for patient safety eg indicators, guidelines and open source assessment forms.However, the central element involving the analysis of medical texts, and how this would be transferred to the coalface in health services, is not sufficiently explained.

Personnel and infrastructure
The personnel and infrastructure is adequate for the scope of the project.

Feasibility within the set timeframe and budget
The project seems feasible within the set timeframe and budget of the proposal.

Comment
The project aims to respond to a very relevant clinical question regarding the safety of antithrombotic drugs, proposing a targeted solution that can improve the detection tools available for adverse drug events. The proposed method of using retrospective analysis of structured as well as unstructured data can deliver a novel mixed approach that would maximise the information available. The design is supported by a Delphi method that is appropriate to identify the key elements for the proposed study. These are clear strengths that are worth to be outlined.
However, major weaknesses affect the overall score: Computational algorithms proposed for the structured and unstructured data mining are absolutely unclear. Sample size calculation methods do not seem sufficiently clear, stable and objective. Other methods based on summary elements eg likelihood ratios, ROC curve and power calculations would have been preferable. Ethical challenges in the management of individual level data should have been better explained How the analysis of medical text would be transferred in health services is not explained.
The absence of a co-applicant with strong biostatistical background is a noticeable pitfall in the composition of the project team, which might have negatively influenced the accuracy of the methodological section.