Abstract:Introduction Adverse drug reaction reports are usually manually assessed by pharmacovigilance experts to detect safety signals associated with drugs. With the recent extension of reporting to patients and the emergence of mass media-related sanitary crises, adverse drug reaction reports currently frequently overwhelm pharmacovigilance networks. Artificial intelligence could help support the work of pharmacovigilance experts during such crises, by automatically coding reports, allowing them to prioritise or accelerate their manual assessment. After a previous study showing first results, we developed and compared state-of-the-art machine learning models using a larger nationwide dataset, aiming to automatically pre-code patients' adverse drug reaction reports. Objectives We aimed to determine the best artificial intelligence model identifying adverse drug reactions and assessing seriousness in patients reports from the French national pharmacovigilance web portal. Methods Reports coded by 27 Pharmacovigilance Centres between March 2017 and December 2020 were selected ( n = 11,633). For each report, the Portable Document Format form containing free-text information filled by the patient, and the corresponding encodings of adverse event symptoms (in Medical Dictionary for Regulatory Activities Preferred Terms) and seriousness were obtained. This encoding by experts was used as the reference to train and evaluate models, which contained input data processing and machine-learning natural language processing to learn and predict encodings. We developed and compared different approaches for data processing and classifiers. Performance was evaluated using receiver operating characteristic area under the curve (AUC), F-measure, sensitivity, specificity and positive predictive value. We used data from 26 Pharmacovigilance Centres for training and internal validation. External validation was performed using data from the remaining Pharmacovigilance Centres during the same period. Results Internal validation: for adverse drug reaction identification, Term Frequency-Inverse Document Frequency (TF-IDF) + Light Gradient Boosted Machine (LGBM) achieved an AUC of 0.97 and an F-measure of 0.80. The Cross-lingual Language Model (XLM) [transformer] obtained an AUC of 0.97 and an F-measure of 0.78. For seriousness assessment, FastText + LGBM achieved an AUC of 0.85 and an F-measure of 0.63. CamemBERT (transformer) + Light Gradient Boosted Machine obtained an AUC of 0.84 and an F-measure of 0.63. External validation for both adverse drug reaction identification and seriousness assessment tasks yielded consistent and robust results. Conclusions Our artificial intelligence models showed promising performance to automatically code patient adverse drug reaction reports, with very similar results across approaches. Our system has been deployed by national health authorities in France since January 2021 to facilitate pharmacovigilance of COVID-19 vaccines. Further studies will be needed to validate the performance of the tool in real-life settings.

Identification of Pregnancy Adverse Drug Reactions in Pharmacovigilance Reporting Systems: A Novel Algorithm Developed in EudraVigilance

Detection and evaluation of signals associated with exposure to individual and combination of medications in pregnancy: a signal detection study protocol

Validation of a Novel Method to Assess the Clinical Quality of Information in Pregnancy-Related Pharmacovigilance Case Reports: A ConcePTION Project

Investigating the Benefit-Risk Profile of Drugs: From Spontaneous Reporting Systems to Real-World Data for Pharmacovigilance

A discovery and verification approach to pharmacovigilance using electronic healthcare data

Core Data Elements for Pregnancy Pharmacovigilance Studies Using Primary Source Data Collection Methods: Recommendations from the IMI ConcePTION Project

Pharmacovigilance in Pregnancy Studies, Exposures and Outcomes Ascertainment, and Findings from Low- and Middle-Income Countries: A Scoping Review

Detection of potential safety signals related to the use of remdesivir and tocilizumab in the COVID era during pregnancy, resorting to open data from the FDA adverse event reporting system (FAERS)

Establishment of a Rapid Identification of Adverse Drug Reaction Program in R Language Implementation Based on Monitoring Data

Surveillance of Medication Use During Pregnancy in the Mini-Sentinel Program

Exploiting heterogeneous publicly available data sources for drug safety surveillance: computational framework and case studies

Development and validation of a machine learning algorithm for predicting the risk of postpartum depression among pregnant women

The Impact of Mandatory Reporting of Non-Serious Safety Reports to EudraVigilance on the Detection of Adverse Reactions

Validation of Artificial Intelligence to Support the Automatic Coding of Patient Adverse Drug Reaction Reports, Using Nationwide Pharmacovigilance Data

Adverse drug reactions in pregnant women: Do they differ from those in non-pregnant women of childbearing age?

Detecting Pharmacovigilance Signals Combining Electronic Medical Records With Spontaneous Reports: A Case Study of Conventional Disease-Modifying Antirheumatic Drugs for Rheumatoid Arthritis

Development and evaluation of standardized pregnancy identification and trimester distribution algorithms in U.S. IBM MarketScan® Commercial and Medicaid data

A Sex- and Gender-Based Analysis of Adverse Drug Reactions: A Scoping Review of Pharmacovigilance Databases

Methods for drug safety signal detection using routinely collected observational electronic health care data: a systematic review

The safety profile of usage of glucagon‐like peptide‐1 receptor agonists in pregnancy: A pharmacovigilance analysis based on the Food and Drug Administration Adverse Event Reporting System

Developing A Deep Learning Natural Language Processing Algorithm For Automated Reporting Of Adverse Drug Reactions