Abstract:Importance:Although critical to patient safety, health care-related allergic reactions are challenging to identify and monitor.Objective:To develop a deep learning model to identify allergic reactions in the free-text narrative of hospital safety reports and evaluate its generalizability, efficiency, productivity, and interpretability.Design, Setting, and Participants:This cross-sectional study analyzed hospital safety reports filed between May 2004 and January 2019 at Brigham and Women's Hospital and between April 2006 and June 2018 at Massachusetts General Hospital in Boston. Training and validating a deep learning model involved extracting safety reports using 101 expert-curated keywords from Massachusetts General Hospital (data set I). The model was then evaluated on 3 data sets: reports without keywords (data set II), reports from a different time frame (data set III), and reports from a different hospital (Brigham and Women's Hospital; data set IV). Statistical analyses were performed between March 1, 2019, and July 18, 2020.Main Outcomes and Measures:The area under the receiver operating characteristic curve and area under the precision-recall curve were used on data set I. The precision at top-k was used on data sets II to IV.Results:A total of 299 028 safety reports with 172 854 patients were included. Of these patients, 86 544 were women (50.1%) and the median (interquartile range [IQR]) age was 59.7 (43.8-71.6) years. The deep learning model achieved an area under the receiver operating characteristic curve of 0.979 (95% CI, 0.973-0.985) and an area under the precision-recall curve of 0.809 (95% CI, 0.773-0.845). The model achieved precisions at the top 100 model-identified cases of 0.930 in data set II, 0.960 in data set III, and 0.990 in data set IV. Compared with the keyword-search approach, the deep learning model reduced the number of cases for manual review by 63.8% and identified 24.2% more cases of confirmed allergic reactions. The model highlighted important words (eg, rash, hives, and Benadryl) in prediction and extended the list of expert-curated keywords through an attention layer.Conclusions and Relevance:This study showed that a deep learning model can accurately and efficiently identify allergic reactions using free-text narratives written by a variety of health care professionals. This model could be used to improve allergy care, potentially enabling real-time event surveillance and guidance for medical errors and system improvement.

Artificial intelligence and machine learning for anaphylaxis algorithms

Current state and prospects of artificial intelligence in allergy

Using machine learning to improve anaphylaxis case identification in medical claims data

A Framework for Augmented Intelligence in Allergy and Immunology Practice and Research-A Work Group Report of the AAAAI Health Informatics, Technology, and Education Committee

Artificial intelligence and machine learning in diabetes research

Artificial Intelligence: Exploring the Future of Innovation in Allergy Immunology

Future of Allergy and Immunology: Is AI the Key in the Digital Era?

Development and Validation of a Deep Learning Model for Detection of Allergic Reactions Using Safety Event Reports Across Hospitals

A review on artificial intelligence and machine learning used in pharmaceutical research

Analytics with Artificial Intelligence to Advance the Treatment of Acute Respiratory Distress Syndrome

Identifying patients at risk of anaphylaxis

EAACI Guidelines on environmental science in allergic diseases and asthma – leveraging artificial intelligence and machine learning to develop a causality model in exposomics

Artificial Intelligence/Machine Learning in Respiratory Medicine and Potential Role in Asthma and COPD Diagnosis

Artificial Intelligence in Heart Failure and Acute Kidney Injury: Emerging Concepts and Controversial Dimensions

Artificial Intelligence and Machine Learning in Chronic Airway Diseases: Focus on Asthma and Chronic Obstructive Pulmonary Disease

A systematic review of validated methods for identifying anaphylaxis, including anaphylactic shock and angioneurotic edema, using administrative and claims data

A Comprehensive Analysis and Review of Artificial Intelligence in Anaesthesia

The Role and Promise of Artificial Intelligence in Medical Toxicology

Artificial Intelligence in Cardiovascular Care — Part 2: Applications: JACC Review Topic of the Week

Application of Artificial Intelligence in Acute Coronary Syndrome: A Brief Literature Review

Advances in critical care nephrology through artificial intelligence