Abstract:Importance: The Sentinel System is a key component of the US Food and Drug Administration (FDA) postmarketing safety surveillance commitment and uses clinical health care data to conduct analyses to inform drug labeling and safety communications, FDA advisory committee meetings, and other regulatory decisions. However, observational data are frequently deemed insufficient for reliable evaluation of safety concerns owing to limitations in underlying data or methodology. Advances in large language models (LLMs) provide new opportunities to address some of these limitations. However, careful consideration is necessary for how and where LLMs can be effectively deployed for these purposes. Observations: LLMs may provide new avenues to support signal-identification activities to identify novel adverse event signals from narrative text of electronic health records. These algorithms may be used to support epidemiologic investigations examining the causal relationship between exposure to a medical product and an adverse event through development of probabilistic phenotyping of health outcomes of interest and extraction of information related to important confounding factors. LLMs may perform like traditional natural language processing tools by annotating text with controlled vocabularies with additional tailored training activities. LLMs offer opportunities for enhancing information extraction from adverse event reports, medical literature, and other biomedical knowledge sources. There are several challenges that must be considered when leveraging LLMs for postmarket surveillance. Prompt engineering is needed to ensure that LLM-extracted associations are accurate and specific. LLMs require extensive infrastructure to use, which many health care systems lack, and this can impact diversity, equity, and inclusion, and result in obscuring significant adverse event patterns in some populations. LLMs are known to generate nonfactual statements, which could lead to false positive signals and downstream evaluation activities by the FDA and other entities, incurring substantial cost. Conclusions and relevance: LLMs represent a novel paradigm that may facilitate generation of information to support medical product postmarket surveillance activities that have not been possible. However, additional work is required to ensure LLMs can be used in a fair and equitable manner, minimize false positive findings, and support the necessary rigor of signal detection needed for regulatory activities.

Automated Extraction of Mortality Information from Publicly Available Sources Using Language Models

Federated Learning of Electronic Health Records Improves Mortality Prediction in Patients Hospitalized with COVID-19.

Tending Unmarked Graves: Classification of Post-mortem Content on Social Media

Proposing Causal Sequence of Death by Neural Machine Translation in Public Health Informatics

Opioid death projections with AI-based forecasts using social media language

Scalable information extraction from free text electronic health records using large language models

Large Language Models versus Classical Machine Learning: Performance in COVID-19 Mortality Prediction Using High-Dimensional Tabular Data

Federated Learning of Electronic Health Records to Improve Mortality Prediction in Hospitalized Patients With COVID-19: Machine Learning Approach

Approach to machine learning for extraction of real-world data variables from electronic health records

Predictive Analytics for Mortality: FSRNCA-FLANN Modeling Using Public Health Inventory Records

Public Health Informatics: Proposing Causal Sequence of Death Using Neural Machine Translation

Evaluating local open-source large language models for data extraction from unstructured reports on mechanical thrombectomy in patients with ischemic stroke

Large language models to identify social determinants of health in electronic health records

Enhancing Postmarketing Surveillance of Medical Products With Large Language Models

Machine Learning Based Early Mortality Prediction in the Emergency Department.

Consensus of state of the art mortality prediction models: From all-cause mortality to sudden death prediction

Automated Extraction of Stroke Severity From Unstructured Electronic Health Records Using Natural Language Processing

Automated Extraction of Patient-Centered Outcomes After Breast Cancer Treatment: An Open-Source Large Language Model-Based Toolkit

Improving palliative and end-of-life care with machine learning and routine data: a rapid review

LCD Benchmark: Long Clinical Document Benchmark on Mortality Prediction for Language Models