Abstract:e13620 Background: Accurate assessment of treatment response is fundamental to advancing cancer patient care, particularly in lung cancer, where treatment modalities are diverse and complex. The extraction of treatment response, particularly disease progression information, from Electronic Health Records (EHRs) is essential for several reasons: It facilitates generation of real-world evidence from real-world data, enables personalized treatment planning, and contributes to a broader understanding of cancer therapeutics. However, much of treatment response information is documented in free-text EHR notes and traditional methods of extracting such information from notes are labor-intensive, error-prone, and inefficient, presenting significant barriers to timely and accurate real-world evidence studies. The advancement of Natural Language Processing (NLP) technologies, especially Large Language Models (LLMs), presents a transformative opportunity to automate the extraction of treatment responses. Methods: We used a cohort of 1,953 primary lung cancer patients identified from UPMC Hillman Cancer Center cancer registry and retrieved ~113,000 clinical notes from UPMC clinical data warehouse. We focused on extracting disease progression information, following the RECIST guidelines which define progression as an increase in tumor size or cancer markers after therapy. We selected ~50 notes to perform manual annotations by an experienced oncologist and created a gold standard dataset to validate the NLP models. We fine-tuned a state-of-the-art open-source LLM named LLAMA-2 and compared against a traditional rule-based NLP system for the automated extraction of disease progression from notes. The process of fine-tuning involved adjusting the model's parameters specifically to improve its ability to recognize and classify instances of disease progression. Results: Our analysis demonstrated a significant enhancement in performance with the LLM compared to the traditional rule-based NLP system. LLM exhibited a remarkable increase in sensitivity by ~37%, indicating its superior ability to accurately identify disease progression. Additionally, the model maintained high specificity and PPV, achieving scores nearly comparable to the rule-based system but with a notable improvement in the F1-score by ~14%. Conclusions: Our research highlights the transformative potential of LLM-based NLP algorithms in automating the extraction of treatment responses from free-text EHRs. This methodology not only provides a scalable and efficient mechanism for processing large volumes of clinical text but also significantly enhances the accuracy of lung cancer treatment response assessments. [Table: see text]

Automating Clinical Phenotyping Using Natural Language Processing: An Application for Crohn's Disease

A Large Language Model Outperforms Other Computational Approaches to the High-Throughput Phenotyping of Physician Notes

High-Throughput Phenotyping of Clinical Text Using Large Language Models

Large Language Models Facilitate the Generation of Electronic Health Record Phenotyping Algorithms

Leveraging GPT-4 for identifying cancer phenotypes in electronic health records: a performance comparison between GPT-4, GPT-3.5-turbo, Flan-T5, Llama-3-8B, and spaCy's rule-based and machine learning-based methods

Toward Cross‐platform Electronic Health Record‐driven Phenotyping Using Clinical Quality Language

Comparing Rule-Based and Deep Learning Models for Patient Phenotyping

Identifying and Extracting Rare Diseases and Their Phenotypes with Large Language Models

Large Language Models with Retrieval-Augmented Generation for Zero-Shot Disease Phenotyping

On the limitations of large language models in clinical diagnosis

Improving Automated Deep Phenotyping Through Large Language Models Using Retrieval Augmented Generation

Utilizing Large Language Models for Enhanced Clinical Trial Matching: A Study on Automation in Patient Screening

Automating the detection of treatment progression in patients with lung cancer using large language models.

Developing a portable natural language processing based phenotyping system

Enhancing Phenotype Recognition in Clinical Notes Using Large Language Models: PhenoBCBERT and PhenoGPT

Large language models for extracting histopathologic diagnoses from electronic health records

Zero-shot Interpretable Phenotyping of Postpartum Hemorrhage Using Large Language Models

High Throughput Phenotyping of Physician Notes with Large Language and Hybrid NLP Models

Investigating Alternative Feature Extraction Pipelines For Clinical Note Phenotyping

Transformative potential of Large Language Models in data mining on Electronic Health Records.