Abstract:e13620 Background: Accurate assessment of treatment response is fundamental to advancing cancer patient care, particularly in lung cancer, where treatment modalities are diverse and complex. The extraction of treatment response, particularly disease progression information, from Electronic Health Records (EHRs) is essential for several reasons: It facilitates generation of real-world evidence from real-world data, enables personalized treatment planning, and contributes to a broader understanding of cancer therapeutics. However, much of treatment response information is documented in free-text EHR notes and traditional methods of extracting such information from notes are labor-intensive, error-prone, and inefficient, presenting significant barriers to timely and accurate real-world evidence studies. The advancement of Natural Language Processing (NLP) technologies, especially Large Language Models (LLMs), presents a transformative opportunity to automate the extraction of treatment responses. Methods: We used a cohort of 1,953 primary lung cancer patients identified from UPMC Hillman Cancer Center cancer registry and retrieved ~113,000 clinical notes from UPMC clinical data warehouse. We focused on extracting disease progression information, following the RECIST guidelines which define progression as an increase in tumor size or cancer markers after therapy. We selected ~50 notes to perform manual annotations by an experienced oncologist and created a gold standard dataset to validate the NLP models. We fine-tuned a state-of-the-art open-source LLM named LLAMA-2 and compared against a traditional rule-based NLP system for the automated extraction of disease progression from notes. The process of fine-tuning involved adjusting the model's parameters specifically to improve its ability to recognize and classify instances of disease progression. Results: Our analysis demonstrated a significant enhancement in performance with the LLM compared to the traditional rule-based NLP system. LLM exhibited a remarkable increase in sensitivity by ~37%, indicating its superior ability to accurately identify disease progression. Additionally, the model maintained high specificity and PPV, achieving scores nearly comparable to the rule-based system but with a notable improvement in the F1-score by ~14%. Conclusions: Our research highlights the transformative potential of LLM-based NLP algorithms in automating the extraction of treatment responses from free-text EHRs. This methodology not only provides a scalable and efficient mechanism for processing large volumes of clinical text but also significantly enhances the accuracy of lung cancer treatment response assessments. [Table: see text]

Large language models for extracting histopathologic diagnoses from electronic health records

Critical Care Studies Using Large Language Models Based on Electronic Healthcare Records: A Technical Note

Large language models for accurate disease detection in electronic health records

Privacy-preserving large language models for structured medical information retrieval

Applications of Large Language Models in Pathology

Development of a privacy preserving large language model for automated data extraction from thyroid cancer pathology reports

From Text to Tables: A Local Privacy Preserving Large Language Model for Structured Information Retrieval from Medical Documents

Extraction and classification of structured data from unstructured hepatobiliary pathology reports using large language models: a feasibility study compared with rules-based natural language processing

Validation of large language models for detecting pathologic complete response in breast cancer using population-based pathology reports

LLMD: A Large Language Model for Interpreting Longitudinal Medical Records

Large Language Models with Retrieval-Augmented Generation for Zero-Shot Disease Phenotyping

Automating the detection of treatment progression in patients with lung cancer using large language models.

Is larger always better? Evaluating and prompting large language models for non-generative medical tasks

Optimizing large language models in digestive disease: strategies and challenges to improve clinical outcomes

Transformative potential of Large Language Models in data mining on Electronic Health Records.

Enhancing Clinical Data Extraction from Pathology Reports: A Comparative Analysis of Large Language Models

Evaluating Large Language Models for Public Health Classification and Extraction Tasks

A Comparative Analysis of Privacy-Preserving Large Language Models For Automated Echocardiography Report Analysis

Scalable information extraction from free text electronic health records using large language models

Large Language Models in Medicine: The Potentials and Pitfalls