Abstract:e13620 Background: Accurate assessment of treatment response is fundamental to advancing cancer patient care, particularly in lung cancer, where treatment modalities are diverse and complex. The extraction of treatment response, particularly disease progression information, from Electronic Health Records (EHRs) is essential for several reasons: It facilitates generation of real-world evidence from real-world data, enables personalized treatment planning, and contributes to a broader understanding of cancer therapeutics. However, much of treatment response information is documented in free-text EHR notes and traditional methods of extracting such information from notes are labor-intensive, error-prone, and inefficient, presenting significant barriers to timely and accurate real-world evidence studies. The advancement of Natural Language Processing (NLP) technologies, especially Large Language Models (LLMs), presents a transformative opportunity to automate the extraction of treatment responses. Methods: We used a cohort of 1,953 primary lung cancer patients identified from UPMC Hillman Cancer Center cancer registry and retrieved ~113,000 clinical notes from UPMC clinical data warehouse. We focused on extracting disease progression information, following the RECIST guidelines which define progression as an increase in tumor size or cancer markers after therapy. We selected ~50 notes to perform manual annotations by an experienced oncologist and created a gold standard dataset to validate the NLP models. We fine-tuned a state-of-the-art open-source LLM named LLAMA-2 and compared against a traditional rule-based NLP system for the automated extraction of disease progression from notes. The process of fine-tuning involved adjusting the model's parameters specifically to improve its ability to recognize and classify instances of disease progression. Results: Our analysis demonstrated a significant enhancement in performance with the LLM compared to the traditional rule-based NLP system. LLM exhibited a remarkable increase in sensitivity by ~37%, indicating its superior ability to accurately identify disease progression. Additionally, the model maintained high specificity and PPV, achieving scores nearly comparable to the rule-based system but with a notable improvement in the F1-score by ~14%. Conclusions: Our research highlights the transformative potential of LLM-based NLP algorithms in automating the extraction of treatment responses from free-text EHRs. This methodology not only provides a scalable and efficient mechanism for processing large volumes of clinical text but also significantly enhances the accuracy of lung cancer treatment response assessments. [Table: see text]

Validation of large language models for detecting pathologic complete response in breast cancer using population-based pathology reports

Large language models for extracting histopathologic diagnoses from electronic health records

A Large Language Model Pipeline for Breast Cancer Oncology

CancerLLM: A Large Language Model in Cancer Domain

Applications of Large Language Models (LLMs) in Breast Cancer Care

Utilizing large language models in breast cancer management: systematic review

Applications of Large Language Models in Pathology

A comparative study of large language model-based zero-shot inference and task-specific supervised classification of breast cancer pathology reports

Assessing Large Language Models for Oncology Data Inference from Radiology Reports

Evaluating Large Language Models for Radiology Natural Language Processing

Large Language Models for Automated Synoptic Reports and Resectability Categorization in Pancreatic Cancer

Evaluation of large language models in breast cancer clinical scenarios: A comparative analysis based on ChatGPT-3.5, ChatGPT-4.0, and Claude2

Automating the detection of treatment progression in patients with lung cancer using large language models.

Extraction and classification of structured data from unstructured hepatobiliary pathology reports using large language models: a feasibility study compared with rules-based natural language processing

Inferring cancer disease response from radiology reports using large language models with data augmentation and prompting

Enhancing Clinical Data Extraction from Pathology Reports: A Comparative Analysis of Large Language Models

Large Language Models in Pathology: A Comparative Study on Multiple Choice Question Performance with Pathology Trainees

Development of a privacy preserving large language model for automated data extraction from thyroid cancer pathology reports

Large language model answers medical questions about standard pathology reports

A comparative study of zero-shot inference with large language models and supervised modeling in breast cancer pathology classification