Abstract:e13620 Background: Accurate assessment of treatment response is fundamental to advancing cancer patient care, particularly in lung cancer, where treatment modalities are diverse and complex. The extraction of treatment response, particularly disease progression information, from Electronic Health Records (EHRs) is essential for several reasons: It facilitates generation of real-world evidence from real-world data, enables personalized treatment planning, and contributes to a broader understanding of cancer therapeutics. However, much of treatment response information is documented in free-text EHR notes and traditional methods of extracting such information from notes are labor-intensive, error-prone, and inefficient, presenting significant barriers to timely and accurate real-world evidence studies. The advancement of Natural Language Processing (NLP) technologies, especially Large Language Models (LLMs), presents a transformative opportunity to automate the extraction of treatment responses. Methods: We used a cohort of 1,953 primary lung cancer patients identified from UPMC Hillman Cancer Center cancer registry and retrieved ~113,000 clinical notes from UPMC clinical data warehouse. We focused on extracting disease progression information, following the RECIST guidelines which define progression as an increase in tumor size or cancer markers after therapy. We selected ~50 notes to perform manual annotations by an experienced oncologist and created a gold standard dataset to validate the NLP models. We fine-tuned a state-of-the-art open-source LLM named LLAMA-2 and compared against a traditional rule-based NLP system for the automated extraction of disease progression from notes. The process of fine-tuning involved adjusting the model's parameters specifically to improve its ability to recognize and classify instances of disease progression. Results: Our analysis demonstrated a significant enhancement in performance with the LLM compared to the traditional rule-based NLP system. LLM exhibited a remarkable increase in sensitivity by ~37%, indicating its superior ability to accurately identify disease progression. Additionally, the model maintained high specificity and PPV, achieving scores nearly comparable to the rule-based system but with a notable improvement in the F1-score by ~14%. Conclusions: Our research highlights the transformative potential of LLM-based NLP algorithms in automating the extraction of treatment responses from free-text EHRs. This methodology not only provides a scalable and efficient mechanism for processing large volumes of clinical text but also significantly enhances the accuracy of lung cancer treatment response assessments. [Table: see text]

Suprasellar epithelial cyst. Case report.

Empirical evaluation of language modeling to ascertain cancer outcomes from clinical text reports

Assessing Large Language Models for Oncology Data Inference from Radiology Reports

Use of Natural Language Processing to Infer Sites of Metastatic Disease From Radiology Reports at Scale

Synoptic Reporting by Summarizing Cancer Pathology Reports using Large Language Models

Large Language Models for Automated Synoptic Reports and Resectability Categorization in Pancreatic Cancer

Evaluating Large Language Models for Radiology Natural Language Processing

Automating the detection of treatment progression in patients with lung cancer using large language models.

Implementing engineering education on the WWW: 3 case studies

Validation of large language models for detecting pathologic complete response in breast cancer using population-based pathology reports

Patient Centric Summarization of Radiology Findings using Large Language Models

A Large Language Model Pipeline for Breast Cancer Oncology

Large language models for extracting histopathologic diagnoses from electronic health records

Using Recurrent Neural Networks to Extract High-Quality Information From Lung Cancer Screening Computerized Tomography Reports for Inter-Radiologist Audit and Feedback Quality Improvement

Applying Natural Language Processing to Single-Report Prediction of Metastatic Disease Response Using the OR-RADS Lexicon

Potential of Multimodal Large Language Models for Data Mining of Medical Images and Free-text Reports

CORAL: Expert-Curated medical Oncology Reports to Advance Language Model Inference

Simple Words over Rich Imaging: Accurate Brain Disease Classification via Language Model Analysis of Radiological Reports

RadOnc-GPT: A Large Language Model for Radiation Oncology

Cancer-Answer: Empowering Cancer Care with Advanced Large Language Models

Narrative Feature or Structured Feature? A Study of Large Language Models to Identify Cancer Patients at Risk of Heart Failure