Abstract:Key Points Purpose: We examined the effectiveness of proprietary and open Large Language Models (LLMs) in detecting disease presence, location, and treatment response in pancreatic cancer from radiology reports. Methods: We analyzed 203 deidentified radiology reports, manually annotated for disease status, location, and indeterminate nodules needing follow-up. Utilizing GPT-4, GPT-3.5-turbo, and open models like Gemma-7B and Llama3-8B, we employed strategies such as ablation and prompt engineering to boost accuracy. Discrepancies between human and model interpretations were reviewed by a secondary oncologist. Results: Among 164 pancreatic adenocarcinoma patients, GPT-4 showed the highest accuracy in inferring disease status, achieving a 75.5% correctness (F1-micro). Open models Mistral-7B and Llama3-8B performed comparably, with accuracies of 68.6% and 61.4%, respectively. Mistral-7B excelled in deriving correct inferences from "Objective Findings" directly. Most tested models demonstrated proficiency in identifying disease containing anatomical locations from a list of choices, with GPT-4 and Llama3-8B showing near parity in precision and recall for disease site identification. However, open models struggled with differentiating benign from malignant post-surgical changes, impacting their precision in identifying findings indeterminate for cancer. A secondary review occasionally favored GPT-3.5's interpretations, indicating the variability in human judgment. Conclusion: LLMs, especially GPT-4, are proficient in deriving oncological insights from radiology reports. Their performance is enhanced by effective summarization strategies, demonstrating their potential in clinical support and healthcare analytics. This study also underscores the possibility of zero-shot open model utility in environments where proprietary models are restricted. Finally, by providing a set of annotated radiology reports, this paper presents a valuable dataset for further LLM research in oncology.

S137 Development and Validation of Large Language Models to Aid Automated Classification of Pancreatic Cysts at Scale

Large Language Models for Automated Synoptic Reports and Resectability Categorization in Pancreatic Cancer

Assessing Large Language Models for Oncology Data Inference from Radiology Reports

Large language models for extracting histopathologic diagnoses from electronic health records

Automated Detection of Pancreatic Cystic Lesions on CT Using Deep Learning

Privacy-preserving large language models for structured medical information retrieval

Validation of large language models for detecting pathologic complete response in breast cancer using population-based pathology reports

Should All Pancreatic Cystic Lesions with Worrisome or High-Risk Features Be Resected? A Clinical and Radiological Machine Learning Model May Help to Answer

CancerLLM: A Large Language Model in Cancer Domain

A Large Language Model Pipeline for Breast Cancer Oncology

Identification of pancreatic cancer risk factors from clinical notes using natural language processing

Evaluating Large Language Models for Radiology Natural Language Processing

Extraction and classification of structured data from unstructured hepatobiliary pathology reports using large language models: a feasibility study compared with rules-based natural language processing

Automatically Detecting Pancreatic Cysts in Autosomal Dominant Polycystic Kidney Disease on MRI Using Deep Learning

Radiomics Boosts Deep Learning Model for IPMN Classification

Large Language Models for Granularized Barrett's Esophagus Diagnosis Classification

Collaborative Enhancement of Consistency and Accuracy in US Diagnosis of Thyroid Nodules Using Large Language Models

Classifying Cancer Stage with Open-Source Clinical Large Language Models

Large Language Models in Medicine: The Potentials and Pitfalls

Classification prediction of pancreatic cystic neoplasms based on radiomics deep learning models