Abstract:Purpose: To develop and evaluate an automated system for extracting structured clinical information from unstructured radiology and pathology reports using open-weights large language models (LMs) and retrieval augmented generation (RAG), and to assess the effects of model configuration variables on extraction performance. Methods and Materials: The study utilized two datasets: 7,294 radiology reports annotated for Brain Tumor Reporting and Data System (BT-RADS) scores and 2,154 pathology reports annotated for isocitrate dehydrogenase (IDH) mutation status. An automated pipeline was developed to benchmark the performance of various LMs and RAG configurations. The impact of model size, quantization, prompting strategies, output formatting, and inference parameters was systematically evaluated. Results: The best performing models achieved over 98% accuracy in extracting BT-RADS scores from radiology reports and over 90% for IDH mutation status extraction from pathology reports. The top model being medical fine-tuned llama3. Larger, newer, and domain fine-tuned models consistently outperformed older and smaller models. Model quantization had minimal impact on performance. Few-shot prompting significantly improved accuracy. RAG improved performance for complex pathology reports but not for shorter radiology reports. Conclusions: Open LMs demonstrate significant potential for automated extraction of structured clinical data from unstructured clinical reports with local privacy-preserving application. Careful model selection, prompt engineering, and semi-automated optimization using annotated data are critical for optimal performance. These approaches could be reliable enough for practical use in research workflows, highlighting the potential for human-machine collaboration in healthcare data extraction.

Development and Testing of Retrieval Augmented Generation in Large Language Models -- A Case Study Report

oRetrieval Augmented Generation for 10 Large Language Models and its Generalizability in Assessing Medical Fitness

Retrieval-Augmented Generation for Large Language Models: A Survey

Enhancing Large Language Models with Domain-specific Retrieval Augment Generation: A Case Study on Long-form Consumer Health Question Answering in Ophthalmology

Deploying Large Language Models With Retrieval Augmented Generation

Language Models and Retrieval Augmented Generation for Automated Structured Data Extraction from Diagnostic Reports

BiomedRAG: A Retrieval Augmented Large Language Model for Biomedicine

Transforming Healthcare Education: Harnessing Large Language Models for Frontline Health Worker Capacity Building using Retrieval-Augmented Generation

Retrieval-augmented large language models for clinical trial screening.

Comprehensive and Practical Evaluation of Retrieval-Augmented Generation Systems for Medical Question Answering

Rationale-Guided Retrieval Augmented Generation for Medical Question Answering

Mitigating Hallucinations in Large Language Models: A Comparative Study of RAG-enhanced vs. Human-Generated Medical Templates

Bailicai: A Domain-Optimized Retrieval-Augmented Generation Framework for Medical Applications

Tool Calling: Enhancing Medication Consultation via Retrieval-Augmented Large Language Models

A Survey on Retrieval-Augmented Text Generation for Large Language Models

Benchmarking Retrieval-Augmented Generation for Medicine

MKRAG: Medical Knowledge Retrieval Augmented Generation for Medical Question Answering

The Geometry of Queries: Query-Based Innovations in Retrieval-Augmented Generation

Retrieval Augmented Generation and Representative Vector Summarization for large unstructured textual data in Medical Education