Abstract:Purpose: To quantitatively and qualitatively evaluate and compare the performance of leading large language models (LLMs), including proprietary models (GPT-4, GPT-3.5 Turbo, Claude-3-Opus, and Gemini Ultra) and open-source models (Mistral-7b and Mistral-8×7b), in simplifying 109 interventional radiology reports. Methods: Qualitative performance was assessed using a five-point Likert scale for accuracy, completeness, clarity, clinical relevance, naturalness, and error rates, including trust-breaking and post-therapy misconduct errors. Quantitative readability was assessed using Flesch Reading Ease (FRE), Flesch-Kincaid Grade Level (FKGL), SMOG Index, and Dale-Chall Readability Score (DCRS). Paired t-tests and Bonferroni-corrected p-values were used for statistical analysis. Results: Qualitative evaluation showed no significant differences between GPT-4 and Claude-3-Opus for any metrics evaluated (all Bonferroni-corrected p-values: p = 1), while they outperformed other assessed models across five qualitative metrics (p < 0.001). GPT-4 had the fewest content and trust-breaking errors, with Claude-3-Opus second. However, all models exhibited some level of trust-breaking and post-therapy misconduct errors, with GPT-4-Turbo and GPT-3.5-Turbo with few-shot prompting showing the lowest error rates, and Mistral-7B and Mistral-8×7B showing the highest. Quantitatively, GPT-4 surpassed Claude-3-Opus in all readability metrics (all p < 0.001), with a median FRE score of 69.01 (IQR: 64.88-73.14) versus 59.74 (IQR: 55.47-64.01) for Claude-3-Opus. GPT-4 also outperformed GPT-3.5-Turbo and Gemini Ultra (both p < 0.001). Inter-rater reliability was strong (κ = 0.77-0.84). Conclusions: GPT-4 and Claude-3-Opus demonstrated superior performance in generating simplified IR reports, but the presence of errors across all models, including trust-breaking errors, highlights the need for further refinement and validation before clinical implementation. Clinical relevance/applications: With the increasing complexity of interventional radiology (IR) procedures and the growing availability of electronic health records, simplifying IR reports is critical to improving patient understanding and clinical decision-making. This study provides insights into the performance of various LLMs in rewriting IR reports, which can help in selecting the most suitable model for clinical patient-centered applications.

Evaluation of Radiology Residents' Reporting Skills Using Large Language Models: An Observational Study

Enhancing radiology training with GPT-4: Pilot analysis of automated feedback in trainee preliminary reports

LLM-RadJudge: Achieving Radiologist-Level Evaluation for X-Ray Report Generation

Constructing a Large Language Model to Generate Impressions from Findings in Radiology Reports

Evaluating Large Language Models for Radiology Natural Language Processing

An open-source fine-tuned large language model for radiological impression generation: a multi-reader performance study

The current status of large language models in summarizing radiology report impressions

Leveraging Professional Radiologists' Expertise to Enhance LLMs' Evaluation for Radiology Reports

Large language models for structured reporting in radiology: past, present, and future

Improving Clinical Expertise in Large Language Models Using Electronic Medical Records

Large Language Models for Simplified Interventional Radiology Reports: A Comparative Analysis

From Revisions to Insights: Converting Radiology Report Revisions into Actionable Educational Feedback Using Generative AI Models

Exploring Multilingual Large Language Models for Enhanced TNM classification of Radiology Report in lung cancer staging

MRScore: Evaluating Radiology Report Generation with LLM-based Reward System

Exploring Multimodal Large Language Models for Radiology Report Error-checking

Evaluation of large language models performance against humans for summarizing MRI knee radiology reports: A feasibility study

Fine-Tuning In-House Large Language Models to Infer Differential Diagnosis from Radiology Reports

From technical to understandable: Artificial Intelligence Large Language Models improve the readability of knee radiology reports

Improving Radiology Report Conciseness and Structure via Local Large Language Models

Advancing radiology practice and research: harnessing the potential of large language models amidst imperfections

Large language models (LLMs) in radiology exams for medical students: Performance and consequences