KARGEN: Knowledge-enhanced Automated Radiology Report Generation Using Large Language Models

Yingshu Li,Zhanyu Wang,Yunyi Liu,Lei Wang,Lingqiao Liu,Luping Zhou

DOI: https://doi.org/10.48550/arXiv.2409.05370

2024-09-09

Abstract:Harnessing the robust capabilities of Large Language Models (LLMs) for narrative generation, logical reasoning, and common-sense knowledge integration, this study delves into utilizing LLMs to enhance automated radiology report generation (R2Gen). Despite the wealth of knowledge within LLMs, efficiently triggering relevant knowledge within these large models for specific tasks like R2Gen poses a critical research challenge. This paper presents KARGEN, a Knowledge-enhanced Automated radiology Report GENeration framework based on LLMs. Utilizing a frozen LLM to generate reports, the framework integrates a knowledge graph to unlock chest disease-related knowledge within the LLM to enhance the clinical utility of generated reports. This is achieved by leveraging the knowledge graph to distill disease-related features in a designed way. Since a radiology report encompasses both normal and disease-related findings, the extracted graph-enhanced disease-related features are integrated with regional image features, attending to both aspects. We explore two fusion methods to automatically prioritize and select the most relevant features. The fused features are employed by LLM to generate reports that are more sensitive to diseases and of improved quality. Our approach demonstrates promising results on the MIMIC-CXR and IU-Xray datasets.

Computer Vision and Pattern Recognition,Artificial Intelligence

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to more effectively utilize the knowledge of large - language models (LLMs) in automated radiology report generation (R2Gen) to improve the quality and clinical practicality of generated reports. Although LLMs possess strong capabilities in narrative generation, logical reasoning, and common - sense knowledge integration, in specific tasks such as R2Gen, how to efficiently trigger relevant knowledge in LLMs is an important research challenge.

Specifically, the paper proposes a framework named KARGEN, which aims to solve this problem in the following ways:

1. **Integrating the medical domain knowledge graph**: Use a frozen LLM to generate reports, and unlock knowledge related to chest diseases by introducing a knowledge graph, thereby enhancing the clinical practicality of the generated reports.

2. **Extracting disease - related features**: Distill disease - related features through the knowledge graph and combine them with regional image features to focus on normal and disease - related findings in radiology reports.

3. **Feature fusion methods**: Explore two fusion methods (element - level fusion and modality - level fusion) to automatically prioritize the most relevant features, making the generated reports more sensitive to diseases and of higher quality.

4. **Verification and evaluation**: Conduct experiments on two public datasets, MIMIC - CXR and IU - Xray. The results show that KARGEN outperforms existing methods on multiple evaluation metrics, especially performing prominently on clinically - relevant evaluation metrics.

Through these improvements, KARGEN can more accurately identify and describe diseases when generating radiology reports, thereby improving the clinical practicality and quality of the reports.