Abstract:Medical imaging is an important tool for clinical diagnosis. Nevertheless, it is very time-consuming and error-prone for physicians to prepare imaging diagnosis reports. Therefore, it is necessary to develop some methods to generate medical imaging reports automatically. Currently, the task of medical imaging report generation is challenging in at least two aspects: (1) medical images are very similar to each other. The differences between normal and abnormal images and between different abnormal images are usually trivial; (2) unrelated or incorrect keywords describing abnormal findings in the generated reports lead to mis-communications. In this paper, we propose a medical image report generation framework composed of four modules, including a Transformer encoder, a MIX-MLP multi-label classification network, a co-attention mechanism (CAM) based semantic and visual feature fusion, and a hierarchical LSTM decoder. The Transformer encoder can be used to learn long-range dependencies between images and labels, effectively extract visual and semantic features of images, and establish long-term dependent relationships between visual and semantic information to accurately extract abnormal features from images. The MIX-MLP multi-label classification network, the co-attention mechanism and the hierarchical LSTM network can better identify abnormalities, achieving visual and text alignment fusion and multi-label diagnostic classification to better facilitate report generation. The results of the experiments performed on two widely used radiology report datasets, IU X-RAY and MIMIC-CXR, show that our proposed framework outperforms current report generation models in terms of both natural linguistic generation metrics and clinical efficacy assessment metrics. The code of this work is available online at https://github.com/watersunhznu/LIFMRG.

An Inclusive Task-Aware Framework for Radiology Report Generation

Automatic Report Generation Method Based on Multiscale Feature Extraction and Word Attention Network.

An Organ-aware Diagnosis Framework for Radiology Report Generation

Radiology Report Generation via Structured Knowledge-Enhanced Multi-modal Attention and Contrastive Learning.

Act Like a Radiologist: Radiology Report Generation across Anatomical Regions

On the Automatic Generation of Medical Imaging Reports

Show, Describe and Conclude: On Exploiting the Structure Information of Chest X-Ray Reports

A Self-boosting Framework for Automated Radiographic Report Generation

An efficient but effective writer: Diffusion-based semi-autoregressive transformer for automated radiology report generation

Radiology Report Generation with a Learned Knowledge Base and Multi-Modal Alignment

A Self-Guided Framework for Radiology Report Generation

Reading Radiology Imaging Like The Radiologist

A label information fused medical image report generation framework

Enhanced Knowledge Injection for Radiology Report Generation

Generating radiology reports via auxiliary signal guidance and a memory-driven network

Beyond Images: An Integrative Multi-modal Approach to Chest X-Ray Report Generation

A Survey of Deep Learning-based Radiology Report Generation Using Multimodal Data

AI Accelerated Human-in-the-loop Structuring of Radiology Reports

Designing a Robust Radiology Report Generation System

Knowledge Matters: Radiology Report Generation with General and Specific Knowledge

Quality Control for Radiology Report Generation Models via Auxiliary Auditing Components