Abstract:Automated radiology report generation is gaining popularity as a means to alleviate the workload of radiologists and prevent misdiagnosis and missed diagnoses. By imitating the working patterns of radiologists, previous report generation approaches have achieved remarkable performance. However, these approaches suffer from two significant problems: (1) lack of visual prior: medical observations in radiology images are interdependent and exhibit certain patterns, and lack of such visual prior can result in reduced accuracy in identifying abnormal regions; (2) lack of alignment between images and texts: the absence of annotations and alignments for regions of interest in the radiology images and reports can lead to inconsistent visual and textual features of the abnormal regions generated by the model. To address these issues, we propose a Visual Prior-based Cross-modal Alignment Network for radiology report generation. First, we propose a novel Contrastive Attention that compares input image with normal images to extract difference information, namely visual prior, which helps to identify abnormalities quickly. Then, to facilitate the alignment of images and texts, we propose a Cross-modal Alignment Network that leverages the cross-modal matrix initialized by the features generated by pre-trained models, to compute cross-modal responses for visual and textual features. Finally, a Visual Prior-guided Multi-Head Attention is proposed to incorporate the visual prior into the generation process. The extensive experimental results on two benchmark datasets, IU-Xray and MIMIC-CXR, illustrate that our proposed model outperforms the state-of-the-art models over almost all metrics, achieving BLEU-4 scores of 0.188 and 0.116 and CIDEr scores of 0.409 and 0.240, respectively.

Co-Occurrence Relationship Driven Hierarchical Attention Network for Brain CT Report Generation

Automatic Report Generation Method Based on Multiscale Feature Extraction and Word Attention Network.

Weakly Guided Attention Model with Hierarchical Interaction for Brain CT Report Generation.

Weakly Guided Hierarchical Encoder-Decoder Network for Brain CT Report Generation.

Granularity Matters: Pathological Graph-driven Cross-modal Alignment for Brain CT Report Generation

See Detail Say Clear: Towards Brain CT Report Generation via Pathological Clue-driven Representation Learning

Memory-based Cross-modal Semantic Alignment Network for Radiology Report Generation

GHCL: Gaussian heuristic curriculum learning for Brain CT report generation

Eye Gaze Guided Cross-Modal Alignment Network for Radiology Report Generation.

Few-Shot Radiology Report Generation via Knowledge Transfer and Multi-modal Alignment.

Visual prior-based cross-modal alignment network for radiology report generation

Prior tissue knowledge-driven contrastive learning for brain CT report generation

Multi-scale Superpixel Based Hierarchical Attention Model for Brain CT Classification

Cross-modal Contrastive Attention Model for Medical Report Generation.

Multifocal region-assisted cross-modality learning for chest X-ray report generation

Semi-supervised Medical Report Generation via Graph-guided Hybrid Feature Consistency

Work like a doctor: Unifying scan localizer and dynamic generator for automated computed tomography report generation

A medical report generation method integrating teacher–student model and encoder–decoder network

Unifying Relational Sentence Generation and Retrieval for Medical Image Report Composition

Deep Cascaded Attention Network for Multi-task Brain Tumor Segmentation.

Attention based automated radiology report generation using CNN and LSTM