Abstract:Medical report generation is a time-consuming and knowledge-intensive task performed by radiologists to describe various regions within medical images. Writing report manually is prone to subjective bias and errors. Consequently, medical report generation automatically has become an important research direction in the field of artificial intelligence. While recent report generation methods have achieved relatively fluent medical reports, several challenges remain: (1) They overlook valuable semantic information from similar cases, which results in insufficient information for accurate reporting; (2) Deep exploration of medical features is lacking, which hampers the understanding of medical terminology and compromises disease prediction accuracy. To address the aforementioned issues, this paper proposes a Similarity Retrieval and Medical Cross-modal Attention based Medical Report Generation Network (SRMCAN). By employing content-based similarity retrieval, SRMCAN filters out interfering information in relevant semantic features, which serves as a complementary feature for the model. SRMCAN constructs a fine-grained alignment loss function, taking similar cases as hard negative samples to enhance the dynamic interaction between cross-modal disease features. A Medical Cross-modal Attention mechanism is designed to capture second-order interactions between cross-modal features, incorporating coordinate attention to calculate attention distributions in two spatial directions. The Medical Cross-modal Attention strengthens the model's understanding and reasoning ability for medical information, improving the accuracy and professionalism of the generated reports. Experimental results on the IU X-Ray dataset demonstrate that SRMCAN improves the fluency, accuracy, and professionalism of medical reports, providing radiologists with more valuable reference medical reports.

Dual-Modality Visual Feature Flow for Medical Report Generation

Automatic Report Generation Method Based on Multiscale Feature Extraction and Word Attention Network.

VMEKNet: Visual Memory and External Knowledge Based Network for Medical Report Generation.

Medical Report Generation Via Multimodal Spatio-Temporal Fusion

Enriching Semantic Features for Medical Report Generation.

CGFTrans: Cross-Modal Global Feature Fusion Transformer for Medical Report Generation

[Research on automatic generation of multimodal medical image reports based on memory driven]

Automatic Medical Image Report Generation with Multi-view and Multi-modal Attention Mechanism

Automatic Medical Report Generation Based on Cross-View Attention and Visual-Semantic Long Short Term Memorys

A label information fused medical image report generation framework

Visual-Textual Attentive Semantic Consistency for Medical Report Generation

Generating Radiology Reports Via Multi-feature Optimization Transformer

Unifying Relational Sentence Generation and Retrieval for Medical Image Report Composition

On the Automatic Generation of Medical Imaging Reports

Ultrasound Report Generation with Cross-Modality Feature Alignment via Unsupervised Guidance

Similarity Retrieval and Medical Cross-Modal Attention Based Medical Report Generation

A medical report generation method integrating teacher–student model and encoder–decoder network

Joint Embedding of Deep Visual and Semantic Features for Medical Image Report Generation

The Long-Term Memory Transformer with Multimodal Fusion for Radiology Report Generation

Multi-modal Fusion with Semantic Supervision for Radiology Report Generation

DAMPER: A Dual-Stage Medical Report Generation Framework with Coarse-Grained MeSH Alignment and Fine-Grained Hypergraph Matching