Abstract:Medical imaging is an important tool for clinical diagnosis. Nevertheless, it is very time-consuming and error-prone for physicians to prepare imaging diagnosis reports. Therefore, it is necessary to develop some methods to generate medical imaging reports automatically. Currently, the task of medical imaging report generation is challenging in at least two aspects: (1) medical images are very similar to each other. The differences between normal and abnormal images and between different abnormal images are usually trivial; (2) unrelated or incorrect keywords describing abnormal findings in the generated reports lead to mis-communications. In this paper, we propose a medical image report generation framework composed of four modules, including a Transformer encoder, a MIX-MLP multi-label classification network, a co-attention mechanism (CAM) based semantic and visual feature fusion, and a hierarchical LSTM decoder. The Transformer encoder can be used to learn long-range dependencies between images and labels, effectively extract visual and semantic features of images, and establish long-term dependent relationships between visual and semantic information to accurately extract abnormal features from images. The MIX-MLP multi-label classification network, the co-attention mechanism and the hierarchical LSTM network can better identify abnormalities, achieving visual and text alignment fusion and multi-label diagnostic classification to better facilitate report generation. The results of the experiments performed on two widely used radiology report datasets, IU X-RAY and MIMIC-CXR, show that our proposed framework outperforms current report generation models in terms of both natural linguistic generation metrics and clinical efficacy assessment metrics. The code of this work is available online at https://github.com/watersunhznu/LIFMRG.

Personality Traits and Sibling Relationships in Emerging Adults

Token Imbalance Adaptation for Radiology Report Generation

A medical report generation method integrating teacher–student model and encoder–decoder network

Complex Mixer for MedMNIST Classification Decathlon

A label information fused medical image report generation framework

AIMNet: Adaptive Image-Tag Merging Network For Automatic Medical Report Generation

SMMix: Self-Motivated Image Mixing for Vision Transformers

Memory-based Cross-modal Semantic Alignment Network for Radiology Report Generation

Adversarial Training with Comprehensive Objective for Medical Image Report Generation.

Mixer is more than just a model

Hierarchical medical image report adversarial generation with hybrid discriminator

Automatic Radiology Reports Generation via Memory Alignment Network

Joint Embedding of Deep Visual and Semantic Features for Medical Image Report Generation

A Self-boosting Framework for Automated Radiographic Report Generation

TokenCompose: Text-to-Image Diffusion with Token-level Supervision

Joint Imbalance Adaptation for Radiology Report Generation

MATNet: Exploiting Multi-Modal Features for Radiology Report Generation.

TiMix: Text-aware Image Mixing for Effective Vision-Language Pre-training

AutoMO-Mixer: An automated multi-objective Mixer model for balanced, safe and robust prediction in medicine

SynthMix: Mixing up Aligned Synthesis for Medical Cross-Modality Domain Adaptation

VLMixer: Unpaired Vision-Language Pre-training via Cross-Modal CutMix