Abstract:Background and Objective: Automatic generation of medical reports reduces both the burden on radiologists and the possibility of errors due to the inexperience of radiologists. The model that utilizes attention mechanism and contrastive learning can generate medical reports by capturing both general and specific semantics. However, existing contrastive learning methods ignore the specificity of medical data, that is, a patient may suffer from multiple diseases at the same time. This means that the lack of fine-grained relationships for contrastive learning will lead to the problem of insufficient specificity. Methods: To address the above problem, a label correlated contrastive learning method is proposed to encourage the model to generate higher-quality reports. Firstly, the refined similarity description matrix of the contrastive relationship between the reports is obtained by calculating the similarities between the multi-label classification of the reports. Secondly, the representations of image features and the embeddings containing semantic information from the decoder are projected into a hidden space. Thirdly, label correlated contrastive learning is performed with the hidden representations of the image, the embeddings of the text, and the similarity matrix. Through contrastive learning, the “hard” negative samples that share more labels with the target sample are being assigned more weights. Finally, label correlated contrastive learning and attention mechanism are combined to generate reports. Results: Comprehensive experiments are conducted on widely used datasets, IU X-ray and MIMIC-CXR. Specifically, on IU X-ray dataset, our method achieves METEOR and ROUGE-L scores of 0.198 and 0.392, respectively. On MIMIC-CXR dataset, our method achieves precision, recall, and F-1 scores of 0.384, 0.376, and 0.304, respectively. The results indicate that proposed method outperforms previous state-of-the-art models. Conclusions: This work improves the performance of automatically generating medical reports, making their application in computer-aided diagnosis feasible.

Enhancing Biomedical Multi-modal Representation Learning with Multi-scale Pre-training and Perturbed Report Discrimination

Improving Medical Vision-Language Contrastive Pretraining with Semantics-aware Triage

MoRE: Multi-Modal Contrastive Pre-training with Transformers on X-Rays, ECGs, and Diagnostic Report

Contrastive pre-training and linear interaction attention-based transformer for universal medical reports generation

A Comparison of Pre-trained Vision-and-Language Models for Multimodal Representation Learning across Medical Images and Reports

Multimodal Pretraining from Monolingual to Multilingual

Medical Vision-Language Representation Learning with Cross-Modal Multi-Teacher Contrastive Distillation

Contrastive Cross-Modal Pre-Training: A General Strategy for Small Sample Medical Imaging

Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training

MITER: Medical Image–TExt joint adaptive pretRaining with multi-level contrastive learning

Multimodal Contrastive Training for Visual Representation Learning

Leveraging Vision-Language Pre-Trained Model and Contrastive Learning for Enhanced Multimodal Sentiment Analysis

Pre-training Multi-task Contrastive Learning Models for Scientific Literature Understanding

Dense Contrastive Visual-Linguistic Pretraining

Clinical-BERT: Vision-Language Pre-training for Radiograph Diagnosis and Reports Generation.

Cross-modal Contrastive Attention Model for Medical Report Generation.

Medical Vision-Language Pre-Training for Brain Abnormalities

Label Correlated Contrastive Learning for Medical Report Generation

Bidirectional Captioning for Clinically Accurate and Interpretable Models

Contrastive Learning of Medical Visual Representations from Paired Images and Text