End-to-End Clustering Enhanced Contrastive Learning for Radiology Reports Generation

Xinyao Liu,Junchang Xin,Qi Shen,Chuangang Li,Zhihong Huang,Zhiqiong Wang

DOI: https://doi.org/10.1109/tetci.2024.3449876

2024-01-01

IEEE Transactions on Emerging Topics in Computational Intelligence

Abstract:With the rapid growth of medical imaging data, radiologists must dedicate a significant amount of time to report writing. Automated generation of radiology reports not only alleviates the heavy workload of physicians but, more importantly, can reduce mistakes and oversights caused by insufficient experience. However, due to substantial data bias in medical data, prior studies using typical cross-entropy loss in encoder-decoder architectures often result in generalized descriptions of normal tissues and may overlook crucial clinical abnormalities. Therefore, we propose a clustering enhanced contrastive learning model named CECL to generate more diverse radiology reports, and it is worth noting that our method is end-to-end trainable. Specifically, an adaptive alignment fusion encoder-decoder network (AAF) is constructed by fusing the image features and text semantic features from the transformer decoder, eliminating information redundancy across different modalities. Moreover, a label-guided contrastive learning (LCL) module is proposed. In detail, clustering is performed on the fused features using Gaussian competition. Supervised contrastive learning is conducted based on the clustering results to enhance feature representation ability. We evaluate the CECL on two widely used publicly available datasets, IU X-ray and MIMIC-CXR, using NLG and CE metrics. The experimental results demonstrate that CECL can produce fluent reports with more descriptions of anomalies, outperforming other state-of-the-art methods with higher clinical correctness.