CDBIFusion: A Cross-Domain Bidirectional Interaction Fusion Network for PET and MRI Images.

Jie Zhang,Bicao Li,Bei Wang,Zhuhong Shao,Jie Huang,Jiaxi Lu

DOI: https://doi.org/10.1007/978-981-99-8558-6_36

2024-01-01

Abstract:The existing methodologies for image fusion in deep learning predominantly focus on convolutional neural networks (CNN). However, recent investigations have explored the utilization of Transformer models to enhance fusion performance. Hence, we propose a novel generative adversarial fusion framework that combines both CNN and Transformer, referred to as the Cross-Domain Bidirectional Interaction Fusion Network (CDBIFusion). Specifically, we devise three distinct pathways for the generator, each serving a unique purpose. The two CNN pathways are employed to capture local information from MRI and PET images, and the other pathway adopts a transformer architecture that cascades both source images as input, enabling the exploitation of global correlations. Moreover, we present a cross-domain bidirectional interaction (CDBI) module that facilitates the retention and interaction of information deactivated by the ReLU activation function between two CNN paths. The interaction operates by cross-cascading ReLU activation features and deactivation features from separate paths by two ReLU rectifiers and then delivering them to the other path, thus reducing the valuable information lost through deactivation. Extensive experiments have demonstrated that our CDBIFusion surpasses other current methods in terms of subjective perception and objective evaluation.