Abstract:If unaligned multimodal medical images can be simultaneously aligned and fused using a single-stage approach within a unified processing framework, it will not only achieve mutual promotion of dual tasks but also help reduce the complexity of the model. However, the design of this model faces the challenge of incompatible requirements for feature fusion and alignment; specifically, feature alignment requires consistency among corresponding features, whereas feature fusion requires the features to be complementary to each other. To address this challenge, this paper proposes an unaligned medical image fusion method called Bidirectional Stepwise Feature Alignment and Fusion (BSFA-F) strategy. To reduce the negative impact of modality differences on cross-modal feature matching, we incorporate the Modal Discrepancy-Free Feature Representation (MDF-FR) method into BSFA-F. MDF-FR utilizes a Modality Feature Representation Head (MFRH) to integrate the global information of the input image. By injecting the information contained in MFRH of the current image into other modality images, it effectively reduces the impact of modality differences on feature alignment while preserving the complementary information carried by different images. In terms of feature alignment, BSFA-F employs a bidirectional stepwise alignment deformation field prediction strategy based on the path independence of vector displacement between two points. This strategy solves the problem of large spans and inaccurate deformation field prediction in single-step alignment. Finally, Multi-Modal Feature Fusion block achieves the fusion of aligned features. The experimental results across multiple datasets demonstrate the effectiveness of our method. The source code is available at <a class="link-external link-https" href="https://github.com/slrl123/BSAFusion" rel="external noopener nofollow">this https URL</a>.

The Role of Spatial Alignment in Multimodal Medical Image Fusion Using Deep Learning for Diagnostic Problems

Multimodal Medical Image Fusion: The Perspective of Deep Learning

Adaptive spatial and frequency experts fusion network for medical image fusion

Interpretation on Deep Multimodal Fusion for Diagnostic Classification

Medical image fusion with deep neural networks

Adaptive Multimodal Image Fusion with a Deep Pyramidal Residual Learning Network

Deep learning and multimodal feature fusion for the aided diagnosis of Alzheimer's disease

Multimodal medical image fusion and classification using deep learning techniques

Hierarchical Progressive Network for Multimodal Medical Image Fusion in Healthcare Systems

Three-Dimensional Medical Image Fusion with Deformable Cross-Attention

Deep learning methods for medical image fusion: A review

Multimodal medical image fusion using convolutional neural network and extreme learning machine

MSAIF-Net: A Multistage Spatial Attention-Based Invertible Fusion Network for MR Images.

BSAFusion: A Bidirectional Stepwise Feature Alignment Network for Unaligned Medical Image Fusion

AdaFuse: Adaptive Medical Image Fusion Based on Spatial-Frequential Cross Attention

An ensemble deep learning model for medical image fusion with Siamese neural networks and VGG-19

A review: Deep learning for medical image segmentation using multi-modality fusion

SS-SSAN: a Self-Supervised Subspace Attentional Network for Multi-Modal Medical Image Fusion

Discriminative Fusion of Moments-Aligned Latent Representation of Multimodality Medical Data.

Multimodal medical image fusion based on interval gradients and convolutional neural networks

Multimodal cross enhanced fusion network for diagnosis of Alzheimer?s disease and