Abstract:Owing to the limitations of imaging sensors, it is challenging to obtain a medical image that simultaneously contains functional metabolic information and structural tissue details. Multimodal medical image fusion, an effective way to merge the complementary information in different modalities, has become a significant technique to facilitate clinical diagnosis and surgical navigation. With powerful feature representation ability, deep learning (DL)-based methods have improved such fusion results but still have not achieved satisfactory performance. Specifically, existing DL-based methods generally depend on convolutional operations, which can well extract local patterns but have limited capability in preserving global context information. To compensate for this defect and achieve accurate fusion, we propose a novel unsupervised method to fuse multimodal medical images via a multiscale adaptive Transformer termed MATR. In the proposed method, instead of directly employing vanilla convolution, we introduce an adaptive convolution for adaptively modulating the convolutional kernel based on the global complementary context. To further model long-range dependencies, an adaptive Transformer is employed to enhance the global semantic extraction capability. Our network architecture is designed in a multiscale fashion so that useful multimodal information can be adequately acquired from the perspective of different scales. Moreover, an objective function composed of a structural loss and a region mutual information loss is devised to construct constraints for information preservation at both the structural-level and the feature-level. Extensive experiments on a mainstream database demonstrate that the proposed method outperforms other representative and state-of-the-art methods in terms of both visual quality and quantitative evaluation. We also extend the proposed method to address other biomedical image fusion issues, and the pleasing fusion results illustrate that MATR has good generalization capability. The code of the proposed method is available at https://github.com/tthinking/MATR.

MACTFusion: Lightweight Cross Transformer for Adaptive Multimodal Medical Image Fusion

MATR: Multimodal Medical Image Fusion Via Multiscale Adaptive Transformer.

Transformer-Based End-to-End Anatomical and Functional Image Fusion

Multi-Modal Image Fusion Via Deep Laplacian Pyramid Hybrid Network

MDC-RHT: Multi-Modal Medical Image Fusion via Multi-Dimensional Dynamic Convolution and Residual Hybrid Transformer

An Improved Hybrid Network With a Transformer Module for Medical Image Fusion

Cross Attention Multi Scale CNN-Transformer Hybrid Encoder is General Medical Image Learner.

Edge-Enhanced Dilated Residual Attention Network for Multimodal Medical Image Fusion

MixFuse: an Iterative Mix-Attention Transformer for Multi-Modal Image Fusion

Multimodal Transformer for Accelerated MR Imaging

RTFusion: A Multimodal Fusion Network with Significant Information Enhancement

MCFT: Multimodal Contrastive Fusion Transformer for Classification of Hyperspectral Image and LiDAR Data

EMOST: A dual-branch hybrid network for medical image fusion via efficient model module and sparse transformer

Multi-Modal Transformer for Accelerated MR Imaging

MRSCFusion: Joint Residual Swin Transformer and Multiscale CNN for Unsupervised Multimodal Medical Image Fusion.

Multi-modal medical image fusion based on densely-connected high-resolution CNN and hybrid transformer

FusionMamba: Dynamic Feature Enhancement for Multimodal Image Fusion with Mamba

AdaFuse: Adaptive Medical Image Fusion Based on Spatial-Frequential Cross Attention

Multimodal Token Fusion for Vision Transformers

M4FNet: Multimodal Medical Image Fusion Network Via Multi-Receptive-field and Multi-Scale Feature Integration.

TransFusion: Multi-view Divergent Fusion for Medical Image Segmentation with Transformers