Abstract:Multimodal medical image fusion aims to improve the clinical practicability of medical images by integrating complementary information from multiple medical images. However, in traditional fusion methods, the fusion rules based on prior knowledge or logic usually cannot match the feature representation perfectly, which results in partial information loss. Furthermore, most deep learning-based fusion methods depend on convolutional operations, which only focus on local features and have limited retention of context information. To address the above issues, we propose an end-to-end dynamic residual focal transformer network for multimodal medical image fusion, termed DRFT. The DRFT framework is an end-to-end network with no need to manually design fusion rules. Firstly, the context-gated convolution is introduced to construct the context dynamic extraction module (CDEM) to extract the key semantic information more accurately from multimodal medical images. Then, a new residual transformer fusion module (RTFM) is designed by incorporating the focal transformer into the residual mechanism, which can not only extract the deep semantic features, but also adaptively learn the optimal fusion scheme. Finally, the nest architecture is employed to extract multiscale features. In addition, a new objective function consisting of global detail loss and fusion enhancement loss is designed to enrich the modal information in the fused image. Notably, the proposed network does not require the two-stage training strategy as opposed to the traditional encoder–decoder fusion structure. Extensive experimental results on mainstream datasets show that, compared with the state-of-the-art methods, the proposed DRFT delivers better performance in both qualitative and quantitative evaluation.

CDRNet: Cascaded Dense Residual Network for Grayscale and Pseudocolor Medical Image Fusion.

SMRFnet: Saliency multi-scale residual fusion network for grayscale and pseudo color medical image fusion

Multi-Modal Medical Image Fusion Based on FusionNet in YIQ Color Space

Multimodal medical image fusion based on interval gradients and convolutional neural networks

MRI and PET/SPECT Image Fusion Based on Residual Network

Edge-Enhanced Dilated Residual Attention Network for Multimodal Medical Image Fusion

MDC-RHT: Multi-Modal Medical Image Fusion via Multi-Dimensional Dynamic Convolution and Residual Hybrid Transformer

Adaptive Multimodal Image Fusion with a Deep Pyramidal Residual Learning Network

An efficient Dense-Resnet for multimodal image fusion using medical image

M4FNet: Multimodal Medical Image Fusion Network Via Multi-Receptive-field and Multi-Scale Feature Integration.

Multi-modal medical image fusion using improved dual-channel PCNN

LRFNet: A real-time medical image fusion method guided by detail information

Medical Image Fusion Based on Adaptive Pulse Coupled Neural Networks and Nonsubsampled Contourlet Transform

Medical Image Fusion Via PCNN Based on Edge Preservation and Improved Sparse Representation in NSST Domain

Multimodal medical image fusion using residual network 50 in non subsampled contourlet transform

Multi-scale Feature Fusion Convolutional Neural Network for Multi-Modal Medical Image Fusion.

MCFNet: Multi-Layer Concatenation Fusion Network for Medical Images Fusion

MBRARN: multibranch residual attention reconstruction network for medical image fusion

End-to-end Dynamic Residual Focal Transformer Network for Multimodal Medical Image Fusion

DFENet: A dual-branch feature enhanced network integrating transformers and convolutional feature learning for multimodal medical image fusion

Medical image fusion via discrete stationary wavelet transform and an enhanced radial basis function neural network