Abstract:Owing to the limitations of imaging sensors, it is challenging to obtain a medical image that simultaneously contains functional metabolic information and structural tissue details. Multimodal medical image fusion, an effective way to merge the complementary information in different modalities, has become a significant technique to facilitate clinical diagnosis and surgical navigation. With powerful feature representation ability, deep learning (DL)-based methods have improved such fusion results but still have not achieved satisfactory performance. Specifically, existing DL-based methods generally depend on convolutional operations, which can well extract local patterns but have limited capability in preserving global context information. To compensate for this defect and achieve accurate fusion, we propose a novel unsupervised method to fuse multimodal medical images via a multiscale adaptive Transformer termed MATR. In the proposed method, instead of directly employing vanilla convolution, we introduce an adaptive convolution for adaptively modulating the convolutional kernel based on the global complementary context. To further model long-range dependencies, an adaptive Transformer is employed to enhance the global semantic extraction capability. Our network architecture is designed in a multiscale fashion so that useful multimodal information can be adequately acquired from the perspective of different scales. Moreover, an objective function composed of a structural loss and a region mutual information loss is devised to construct constraints for information preservation at both the structural-level and the feature-level. Extensive experiments on a mainstream database demonstrate that the proposed method outperforms other representative and state-of-the-art methods in terms of both visual quality and quantitative evaluation. We also extend the proposed method to address other biomedical image fusion issues, and the pleasing fusion results illustrate that MATR has good generalization capability. The code of the proposed method is available at https://github.com/tthinking/MATR.

Discriminative Fusion of Moments-Aligned Latent Representation of Multimodality Medical Data.

Interpretation on Deep Multimodal Fusion for Diagnostic Classification

Multimodal risk prediction with physiological signals, medical images and clinical notes

Missing-modality Enabled Multi-modal Fusion Architecture for Medical Data

Multimodal Medical Image Fusion: The Perspective of Deep Learning

Research on Multimodal Fusion of Temporal Electronic Medical Records

DrFuse: Learning Disentangled Representation for Clinical Multi-Modal Fusion with Missing Modality and Modal Inconsistency

Multimodal learning for fetal distress diagnosis using a multimodal medical information fusion framework

Coupled Feature Learning for Multimodal Medical Image Fusion

Supervised Multi-Modal Fission Learning

Weakly paired multimodal fusion using multilayer extreme learning machine

Multimodal sensor fusion in the latent representation space

Multimodal Fusion Learning with Dual Attention for Medical Imaging

Deep Multi-modal Latent Representation Learning for Automated Dementia Diagnosis

Multi-modal Multi-instance Learning Using Weakly Correlated Histopathological Images and Tabular Clinical Information

Multi-modal Neuroimaging Data Fusion Via Latent Space Learning for Alzheimer’s Disease Diagnosis

The Role of Spatial Alignment in Multimodal Medical Image Fusion Using Deep Learning for Diagnostic Problems

MDC-RHT: Multi-Modal Medical Image Fusion via Multi-Dimensional Dynamic Convolution and Residual Hybrid Transformer

Multimodal medical image fusion using convolutional neural network and extreme learning machine

MATR: Multimodal Medical Image Fusion Via Multiscale Adaptive Transformer.

Application of Multimodal Fusion Deep Learning Model in Disease Recognition