Abstract:Medical image fusion aims to integrate the complementary information captured by images of different modalities into a more informative composite image. However, current study on medical image fusion suffers from several drawbacks: 1) existing methods are mostly designed for 2-D slice fusion, and they tend to lose spatial contextual information when fusing medical images with volumetric structure slice by slice individually; 2) the few existing 3-D medical image fusion methods fail in considering the characteristics of source modalities sufficiently, leading to the loss of important modality information; and 3) most existing works concentrate on pursuing good performance on visual perception and objective evaluation, while there is a severe lack of clinical problem-oriented study. In this article, to address these issues, we propose a multimodal MRI volumetric data fusion method based on an end-to-end convolutional neural network (CNN). In our network, an attention-based multimodal feature fusion (MMFF) module is presented for more effective feature learning. In addition, a specific loss function that considers the characteristics of different MRI modalities is designed to preserve the modality information. Experimental results demonstrate that the proposed method can obtain more competitive results on both visual quality and objective assessment when compared with some representative 3-D and 2-D medical image fusion methods. We further verify the significance of the proposed method for brain tumor segmentation by enriching the input modalities, and the results show that it is helpful to improve the segmentation accuracy. The source code of our fusion method is available at https://github.com/yuliu316316/3D-CNN-Fusion.

Advances in multimodality data fusion in neuroimaging

Advances in multimodal data fusion in neuroimaging: Overview, challenges, and novel orientation

Multimodal Fusion of Brain Imaging Data: Methods and Applications

A review of multivariate methods for multimodal fusion of brain imaging data

Multimodal MRI Volumetric Data Fusion With Convolutional Neural Networks

Editorial: Multimodal Brain Image Fusion: Methods, Evaluations, and Applications

The Application and Progress of Multimodality Image Fusion in Neurosurgery

Deep Multimodal Data Fusion

Multimodal Fusion on Low-quality Data: A Comprehensive Survey

A Survey on Deep Learning for Multimodal Data Fusion

Multi-modal bioelectrical signal fusion analysis based on different acquisition devices and scene settings: Overview, challenges, and novel orientation

Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects

A Review of Multimodal Medical Image Fusion Techniques

Analysis of Multimodal Data Fusion from an Information Theory Perspective

Multimodal Medical Image Fusion: The Perspective of Deep Learning

Fusing DTI and fMRI data: A survey of methods and applications

A Comprehensive Survey on Deep Learning Multi-Modal Fusion: Methods, Technologies and Applications

Interpretation on Deep Multimodal Fusion for Diagnostic Classification

Chromatic fusion: generative multimodal neuroimaging data fusion provides multi-informed insights into schizophrenia

Multimodal medical image fusion algorithm in the era of big data

Multimodal medical image fusion review: Theoretical background and recent advances