Abstract:Medical image fusion aims to integrate the complementary information captured by images of different modalities into a more informative composite image. However, current study on medical image fusion suffers from several drawbacks: 1) existing methods are mostly designed for 2-D slice fusion, and they tend to lose spatial contextual information when fusing medical images with volumetric structure slice by slice individually; 2) the few existing 3-D medical image fusion methods fail in considering the characteristics of source modalities sufficiently, leading to the loss of important modality information; and 3) most existing works concentrate on pursuing good performance on visual perception and objective evaluation, while there is a severe lack of clinical problem-oriented study. In this article, to address these issues, we propose a multimodal MRI volumetric data fusion method based on an end-to-end convolutional neural network (CNN). In our network, an attention-based multimodal feature fusion (MMFF) module is presented for more effective feature learning. In addition, a specific loss function that considers the characteristics of different MRI modalities is designed to preserve the modality information. Experimental results demonstrate that the proposed method can obtain more competitive results on both visual quality and objective assessment when compared with some representative 3-D and 2-D medical image fusion methods. We further verify the significance of the proposed method for brain tumor segmentation by enriching the input modalities, and the results show that it is helpful to improve the segmentation accuracy. The source code of our fusion method is available at https://github.com/yuliu316316/3D-CNN-Fusion.

Multimodal Medical Image Fusion Based on Nonsubsampled Contourlet Transform Using Improved Pcnn

Pulse Coupled Neural Network-Based Multimodal Medical Image Fusion via Guided Filtering and WSEML in NSCT Domain

Multi-modality Anatomical and Functional Medical Image Fusion Based on Simplified-spatial Frequency-pulse Coupled Neural Networks and Region Energy-weighted Average Strategy in Non-sub Sampled Contourlet Transform Domain.

Multimodal medical image fusion algorithm based on pulse coupled neural networks and nonsubsampled contourlet transform

Medical Image Fusion with Parameter-Adaptive Pulse Coupled Neural Network in Nonsubsampled Shearlet Transform Domain

Multi-modal medical image fusion using improved dual-channel PCNN

Research of Multimodal Medical Image Fusion Based on Parameter-Adaptive Pulse-Coupled Neural Network and Convolutional Sparse Representation

A Medical Image Fusion Method Based on Convolutional Neural Networks

Feature-Motivated Simplified Adaptive PCNN-Based Medical Image Fusion Algorithm in NSST Domain

Multimodal MRI Volumetric Data Fusion With Convolutional Neural Networks

A novel fusion method based on dynamic threshold neural P systems and nonsubsampled contourlet transform for multi-modality medical images

A novel multi-focus image fusion method using PCNN in nonsubsampled contourlet transform domain

Medical Image Fusion Method Based on Coupled Neural P Systems in Nonsubsampled Shearlet Transform Domain

Log-Gabor energy based multimodal medical image fusion in NSCT domain

Multimodal medical image fusion based on interval gradients and convolutional neural networks

Medical Image Segmentation Based on Multi-Modal Convolutional Neural Network: Study on Image Fusion Schemes

Multimodal medical image fusion based on IHS and PCA

Multimodal medical image fusion using residual network 50 in non subsampled contourlet transform

Multimodal medical image fusion with convolution sparse representation and mutual information correlation in NSST domain

Detail-Enhanced Medical Image Fusion in NSCT Domain

A new method of medical image fusion based on nonsubsampled contourlet transform