Abstract:Background: Recent advances in artificial intelligence and digital image processing have inspired the use of deep neural networks for segmentation tasks in multimodal medical imaging. Unlike natural images, multimodal medical images contain much richer information regarding different modal properties and therefore present more challenges for semantic segmentation. However, there is no report on systematic research that integrates multi-scaled and structured analysis of single-modal and multimodal medical images. Methods: We propose a deep neural network, named as Modality Preserving U-Net (MPU-Net), for modality-preserving analysis and segmentation of medical targets from multimodal medical images. The proposed MPU-Net consists of a modality preservation encoder (MPE) module that preserves the feature independency among the modalities and a modality fusion decoder (MFD) module that performs a multiscale feature fusion analysis for each modality in order to provide a rich feature representation for the final task. The effectiveness of such a single-modal preservation and multimodal fusion feature extraction approach is verified by multimodal segmentation experiments and an ablation study using brain tumor and prostate datasets from Medical Segmentation Decathlon (MSD). Results: The segmentation experiments demonstrated the superiority of MPU-Net over other methods in the segmentation tasks for multimodal medical images. In the brain tumor segmentation tasks, the Dice scores (DSCs) for the whole tumor (WT), the tumor core (TC) and the enhancing tumor (ET) regions were 89.42%, 86.92%, and 84.59%, respectively. In the meanwhile, the 95% Hausdorff distance (HD95) results were 3.530, 4.899 and 2.555, respectively. In the prostate segmentation tasks, the DSCs for the peripheral zone (PZ) and the transitional zone (TZ) of the prostate were 71.20% and 90.38%, respectively. In the meanwhile, the 95% HD95 results were 6.367 and 4.766, respectively. The ablation study showed that the combination of single-modal preservation and multimodal fusion methods improved the performance of multimodal medical image feature analysis. Conclusions: In the segmentation tasks using brain tumor and prostate datasets, the MPU-Net method has achieved the improved performance in comparison with the conventional methods, indicating its potential application for other segmentation tasks in multimodal medical images.

Multimodal Biomedical Image Segmentation using Multi-Dimensional U-Convolutional Neural Network

MultiResUNet : Rethinking the U-Net Architecture for Multimodal Biomedical Image Segmentation

3D U$$^$$-Net: A 3D Universal U-Net for Multi-domain Medical Image Segmentation

U-Net Based Architecture for an Improved Multiresolution Segmentation in Medical Images

Mu-Net: Multi-Path Upsampling Convolution Network for Medical Image Segmentation

MSRD-Unet: Multiscale Residual Dilated U-Net for Medical Image Segmentation

A Multi-Scale Context Aware Attention Model for Medical Image Segmentation

CMM-Net: Contextual multi-scale multi-level network for efficient biomedical image segmentation

DC-UNet: Rethinking the U-Net Architecture with Dual Channel Efficient CNN for Medical Images Segmentation

AID-U-Net: An Innovative Deep Convolutional Architecture for Semantic Segmentation of Biomedical Images

ConnectedUNets++: Mass Segmentation from Whole Mammographic Images

DoubleU-NetPlus: A Novel Attention and Context Guided Dual U-Net with Multi-Scale Residual Feature Fusion Network for Semantic Segmentation of Medical Images

MDU-Net: Multi-scale Densely Connected U-Net for biomedical image segmentation

Attention Based Multi-Scale Nested Network for Biomedical Image Segmentation

MH UNet: A Multi-Scale Hierarchical Based Architecture for Medical Image Segmentation

Modality preserving U-Net for segmentation of multimodal medical images

A multi-attention and depthwise separable convolution network for medical image segmentation

MDA-Unet: A Multi-Scale Dilated Attention U-Net for Medical Image Segmentation

R2U++: a multiscale recurrent residual U-Net with dense skip connections for medical image segmentation

U-Net-Based Models towards Optimal MR Brain Image Segmentation