Abstract:Background: Recent advances in artificial intelligence and digital image processing have inspired the use of deep neural networks for segmentation tasks in multimodal medical imaging. Unlike natural images, multimodal medical images contain much richer information regarding different modal properties and therefore present more challenges for semantic segmentation. However, there is no report on systematic research that integrates multi-scaled and structured analysis of single-modal and multimodal medical images. Methods: We propose a deep neural network, named as Modality Preserving U-Net (MPU-Net), for modality-preserving analysis and segmentation of medical targets from multimodal medical images. The proposed MPU-Net consists of a modality preservation encoder (MPE) module that preserves the feature independency among the modalities and a modality fusion decoder (MFD) module that performs a multiscale feature fusion analysis for each modality in order to provide a rich feature representation for the final task. The effectiveness of such a single-modal preservation and multimodal fusion feature extraction approach is verified by multimodal segmentation experiments and an ablation study using brain tumor and prostate datasets from Medical Segmentation Decathlon (MSD). Results: The segmentation experiments demonstrated the superiority of MPU-Net over other methods in the segmentation tasks for multimodal medical images. In the brain tumor segmentation tasks, the Dice scores (DSCs) for the whole tumor (WT), the tumor core (TC) and the enhancing tumor (ET) regions were 89.42%, 86.92%, and 84.59%, respectively. In the meanwhile, the 95% Hausdorff distance (HD95) results were 3.530, 4.899 and 2.555, respectively. In the prostate segmentation tasks, the DSCs for the peripheral zone (PZ) and the transitional zone (TZ) of the prostate were 71.20% and 90.38%, respectively. In the meanwhile, the 95% HD95 results were 6.367 and 4.766, respectively. The ablation study showed that the combination of single-modal preservation and multimodal fusion methods improved the performance of multimodal medical image feature analysis. Conclusions: In the segmentation tasks using brain tumor and prostate datasets, the MPU-Net method has achieved the improved performance in comparison with the conventional methods, indicating its potential application for other segmentation tasks in multimodal medical images.

MP-Net: A Multi-Center Privacy-Preserving Network for Medical Image Segmentation

Mmy-net: a Multimodal Network Exploiting Image and Patient Metadata for Simultaneous Segmentation and Diagnosis

WNET: an End-to-End Atlas-Guided and Boundary-Enhanced Network for Medical Image Segmentation

Split-U-Net: Preventing Data Leakage in Split Learning for Collaborative Multi-Modal Brain Tumor Segmentation

Mixup-Privacy: A simple yet effective approach for privacy-preserving segmentation

A novel medical image segmentation approach by using multi-branch segmentation network based on local and global information synchronous learning

MSMCNet: Differential context drives accurate localization and edge smoothing of lesions for medical image segmentation

BUNET: Blind Medical Image Segmentation Based on Secure UNET

MFLUnet: multi-scale fusion lightweight Unet for medical image segmentation

Towards Privacy-Preserving Medical Imaging: Federated Learning with Differential Privacy and Secure Aggregation Using a Modified ResNet Architecture

Modality preserving U-Net for segmentation of multimodal medical images

TransCS-Net: A hybrid transformer-based privacy-protecting network using compressed sensing for medical image segmentation

A separable privacy-preserving technique based on reversible medical data hiding in plaintext encrypted images using neural network

APRNet: A 3D Anisotropic Pyramidal Reversible Network with Multi-Modal Cross-Dimension Attention for Brain Tissue Segmentation in MR Images.

Multi-perspective feature compensation enhanced network for medical image segmentation

Mu-Net: Multi-Path Upsampling Convolution Network for Medical Image Segmentation

CM-SegNet: A Deep Learning-Based Automatic Segmentation Approach for Medical Images by Combining Convolution and Multilayer Perceptron

A Privacy Protection Framework for Medical Image Security without Key Dependency Based on Visual Cryptography and Trusted Computing

MpMsCFMA-Net: Multi-path Multi-scale Context Feature Mixup and Aggregation Network for medical image segmentation

TMS-Net: A Segmentation Network Coupled With A Run-time Quality Control Method For Robust Cardiac Image Segmentation

NMNet: Learning Multi-level Semantic Information from Scale Extension Domain for Improved Medical Image Segmentation