Abstract:The accurate whole heart segmentation (WHS) of multi-modality medical images including magnetic resonance image (MRI) and computed tomography (CT) plays an important role in many clinical applications, such as accurate preoperative diagnosis planning and intraoperative treatment. Considering that the shape information of each component of the whole heart is complementary, we can extract multi-modality features and obtain the final segmentation results by fusing MRI and CT images. In this paper, we proposed a multi-modality transfer learning network with adversarial training (MMTLNet) for 3D multi-modality whole heart segmentation. Firstly, the network transfers the source domain (MRI domain) to the target domain (CT domain) by reconstructing the MRI images with a generator network and optimizing the reconstructed MRI images with a discriminator network, which enables us to fuse the MRI images with CT images to fully utilize the useful information from images in multi-modality for segmentation task. Secondly, to retain the useful information and remove the redundant information for accurate segmentation, we introduce the spatial attention mechanism into the backbone connection of UNet network to optimize the feature extraction between layers, and add channel attention mechanism at the jump connection to optimize the information extracted from the low-level feature map. Thirdly, we propose a new loss function in the adversarial training by introducing a weighted coefficient to distribute the proportion between Dice coefficient loss and generator loss, which can not only ensure the images to be correctly transferred from MRI domain to CT domain, but also achieve accurate segmentation with the transferred domain. We extensively evaluated our method on the data set of the multi-modality whole heart segmentation (MM-WHS) challenge, in conjunction with MICCAI 2017. The dice values of whole heart segmentation are 0.914 (CT images) and 0.890 (MRI images), which are both higher than the state-of-the-art.

Multitask Learning for Estimating Multitype Cardiac Indices in MRI and CT Based on Adversarial Reverse Mapping

Simulating cross‐modal medical images using multi‐task adversarial learning of a deep convolutional neural network

Direct Multitype Cardiac Indices Estimation via Joint Representation and Regression Learning

A Multi-Task Network For Cardiac Magnetic Resonance Image Segmentation And Classification

Spatio-temporal Multi-Task Network Cascade for Accurate Assessment of Cardiac CT Perfusion.

Iterative Temporal-Spatial Transformer-based Cardiac T1 Mapping MRI Reconstruction

Multi-task Deep Learning for Cerebrovascular Disease Classification and MRI-to-PET Translation

A Multi-Task Cross-Task Learning Architecture for Ad-hoc Uncertainty Estimation in 3D Cardiac MRI Image Segmentation

Enhancing CT Image synthesis from multi-modal MRI data based on a multi-task neural network framework

Recognition and standardization of cardiac MRI orientation via multi-tasking learning and deep neural networks

Multitask Learning for Improved Late Mechanical Activation Detection of Heart from Cine DENSE MRI

Using Multi-Task Learning to Improve Diagnostic Performance of Convolutional Neural Networks

A Framework for Deep Multitask Learning With Multiparametric Magnetic Resonance Imaging for the Joint Prediction of Histological Characteristics in Breast Cancer

One-stage Multi-task Detector for 3D Cardiac MR Imaging

Toward Accurate Cardiac MRI Segmentation With Variational Autoencoder-Based Unsupervised Domain Adaptation

Multi-sequence Cardiac MR Segmentation with Adversarial Domain Adaptation Network

Region-focused Multi-View Transformer-Based Generative Adversarial Network for Cardiac Cine MRI Reconstruction

Multi-level Temporal Information Sharing Transformer-Based Feature Reuse Network for Cardiac MRI Reconstruction.

Alternating low‐rank tensor reconstruction for improved multiparametric mapping with cardiovascular MR Multitasking

MMTLNet: Multi-Modality Transfer Learning Network with adversarial training for 3D whole heart segmentation

Unsupervised Domain Adaptation from Axial to Short-Axis Multi-Slice Cardiac MR Images by Incorporating Pretrained Task Networks