Abstract:The accurate whole heart segmentation (WHS) of multi-modality medical images including magnetic resonance image (MRI) and computed tomography (CT) plays an important role in many clinical applications, such as accurate preoperative diagnosis planning and intraoperative treatment. Considering that the shape information of each component of the whole heart is complementary, we can extract multi-modality features and obtain the final segmentation results by fusing MRI and CT images. In this paper, we proposed a multi-modality transfer learning network with adversarial training (MMTLNet) for 3D multi-modality whole heart segmentation. Firstly, the network transfers the source domain (MRI domain) to the target domain (CT domain) by reconstructing the MRI images with a generator network and optimizing the reconstructed MRI images with a discriminator network, which enables us to fuse the MRI images with CT images to fully utilize the useful information from images in multi-modality for segmentation task. Secondly, to retain the useful information and remove the redundant information for accurate segmentation, we introduce the spatial attention mechanism into the backbone connection of UNet network to optimize the feature extraction between layers, and add channel attention mechanism at the jump connection to optimize the information extracted from the low-level feature map. Thirdly, we propose a new loss function in the adversarial training by introducing a weighted coefficient to distribute the proportion between Dice coefficient loss and generator loss, which can not only ensure the images to be correctly transferred from MRI domain to CT domain, but also achieve accurate segmentation with the transferred domain. We extensively evaluated our method on the data set of the multi-modality whole heart segmentation (MM-WHS) challenge, in conjunction with MICCAI 2017. The dice values of whole heart segmentation are 0.914 (CT images) and 0.890 (MRI images), which are both higher than the state-of-the-art.

Attention-guided model for mitral regurgitation analysis based on multi-task learning

A Multi-Scale Residual Attention Network for Retinal Vessel Segmentation

Mixed Transformer U-Net for Medical Image Segmentation

SK-Unet: an Improved U-net Model with Selective Kernel for the Segmentation of Multi-sequence Cardiac MR

Segmentation of Echocardiography Based on Deep Learning Model

Multiview Two-Task Recursive Attention Model for Left Atrium and Atrial Scars Segmentation

H2MaT-Unet:Hierarchical hybrid multi-axis transformer based Unet for medical image segmentation

Mitral Valve Segmentation and Tracking from Transthoracic Echocardiography Using Deep Learning

Residual UNet with spatial and channel attention for automatic magnetic resonance image segmentation of rectal cancer

Mitral Regurgitation Quantification from Multi-channel Ultrasound Images via Deep Learning

NVTrans-UNet: Neighborhood vision transformer based U-Net for multi-modal cardiac MR image segmentation

Multi-task Learning Framework for Echocardiography Segmentation

SK-Unet: An Improved U-Net Model With Selective Kernel for the Segmentation of LGE Cardiac MR Images

A Deep Learning-Based Fully Automated Pipeline for Regurgitant Mitral Valve Anatomy Analysis From 3D Echocardiography

MMTLNet: Multi-Modality Transfer Learning Network with adversarial training for 3D whole heart segmentation

Multiresolution Aggregation Transformer UNet Based on Multiscale Input and Coordinate Attention for Medical Image Segmentation

Machine Learning for Automated Mitral Regurgitation Detection from Cardiac Imaging

A Deep Learning-Based and Fully Automated Pipeline for Regurgitant Mitral Valve Anatomy Analysis from 3D Echocardiography

Automatic 3D mitral valve leaflet segmentation and validation of quantitative measurement

AHU-MultiNet: Adaptive loss balancing based on homoscedastic uncertainty in multi-task medical image segmentation network

Cross-modality Guidance-aided Multi-modal Learning with Dual Attention for MRI Brain Tumor Grading