Abstract:The state-of-the-art models for medical image segmentation are variants of U-Net and fully convolutional networks (FCN). Despite their success, these models have two limitations: (1) their optimal depth is apriori unknown, requiring extensive architecture search or inefficient ensemble of models of varying depths; and (2) their skip connections impose an unnecessarily restrictive fusion scheme, forcing aggregation only at the same-scale feature maps of the encoder and decoder sub-networks. To overcome these two limitations, we propose UNet++, a new neural architecture for semantic and instance segmentation, by (1) alleviating the unknown network depth with an efficient ensemble of U-Nets of varying depths, which partially share an encoder and co-learn simultaneously using deep supervision; (2) redesigning skip connections to aggregate features of varying semantic scales at the decoder sub-networks, leading to a highly flexible feature fusion scheme; and (3) devising a pruning scheme to accelerate the inference speed of UNet++. We have evaluated UNet++ using six different medical image segmentation datasets, covering multiple imaging modalities such as computed tomography (CT), magnetic resonance imaging (MRI), and electron microscopy (EM), and demonstrating that (1) UNet++ consistently outperforms the baseline models for the task of semantic segmentation across different datasets and backbone architectures; (2) UNet++ enhances segmentation quality of varying-size objects—an improvement over the fixed-depth U-Net; (3) Mask RCNN++ (Mask R-CNN with UNet++ design) outperforms the original Mask R-CNN for the task of instance segmen-ation; and (4) pruned UNet++ models achieve significant speedup while showing only modest performance degradation. Our implementation and pre-trained models are available at https://github.com/MrGiovanni/UNetPlusPlus.

Competition vs. Concatenation in Skip Connections of Fully Convolutional Networks

Select, Attend, and Transfer: Light, Learnable Skip Connections

The Importance of Skip Connections in Biomedical Image Segmentation

Fully Convolutional Networks for Semantic Segmentation

Enhanced Multi-Scale Feature Adaptive Fusion Sparse Convolutional Network for Large-Scale Scenes Semantic Segmentation

Translated Skip Connections -- Expanding the Receptive Fields of Fully Convolutional Neural Networks

Concurrent Spatial and Channel Squeeze & Excitation in Fully Convolutional Networks

A Long Skip Connection for Enhanced Color Selectivity in CNN Architectures

FSC-UNet: a Lightweight Medical Image Segmentation Algorithm Fused with Skip Connections

Faster SCDNet: Real-Time Semantic Segmentation Network with Split Connection and Flexible Dilated Convolution

SSFCN-DCRF

A Convolutional Encoder-Decoder Network with Skip Connections for Saliency Prediction

Constrained Multi-scale Dense Connections for Accurate Biomedical Image Segmentation

Rethinking Skip Connections in Encoder-decoder Networks for Monocular Depth Estimation

Real-time Foreground Object Segmentation Networks Using Long and Short Skip Connections

USCT-UNet: Rethinking the Semantic Gap in U-Net Network From U-Shaped Skip Connections With Multichannel Fusion Transformer

Dense Convolutional Networks for Semantic Segmentation.

Convolutional Networks with Dense Connectivity

Towards bi-directional skip connections in encoder-decoder architectures and beyond

BEA-Net: Body and Edge Aware Network with Multi-Scale Short-Term Concatenation for Medical Image Segmentation.

UNet++: Redesigning Skip Connections to Exploit Multiscale Features in Image Segmentation