Abstract:Recently, progress on semantic image segmentation is substantial, benefiting from the rapid development of Convolutional Neural Networks. Semantic image segmentation approaches proposed lately have been mostly based on Fully convolutional Networks (FCNs). However, these FCN-based methods use large receptive fields and too many pooling layers to depict the discriminative semantic information of the images. Specifically, on one hand, convolutional kernel with large receptive field smooth the detailed edges, since too much contexture information is used to depict the “center pixel.” However, the pooling layer increases the receptive field through zooming out the latest feature maps, which loses many detailed information of the image, especially in the deeper layers of the network. These operations often cause low spatial resolution inside deep layers, which leads to spatially fragmented prediction. To address this problem, we exploit the inherent multi-scale and pyramidal hierarchy of deep convolutional networks to extract the feature maps with different resolutions and take full advantages of these feature maps via a gradually stacked fusing way. Specifically, for two adjacent convolutional layers, we upsample the features from deeper layer with stride of 2 and then stack them on the features from shallower layer. Then, a convolutional layer with kernels of 1× 1 is followed to fuse these stacked features. The fused feature preserves the spatial structure information of the image; meanwhile, it owns strong discriminative capability for pixel classification. Additionally, to further preserve the spatial structure information and regional connectivity of the predicted category label map, we propose a novel loss term for the network. In detail, two graph model-based spatial affinity matrixes are proposed, which are used to depict the pixel-level relationships in the input image and predicted category label map respectively, and then their cosine distance is backward propagated to the network. The proposed architecture, called spatial structure preserving feature pyramid network, significantly improves the spatial resolution of the predicted category label map for semantic image segmentation. The proposed method achieves state-of-the-art results on three public and challenging datasets for semantic image segmentation.

Multi-branch Input Structure for Pyramid Scene Parsing Network

MBNet: A Multi-Resolution Branch Network for Semantic Segmentation Of Ultra-High Resolution Images

Scene Classification Via Learning a Multi-Branch Convolutional Network.

Semantic Segmentation Network with Multi-Path Structure, Attention Reweighting and Multi-Scale Encoding

Cross Guided and Pyramid Aggregation Networks for Real-time Semantic Segmentation

Multi‐stream Densely Connected Network for Semantic Segmentation

A Multi-Branch Feature Fusion Strategy Based on an Attention Mechanism for Remote Sensing Image Scene Classification

Spatial Structure Preserving Feature Pyramid Network for Semantic Image Segmentation

Real-Time Semantic Segmentation via an Efficient Multi-Column Network

DPNet: Dual-Pyramid Semantic Segmentation Network Based on Improved Deeplabv3 Plus

Multi-Path Spatial Detail-Aware Network for Semantic Segmentation

Synthetical application of multi-feature map detection and multi-branch convolution

Deep Multi-Branch Aggregation Network for Real-Time Semantic Segmentation in Street Scenes.

Attention Pyramid Module for Scene Recognition

FCPFNet: Feature Complementation Network with Pyramid Fusion for Semantic Segmentation

Pyramid Architecture for Multi-Scale Processing in Point Cloud Segmentation

Bilateral Network with Residual U-blocks and Dual-Guided Attention for Real-time Semantic Segmentation

PCANet: Pyramid convolutional attention network for semantic segmentation

High Resolution Scene Parsing Network Based on Semantic Segmentation

Quadtree Generating Networks: Efficient Hierarchical Scene Parsing with Sparse Convolutions

Multi-module Spatial Semantic Network for Semantic Segmentation