Abstract:Humans are able to recognize structured relations in observation, allowing us to decompose complex scenes into simpler parts and abstract the visual world at multiple levels. However, such hierarchical reasoning ability of human perception remains largely unexplored in current literature of semantic segmentation. Existing works are often aware of flatten labels and distinguish all the semantic categories exclusively for each pixel. In this work, we instead address hierarchical semantic segmentation (HSS), with the aim of providing a structured, pixel-wise description of visual observation in terms of a class hierarchy. We devise Hssn , a general HSS framework that tackles two critical issues in this task: i) how to efficiently adapt existing hierarchy-agnostic segmentation networks to the HSS setting, and ii) how to leverage the class hierarchy to regularize HSS network learning. To address i) , Hssn directly casts HSS as a pixel-wise multi-label classification task, only bringing minimal architecture change to current segmentation models. To solve ii) , Hssn first explores inherent properties of the hierarchy as a training objective, which enforces segmentation predictions to obey the hierarchy structure. Furthermore, with a set of hierarchy-induced margin constraints, Hssn efficiently reshapes the learned pixel embedding space, so as to generate hierarchy-aware pixel representations and facilitate structured segmentation eventually. Building upon Hssn , we further exploit the mutual exclusion relation between semantic labels and strengthen the margin based regularization strategy with more meaningful constrains, leading to Hssn +, a more effective framework for HSS. We conduct extensive experiments on six semantic segmentation datasets (i.e., Mapillary Vistas 2.0, Cityscapes, LIP, PASCAL-Person-Part, PASCAL-Part-58, and PASCAL-Part-108), with different class hierarchies, network architectures, and backbones, and the results confirm the generalization and superiority of our algorithms.

Image object's semantic hierarchy and its unsupervised learning algrithm

Semantic Image Segmentation Based on Spatial Context Relations

Learning Object Spatial Relationship from Demonstration

SSC: Semantic Scan Context for Large-Scale Place Recognition

Semantic Hierarchy-Aware Segmentation

Semantic and Spatial Co-Occurrence Analysis on Object Pairs for Urban Scene Classification.

Semantic-Spatial Matching for Image Classification

Deep Hierarchical Semantic Segmentation

Semantic image classification using statistical local spatial relations model

Unsupervised object category discovery via information bottleneck method.

On Support Relations Inference and Scene Hierarchy Graph Construction from Point Cloud in Clustered Environments

SENSC Algorithm for Object and Scene Categorization.

Object coding on the semantic graph for scene classification.

Vocabulary Hierarchy Optimisation Based on Spatial Context and Category Information

Spatial Structure Constraints for Weakly Supervised Semantic Segmentation

Learning Hierarchical Space Tiling for Scene Modeling, Parsing and Attribute Tagging

Hierarchical object representations for visual recognition via weakly supervised learning

Image Annotations Based on Semi-supervised Clustering with Semantic Soft Constraints.

Semantic-Enhanced Image Clustering

Unsupervised Hierarchical Semantic Segmentation with Multiview Cosegmentation and Clustering Transformers

Learning a Probabilistic Topology Discovering Model for Scene Categorization.