Abstract:Humans are able to recognize structured relations in observation, allowing us to decompose complex scenes into simpler parts and abstract the visual world at multiple levels. However, such hierarchical reasoning ability of human perception remains largely unexplored in current literature of semantic segmentation. Existing works are often aware of flatten labels and distinguish all the semantic categories exclusively for each pixel. In this work, we instead address hierarchical semantic segmentation (HSS), with the aim of providing a structured, pixel-wise description of visual observation in terms of a class hierarchy. We devise Hssn , a general HSS framework that tackles two critical issues in this task: i) how to efficiently adapt existing hierarchy-agnostic segmentation networks to the HSS setting, and ii) how to leverage the class hierarchy to regularize HSS network learning. To address i) , Hssn directly casts HSS as a pixel-wise multi-label classification task, only bringing minimal architecture change to current segmentation models. To solve ii) , Hssn first explores inherent properties of the hierarchy as a training objective, which enforces segmentation predictions to obey the hierarchy structure. Furthermore, with a set of hierarchy-induced margin constraints, Hssn efficiently reshapes the learned pixel embedding space, so as to generate hierarchy-aware pixel representations and facilitate structured segmentation eventually. Building upon Hssn , we further exploit the mutual exclusion relation between semantic labels and strengthen the margin based regularization strategy with more meaningful constrains, leading to Hssn +, a more effective framework for HSS. We conduct extensive experiments on six semantic segmentation datasets (i.e., Mapillary Vistas 2.0, Cityscapes, LIP, PASCAL-Person-Part, PASCAL-Part-58, and PASCAL-Part-108), with different class hierarchies, network architectures, and backbones, and the results confirm the generalization and superiority of our algorithms.

Hierarchy-associated Semantic-Rule Inference Framework for Classifying Indoor Scenes

Automatic Semantic Modeling of Indoor Scenes from Low-Quality RGB-D Data Using Contextual Information

Indoor Scene Recognition in 3D

Scene Parsing by Integrating Function, Geometry and Appearance Models

Recognition of Indoor Scenes Using 3-D Scene Graphs

Exploiting Object-based and Segmentation-based Semantic Features for Deep Learning-based Indoor Scene Classification

Scene Classification in Indoor Environments for Robots using Context Based Word Embeddings

Semantic-aware Room-Level Indoor Modeling from Point Clouds

Semantic-guided modeling of spatial relation and object co-occurrence for indoor scene recognition

An Interactive Approach to Semantic Modeling of Indoor Scenes with an RGBD Camera

On Support Relations Inference and Scene Hierarchy Graph Construction from Point Cloud in Clustered Environments

Indoor scene recognition through object detection

Indoor Scene Classification by Incorporating Predicted Depth Descriptor.

Indoor Scene Recognition via Object Detection and TF-IDF

SRRM: Semantic Region Relation Model for Indoor Scene Recognition

CONSTRUCTION OF A DUAL-TASK MODEL FOR INDOOR SCENE RECOGNITION AND SEMANTIC SEGMENTATION BASED ON POINT CLOUDS

Semantic Hierarchy-Aware Segmentation

3D Semantic Segmentation Algorithm for Indoor Scenes based on Long-term Memory

An Approach for Construct Semantic Map with Scene Classification and Object Semantic Segmentation

Semantic 3D Indoor Scene Enhancement Using Guide Words

3D Indoor Environment Modeling and Detection of Moving Object and Scene Understanding