Abstract:Humans are able to recognize structured relations in observation, allowing us to decompose complex scenes into simpler parts and abstract the visual world at multiple levels. However, such hierarchical reasoning ability of human perception remains largely unexplored in current literature of semantic segmentation. Existing works are often aware of flatten labels and distinguish all the semantic categories exclusively for each pixel. In this work, we instead address hierarchical semantic segmentation (HSS), with the aim of providing a structured, pixel-wise description of visual observation in terms of a class hierarchy. We devise Hssn , a general HSS framework that tackles two critical issues in this task: i) how to efficiently adapt existing hierarchy-agnostic segmentation networks to the HSS setting, and ii) how to leverage the class hierarchy to regularize HSS network learning. To address i) , Hssn directly casts HSS as a pixel-wise multi-label classification task, only bringing minimal architecture change to current segmentation models. To solve ii) , Hssn first explores inherent properties of the hierarchy as a training objective, which enforces segmentation predictions to obey the hierarchy structure. Furthermore, with a set of hierarchy-induced margin constraints, Hssn efficiently reshapes the learned pixel embedding space, so as to generate hierarchy-aware pixel representations and facilitate structured segmentation eventually. Building upon Hssn , we further exploit the mutual exclusion relation between semantic labels and strengthen the margin based regularization strategy with more meaningful constrains, leading to Hssn +, a more effective framework for HSS. We conduct extensive experiments on six semantic segmentation datasets (i.e., Mapillary Vistas 2.0, Cityscapes, LIP, PASCAL-Person-Part, PASCAL-Part-58, and PASCAL-Part-108), with different class hierarchies, network architectures, and backbones, and the results confirm the generalization and superiority of our algorithms.

Shape Segmentation by Hierarchical Splat Clustering.

Shape Representation and Recognition in High Dimensional Feature Space

Adaptive Decomposition and Shared Weight Volumetric Transformer Blocks for Efficient Patch-Free 3D Medical Image Segmentation.

Part-to-Surface Mesh Segmentation for Mechanical Models Based on Multi-Stage Clustering

Volumetric Shape Contexts for Mesh Co-Segmentation.

Scribble-Based 3D Shape Segmentation via Weakly-Supervised Learning

User-controllable Mesh Segmentation Using Shape Harmonic Signature

Hierarchical Aggregation for 3D Instance Segmentation

TetSphere Splatting: Representing High-Quality Geometry with Lagrangian Volumetric Meshes

Unsupervised Co-Segmentation of 3D Shapes Via Affinity Aggregation Spectral Clustering.

Associate Semantic-Instance Segmentation of 3D Point Clouds Based on Local Feature Extraction

CAD Mesh Model Segmentation by Clustering

Deep Hierarchical Semantic Segmentation

PartNet: A Recursive Part Decomposition Network for Fine-grained and Hierarchical Shape Segmentation

Automatic Mesh Segmentation Based on Hierarchical Clustering

Hierarchical Insights: Exploiting Structural Similarities for Reliable 3D Semantic Segmentation

Semantic Hierarchy-Aware Segmentation

3D Shape Segmentation Using Soft Density Peak Clustering and Semi-Supervised Learning

Hierarchical 3d shape model for segmentation of 4d MR cardiac images

Spectral 3D Mesh Segmentation with a Novel Single Segmentation Field.

HiSplat: Hierarchical 3D Gaussian Splatting for Generalizable Sparse-View Reconstruction