Abstract:Humans are able to recognize structured relations in observation, allowing us to decompose complex scenes into simpler parts and abstract the visual world at multiple levels. However, such hierarchical reasoning ability of human perception remains largely unexplored in current literature of semantic segmentation. Existing works are often aware of flatten labels and distinguish all the semantic categories exclusively for each pixel. In this work, we instead address hierarchical semantic segmentation (HSS), with the aim of providing a structured, pixel-wise description of visual observation in terms of a class hierarchy. We devise Hssn , a general HSS framework that tackles two critical issues in this task: i) how to efficiently adapt existing hierarchy-agnostic segmentation networks to the HSS setting, and ii) how to leverage the class hierarchy to regularize HSS network learning. To address i) , Hssn directly casts HSS as a pixel-wise multi-label classification task, only bringing minimal architecture change to current segmentation models. To solve ii) , Hssn first explores inherent properties of the hierarchy as a training objective, which enforces segmentation predictions to obey the hierarchy structure. Furthermore, with a set of hierarchy-induced margin constraints, Hssn efficiently reshapes the learned pixel embedding space, so as to generate hierarchy-aware pixel representations and facilitate structured segmentation eventually. Building upon Hssn , we further exploit the mutual exclusion relation between semantic labels and strengthen the margin based regularization strategy with more meaningful constrains, leading to Hssn +, a more effective framework for HSS. We conduct extensive experiments on six semantic segmentation datasets (i.e., Mapillary Vistas 2.0, Cityscapes, LIP, PASCAL-Person-Part, PASCAL-Part-58, and PASCAL-Part-108), with different class hierarchies, network architectures, and backbones, and the results confirm the generalization and superiority of our algorithms.

Constructing Semantic Hierarchies Via Fusion Learning Architecture

Learning Semantic Hierarchies Via Word Embeddings.

Learning Semantic Hierarchies: a Continuous Vector Space Approach

Lexical semantics enhanced neural word embeddings

Research of improving semantic image segmentation based on a feature fusion model

ExFuse: Enhancing Feature Fusion for Semantic Segmentation

Aspect term extraction via adaptive fusion of sequential and hierarchical representation

Enhancing Feature Fusion with Spatial Aggregation and Channel Fusion for Semantic Segmentation

Learning Visual Hierarchies with Hyperbolic Embeddings

Semantic-Aware Fusion Network Based on Super-Resolution

GFF: Gated Fully Fusion for Semantic Segmentation

Hierarchical Label with Imbalance and Attributed Network Structure Fusion for Network Embedding

Multi-level Fusion of Multi-modal Semantic Embeddings for Zero Shot Learning

Efficient Heuristic Methods for Multimodal Fusion and Concept Fusion in Video Concept Detection

Syntactic Fusion: Enhancing Aspect-Level Sentiment Analysis Through Multi-Tree Graph Integration

Semantic Hierarchy-Aware Segmentation

BuildingFusion: Semantic-Aware Structural Building-Scale 3D Reconstruction.

Multi-task Hierarchical Heterogeneous Fusion Framework for multimodal summarization

Unified Lattice Graph Fusion for Chinese Named Entity Recognition

Infrared and Visible Image Fusion with Hierarchical Human Perception

Heterogeneous Hierarchical Fusion Network for Multimodal Sentiment Analysis in Real-World Environments