Abstract:Scene graph generation aims to detect visual entities and relationships between them from an image. The object-level visual information is of vital importance for predicting accurate relationships. However, most existing methods essentially encode visual information with coarse supervised information, since they regard different relationships as mutually exclusive semantics with equal-distance labels by taking cross-entropy function as the main training loss. Intuitively, different relationship semantics naturally have their own similarity and dissimilarity with different level distances, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> , the topological information of relationship semantics. It can serve as an inspiring hint to aid learning to grasp the key related visual information. Accordingly, we propose a Semantically Similarity-wise Dual-branch Network (SSDN) which introduces topological information of relationship semantics as extra supervision to aid learning extracting and encoding relationship-related visual information. To avoid possible chaotic feature learning and enable the introduced knowledge to be better absorbed during inference, we design a dual-branch framework consisting of an auxiliary branch and an inference branch. The topological information extracted from the groundtruth is introduced at the front end of the auxiliary branch which then generates a soft embedding to be propagated to the inference branch in a knowledge distillation manner. Extensive experiments show that our model averagely outperforms state-of-the-art approaches on benchmark Visual Genome and VRD significantly, which demonstrates its effectiveness and superiority.

On support relations and semantic scene graphs

Towards Robotic Semantic Segmentation of Supporting Surfaces

On Support Relations Inference and Scene Hierarchy Graph Construction from Point Cloud in Clustered Environments

Attention Redirection Transformer with Semantic Oriented Learning for Unbiased Scene Graph Generation

Learning 3D Semantic Scene Graphs From 3D Indoor Reconstructions

Indoor Scene Understanding with Geometric and Semantic Contexts

Learning to Generate Language-Supervised and Open-Vocabulary Scene Graph Using Pre-Trained Visual-Semantic Space

Semantic Modeling of Indoor Scenes with Support Inference from a Single Photograph

3D Scene Graph Generation from Point Clouds

Automatic Semantic Modeling of Indoor Scenes from Low-Quality RGB-D Data Using Contextual Information

3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera

Bridging Visual and Textual Semantics: Towards Consistency for Unbiased Scene Graph Generation

Online Scene Semantic Understanding Based on Sparsely Correlated Network for AR

Enhancing Scene Graph Generation with Hierarchical Relationships and Commonsense Knowledge

Attentive Relational Networks for Mapping Images to Scene Graphs

Semantically Similarity-Wise Dual-Branch Network for Scene Graph Generation

Learning Canonical Representations for Scene Graph to Image Generation

Semantic Relation Model and Dataset for Remote Sensing Scene Understanding

Semantic-guided modeling of spatial relation and object co-occurrence for indoor scene recognition

Expanding Scene Graph Boundaries: Fully Open-vocabulary Scene Graph Generation via Visual-Concept Alignment and Retention

Recognition of Indoor Scenes Using 3-D Scene Graphs