Abstract:Scene graph aims to faithfully reveal humans' perception of image content. When humans look at a scene, they usually focus on their interested parts in a special priority. This innate habit indicates a hierarchical preference about human perception. Therefore, we argue to generate the Scene Graph of Interest which should be hierarchically constructed, so that the important primary content is firstly presented while the secondary one is presented on demand. To achieve this goal, we propose the Tree–Guided Importance Ranking (TGIR) model. We represent the scene with a hierarchical structure by firstly detecting objects in the scene and organizing them into a Hierarchical Entity Tree (HET) according to their spatial scale, considering that larger objects are more likely to be noticed instantly. After that, the scene graph is generated guided by structural information of HET which is modeled by the elaborately designed Hierarchical Contextual Propagation (HCP) module. To further highlight the key relationship in the scene graph, all relationships are re-ranked through additionally estimating their importance by the Relationship Ranking Module (RRM). To train RRM, the most direct way is to collect the key relationship annotation, which is the so-called Direct Supervision scheme. As collecting annotation may be cumbersome, we further utilize two intuitive and effective cues, visual saliency and spatial scale, and treat them as Approximate Supervision, according to the findings that these cues are positively correlated with relationship importance. With these readily available cues, the RRM is still able to estimate the importance even without key relationship annotation. Experiments indicate that our method not only achieves state-of-the-art performances on scene graph generation, but also is expert in mining image-specific relationships which play a great role in serving subsequent tasks such as image captioning and cross-modal retrieval.

Scene Graph Generation Based On Node-Relation Context Module

Scene Graph Generation With Hierarchical Context

Attention Redirection Transformer with Semantic Oriented Learning for Unbiased Scene Graph Generation

Complex Relation Embedding for Scene Graph Generation

Enhancing Scene Graph Generation with Hierarchical Relationships and Commonsense Knowledge

Attentive Relational Networks for Mapping Images to Scene Graphs

Sketching Image Gist: Human-Mimetic Hierarchical Scene Graph Generation

Exploiting Relationship for Complex-scene Image Generation

Iterative Visual Relationship Detection via Commonsense Knowledge Graph

Relationship-Aware Primal-Dual Graph Attention Network For Scene Graph Generation.

Improving Rare Relation Inferring for Scene Graph Generation Using Bipartite Graph Network

Fine‐Grained Scene Graph Generation with Overlap Region and Geometrical Center

Boosting Scene Graph Generation with Contextual Information

Hypercomplex context guided interaction modeling for scene graph generation

Scene Graph Generation Via Convolutional Message Passing and Class-Aware Memory Embeddings

Knowledge-Based Scene Graph Generation with Visual Contextual Dependency

Scene Graph Generation Via Multi-Relation Classification and Cross-Modal Attention Coordinator.

Exploring Correlation of Relationship Reasoning for Scene Graph Generation.

Scene Graph Generation with Geometric Context

A Novel Framework for Scene Graph Generation Via Prior Knowledge

Importance First: Generating Scene Graph of Human Interest