Abstract:Scene graph generation (SGGen) is a challenging task due to a complex visual context of an image. Intuitively, the human visual system can volitionally focus on attended regions by salient stimuli associated with visual cues. For example, to infer the relationship between man and horse, the interaction between human leg and horseback can provide strong visual evidence to predict the predicate ride. Besides, the attended region face can also help to determine the object man. Till now, most of the existing works studied the SGGen by extracting coarse-grained bounding box features while understanding fine-grained visual regions received limited attention. To mitigate the drawback, this article proposes a region-aware attention learning method. The key idea is to explicitly construct the attention space to explore salient regions with the object and predicate inferences. First, we extract a set of regions in an image with the standard detection pipeline. Each region regresses to an object. Second, we propose the object-wise attention graph neural network (GNN), which incorporates attention modules into the graph structure to discover attended regions for object inference. Third, we build the predicate-wise co-attention GNN to jointly highlight subject’s and object’s attended regions for predicate inference. Particularly, each subject-object pair is connected with one of the latent predicates to construct one triplet. The proposed intra-triplet and inter-triplet learning mechanism can help discover the pair-wise attended regions to infer predicates. Extensive experiments on two popular benchmarks demonstrate the superiority of the proposed method. Additional ablation studies and visualization further validate its effectiveness.

Multi-Scale Graph Attention Network for Scene Graph Generation

Scene Graph Generation using Depth-based Multimodal Network.

Attention Redirection Transformer with Semantic Oriented Learning for Unbiased Scene Graph Generation

Scene Graph Generation Via Multi-Relation Classification and Cross-Modal Attention Coordinator.

Attentive Gated Graph Neural Network for Image Scene Graph Generation

Toward Region-Aware Attention Learning for Scene Graph Generation

A Unified Deep Sparse Graph Attention Network for Scene Graph Generation

Counterfactual Critic Multi-Agent Training for Scene Graph Generation

Attention LSTM for Scene Graph Generation

Scene Graph Generation by Multi-Level Semantic Tasks.

Deep relational self-Attention networks for scene graph generation

MLMG-SGG: Multilabel Scene Graph Generation with Multigrained Features.

Relationship-Aware Primal-Dual Graph Attention Network For Scene Graph Generation.

Multiview Scene Graph

Fast Contextual Scene Graph Generation with Unbiased Context Augmentation.

Scene Dynamics: Counterfactual Critic Multi-Agent Training for Scene Graph Generation.

Attentive Relational Networks for Mapping Images to Scene Graphs

A Multi-scale Semantic Attention Representation for Multi-Label Image Recognition with Graph Networks

Rethinking Image Generation from Scene Graphs with Attention Mechanism

Scene Graph Captioner: Image Captioning Based on Structural Visual Representation

Scene Graph Inference Via Multi-Scale Context Modeling