Abstract:Scene graph generation (SGG) task is plagued by insufficient and long-tailed training samples. Thus external text knowledge has been introduced into SGG task, which augments the training dataset. However, rare relations or low-frequency events, such as , are still hard to be discovered since they also rarely appear in external texts. Addressed these issues, this paper proposes a model-agnostic Bipartite Graph Network with Dual-Group Message Passing (DG-BGN). It extends the relation space of each object pair in training dataset based on the similarity of different object pairs, which cooperates external text knowledge and internal visual information. Specifically, the main framework of DG-BGN consists of three parts, that is, bipartite graph construction (BGC), dual-group message propagation (DG-MP), and pseudo label generation (PLG). For BGC, alignment between external texts and visual dataset is performed to mine information from multimodal resources. Taking object pair as the index of the grouping, dual bipartite graph containing intra/inter-group graph is established to represent predicate label probability distribution of object pair instances. Among them, inter-group bipartite graph is built based on the similarity between object pair instance. For DG-MP, efficient message passing is running on both kinds of bipartite graph through Graph Convolution Neural Networks to refine the graphs, in order to let each object pair instance learn from its similar instances. Finally in PLG, pseudo labels, which are used to train SGG models in fully supervised way, are obtained by aggregating these refined graphs. Systematic experiments on Visual Genome dataset and Conceptual Captions show that our method performs better on discovering rare relations.

Relationship-Aware Primal-Dual Graph Attention Network For Scene Graph Generation.

Visual relationship detection with a deep convolutional relationship network

Attentive Relational Networks for Mapping Images to Scene Graphs

Hierarchical Graph Attention Network for Visual Relationship Detection

A Hierarchical Recurrent Approach To Predict Scene Graphs From A Visual-Attention-Oriented Perspective

Deep relational self-Attention networks for scene graph generation

Improving Rare Relation Inferring for Scene Graph Generation Using Bipartite Graph Network

Semantically Similarity-Wise Dual-Branch Network for Scene Graph Generation

Multi-Scale Graph Attention Network for Scene Graph Generation

Toward Region-Aware Attention Learning for Scene Graph Generation

Scene Graph Generation Via Multi-Relation Classification and Cross-Modal Attention Coordinator.

Reasoning in Different Directions: Triplet Learning for Scene Graph Generation

Leveraging Predicate and Triplet Learning for Scene Graph Generation

Hyper-relationship Learning Network for Scene Graph Generation

Relation Regularized Scene Graph Generation

Multi-Granularity Sparse Relationship Matrix Prediction Network for End-to-End Scene Graph Generation

Knowledge-Embedded Routing Network for Scene Graph Generation

Boosting Scene Graph Generation with Visual Relation Saliency

Scene Graph Generation With Hierarchical Context

DSGG: Dense Relation Transformer for an End-to-end Scene Graph Generation

Joint Modeling of Visual Objects and Relations for Scene Graph Generation.