Abstract:As a structured representation of the image content, the visual scene graph (visual relationship) acts as a bridge between computer vision and natural language processing. Existing models on the scene graph generation task notoriously require tens or hundreds of labeled samples. By contrast, human beings can learn visual relationships from a few or even one example. Inspired by this, we design a task named One-Shot Scene Graph Generation, where each relationship triplet (e.g., "dog-has-head") comes from only one labeled example. The key insight is that rather than learning from scratch, one can utilize rich prior knowledge. In this paper, we propose Multiple Structured Knowledge (Relational Knowledge and Commonsense Knowledge) for the one-shot scene graph generation task. Specifically, the Relational Knowledge represents the prior knowledge of relationships between entities extracted from the visual content, e.g., the visual relationships "standing in", "sitting in", and "lying in" may exist between "dog" and "yard", while the Commonsense Knowledge encodes "sense-making" knowledge like "dog can guard yard". By organizing these two kinds of knowledge in a graph structure, Graph Convolution Networks (GCNs) are used to extract knowledge-embedded semantic features of the entities. Besides, instead of extracting isolated visual features from each entity generated by Faster R-CNN, we utilize an Instance Relation Transformer encoder to fully explore their context information. Based on a constructed one-shot dataset, the experimental results show that our method significantly outperforms existing state-of-the-art methods by a large margin. Ablation studies also verify the effectiveness of the Instance Relation Transformer encoder and the Multiple Structured Knowledge.

Graph R-CNN for Scene Graph Generation

SceneSketcher-v2: Fine-Grained Scene-Level Sketch-Based Image Retrieval Using Adaptive GCNs

Structured Sparse R-CNN for Direct Scene Graph Generation

One-shot Scene Graph Generation

Using Scene Graph Context to Improve Image Generation

Relation Regularized Scene Graph Generation

Counterfactual Critic Multi-Agent Training for Scene Graph Generation

Scene Dynamics: Counterfactual Critic Multi-Agent Training for Scene Graph Generation.

3D Scene Graph Generation from Point Clouds

Factorizable Net: An Efficient Subgraph-based Framework for Scene Graph Generation

Scene Graph Generation with Geometric Context

Reasoning in Different Directions: Triplet Learning for Scene Graph Generation

Learning Canonical Representations for Scene Graph to Image Generation

Deep Generative Probabilistic Graph Neural Networks for Scene Graph Generation

Attentive Relational Networks for Mapping Images to Scene Graphs

Deep relational self-Attention networks for scene graph generation

R3CD: Scene Graph to Image Generation with Relation-Aware Compositional Contrastive Control Diffusion

Scene Graph Generation With Hierarchical Context

Unconditional Scene Graph Generation

Pixels to Graphs by Associative Embedding

Image Generation from Scene Graph with Object Edges