Abstract:As a structured representation of the image content, the visual scene graph (visual relationship) acts as a bridge between computer vision and natural language processing. Existing models on the scene graph generation task notoriously require tens or hundreds of labeled samples. By contrast, human beings can learn visual relationships from a few or even one example. Inspired by this, we design a task named One-Shot Scene Graph Generation, where each relationship triplet (e.g., "dog-has-head") comes from only one labeled example. The key insight is that rather than learning from scratch, one can utilize rich prior knowledge. In this paper, we propose Multiple Structured Knowledge (Relational Knowledge and Commonsense Knowledge) for the one-shot scene graph generation task. Specifically, the Relational Knowledge represents the prior knowledge of relationships between entities extracted from the visual content, e.g., the visual relationships "standing in", "sitting in", and "lying in" may exist between "dog" and "yard", while the Commonsense Knowledge encodes "sense-making" knowledge like "dog can guard yard". By organizing these two kinds of knowledge in a graph structure, Graph Convolution Networks (GCNs) are used to extract knowledge-embedded semantic features of the entities. Besides, instead of extracting isolated visual features from each entity generated by Faster R-CNN, we utilize an Instance Relation Transformer encoder to fully explore their context information. Based on a constructed one-shot dataset, the experimental results show that our method significantly outperforms existing state-of-the-art methods by a large margin. Ablation studies also verify the effectiveness of the Instance Relation Transformer encoder and the Multiple Structured Knowledge.

A Novel End-to-End Transformer for Scene Graph Generation

Iterative Scene Graph Generation with Generative Transformers

SGTR+: End-to-end Scene Graph Generation with Transformer

Transformer-based Image Generation from Scene Graphs

DSGG: Dense Relation Transformer for an End-to-end Scene Graph Generation

One-shot Scene Graph Generation

SGEITL: Scene Graph Enhanced Image-Text Learning for Visual Commonsense Reasoning

Counterfactual Critic Multi-Agent Training for Scene Graph Generation

Reasoning in Different Directions: Triplet Learning for Scene Graph Generation

SPAN: Learning Similarity between Scene Graphs and Images with Transformers

Learning to Generate Scene Graph from Natural Language Supervision

Learning Canonical Representations for Scene Graph to Image Generation

Scene Dynamics: Counterfactual Critic Multi-Agent Training for Scene Graph Generation.

BGT-Net: Bidirectional GRU Transformer Network for Scene Graph Generation

Learning Visual Commonsense for Robust Scene Graph Generation

Towards a Unified Transformer-based Framework for Scene Graph Generation and Human-object Interaction Detection

Toward a Unified Transformer-Based Framework for Scene Graph Generation and Human-Object Interaction Detection

Spatial-Temporal Knowledge-Embedded Transformer for Video Scene Graph Generation

RelTR: Relation Transformer for Scene Graph Generation

From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models

Unconditional Scene Graph Generation