Abstract:Despite progress in visual perception tasks such as image classification and detection, computers still struggle to understand the interdependency of objects in the scene as a whole, e.g., relations between objects or their attributes. Existing methods often ignore global context cues capturing the interactions among different object instances, and can only recognize a handful of types by exhaustively training individual detectors for all possible relationships. To capture such global interdependency, we propose a deep Variation-structured Reinforcement Learning (VRL) framework to sequentially discover object relationships and attributes in the whole image. First, a directed semantic action graph is built using language priors to provide a rich and compact representation of semantic correlations between object categories, predicates, and attributes. Next, we use a variation-structured traversal over the action graph to construct a small, adaptive action set for each step based on the current state and historical actions. In particular, an ambiguity-aware object mining scheme is used to resolve semantic ambiguity among object categories that the object detector fails to distinguish. We then make sequential predictions using a deep RL framework, incorporating global context cues and semantic embeddings of previously extracted phrases in the state vector. Our experiments on the Visual Relationship Detection (VRD) dataset and the large-scale Visual Genome dataset validate the superiority of VRL, which can achieve significantly better detection results on datasets involving thousands of relationship and attribute types. We also demonstrate that VRL is able to predict unseen types embedded in our action graph by learning correlations on shared graph nodes.

AVR: Attention based Salient Visual Relationship Detection

Visual relationship detection with a deep convolutional relationship network

Visual Relationship Recognition Via Language And Position Guided Attention

Visual Relation Detection with Multi-Level Attention

Attention Guided Relation Detection Approach for Video Visual Relation Detection

Visual Relationship Detection: A Survey

Visual relationship detection with recurrent attention and negative sampling

Relationship Detection Based on Object Semantic Inference and Attention Mechanisms

Visual Spatial Attention Network for Relationship Detection.

Rethinking Visual Relationships for High-level Image Understanding.

Vrr-Vg: Refocusing Visually-Relevant Relationships

Hierarchical Graph Attention Network for Visual Relationship Detection

Towards Overcoming False Positives in Visual Relationship Detection.

Dual Attentional Transformer for Video Visual Relation Prediction

Visual relationship detection with region topology structure

Hierarchical Visual Relationship Detection

Visual Relationship Detection With Visual-Linguistic Knowledge From Multimodal Representations

Deep Variation-structured Reinforcement Learning for Visual Relationship and Attribute Detection

Visual Relationship Detection with Relative Location Mining

AVR (advancing video retrieval): A new framework guided by multi-level fusion of visual and semantic Features for deep learning-based concept detection

Detecting Visual Relationships Using Box Attention