Abstract:Automatically interpreting social relations, e.g., friendship, kinship, etc., from visual scenes has huge potential application value in areas such as knowledge graphs construction, person behavior and emotion analysis, entertainment ecology, etc. Great progress has been made in social analysis based on structured data. However, existing video-based methods consider social relationship extraction as a general classification task and categorize videos into only predefined types. Such methods are unable to recognize multiple relations in multi-person videos, which is obviously not consistent with the actual application scenarios. At the same time, videos are inherently multimodal. Subtitles in the video also provide abundant cues for relationship recognition that is often ignored by researchers. In this paper, we introduce and define a new task named "Multiple-Relation Extraction in Videos (MREV)". To solve the MREV task, we propose the Visual-Textual Fusion (VTF) framework for jointly modeling visual and textual information. For the spatial representation, we not only adopt a SlowFast network to learn global action and scene information, but also exploit the unique cues of face, body and dialogue between characters. For the temporal domain, we propose a Temporal Feature Aggregation module to perform temporal reasoning, which assesses the quality of different frames adaptively. After that, we use a Multi-Conv Attention module to capture the inter-modal correlation and map the features of different modes to a coordinated feature space. By this means, our VTF framework comprehensively exploits abundant multimodal cues for the MREV task and achieves 49.2% and 50.4% average accuracy on a self-constructed Video Multiple-Relation(VMR) dataset and ViSR dataset, respectively. Extensive experiments on VMR dataset and ViSR dataset demonstrate the effectiveness of the proposed framework.

Social Relation Graph Generation on Untrimmed Video.

Video Relation Detection with Spatio-Temporal Graph

CAGNet: a context-aware graph neural network for detecting social relationships in videos

Relation Extraction from Videos Based on IoT Intelligent Collaboration Framework

Video Captioning Via Relation-Aware Graph Learning

Relational Graph Learning for Grounded Video Description Generation

InteractNet: Social Interaction Recognition for Semantic-rich Videos

Classification-Then-Grounding: Reformulating Video Scene Graphs As Temporal Bipartite Graphs

Video Scene Graph Generation from Single-Frame Weak Supervision.

Toward jointly understanding social relationships and characters from videos

Socializing the Videos: A Multimodal Approach for Social Relation Recognition

A Multimodal Approach for Multiple-Relation Extraction in Videos

Triple Correlations-Guided Label Supplementation for Unbiased Video Scene Graph Generation

Visual Relation Grounding in Videos

Graph-Based Social Relation Reasoning

Target Adaptive Context Aggregation for Video Scene Graph Generation

Dual Variational Graph Reconstruction Learning for Social Recommendation

Unified Static and Dynamic Network: Efficient Temporal Filtering for Video Grounding

Beyond Short-Term Snippet: Video Relation Detection With Spatio-Temporal Global Context

When I Fall in Love: Capturing Video-oriented Social Relationship Evolution Via Attentive GNN

Multi-Modal Relational Graph for Cross-Modal Video Moment Retrieval