Abstract:Social interaction is a common phenomenon in human societies. Different from discovering groups based on the similarity of individuals’ actions, social interaction focuses more on the mutual influence between people. Although people can easily judge whether or not there are social interactions in a real-world scene, it is difficult for an intelligent system to discover social interactions. Initiating and concluding social interactions are greatly influenced by an individual’s social cognition and the surrounding environment, which are closely related to psychology. Thus, converting the psychological factors that impact social interactions into quantifiable visual representations and creating a model for interaction relationships poses a significant challenge. To this end, we propose a Psychology-Guided Environment Aware Network (PEAN) that models social interaction among people in videos using supervised learning. Specifically, we divide the surrounding environment into scene-aware visual-based and human-aware visual-based descriptions. For the scene-aware visual clue, we utilize 3D features as global visual representations. For the human-aware visual clue, we consider instance-based location and behaviour-related visual representations to map human-centered interaction elements in social psychology: distance, openness and orientation. In addition, we design an environment aware mechanism to integrate features from visual clues, with a Transformer to explore the relation between individuals and construct pairwise interaction strength features. The interaction intensity matrix reflecting the mutual nature of the interaction is obtained by processing the interaction strength features with the interaction discovery module. An interaction constrained loss function composed of interaction critical loss function and smooth F β loss function is proposed to optimize the whole framework to improve the distinction of the interaction matrix and alleviate class imbalance caused by pairwise interaction sparsity. Given the diversity of real-world interactions, we collect a new dataset named Social Basketball Activity Dataset (Soical-BAD), covering complex social interactions. Our method achieves the best performance among social-CAD, social-BAD, and their combined dataset named Video Social Interaction Dataset (VSID).

Toward jointly understanding social relationships and characters from videos

Socializing the Videos: A Multimodal Approach for Social Relation Recognition

InteractNet: Social Interaction Recognition for Semantic-rich Videos

Learning Interactions and Relationships Between Movie Characters

CAGNet: a context-aware graph neural network for detecting social relationships in videos

Social Context-aware Person Search in Videos via Multi-modal Cues

Video Captioning Via Relation-Aware Graph Learning

Spatial-Temporal Correlation and Topology Learning for Person Re-Identification in Videos

Multi-Granularity Reasoning for Social Relation Recognition From Images

Character Matters: Video Story Understanding with Character-Aware Relations

Improved Actor Relation Graph based Group Activity Recognition

Graph-Based Social Relation Reasoning

Inferring Social Roles In Long Timespan Video Sequence

Joint Learning for Relationship and Interaction Analysis in Video with Multimodal Feature Fusion

Human Interaction Understanding With Joint Graph Decomposition and Node Labeling

Psychology-Guided Environment Aware Network for Discovering Social Interaction Groups from Videos

Joint Learning of Social Groups, Individuals Action and Sub-group Activities in Videos

Video Relationship Reasoning using Gated Spatio-Temporal Energy Graph

When I Fall in Love: Capturing Video-oriented Social Relationship Evolution Via Attentive GNN

Video Relation Detection with Spatio-Temporal Graph

Non-parametric Contextual Relationship Learning for Semantic Video Object Segmentation