Abstract:People’s interactions with each other form the social relations in society. Understanding human social relations in the public space is of great importance for supporting the public administrations. Recognizing social relations through visual data captured by remote sensing cameras is one of the most efficient ways to observe human interactions in a public space. Generally speaking, persons in the same scene tend to know each other, and the relations between person pairs are strongly correlated. The scene information in which people interact is also one of the important cues for social relation recognition (SRR). The existing works have not explored the correlations between the scene information and people’s interactions. The scene information has only been extracted on a simple level and high level semantic features to support social relation understanding are lacking. To address this issue, we propose a social relation structure-aware local–global model for SRR to exploit the high-level semantic global information of the scene where the social relation structure is explored. In our proposed model, the graph neural networks (GNNs) are employed to reason through the interactions (local information) between social relations and the global contextual information contained in the constructed scene-relation graph. Experiments demonstrate that our proposed local–global information-reasoned social relation recognition model (SRR-LGR) can reason through the local–global information. Further, the results of the final model show that our method outperforms the state-of-the-art methods. In addition, we have further discussed whether the global information contributes equally to different social relations in the same scene, by exploiting an attention mechanism in our proposed model. Further applications of SRR for human-observation are also exploited.

Progressive Graph Reasoning-Based Social Relation Recognition

Graph-Based Social Relation Reasoning

Multi-Granularity Reasoning for Social Relation Recognition From Images

Principal Relation Component Reasoning-Enhanced Social Relation Recognition

HF-SRGR: a New Hybrid Feature-Driven Social Relation Graph Reasoning Model

Deep Reasoning with Knowledge Graph for Social Relationship Understanding

Multi-Level Transformer-Based Social Relation Recognition

Social Relation Recognition from Videos Via Multi-Scale Spatial-Temporal Reasoning

Multi-scale Features Based Interpersonal Relation Recognition Using Higher-Order Graph Neural Network.

Learning Social Spatio-Temporal Relation Graph in the Wild and a Video Benchmark.

SRR-LGR: Local–Global Information-Reasoned Social Relation Recognition for Human-Oriented Observation

Social Relation Reasoning Based on Triangular Constraints

Graph-based social relation inference with multi-level conditional attention

An End-To-End Network for Generating Social Relationship Graphs

Enhancing Social Relation Inference with Concise Interaction Graph and Discriminative Scene Representation

Relation-Aware Reasoning with Graph Convolutional Network.

Progressive Relation Learning for Group Activity Recognition

A Domain Based Approach to Social Relation Recognition

Configurable Graph Reasoning for Visual Relationship Detection.

Social Relation Analysis from Videos Via Multi-entity Reasoning

Unveiling Social Relations: Leveraging Interpersonal Similarity Learning for Social Relation Recognition.