Abstract:People’s interactions with each other form the social relations in society. Understanding human social relations in the public space is of great importance for supporting the public administrations. Recognizing social relations through visual data captured by remote sensing cameras is one of the most efficient ways to observe human interactions in a public space. Generally speaking, persons in the same scene tend to know each other, and the relations between person pairs are strongly correlated. The scene information in which people interact is also one of the important cues for social relation recognition (SRR). The existing works have not explored the correlations between the scene information and people’s interactions. The scene information has only been extracted on a simple level and high level semantic features to support social relation understanding are lacking. To address this issue, we propose a social relation structure-aware local–global model for SRR to exploit the high-level semantic global information of the scene where the social relation structure is explored. In our proposed model, the graph neural networks (GNNs) are employed to reason through the interactions (local information) between social relations and the global contextual information contained in the constructed scene-relation graph. Experiments demonstrate that our proposed local–global information-reasoned social relation recognition model (SRR-LGR) can reason through the local–global information. Further, the results of the final model show that our method outperforms the state-of-the-art methods. In addition, we have further discussed whether the global information contributes equally to different social relations in the same scene, by exploiting an attention mechanism in our proposed model. Further applications of SRR for human-observation are also exploited.

Linking People Across Text and Images Based on Social Relation Reasoning.

Principal Relation Component Reasoning-Enhanced Social Relation Recognition

Multi-Granularity Reasoning for Social Relation Recognition From Images

SRR-LGR: Local–Global Information-Reasoned Social Relation Recognition for Human-Oriented Observation

Cross-Modal Implicit Relation Reasoning and Aligning for Text-to-Image Person Retrieval

Rethinking referring relationships from a perspective of mask-level relational reasoning

Unveiling Social Relations: Leveraging Interpersonal Similarity Learning for Social Relation Recognition.

Progressive Graph Reasoning-Based Social Relation Recognition

Context‐aware relation enhancement and similarity reasoning for image‐text retrieval

Towards Bridged Vision and Language: Learning Cross-modal Knowledge Representation for Relation Extraction

Graph-Based Social Relation Reasoning

Exploring Uni-Modal Feature Learning on Entities and Relations for Remote Sensing Cross-Modal Text-Image Retrieval

Remote Sensing Image-Text Retrieval With Implicit-Explicit Relation Reasoning

HF-SRGR: a New Hybrid Feature-Driven Social Relation Graph Reasoning Model

Social Relation Recognition from Videos Via Multi-Scale Spatial-Temporal Reasoning

Visual Relationship Detection With Visual-Linguistic Knowledge From Multimodal Representations

Multi-Level Transformer-Based Social Relation Recognition

Person Re-Identification with Visual Semantic Representation Mining and Reasoning

Learning Social Spatio-Temporal Relation Graph in the Wild and a Video Benchmark.

Dual-graph Hierarchical Interaction Network for Referring Image Segmentation

Collaborative Position Reasoning Network for Referring Image Segmentation