Abstract:Automated social behaviour analysis of mice has become an increasingly popular research area in behavioural neuroscience. Recently, pose information (i.e., locations of keypoints or skeleton) has been used to interpret social behaviours of mice. Nevertheless, effective encoding and decoding of social interaction information underlying the keypoints of mice has been rarely investigated in the existing methods. In particular, it is challenging to model complex social interactions between mice due to highly deformable body shapes and ambiguous movement patterns. To deal with the interaction modelling problem, we here propose a Cross-Skeleton Interaction Graph Aggregation Network (CS-IGANet) to learn abundant dynamics of freely interacting mice, where a Cross-Skeleton Node-level Interaction module (CS-NLI) is used to model multi-level interactions (i.e., intra-, inter- and cross-skeleton interactions). Furthermore, we design a novel Interaction-Aware Transformer (IAT) to dynamically learn the graph-level representation of social behaviours and update the node-level representation, guided by our proposed interaction-aware self-attention mechanism. Finally, to enhance the representation ability of our model, an auxiliary self-supervised learning task is proposed for measuring the similarity between cross-skeleton nodes. Experimental results on the standard CRMI13-Skeleton and our PDMB-Skeleton datasets show that our proposed model outperforms several other state-of-the-art approaches.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the problem of automatically analyzing complex interactions in mouse social behaviors. Specifically, the author points out that current methods do not go deep enough in encoding and decoding the social interaction information contained in mouse key points (such as skeletal joint positions), especially in modeling complex social interactions due to highly deformable body shapes and ambiguous movement patterns. ### Core of the Problem 1. **Limitations of Existing Methods**: - Most of the existing methods rely on shallow features designed manually (such as the distance between two noses), and these features are not sufficient to describe the dependency relationships between key points. - Although Graph Convolutional Networks (GCNs) perform well in action recognition of a single object, they are not effective in dealing with multiple interacting subjects, especially in capturing complex social interactions. 2. **Challenges**: - Modeling complex social interactions between mice, especially due to the high variability of mouse bodies and the ambiguity of movement patterns. - Automatically learning rich spatio - temporal dynamic relationships from key - point information. ### Proposed Solution To solve the above problems, the author proposes a new model - **Cross - Skeleton Interaction Graph Aggregation Network (CS - IGANet)**. The main innovations of this model include: 1. **Cross - Skeleton Node - Level Interaction (CS - NLI) Module**: - It is used to model multi - level interactions, including intra - skeleton, inter - skeleton, and cross - skeleton interactions. - Infer the corresponding interaction patterns by fusing multi - order dense information. 2. **Interaction - Aware Transformer (IAT)**: - Dynamically learn the graph - level representation of social behaviors and update the node - level representation to extract higher - level features. - Use the interaction - aware self - attention mechanism to enhance the model's focus on important nodes. 3. **Auxiliary Self - Supervised Learning Strategy**: - By optimizing the self - supervised objective function and the traditional classification loss function (such as cross - entropy loss), the model can focus more on the similarities between node pairs between different skeletons, thereby enhancing the model's representation ability. ### Summary This paper aims to solve the deficiencies of existing methods in automatically analyzing mouse social behaviors, especially the challenges in modeling complex social interactions, by introducing the CS - IGANet model. By combining graph neural networks, self - attention mechanisms, and self - supervised learning, this model can effectively learn the rich dynamic relationships of mouse social behaviors in long - video.

Cross-Skeleton Interaction Graph Aggregation Network for Representation Learning of Mouse Social Behaviour

Interactive semantics neural networks for skeleton-based human interaction recognition

Learning Multi-View Interactional Skeleton Graph for Action Recognition

Social Behavior Recognition in Mouse Video Using Agent Embedding and LSTM Modelling.

MVHANet: Multi-view Hierarchical Aggregation Network for Skeleton-Based Hand Gesture Recognition

Muti-view Mouse Social Behaviour Recognition with Deep Graphical Model

Multi-View Mouse Social Behaviour Recognition With Deep Graphic Model

Interweaved Graph and Attention Network for 3D Human Pose Estimation

Structured Context Enhancement Network for Mouse Pose Estimation

A Novel Multi-Stream Hand-Object Interaction Network for Assembly Action Recognition

Symbiotic Graph Neural Networks for 3D Skeleton-based Human Action Recognition and Motion Prediction

Multi-Level Graph Encoding with Structural-Collaborative Relation Learning for Skeleton-Based Person Re-Identification

Multi-Modal Enhancement Transformer Network for Skeleton-Based Human Interaction Recognition

Human-Robot Collaboration Through a Multi-Scale Graph Convolution Neural Network With Temporal Attention

MGSAN: multimodal graph self-attention network for skeleton-based action recognition

Dyadic relational graph convolutional networks for skeleton-based human interaction recognition

A Skeleton-aware Graph Convolutional Network for Human-Object Interaction Detection

Two-person Graph Convolutional Network for Skeleton-based Human Interaction Recognition.

Behaviour Modelling of Social Animals via Causal Structure Discovery and Graph Neural Networks

Spatiotemporal Progressive Inward-Outward Aggregation Network for skeleton-based action recognition

STPoseNet: A real-time spatiotemporal network model for robust mouse pose estimation