Abstract:In the fields of neuroscience and pharmacology, understanding rodent behavior is of vital importance for studying the effects of genetic operations and pharmacological therapies. Conventional behavior recognition methods based on raw images often struggle with noise, such as changes in the lighting conditions and the image backgrounds. On the other hand, pose-based approaches have demonstrated robustness against these challenges. However, existing methods rely on manually constructed features, which are time-consuming and may not fully exploit the potential of the pose data. In this work, we propose the hierarchical spatial–temporal window transformer network (HSTWFormer), a novel approach that efficiently extracts multiscale and cross-spacetime features from rodent pose data. By adopting a pure Transformer structure, HSTWFormer not only avoids the need for a predefined skeletal topology, but also enables adaptive recognition of interactive behaviors between multiple rodents. By merging the features of temporal neighbors, we construct a hierarchical structure with different receptive fields that retain essential information of all scales, enabling the extraction of semantic features from low to high level. Furthermore, a spatial–temporal window attention (STWA) block is introduced to capture correlations between different key points across frames. The STWA blocks facilitate the extraction of both short-term and long-term cross-spacetime features by enabling interactions between window information through window shifting, enhancing the network's modeling performance. The effectiveness of the proposed HSTWFormer is demonstrated on two datasets, CRIM13 and CalMS21. We achieved accuracies of 79.3% and 69.8% for interactive and overall behaviors in the CRIM13 dataset, and 76.4% accuracy in the CalMS21 dataset. Our method harnesses the wealth of information embedded in key points, showcasing robust modeling capabilities for accurate rodent behavior recognition, and provides a novel and effective approach to assist researchers in neuroscience and pharmacology in better quantifying rodent behavior.

Hierarchical Spatial–Temporal Window Transformer for Pose-Based Rodent Behavior Recognition

A Spatio-Temporal Transformer Network for Human Motion Prediction in Human-Robot Collaboration

Automatic Recognition of Rat's Postures Based on Contour Curvature and Hierarchical Clustering Analysis

Markerless Rat Behavior Quantification With Cascade Neural Network

Automatic Recognition of Rat's Postures Based on Naive Bayes Classifier

STPoseNet: A real-time spatiotemporal network model for robust mouse pose estimation

Context-Aware Mouse Behavior Recognition Using Hidden Markov Models

An Event-Driven Object Recognition Model Using Activated Connected Domain Detection

Murine Motion Behavior Recognition Based on DeepLabCut and Convolutional Long Short-Term Memory Network

Human Behavior Recognition Based on Multiscale Convolutional Neural Network.

Wavelet Tree Transformer: Multihead Attention With Frequency-Selective Representation and Interaction for Remote Sensing Object Detection

STMT: A Spatial-Temporal Mesh Transformer for MoCap-Based Action Recognition

Emotion recognition using hierarchical spatial-temporal learning transformer from regional to global brain

Unsupervised identification of rat behavioral motifs across timescales

HRPoseFormer: High-Resolution Transformer for Human Pose Estimation Via Multi-Scale Token Aggregation

Social Behavior Recognition in Mouse Video Using Agent Embedding and LSTM Modelling.

Efficient Aggressive Behavior Recognition of Pigs Based on Temporal Shift Module

Learning Adaptive Spatio-Temporal Inference Transformer for Coarse-to-Fine Animal Visual Tracking: Algorithm and Benchmark

Anti-drift pose tracker (ADPT): A transformer-based network for robust animal pose estimation cross-species

An attention-based temporal convolutional network for rodent sleep stage classification across species, mutants and experimental environments with single-channel electroencephalogram

A Spatiotemporal Convolutional Network for Multi-Behavior Recognition of Pigs