Abstract:In the fields of neuroscience and pharmacology, understanding rodent behavior is of vital importance for studying the effects of genetic operations and pharmacological therapies. Conventional behavior recognition methods based on raw images often struggle with noise, such as changes in the lighting conditions and the image backgrounds. On the other hand, pose-based approaches have demonstrated robustness against these challenges. However, existing methods rely on manually constructed features, which are time-consuming and may not fully exploit the potential of the pose data. In this work, we propose the hierarchical spatial–temporal window transformer network (HSTWFormer), a novel approach that efficiently extracts multiscale and cross-spacetime features from rodent pose data. By adopting a pure Transformer structure, HSTWFormer not only avoids the need for a predefined skeletal topology, but also enables adaptive recognition of interactive behaviors between multiple rodents. By merging the features of temporal neighbors, we construct a hierarchical structure with different receptive fields that retain essential information of all scales, enabling the extraction of semantic features from low to high level. Furthermore, a spatial–temporal window attention (STWA) block is introduced to capture correlations between different key points across frames. The STWA blocks facilitate the extraction of both short-term and long-term cross-spacetime features by enabling interactions between window information through window shifting, enhancing the network's modeling performance. The effectiveness of the proposed HSTWFormer is demonstrated on two datasets, CRIM13 and CalMS21. We achieved accuracies of 79.3% and 69.8% for interactive and overall behaviors in the CRIM13 dataset, and 76.4% accuracy in the CalMS21 dataset. Our method harnesses the wealth of information embedded in key points, showcasing robust modeling capabilities for accurate rodent behavior recognition, and provides a novel and effective approach to assist researchers in neuroscience and pharmacology in better quantifying rodent behavior.

Automatic Recognition of Rat's Postures Based on Contour Curvature and Hierarchical Clustering Analysis

Automatic Recognition of Rat's Postures Based on Naive Bayes Classifier

A Posture Recognition System for Rat Cyborg Automated Navigation

Hand Posture Recognition and Tracking Based on Bag-of-Words for Human Robot Interaction

Based on cluster tree human action recognition algorithm for monocular video

Hierarchical Bag-Of-Features For Hand Posture Recognition

A Hierarchical Algorithm for Human Posture Recognition Based on Spatial and Frequency Domain Features

Classroom Student Posture Recognition Based on an Improved High-Resolution Network.

Human Articulated Body Recognition Method in High-Resolution Monitoring Images

Automatic Analysis System of Motion in Rat Based on Vision

Human posture recognition based on multiple features and rule learning

Validation of a system for automatic quantitative analysis of laboratory mice behavior based on locomotor pose

Markerless Rat Behavior Quantification With Cascade Neural Network

RGB-D Camera based Human Limb Movement Recognition and Tracking in Supine Positions.

Apparatus and methods for mouse behavior recognition on foot contact features

Are You Sitting Right?-Sitting Posture Recognition Using RF Signals

Pig-Posture Recognition Based on Computer Vision: Dataset and Exploration

Hierarchical Spatial–Temporal Window Transformer for Pose-Based Rodent Behavior Recognition

Partially Occluded Head Posture Estimation for 2D Images Using Pyramid HoG Features

Head-reference Human Contour Model

Human Motion Posture Detection Algorithm Using Deep Reinforcement Learning