Abstract:Intelligent agents, such as robots, have to serve a multitude of autonomous functions. Examples are, e.g., collision avoidance, navigation and route planning, active sensing of its environment, or the interaction and non-verbal communication with people in the extended reach space. Here, we focus on the recognition of the action of a human agent based on a biologically inspired visual architecture of analyzing articulated movements. The proposed processing architecture builds upon coarsely segregated streams of sensory processing along different pathways which separately process form and motion information (Layher et al., 2014). Action recognition is performed in an event-based scheme by identifying representations of characteristic pose configurations (key poses) in an image sequence. In line with perceptual studies, key poses are selected unsupervised utilizing a feature-driven criterion which combines extrema in the motion energy with the horizontal and the vertical extendedness of a body shape. Per class representations of key pose frames are learned using a deep convolutional neural network consisting of 15 convolutional layers. The network is trained using the energy-efficient deep neuromorphic networks (Eedn) framework (Esser et al., 2016), which realizes the mapping of the trained synaptic weights onto the IBM Neurosynaptic System platform (Merolla et al., 2014). After the mapping, the trained network achieves real-time capabilities for processing input streams and classify input images at about 1,000 frames per second while the computational stages only consume about 70 mW of energy (without spike transduction). Particularly regarding mobile robotic systems, a low energy profile might be crucial in a variety of application scenarios. Cross-validation results are reported for two different datasets and compared to state-of-the-art action recognition approaches. The results demonstrate, that (I) the presented approach is on par with other key pose based methods described in the literature, which select key pose frames by optimizing classification accuracy, (II) compared to the training on the full set of frames, representations trained on key pose frames result in a higher confidence in class assignments, and (III) key pose representations show promising generalization capabilities in a cross-dataset evaluation.

CNN-Based Action Recognition and Pose Estimation for Classifying Animal Behavior from Videos: A Survey

Modelling Human Body Pose for Action Recognition Using Deep Neural Networks

Human Action Recognition Using Deep Learning Methods.

Human Pose-based Estimation, Tracking and Action Recognition with Deep Learning: A Survey

SuperAnimal pretrained pose estimation models for behavioral analysis

A study of animal action segmentation algorithms across supervised, unsupervised, and semi-supervised learning paradigms

Event-based Action Recognition Using Motion Information and Spiking Neural Networks

Animal Behavior Analysis Methods Using Deep Learning: A Survey

Mouse Behavior Recognition Based On Convolution Neural Networ

Pose Recognition in the Wild: Animal pose estimation using Agglomerative Clustering and Contrastive Learning

Analyzing animal behavior via classifying each video frame using convolutional neural networks

An Overview of Behavioral Recognition

Animal Recognition System Based on Convolutional Neural Network

Human Behavior Recognition Based on Multiscale Convolutional Neural Network.

Markerless Dog Pose Recognition in the Wild Using ResNet Deep Learning Model

A review of Convolutional-Neural-Network-based action recognition

Human Behavior Analysis: A Survey on Action Recognition

Real-Time Biologically Inspired Action Recognition from Key Poses Using a Neuromorphic Architecture

RNNs, CNNs and Transformers in Human Action Recognition: A Survey and a Hybrid Model

Empowering Efficient Spatio-Temporal Learning with a 3D CNN for Pose-Based Action Recognition