Abstract:Intelligent agents, such as robots, have to serve a multitude of autonomous functions. Examples are, e.g., collision avoidance, navigation and route planning, active sensing of its environment, or the interaction and non-verbal communication with people in the extended reach space. Here, we focus on the recognition of the action of a human agent based on a biologically inspired visual architecture of analyzing articulated movements. The proposed processing architecture builds upon coarsely segregated streams of sensory processing along different pathways which separately process form and motion information (Layher et al., 2014). Action recognition is performed in an event-based scheme by identifying representations of characteristic pose configurations (key poses) in an image sequence. In line with perceptual studies, key poses are selected unsupervised utilizing a feature-driven criterion which combines extrema in the motion energy with the horizontal and the vertical extendedness of a body shape. Per class representations of key pose frames are learned using a deep convolutional neural network consisting of 15 convolutional layers. The network is trained using the energy-efficient deep neuromorphic networks (Eedn) framework (Esser et al., 2016), which realizes the mapping of the trained synaptic weights onto the IBM Neurosynaptic System platform (Merolla et al., 2014). After the mapping, the trained network achieves real-time capabilities for processing input streams and classify input images at about 1,000 frames per second while the computational stages only consume about 70 mW of energy (without spike transduction). Particularly regarding mobile robotic systems, a low energy profile might be crucial in a variety of application scenarios. Cross-validation results are reported for two different datasets and compared to state-of-the-art action recognition approaches. The results demonstrate, that (I) the presented approach is on par with other key pose based methods described in the literature, which select key pose frames by optimizing classification accuracy, (II) compared to the training on the full set of frames, representations trained on key pose frames result in a higher confidence in class assignments, and (III) key pose representations show promising generalization capabilities in a cross-dataset evaluation.

Overcoming the Domain Gap in Neural Action Representations

Overcoming the Domain Gap in Contrastive Learning of Neural Action Representations

Real-Time Biologically Inspired Action Recognition from Key Poses Using a Neuromorphic Architecture

Seeing eye-to-eye? A comparison of object recognition performance in humans and deep convolutional neural networks under image manipulation

Deformation-aware Unpaired Image Translation for Pose Estimation on Laboratory Animals

Exploiting Multiple EEG Data Domains with Adversarial Learning

Resolving Domain Shift For Representations Of Speech In Non-Invasive Brain Recordings

Ecologically mapped neuronal identity: Towards standardizing activity across heterogeneous experiments

Brain-inspired Computational Modeling of Action Recognition with Recurrent Spiking Neural Networks Equipped with Reinforcement Delay Learning

How to optimize neuroscience data utilization and experiment design for advancing primate visual and linguistic brain models?

Achieving More Human Brain-Like Vision via Human EEG Representational Alignment

Exploring biometric domain adaptation in human action recognition models for unconstrained environments

CNN-Based Action Recognition and Pose Estimation for Classifying Animal Behavior from Videos: A Survey

Neuralizer: General Neuroimage Analysis without Re-Training

The Algonauts Project 2023 Challenge: How the Human Brain Makes Sense of Natural Scenes

A study of animal action segmentation algorithms across supervised, unsupervised, and semi-supervised learning paradigms

Neural networks and neuroscience-inspired computer vision

Exploiting Large Neuroimaging Datasets to Create Connectome-Constrained Approaches for more Robust, Efficient, and Adaptable Artificial Intelligence

Automatic mapping of multiplexed social receptive fields by deep learning and GPU-accelerated 3D videography

Generalizability analysis of deep learning predictions of human brain responses to augmented and semantically novel visual stimuli

NeuroCine: Decoding Vivid Video Sequences from Human Brain Activties