Abstract:Intelligent agents, such as robots, have to serve a multitude of autonomous functions. Examples are, e.g., collision avoidance, navigation and route planning, active sensing of its environment, or the interaction and non-verbal communication with people in the extended reach space. Here, we focus on the recognition of the action of a human agent based on a biologically inspired visual architecture of analyzing articulated movements. The proposed processing architecture builds upon coarsely segregated streams of sensory processing along different pathways which separately process form and motion information (Layher et al., 2014). Action recognition is performed in an event-based scheme by identifying representations of characteristic pose configurations (key poses) in an image sequence. In line with perceptual studies, key poses are selected unsupervised utilizing a feature-driven criterion which combines extrema in the motion energy with the horizontal and the vertical extendedness of a body shape. Per class representations of key pose frames are learned using a deep convolutional neural network consisting of 15 convolutional layers. The network is trained using the energy-efficient deep neuromorphic networks (Eedn) framework (Esser et al., 2016), which realizes the mapping of the trained synaptic weights onto the IBM Neurosynaptic System platform (Merolla et al., 2014). After the mapping, the trained network achieves real-time capabilities for processing input streams and classify input images at about 1,000 frames per second while the computational stages only consume about 70 mW of energy (without spike transduction). Particularly regarding mobile robotic systems, a low energy profile might be crucial in a variety of application scenarios. Cross-validation results are reported for two different datasets and compared to state-of-the-art action recognition approaches. The results demonstrate, that (I) the presented approach is on par with other key pose based methods described in the literature, which select key pose frames by optimizing classification accuracy, (II) compared to the training on the full set of frames, representations trained on key pose frames result in a higher confidence in class assignments, and (III) key pose representations show promising generalization capabilities in a cross-dataset evaluation.

E-BabyNet: Enhanced Action Recognition of Infant Reaching in Unconstrained Environments

Challenges in Video-Based Infant Action Recognition: A Critical Examination of the State of the Art

Baby Physical Safety Monitoring in Smart Home Using Action Recognition System

Computational Baby Learning

Artificial intelligence detects awareness of functional relation with the environment in 3 month old babies

Activity Recognition From Newborn Resuscitation Videos

Deep-learning for automated markerless tracking of infants general movements

A Video-based End-to-end Pipeline for Non-nutritive Sucking Action Recognition and Segmentation in Young Infants

Adapting Action Recognition Neural Networks for Automated Infantile Spasm Detection

MIE-Net: Motion Information Enhancement Network for Fine-Grained Action Recognition Using RGB Sensors

Vision-based activity recognition in children with autism-related behaviors

Towards Computational Baby Learning: A Weakly-Supervised Approach for Object Detection

Preterm infants' pose estimation with spatio-temporal features

Automatic infant 2D pose estimation from videos: comparing seven deep neural network methods

Cascaded Interactional Targeting Network for Egocentric Video Analysis.

A Jeap-BiLSTM Neural Network for Action Recognition

Robotic 3D reaching through a development-driven double neural network architecture

ResLNet: Deep Residual LSTM Network with Longer Input for Action Recognition

SonoNet: Real-Time Detection and Localisation of Fetal Standard Scan Planes in Freehand Ultrasound

A Spatio-temporal Attention-based Model for Infant Movement Assessment from Videos

Real-Time Biologically Inspired Action Recognition from Key Poses Using a Neuromorphic Architecture