Abstract:Traditionally the Perception Action cycle is the first stage of building an autonomous robotic system and a practical way to implement a low latency reactive system within a low Size, Weight and Power (SWaP) package. However, within complex scenarios, this method can lack contextual understanding about the scene, such as object recognition-based tracking or system attention. Object detection, identification and tracking along with semantic segmentation and attention are all modern computer vision tasks in which Convolutional Neural Networks (CNN) have shown significant success, although such networks often have a large computational overhead and power requirements, which are not ideal in smaller robotics tasks. Furthermore, cloud computing and massively parallel processing like in Graphic Processing Units (GPUs) are outside the specification of many tasks due to their respective latency and SWaP constraints. In response to this, Spiking Convolutional Neural Networks (SCNNs) look to provide the feature extraction benefits of CNNs, while maintaining low latency and power overhead thanks to their asynchronous spiking event-based processing. A novel Neuromorphic Perception Understanding Action (PUA) system is presented, that aims to combine the feature extraction benefits of CNNs with low latency processing of SCNNs. The PUA utilizes a Neuromorphic Vision Sensor for Perception that facilitates asynchronous processing within a Spiking fully Convolutional Neural Network (SpikeCNN) to provide semantic segmentation and Understanding of the scene. The output is fed to a spiking control system providing Actions. With this approach, the aim is to bring features of deep learning into the lower levels of autonomous robotics, while maintaining a biologically plausible STDP rule throughout the learned encoding part of the network. The network will be shown to provide a more robust and predictable management of spiking activity with an improved thresholding response. The reported experiments show that this system can deliver robust results of over 96 and 81% for accuracy and Intersection over Union, ensuring such a system can be successfully used within object recognition, classification and tracking problem. This demonstrates that the attention of the system can be tracked accurately, while the asynchronous processing means the controller can give precise track updates with minimal latency.

How the Brain Achieves Real-Time Vision: A Spiking Position Perception Model

Real-time vision system for a Ping-Pong robot

RSNN: Recurrent Spiking Neural Networks for Dynamic Spatial-Temporal Information Processing

Embodied Neuromorphic Vision with Event-Driven Random Backpropagation

Hierarchical Spiking-Based Model for Efficient Image Classification with Enhanced Feature Extraction and Encoding.

Spiking-timing based pattern recognition with real-world visual stimuli

Real-Time Target Tracking System with Spiking Neural Networks Implemented on Neuromorphic Chips

Perception Understanding Action: Adding Understanding to the Perception Action Cycle With Spiking Segmentation

A Spiking Artificial Vision Architecture Based on Fully Emulating the Human Vision

A dynamic vision sensor object recognition model based on trainable event-driven convolution and spiking attention mechanism

Research on Brain-inspired Vision Based on Dynamic Vision Sensor Cameras

Toward Efficient Processing and Learning with Spikes: New Approaches for Multispike Learning

NeuroVE: Brain-inspired Linear-Angular Velocity Estimation with Spiking Neural Networks

Training Robust Spiking Neural Networks with ViewPoint Transform and SpatioTemporal Stretching

A Spiking Visual Neuron for Depth Perceptual Systems

Real-time Tracking Based on Neuromrophic Vision

Towards the Next Generation of Retinal Neuroprosthesis: Visual Computation with Spikes

Dynamic Spatiotemporal Pattern Recognition with Recurrent Spiking Neural Network

A Spatial–Channel–Temporal-Fused Attention for Spiking Neural Networks

Spiking Elementary Motion Detector in Neuromorphic Systems

Comprehensive Online Training and Deployment for Spiking Neural Networks