Abstract:Event-based cameras have recently shown great potential for high-speed motion estimation owing to their ability to capture temporally rich information asynchronously. Spiking Neural Networks (SNNs), with their neuro-inspired event-driven processing can efficiently handle such asynchronous data, while neuron models such as the leaky-integrate and fire (LIF) can keep track of the quintessential timing information contained in the inputs. SNNs achieve this by maintaining a dynamic state in the neuron memory, retaining important information while forgetting redundant data over time. Thus, we posit that SNNs would allow for better performance on sequential regression tasks compared to similarly sized Analog Neural Networks (ANNs). However, deep SNNs are difficult to train due to vanishing spikes at later layers. To that effect, we propose an adaptive fully-spiking framework with learnable neuronal dynamics to alleviate the spike vanishing problem. We utilize surrogate gradient-based backpropagation through time (BPTT) to train our deep SNNs from scratch. We validate our approach for the task of optical flow estimation on the Multi-Vehicle Stereo Event-Camera (MVSEC) dataset and the DSEC-Flow dataset. Our experiments on these datasets show an average reduction of 13% in average endpoint error (AEE) compared to state-of-the-art ANNs. We also explore several down-scaled models and observe that our SNN models consistently outperform similarly sized ANNs offering 10%-16% lower AEE. These results demonstrate the importance of SNNs for smaller models and their suitability at the edge. In terms of efficiency, our SNNs offer substantial savings in network parameters (48.3x) and computational energy (10.2x) while attaining ~10% lower EPE compared to the state-of-the-art ANN implementations.

Spiking Neural Network Recognition Method Based on Dynamic Visual Motion Features

RSNN: Recurrent Spiking Neural Networks for Dynamic Spatial-Temporal Information Processing

An Event-based Feature Representation Method for Event Stream Classification Using Deep Spiking Neural Networks

Event-based Action Recognition Using Motion Information and Spiking Neural Networks

Event-driven Spiking Neural Networks with Spike-Based Learning

A dynamic vision sensor object recognition model based on trainable event-driven convolution and spiking attention mechanism

Deep CovDenseSNN: A Hierarchical Event-Driven Dynamic Framework with Spiking Neurons in Noisy Environment

Sparser spiking activity can be better: Feature Refine-and-Mask spiking neural network for event-based visual recognition

Spiking Neural Networks for event-based action recognition: A new task to understand their advantage

An Event-Driven Computational System With Spiking Neurons For Object Recognition

A Spike Learning System for Event-driven Object Recognition

Hierarchical Spiking-Based Model for Efficient Image Classification with Enhanced Feature Extraction and Encoding.

Dynamic Spatiotemporal Pattern Recognition with Recurrent Spiking Neural Network

An Event-based Categorization Model Using Spatio-temporal Features in a Spiking Neural Network.

Spiking Neural Network as Adaptive Event Stream Slicer

ReSpike: Residual Frames-based Hybrid Spiking Neural Networks for Efficient Action Recognition

SpikePoint: An Efficient Point-based Spiking Neural Network for Event Cameras Action Recognition

Adaptive-SpikeNet: Event-based Optical Flow Estimation using Spiking Neural Networks with Learnable Neuronal Dynamics

Sparse Temporal Encoding of Visual Features for Robust Object Recognition by Spiking Neurons

CSNN: an Augmented Spiking Based Framework with Perceptron-Inception