Abstract:Event-based cameras have recently shown great potential for high-speed motion estimation owing to their ability to capture temporally rich information asynchronously. Spiking Neural Networks (SNNs), with their neuro-inspired event-driven processing can efficiently handle such asynchronous data, while neuron models such as the leaky-integrate and fire (LIF) can keep track of the quintessential timing information contained in the inputs. SNNs achieve this by maintaining a dynamic state in the neuron memory, retaining important information while forgetting redundant data over time. Thus, we posit that SNNs would allow for better performance on sequential regression tasks compared to similarly sized Analog Neural Networks (ANNs). However, deep SNNs are difficult to train due to vanishing spikes at later layers. To that effect, we propose an adaptive fully-spiking framework with learnable neuronal dynamics to alleviate the spike vanishing problem. We utilize surrogate gradient-based backpropagation through time (BPTT) to train our deep SNNs from scratch. We validate our approach for the task of optical flow estimation on the Multi-Vehicle Stereo Event-Camera (MVSEC) dataset and the DSEC-Flow dataset. Our experiments on these datasets show an average reduction of 13% in average endpoint error (AEE) compared to state-of-the-art ANNs. We also explore several down-scaled models and observe that our SNN models consistently outperform similarly sized ANNs offering 10%-16% lower AEE. These results demonstrate the importance of SNNs for smaller models and their suitability at the edge. In terms of efficiency, our SNNs offer substantial savings in network parameters (48.3x) and computational energy (10.2x) while attaining ~10% lower EPE compared to the state-of-the-art ANN implementations.

An SNN-Based and Neuromorphic-Hardware-Implementable Noise Filter with Self-adaptive Time Window for Event-Based Vision Sensor.

RSNN: Recurrent Spiking Neural Networks for Dynamic Spatial-Temporal Information Processing

Deep CovDenseSNN: A Hierarchical Event-Driven Dynamic Framework with Spiking Neurons in Noisy Environment

A Noise Filter for Dynamic Vision Sensors using Self-adjusting Threshold

Hierarchical Spiking-Based Model for Efficient Image Classification with Enhanced Feature Extraction and Encoding.

A dynamic vision sensor object recognition model based on trainable event-driven convolution and spiking attention mechanism

SeqXFilter: A Memory-efficient Denoising Filter for Dynamic Vision Sensors

CSNN: an Augmented Spiking Based Framework with Perceptron-Inception

Exploiting Neuron and Synapse Filter Dynamics in Spatial Temporal Learning of Deep Spiking Neural Network

Computational event-driven vision sensors for in-sensor spiking neural networks

Event-based Video Reconstruction via Potential-assisted Spiking Neural Network

Event-driven Spiking Neural Networks with Spike-Based Learning

Event-Based Multimodal Spiking Neural Network with Attention Mechanism

RN-Net: Reservoir Nodes-Enabled Neuromorphic Vision Sensing Network

VTSNN: a virtual temporal spiking neural network

A 1024-Neuron 1M-Synapse Event-Driven SNN Accelerator for DVS Applications

Sparser spiking activity can be better: Feature Refine-and-Mask spiking neural network for event-based visual recognition

DVS-Attacks: Adversarial Attacks on Dynamic Vision Sensors for Spiking Neural Networks

An Event Coding Method Based on Frame Images with Dynamic Vision Sensor Modeling

Adaptive-SpikeNet: Event-based Optical Flow Estimation using Spiking Neural Networks with Learnable Neuronal Dynamics

Ultra-low Latency Spiking Neural Networks with Spatio-Temporal Compression and Synaptic Convolutional Block