Abstract:Event-based cameras have recently shown great potential for high-speed motion estimation owing to their ability to capture temporally rich information asynchronously. Spiking Neural Networks (SNNs), with their neuro-inspired event-driven processing can efficiently handle such asynchronous data, while neuron models such as the leaky-integrate and fire (LIF) can keep track of the quintessential timing information contained in the inputs. SNNs achieve this by maintaining a dynamic state in the neuron memory, retaining important information while forgetting redundant data over time. Thus, we posit that SNNs would allow for better performance on sequential regression tasks compared to similarly sized Analog Neural Networks (ANNs). However, deep SNNs are difficult to train due to vanishing spikes at later layers. To that effect, we propose an adaptive fully-spiking framework with learnable neuronal dynamics to alleviate the spike vanishing problem. We utilize surrogate gradient-based backpropagation through time (BPTT) to train our deep SNNs from scratch. We validate our approach for the task of optical flow estimation on the Multi-Vehicle Stereo Event-Camera (MVSEC) dataset and the DSEC-Flow dataset. Our experiments on these datasets show an average reduction of 13% in average endpoint error (AEE) compared to state-of-the-art ANNs. We also explore several down-scaled models and observe that our SNN models consistently outperform similarly sized ANNs offering 10%-16% lower AEE. These results demonstrate the importance of SNNs for smaller models and their suitability at the edge. In terms of efficiency, our SNNs offer substantial savings in network parameters (48.3x) and computational energy (10.2x) while attaining ~10% lower EPE compared to the state-of-the-art ANN implementations.

Learning Optical Flow from Continuous Spike Streams

Optical Flow for Spike Camera with Hierarchical Spatial-Temporal Spike Fusion

Unsupervised Optical Flow Estimation with Dynamic Timing Representation for Spike Camera

Self-Supervised Joint Dynamic Scene Reconstruction and Optical Flow Estimation for Spiking Camera

Spike-FlowNet: Event-based Optical Flow Estimation with Energy-Efficient Hybrid Neural Networks

Optical flow estimation from event-based cameras and spiking neural networks

SDformerFlow: Spatiotemporal swin spikeformer for event-based optical flow estimation

Optical Flow Dataset and Benchmark for Visual Crowd Analysis

SpikeCV: Open a Continuous Computer Vision Era

Spatio-Temporal Recurrent Networks for Event-Based Optical Flow Estimation

Motion Estimation for Spike Camera Data Sequence Via Spike Interval Analysis.

Adaptive-SpikeNet: Event-based Optical Flow Estimation using Spiking Neural Networks with Learnable Neuronal Dynamics

Learning Temporal-Ordered Representation for Spike Streams Based on Discrete Wavelet Transforms.

Neuromorphic Optical Flow and Real-time Implementation with Event Cameras

Dense Continuous-Time Optical Flow from Event Cameras

Boosting Spike Camera Image Reconstruction from a Perspective of Dealing with Spike Fluctuations

SpikeCodec: An End-to-end Learned Compression Framework for Spiking Camera

Unsupervised Learning of a Hierarchical Spiking Neural Network for Optical Flow Estimation: From Events to Global Motion Perception

High-Speed Motion Scene Reconstruction for Spike Camera via Motion Aligned Filtering

SKFlow: Learning Optical Flow with Super Kernels

Recognizing Ultra-High-Speed Moving Objects with Bio-Inspired Spike Camera