Abstract:Event cameras form a fundamental foundation for visual perception in scenes characterized by high speed and a wide dynamic range. Although deep learning techniques have achieved remarkable success in estimating event-based optical flow, existing methods have not adequately addressed the significance of temporal information in capturing spatiotemporal features. Due to the dynamics of spiking neurons in SNNs, which preserve important information while forgetting redundant information over time, they are expected to outperform analog neural networks (ANNs) with the same architecture and size in sequential regression tasks. In addition, SNNs on neuromorphic hardware achieve advantages of extremely low power consumption. However, present SNN architectures encounter issues related to limited generalization and robustness during training, particularly in noisy scenes. To tackle these problems, this study introduces an innovative spike-based self-supervised learning algorithm known as SeLHIB, which leverages the information bottleneck theory. By utilizing event-based camera inputs, SeLHIB enables robust estimation of optical flow in the presence of noise. To the best of our knowledge, this is the first proposal of a self-supervised information bottleneck learning strategy based on spiking neural networks. Furthermore, we develop a spike-based self-supervised algorithm for higher-order information bottleneck learning that employs nonlinear mutual information to enhance the extraction of relevant information and eliminate redundancy. The proposed network is trained end-to-end using self-supervised learning on a dataset from a multi-vehicle stereo event camera. We demonstrate that SeLHIB significantly enhances the generalization and robustness of optical flow estimation in various noise conditions. In terms of energy efficiency, SeLHIB achieves 90.44% and 45.70% cut down of energy consumption compared to its counterpart ANN and counterpart SNN models, while attaining 33.78% lower AEE (MVSEC), 5.86% lower RAST (ECD) and 6.22% lower RAST (HQF) compared to the state-of-the-art counterpart ANN implementations with the same sizes and architectures.

A Visually Inspired Computational Model for Recognition of Optic Flow

RSNN: Recurrent Spiking Neural Networks for Dynamic Spatial-Temporal Information Processing

Hierarchical Spiking-Based Model for Efficient Image Classification with Enhanced Feature Extraction and Encoding.

Deep CovDenseSNN: A Hierarchical Event-Driven Dynamic Framework with Spiking Neurons in Noisy Environment

Pattern Recognition of a Spiking Neural Network Based on Visual Motion Model

Biologically Inspired Structure Learning with Reverse Knowledge Distillation for Spiking Neural Networks

CSNN: an Augmented Spiking Based Framework with Perceptron-Inception

Rapid Feedforward Computation by Temporal Encoding and Learning with Spiking Neurons

A Spike-Timing Based Integrated Model for Pattern Recognition

Sparse Temporal Encoding of Visual Features for Robust Object Recognition by Spiking Neurons

Adaptive-SpikeNet: Event-based Optical Flow Estimation using Spiking Neural Networks with Learnable Neuronal Dynamics

RN-Net: Reservoir Nodes-Enabled Neuromorphic Vision Sensing Network

A Spiking Neural Network Based Cortex-Like Mechanism and Application to Facial Expression Recognition

A dynamic vision sensor object recognition model based on trainable event-driven convolution and spiking attention mechanism

Unsupervised Learning of a Hierarchical Spiking Neural Network for Optical Flow Estimation: From Events to Global Motion Perception

Self-Supervised High-order Information Bottleneck Learning of Spiking Neural Network for Robust Event-Based Optical Flow Estimation

Spike-Visnet: A Novel Framework for Visual Recognition with Focuslayer-Stdp Learning

A Bio-Inspired Hierarchical Spiking Neural Network With Reward-Modulated STDP Learning Rule for AER Object Recognition

Reconstruction of Visual Image from Functional Magnetic Resonance Imaging Using Spiking Neuron Model

Modelling Human Visual Motion Processing with Trainable Motion Energy Sensing and a Self-attention Network

Optical flow estimation from event-based cameras and spiking neural networks