Abstract:Event cameras form a fundamental foundation for visual perception in scenes characterized by high speed and a wide dynamic range. Although deep learning techniques have achieved remarkable success in estimating event-based optical flow, existing methods have not adequately addressed the significance of temporal information in capturing spatiotemporal features. Due to the dynamics of spiking neurons in SNNs, which preserve important information while forgetting redundant information over time, they are expected to outperform analog neural networks (ANNs) with the same architecture and size in sequential regression tasks. In addition, SNNs on neuromorphic hardware achieve advantages of extremely low power consumption. However, present SNN architectures encounter issues related to limited generalization and robustness during training, particularly in noisy scenes. To tackle these problems, this study introduces an innovative spike-based self-supervised learning algorithm known as SeLHIB, which leverages the information bottleneck theory. By utilizing event-based camera inputs, SeLHIB enables robust estimation of optical flow in the presence of noise. To the best of our knowledge, this is the first proposal of a self-supervised information bottleneck learning strategy based on spiking neural networks. Furthermore, we develop a spike-based self-supervised algorithm for higher-order information bottleneck learning that employs nonlinear mutual information to enhance the extraction of relevant information and eliminate redundancy. The proposed network is trained end-to-end using self-supervised learning on a dataset from a multi-vehicle stereo event camera. We demonstrate that SeLHIB significantly enhances the generalization and robustness of optical flow estimation in various noise conditions. In terms of energy efficiency, SeLHIB achieves 90.44% and 45.70% cut down of energy consumption compared to its counterpart ANN and counterpart SNN models, while attaining 33.78% lower AEE (MVSEC), 5.86% lower RAST (ECD) and 6.22% lower RAST (HQF) compared to the state-of-the-art counterpart ANN implementations with the same sizes and architectures.

Training a spiking neural network on an event-based label-free flow cytometry dataset

Spike-FlowNet: Event-based Optical Flow Estimation with Energy-Efficient Hybrid Neural Networks

Photonic Neuromorphic Accelerators for Event-Based Imaging Flow Cytometry

Adaptive-SpikeNet: Event-based Optical Flow Estimation using Spiking Neural Networks with Learnable Neuronal Dynamics

GateNet: A novel Neural Network Architecture for Automated Flow Cytometry Gating

Optical flow estimation from event-based cameras and spiking neural networks

Hierarchical Spiking-Based Model for Efficient Image Classification with Enhanced Feature Extraction and Encoding.

AutoGater: a weakly supervised neural network model to gate cells in flow cytometric analyses

SDformerFlow: Spatiotemporal swin spikeformer for event-based optical flow estimation

Spiking Transfer Learning From RGB Image to Neuromorphic Event Stream

An Efficient Learning Algorithm for Direct Training Deep Spiking Neural Networks

Unsupervised Learning of a Hierarchical Spiking Neural Network for Optical Flow Estimation: From Events to Global Motion Perception

Label-free cell cycle analysis for high-throughput imaging flow cytometry

An Event-based Categorization Model Using Spatio-temporal Features in a Spiking Neural Network.

Self-Supervised High-order Information Bottleneck Learning of Spiking Neural Network for Robust Event-Based Optical Flow Estimation

Coupling a Recurrent Neural Network to SPAD TCSPC Systems for Real-time Fluorescence Lifetime Imaging

Deep Cytometry: Deep learning with Real-time Inference in Cell Sorting and Flow Cytometry

FlowCyt: A Comparative Study of Deep Learning Approaches for Multi-Class Classification in Flow Cytometry Benchmarking

SA‐FlowNet: Event‐based Self‐attention Optical Flow Estimation with Spiking‐analogue Neural Networks

Neuromorphic Optical Flow and Real-time Implementation with Event Cameras

Spatio-Temporal Recurrent Networks for Event-Based Optical Flow Estimation