Abstract:Event cameras, with their high dynamic range and temporal resolution, are ideally suited for object detection, especially under scenarios with motion blur and challenging lighting conditions. However, while most existing approaches prioritize optimizing spatiotemporal representations with advanced detection backbones and early aggregation functions, the crucial issue of adaptive event sampling remains largely unaddressed. Spiking Neural Networks (SNNs), which operate on an event-driven paradigm through sparse spike communication, emerge as a natural fit for addressing this challenge. In this study, we discover that the neural dynamics of spiking neurons align closely with the behavior of an ideal temporal event sampler. Motivated by this insight, we propose a novel adaptive sampling module that leverages recurrent convolutional SNNs enhanced with temporal memory, facilitating a fully end-to-end learnable framework for event-based detection. Additionally, we introduce Residual Potential Dropout (RPD) and Spike-Aware Training (SAT) to regulate potential distribution and address performance degradation encountered in spike-based sampling modules. Empirical evaluation on neuromorphic detection datasets demonstrates that our approach outperforms existing state-of-the-art spike-based methods with significantly fewer parameters and time steps. For instance, our method yields a 4.4\% mAP improvement on the Gen1 dataset, while requiring 38\% fewer parameters and only three time steps. Moreover, the applicability and effectiveness of our adaptive sampling methodology extend beyond SNNs, as demonstrated through further validation on conventional non-spiking models. Code is available at <a class="link-external link-https" href="https://github.com/Windere/EAS-SNN" rel="external noopener nofollow">this https URL</a>.

Event-based Action Recognition Using Motion Information and Spiking Neural Networks

Spiking Neural Network Recognition Method Based on Dynamic Visual Motion Features

RSNN: Recurrent Spiking Neural Networks for Dynamic Spatial-Temporal Information Processing

Learning SpatioTemporal and Motion Features in a Unified 2D Network for Action Recognition

Event-Based Multimodal Spiking Neural Network with Attention Mechanism

Deep CovDenseSNN: A Hierarchical Event-Driven Dynamic Framework with Spiking Neurons in Noisy Environment

An Event-Driven Object Recognition Model Using Activated Connected Domain Detection

SpikePoint: An Efficient Point-based Spiking Neural Network for Event Cameras Action Recognition

An Event-based Feature Representation Method for Event Stream Classification Using Deep Spiking Neural Networks

Spiking Neural Networks for event-based action recognition: A new task to understand their advantage

[A bio-inspired hierarchical spiking neural network with biological synaptic plasticity for event camera object recognition].

Towards Event Camera Signal Recognition Using a Lightweight Spiking Neural Network

CSNN: an Augmented Spiking Based Framework with Perceptron-Inception

Hierarchical Spiking-Based Model for Efficient Image Classification with Enhanced Feature Extraction and Encoding.

An Event-based Categorization Model Using Spatio-temporal Features in a Spiking Neural Network.

A Gesture Recognition Method Based On Spiking Neural Networks For Cognition Development

ReSpike: Residual Frames-based Hybrid Spiking Neural Networks for Efficient Action Recognition

EAS-SNN: End-to-End Adaptive Sampling and Representation for Event-based Detection with Recurrent Spiking Neural Networks

Towards event camera signal recognition with lightweight spiking

Sparser spiking activity can be better: Feature Refine-and-Mask spiking neural network for event-based visual recognition