Abstract:Spiking neural networks (SNNs) are rich in spatio-temporal dynamics and are suitable for processing event-based neuromorphic data. However, event-based datasets are usually less annotated than static datasets. This small data scale makes SNNs prone to overfitting and limits their performance. In order to improve the generalization ability of SNNs on event-based datasets, we use static images to assist SNN training on event data. In this paper, we first discuss the domain mismatch problem encountered when directly transferring networks trained on static datasets to event data. We argue that the inconsistency of feature distributions becomes a major factor hindering the effective transfer of knowledge from static images to event data. To address this problem, we propose solutions in terms of two aspects: feature distribution and training strategy. Firstly, we propose a knowledge transfer loss, which consists of domain alignment loss and spatio-temporal regularization. The domain alignment loss learns domain-invariant spatial features by reducing the marginal distribution distance between the static image and the event data. Spatio-temporal regularization provides dynamically learnable coefficients for domain alignment loss by using the output features of the event data at each time step as a regularization term. In addition, we propose a sliding training strategy, which gradually replaces static image inputs probabilistically with event data, resulting in a smoother and more stable training for the network. We validate our method on neuromorphic datasets, including N-Caltech101, CEP-DVS, and N-Omniglot. The experimental results show that our proposed method achieves better performance on all datasets compared to the current state-of-the-art methods. Code is available at <a class="link-external link-https" href="https://github.com/Brain-Cog-Lab/Transfer-for-DVS" rel="external noopener nofollow">this https URL</a>.

Spiking Transfer Learning From RGB Image to Neuromorphic Event Stream

An Efficient Knowledge Transfer Strategy for Spiking Neural Networks from Static to Event Domain

An Event-based Feature Representation Method for Event Stream Classification Using Deep Spiking Neural Networks

Hierarchical Spiking-Based Model for Efficient Image Classification with Enhanced Feature Extraction and Encoding.

An Event-based Categorization Model Using Spatio-temporal Features in a Spiking Neural Network.

An Event-Driven Computational System With Spiking Neurons For Object Recognition

Event-based Action Recognition Using Motion Information and Spiking Neural Networks

SSTFormer: Bridging Spiking Neural Network and Memory Support Transformer for Frame-Event based Recognition

Towards Event Camera Signal Recognition Using a Lightweight Spiking Neural Network

Towards event camera signal recognition with lightweight spiking

[A bio-inspired hierarchical spiking neural network with biological synaptic plasticity for event camera object recognition].

Deep CovDenseSNN: A Hierarchical Event-Driven Dynamic Framework with Spiking Neurons in Noisy Environment

Training Robust Spiking Neural Networks on Neuromorphic Data with Spatiotemporal Fragments

A Bio-Inspired Hierarchical Spiking Neural Network With Reward-Modulated STDP Learning Rule for AER Object Recognition

Event-driven Spiking Neural Networks with Spike-Based Learning

Representation Learning using Event-based STDP

Effective AER Object Classification Using Segmented Probability-Maximization Learning in Spiking Neural Networks

Improving NeuCube Spiking Neural Network for EEG-based Pattern Recognition Using Transfer Learning

EventRPG: Event Data Augmentation with Relevance Propagation Guidance

NeuSpike-Net: High Speed Video Reconstruction via Bio-inspired Neuromorphic Cameras

Ultra-low Latency Spiking Neural Networks with Spatio-Temporal Compression and Synaptic Convolutional Block