Abstract:Spiking neural networks (SNNs) are rich in spatio-temporal dynamics and are suitable for processing event-based neuromorphic data. However, event-based datasets are usually less annotated than static datasets. This small data scale makes SNNs prone to overfitting and limits their performance. In order to improve the generalization ability of SNNs on event-based datasets, we use static images to assist SNN training on event data. In this paper, we first discuss the domain mismatch problem encountered when directly transferring networks trained on static datasets to event data. We argue that the inconsistency of feature distributions becomes a major factor hindering the effective transfer of knowledge from static images to event data. To address this problem, we propose solutions in terms of two aspects: feature distribution and training strategy. Firstly, we propose a knowledge transfer loss, which consists of domain alignment loss and spatio-temporal regularization. The domain alignment loss learns domain-invariant spatial features by reducing the marginal distribution distance between the static image and the event data. Spatio-temporal regularization provides dynamically learnable coefficients for domain alignment loss by using the output features of the event data at each time step as a regularization term. In addition, we propose a sliding training strategy, which gradually replaces static image inputs probabilistically with event data, resulting in a smoother and more stable training for the network. We validate our method on neuromorphic datasets, including N-Caltech101, CEP-DVS, and N-Omniglot. The experimental results show that our proposed method achieves better performance on all datasets compared to the current state-of-the-art methods. Code is available at <a class="link-external link-https" href="https://github.com/Brain-Cog-Lab/Transfer-for-DVS" rel="external noopener nofollow">this https URL</a>.
What problem does this paper attempt to address?
The main problem that this paper attempts to solve is: **How to improve the generalization ability of spiking neural networks (SNNs) on event - domain data**, especially when the event data sets are small in scale and have few labels. Specifically, the paper focuses on the domain - mismatch problem between static image data and event data, that is, the problem of poor performance when directly transferring models trained on static images to event data.
### Problem Background
1. **Characteristics of Event Data**
- Event data (such as data collected by DVS sensors) has the advantages of high dynamic range, high temporal resolution, and low latency.
- However, the process of obtaining event data is complex and expensive, resulting in event data sets usually being small in scale and having few labels.
2. **Advantages of Static Data**
- Static data sets (such as static images) are large in scale, easy to obtain, and pre - trained deep neural networks can be well transferred to other static data sets.
- Although static images provide rich spatial information, directly transferring pre - trained models on static images to event data often has poor results because there is a domain - mismatch problem between them.
### Core Problems of the Paper
- **Domain - Mismatch Problem**: The feature distributions between static images and event data are quite different, resulting in difficulties in knowledge transfer. Specific manifestations are:
- Models trained directly on static data perform poorly on event data.
- The feature representations of models on different data are significantly different, especially in membrane potential distribution.
### Solutions
To overcome this problem, the paper proposes the following solutions:
1. **Knowledge Transfer Loss Function (Knowledge Transfer Loss)**
- **Domain Alignment Loss**: By minimizing the difference in feature distributions between static images and event data, learn domain - invariant spatial features.
- **Spatio - Temporal Regularization**: Provide a dynamically adjusted coefficient for the domain alignment loss to better capture the temporal characteristics in event data.
2. **Sliding Training Strategy (Sliding Training Strategy)**
- Gradually replace static image inputs with event data during the training process, so that the network can smoothly transition from relying on static images to processing event data.
- This progressive replacement strategy ensures the stability of the training process and avoids training instability or convergence difficulties that may be caused by directly switching data.
### Experimental Verification
The paper conducted experiments on several commonly used event data sets (such as N - Caltech101, CEP - DVS, N - Omniglot), and the results show that the proposed method outperforms the existing state - of - the - art methods on all data sets.
### Summary
The main contributions of this paper are:
- Proposing an effective knowledge transfer loss function that can learn domain - invariant features in the spatio - temporal dimension.
- Designing a sliding training strategy that enables the network to smoothly adapt to event data, thereby improving the generalization ability of SNNs on event data.
Through these methods, the paper successfully solves the domain - mismatch problem between static images and event data and improves the performance of SNNs on event data.