Abstract:Spiking Neural Networks (SNNs) are considered to have enormous potential in the future development of Artificial Intelligence (AI) due to their brain-inspired and energy-efficient properties. In the current supervised learning domain of SNNs, compared to vanilla Spatial-Temporal Back-propagation (STBP) training, online training can effectively overcome the risk of GPU memory explosion and has received widespread academic attention. However, the current proposed online training methods cannot tackle the inseparability problem of temporal dependent gradients and merely aim to optimize the training memory, resulting in no performance advantages compared to the STBP training models in the inference phase. To address the aforementioned challenges, we propose Efficient Multi-Precision Firing (EM-PF) model, which is a family of advanced spiking models based on floating-point spikes and binary synaptic weights. We point out that EM-PF model can effectively separate temporal gradients and achieve full-stage optimization towards computation speed and memory footprint. Experimental results have demonstrated that EM-PF model can be flexibly combined with various techniques including random back-propagation, parallel computation and channel attention mechanism, to achieve state-of-the-art performance with extremely low computational overhead in the field of online learning.

What problem does this paper attempt to address?

The problems that this paper attempts to solve are the two main drawbacks encountered by Spiking Neural Networks (SNNs) during online training: the inconsistency between forward propagation and backward propagation, and the lack of effectiveness in online deployment. Specifically: 1. **Inconsistency between forward propagation and backward propagation**: - Current online learning frameworks will cause consistency problems between forward propagation and backward propagation when separating time - dependent gradients. This is because the surrogate functions of spiking neurons are usually related to membrane potential values, and spike trains are unevenly distributed in the time dimension, making it impossible to incorporate time - dependent gradients into the backward propagation chain in the spatial dimension. This leads to a decline in learning performance. 2. **Lack of effectiveness in online deployment**: - Current online learning methods mainly focus on optimizing training memory, but cannot bring any optimization in terms of computation time and memory usage during the inference stage. This is because in the framework of binary spike firing, it is difficult to introduce parallel computing or weight quantization techniques to improve the inference speed or optimize memory usage without sacrificing learning accuracy. To overcome these problems, the authors propose the Efficient Multi - Precision Firing (EM - PF) model, which is an advanced spiking model based on floating - point spikes and binary synaptic weights. The EM - PF model solves the above problems in the following ways: 1. **Enhancing the separability of backward gradients**: - The EM - PF model solves the non - differentiable problem in the spike - firing process and enhances the uniformity of spike trains by adopting floating - point spikes and binary synaptic weights, significantly improving the separability of backward gradients. 2. **Flexibly combining various optimization techniques**: - The EM - PF model can flexibly combine techniques such as stochastic backpropagation, parallel computing, and channel - attention mechanisms to achieve comprehensive optimization from the training to the inference stage. 3. **Theoretical analysis and experimental verification**: - The authors prove through theoretical analysis that the EM - PF model satisfies the conditions for separable backward gradients under specific conditions, and verify through experiments the superior performance of the EM - PF model on different datasets, especially in terms of parameter memory occupation and learning accuracy. In conclusion, this paper proposes a new online training framework, aiming to solve the problems of forward - propagation - backward - propagation inconsistency and online - deployment effectiveness in SNNs during online training, thereby achieving more efficient online learning and deployment.

Comprehensive Online Training and Deployment for Spiking Neural Networks