Abstract:Spiking neural networks (SNNs) are promising brain-inspired energy-efficient models. Recent progress in training methods has enabled successful deep SNNs on large-scale tasks with low latency. Particularly, backpropagation through time (BPTT) with surrogate gradients (SG) is popularly used to achieve high performance in a very small number of time steps. However, it is at the cost of large memory consumption for training, lack of theoretical clarity for optimization, and inconsistency with the online property of biological learning and rules on neuromorphic hardware. Other works connect spike representations of SNNs with equivalent artificial neural network formulation and train SNNs by gradients from equivalent mappings to ensure descent directions. But they fail to achieve low latency and are also not online. In this work, we propose online training through time (OTTT) for SNNs, which is derived from BPTT to enable forward-in-time learning by tracking presynaptic activities and leveraging instantaneous loss and gradients. Meanwhile, we theoretically analyze and prove that gradients of OTTT can provide a similar descent direction for optimization as gradients based on spike representations under both feedforward and recurrent conditions. OTTT only requires constant training memory costs agnostic to time steps, avoiding the significant memory costs of BPTT for GPU training. Furthermore, the update rule of OTTT is in the form of three-factor Hebbian learning, which could pave a path for online on-chip learning. With OTTT, it is the first time that two mainstream supervised SNN training methods, BPTT with SG and spike representation-based training, are connected, and meanwhile in a biologically plausible form. Experiments on CIFAR-10, CIFAR-100, ImageNet, and CIFAR10-DVS demonstrate the superior performance of our method on large-scale static and neuromorphic datasets in small time steps.

A Quantum-Inspired Online Spiking Neural Network for Time-Series Predictions

RSNN: Recurrent Spiking Neural Networks for Dynamic Spatial-Temporal Information Processing

Efficient and Effective Time-Series Forecasting with Spiking Neural Networks

Quantum superposition inspired spiking neural network

Deep CovDenseSNN: A Hierarchical Event-Driven Dynamic Framework with Spiking Neurons in Noisy Environment

Q-SNNs: Quantized Spiking Neural Networks

CSNN: an Augmented Spiking Based Framework with Perceptron-Inception

Comprehensive Online Training and Deployment for Spiking Neural Networks

Towards Energy-Preserving Natural Language Understanding with Spiking Neural Networks

Online Stabilization of Spiking Neural Networks.

A frugal Spiking Neural Network for unsupervised classification of continuous multivariate temporal data

Quantum recurrent neural networks for sequential learning

Spiking neural networks for predictive and explainable modelling of multimodal streaming data with a case study on financial time series and online news

PT-Spike: A Precise-Time-Dependent Single Spike Neuromorphic Architecture with Efficient Supervised Learning

A Time Series Forecasting Approach Based on Nonlinear Spiking Neural Systems

Unsupervised SFQ-Based Spiking Neural Network

Spatial-Temporal Search for Spiking Neural Networks

Quantum Kernel-Based Long Short-term Memory for Climate Time-Series Forecasting

Online Training Through Time for Spiking Neural Networks

Methodology based on spiking neural networks for univariate time-series forecasting