Abstract:Spiking neural networks (SNNs) have attracted attention due to their biological plausibility and the potential for low-energy applications on neuromorphic hardware. Two mainstream approaches are commonly used to obtain SNNs, i.e. , ANN-to-SNN conversion methods, and Directly-trained-SNN methods. However, the former achieve excellent performance at the cost of a large number of time steps ( i.e. , latency), while the latter exhibit lower latency but suffers from suboptimal performance. To tackle the performance-latency trade-off, we propose Self-Architectural Knowledge Distillation (SAKD), an intuitive and effective method for SNNs leveraging Knowledge Distillation (KD). We adopt a bilevel teacher-student training strategy in SAKD, i.e. , level-1 involves directly transferring same-architectural pre-trained ANN weights to SNNs, and level-2 encourages the SNNs to mimic ANN's behavior, considering both final responses and intermediate features aspects. Learning with informative supervision signals fostered by labels and ANNs, our SAKD achieves new state-of-the-art (SOTA) performance with a few time steps on widely-used classification benchmark datasets. On ImageNet-1K, with only 4 time steps, our Spiking-ResNet34 model attains a Top-1 accuracy of 70.04%, outperforming the previous same-architectural SOTA methods. Notably, our SEW-ResNet152 model reaches a Top-1 accuracy of 77.30% on ImageNet-1K, setting a new SOTA benchmark for SNNs. Furthermore, we apply our SAKD to various dense prediction downstream tasks, such as object detection and semantic segmentation, demonstrating strong generalization ability and superior performance. In conclusion, our proposed SAKD framework presents a promising approach for achieving both high performance and low latency in SNNs, potentially paving the way for future advancements in the field.

Spiking generative networks empowered by multiple dynamic experts for lifelong learning

Adaptive Deep Spiking Neural Network with Global-Local Learning Via Balanced Excitatory and Inhibitory Mechanism

Efficient Spiking Neural Networks with Sparse Selective Activation for Continual Learning

Adaptive Multi-Level Firing for Direct Training Deep Spiking Neural Networks

Adaptive Smoothing Gradient Learning for Spiking Neural Networks.

Spiking Generative Adversarial Networks With a Neural Network Discriminator: Local Training, Bayesian Models, and Continual Meta-Learning

An Efficient Learning Algorithm for Direct Training Deep Spiking Neural Networks

Learnable Surrogate Gradient for Direct Training Spiking Neural Networks

Few-Shot Learning in Spiking Neural Networks by Multi-Timescale Optimization

SiGNN: A Spike-induced Graph Neural Network for Dynamic Graph Representation Learning

Enhancing Efficient Continual Learning with Dynamic Structure Development of Spiking Neural Networks

Spiking generative adversarial network with attention scoring decoding

Dynamic Spiking Framework for Graph Neural Networks

Evolving Efficient Genetic Encoding for Deep Spiking Neural Networks

Spiking Diffusion Models

Lifelong Dual Generative Adversarial Nets Learning in Tandem

Neurogenesis Dynamics-inspired Spiking Neural Network Training Acceleration

Toward Scalable, Efficient, and Accurate Deep Spiking Neural Networks With Backward Residual Connections, Stochastic Softmax, and Hybridization

Training Deep Convolutional Spiking Neural Networks With Spike Probabilistic Global Pooling

Comprehensive Online Training and Deployment for Spiking Neural Networks

Self-architectural knowledge distillation for spiking neural networks