Abstract:Spiking neural networks (SNNs) have attracted attention due to their biological plausibility and the potential for low-energy applications on neuromorphic hardware. Two mainstream approaches are commonly used to obtain SNNs, i.e. , ANN-to-SNN conversion methods, and Directly-trained-SNN methods. However, the former achieve excellent performance at the cost of a large number of time steps ( i.e. , latency), while the latter exhibit lower latency but suffers from suboptimal performance. To tackle the performance-latency trade-off, we propose Self-Architectural Knowledge Distillation (SAKD), an intuitive and effective method for SNNs leveraging Knowledge Distillation (KD). We adopt a bilevel teacher-student training strategy in SAKD, i.e. , level-1 involves directly transferring same-architectural pre-trained ANN weights to SNNs, and level-2 encourages the SNNs to mimic ANN's behavior, considering both final responses and intermediate features aspects. Learning with informative supervision signals fostered by labels and ANNs, our SAKD achieves new state-of-the-art (SOTA) performance with a few time steps on widely-used classification benchmark datasets. On ImageNet-1K, with only 4 time steps, our Spiking-ResNet34 model attains a Top-1 accuracy of 70.04%, outperforming the previous same-architectural SOTA methods. Notably, our SEW-ResNet152 model reaches a Top-1 accuracy of 77.30% on ImageNet-1K, setting a new SOTA benchmark for SNNs. Furthermore, we apply our SAKD to various dense prediction downstream tasks, such as object detection and semantic segmentation, demonstrating strong generalization ability and superior performance. In conclusion, our proposed SAKD framework presents a promising approach for achieving both high performance and low latency in SNNs, potentially paving the way for future advancements in the field.

LaSNN: Layer-wise ANN-to-SNN Distillation for Effective and Efficient Training in Deep Spiking Neural Networks

IDSNN: Towards High-Performance and Low-Latency SNN Training Via Initialization and Distillation.

Joint A-SNN: Joint Training of Artificial and Spiking Neural Networks via Self-Distillation and Weight Factorization

Spiking Deep Residual Networks.

Adaptive Multi-Level Firing for Direct Training Deep Spiking Neural Networks

Effective Active Learning Method for Spiking Neural Networks.

High-accuracy deep ANN-to-SNN conversion using quantization-aware training framework and calcium-gated bipolar leaky integrate and fire neuron

Constructing Deep Spiking Neural Networks from Artificial Neural Networks with Knowledge Distillation

Optimal ANN-SNN Conversion for Fast and Accurate Inference in Deep Spiking Neural Networks

Biologically Inspired Structure Learning with Reverse Knowledge Distillation for Spiking Neural Networks

A universal ANN-to-SNN framework for achieving high accuracy and low latency deep Spiking Neural Networks

An Efficient Learning Algorithm for Direct Training Deep Spiking Neural Networks

A New ANN-SNN Conversion Method with High Accuracy, Low Latency and Good Robustness

Self-architectural knowledge distillation for spiking neural networks

Training much deeper spiking neural networks with a small number of time-steps

Spike Trains Encoding and Threshold Rescaling Method for Deep Spiking Neural Networks

Deep CovDenseSNN: A Hierarchical Event-Driven Dynamic Framework with Spiking Neurons in Noisy Environment

Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency Spiking Neural Networks

Optimized Potential Initialization for Low-latency Spiking Neural Networks

Toward High-Accuracy and Low-Latency Spiking Neural Networks With Two-Stage Optimization

Optimal Conversion of Conventional Artificial Neural Networks to Spiking Neural Networks