Abstract:Spiking Neural Networks (SNNs) are considered as a potential candidate for the next generation of artificial intelligence with appealing characteristics such as sparse computation and inherent temporal dynamics. By adopting architectures of Artificial Neural Networks (ANNs), SNNs achieve competitive performances on benchmark tasks like image classification. However, successful architectures of ANNs are not optimal for SNNs. In this work, we apply Neural Architecture Search (NAS) to find suitable architectures for SNNs. Previous NAS methods for SNNs focus primarily on the spatial dimension, with a notable lack of consideration for the temporal dynamics that are of critical importance for SNNs. Drawing inspiration from the heterogeneity of biological neural networks, we propose a differentiable approach to optimize SNN on both spatial and temporal dimensions. At spatial level, we have developed a spike-based differentiable hierarchical search (SpikeDHS) framework, where spike-based operation is optimized on both the cell and the layer level under computational constraints. We further propose a differentiable surrogate gradient search (DGS) method to evolve local SG functions independently during training. At temporal level, we explore an optimal configuration of diverse temporal dynamics on different types of spiking neurons by evolving their time constants, based on which we further develop hybrid networks combining SNN and ANN, balancing both accuracy and efficiency. Our methods achieve comparable classification performance of CIFAR10/100 and ImageNet with accuracies of 96.43%, 78.96%, and 70.21%, respectively. On event-based deep stereo, our methods find optimal layer variation and surpass the accuracy of specially designed ANNs with 26$\times$ lower computational cost ($6.7\mathrm{mJ}$), demonstrating the potential of SNN in processing highly sparse and dynamic signals.

Efficient Spiking Neural Networks with Sparse Selective Activation for Continual Learning

Adaptive Deep Spiking Neural Network with Global-Local Learning Via Balanced Excitatory and Inhibitory Mechanism

Similarity-based context aware continual learning for spiking neural networks

Effective Active Learning Method for Spiking Neural Networks.

Adaptive Multi-Level Firing for Direct Training Deep Spiking Neural Networks

Deep CovDenseSNN: A Hierarchical Event-Driven Dynamic Framework with Spiking Neurons in Noisy Environment

Spiking Deep Residual Networks.

An Efficient Learning Algorithm for Direct Training Deep Spiking Neural Networks

Deep Spiking Neural Network Using Spatio-temporal Backpropagation with Variable Resistance.

Efficient Spiking Neural Networks with Logarithmic Temporal Coding.

Adaptive Reorganization of Neural Pathways for Continual Learning with Spiking Neural Networks

CSNN: an Augmented Spiking Based Framework with Perceptron-Inception

Enhancing Efficient Continual Learning with Dynamic Structure Development of Spiking Neural Networks

Toward Efficient Processing and Learning with Spikes: New Approaches for Multispike Learning

Efficient Training of Supervised Spiking Neural Network Via Accurate Synaptic-Efficiency Adjustment Method

Adaptive Synaptic Scaling in Spiking Networks for Continual Learning and Enhanced Robustness

Training a General Spiking Neural Network with Improved Efficiency and Minimum Latency

Learnable Leakage and Onset-Spiking Self-Attention in SNNs with Local Error Signals

Spatial-Temporal Search for Spiking Neural Networks

Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting

Attention Spiking Neural Networks