Abstract:Spiking Neural Networks (SNNs) are considered as a potential candidate for the next generation of artificial intelligence with appealing characteristics such as sparse computation and inherent temporal dynamics. By adopting architectures of Artificial Neural Networks (ANNs), SNNs achieve competitive performances on benchmark tasks like image classification. However, successful architectures of ANNs are not optimal for SNNs. In this work, we apply Neural Architecture Search (NAS) to find suitable architectures for SNNs. Previous NAS methods for SNNs focus primarily on the spatial dimension, with a notable lack of consideration for the temporal dynamics that are of critical importance for SNNs. Drawing inspiration from the heterogeneity of biological neural networks, we propose a differentiable approach to optimize SNN on both spatial and temporal dimensions. At spatial level, we have developed a spike-based differentiable hierarchical search (SpikeDHS) framework, where spike-based operation is optimized on both the cell and the layer level under computational constraints. We further propose a differentiable surrogate gradient search (DGS) method to evolve local SG functions independently during training. At temporal level, we explore an optimal configuration of diverse temporal dynamics on different types of spiking neurons by evolving their time constants, based on which we further develop hybrid networks combining SNN and ANN, balancing both accuracy and efficiency. Our methods achieve comparable classification performance of CIFAR10/100 and ImageNet with accuracies of 96.43%, 78.96%, and 70.21%, respectively. On event-based deep stereo, our methods find optimal layer variation and surpass the accuracy of specially designed ANNs with 26$\times$ lower computational cost ($6.7\mathrm{mJ}$), demonstrating the potential of SNN in processing highly sparse and dynamic signals.

Similarity-based context aware continual learning for spiking neural networks

Efficient Spiking Neural Networks with Sparse Selective Activation for Continual Learning

Adaptive Deep Spiking Neural Network with Global-Local Learning Via Balanced Excitatory and Inhibitory Mechanism

Unleashing the Potential of Spiking Neural Networks for Sequential Modeling with Contextual Embedding.

Adaptive Reorganization of Neural Pathways for Continual Learning with Spiking Neural Networks

Enhancing Efficient Continual Learning with Dynamic Structure Development of Spiking Neural Networks

Deep CovDenseSNN: A Hierarchical Event-Driven Dynamic Framework with Spiking Neurons in Noisy Environment

CSNN: an Augmented Spiking Based Framework with Perceptron-Inception

An Efficient Learning Algorithm for Direct Training Deep Spiking Neural Networks

Context Gating in Spiking Neural Networks: Achieving Lifelong Learning through Integration of Local and Global Plasticity

Adaptive Multi-Level Firing for Direct Training Deep Spiking Neural Networks

Hierarchical Spiking-Based Model for Efficient Image Classification with Enhanced Feature Extraction and Encoding.

ALADE-SNN: Adaptive Logit Alignment in Dynamically Expandable Spiking Neural Networks for Class Incremental Learning

A Brain-Inspired Spiking Neural Network Model with Temporal Encoding and Learning

Adaptive Smoothing Gradient Learning for Spiking Neural Networks.

Temporal Contrastive Learning for Spiking Neural Networks

Spatial-Temporal Search for Spiking Neural Networks

Training Spiking Neural Networks for Cognitive Tasks: A Versatile Framework Compatible with Various Temporal Codes.

Effect Exploration of the Spiking Neuron Property and Heterogeneity on Brain-Inspired Context-Dependent Learning

A Curiosity-Based Learning Method for Spiking Neural Networks

Training Stronger Spiking Neural Networks with Biomimetic Adaptive Internal Association Neurons