Abstract:Spiking Neural Networks (SNNs) are considered as a potential candidate for the next generation of artificial intelligence with appealing characteristics such as sparse computation and inherent temporal dynamics. By adopting architectures of Artificial Neural Networks (ANNs), SNNs achieve competitive performances on benchmark tasks like image classification. However, successful architectures of ANNs are not optimal for SNNs. In this work, we apply Neural Architecture Search (NAS) to find suitable architectures for SNNs. Previous NAS methods for SNNs focus primarily on the spatial dimension, with a notable lack of consideration for the temporal dynamics that are of critical importance for SNNs. Drawing inspiration from the heterogeneity of biological neural networks, we propose a differentiable approach to optimize SNN on both spatial and temporal dimensions. At spatial level, we have developed a spike-based differentiable hierarchical search (SpikeDHS) framework, where spike-based operation is optimized on both the cell and the layer level under computational constraints. We further propose a differentiable surrogate gradient search (DGS) method to evolve local SG functions independently during training. At temporal level, we explore an optimal configuration of diverse temporal dynamics on different types of spiking neurons by evolving their time constants, based on which we further develop hybrid networks combining SNN and ANN, balancing both accuracy and efficiency. Our methods achieve comparable classification performance of CIFAR10/100 and ImageNet with accuracies of 96.43%, 78.96%, and 70.21%, respectively. On event-based deep stereo, our methods find optimal layer variation and surpass the accuracy of specially designed ANNs with 26$\times$ lower computational cost ($6.7\mathrm{mJ}$), demonstrating the potential of SNN in processing highly sparse and dynamic signals.

Surrogate-Assisted Evolutionary Search of Spiking Neural Architectures in Liquid State Machines

Surrogate-Assisted Cooperative Co-evolutionary Reservoir Architecture Search for Liquid State Machines

Evolutionary Optimization Of Liquid State Machines For Robust Learning

Adaptive structure evolution and biologically plausible synaptic plasticity for recurrent spiking neural networks

Learnable Surrogate Gradient for Direct Training Spiking Neural Networks

Adaptive Multi-Level Firing for Direct Training Deep Spiking Neural Networks

Increasing Liquid State Machine Performance with Edge-of-Chaos Dynamics Organized by Astrocyte-modulated Plasticity

Improving Liquid State Machine with Hybrid Plasticity

Spatial-Temporal Search for Spiking Neural Networks

An Efficient Learning Algorithm for Direct Training Deep Spiking Neural Networks

Dynamic Training of Liquid State Machines

Emergence of brain-inspired small-world spiking neural network through neuroevolution

Multi-scale Evolutionary Neural Architecture Search for Deep Spiking Neural Networks

Efficient spiking neural network design via neural architecture search

A Classification Surrogate Model based Evolutionary Algorithm for Neural Network Structure Learning

Surrogate-assisted Evolutionary Neural Architecture Search with Network Embedding

Neural Architecture Search using Covariance Matrix Adaptation Evolution Strategy

SpikingSSMs: Learning Long Sequences with Sparse and Parallel Spiking State Space Models

Efficient Multi-objective Neural Architecture Search via Lamarckian Evolution

Toward Less Constrained Macro-Neural Architecture Search