Direct Training High-Performance Deep Spiking Neural Networks: A Review of Theories and Methods

Chenlin Zhou,Han Zhang,Liutao Yu,Yumin Ye,Zhaokun Zhou,Liwei Huang,Zhengyu Ma,Xiaopeng Fan,Huihui Zhou,Yonghong Tian

2024-07-10

Abstract:Spiking neural networks (SNNs) offer a promising energy-efficient alternative to artificial neural networks (ANNs), in virtue of their high biological plausibility, rich spatial-temporal dynamics, and event-driven computation. The direct training algorithms based on the surrogate gradient method provide sufficient flexibility to design novel SNN architectures and explore the spatial-temporal dynamics of SNNs. According to previous studies, the performance of models is highly dependent on their sizes. Recently, direct training deep SNNs have achieved great progress on both neuromorphic datasets and large-scale static datasets. Notably, transformer-based SNNs show comparable performance with their ANN counterparts. In this paper, we provide a new perspective to summarize the theories and methods for training deep SNNs with high performance in a systematic and comprehensive way, including theory fundamentals, spiking neuron models, advanced SNN models and residual architectures, software frameworks and neuromorphic hardware, applications, and future trends. The reviewed papers are collected at <a class="link-external link-https" href="https://github.com/zhouchenlin2096/Awesome-Spiking-Neural-Networks" rel="external noopener nofollow">this https URL</a>

Neural and Evolutionary Computing

What problem does this paper attempt to address?

This paper provides a comprehensive overview of the theory and methods for directly training high-performance Deep Spiking Neural Networks (SNNs). SNNs, as a biologically inspired computational model, are considered potential competitors to traditional Artificial Neural Networks (ANNs) due to their low energy consumption and similarity to the human brain. The paper focuses on direct training of SNNs rather than the methods for converting ANNs to SNNs, as conversion methods are limited by the original ANN architecture and are not suitable for dynamic signal processing.

In the methods for direct training of SNNs, proxy gradient methods are commonly used for backpropagation due to the non-differentiable nature of spiking neurons. This approach allows SNNs to handle time-dependent data and achieve good performance on large static datasets, while also providing flexibility in designing novel SNN architectures and directly exploring SNN characteristics. In recent years, transformer-based SNNs have reached performance levels comparable to ANNs.

The paper is organized as follows: it introduces the development and latest advances in spiking neuron models, elaborates on the fundamental principles of SNNs, discusses recent advanced SNN models and architectures, particularly transformer-based SNNs, summarizes the development of software frameworks and neuromorphic hardware, provides an overview of SNN applications, and identifies future research trends.

The article also discusses different types of spiking neuron models, such as trainable parameter spiking neurons and parallel spiking neurons, and how they improve the performance of SNNs by enhancing membrane potential and thresholds. Additionally, the paper covers information encoding in SNNs, network training methods (such as proxy gradients and loss functions), the application of batch normalization in SNNs, and the latest developments in SNN architectures, such as residual connections and transformer-based structures.

In conclusion, this paper aims to systematically summarize the theory and methods for directly training deep SNNs, providing a comprehensive perspective for research in this field.