Abstract:Spiking neural networks (SNNs) provide an alternative solution to conventional artificial neural networks with energy-saving and high-efficiency characteristics after hardware implantation. However, due to its non-differentiable activation function and the temporally delayed accumulation in outputs, the direct training of SNNs is extraordinarily tough even adopting a surrogate gradient to mimic the backpropagation. For SNN training, this non-differentiability causes the intrinsic gradient error that would be magnified through layerwise backpropagation, especially through multiple layers. In this paper, we propose a novel approach to reducing gradient error from a new perspective called surrogate module learning (SML). Surrogate module learning tries to construct a shortcut path to back-propagate a more accurate gradient to a certain SNN part utilizing the surrogate modules. Then, we develop a new loss function for concurrently training the network and enhancing the surrogate modules’ surrogate capacity. We demonstrate that when the outputs of surrogate modules are close to the SNN output, the fraction of the gradient error drops significantly. Our method consistently and significantly enhances the performance of SNNs on all experiment datasets, including CIFAR-10/100, ImageNet, and ES-ImageNet. For example, for spiking ResNet-34 architecture on ImageNet, we increased the SNN accuracy by 3.46%. Codes are available at https://github. com/brain-intelligence-lab/ surrogate_module_learning .

Solving Flat-Spot Problem in Back-Propagation Learning Algorithm Based on Magnified Error

An Algorithm for Fast Convergence of Back Propagation by Enlarging Error

Back Propagation Based on Selective Attention for Fast Convergence of Training Neural Network

Memorized Sparse Backpropagation.

A Fast Learning Algorithm of Neural Networks by Changing Error Functions

Fast Learning Algorithms for Feedforward Neural Networks

A Hybrid Algorithm Based on Attention Model

A Hybrid Algorithm Based on Chips

An Efficient Learning Algorithm for Direct Training Deep Spiking Neural Networks

The Improved Rapid Convergence Algorithm of the Connecting Rights in the BP Network

Optimal Algorithm for a Multilayer Feedforward Net and Its Application

Magnitude Matters: Fixing SIGNSGD Through Magnitude-Aware Sparsification and Error Feedback in the Presence of Data Heterogeneity

Training Feed-forward Neural Networks Using the Gradient Descent Method with the Optimal Stepsize

New and Rapidly Converging BP Neural Network Algorithm

Research on Back-Propagation Learning Algorithm Based on Weight Adjustment Parameters

A Fast Learning Algorithm of Feedforward Neural Networks by Using Novel Error Functions

An Advanced NN Learning Algorithm Based on DFP Method

Bp Learning Algorithm Based on Dfp and Trust Region Method

Improving Steepest Descent Method by Learning Rate Annealing and Momentum in Neural Network

Surrogate Module Learning: Reduce the Gradient Error Accumulation in Training Spiking Neural Networks

Global Convergence of Online Bp Training with Dynamic Learning Rate