Abstract:Spiking neural networks (SNNs) are brain-inspired models that enable energy-efficient implementation on neuromorphic hardware. However, the supervised training of SNNs remains a hard problem due to the discontinuity of the spiking neuron model. Most existing methods imitate the backpropagation framework and feedforward architectures for artificial neural networks, and use surrogate derivatives or compute gradients with respect to the spiking time to deal with the problem. These approaches either accumulate approximation errors or only propagate information limitedly through existing spikes, and usually require information propagation along time steps with large memory costs and biological implausibility. In this work, we consider feedback spiking neural networks, which are more brainlike, and propose a novel training method that does not rely on the exact reverse of the forward computation. First, we show that the average firing rates of SNNs with feedback connections would gradually evolve to an equilibrium state along time, which follows a fixed-point equation. Then by viewing the forward computation of feedback SNNs as a black-box solver for this equation, and leveraging the implicit differentiation on the equation, we can compute the gradient for parameters without considering the exact forward procedure. In this way, the forward and backward procedures are decoupled and therefore the problem of non-differentiable spiking functions is avoided. We also briefly discuss the biological plausibility of implicit differentiation, which only requires computing another equilibrium. Extensive experiments on MNIST, Fashion-MNIST, N-MNIST, CIFAR-10, and CIFAR-100 demonstrate the superior performance of our method for feedback models with fewer neurons and parameters in a small number of time steps. Our code is available at https://github.com/pkuxmq/IDE- FSNN.

A Forward Learning Algorithm for Neural Memory Ordinary Differential Equations

Nmode: Neural Memory Ordinary Differential Equation

A neuron-weighted learning algorithm and its hardware implementation in associative memories

Convolutional Neural Networks with Neural Memory Ordinary Differential Equations

A Bidirectional Feedforward Neural Network Architecture Using the Discretized Neural Memory Ordinary Differential Equation

The Integrated Forward-Forward Algorithm: Integrating Forward-Forward and Shallow Backpropagation With Local Losses

An Efficient Learning Algorithm for Direct Training Deep Spiking Neural Networks

The Cascaded Forward Algorithm for Neural Network Training

Emerging NeoHebbian Dynamics in Forward-Forward Learning: Implications for Neuromorphic Computing

Dendritic Neuron Model with Effective Learning Algorithms for Classification, Approximation, and Prediction

Memory Networks: Towards Fully Biologically Plausible Learning

Enhancing Robustness of Medical Image Segmentation Model with Neural Memory Ordinary Differential Equation.

Error Forward-Propagation: Reusing Feedforward Connections to Propagate Errors in Deep Learning

Improved Forward-Forward Contrastive Learning

Forward-Backward Stochastic Neural Networks: Deep Learning of High-dimensional Partial Differential Equations

Neural Memory Plasticity for Anomaly Detection

Training Feedback Spiking Neural Networks by Implicit Differentiation on the Equilibrium State

Do Residual Neural Networks discretize Neural Ordinary Differential Equations?

Hardware Implementation of Nmode on FPGA

A stable mapping of nmODE

Neural Delay Differential Equations: System Reconstruction and Image Classification