A Comprehensive Review of Spiking Neural Networks: Interpretation, Optimization, Efficiency, and Best Practices

Kai Malcolm,Josue Casco-Rodriguez

2023-03-22

Abstract:Biological neural networks continue to inspire breakthroughs in neural network performance. And yet, one key area of neural computation that has been under-appreciated and under-investigated is biologically plausible, energy-efficient spiking neural networks, whose potential is especially attractive for low-power, mobile, or otherwise hardware-constrained settings. We present a literature review of recent developments in the interpretation, optimization, efficiency, and accuracy of spiking neural networks. Key contributions include identification, discussion, and comparison of cutting-edge methods in spiking neural network optimization, energy-efficiency, and evaluation, starting from first principles so as to be accessible to new practitioners.

Neural and Evolutionary Computing,Machine Learning,Image and Video Processing

What problem does this paper attempt to address?

### What problem does this paper attempt to solve?

The paper "Spiking Neural Networks: Interpretation, Optimization, Efficiency, and Best Practices" primarily addresses the following issues:

1. **Optimization Challenges**:
- It presents the main challenges faced by Spiking Neural Networks (SNN) during optimization, including how to perform gradient descent on non-differentiable spike actions and leveraging the intrinsic properties of neuromorphic computing to achieve more efficient SNN training methods.

2. **Biological Plausibility and Interpretability**:
- It demonstrates that SNN models are not only easy for humans to understand but also serve as a reasonable approximation of biological neural networks. The paper discusses two main types of information encoding: rate coding and time-to-first-spike (TTFS), analyzing their respective advantages and disadvantages.

3. **Review of Recent Advances**:
- It provides an overview of the latest advancements in SNN optimization, particularly regarding surrogate gradients. For example, the Differentiable Spike method proposed by Li et al., and the Information Maximization Loss (IM-Loss) and Evolutionary Surrogate Gradient (ESG) introduced by Guo et al.

4. **Sparsity Utilization**:
- Perez-Nieves and Goodman proposed a novel surrogate gradient approximation method that significantly improves computational efficiency and speed by leveraging the sparsity of spike sequences.

5. **Learning Rules**:
- It explores several common learning rules used for SNN training, such as Spike-Timing-Dependent Plasticity (STDP) and its extension R-STDP, as well as event-driven optimization.

In summary, this literature review aims to systematically summarize the key advancements in the field of SNNs, particularly in terms of optimization, energy efficiency, and biological plausibility, to provide researchers with a comprehensive understanding framework.