Reconsidering the energy efficiency of spiking neural networks

Zhanglu Yan,Zhenyu Bai,Weng-Fai Wong

2024-08-29

Abstract:Spiking neural networks (SNNs) are generally regarded as more energy-efficient because they do not use multiplications. However, most SNN works only consider the counting of additions to evaluate energy consumption, neglecting other overheads such as memory accesses and data movement operations. This oversight can lead to a misleading perception of efficiency, especially when state-of-the-art SNN accelerators operate with very small time window sizes. In this paper, we present a detailed comparison of the energy consumption of artificial neural networks (ANNs) and SNNs from a hardware perspective. We provide accurate formulas for energy consumption based on classical multi-level memory hierarchy architectures, commonly used neuromorphic dataflow architectures, and our proposed improved spatial-dataflow architecture. Our research demonstrates that to achieve comparable accuracy and greater energy efficiency than ANNs, SNNs require strict limitations on both time window size T and sparsity s. For instance, with the VGG16 model and a fixed T of 6, the neuron sparsity rate must exceed 93% to ensure energy efficiency across most architectures. Inspired by our findings, we explore strategies to enhance energy efficiency by increasing sparsity. We introduce two regularization terms during training that constrain weights and activations, effectively boosting the sparsity rate. Our experiments on the CIFAR-10 dataset, using T of 6, show that our SNNs consume 69% of the energy used by optimized ANNs on spatial-dataflow architectures, while maintaining an SNN accuracy of 94.18%. This framework, developed using PyTorch, is publicly available for use and further research.

Neural and Evolutionary Computing,Artificial Intelligence,Machine Learning

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve

This paper aims to reassess the energy efficiency of Spiking Neural Networks (SNNs). Although SNNs are generally considered more energy-efficient than Artificial Neural Networks (ANNs) because they do not use multiplication operations, most studies only consider the number of addition operations to evaluate energy consumption, neglecting other overheads such as memory access and data movement operations. This neglect may lead to misunderstandings about efficiency, especially when state-of-the-art SNN accelerators use very small time windows.

Specifically, the paper compares the energy consumption of SNNs and ANNs in detail from a hardware perspective and proposes energy consumption formulas based on classical multi-level memory hierarchy architectures, commonly used neuromorphic dataflow architectures, and improved spatial dataflow architectures. The study shows that to achieve higher energy efficiency than ANNs with comparable accuracy, SNNs need to strictly limit the time window size \(T\) and sparsity \(s\). For example, when using the VGG16 model and fixing \(T=6\), the neuron sparsity rate must exceed 93% to ensure energy efficiency on most architectures.

Additionally, the authors explore strategies to improve energy efficiency by increasing sparsity, introducing two regularization terms to constrain weights and activations, thereby effectively enhancing the sparsity rate. Experimental results show that optimized SNNs, when using \(T=6\) on the CIFAR-10 dataset, consume only 69% of the energy of optimized ANNs on spatial dataflow architectures while maintaining 94.18% SNN accuracy.