Abstract:With large deep neural networks (DNNs) necessary to solve complex and data-intensive problems, energy efficiency is a key bottleneck for effectively deploying DL in the real world. Deep spiking NNs have gained much research attention recently due to the interest in building biological neural networks and the availability of neuromorphic platforms, which can be orders of magnitude more energy efficient compared to CPUs and GPUs. Although spiking NNs have proven to be an efficient technique for solving many machine learning and computer vision problems, to the best of our knowledge, this is the first attempt to adapt spiking NNs to sparse datasets. In this paper, we study the behaviour of spiking NNs in handling NLP datasets and the sparsity in their data representation. Then, we propose a novel framework for spiking NN using the concept of stochastic computing. Specifically, instead of generating spike trains with firing rates proportional to the intensity of each value in the feature set separately, the whole feature set is treated as a distribution function and a stochastic spiking train that follow this distribution is generated. This framework reduces the connectivity between NN layers from $\mathrm{O}(N)$ to $\mathrm{O}(\log N)$. Also, it encodes input data differently and make suitable to handle sparse datasets. Finally, the framework achieves high energy efficiency since it uses Integrate and Fire neurons same as conventional spiking NNs. The results show that our proposed stochastic-based SNN achieves nearly the same accuracy as the original DNN on MNIST dataset, and it has better performance than state-of-the-art SNN. Besides that stochastic-based SNN is energy efficient, where the fully connected DNN, the conventional SNN, and the data normalized SNN consume 38.24, 1.83, and 1.85-times more energy than the stochastic-based SNN, respectively. For sparse datasets, including IMDb and In-House clinical datasets, stochastic-based SNN achieves performance comparable to that of the conventional DNN. However, the conventional spiking NN has a significant decline in classification accuracy.

Spiking neural networks for nonlinear regression

BitSNNs: Revisiting Energy-efficient Spiking Neural Networks

Spiking recurrent neural networks for neuromorphic computing in nonlinear structural mechanics

Recent Advances in Scalable Energy-Efficient and Trustworthy Spiking Neural networks: from Algorithms to Technology

Spiking Deep Residual Networks.

Towards Energy-Preserving Natural Language Understanding with Spiking Neural Networks

You Only Spike Once: Improving Energy-Efficient Neuromorphic Inference to ANN-Level Accuracy

Optimizing Power Efficiency: Converting Recurrent Neural Networks to Spiking Neural Networks for Time-Domain Analysis

Efficient Spiking Neural Networks with Sparse Selective Activation for Continual Learning

Are SNNs Really More Energy-Efficient Than ANNs? an In-Depth Hardware-Aware Study

An Efficient Software-Hardware Design Framework for Spiking Neural Network Systems

Reconsidering the energy efficiency of spiking neural networks

30.2 A 22nm 0.26nW/Synapse Spike-Driven Spiking Neural Network Processing Unit Using Time-Step-First Dataflow and Sparsity-Adaptive In-Memory Computing

Energy-Efficient High-Accuracy Spiking Neural Network Inference Using Time-Domain Neurons

Efficient spiking neural network design via neural architecture search

Analysis of biologically plausible neuron models for regression with spiking neural networks

Training a General Spiking Neural Network with Improved Efficiency and Minimum Latency

PT-Spike: A Precise-Time-Dependent Single Spike Neuromorphic Architecture with Efficient Supervised Learning

SNNLP: Energy-Efficient Natural Language Processing Using Spiking Neural Networks

Energy efficient stochastic-based deep spiking neural networks for sparse datasets

Hybrid variable spiking graph neural networks for energy-efficient scientific machine learning