Abstract:Spiking neural networks (SNNs), which are a form of neuromorphic, brain-inspired AI, have the potential to be a power-efficient alternative to artificial neural networks (ANNs). Spikes that occur in SNN systems, also known as activations, tend to be extremely sparse, and low in number. This minimizes the number of data accesses typically needed for processing. In addition, SNN systems are typically designed to use addition operations which consume much less energy than the typical multiply and accumulate operations used in DNN systems. The vast majority of neuromorphic hardware designs support rate-based SNNs, where the information is encoded by spike rates. Generally, rate-based SNNs can be inefficient as a large number of spikes will be transmitted and processed during inference. One coding scheme that has the potential to improve efficiency is the time-to-first-spike (TTFS) coding, where the information isn't presented through the frequency of spikes, but instead through the relative spike arrival time. In TTFS-based SNNs, each neuron can only spike once during the entire inference process, and this results in high sparsity. The activation sparsity of TTFS-based SNNs is higher than rate-based SNNs, but TTFS-based SNNs have yet to achieve the same accuracy as rate-based SNNs. In this work, we propose two key improvements for TTFS-based SNN systems: (1) a novel optimization algorithm to improve the accuracy of TTFS-based SNNs and (2) a novel hardware accelerator for TTFS-based SNNs that uses a scalable and low-power design. Our work in TTFS coding and training improves the accuracy of TTFS-based SNNs to achieve state-of-the-art results on the MNIST and Fashion-MNIST datasets. Meanwhile, our work reduces the power consumption by at least 2.4×, 25.9×, and 38.4× over the state-of-the-art neuromorphic hardware on MNIST, Fashion-MNIST, and CIFAR10, respectively.

AT-SNN: Adaptive Tokens for Vision Transformer on Spiking Neural Network

Spiking Neural Networks with Dynamic Time Steps for Vision Transformers

Making Vision Transformers Efficient from A Token Sparsification View

Spikeformer: Training high-performance spiking neural network with transformer

SPVT: Spiked Pyramid Vision Transformer

Scaling Spike-driven Transformer with Efficient Spike Firing Approximation Training

Spiking Transformer with Spatial-Temporal Attention

Twin Network Augmentation: A Novel Training Strategy for Improved Spiking Neural Networks and Efficient Weight Quantization

Toward High-Accuracy and Low-Latency Spiking Neural Networks With Two-Stage Optimization

SpikingResformer: Bridging ResNet and Vision Transformer in Spiking Neural Networks

VTSNN: a virtual temporal spiking neural network

Towards Energy-Efficient, Low-Latency and Accurate Spiking LSTMs

Adaptive Sparse ViT: Towards Learnable Adaptive Token Pruning by Fully Exploiting Self-Attention

DS2TA: Denoising Spiking Transformer with Attenuated Spatiotemporal Attention

ATC: Approximate Temporal Coding for Efficient Implementations of Spiking Neural Networks.

Spatial-Temporal Spiking Feature Pruning in Spiking Transformer

TT-SNN: Tensor Train Decomposition for Efficient Spiking Neural Network Training

A TTFS-based energy and utilization efficient neuromorphic CNN accelerator

DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification.

Adaptive Token Sampling For Efficient Vision Transformers

SkipViT: Speeding Up Vision Transformers with a Token-Level Skip Connection