Abstract:Brain-inspired spiking neural networks (SNNs) are becoming a promising energy-efficient alternative to traditional artificial neural networks (ANNs). However, the performance gap between SNNs and ANNs has been a significant hindrance to deploying SNNs ubiquitously. To leverage the full potential of SNNs, in this paper we study the attention mechanisms, which can help human focus on important information. We present our idea of attention in SNNs with a multi-dimensional attention module, which infers attention weights along the temporal, channel, as well as spatial dimension separately or simultaneously. Based on the existing neuroscience theories, we exploit the attention weights to optimize membrane potentials, which in turn regulate the spiking response. Extensive experimental results on event-based action recognition and image classification datasets demonstrate that attention facilitates vanilla SNNs to achieve sparser spiking firing, better performance, and energy efficiency concurrently. In particular, we achieve top-1 accuracy of 75.92% and 77.08% on ImageNet-1 K with single/4-step Res-SNN-104, which are state-of-the-art results in SNNs. Compared with counterpart Res-ANN-104, the performance gap becomes -0.95/+0.21 percent and the energy efficiency is 31.8×/7.4×. To analyze the effectiveness of attention SNNs, we theoretically prove that the spiking degradation or the gradient vanishing, which usually holds in general SNNs, can be resolved by introducing the block dynamical isometry theory. We also analyze the efficiency of attention SNNs based on our proposed spiking response visualization method. Our work lights up SNN's potential as a general backbone to support various applications in the field of SNN research, with a great balance between effectiveness and energy efficiency.

ReSpike: Residual Frames-based Hybrid Spiking Neural Networks for Efficient Action Recognition

RSNN: Recurrent Spiking Neural Networks for Dynamic Spatial-Temporal Information Processing

Event-based Action Recognition Using Motion Information and Spiking Neural Networks

Spiking Neural Network Recognition Method Based on Dynamic Visual Motion Features

Hierarchical Spiking-Based Model for Efficient Image Classification with Enhanced Feature Extraction and Encoding.

SpikePoint: An Efficient Point-based Spiking Neural Network for Event Cameras Action Recognition

Spiking Neural Networks for event-based action recognition: A new task to understand their advantage

Spike-HAR++: an Energy-Efficient and Lightweight Parallel Spiking Transformer for Event-Based Human Action Recognition

Converting Artificial Neural Networks to Ultra-Low-Latency Spiking Neural Networks for Action Recognition

Spiking Deep Residual Networks.

Deep Spiking Neural Network for High-Accuracy and Energy-Efficient Face Action Unit Recognition.

Sparser spiking activity can be better: Feature Refine-and-Mask spiking neural network for event-based visual recognition

Toward Efficient Processing and Learning with Spikes: New Approaches for Multispike Learning

Spiking Neural Network as Adaptive Event Stream Slicer

SpikeAtConv: An Integrated Spiking-Convolutional Attention Architecture for Energy-Efficient Neuromorphic Vision Processing

Attention Spiking Neural Networks

Integer-Valued Training and Spike-Driven Inference Spiking Neural Network for High-performance and Energy-efficient Object Detection

Adaptive-SpikeNet: Event-based Optical Flow Estimation using Spiking Neural Networks with Learnable Neuronal Dynamics

A Study On the Effects of Pre-processing On Spatio-temporal Action Recognition Using Spiking Neural Networks Trained with STDP

Toward Scalable, Efficient, and Accurate Deep Spiking Neural Networks With Backward Residual Connections, Stochastic Softmax, and Hybridization