Abstract:Brain-inspired spiking neural networks (SNNs) are becoming a promising energy-efficient alternative to traditional artificial neural networks (ANNs). However, the performance gap between SNNs and ANNs has been a significant hindrance to deploying SNNs ubiquitously. To leverage the full potential of SNNs, in this paper we study the attention mechanisms, which can help human focus on important information. We present our idea of attention in SNNs with a multi-dimensional attention module, which infers attention weights along the temporal, channel, as well as spatial dimension separately or simultaneously. Based on the existing neuroscience theories, we exploit the attention weights to optimize membrane potentials, which in turn regulate the spiking response. Extensive experimental results on event-based action recognition and image classification datasets demonstrate that attention facilitates vanilla SNNs to achieve sparser spiking firing, better performance, and energy efficiency concurrently. In particular, we achieve top-1 accuracy of 75.92% and 77.08% on ImageNet-1 K with single/4-step Res-SNN-104, which are state-of-the-art results in SNNs. Compared with counterpart Res-ANN-104, the performance gap becomes -0.95/+0.21 percent and the energy efficiency is 31.8×/7.4×. To analyze the effectiveness of attention SNNs, we theoretically prove that the spiking degradation or the gradient vanishing, which usually holds in general SNNs, can be resolved by introducing the block dynamical isometry theory. We also analyze the efficiency of attention SNNs based on our proposed spiking response visualization method. Our work lights up SNN's potential as a general backbone to support various applications in the field of SNN research, with a great balance between effectiveness and energy efficiency.

Spiking generative adversarial network with attention scoring decoding

Spiking Generative Adversarial Network with Attention Scoring Decoding

Temporal Spiking Generative Adversarial Networks for Heading Direction Decoding

Gated Attention Coding for Training High-performance and Efficient Spiking Neural Networks

Spiking-GAN: A Spiking Generative Adversarial Network Using Time-To-First-Spike Coding

Spiking Generative Adversarial Networks With a Neural Network Discriminator: Local Training, Bayesian Models, and Continual Meta-Learning

Hierarchical Spiking-Based Model for Efficient Image Classification with Enhanced Feature Extraction and Encoding.

When Spiking neural networks meet temporal attention image decoding and adaptive spiking neuron

An Efficient Learning Algorithm for Direct Training Deep Spiking Neural Networks

Distributed Spatio-Temporal Generative Adversarial Networks

Adaptive Multi-Level Firing for Direct Training Deep Spiking Neural Networks

Spiking generative networks empowered by multiple dynamic experts for lifelong learning

Spiking Diffusion Models

Evolving Efficient Genetic Encoding for Deep Spiking Neural Networks

Neuromorphic Data Augmentation for Training Spiking Neural Networks

A Spatial–Channel–Temporal-Fused Attention for Spiking Neural Networks

Synthetic-Neuroscore: Using A Neuro-AI Interface for Evaluating Generative Adversarial Networks

Attention Spiking Neural Networks

Spiking-Diffusion: Vector Quantized Discrete Diffusion Model with Spiking Neural Networks

Decoding Spiking Mechanism with Dynamic Learning on Neuron Population

SpikeGraphormer: A High-Performance Graph Transformer with Spiking Graph Attention