Abstract:Spiking neural networks (SNNs) are attracting widespread interest due to their biological plausibility, energy efficiency, and powerful spatiotemporal information representation ability. Given the critical role of attention mechanisms in enhancing neural network performance, the integration of SNNs and attention mechanisms exhibits tremendous potential to deliver energy-efficient and high-performance computing paradigms. In this article, we present a novel temporal-channel joint attention mechanism for SNNs, referred to as TCJA-SNN. The proposed TCJA-SNN framework can effectively assess the significance of spike sequence from both spatial and temporal dimensions. More specifically, our essential technical contribution lies on: 1) we employ the squeeze operation to compress the spike stream into an average matrix. Then, we leverage two local attention mechanisms based on efficient 1-D convolutions to facilitate comprehensive feature extraction at the temporal and channel levels independently and 2) we introduce the cross-convolutional fusion (CCF) layer as a novel approach to model the interdependencies between the temporal and channel scopes. This layer effectively breaks the independence of these two dimensions and enables the interaction between features. Experimental results demonstrate that the proposed TCJA-SNN outperforms the state-of-the-art (SOTA) on all standard static and neuromorphic datasets, including Fashion-MNIST, CIFAR10, CIFAR100, CIFAR10-DVS, N-Caltech 101, and DVS128 Gesture. Furthermore, we effectively apply the TCJA-SNN framework to image generation tasks by leveraging a variation autoencoder. To the best of our knowledge, this study is the first instance where the SNN-attention mechanism has been employed for high-level classification and low-level generation tasks. Our implementation codes are available at https://github.com/ridgerchu/TCJA.

Spiking Transformer with Spatial-Temporal Attention

Spatial-Temporal Self-Attention for Asynchronous Spiking Neural Networks

SpikingResformer: Bridging ResNet and Vision Transformer in Spiking Neural Networks

Spikformer: When Spiking Neural Network Meets Transformer

Combining Aggregated Attention and Transformer Architecture for Accurate and Efficient Performance of Spiking Neural Networks

Spikeformer: Training high-performance spiking neural network with transformer

DS2TA: Denoising Spiking Transformer with Attenuated Spatiotemporal Attention

A Spatial–Channel–Temporal-Fused Attention for Spiking Neural Networks

TE-Spikformer:Temporal-enhanced spiking neural network with transformer

SpikeGraphormer: A High-Performance Graph Transformer with Spiking Graph Attention

Stochastic Spiking Attention: Accelerating Attention with Stochastic Computing in Spiking Networks

STCA-SNN: Self-Attention-based Temporal-Channel Joint Attention for Spiking Neural Networks

Spiking Transformers for Event-based Single Object Tracking

Scaling Spike-driven Transformer with Efficient Spike Firing Approximation Training

Spatial-Temporal Spiking Feature Pruning in Spiking Transformer

Spiking Wavelet Transformer

TCJA-SNN: Temporal-Channel Joint Attention for Spiking Neural Networks

Spiking Neural Networks with Dynamic Time Steps for Vision Transformers

Attention Spiking Neural Networks

QKFormer: Hierarchical Spiking Transformer using Q-K Attention