Abstract:Spiking Neural Networks (SNNs) have indeed shown remarkable promise in the field of computer vision, emerging as a low-energy alternative to traditional Artificial Neural Networks (ANNs). However, SNNs also face several challenges: i) Existing SNNs are not purely additive and involve a substantial amount of floating-point computations, which contradicts the original design intention of adapting to neuromorphic chips; ii) The incorrect positioning of convolutional and pooling layers relative to spiking layers leads to reduced accuracy; iii) Leaky Integrate-and-Fire (LIF) neurons have limited capability in representing local information, which is disadvantageous for downstream visual tasks like semantic segmentation. To address the challenges in SNNs, i) we introduce Pure Sparse Self Attention (PSSA) and Dynamic Spiking Membrane Shortcut (DSMS), combining them to tackle the issue of floating-point computations; ii) the Spiking Precise Gradient downsampling (SPG-down) method is proposed for accurate gradient transmission; iii) the Group-LIF neuron concept is introduced to ensure LIF neurons' capability in representing local information both horizontally and vertically, enhancing their applicability in semantic segmentation tasks. Ultimately, these three solutions are integrated into the Powerful Sparse-Spike-Driven Transformer (PSSD-Transformer), effectively handling semantic segmentation tasks and addressing the challenges inherent in SNNs. The experimental results demonstrate that our model outperforms previous results on standard classification datasets and also shows commendable performance on semantic segmentation datasets. Up to this point, PSSD is the first model in the SNN field to perform semantic segmentation on large datasets. The code will be made publicly available after the paper is accepted for publication.

Spatial-Temporal Spiking Feature Pruning in Spiking Transformer

Efficient Structure Slimming for Spiking Neural Networks

Loss Constrains Added Squeeze and Excitation Blocks for Pruning Deep Neural Networks

SparseSpikformer: A Co-Design Framework for Token and Weight Pruning in Spiking Transformer

Scaling Spike-driven Transformer with Efficient Spike Firing Approximation Training

An efficient pruning and fine-tuning method for deep spiking neural network

Towards Efficient Deep Spiking Neural Networks Construction with Spiking Activity based Pruning

An Attention-Based Token Pruning Method for Vision Transformers

QKFormer: Hierarchical Spiking Transformer using Q-K Attention

Spiking Transformer with Spatial-Temporal Attention

Towards Energy Efficient Spiking Neural Networks: an Unstructured Pruning Framework.

Brain-Inspired Efficient Pruning: Exploiting Criticality in Spiking Neural Networks

Spatio-Temporal Pruning and Quantization for Low-latency Spiking Neural Networks

Trimming Down Large Spiking Vision Transformers via Heterogeneous Quantization Search

SpikingMiniLM: Energy-Efficient Spiking Transformer for Natural Language Understanding

PSSD-Transformer: Powerful Sparse Spike-Driven Transformer for Image Semantic Segmentation

Spatio-Temporal Pruning for Training Ultra-Low-Latency Spiking Neural Networks in Remote Sensing Scene Classification

SpikingResformer: Bridging ResNet and Vision Transformer in Spiking Neural Networks

TE-Spikformer:Temporal-enhanced spiking neural network with transformer

Spikformer: When Spiking Neural Network Meets Transformer

Masked Spiking Transformer