Abstract:Spiking neural networks (SNNs), as the brain-inspired neural networks, encode information in spatio-temporal dynamics. They have the potential to serve as low-power alternatives to artificial neural networks (ANNs) due to their sparse and event-driven nature. However, existing SNN-based models for pixel-level semantic segmentation tasks suffer from poor performance and high memory overhead, failing to fully exploit the computational effectiveness and efficiency of SNNs. To address these challenges, we propose the multi-scale and full spike segmentation network (MFS-Seg), which is based on the deep direct trained SNN and represents the first attempt to train a deep SNN with surrogate gradients for semantic segmentation. Specifically, we design an efficient fully-spike residual block (EFS-Res) to alleviate representation issues caused by spiking noise on different channels. EFS-Res utilizes depthwise separable convolution to improve the distributions of spiking feature maps. The visualization shows that our model can effectively extract the edge features of segmented objects. Furthermore, it can significantly reduce the memory overhead and energy consumption of the network. In addition, we theoretically analyze and prove that EFS-Res can avoid the degradation problem based on block dynamical isometry theory. Experimental results on the Camvid dataset, the DDD17 dataset, and the DSEC-Semantic dataset show that our model achieves comparable performance to the mainstream UNet network with up to 31× fewer parameters, while significantly reducing power consumption by over 13×. Overall, our MFS-Seg model demonstrates promising results in terms of performance, memory efficiency, and energy consumption, showcasing the potential of deep SNNs for semantic segmentation tasks. Our code is available in https://github.com/BICLab/MFS-Seg.

Energy-Efficient Spiking Segmenter for Frame and Event-Based Images.

Spike-BRGNet: Efficient and Accurate Event-based Semantic Segmentation with Boundary Region-Guided Spiking Neural Networks

RSNN: Recurrent Spiking Neural Networks for Dynamic Spatial-Temporal Information Processing

BitSNNs: Revisiting Energy-efficient Spiking Neural Networks

Hierarchical Spiking-Based Model for Efficient Image Classification with Enhanced Feature Extraction and Encoding.

A Spiking Neural Network for Image Segmentation

Beyond Classification: Directly Training Spiking Neural Networks for Semantic Segmentation

Efficient Event-based Semantic Segmentation with Spike-driven Lightweight Transformer-based Networks

A 67.5μJ/Prediction Accelerator for Spiking Neural Networks in Image Segmentation

Multi-scale Full Spike Pattern for Semantic Segmentation

Accurate and Efficient Event-based Semantic Segmentation Using Adaptive Spiking Encoder-Decoder Network

EvSegSNN: Neuromorphic Semantic Segmentation for Event Data

Spike2Former: Efficient Spiking Transformer for High-performance Image Segmentation

Spiking neural networks fine-tuning for brain image segmentation

CSNN: an Augmented Spiking Based Framework with Perceptron-Inception

Spiking-NeRF: Spiking Neural Network for Energy-Efficient Neural Rendering

PSSD-Transformer: Powerful Sparse Spike-Driven Transformer for Image Semantic Segmentation

Training a General Spiking Neural Network with Improved Efficiency and Minimum Latency

ReSpike: Residual Frames-based Hybrid Spiking Neural Networks for Efficient Action Recognition

Reconsidering the energy efficiency of spiking neural networks

A Scatter-and-Gather Spiking Convolutional Neural Network on a Reconfigurable Neuromorphic Hardware