Stochastic Spiking Neural Networks with First-to-Spike Coding

Yi Jiang,Sen Lu,Abhronil Sengupta

2024-07-02

Abstract:Spiking Neural Networks (SNNs), recognized as the third generation of neural networks, are known for their bio-plausibility and energy efficiency, especially when implemented on neuromorphic hardware. However, the majority of existing studies on SNNs have concentrated on deterministic neurons with rate coding, a method that incurs substantial computational overhead due to lengthy information integration times and fails to fully harness the brain's probabilistic inference capabilities and temporal dynamics. In this work, we explore the merger of novel computing and information encoding schemes in SNN architectures where we integrate stochastic spiking neuron models with temporal coding techniques. Through extensive benchmarking with other deterministic SNNs and rate-based coding, we investigate the tradeoffs of our proposal in terms of accuracy, inference latency, spiking sparsity, energy consumption, and robustness. Our work is the first to extend the scalability of direct training approaches of stochastic SNNs with temporal encoding to VGG architectures and beyond-MNIST datasets.

Neural and Evolutionary Computing

What problem does this paper attempt to address?

This paper primarily explores how to combine randomness and time encoding in Spiking Neural Networks (SNNs) to improve efficiency and performance. Traditional SNNs typically use deterministic neurons and rate coding, but this approach has issues such as high computational overhead and long information integration time, and also fails to fully utilize the probabilistic reasoning and temporal dynamics of the brain. The researchers propose a new computing and information encoding method that combines stochastic spiking neuron models with First-to-Spike Coding to reduce latency, improve energy efficiency, and enhance robustness.

The main contributions of the paper include:
1. The proposal of an algorithm framework for directly training random SNNs with First-to-Spike encoding, as well as a deterministic SNN training framework as a baseline for comparison.
2. The demonstration of direct training of deep networks such as the VGG architecture on MNIST and CIFAR-10 datasets, which is the first time that the VGG architecture of SNNs trained with First-to-Spike encoding has been applied to the CIFAR dataset.
3. A comprehensive quantitative analysis of random time encoding SNNs, exploring the trade-offs between accuracy, latency, sparsity, energy efficiency, and robustness.

The paper also compares different encoding methods (such as Time-to-First-Spike coding) and training strategies (such as ANN-SNN conversion and direct SNN training), and highlights the potential of direct training of random SNNs with time encoding on complex datasets, particularly in reducing latency. However, the current work mainly focuses on shallow networks and simple datasets, and extending it to deep networks and complex datasets still poses challenges.

In the experimental section, the paper demonstrates the performance of different models (including conversion methods and direct training methods) on MNIST and CIFAR-10 datasets, including metrics such as accuracy, latency, energy cost, and sparsity. It is found that randomness can improve the accuracy of complex datasets, while First-to-Spike encoding can significantly reduce latency. Although random SNNs have higher activation rates in some layers, they still achieve higher energy efficiency overall.