Abstract:Spiking neural networks (SNNs) can do inference with low power consumption due to their spike sparsity. Although SNNs can be combined with neuromorphic hardware to achieve efficient inference, they are often difficult to train directly due to discrete non-differentiable spikes. As an alternative, ANN-SNN conversion is an efficient way to achieve deep SNNs by converting well-trained artificial neural networks (ANNs). However, the existing methods commonly use constant threshold for conversion. A high constant threshold value prevents neurons from rapidly delivering spikes to deeper layers and causes high time delay. In addition, the same response for different inputs may result in information loss during the information transmission. Inspired by the biological adaptive threshold mechanism, we propose a multistage adaptive threshold (MSAT) method to alleviate this problem. Instead of using a single, constant value, the threshold is adjusted in multistages, adapting to each neuron’s firing history and input properties. Specifically, for each neuron, the dynamic threshold is positively correlated with the average membrane potential and negatively correlated with the rate of depolarization. The adaptation to membrane potential and input allows a timely adjustment of the threshold to fire spikes faster and transmit more information. Moreover, we analyze the spikes of inactivated neurons error, which is pervasive in early time steps. We also propose spike confidence accordingly to measure confidence about the neurons that correctly deliver spikes. Such spike confidence in early time steps is used to determine whether to elicit the spike to alleviate the spikes of inactivated neurons error. Combined with the proposed methods, we examine the performance on CIFAR-10, CIFAR-100, and ImageNet datasets. We also conduct sentiment classification and speech recognition experiments on the IDBM and Google speech commands datasets, respectively. Experiments show that our methods can achieve near-lossless and lower latency ANN-SNN conversion. In summary, we build a biologically inspired multistage adaptive threshold for converted SNN, with comparable performance to state-of-the-art methods while improving energy efficiency.

Multi-scale Adaptive Networks for Efficient Inference

Multi-Scale Adaptive Network for Single Image Denoising

Dynamic Multi-path Neural Network

Adaptive Multi-Level Firing for Direct Training Deep Spiking Neural Networks

Resolution Adaptive Networks for Efficient Inference

Multi-scale attention network for image super-resolution

Improved Techniques for Training Adaptive Deep Networks

Multi-Scale Fusion With Matching Attention Model: A Novel Decoding Network Cooperated With NAS for Real-Time Semantic Segmentation

Distilling Multi-Scale Neural Mechanisms from Diverse Unlabeled Experimental Data Using Deep Domain-Adaptive Inference Framework

Dynamic Deep Neural Network Inference Via Adaptive Channel Skipping.

Lightweight multi-scale distillation attention network for image super-resolution

Adaptive Deep Inference Framework for Cloud-Edge Collaboration

S2DNAS: Transforming Static CNN Model for Dynamic Inference via Neural Architecture Search

MSD: Multi-Self-Distillation Learning via Multi-classifiers within Deep Neural Networks

SCAN: A Scalable Neural Networks Framework Towards Compact and Efficient Models

MS3DAAM: Multi-scale 3-D Analytic Attention Module for Convolutional Neural Networks

MSAT: biologically inspired multistage adaptive threshold for conversion of spiking neural networks

Sample-Adaptive Classification Inference Network

MSAT: Biologically Inspired Multi-Stage Adaptive Threshold for Conversion of Spiking Neural Networks

Automatic Generation of Dynamic Inference Architecture for Deep Neural Networks

Self-Distillation: Towards Efficient and Compact Neural Networks