Abstract:Elements of neural networks, both biological and artificial, can be described by their selectivity for specific cognitive features. Understanding these features is important for understanding the inner workings of neural networks. For a living system, such as a neuron, whose response to a stimulus is unknown and not differentiable, the only way to reveal these features is through a feedback loop that exposes it to a large set of different stimuli. The properties of these stimuli should be varied iteratively in order to maximize the neuronal response. To utilize this feedback loop for a biological neural network, it is important to run it quickly and efficiently in order to reach the stimuli that maximizes certain neurons' activation with the least number of iterations possible. Here we present a framework with an efficient design for such a loop. We successfully tested it on an artificial spiking neural network (SNN), which is a model that simulates the asynchronous spiking activity of neurons in living brains. Our optimization method for activation maximization is based on the low-rank Tensor Train decomposition of the discrete activation function. The optimization space is the latent parameter space of images generated by SN-GAN or VQ-VAE generative models. To our knowledge, this is the first time that effective AM has been applied to SNNs. We track changes in the optimal stimuli for artificial neurons during training and show that highly selective neurons can form already in the early epochs of training and in the early layers of a convolutional spiking network. This formation of refined optimal stimuli is associated with an increase in classification accuracy. Some neurons, especially in the deeper layers, may gradually change the concepts they are selective for during learning, potentially explaining their importance for model performance.

Fast gradient-free activation maximization for neurons in spiking neural networks

Efficient Spiking Neural Networks with Sparse Selective Activation for Continual Learning

Gradient-free activation maximization for identifying effective stimuli

Adaptive Multi-Level Firing for Direct Training Deep Spiking Neural Networks

An Efficient Learning Algorithm for Direct Training Deep Spiking Neural Networks

IM-LIF: Improved Neuronal Dynamics with Attention Mechanism for Direct Training Deep Spiking Neural Network

Optimal ANN-SNN Conversion for Fast and Accurate Inference in Deep Spiking Neural Networks

Hierarchical Spiking-Based Model for Efficient Image Classification with Enhanced Feature Extraction and Encoding.

Toward Scalable, Efficient, and Accurate Deep Spiking Neural Networks With Backward Residual Connections, Stochastic Softmax, and Hybridization

Attention Spiking Neural Networks

Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting

High-performance deep spiking neural networks with 0.3 spikes per neuron

Optimized Potential Initialization for Low-latency Spiking Neural Networks

Leaky ReLUs That Differ in Forward and Backward Pass Facilitate Activation Maximization in Deep Neural Networks

High-parallelism Inception-like Spiking Neural Networks for Unsupervised Feature Learning

Toward High-Accuracy and Low-Latency Spiking Neural Networks With Two-Stage Optimization

Training a General Spiking Neural Network with Improved Efficiency and Minimum Latency

Activate or Not: Learning Customized Activation

Towards Energy-Efficient, Low-Latency and Accurate Spiking LSTMs

A Unified Optimization Framework of ANN-SNN Conversion: Towards Optimal Mapping from Activation Values to Firing Rates.