Abstract:Elements of neural networks, both biological and artificial, can be described by their selectivity for specific cognitive features. Understanding these features is important for understanding the inner workings of neural networks. For a living system, such as a neuron, whose response to a stimulus is unknown and not differentiable, the only way to reveal these features is through a feedback loop that exposes it to a large set of different stimuli. The properties of these stimuli should be varied iteratively in order to maximize the neuronal response. To utilize this feedback loop for a biological neural network, it is important to run it quickly and efficiently in order to reach the stimuli that maximizes certain neurons' activation with the least number of iterations possible. Here we present a framework with an efficient design for such a loop. We successfully tested it on an artificial spiking neural network (SNN), which is a model that simulates the asynchronous spiking activity of neurons in living brains. Our optimization method for activation maximization is based on the low-rank Tensor Train decomposition of the discrete activation function. The optimization space is the latent parameter space of images generated by SN-GAN or VQ-VAE generative models. To our knowledge, this is the first time that effective AM has been applied to SNNs. We track changes in the optimal stimuli for artificial neurons during training and show that highly selective neurons can form already in the early epochs of training and in the early layers of a convolutional spiking network. This formation of refined optimal stimuli is associated with an increase in classification accuracy. Some neurons, especially in the deeper layers, may gradually change the concepts they are selective for during learning, potentially explaining their importance for model performance.

Evolving activation dynamics in feedforward neural networks

Importance of Optimizing Neural Activation Function Types

Neuroevolution of Recurrent Architectures on Control Tasks

Evolving neural networks to follow trajectories of arbitrary complexity

Tracking Functional Changes in Nonstationary Signals with Evolutionary Ensemble Bayesian Model for Robust Neural Decoding

Normalized Activation Function: Toward Better Convergence

Adaptive Activation Functions for Predictive Modeling with Sparse Experimental Data

Single-unit activations confer inductive biases for emergent circuit solutions to cognitive tasks

An Artificial Neural Network Functionalized by Evolution

Brain-Inspired Constructive Learning Algorithms with Evolutionally Additive Nonlinear Neurons

An overview of the activation functions used in deep learning algorithms

Fast gradient-free activation maximization for neurons in spiking neural networks

Evolution of Novel Activation Functions in Neural Network Training with Applications to Classification of Exoplanets

How important are activation functions in regression and classification? A survey, performance comparison, and future directions

Evolving Generalized Modulatory Learning: Unifying Neuromodulation and Synaptic Plasticity

Evolving Transferable Neural Pruning Functions

Controllability, Multiplexing, and Transfer Learning in Networks using Evolutionary Learning

First Steps Towards a Runtime Analysis of Neuroevolution

Coupled Hebbian learning and evolutionary dynamics in a formal model for structural synaptic plasticity

Improving the performance of mutation-based evolving artificial neural networks with self-adaptive mutations

Activation function optimization scheme for image classification