Abstract:Spiking Neural Networks (SNN). SNNs are based on a more biologically inspired approach than usual artificial neural networks. Such models are characterized by complex dynamics between neurons and spikes. These are very sensitive to the hyperparameters, making their optimization challenging. To tackle hyperparameter optimization of SNNs, we initially extended the signal loss issue of SNNs to what we call silent networks. These networks fail to emit enough spikes at their outputs due to mistuned hyperparameters or architecture. Generally, search spaces are heavily restrained, sometimes even discretized, to prevent the sampling of such networks. By defining an early stopping criterion detecting silent networks and by designing specific constraints, we were able to instantiate larger and more flexible search spaces. We applied a constrained Bayesian optimization technique, which was asynchronously parallelized, as the evaluation time of a SNN is highly stochastic. Large-scale experiments were carried-out on a multi-GPU Petascale architecture. By leveraging silent networks, results show an acceleration of the search, while maintaining good performances of both the optimization algorithm and the best solution obtained. We were able to apply our methodology to two popular training algorithms, known as spike timing dependent plasticity and surrogate gradient. Early detection allowed us to prevent worthless and costly computation, directing the search toward promising hyperparameter combinations. Our methodology could be applied to multi-objective problems, where the spiking activity is often minimized to reduce the energy consumption. In this scenario, it becomes essential to find the delicate frontier between low-spiking and silent networks. Finally, our approach may have implications for neural architecture search, particularly in defining suitable spiking architectures.

A Generalized Spiking Locally Competitive Algorithm for Multiple Optimization Problems

Toward Efficient Processing and Learning with Spikes: New Approaches for Multispike Learning

Neurodynamic Algorithms with Finite/Fixed-Time Convergence for Sparse Optimization Via ℓ1 Regularization

Implementing and Benchmarking the Locally Competitive Algorithm on the Loihi 2 Neuromorphic Processor

Adaptive Multi-Level Firing for Direct Training Deep Spiking Neural Networks

An Efficient Learning Algorithm for Direct Training Deep Spiking Neural Networks

Fast and Accurate Classification with a Multi-Spike Learning Algorithm for Spiking Neurons.

A Supervised Multi-Spike Learning Algorithm for Spiking Neural Networks

Efficient Sparse Coding with the Adaptive Locally Competitive Algorithm for Speech Classification

A 5.37-TSOPS/W Reconfigurable Neuron Array with Dual-mode Neurons and Asynchronous Synapses for Energy-Efficient Inference and Biological Neural Network Simulation

Combinatorial optimization solving by coherent Ising machines based on spiking neural networks

Surrogate-Assisted Evolutionary Search of Spiking Neural Architectures in Liquid State Machines

D-SELD: Dataset-Scalable Exemplar LCA-Decoder

Effective and Efficient Intracortical Brain Signal Decoding with Spiking Neural Networks

MT-Spike: A Multilayer Time-based Spiking Neuromorphic Architecture with Temporal Error Backpropagation

Parallel Hyperparameter Optimization Of Spiking Neural Network

Toward High-Accuracy and Low-Latency Spiking Neural Networks With Two-Stage Optimization

AutoML for neuromorphic computing and application-driven co-design: asynchronous, massively parallel optimization of spiking architectures

Spike-driven Multi-Scale Learning with Hybrid Mechanisms of Spiking Dendrites.

Improving Spiking Sparse Recovery via Non-Convex Penalties

Multilayer Photonic Spiking Neural Networks: Generalized Supervised Learning Algorithm and Network Optimization