Abstract:Spiking Neural Networks (SNN). SNNs are based on a more biologically inspired approach than usual artificial neural networks. Such models are characterized by complex dynamics between neurons and spikes. These are very sensitive to the hyperparameters, making their optimization challenging. To tackle hyperparameter optimization of SNNs, we initially extended the signal loss issue of SNNs to what we call silent networks. These networks fail to emit enough spikes at their outputs due to mistuned hyperparameters or architecture. Generally, search spaces are heavily restrained, sometimes even discretized, to prevent the sampling of such networks. By defining an early stopping criterion detecting silent networks and by designing specific constraints, we were able to instantiate larger and more flexible search spaces. We applied a constrained Bayesian optimization technique, which was asynchronously parallelized, as the evaluation time of a SNN is highly stochastic. Large-scale experiments were carried-out on a multi-GPU Petascale architecture. By leveraging silent networks, results show an acceleration of the search, while maintaining good performances of both the optimization algorithm and the best solution obtained. We were able to apply our methodology to two popular training algorithms, known as spike timing dependent plasticity and surrogate gradient. Early detection allowed us to prevent worthless and costly computation, directing the search toward promising hyperparameter combinations. Our methodology could be applied to multi-objective problems, where the spiking activity is often minimized to reduce the energy consumption. In this scenario, it becomes essential to find the delicate frontier between low-spiking and silent networks. Finally, our approach may have implications for neural architecture search, particularly in defining suitable spiking architectures.

Hyper-parameter selection in deep neural networks using parallel particle swarm optimization

Particle swarm optimization-based automatic parameter selection for deep neural networks and its applications in large-scale and high-dimensional data

Parameters Optimization of Deep Learning Models using Particle Swarm Optimization

A Method of Parameter Optimization for Particle Swarm Optimization Based on Stochastic Processes

Enhancing Particle Swarm Optimization via probabilistic models

Efficient Hyperparameter Optimization for Convolution Neural Networks in Deep Learning: A Distributed Particle Swarm Optimization Approach

A Multimodal Improved Particle Swarm Optimization for High Dimensional Problems in Electromagnetic Devices

Comparison of Bayesian and particle swarm algorithms for hyperparameter optimisation in machine learning applications in high energy physics

Parallel Hyperparameter Optimization Of Spiking Neural Network

A Particle Swarm Optimization Method Applied to Global Optimization of Inverse Problem

Particle swarm optimization of deep neural networks architectures for image classification

Research on hybrid strategy Particle Swarm Optimization algorithm and its applications

A Modified Particle Swarm Optimization with a Smart Particle for Inverse Problems in Electromagnetic Devices

Designing neural networks using hybrid particle swarm optimization

A Modified Particle Swarm Optimization Algorithm for Global Optimizations of Inverse Problems

Parallel hyperparameter optimization of spiking neural networks

Simplifying Particle Swarm Optimization

PSO-PINN: Physics-Informed Neural Networks Trained with Particle Swarm Optimization

A Survey on Parallel Particle Swarm Optimization Algorithms

Neural Architecture Search using Particle Swarm and Ant Colony Optimization

A K-means-based Multi-subpopulation Particle Swarm Optimization for Neural Network Ensemble