Abstract:In this paper, we propose a new automatic hyperparameter selection approach for determining the optimal network configuration (network structure and hyperparameters) for deep neural networks using particle swarm optimization (PSO) in combination with a steepest gradient descent algorithm. In the proposed approach, network configurations were coded as a set of real-number m-dimensional vectors as the individuals of the PSO algorithm in the search procedure. During the search procedure, the PSO algorithm is employed to search for optimal network configurations via the particles moving in a finite search space, and the steepest gradient descent algorithm is used to train the DNN classifier with a few training epochs (to find a local optimal solution) during the population evaluation of PSO. After the optimization scheme, the steepest gradient descent algorithm is performed with more epochs and the final solutions (pbest and gbest) of the PSO algorithm to train a final ensemble model and individual DNN classifiers, respectively. The local search ability of the steepest gradient descent algorithm and the global search capabilities of the PSO algorithm are exploited to determine an optimal solution that is close to the global optimum. We constructed several experiments on hand-written characters and biological activity prediction datasets to show that the DNN classifiers trained by the network configurations expressed by the final solutions of the PSO algorithm, employed to construct an ensemble model and individual classifier, outperform the random approach in terms of the generalization performance. Therefore, the proposed approach can be regarded an alternative tool for automatic network structure and parameter selection for deep neural networks.

Using ensemble information in swarming artificial neural networks

Optimizing Weights by Genetic Algorithm for Neural Network Ensemble

Cooperative Bare-Bone Particle Swarm Optimization for Data Clustering

Artificial Neural Network Ensemble For Land Cover Classification

Neural Networks Learning Using Vbest Model Particle Swarm Optimisation

Enhancing Particle Swarm Optimization via probabilistic models

Swarm Intelligence Based Ensemble Learning of Deep Neural Networks

Using ensemble method to improve the performance of genetic algorithm

Designing neural networks using hybrid particle swarm optimization

A novel adaptive algorithm of particle swarm optimization based on the human social learning intelligence

Application of Artificial Neural Network and Particle Swarm Optimization in Car-Following Model

Dimensionality reduction for evolving RBF networks with particle swarms

Genetic Algorithm based Selective Neural Network Ensemble.

A K-means-based Multi-subpopulation Particle Swarm Optimization for Neural Network Ensemble

Particle swarm optimization-based automatic parameter selection for deep neural networks and its applications in large-scale and high-dimensional data

Support Vector Machines Ensemble With Optimizing Weights By Genetic Algorithm

An agent-assisted heterogeneous learning swarm optimizer for large-scale optimization

Training Radial Basis Function Networks with Particle Swarms

PSO-ANE: Adaptive Network Embedding with Particle Swarm Optimization

Negatively correlated neural network ensemble with multi-population particle swarm optimization

Ensembling Neural Networks: Many Could Be Better Than All