Abstract:In this paper, we propose a new automatic hyperparameter selection approach for determining the optimal network configuration (network structure and hyperparameters) for deep neural networks using particle swarm optimization (PSO) in combination with a steepest gradient descent algorithm. In the proposed approach, network configurations were coded as a set of real-number m-dimensional vectors as the individuals of the PSO algorithm in the search procedure. During the search procedure, the PSO algorithm is employed to search for optimal network configurations via the particles moving in a finite search space, and the steepest gradient descent algorithm is used to train the DNN classifier with a few training epochs (to find a local optimal solution) during the population evaluation of PSO. After the optimization scheme, the steepest gradient descent algorithm is performed with more epochs and the final solutions (pbest and gbest) of the PSO algorithm to train a final ensemble model and individual DNN classifiers, respectively. The local search ability of the steepest gradient descent algorithm and the global search capabilities of the PSO algorithm are exploited to determine an optimal solution that is close to the global optimum. We constructed several experiments on hand-written characters and biological activity prediction datasets to show that the DNN classifiers trained by the network configurations expressed by the final solutions of the PSO algorithm, employed to construct an ensemble model and individual classifier, outperform the random approach in terms of the generalization performance. Therefore, the proposed approach can be regarded an alternative tool for automatic network structure and parameter selection for deep neural networks.

Feature selection with particle swarms

Dimensionality reduction for evolving RBF networks with particle swarms

A Novel Feature Selection Method Based on Extreme Learning Machine and Fractional-Order Darwinian PSO.

Feature Selection Using Enhanced Particle Swarm Optimisation for Classification Models

Feature selection using feature ranking, correlation analysis and chaotic binary particle swarm optimization

Feature selection using bare-bones particle swarm optimization with mutual information

Training RBF neural networks with PSO and improved subtractive clustering algorithms

Feature Selection Method With Proportionate Fitness Based Binary Particle Swarm Optimization

Training Radial Basis Function Networks with Particle Swarms

A Novel Feature-Selection Approach Based on Particle Swarm Optimization Algorithm for Intrusion Detection Systems (Workshop Paper)

Feature subset selection via an improved discretization-based particle swarm optimization

An evolutionary feature selection method based on probability-based initialized particle swarm optimization

A particle swarm optimisation based multi-objective filter approach to feature selection for classification

Evolving RBF neural networks for pattern classification

A Multi-Objective Particle Swarm Optimisation for Filter-Based Feature Selection in Classification Problems

Feature Selection Based on Adaptive Particle Swarm Optimization with Leadership Learning

Hybrid particle swarm optimization algorithm for text feature selection problems

Information gain ratio-based subfeature grouping empowers particle swarm optimization for feature selection

Improving the Particle Swarm Algorithm and Optimizing the Network Intrusion Detection of Neural Network

Particle swarm optimization-based automatic parameter selection for deep neural networks and its applications in large-scale and high-dimensional data

Hybrid firefly particle swarm optimisation algorithm for feature selection problems