Abstract:The choice of parameters, and the design of the network architecture are important factors affecting the performance of deep neural networks. Genetic Algorithms (GA) have been used before to determine parameters of a network. Yet, GAs perform a finite search over a discrete set of pre-defined candidates, and cannot, in general, generate unseen configurations. In this paper, to move from exploration to exploitation, we propose a novel and systematic method that autonomously and simultaneously optimizes multiple parameters of any deep neural network by using a GA aided by a bi-generative adversarial network (Bi-GAN). The proposed Bi-GAN allows the autonomous exploitation and choice of the number of neurons, for fully-connected layers, and number of filters, for convolutional layers, from a large range of values. Our proposed Bi-GAN involves two generators, and two different models compete and improve each other progressively with a GAN-based strategy to optimize the networks during GA evolution. Our proposed approach can be used to autonomously refine the number of convolutional layers and dense layers, number and size of kernels, and the number of neurons for the dense layers; choose the type of the activation function; and decide whether to use dropout and batch normalization or not, to improve the accuracy of different deep neural network architectures. Without loss of generality, the proposed method has been tested with the ModelNet database, and compared with the 3D Shapenets and two GA-only methods. The results show that the presented approach can simultaneously and successfully optimize multiple neural network parameters, and achieve higher accuracy even with shallower networks.

A Method of Adaptive Hyperparameter Optimization for Deep Generative Models

Pre-training the Deep Generative Models with Adaptive Hyperparameter Optimization

Hyperparameters Adaptation for Restricted Boltzmann Machines Based on Free Energy

Fast Model Selection and Hyperparameter Tuning for Generative Models

A Unified Gaussian Process for Branching and Nested Hyperparameter Optimization

Hyperparameter Optimization for Machine Learning Models Based on Bayesian Optimization

Optimal Designs of Gaussian Processes with Budgets for Hyperparameter Optimization

Adaptive Optimizer for Automated Hyperparameter Optimization Problem

Autonomously and Simultaneously Refining Deep Neural Network Parameters by a Bi-Generative Adversarial Network Aided Genetic Algorithm

Learning Deep Generative Models with Doubly Stochastic Gradient MCMC

A Generic Approach for Enhancing GANs by Regularized Latent Optimization

Analyzing the Training Processes of Deep Generative Models.

Efficient Hyperparameter Optimization in Deep Learning Using a Variable Length Genetic Algorithm

Derivative-Free Optimization with Adaptive Experience for Efficient Hyper-Parameter Tuning.

Bayesian Hyperparameter Optimization with BoTorch, GPyTorch and Ax

DP-HyPO: An Adaptive Private Hyperparameter Optimization Framework

HyperDomainNet: Universal Domain Adaptation for Generative Adversarial Networks

An automatic hyperparameter optimization DNN model for precipitation prediction

Efficient Hyperparameter Optimization for Deep Learning Algorithms Using Deterministic RBF Surrogates

Hyperparameters Optimization in Deep Convolutional Neural Network / Bayesian Approach with Gaussian Process Prior

Further Analysis of Outlier Detection with Deep Generative Models.