Abstract:Dropout is an effective method of mitigating over-fitting while training deep neural networks (DNNs). This method consists of switching off (dropping) some of the neurons of the DNN and training it by keeping the remaining neurons active. This approach makes the DNN general and resilient to changes in its inputs. However, the probability of a neuron belonging to a layer to be dropped, the 'dropout rate', is a hard-to-tune parameter that affects the performance of the trained model. Moreover, there is no reason, besides being more practical during parameter tuning, why the dropout rate should be the same for all neurons across a layer. This paper proposes a novel method to guide the dropout rate based on an evolutionary algorithm. In contrast to previous studies, we associate a dropout with each individual neuron of the network, thus allowing more flexibility in the training phase. The vector encoding the dropouts for the entire network is interpreted as the candidate solution of a bi-objective optimisation problem, where the first objective is the error reduction due to a set of dropout rates for a given data batch, while the second objective is the distance of the used dropout rates from a pre-arranged constant. The second objective is used to control the dropout rates and prevent them from becoming too small, hence ineffective; or too large, thereby dropping a too-large portion of the network. Experimental results show that the proposed method, namely GADropout, produces DNNs that consistently outperform DNNs designed by other dropout methods, some of them being modern advanced dropout methods representing the state-of-the-art. GADroput has been tested on multiple datasets and network architectures.

Adaptive Dropout Method Based on Biological Principles

Advanced Dropout: A Model-free Methodology for Bayesian Dropout Optimization.

Adaptive Sparse Dropout: Learning the Certainty and Uncertainty in Deep Neural Networks

Continuous Dropout

Continuously evolving dropout with multi-objective evolutionary optimisation

Dropout neuronal unit with tunable probability based on NbO

Excitation Dropout: Encouraging Plasticity in Deep Neural Networks

Self-Balanced Dropout

Guided Dropout: Improving Deep Networks Without Increased Computation

Dropout neuronal unit with tunable probability based on NbO<sub>x</sub> stochastic memristor for efficient suppression of overfitting

Surrogate Dropout: Learning Optimal Drop Rate Through Proxy.

Dropout, a basic and effective regularization method for a deep learning model: a case study

NFLB Dropout: Improve Generalization Ability by Dropping out the Best -A Biologically Inspired Adaptive Dropout Method for Unsupervised Learning.

Variational Bayesian Dropout With A Hierarchical Prior

Rethinking the Usage of Batch Normalization and Dropout in the Training of Deep Neural Networks

Rademacher Dropout: an Adaptive Dropout for Deep Neural Network Via Optimizing Generalization Gap.

&Lt;inline-Formula> &Lt;tex-Math Notation="latex">$\beta$ &Lt;/tex-Math> &Lt;/inline-Formula>-dropout: A Unified Dropout

Stochastic Modified Equations and Dynamics of Dropout Algorithm

A Theoretically Grounded Application of Dropout in Recurrent Neural Networks