Abstract:Due to lack of data, overfitting ubiquitously exists in real-world applications of deep neural networks (DNNs). We propose advanced dropout, a model-free methodology, to mitigate overfitting and improve the performance of DNNs. The advanced dropout technique applies a model-free and easily implemented distribution with parametric prior, and adaptively adjusts dropout rate. Specifically, the distribution parameters are optimized by stochastic gradient variational Bayes in order to carry out an end-to-end training. We evaluate the effectiveness of the advanced dropout against nine dropout techniques on seven computer vision datasets (five small-scale datasets and two large-scale datasets) with various base models. The advanced dropout outperforms all the referred techniques on all the datasets. We further compare the effectiveness ratios and find that advanced dropout achieves the highest one on most cases. Next, we conduct a set of analysis of dropout rate characteristics, including convergence of the adaptive dropout rate, the learned distributions of dropout masks, and a comparison with dropout rate generation without an explicit distribution. In addition, the ability of overfitting prevention is evaluated and confirmed. Finally, we extend the application of the advanced dropout to uncertainty inference, network pruning, text classification, and regression. The proposed advanced dropout is also superior to the corresponding referred methods. Codes are available at https://github.com/PRIS-CV/AdvancedDropout .

Interpreting and Boosting Dropout from a Game-Theoretic View

Towards Understanding and Improving Dropout in Game Theory

Wordreg: Mitigating the Gap Between Training and Inference with Worst-Case Drop Regularization

Rethinking the Usage of Batch Normalization and Dropout in the Training of Deep Neural Networks

Stochastic Modified Equations and Dynamics of Dropout Algorithm

A Theoretically Grounded Application of Dropout in Recurrent Neural Networks

Implicit Regularization of Dropout

Guided Dropout: Improving Deep Networks Without Increased Computation

Advanced Dropout: A Model-free Methodology for Bayesian Dropout Optimization.

Investigating the Synergistic Effects of Dropout and Residual Connections on Language Model Training

Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning

Dropout Reduces Underfitting

Interpretability of Neural Networks Based on Game-theoretic Interactions

Functional Network: A Novel Framework for Interpretability of Deep Neural Networks

NFLB Dropout: Improve Generalization Ability by Dropping out the Best -A Biologically Inspired Adaptive Dropout Method for Unsupervised Learning.

Flipover outperforms dropout in deep learning

A Bayesian encourages dropout

Rademacher Dropout: an Adaptive Dropout for Deep Neural Network Via Optimizing Generalization Gap.

Self-Balanced Dropout

Adaptive Dropout Method Based on Biological Principles

R-Drop: Regularized Dropout for Neural Networks.