Abstract:Convolutional Neural networks (CNNs) based applications have become ubiquitous, where proper regularization is greatly needed. To prevent large neural network models from overfitting, dropout has been widely used as an efficient regularization technique in practice. However, many recent works show that the standard dropout is ineffective or even detrimental to the training of CNNs. In this paper, we revisit this issue and examine various dropout variants in an attempt to improve existing dropout-based regularization techniques for CNNs. We attribute the failure of standard dropout to the conflict between the stochasticity of dropout and its following Batch Normalization (BN), and propose to reduce the conflict by placing dropout operations right before the convolutional operation instead of BN, or totally address this issue by replacing BN with Group Normalization (GN). We further introduce a structurally more suited dropout variant Drop-Conv2d, which provides more efficient and effective regularization for deep CNNs. These dropout variants can be readily integrated into the building blocks of CNNs and implemented in existing deep learning platforms. Extensive experiments on benchmark datasets including CIFAR, SVHN and ImageNet are conducted to compare the existing building blocks and the proposed ones with dropout training. Results show that our building blocks improve over state-of-the-art CNNs significantly, which is mainly due to the better regularization and implicit model ensemble effect.

Effective and Efficient Dropout for Deep Convolutional Neural Networks

Rethinking the Usage of Batch Normalization and Dropout in the Training of Deep Neural Networks

Continuous Dropout

Wordreg: Mitigating the Gap Between Training and Inference with Worst-Case Drop Regularization

Shakeout: A New Approach to Regularized Deep Neural Network Training

Attentiondrop For Convolutional Neural Networks

Correlation-based Structural Dropout for Convolutional Neural Networks

Dropout Reduces Underfitting

Dropout, a basic and effective regularization method for a deep learning model: a case study

LocalDrop: A Hybrid Regularization for Deep Neural Networks

Towards Dropout Training for Convolutional Neural Networks

TargetDrop: A Targeted Regularization Method for Convolutional Neural Networks

R-Drop: Regularized Dropout for Neural Networks.

Flip-Rotate-Pooling Convolution and Split Dropout on Convolution Neural Networks for Image Classification

Survey of Dropout Methods for Deep Neural Networks

Convolutional Neural Networks With Dynamic Regularization

R-Block: Regularized Block of Dropout for convolutional networks

Regularizing neural networks with adaptive local drop

Nesterov Accelerated Gradient Descent-Based Convolution Neural Network with Dropout for Facial Expression Recognition.

How to Use Dropout Correctly on Residual Networks with Batch Normalization

Applying Monte Carlo Dropout to Quantify the Uncertainty of Skip Connection-Based Convolutional Neural Networks Optimized by Big Data