Abstract:Randomness is widely introduced in neural network training to simplify model optimization or avoid the over-fitting problem. Among them, dropout and its variations in different aspects (e.g., data, model structure) are prevalent in regularizing the training of deep neural networks. Though effective and performing well, the randomness introduced by these dropout-based methods causes nonnegligible inconsistency between training and inference. In this paper, we introduce a simple consistency training strategy to regularize such randomness, namely R-Drop, which forces two output distributions sampled by each type of randomness to be consistent. Specifically, R-Drop minimizes the bidirectional KL-divergence between two output distributions produced by dropout-based randomness for each training sample. Theoretical analysis reveals that R-Drop can reduce the above inconsistency by reducing the inconsistency among the sampled sub structures and bridging the gap between the loss calculated by the full model and sub structures. Experiments on 7 widely-used deep learning tasks ( 23 datasets in total) demonstrate that R-Drop is universally effective for different types of neural networks (i.e., feed-forward, recurrent, and graph neural networks) and different learning paradigms (supervised, parameter-efficient, and semi-supervised). In particular, it achieves state-of-the-art performances with the vanilla Transformer model on WMT14 English → German translation ( 30.91 BLEU) and WMT14 English → French translation ( 43.95 BLEU), even surpassing models trained with extra large-scale data and expert-designed advanced variants of Transformer models. Our code is available at GitHub https://github.com/dropreg/R-Drop.

Energy-Based Dropout in Restricted Boltzmann Machines: Why Not Go Random

Hyperparameters Adaptation for Restricted Boltzmann Machines Based on Free Energy

R-Drop: Regularized Dropout for Neural Networks.

Randomness Regularization with Simple Consistency Training for Neural Networks

Excitation Dropout: Encouraging Plasticity in Deep Neural Networks

Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning

Approximate Random Dropout.

Partitioning Large Scale Deep Belief Networks Using Dropout

Stochastic Modified Equations and Dynamics of Dropout Algorithm

Continuous Dropout

Y-Drop: A Conductance based Dropout for fully connected layers

A Theoretically Grounded Application of Dropout in Recurrent Neural Networks

Survey of Dropout Methods for Deep Neural Networks

Dropout neuronal unit with tunable probability based on NbO<sub>x</sub> stochastic memristor for efficient suppression of overfitting

Dropout, a basic and effective regularization method for a deep learning model: a case study

A random energy approach to deep learning

Dropout neuronal unit with tunable probability based on NbO

Deep Network Regularization via Bayesian Inference of Synaptic Connectivity

Properties and Bayesian fitting of restricted Boltzmann machines

AutoDropout: Learning Dropout Patterns to Regularize Deep Networks

One Memristor-One Electrolyte-Gated Transistor based High Energy-Efficient Dropout Neuronal Units