Abstract:Graph neural network (GNN) has demonstrated superior performance on graph learning tasks. GNN captures the data dependencies via message passing amid neural networks. Hence the prediction of a node label can utilize information from its neighbors in a graph. Dropout is a regularization as well as an ensemble method for convolutional neural network (CNN), which has been carefully studied. However, there are few existing works that focused on dropout schemes for GNN. Although GNN and CNN share similar model architecture, both with convolutional layers and fully connected layers, the input data structure for GNN and CNN are different and convolution operation differs. This suggests the dropout schemes for CNN should not be directly applied to GNN without a good understanding of the impact. In this paper, we divide the existing dropout schemes for GNN into two categories: (1) dropout on feature maps and (2) dropout on graph structure. Based on the drawbacks of current GNN dropout models, we propose a novel layer compensation dropout and a novel adaptive heteroscadestic Gaussian dropout, which can be applied to any type of GNN models and outperforms their corresponding baselines in shallow GNNs. Then an experimental study shows Bernoulli dropout generalize better while Gaussian dropout is slightly stronger in transductive performance. At last, we theoretically study how different dropout schemes mitigate over-smoothing problems and experimental results shows that layer compensation dropout allows a GNN model to maintain or slightly improve its performance as the GNN model adds more layers while all the other dropout models suffer from performance degradation when GNN goes deep.

Dropout Rademacher Complexity of Deep Neural Networks.

Rademacher Dropout: an Adaptive Dropout for Deep Neural Network Via Optimizing Generalization Gap.

Stochastic Modified Equations and Dynamics of Dropout Algorithm

Rethinking the Usage of Batch Normalization and Dropout in the Training of Deep Neural Networks

Implicit Regularization of Dropout

Beyond Dropout: Feature Map Distortion to Regularize Deep Neural Networks

Dropout in Training Neural Networks: Flatness of Solution and Noise Structure

Dropout Training, Data-dependent Regularization, and Generalization Bounds.

Guided Dropout: Improving Deep Networks Without Increased Computation

Dropout: Explicit Forms and Capacity Control

Adaptive Sparse Dropout: Learning the Certainty and Uncertainty in Deep Neural Networks

Asymptotic Convergence Rate of Dropout on Shallow Linear Neural Networks

Dropout, a basic and effective regularization method for a deep learning model: a case study

R-Drop: Regularized Dropout for Neural Networks.

Towards Understanding and Improving Dropout in Game Theory

Effective and Efficient Dropout for Deep Convolutional Neural Networks

Approximate Random Dropout.

Survey of Dropout Methods for Deep Neural Networks

Understanding Dropout for Graph Neural Networks

Probing the Structure and Functional Properties of the Dropout-Induced Correlated Variability in Convolutional Neural Networks

Information Plane Analysis for Dropout Neural Networks