Abstract:Deep learning models learn to fit training data while they are highly expected to generalize well to testing data. Most works aim at finding such models by creatively designing architectures and fine-tuning parameters. To adapt to particular tasks, hand-crafted information such as image prior has also been incorporated into end-to-end learning. However, very little progress has been made on investigating how an individual training sample will influence the generalization ability of a model. In other words, to achieve high generalization accuracy, do we really need all the samples in a training dataset? In this paper, we demonstrate that deep learning models such as convolutional neural networks may not favor all training samples, and generalization accuracy can be further improved by dropping those unfavorable samples. Specifically, the influence of removing a training sample is quantifiable, and we propose a Two-Round Training approach, aiming to achieve higher generalization accuracy. We locate unfavorable samples after the first round of training, and then retrain the model from scratch with the reduced training dataset in the second round. Since our approach is essentially different from fine-tuning or further training, the computational cost should not be a concern. Our extensive experimental results indicate that, with identical settings, the proposed approach can boost performance of the well-known networks on both high-level computer vision problems such as image classification, and low-level vision problems such as image denoising.

TargetDrop: A Targeted Regularization Method for Convolutional Neural Networks

Attentiondrop For Convolutional Neural Networks

Wordreg: Mitigating the Gap Between Training and Inference with Worst-Case Drop Regularization

FocusedDropout for Convolutional Neural Network

Correlation-based Structural Dropout for Convolutional Neural Networks

Effective and Efficient Dropout for Deep Convolutional Neural Networks

R-Drop: Regularized Dropout for Neural Networks.

DropConnect Regularization Method with Sparsity Constraint for Neural Networks

DropAttention: A Regularization Method for Fully-Connected Self-Attention Networks

CamDrop

LGST-Drop: Label-Guided Structural Dropout for Spatial–temporal Convolutional Neural Networks

Weighted Channel Dropout for Regularization of Deep Convolutional Neural Network

Regularizing neural networks with adaptive local drop

LocalDrop: A Hybrid Regularization for Deep Neural Networks

AutoDropout: Learning Dropout Patterns to Regularize Deep Networks

DropMask: A data augmentation method for convolutional networks

The Overfocusing Bias of Convolutional Neural Networks: A Saliency-Guided Regularization Approach

Dropout Reduces Underfitting

CLDM: Convolutional Layer Dropout Module.

Data Dropout: Optimizing Training Data for Convolutional Neural Networks

On Dropping Clusters to Regularize Graph Convolutional Neural Networks