Abstract:Supervised learning can be viewed as distilling relevant information from input data into feature representations. This process becomes difficult when supervision is noisy as the distilled information might not be relevant. In fact, recent research shows that networks can easily overfit all labels including those that are corrupted, and hence can hardly generalize to clean datasets. In this article, we focus on the problem of learning with noisy labels and introduce compression inductive bias to network architectures to alleviate this overfitting problem. More precisely, we revisit one classical regularization named Dropout and its variant Nested Dropout. Dropout can serve as a compression constraint for its feature dropping mechanism, while Nested Dropout further learns ordered feature representations with respect to feature importance. Moreover, the trained models with compression regularization are further combined with co-teaching for performance boost. Theoretically, we conduct bias variance decomposition of the objective function under compression regularization. We analyze it for both single model and co-teaching. This decomposition provides three insights: 1) it shows that overfitting is indeed an issue in learning with noisy labels; 2) through an information bottleneck formulation, it explains why the proposed feature compression helps in combating label noise; and 3) it gives explanations on the performance boost brought by incorporating compression regularization into co-teaching. Experiments show that our simple approach can have comparable or even better performance than the state-of-the-art methods on benchmarks with real-world label noise including Clothing1M and ANIMAL-10N. Our implementation is available at https://yingyichen-cyy.github.io/ CompressFeatNoisyLabels/.

Combating Noisy Labels by Agreement: A Joint Training Method with Co-Regularization

Combating noisy labels by agreement: A joint training method with co-regularization

Co-teaching + : Towards Training of Robust Deep Networks with Noisy Labels

Consensus Co-teaching for Dynamically Learning with Noisy Labels

Gradient Agreement Hinders the Memorization of Noisy Labels

Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels

Agreement or Disagreement in Noise-tolerant Mutual Learning?

Asymmetric Co-teaching with Multi-view Consensus for Noisy Label Learning

Enhancing Robustness in Learning with Noisy Labels: an Asymmetric Co-Training Approach

Co-learning: Learning from Noisy Labels with Self-supervision

How does Disagreement Help Generalization against Label Corruption?

Co-LDL: A Co-Training-Based Label Distribution Learning Method for Tackling Label Noise

CJC-net: A cyclical training method with joint loss and co-teaching strategy net for deep learning under noisy labels

Noise is Also Useful: Negative Correlation-Steered Latent Contrastive Learning

Boosting Co-teaching with Compression Regularization for Label Noise

Jo-SRC: A Contrastive Approach for Combating Noisy Labels

Learning with Neighbor Consistency for Noisy Labels

Cooperative Learning for Noisy Supervision.

Compressing Features for Learning With Noisy Labels

Synergistic Network Learning and Label Correction for Noise-robust Image Classification

Learning with Noisy Labels Via Self-supervised Adversarial Noisy Masking