Abstract:Supervised learning can be viewed as distilling relevant information from input data into feature representations. This process becomes difficult when supervision is noisy as the distilled information might not be relevant. In fact, recent research shows that networks can easily overfit all labels including those that are corrupted, and hence can hardly generalize to clean datasets. In this article, we focus on the problem of learning with noisy labels and introduce compression inductive bias to network architectures to alleviate this overfitting problem. More precisely, we revisit one classical regularization named Dropout and its variant Nested Dropout. Dropout can serve as a compression constraint for its feature dropping mechanism, while Nested Dropout further learns ordered feature representations with respect to feature importance. Moreover, the trained models with compression regularization are further combined with co-teaching for performance boost. Theoretically, we conduct bias variance decomposition of the objective function under compression regularization. We analyze it for both single model and co-teaching. This decomposition provides three insights: 1) it shows that overfitting is indeed an issue in learning with noisy labels; 2) through an information bottleneck formulation, it explains why the proposed feature compression helps in combating label noise; and 3) it gives explanations on the performance boost brought by incorporating compression regularization into co-teaching. Experiments show that our simple approach can have comparable or even better performance than the state-of-the-art methods on benchmarks with real-world label noise including Clothing1M and ANIMAL-10N. Our implementation is available at https://yingyichen-cyy.github.io/ CompressFeatNoisyLabels/.

CJC-net: A cyclical training method with joint loss and co-teaching strategy net for deep learning under noisy labels

Noise is the Fatal Poison: A Noise-aware Network for Noisy Dataset Classification

FGCM: Noisy Label Learning via Fine-Grained Confidence Modeling

Gradient Agreement Hinders the Memorization of Noisy Labels

Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels

An joint end-to-end framework for learning with noisy labels

Co-LDL: A Co-Training-Based Label Distribution Learning Method for Tackling Label Noise

Iterative Learning with Open-set Noisy Labels

Learning with Noisy Labels Via Self-supervised Adversarial Noisy Masking

Boosting Co-teaching with Compression Regularization for Label Noise

Cooperative Learning for Noisy Supervision.

Mutual calibration training: Training deep neural networks with noisy labels using dual-models

Synergistic Network Learning and Label Correction for Noise-robust Image Classification

Joint Optimization Framework for Learning with Noisy Labels

Cross-to-merge training with class balance strategy for learning with noisy labels

Agreement or Disagreement in Noise-tolerant Mutual Learning?

Co-learning: Learning from Noisy Labels with Self-supervision

Compressing Features for Learning With Noisy Labels

Iterative Cross Learning on Noisy Labels

Coresets for Robust Training of Neural Networks against Noisy Labels

Asymmetric Co-teaching with Multi-view Consensus for Noisy Label Learning