Abstract:Unsupervised feature learning has made great strides with contrastive learning based on instance discrimination and invariant mapping, as benchmarked on curated class-balanced datasets. However, natural data could be highly correlated and long-tail distributed. Natural between-instance similarity conflicts with the presumed instance distinction, causing unstable training and poor performance. Our idea is to discover and integrate between-instance similarity into contrastive learning, not directly by instance grouping, but by cross-level discrimination (CLD) between instances and local instance groups. While invariant mapping of each instance is imposed by attraction within its augmented views, between-instance similarity could emerge from common repulsion against instance groups. Our batch-wise and cross-view comparisons also greatly improve the positive/negative sample ratio of contrastive learning and achieve better invariant mapping. To effect both grouping and discrimination objectives, we impose them on features separately derived from a shared representation. In addition, we propose normalized projection heads and unsupervised hyper-parameter tuning for the first time. Our extensive experimentation demonstrates that CLD is a lean and powerful add-on to existing methods such as NPID, MoCo, InfoMin, and BYOL on highly correlated, long-tail, or balanced datasets. It not only achieves new state-of-the-art on self-supervision, semi-supervision, and transfer learning benchmarks, but also beats MoCo v2 and SimCLR on every reported performance attained with a much larger compute. CLD effectively brings unsupervised learning closer to natural data and real-world applications. Our code is publicly available at: <a class="link-external link-https" href="https://github.com/frank-xwang/CLD-UnsupervisedLearning" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the challenges encountered in unsupervised feature learning in natural data. Specifically, natural data may be highly correlated and have a long - tailed distribution, which is different from the current benchmark datasets where instances are clearly distinguishable and the classes are balanced. Therefore, traditional contrastive learning methods based on instance discrimination are unstable and perform poorly when dealing with natural data. The paper proposes a new method - Cross - Level Instance - Group Discrimination (CLD), aiming to discover and integrate the similarities between instances into contrastive learning, rather than just achieving this through instance grouping. CLD solves the problem of similarities between instances in natural data by performing cross - level discrimination between instances and local instance groups, thereby improving the stability and performance of feature learning. The main contributions of CLD include: 1. Extending unsupervised feature learning to natural data with high correlation and long - tailed distribution. 2. Proposing a method of cross - level discrimination between instances and local groups to discover and integrate the similarities between instances into contrastive learning. At the same time, a normalized projection head and unsupervised hyperparameter tuning are proposed. 3. Experiments show that adding CLD to existing methods has almost no additional overhead, but can significantly improve performance, reaching a new state - of - the - art level in all benchmarks, and surpassing every reported performance of MoCo v2 and SimCLR when using larger computing resources. In conclusion, through the introduction of CLD, this paper effectively solves the key problem of unsupervised feature learning in natural data, promoting the development of unsupervised learning towards closer to real - world applications.

Unsupervised Feature Learning by Cross-Level Instance-Group Discrimination

DCS: Debiased Contrastive Learning with Weak Supervision for Time Series Classification

Debiased Graph Contrastive Learning.

Learning the Unlearned: Mitigating Feature Suppression in Contrastive Learning

Prototypical Contrastive Learning of Unsupervised Representations

Unsupervised Cross-view Subspace Clustering via Adaptive Contrastive Learning

Learning the Relation between Similarity Loss and Clustering Loss in Self-Supervised Learning

Cross-domain Contrastive Learning for Unsupervised Domain Adaptation

DenseCL: A Simple Framework for Self-Supervised Dense Visual Pre-Training

Debiased Contrastive Learning of Unsupervised Sentence Representations

Dense Contrastive Learning for Self-Supervised Visual Pre-Training

Beyond Instance Discrimination: Relation-aware Contrastive Self-supervised Learning

NaCL: noise-robust cross-domain contrastive learning for unsupervised domain adaptation

L^2CL: Embarrassingly Simple Layer-to-Layer Contrastive Learning for Graph Collaborative Filtering

InfoUCL: Learning Informative Representations for Unsupervised Continual Learning

Continuous Contrastive Learning for Long-Tailed Semi-Supervised Recognition

Weakly-supervised Contrastive Learning for Unsupervised Object Discovery

Time-Series Contrastive Learning against False Negatives and Class Imbalance

Class-wise Contrastive Prototype Learning for Semi-Supervised Classification under Intersectional Class Mismatch

Enhancing Hyperspectral Image Classification: Leveraging Unsupervised Information With Guided Group Contrastive Learning

Task-oriented Contrastive Learning for Unsupervised Domain Adaptation.