Abstract:Long-tailed semi-supervised learning poses a significant challenge in training models with limited labeled data exhibiting a long-tailed label distribution. Current state-of-the-art LTSSL approaches heavily rely on high-quality pseudo-labels for large-scale unlabeled data. However, these methods often neglect the impact of representations learned by the neural network and struggle with real-world unlabeled data, which typically follows a different distribution than labeled data. This paper introduces a novel probabilistic framework that unifies various recent proposals in long-tail learning. Our framework derives the class-balanced contrastive loss through Gaussian kernel density estimation. We introduce a continuous contrastive learning method, CCL, extending our framework to unlabeled data using reliable and smoothed pseudo-labels. By progressively estimating the underlying label distribution and optimizing its alignment with model predictions, we tackle the diverse distribution of unlabeled data in real-world scenarios. Extensive experiments across multiple datasets with varying unlabeled data distributions demonstrate that CCL consistently outperforms prior state-of-the-art methods, achieving over 4% improvement on the ImageNet-127 dataset. Our source code is available at <a class="link-external link-https" href="https://github.com/zhouzihao11/CCL" rel="external noopener nofollow">this https URL</a>

What problem does this paper attempt to address?

This paper attempts to solve two main problems in long - tailed semi - supervised learning (LTSSL): 1. **Imbalanced data distribution**: In practical applications, labeled data and unlabeled data usually present a long - tailed distribution, that is, a few classes have a large number of samples, while most classes have only a small number of samples. This imbalance leads to the deviation of pseudo - label generation, which is biased towards the majority class, thus affecting the model performance. 2. **Unlabeled data distribution difference**: Existing LTSSL methods often assume that the distribution of unlabeled data is the same as that of labeled data or rely on predefined anchor distributions to estimate the distribution of unlabeled data. However, in real - world scenarios, the distribution of unlabeled data may be different from that of labeled data, which further exacerbates the difficulty of model training. To solve these problems, the authors propose a new framework based on continuous contrastive learning (CCL). By introducing reliable and smooth pseudo - labels and combining the deep variational information bottleneck method from the perspective of information theory, this framework aims to improve the robustness of the model to imbalanced data and better adapt to the real distribution of unlabeled data. ### Specific solutions 1. **Unifying existing methods with a probability framework**: - The authors propose a probability framework, which derives class - balanced contrastive loss through Gaussian kernel density estimation and unifies several recently proposed long - tailed learning methods. For example, the popular class - balanced contrastive learning method can be regarded as a special case of this framework when using Gaussian kernels to approximate density. 2. **Continuous contrastive learning (CCL)**: - CCL extends the supervised contrastive loss to unlabeled data, uses "continuous pseudo - labels" obtained from model predictions and propagated labels to alleviate confirmation bias, and gradually estimates the label distribution of unlabeled data and adjusts model predictions to align with the estimated distribution. 3. **Experimental verification**: - The authors conducted extensive experiments on multiple LTSSL datasets with different unlabeled data distributions. The results show that CCL significantly outperforms previous state - of - the - art methods, especially achieving more than 4% improvement on the ImageNet - 127 dataset. ### Key contributions 1. Proposed a probability framework that unifies many recent long - tailed learning proposals. 2. Introduced a continuous contrastive learning method based on reliable and smooth pseudo - labels, which solves the confirmation bias and improves the quality of representation learning. 3. Conducted extensive experiments on multiple LTSSL datasets, demonstrating the effectiveness of the proposed method. Through these innovations, CCL can handle long - tailed distributed data more effectively and maintain good performance when the unlabeled data distributions are different.

Continuous Contrastive Learning for Long-Tailed Semi-Supervised Recognition

Decoupled Contrastive Learning for Long-Tailed Recognition

Balanced Contrastive Learning for Long-Tailed Visual Recognition

Contrastive Learning with Hallucinating Data for Long-Tailed Face Recognition.

Can Continual Learning Improve Long-Tailed Recognition? Toward a Unified Framework

Targeted Supervised Contrastive Learning for Long-Tailed Recognition

Bt-Vmf Contrastive and Collaborative Learning for Long-Tailed Visual Recognition

SMCL: Saliency Masked Contrastive Learning for Long-tailed Recognition

A dual-branch model with inter- and intra-branch contrastive loss for long-tailed recognition

Exploring Contrastive Learning for Long-Tailed Multi-Label Text Classification

Class-Aware Contrastive Semi-Supervised Learning

Label Anchored Contrastive Learning for Language Understanding

Towards Realistic Long-Tailed Semi-Supervised Learning: Consistency is All You Need

Long-Tailed Recognition by Mutual Information Maximization between Latent Features and Ground-Truth Labels

NCL++: Nested Collaborative Learning for long-tailed visual recognition

Three Heads Are Better Than One: Complementary Experts for Long-Tailed Semi-supervised Learning

Contrastive Attraction and Contrastive Repulsion for Representation Learning

A Simple Long-Tailed Recognition Baseline via Vision-Language Model

Interpolation-Based Contrastive Learning for Few-Label Semi-Supervised Learning

Probabilistic Contrastive Learning for Long-Tailed Visual Recognition