Abstract:Integrating supervised contrastive loss to cross entropy-based communication has recently been proposed as a solution to address the long-tail learning problem. However, when the class imbalance ratio is high, it requires adjusting the supervised contrastive loss to support the tail classes, as the conventional contrastive learning is biased towards head classes by default. To this end, we present Rebalanced Contrastive Learning (RCL), an efficient means to increase the long tail classification accuracy by addressing three main aspects: 1. Feature space balancedness - Equal division of the feature space among all the classes, 2. Intra-Class compactness - Reducing the distance between same-class embeddings, 3. Regularization - Enforcing larger margins for tail classes to reduce overfitting. RCL adopts class frequency-based SoftMax loss balancing to supervised contrastive learning loss and exploits scalar multiplied features fed to the contrastive learning loss to enforce compactness. We implement RCL on the Balanced Contrastive Learning (BCL) Framework, which has the SOTA performance. Our experiments on three benchmark datasets demonstrate the richness of the learnt embeddings and increased top-1 balanced accuracy RCL provides to the BCL framework. We further demonstrate that the performance of RCL as a standalone loss also achieves state-of-the-art level accuracy.

What problem does this paper attempt to address?

This paper attempts to solve the classification problem in long - tail distribution. Specifically, the author focuses on how to improve the classification accuracy of tail classes without sacrificing the performance of head classes in the case of class imbalance. The following are the main problems and solutions in the paper: ### Problems the paper attempts to solve: 1. **Class imbalance problem**: In many practical applications, such as pedestrian classification, fraud detection, and spam filtering, the number of samples of different classes in the dataset varies greatly, causing the model to tend to predict the majority class (head class) while ignoring the minority class (tail class). 2. **Challenges in feature representation learning**: Traditional contrastive learning methods perform poorly when dealing with long - tail distributions because they are default - biased towards head classes and cannot fully support the learning of tail classes. 3. **Insufficient generalization ability**: The distribution difference between the training set and the test set (from long - tail distribution to uniform or inverse long - tail distribution) will lead to a decline in the model's generalization ability. ### Proposed solutions: To solve the above problems, the author proposes the **Rebalanced Contrastive Learning (RCL)** method to optimize feature representation and classification performance through the following three main aspects: 1. **Feature space balance**: - Ensure that all classes occupy an equal proportion in the feature space. - By adjusting the supervised contrastive loss, make the contribution of each class to the optimization process more balanced. 2. **Intra - class compactness**: - Reduce the distance between samples of the same class, making samples of the same class gather more closely together. - By scaling the feature vectors and inputting them into the contrastive loss function, enhance the intra - class compactness. 3. **Regularization**: - Apply a larger margin to the tail classes to reduce over - fitting. - Use the SoftMax loss balancing method based on class frequency to ensure that the tail classes have sufficient regularization. ### Experimental results: The author conducted experiments on three benchmark datasets (CIFAR10 Lt, CIFAR100 Lt and ImageNet Lt) to verify the effectiveness of RCL. The experimental results show that RCL not only improves the overall classification accuracy, but also significantly improves the performance of the tail classes, especially in the case of extreme class imbalance. ### Summary: RCL effectively solves the classification problem in long - tail distribution by optimizing the balance of the feature space, enhancing the intra - class compactness and increasing the margin of the tail classes. The method outperforms the existing advanced methods on multiple benchmark datasets, showing its potential in dealing with class imbalance problems. ### Formula summary: - **Balanced SoftMax Function**: \[ \text{Balanced SoftMax Function} = -\log \left( \frac{n_y e^{f_y(x)}}{\sum_{i \in L} n_i e^{f_i(x)}} \right) \] - **Logit Adjusted Loss**: \[ \text{Logit Adjusted Loss} = -\log \left( \frac{e^{f_y(x) + \tau \log \pi_y}}{\sum_{i \in L} e^{f_i(x) + \tau \log \pi_i}} \right) \] - **Rebalanced Contrastive Loss (RCL)**: \[ L_{\text{RCL}} = -\frac{1}{|B_y|} \sum_{p \in B_y \setminus \{i\}} \log \left( \frac{n_y e^{\tilde{z}_i \cdot \tilde{z}_p}}{\sum_

Long-Tail Learning with Rebalanced Contrastive Loss

Balanced Contrastive Learning for Long-Tailed Visual Recognition

Debiased Graph Contrastive Learning.

An Asymmetric Contrastive Loss for Handling Imbalanced Datasets

Decoupled Contrastive Learning for Long-Tailed Recognition

SMCL: Saliency Masked Contrastive Learning for Long-tailed Recognition

Self-Damaging Contrastive Learning

Bt-Vmf Contrastive and Collaborative Learning for Long-Tailed Visual Recognition

Continuous Contrastive Learning for Long-Tailed Semi-Supervised Recognition

A dual-branch model with inter- and intra-branch contrastive loss for long-tailed recognition

Simple-Sampling and Hard-Mixup with Prototypes to Rebalance Contrastive Learning for Text Classification

Contrastive Learning with Boosted Memorization.

CLAF: Contrastive Learning with Augmented Features for Imbalanced Semi-Supervised Learning

ECL: Class-Enhancement Contrastive Learning for Long-tailed Skin Lesion Classification

Learning the Unlearned: Mitigating Feature Suppression in Contrastive Learning

Self Contrastive Learning for Session-based Recommendation

Contrastive Attraction and Contrastive Repulsion for Representation Learning

Adaptive Multi-head Contrastive Learning

BaCon: Boosting Imbalanced Semi-supervised Learning via Balanced Feature-Level Contrastive Learning

Exploring Contrastive Learning for Long-Tailed Multi-Label Text Classification

Balanced complement loss for long-tailed image classification