Simple-Sampling and Hard-Mixup with Prototypes to Rebalance Contrastive Learning for Text Classification

Mengyu Li,Yonghao Liu,Fausto Giunchiglia,Xiaoyue Feng,Renchu Guan

2024-05-19

Abstract:Text classification is a crucial and fundamental task in natural language processing. Compared with the previous learning paradigm of pre-training and fine-tuning by cross entropy loss, the recently proposed supervised contrastive learning approach has received tremendous attention due to its powerful feature learning capability and robustness. Although several studies have incorporated this technique for text classification, some limitations remain. First, many text datasets are imbalanced, and the learning mechanism of supervised contrastive learning is sensitive to data imbalance, which may harm the model performance. Moreover, these models leverage separate classification branch with cross entropy and supervised contrastive learning branch without explicit mutual guidance. To this end, we propose a novel model named SharpReCL for imbalanced text classification tasks. First, we obtain the prototype vector of each class in the balanced classification branch to act as a representation of each class. Then, by further explicitly leveraging the prototype vectors, we construct a proper and sufficient target sample set with the same size for each class to perform the supervised contrastive learning procedure. The empirical results show the effectiveness of our model, which even outperforms popular large language models across several datasets.

Computation and Language

What problem does this paper attempt to address?

This paper focuses on solving the issues of data imbalance and utilizing Supervised Contrastive Learning (SCL) in text classification tasks. The traditional cross-entropy loss function may lead to performance degradation when dealing with imbalanced data, while SCL is sensitive to data imbalance despite its excellent feature learning performance. To address this, the paper proposes a new model called SharpReCL, specifically designed for handling imbalanced text classification tasks.

The SharpReCL model improves SCL in the following ways:
1. It calculates prototype vectors for each class as class representatives to balance the classification branch.
2. It constructs a suitably-sized target sample set using these prototype vectors for supervised contrastive learning.
3. It uses simple sampling and Hard-Mixup techniques to supplement prototype vectors, ensuring that each class is sampled at least once while increasing the diversity of contrast pairs, particularly for addressing the issue of insufficient samples in minority classes.
4. The two learning branches (classification branch and contrastive learning branch) interact and guide each other through prototype vectors, enhancing model training.

Experimental results demonstrate that the SharpReCL model outperforms other methods, including large-scale language models, on multiple imbalanced datasets, validating its effectiveness in handling imbalanced text data.