Abstract:State-of-the-art text classification models are dominated by deep neural networks, but they still struggle to the issue of poor generalization ability when using cross entropy loss for training. One of the reasons is the training samples are usually annotated with hard labels, ignoring the inter-label relationship among these labels. This results in the outputted distributions to violate the label-correlation. Although the widely used contrastive learning is able to learn highly-expressive feature representations that generalize well across the training and test sets, contrastive learning in the embedding space is difficult to model the label correlation and may still output unreasonable label distributions. In this paper, we suggest con-trastive learning using label distributions and present a novel label-level contrastive learning (LLCL) para-digm, which can constrain the unreasonable label distributions from model outputs. We hypothesize that the label distributions of the instances in the same class are more similar than those from other classes. We introduce two label-level contrastive learning losses, namely supervised contrastive learning and self-supervised contrastive learning. After adding our proposed losses to the cross-entropy loss as regu-larizer for the training text classification model, our model obtains the average improvement of 0.74% over the strong RoBERTa-Large baseline on ten datasets. Particularly, our contrastive learning in the label space can effectively capture label correlations better than that in the embedding space, achieving an improvement of 6.2% in the top-2 accuracy on the SST-5 dataset. We also demonstrate that our proposed method is more effective, especially in the text classification tasks with a large label space or limited labeled data. Last but not least, our model does not rely on any kind of specialized architectures, data aug-mentation methods, or additional unsupervised data.CO 2022 Elsevier B.V. All rights reserved.

Supervised contrastive representation learning with tree-structured parzen estimator Bayesian optimization for imbalanced tabular data

OssCSE: Overcoming Surface Structure Bias in Contrastive Learning for Unsupervised Sentence Embedding

Towards Unsupervised Time Series Representation Learning: A Decomposition Perspective

Class-aware and Augmentation-free Contrastive Learning from Label Proportion

DABaCLT: A Data Augmentation Bias-Aware Contrastive Learning Framework for Time Series Representation

ReConTab: Regularized Contrastive Representation Learning for Tabular Data

Attention versus Contrastive Learning of Tabular Data -- A Data-centric Benchmarking

Balanced Contrastive Learning for Long-Tailed Visual Recognition

Parametric Augmentation for Time Series Contrastive Learning

PTab: Using the Pre-trained Language Model for Modeling Tabular Data

Tabular Data Contrastive Learning via Class-Conditioned and Feature-Correlation Based Augmentation

Probabilistic Contrastive Learning for Long-Tailed Visual Recognition

SwitchTab: Switched Autoencoders Are Effective Tabular Learners

Probabilistic Contrastive Learning for Domain Adaptation

Contrastive Learning from Label Distribution: A Case Study on Text Classification

Improving Event Temporal Relation Classification via Auxiliary Label-Aware Contrastive Learning.

Conditional Supervised Contrastive Learning for Fair Text Classification

Enhancing Aspect-Based Sentiment Analysis with Supervised Contrastive Learning

TuneTables: Context Optimization for Scalable Prior-Data Fitted Networks

Debiased Contrastive Learning of Unsupervised Sentence Representations

An intra-class distribution-focused generative adversarial network approach for imbalanced tabular data learning