Abstract:State-of-the-art text classification models are dominated by deep neural networks, but they still struggle to the issue of poor generalization ability when using cross entropy loss for training. One of the reasons is the training samples are usually annotated with hard labels, ignoring the inter-label relationship among these labels. This results in the outputted distributions to violate the label-correlation. Although the widely used contrastive learning is able to learn highly-expressive feature representations that generalize well across the training and test sets, contrastive learning in the embedding space is difficult to model the label correlation and may still output unreasonable label distributions. In this paper, we suggest con-trastive learning using label distributions and present a novel label-level contrastive learning (LLCL) para-digm, which can constrain the unreasonable label distributions from model outputs. We hypothesize that the label distributions of the instances in the same class are more similar than those from other classes. We introduce two label-level contrastive learning losses, namely supervised contrastive learning and self-supervised contrastive learning. After adding our proposed losses to the cross-entropy loss as regu-larizer for the training text classification model, our model obtains the average improvement of 0.74% over the strong RoBERTa-Large baseline on ten datasets. Particularly, our contrastive learning in the label space can effectively capture label correlations better than that in the embedding space, achieving an improvement of 6.2% in the top-2 accuracy on the SST-5 dataset. We also demonstrate that our proposed method is more effective, especially in the text classification tasks with a large label space or limited labeled data. Last but not least, our model does not rely on any kind of specialized architectures, data aug-mentation methods, or additional unsupervised data.CO 2022 Elsevier B.V. All rights reserved.

Simple Flow-Based Contrastive Learning for BERT Sentence Representations

OssCSE: Overcoming Surface Structure Bias in Contrastive Learning for Unsupervised Sentence Embedding

On the Sentence Embeddings from Pre-trained Language Models

Contrastive Learning Models for Sentence Representations

ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer

Contrastive Learning in Distilled Models

A Sentence is Worth 128 Pseudo Tokens: A Semantic-Aware Contrastive Learning Framework for Sentence Embeddings

Ditto: A Simple and Efficient Approach to Improve Sentence Embeddings

A Contrastive Framework to Enhance Unsupervised Sentence Representation Learning

Advancing Semantic Textual Similarity Modeling: A Regression Framework with Translated ReLU and Smooth K2 Loss

Alleviating Over-smoothing for Unsupervised Sentence Representation.

Debiased Contrastive Learning of Unsupervised Sentence Representations

Distilling Semantic Concept Embeddings from Contrastively Fine-Tuned Language Models

Instance Smoothed Contrastive Learning for Unsupervised Sentence Embedding

CoT-BERT: Enhancing Unsupervised Sentence Representation through Chain-of-Thought

Unsupervised Sentence Embedding Model Based on Contrastive Learning

Simple Contrastive Representation Adversarial Learning for NLP Tasks

Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning

Contrastive Learning from Label Distribution: A Case Study on Text Classification

Differentiable Data Augmentation for Contrastive Sentence Representation Learning

Improving Contrastive Learning of Sentence Embeddings from AI Feedback