Abstract:Self-supervised learning approach like contrastive learning is attached great attention in natural language processing. It uses pairs of training data augmentations to build a classification task for an encoder with well representation ability. However, the construction of learning pairs over contrastive learning is much harder in NLP tasks. Previous works generate word-level changes to form pairs, but small transforms may cause notable changes on the meaning of sentences as the discrete and sparse nature of natural language. In this paper, adversarial training is performed to generate challenging and harder learning adversarial examples over the embedding space of NLP as learning pairs. Using contrastive learning improves the generalization ability of adversarial training because contrastive loss can uniform the sample distribution. And at the same time, adversarial training also enhances the robustness of contrastive learning. Two novel frameworks, supervised contrastive adversarial learning (SCAL) and unsupervised SCAL (USCAL), are proposed, which yields learning pairs by utilizing the adversarial training for contrastive learning. The label-based loss of supervised tasks is exploited to generate adversarial examples while unsupervised tasks bring contrastive loss. To validate the effectiveness of the proposed framework, we employ it to Transformer-based models for natural language understanding, sentence semantic textual similarity and adversarial learning tasks. Experimental results on GLUE benchmark tasks show that our fine-tuned supervised method outperforms BERT$_{base}$ over 1.75\%. We also evaluate our unsupervised method on semantic textual similarity (STS) tasks, and our method gets 77.29\% with BERT$_{base}$. The robustness of our approach conducts state-of-the-art results under multiple adversarial datasets on NLI tasks.

Curricular Contrastive Regularization for Speech Enhancement with Self-Supervised Representations.

All Information is Necessary: Integrating Speech Positive and Negative Information by Contrastive Learning for Speech Enhancement

Time Domain Speech Enhancement Using Self-Attention-Based Subspace Projection

OssCSE: Overcoming Surface Structure Bias in Contrastive Learning for Unsupervised Sentence Embedding

Self-Supervised Learning from Contrastive Mixtures for Personalized Speech Enhancement

Semantic-Enhanced Supervised Contrastive Learning.

Speech SIMCLR: Combining Contrastive and Reconstruction Objective for Self-supervised Speech Representation Learning

Robust Data2vec: Noise-robust Speech Representation Learning for ASR by Combining Regression and Improved Contrastive Learning

Self-Damaging Contrastive Learning

Debiased Contrastive Learning of Unsupervised Sentence Representations

IMPROVING MULTIMODAL SPEECH ENHANCEMENT BY INCORPORATING SELF-SUPERVISED AND CURRICULUM LEARNING

Learning the Unlearned: Mitigating Feature Suppression in Contrastive Learning

CLESSR-VC: Contrastive learning enhanced self-supervised representations for one-shot voice conversion

Alleviating Over-smoothing for Unsupervised Sentence Representation.

Learning Speech Representation From Contrastive Token-Acoustic Pretraining

Contrastive sentence representation learning with adaptive false negative cancellation

Simple Contrastive Representation Adversarial Learning for NLP Tasks

Improving Noise Robustness of Contrastive Speech Representation Learning with Speech Reconstruction

Regularizing Contrastive Predictive Coding for Speech Applications

Boosting Speech Enhancement with Clean Self-Supervised Features Via Conditional Variational Autoencoders

Contrastive and Consistency Learning for Neural Noisy-Channel Model in Spoken Language Understanding