Abstract:Self-supervised learning approach like contrastive learning is attached great attention in natural language processing. It uses pairs of training data augmentations to build a classification task for an encoder with well representation ability. However, the construction of learning pairs over contrastive learning is much harder in NLP tasks. Previous works generate word-level changes to form pairs, but small transforms may cause notable changes on the meaning of sentences as the discrete and sparse nature of natural language. In this paper, adversarial training is performed to generate challenging and harder learning adversarial examples over the embedding space of NLP as learning pairs. Using contrastive learning improves the generalization ability of adversarial training because contrastive loss can uniform the sample distribution. And at the same time, adversarial training also enhances the robustness of contrastive learning. Two novel frameworks, supervised contrastive adversarial learning (SCAL) and unsupervised SCAL (USCAL), are proposed, which yields learning pairs by utilizing the adversarial training for contrastive learning. The label-based loss of supervised tasks is exploited to generate adversarial examples while unsupervised tasks bring contrastive loss. To validate the effectiveness of the proposed framework, we employ it to Transformer-based models for natural language understanding, sentence semantic textual similarity and adversarial learning tasks. Experimental results on GLUE benchmark tasks show that our fine-tuned supervised method outperforms BERT$_{base}$ over 1.75\%. We also evaluate our unsupervised method on semantic textual similarity (STS) tasks, and our method gets 77.29\% with BERT$_{base}$. The robustness of our approach conducts state-of-the-art results under multiple adversarial datasets on NLI tasks.

SCAT: Robust Self-supervised Contrastive Learning via Adversarial Training for Text Classification

Adversarial Supervised Contrastive Learning

Contrastive adversarial learning in text classification tasks

Self-Supervised Contrastive Learning with Adversarial Perturbations for Defending Word Substitution-based Attacks

Simple Contrastive Representation Adversarial Learning for NLP Tasks

Introducing Adaptive Continuous Adversarial Training (ACAT) to Enhance ML Robustness

Combating Word-level Adversarial Text with Robust Adversarial Training.

ValCAT: Variable-Length Contextualized Adversarial Transformations Using Encoder-Decoder Language Model

CAT: Collaborative Adversarial Training.

CAT:Collaborative Adversarial Training

Contrastive Adversarial Training for Unsupervised Domain Adaptation

Improving Gradient-based Adversarial Training for Text Classification by Contrastive Learning and Auto-Encoder.

CAT-Gen: Improving Robustness in NLP Models via Controlled Adversarial Text Generation

Robust Pre-Training by Adversarial Contrastive Learning

Strongly Concealed Adversarial Attack Against Text Classification Models with Limited Queries

Counterfactual Contrastive Learning for Robust Text Classification

LST2A: Lexical-Syntactic Targeted Adversarial Attack for Texts

CAT: Customized Adversarial Training for Improved Robustness

CRAT-XML: Contrastive Representation Adversarial Training for Extremely Multi-Label Text Classification

Contrastive learning with text augmentation for text classification

Rethinking the Effect of Data Augmentation in Adversarial Contrastive Learning