Abstract:Domain adversarial training has shown its effective capability for finding domain invariant feature representations and been successfully adopted for various domain adaptation tasks. However, recent advances of large models (e.g., vision transformers) and emerging of complex adaptation scenarios (e.g., DomainNet) make adversarial training being easily biased towards source domain and hardly adapted to target domain. The reason is twofold: relying on large amount of labelled data from source domain for large model training and lacking of labelled data from target domain for fine-tuning. Existing approaches widely focused on either enhancing discriminator or improving the training stability for the backbone networks. Due to unbalanced competition between the feature extractor and the discriminator during the adversarial training, existing solutions fail to function well on complex datasets. To address this issue, we proposed a novel contrastive adversarial training (CAT) approach that leverages the labeled source domain samples to reinforce and regulate the feature generation for target domain. Typically, the regulation forces the target feature distribution being similar to the source feature distribution. CAT addressed three major challenges in adversarial learning: 1) ensure the feature distributions from two domains as indistinguishable as possible for the discriminator, resulting in a more robust domain-invariant feature generation; 2) encourage target samples moving closer to the source in the feature space, reducing the requirement for generalizing classifier trained on the labeled source domain to unlabeled target domain; 3) avoid directly aligning unpaired source and target samples within mini-batch. CAT can be easily plugged into existing models and exhibits significant performance improvements.

Adversarial momentum-contrastive pre-training

Adversarial Pretraining of Self-Supervised Deep Networks: Past, Present and Future

AdCo: Adversarial Contrast for Efficient Learning of Unsupervised Representations from Self-Trained Negative Adversaries

Robust Pre-Training by Adversarial Contrastive Learning

From Pretext to Purpose: Batch-Adaptive Self-Supervised Learning

Adversarial Supervised Contrastive Learning

Disentangled Contrastive Learning for Learning Robust Textual Representations

Better Representations via Adversarial Training in Pre-Training: A Theoretical Perspective

ProSampler: Improving Contrastive Learning by Better Mini-batch Sampling

Discovering Adversarial Examples with Momentum.

When Does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?

BatchSampler: Sampling Mini-Batches for Contrastive Learning in Vision, Language, and Graphs

Dense Contrastive Learning for Self-Supervised Visual Pre-Training

Concurrent Adversarial Learning for Large-Batch Training

Adversarial Dense Contrastive Learning for Semi-Supervised Semantic Segmentation

Boosting Adversarial Attacks with Momentum

Adversarial Robust Memory-Based Continual Learner

Establishing a stronger baseline for lightweight contrastive models

Contrastive Adversarial Training for Unsupervised Domain Adaptation

Fast Propagation is Better: Accelerating Single-Step Adversarial Training via Sampling Subnetworks

Resource and data efficient self supervised learning