Abstract:Vision-language pre-training (VLP) models demonstrate impressive abilities in processing both images and text. However, they are vulnerable to multi-modal adversarial examples (AEs). Investigating the generation of high-transferability adversarial examples is crucial for uncovering VLP models' vulnerabilities in practical scenarios. Recent works have indicated that leveraging data augmentation and image-text modal interactions can enhance the transferability of adversarial examples for VLP models significantly. However, they do not consider the optimal alignment problem between dataaugmented image-text pairs. This oversight leads to adversarial examples that are overly tailored to the source model, thus limiting improvements in transferability. In our research, we first explore the interplay between image sets produced through data augmentation and their corresponding text sets. We find that augmented image samples can align optimally with certain texts while exhibiting less relevance to others. Motivated by this, we propose an Optimal Transport-based Adversarial Attack, dubbed OT-Attack. The proposed method formulates the features of image and text sets as two distinct distributions and employs optimal transport theory to determine the most efficient mapping between them. This optimal mapping informs our generation of adversarial examples to effectively counteract the overfitting issues. Extensive experiments across various network architectures and datasets in image-text matching tasks reveal that our OT-Attack outperforms existing state-of-the-art methods in terms of adversarial transferability.

Vocabulary Learning Via Optimal Transport for Neural Machine Translation

Mutual Information and Diverse Decoding Improve Neural Machine Translation.

Vocabulary Selection Strategies for Neural Machine Translation

Finding the Optimal Vocabulary Size for Neural Machine Translation

On Using Very Large Target Vocabulary for Neural Machine Translation

Vocabulary Manipulation for Neural Machine Translation

Finding Better Subword Segmentation for Neural Machine Translation

Entropy-Based Vocabulary Substitution for Incremental Learning in Multilingual Neural Machine Translation

OT-VP: Optimal Transport-guided Visual Prompting for Test-Time Adaptation

vONTSS: vMF based semi-supervised neural topic modeling with optimal transport

On the Off-Target Problem of Zero-Shot Multilingual Neural Machine Translation

A Systematic Analysis of Vocabulary and BPE Settings for Optimal Fine-tuning of NMT: A Case Study of In-domain Translation

Beyond Shared Vocabulary: Increasing Representational Word Similarities across Languages for Multilingual Machine Translation

OT-Attack: Enhancing Adversarial Transferability of Vision-Language Models via Optimal Transport Optimization

Augmenting Statistical Machine Translation with Subword Translation of Out-of-Vocabulary Words

Optimal Transport Posterior Alignment for Cross-lingual Semantic Parsing

Towards Continual Learning for Multilingual Machine Translation via Vocabulary Substitution

Massive Exploration of Neural Machine Translation Architectures

How Effective is Byte Pair Encoding for Out-Of-Vocabulary Words in Neural Machine Translation?

OT-net: a reusable neural optimal transport solver

Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Models