Abstract:This work addresses the challenge of achieving zero-shot adversarial robustness while preserving zero-shot generalization in large-scale foundation models, with a focus on the popular Contrastive Language-Image Pre-training (CLIP). Although foundation models were reported to have exceptional zero-shot generalization, they are highly vulnerable to adversarial perturbations. Existing methods achieve a comparable good tradeoff between zero-shot adversarial robustness and generalization under small adversarial perturbations. However, they fail to achieve a good tradeoff under large adversarial perturbations. To this end, we propose a novel Text-Image Mutual Awareness (TIMA) method that strikes a balance between zero-shot adversarial robustness and generalization. More precisely, we propose an Image-Aware Text (IAT) tuning mechanism that increases the inter-class distance of text embeddings by incorporating the Minimum Hyperspherical Energy (MHE). Simultaneously, fixed pre-trained image embeddings are used as cross-modal auxiliary supervision to maintain the similarity between the MHE-tuned and original text embeddings by the knowledge distillation, preserving semantic information between different classes. Besides, we introduce a Text-Aware Image (TAI) tuning mechanism, which increases inter-class distance between image embeddings during the training stage by Text-distance based Adaptive Margin (TAM). Similarly, a knowledge distillation is utilized to retain the similarity between fine-tuned and pre-trained image embeddings. Extensive experimental results demonstrate the effectiveness of our approach, showing impressive zero-shot performance against a wide range of adversarial perturbations while preserving the zero-shot generalization capabilities of the original CLIP model.

Text and Image Are Mutually Beneficial: Enhancing Training-Free Few-Shot Classification with CLIP

TIMA: Text-Image Mutual Awareness for Balancing Zero-Shot Adversarial Robustness and Generalization Ability

ProtoCLIP: Prototypical Contrastive Language Image Pretraining

DiffCLIP: Few-shot Language-driven Multimodal Classifier

Iclip: Bridging Image Classification and Contrastive Language-Image Pre-Training for Visual Recognition

Distinguishing Textual Prompt Importance: Image-Guided Text Weighting for CLIP-Based Few-shot Learning

Adversarial Domain Adaptation with CLIP for Few-Shot Image Classification

Improving CLIP Training with Language Rewrites

VT-CLIP: Enhancing Vision-Language Models with Visual-guided Texts

Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification

SoftCLIP: Softer Cross-modal Alignment Makes CLIP Stronger.

CLIP-PING: Boosting Lightweight Vision-Language Models with Proximus Intrinsic Neighbors Guidance

Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm

OT-CLIP: Understanding and Generalizing CLIP Via Optimal Transport

TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives

Democratizing Contrastive Language-Image Pre-training: A CLIP Benchmark of Data, Model, and Supervision

Text-Guided Mixup Towards Long-Tailed Image Categorization

Less is More: Removing Text-regions Improves CLIP Training Efficiency and Robustness

TextCLIP: Text-Guided Face Image Generation And Manipulation Without Adversarial Training