Abstract:Vision-language foundation models like CLIP have shown impressive zero-shot generalization, but finetuning on downstream datasets can cause overfitting and loss of its generalization ability on unseen domains. Although collecting additional data from new domains of interest is possible, this method is often impractical due to the challenges in obtaining annotated data. To address this, we propose a plug-and-play feature synthesis method called LDFS (Language-Guided Diverse Feature Synthesis) to synthesize new domain features and improve existing CLIP fine-tuning strategies. LDFS has three main contributions: 1) To synthesize novel domain features and promote diversity, we propose an instance-conditional feature augmentation strategy based on a text-guided feature augmentation loss. 2) To maintain feature quality after augmenting, we introduce a pairwise regularizer to preserve augmented feature coherence within the CLIP feature space. 3) We propose to use stochastic text feature augmentation to reduce the modality gap and further facilitate the process of text-guided feature synthesis. Extensive experiments show LDFS superiority in improving CLIP generalization ability on unseen domains without collecting data from those domains. The code will be made publicly available.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to improve the generalization ability of the model for unseen domains without collecting new - domain data when fine - tuning on vision - language foundation models (such as CLIP). Specifically, the paper points out that although models such as CLIP perform well in zero - shot scenarios, when adapting to downstream tasks through fine - tuning, it may lead to over - fitting, thereby reducing their generalization performance in unseen domains. In addition, collecting annotated data in new domains is often impractical because it involves high costs and difficulties in data acquisition. To address this challenge, the paper proposes a method named LDFS (Language - Guided Diverse Feature Synthesis), which enhances the generalization ability of CLIP through the following three main contributions: 1. **Promote feature diversity**: A text - guided instance - conditional feature enhancement strategy is proposed. By introducing an instance - conditional feature enhancement loss to synthesize new domain features, the diversity of features is promoted. 2. **Maintain feature quality**: A pairwise regularizer is introduced to maintain the consistency of enhanced features within the CLIP feature space, thereby maintaining the quality of features. 3. **Reduce modality gap**: Random text feature enhancement is used to reduce the modality gap between text and image, further promoting the text - guided feature synthesis process. Through these methods, LDFS can significantly improve the generalization ability of CLIP in unseen domains without relying on target - domain data. The paper verifies the effectiveness of LDFS through experiments on multiple challenging domain - transfer benchmark datasets, including PACS, Office - Home, DomainNet and NICO++.

Enhancing Vision-Language Models Generalization via Diversity-Driven Novel Feature Synthesis

Distilling Vision-Language Foundation Models: A Data-Free Approach via Prompt Diversification

Robust Fine-Tuning of Vision-Language Models for Domain Generalization

DFS: A Diverse Feature Synthesis Model for Generalized Zero-Shot Learning

SYNC-CLIP: Synthetic Data Make CLIP Generalize Better in Data-Limited Scenarios

Prompt as Free Lunch: Enhancing Diversity in Source-Free Cross-domain Few-shot Learning through Semantic-Guided Prompting

SynthesizRR: Generating Diverse Datasets with Retrieval Augmentation

UMFC: Unsupervised Multi-Domain Feature Calibration for Vision-Language Models

On the Diversity of Synthetic Data and its Impact on Training Large Language Models

Diversify, Don't Fine-Tune: Scaling Up Visual Recognition Training with Synthetic Images

Efficient and Long-Tailed Generalization for Pre-trained Vision-Language Model

PromptSync: Bridging Domain Gaps in Vision-Language Models through Class-Aware Prototype Alignment and Discrimination

Spurious Feature Eraser: Stabilizing Test-Time Adaptation for Vision-Language Foundation Model

In the Era of Prompt Learning with Vision-Language Models

Improving Zero-Shot Generalization for CLIP with Variational Adapter

A Sentence Speaks a Thousand Images: Domain Generalization through Distilling CLIP with Language Guidance

Interactive Data Synthesis for Systematic Vision Adaptation via LLMs-AIGCs Collaboration

Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation

Enhancing Zero-Shot Vision Models by Label-Free Prompt Distribution Learning and Bias Correcting

Refining Skewed Perceptions in Vision-Language Models through Visual Representations