Abstract:Prompt learning stands out as one of the most efficient approaches for adapting powerful vision-language foundational models like CLIP to downstream datasets by tuning learnable prompt vectors with very few samples. However, despite its success in achieving remarkable performance on in-domain data, prompt learning still faces the significant challenge of effectively generalizing to novel classes and domains. Some existing methods address this concern by dynamically generating distinct prompts for different domains. Yet, they overlook the inherent potential of prompts to generalize across unseen domains. To address these limitations, our study introduces an innovative prompt learning paradigm, called MetaPrompt, aiming to directly learn domain invariant prompt in few-shot scenarios. To facilitate learning prompts for image and text inputs independently, we present a dual-modality prompt tuning network comprising two pairs of coupled encoders. Our study centers on an alternate episodic training algorithm to enrich the generalization capacity of the learned prompts. In contrast to traditional episodic training algorithms, our approach incorporates both in-domain updates and domain-split updates in a batch-wise manner. For in-domain updates, we introduce a novel asymmetric contrastive learning paradigm, where representations from the pre-trained encoder assume supervision to regularize prompts from the prompted encoder. To enhance performance on out-of-domain distribution, we propose a domain-split optimization on visual prompts for cross-domain tasks or textual prompts for cross-class tasks during domain-split updates. Extensive experiments across 11 datasets for base-to-new generalization and 4 datasets for domain generalization exhibit favorable performance. Compared with the state-of-the-art method, MetaPrompt achieves an absolute gain of 1.02% on the overall harmonic mean in base-to-new generalization and consistently demonstrates superiority over all benchmarks in domain generalization.

Compositional Prompting for Anti-Forgetting in Domain Incremental Learning

CP-Prompt: Composition-Based Cross-modal Prompting for Domain-Incremental Continual Learning

S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning

Prompting to Prompt for Rehearsal-Free Class Incremental Learning.

INCPrompt: Task-Aware incremental Prompting for Rehearsal-Free Class-incremental Learning

Hierarchical Prompts for Rehearsal-free Continual Learning

Not Just Object, But State: Compositional Incremental Learning without Forgetting

Multi-level Prompt As Extra-Embedding for Incremental Learning

PromptFusion: Decoupling Stability and Plasticity for Continual Learning

Learning Domain Invariant Prompt for Vision-Language Models

KC-Prompt: End-To-End Knowledge-Complementary Prompting for Rehearsal-Free Continual Learning

Decorate the Newcomers: Visual Domain Prompt for Continual Test Time Adaptation

Federated Class-Incremental Learning with Prompting

Continual Learning for Remote Sensing Image Scene Classification With Prompt Learning

Domain-Controlled Prompt Learning

CAPrompt: Cyclic Prompt Aggregation for Pre-Trained Model Based Class Incremental Learning

Space-time Prompting for Video Class-incremental Learning

Hierarchical Decomposition of Prompt-Based Continual Learning: Rethinking Obscured Sub-optimality

When Prompt-based Incremental Learning Does Not Meet Strong Pretraining

Dynamic Prompt Allocation and Tuning for Continual Test-Time Adaptation