Abstract:As powerful pre-trained vision-language models (VLMs) like CLIP gain prominence, numerous studies have attempted to combine VLMs for downstream tasks. Among these, prompt learning has been validated as an effective method for adapting to new tasks, which only requiring a small number of parameters. However, current prompt learning methods face two challenges: first, a single soft prompt struggles to capture the diverse styles and patterns within a dataset; second, fine-tuning soft prompts is prone to overfitting. To address these challenges, we propose a mixture of soft prompt learning method incorporating a routing module. This module is able to capture a dataset's varied styles and dynamically selects the most suitable prompts for each instance. Additionally, we introduce a novel gating mechanism to ensure the router selects prompts based on their similarity to hard prompt templates, which both retaining knowledge from hard prompts and improving selection accuracy. We also implement semantically grouped text-level supervision, initializing each soft prompt with the token embeddings of manually designed templates from its group and applied a contrastive loss between the resulted text feature and hard prompt encoded text feature. This supervision ensures that the text features derived from soft prompts remain close to those from their corresponding hard prompts, preserving initial knowledge and mitigating overfitting. Our method has been validated on 11 datasets, demonstrating evident improvements in few-shot learning, domain generalization, and base-to-new generalization scenarios compared to existing baselines. The code will be available at \url{https://anonymous.4open.science/r/mocoop-6387}

Mixture of Experts Meets Prompt-Based Continual Learning

Evolving Parameterized Prompt Memory for Continual Learning

Convolutional Prompting meets Language Models for Continual Learning

KC-Prompt: End-To-End Knowledge-Complementary Prompting for Rehearsal-Free Continual Learning

When Prompt-based Incremental Learning Does Not Meet Strong Pretraining

Vector Quantization Prompting for Continual Learning

Generating Prompts in Latent Space for Rehearsal-free Continual Learning

Hierarchical Prompts for Rehearsal-free Continual Learning

S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning

Progressive Prompts: Continual Learning for Language Models

Consistent Prompting for Rehearsal-Free Continual Learning

Prompt Gradient Projection for Continual Learning.

Hierarchical Decomposition of Prompt-Based Continual Learning: Rethinking Obscured Sub-optimality

Unleashing the Power of Visual Prompting At the Pixel Level

PromptFusion: Decoupling Stability and Plasticity for Continual Learning

CP-Prompt: Composition-Based Cross-modal Prompting for Domain-Incremental Continual Learning

Leveraging Hierarchical Taxonomies in Prompt-based Continual Learning

Mixture of Prompt Learning for Vision Language Models

INCPrompt: Task-Aware incremental Prompting for Rehearsal-Free Class-incremental Learning