CP-Prompt: Composition-Based Cross-modal Prompting for Domain-Incremental Continual Learning

Yu Feng,Zhen Tian,Yifan Zhu,Zongfu Han,Haoran Luo,Guangwei Zhang,Meina Song

2024-08-02

Abstract:The key challenge of cross-modal domain-incremental learning (DIL) is to enable the learning model to continuously learn from novel data with different feature distributions under the same task without forgetting old ones. However, existing top-performing methods still cause high forgetting rates, by lacking intra-domain knowledge extraction and inter-domain common prompting strategy. In this paper, we propose a simple yet effective framework, CP-Prompt, by training limited parameters to instruct a pre-trained model to learn new domains and avoid forgetting existing feature distributions. CP-Prompt captures intra-domain knowledge by compositionally inserting personalized prompts on multi-head self-attention layers and then learns the inter-domain knowledge with a common prompting strategy. CP-Prompt shows superiority compared with state-of-the-art baselines among three widely evaluated DIL tasks. The source code is available at <a class="link-external link-https" href="https://github.com/dannis97500/CP_Prompt" rel="external noopener nofollow">this https URL</a>.

Computation and Language,Artificial Intelligence,Machine Learning

What problem does this paper attempt to address?

The problem that this paper attempts to solve is in domain - incremental learning (DIL) in the cross - modal field, how to make the learning model continuously learn from new data with different feature distributions without forgetting old knowledge. Specifically, the paper focuses on how to avoid the model forgetting the previously learned knowledge when dealing with data in new domains, while effectively extracting knowledge in new domains. This involves two main challenges:

1. **How to balance general knowledge and personalized knowledge during the DIL process**: that is, how to maintain the memory of the learned domains while learning new domains, and at the same time enhance the personalized understanding of each specific domain.
2. **How to describe the influence of the domain context on the embedding tokens**: especially when using the Transformer architecture, how to guide the model to make full use of domain knowledge at different semantic levels to adapt to the change of feature distributions in the domain - incremental learning process.

To solve these problems, the authors propose a framework named CP - Prompt (Composition - Based Cross - modal Prompting for Domain - Incremental Continual Learning). By inserting personalized prompts into the pre - trained model, it guides the model to learn new domain data while avoiding forgetting existing knowledge. CP - Prompt adopts a dual - prompt strategy, namely shared general prompts and personalized prompts, which are respectively used to capture cross - domain general knowledge and domain - specific personalized knowledge. This method not only improves the performance of the model on new domains, but also significantly reduces the adjustment of the original model parameters, thereby improving the parameter efficiency and accuracy of the model.