Efficient Prompt Tuning by Multi-Space Projection and Prompt Fusion

Pengxiang Lan,Enneng Yang,Yuting Liu,Guibing Guo,Linying Jiang,Jianzhe Zhao,Xingwei Wang
2024-07-01
Abstract:Prompt tuning is a promising method to fine-tune a pre-trained language model without retraining its large-scale parameters. Instead, it attaches a soft prompt to the input text, whereby downstream tasks can be well adapted by merely learning the embeddings of prompt tokens. Nevertheless, existing methods still suffer from two challenges: (i) they are hard to balance accuracy and efficiency. A longer (shorter) soft prompt generally leads to a better(worse) accuracy but at the cost of more (less) training time. (ii)The performance may not be consistent when adapting to different downstream tasks. We attribute it to the same embedding space but responsible for different requirements of downstream tasks. To address these issues, we propose an Efficient Prompt Tuning method (EPT) by multi-space projection and prompt fusion. Specifically, it decomposes a given soft prompt into a shorter prompt and two low-rank matrices, significantly reducing the training time. Accuracy is also enhanced by leveraging low-rank matrices and the short prompt as additional knowledge sources to enrich the semantics of the original short prompt. In addition, we project the soft prompt into multiple subspaces to improve the performance consistency, and then adaptively learn the combination weights of different spaces through a gating network. Experiments on 13 natural language processing downstream tasks show that our method significantly and consistently outperforms 11 comparison methods with the relative percentage of improvements up to 12.9%, and training time decreased by 14%.
Computation and Language,Artificial Intelligence,Machine Learning
What problem does this paper attempt to address?
### The Problems Addressed by This Paper This paper primarily addresses the following two issues: 1. **Trade-off between Accuracy and Efficiency**: - Existing prompt tuning methods struggle to find a balance between accuracy and efficiency. Longer (shorter) soft prompts usually result in better (worse) accuracy but increase (decrease) training time. 2. **Performance Inconsistency across Different Downstream Tasks**: - When adapting to different downstream tasks, the performance of existing methods can be unstable. The reason is that these methods attempt to satisfy the needs of different downstream tasks using the same embedding space, while the text information in natural language processing involves different types and levels of difficulty, and the model's attention to semantics in short prompts is limited. To address these issues, the authors propose a method called **Efficient Prompt Tuning (EPT)**, which improves the aforementioned problems through multi-space projection and prompt fusion techniques. Specifically, EPT decomposes soft prompts into shorter prompts and two low-rank matrices, significantly reducing training time, and enhances adaptability to different downstream tasks through a multi-space projection module. Experimental results show that EPT significantly outperforms 11 other comparison methods on multiple natural language processing tasks, with performance improvements of up to 12.9% on some benchmarks, while reducing training time by 14%.