Abstract:Prompt tuning is a promising method to fine-tune a pre-trained language model without retraining its large-scale parameters. Instead, it attaches a soft prompt to the input text, whereby downstream tasks can be well adapted by merely learning the embeddings of prompt tokens. Nevertheless, existing methods still suffer from two challenges: (i) they are hard to balance accuracy and efficiency. A longer (shorter) soft prompt generally leads to a better(worse) accuracy but at the cost of more (less) training time. (ii)The performance may not be consistent when adapting to different downstream tasks. We attribute it to the same embedding space but responsible for different requirements of downstream tasks. To address these issues, we propose an Efficient Prompt Tuning method (EPT) by multi-space projection and prompt fusion. Specifically, it decomposes a given soft prompt into a shorter prompt and two low-rank matrices, significantly reducing the training time. Accuracy is also enhanced by leveraging low-rank matrices and the short prompt as additional knowledge sources to enrich the semantics of the original short prompt. In addition, we project the soft prompt into multiple subspaces to improve the performance consistency, and then adaptively learn the combination weights of different spaces through a gating network. Experiments on 13 natural language processing downstream tasks show that our method significantly and consistently outperforms 11 comparison methods with the relative percentage of improvements up to 12.9%, and training time decreased by 14%.

STT: Soft Template Tuning for Few-Shot Adaptation

PPT: Pre-trained Prompt Tuning for Few-shot Learning

SPT: Learning to Selectively Insert Prompts for Better Prompt Tuning

Iterative Soft Prompt-Tuning for Unsupervised Domain Adaptation

APrompt: Attention Prompt Tuning for Efficient Adaptation of Pre-trained Language Models

Towards Unified Prompt Tuning for Few-shot Text Classification

Hard Sample Aware Prompt-Tuning

Prompt Tuning with Soft Context Sharing for Vision-Language Models

Self-supervised Meta-Prompt Learning with Meta-Gradient Regularization for Few-shot Generalization

Generating Synthetic Datasets for Few-shot Prompt Tuning

Improving Prompt Tuning with Learned Prompting Layers

Short text classification with Soft Knowledgeable Prompt-tuning

Adaptive Prompt Tuning: Vision Guided Prompt Tuning with Cross-Attention for Fine-Grained Few-Shot Learning

Multitask Pre-training of Modular Prompt for Chinese Few-Shot Learning

Multi-Task Pre-Training of Modular Prompt for Few-Shot Learning

Exploring Soft Prompt Initialization Strategy for Few-Shot Continual Text Classification

Efficient Prompt Tuning by Multi-Space Projection and Prompt Fusion

Structured Prompt Tuning

StablePT: Towards Stable Prompting for Few-shot Learning via Input Separation

STPrompt: Semantic-guided and Task-driven prompts for Effective Few-shot Classification

Hierarchical Prompt Tuning for Few-Shot Multi-Task Learning