Abstract:Recent studies have shown that prompt-based fine-tuning improves the performance of large Pre-trained Language Models (PLMs) for few-shot text classification. Specifically, this type of method transforms the text classification task into the inherent Masked Language Modeling (MLM) with some natural language prompts and verbalizers to bridge the gap between the pre-training stage and the fine-tuning stage. Yet, it is unclear how the prompting knowledge can be transferred across similar or distant NLP tasks, for the purpose of mutual reinforcement. Based on continuous prompt embeddings, we propose TransPrompt v2 , a novel transferable prompting framework for few-shot learning across similar or distant text classification tasks. For transferable prompt learning across similar tasks, we employ a multi-task meta-knowledge acquisition procedure to train a meta-learner that captures the cross-task transferable knowledge. For learning across distant tasks, we further introduce the task type descriptions and propose intra-type and inter-type prompt encoders to capture implicit mutual relations among multiple distant tasks. Additionally, two de-biasing techniques are further designed to make the trained meta-learner more task-agnostic and unbiased towards any tasks. After that, the meta-learner can be fine-tuned for specific tasks with better parameters initialization. Extensive experiments show that TransPrompt v2 outperforms single-task and cross-task strong baselines over multiple NLP tasks and datasets. We further show that the meta-learner can effectively improve the performance of PLMs on previously unseen tasks. In addition, TransPrompt v2 also outperforms strong fine-tuning baselines when learning with full training sets.

Hierarchical Prompt Tuning for Few-Shot Multi-Task Learning

Multi-Task Pre-Training of Modular Prompt for Few-Shot Learning

Multitask Pre-training of Modular Prompt for Chinese Few-Shot Learning

Towards Unified Prompt Tuning for Few-shot Text Classification

Helping Language Models Learn More: Multi-dimensional Task Prompt for Few-shot Tuning

PPT: Pre-trained Prompt Tuning for Few-shot Learning

Enhancing Few-Shot Transfer Learning with Optimized Multi-Task Prompt Tuning through Modular Prompt Composition

Hierarchical Prompt Learning for Multi-Task Learning

Unified Prompt Learning Makes Pre-Trained Language Models Better Few-Shot Learners

Prompt Tuning with Soft Context Sharing for Vision-Language Models

Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning

TransPrompt V2: Transferable Prompt-based Fine-tuning for Few-shot Text Classification

APrompt: Attention Prompt Tuning for Efficient Adaptation of Pre-trained Language Models

Ontology-enhanced Prompt-tuning for Few-shot Learning

TransPrompt - Towards an Automatic Transferable Prompting Framework for Few-shot Text Classification.

Instance-wise Prompt Tuning for Pretrained Language Models

Dynamic Prompting: A Unified Framework for Prompt Tuning

Self-supervised Meta-Prompt Learning with Meta-Gradient Regularization for Few-shot Generalization

Making Pre-trained Language Models End-to-end Few-shot Learners with Contrastive Prompt Tuning

Exploring Universal Intrinsic Task Subspace for Few-shot Learning Via Prompt Tuning