Abstract: Recently, prompt tuning \cite{lester2021power} has gradually become a new paradigm for NLP, which only depends on the representation of the words by freezing the parameters of pre-trained language models (PLMs) to obtain remarkable performance on downstream tasks. It maintains the consistency of Masked Language Model (MLM) \cite{devlin2018bert} task in the process of pre-training, and avoids some issues that may happened during fine-tuning. Naturally, we consider that the "[MASK]" tokens carry more useful information than other tokens because the model combines with context to predict the masked tokens. Among the current prompt tuning methods, there will be a serious problem of random composition of the answer tokens in prediction when they predict multiple words so that they have to map tokens to labels with the help verbalizer. In response to the above issue, we propose a new \textbf{Pro}mpt \textbf{Tu}ning based on "[\textbf{M}ASK]" (\textbf{Protum}) method in this paper, which constructs a classification task through the information carried by the hidden layer of "[MASK]" tokens and then predicts the labels directly rather than the answer tokens. At the same time, we explore how different hidden layers under "[MASK]" impact on our classification model on many different data sets. Finally, we find that our \textbf{Protum} can achieve much better performance than fine-tuning after continuous pre-training with less time consumption. Our model facilitates the practical application of large models in NLP.

Masked Siamese Prompt Tuning for Few-Shot Natural Language Understanding

PPT: Pre-trained Prompt Tuning for Few-shot Learning

GPT understands, too

Towards Unified Prompt Tuning for Few-shot Text Classification

Making Pre-trained Language Models End-to-end Few-shot Learners with Contrastive Prompt Tuning

Improving Prompt Tuning with Learned Prompting Layers

Protum: A New Method For Prompt Tuning Based on "[MASK]"

Multi-Task Pre-Training of Modular Prompt for Few-Shot Learning

P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks

Self-supervised Meta-Prompt Learning with Meta-Gradient Regularization for Few-shot Generalization

POUF: Prompt-oriented unsupervised fine-tuning for large pre-trained models

Multitask Pre-training of Modular Prompt for Chinese Few-Shot Learning

Hierarchical Prompt Tuning for Few-Shot Multi-Task Learning

Helping Language Models Learn More: Multi-dimensional Task Prompt for Few-shot Tuning

APrompt: Attention Prompt Tuning for Efficient Adaptation of Pre-trained Language Models

Scaled Prompt-Tuning for Few-Shot Natural Language Generation

Q-Tuning: Queue-based Prompt Tuning for Lifelong Few-shot Language Learning

MuDPT: Multi-modal Deep-symphysis Prompt Tuning for Large Pre-trained Vision-Language Models

An Exploration of Prompt Tuning on Generative Spoken Language Model for Speech Processing Tasks

Prompt Tuning with Soft Context Sharing for Vision-Language Models