Abstract:Language models (LMs) trained on vast quantities of unlabelled data have greatly advanced the field of natural language processing (NLP). In this study, we re-visit the widely accepted notion in NLP that continued pre-training LMs on task-related texts improves the performance of fine-tuning (FT) in downstream tasks. Through experiments on eight single-sentence tasks and eight sentence-pair tasks in both semi-supervised and fully-supervised settings, we find that conventional continued pre-training does not consistently provide benefits and can even be detrimental for sentence-pair tasks or when prompt-based FT is used. To tackle these issues, we propose Prompt-based Continued Pre-training (PCP), which combines the idea of instruction tuning with conventional continued pre-training. Our approach aims to improve the performance of prompt-based FT by presenting both task-related texts and prompt templates to LMs through unsupervised pre-training objectives before fine-tuning for the target task. Our empirical evaluations on 21 benchmarks demonstrate that the PCP consistently improves the performance of state-of-the-art prompt-based FT approaches (up to 20.1% absolute) in both semi-supervised and fully-supervised settings, even with only hundreds of unlabelled examples. Additionally, prompt-based FT with the PCP outperforms state-of-the-art semi-supervised approaches with greater simplicity, eliminating the need for an iterative process and extra data augmentation. Our further analysis explores the performance lower bound of the PCP and reveals that the advantages of PCP persist across different sizes of models and datasets.

Exploring Soft Prompt Initialization Strategy for Few-Shot Continual Text Classification

Ontology-enhanced Prompt-tuning for Few-shot Learning

Investigating Prompt Learning for Chinese Few-Shot Text Classification with Pre-Trained Language Models

Brain-Inspired Fast-and Slow-Update Prompt Tuning for Few-Shot Class-Incremental Learning

StablePT: Towards Stable Prompting for Few-shot Learning via Input Separation

Hierarchy-Aware Interactive Prompt Learning for Few-Shot Classification

STT: Soft Template Tuning for Few-Shot Adaptation

Automated Few-shot Classification with Instruction-Finetuned Language Models

Adaptive Prompt Tuning: Vision Guided Prompt Tuning with Cross-Attention for Fine-Grained Few-Shot Learning

Knowledge-Enhanced Prompt Learning for Few-Shot Text Classification

Towards Unified Prompt Tuning for Few-shot Text Classification

PPT: Pre-trained Prompt Tuning for Few-shot Learning

An Enhanced Few-Shot Text Classification Approach by Integrating Topic Modeling and Prompt-tuning

Enhanced Prompt Learning for Few-shot Text Classification Method

Visual Prompt Tuning for Few-Shot Text Classification.

Distinguishing Textual Prompt Importance: Image-Guided Text Weighting for CLIP-Based Few-shot Learning

Exploring Cross-Domain Few-Shot Classification via Frequency-Aware Prompting

Rethinking the Effect of Uninformative Class Name in Prompt Learning

STPrompt: Semantic-guided and Task-driven prompts for Effective Few-shot Classification

Q-Tuning: Queue-based Prompt Tuning for Lifelong Few-shot Language Learning

Don't Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learner