Abstract:Pre-trained models have been shown effective in many code intelligence tasks, such as automatic code summarization and defect prediction. These models are pre-trained on large-scale unlabeled corpus and then fine-tuned in downstream tasks. However, as the inputs to pre-training and downstream tasks are in different forms, it is hard to fully explore the knowledge of pre-trained models. Besides, the performance of fine-tuning strongly relies on the amount of downstream task data, while in practice, the data scarcity scenarios are common. Recent studies in the natural language processing (NLP) field show that prompt tuning, a new paradigm for tuning, alleviates the above issues and achieves promising results in various NLP tasks. In prompt tuning, the prompts inserted during tuning provide task-specific knowledge, which is especially beneficial for tasks with relatively scarce data. In this article, we empirically evaluate the usage and effect of prompt tuning in code intelligence tasks. We conduct prompt tuning on popular pre-trained models CodeBERT and CodeT5 and experiment with four code intelligence tasks including defect prediction, code search, code summarization, and code translation. Our experimental results show that prompt tuning consistently outperforms fine-tuning in all four tasks. In addition, prompt tuning shows great potential in low-resource scenarios, e.g., improving the BLEU scores of fine-tuning by more than 26% on average for code summarization. Our results suggest that instead of fine-tuning, we could adapt prompt tuning for code intelligence tasks to achieve better performance, especially when lacking task-specific data. We also discuss the implications for adapting prompt tuning in code intelligence tasks.

DePT: Decoupled Prompt Tuning

Dynamic Prompting: A Unified Framework for Prompt Tuning

DeCoOp: Robust Prompt Tuning with Out-of-Distribution Detection

P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks

Improving Prompt Tuning with Learned Prompting Layers

P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks

FPT: Improving Prompt Tuning Efficiency Via Progressive Training.

Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning

Prompt Tuning for Unified Multimodal Pretrained Models.

Prompt Tuning in Code Intelligence: An Experimental Evaluation

Efficient Federated Prompt Tuning for Black-box Large Pre-trained Models

No More Fine-Tuning? An Experimental Evaluation of Prompt Tuning in Code Intelligence

Bridging the Gap: Neural Collapse Inspired Prompt Tuning for Generalization under Class Imbalance

PanDa: Prompt Transfer Meets Knowledge Distillation for Efficient Model Adaptation

Dual Modality Prompt Tuning for Vision-Language Pre-Trained Model

Efficient Prompt Tuning by Multi-Space Projection and Prompt Fusion

Pro-tuning: Unified Prompt Tuning for Vision Tasks

Enhancing Few-Shot Transfer Learning with Optimized Multi-Task Prompt Tuning through Modular Prompt Composition

Multi-Task Pre-Training of Modular Prompt for Few-Shot Learning

BayesPrompt: Prompting Large-Scale Pre-Trained Language Models on Few-shot Inference via Debiased Domain Abstraction

MuDPT: Multi-modal Deep-symphysis Prompt Tuning for Large Pre-trained Vision-Language Models