Abstract:Pre-trained code models (e.g. CodeBERT and CodeT5) have demonstrated their code intelligence in various software engineering tasks, such as code summarization. And full fine-tuning has become the typical approach to adapting these models to downstream tasks. However, full fine-tuning these large models can be computationally expensive and memory-intensive, particularly when training for multiple tasks. To alleviate this issue, several parameter-efficient fine-tuning methods (e.g. Adapter and LoRA) have been proposed to only train a small number of additional parameters, while keeping the original pre-trained parameters frozen. Although these methods claim superiority over the prior techniques, they seldom make a comprehensive and fair comparison on multiple software engineering tasks. Moreover, besides their potential in reducing fine-tuning costs and maintaining approximate performance, the effectiveness of these methods in low-resource, cross-language, and cross-project scenarios is inadequately studied. To this end, we first conduct experiments by fine-tuning state-of-the-art code models with these methods on both code understanding tasks and code generation tasks. The results show that, by tuning only 0.5% additional parameters, these methods may achieve comparable or higher performance than full fine-tuning in code understanding tasks, but they may exhibit slightly weaker performance in code generation tasks. We also investigate the impact of these methods with varying numbers of training samples and find that, a considerable number of samples (e.g. 1000 for clone detection) may be required for them to approximate the performance of full fine-tuning. Our experimental results in cross-language and cross-project scenarios demonstrate that by freezing most pre-trained parameters and tuning only 0.5% additional parameters, these methods achieve consistent improvements in models' transfer learning ability in comparison to full fine-tuning. Our code and data are available at https://github.com/anonymous-ase23/ CodeModelParameterEfficientFinetuning.

Delving into Parameter-Efficient Fine-Tuning in Code Change Learning: an Empirical Study

A Comprehensive Evaluation of Parameter-Efficient Fine-Tuning on Method-Level Code Smell Detection

A Comprehensive Evaluation of Parameter-Efficient Fine-Tuning on Software Engineering Tasks

Parameter-Efficient Fine-Tuning of Large Language Models for Unit Test Generation: An Empirical Study

Exploring Parameter-Efficient Fine-Tuning Techniques for Code Generation with Large Language Models

Empirical Analysis of the Strengths and Weaknesses of PEFT Techniques for LLMs

See Further for Parameter Efficient Fine-tuning by Standing on the Shoulders of Decomposition

Parameter Efficient Fine Tuning: A Comprehensive Analysis Across Applications

An Empirical Study of Parameter-Efficient Fine-Tuning Methods for Pre-Trained Code Models.

Parameter Efficient Instruction Tuning: An Empirical Study

Exploring Parameter-Efficient Fine-Tuning of Large Language Model on Automated Program Repair

Non-Intrusive Adaptation: Input-Centric Parameter-efficient Fine-Tuning for Versatile Multimodal Modeling

Parameter-Efficient Fine-Tuning via Selective Discrete Cosine Transform

Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey

Unlocking Parameter-Efficient Fine-Tuning for Low-Resource Language Translation

Empirical Studies of Parameter Efficient Methods for Large Language Models of Code and Knowledge Transfer to R

An Empirical Study on Parameter-Efficient Fine-Tuning for MultiModal Large Language Models

Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologies

Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment

ADT: an Additive Delta-Tuning Approach for Parameter-Efficient Tuning in Pre-Trained Language Models

BIPEFT: Budget-Guided Iterative Search for Parameter Efficient Fine-Tuning of Large Pretrained Language Models