Abstract:Large Language Models (LLMs) demonstrate impressive capabilities to generate accurate code snippets given natural language intents in zero-shot, i.e., without the need for specific fine-tuning. While prior studies have highlighted the advantages of fine-tuning LLMs, this process incurs high computational costs, making it impractical in resource-scarce environments, particularly for models with billions of parameters. To address these challenges, previous research explored In-Context Learning (ICL) as a strategy to guide the LLM generative process with task-specific prompt examples. However, ICL introduces inconveniences, such as the need for designing contextually relevant prompts and the absence of learning task-specific parameters, thereby limiting downstream task performance. In this context, we foresee Parameter-Efficient Fine-Tuning (PEFT) techniques as a promising approach to efficiently specialize LLMs to task-specific data while maintaining reasonable resource consumption. In this paper, we deliver a comprehensive study of PEFT techniques for LLMs under the automated code generation scenario. Our comprehensive investigation of PEFT techniques for LLMs reveals their superiority and potential over ICL across a diverse set of LLMs. Additionally, we demonstrate the extended capabilities of PEFT, showcasing its ability to learn from two distinct datasets jointly without compromising performance. Furthermore, our study highlights the potential for tuning larger LLMs and significant reductions in memory usage by combining PEFT with quantization. Therefore, this study opens opportunities for broader applications of PEFT in software engineering scenarios. Our code is available at <a class="link-external link-https" href="https://github.com/martin-wey/peft-llm-code/" rel="external noopener nofollow">this https URL</a>.

Train More Parameters But Mind Their Placement: Insights into Language Adaptation with PEFT

Non-Intrusive Adaptation: Input-Centric Parameter-efficient Fine-Tuning for Versatile Multimodal Modeling

Customizing Large Language Model Generation Style using Parameter-Efficient Finetuning

Challenges in Adapting Multilingual LLMs to Low-Resource Languages using LoRA PEFT Tuning

MAPLE: Multilingual Evaluation of Parameter Efficient Finetuning of Large Language Models

Unlocking Parameter-Efficient Fine-Tuning for Low-Resource Language Translation

Empirical Analysis of the Strengths and Weaknesses of PEFT Techniques for LLMs

An Empirical Study on Parameter-Efficient Fine-Tuning for MultiModal Large Language Models

Empirical Studies of Parameter Efficient Methods for Large Language Models of Code and Knowledge Transfer to R

Targeted Multilingual Adaptation for Low-resource Language Families

Parameter-efficient Tuning for Large Language Model Without Calculating Its Gradients

LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models

A Parameter-efficient Language Extension Framework for Multilingual ASR

Exploring Parameter-Efficient Fine-Tuning Techniques for Code Generation with Large Language Models

Parameter-Efficient Fine-Tuning of Large Language Models for Unit Test Generation: An Empirical Study

Arbitrary Few Parameters Are Good Enough for Adapting Large-scale Pre-trained Language Models

Investigating Multilingual Instruction-Tuning: Do Polyglot Models Demand for Multilingual Instructions?

PEMA: An Offsite-Tunable Plug-in External Memory Adaptation for Language Models

Large language models capsule: A research analysis of In-Context Learning (ICL) and Parameter-Efficient Fine-Tuning (PEFT) methods

Empirical Analysis of Efficient Fine-Tuning Methods for Large Pre-Trained Language Models

Parameter-Efficient Fine-Tuning With Adapters